The ICARTT (International Consortium for Atmospheric Research on Transport and Transformation) standards were developed to fulfill data management needs for the ICARTT campaign in 2004. The ICARTT file format is text-based and composed of a header with important data description information and the data section. The ICARTT format, built on the NASA Ames and GTE data formats, was created to facilitate data exchange and promote collaborations among the science teams for achieving the ICARTT campaign goals. Due to the success of the ICARTT campaign, the ICARTT file format was exposed to a broad range of airborne researchers and was adopted for use in many other field campaigns sponsored by NASA and other partner agencies. The ICARTT format standards became a NASA standard in 2010 and was amended in January 2017 providing many enhancements, including the requirement for variable standard names. Primarily designed for airborne field studies, ICARTT has been further utilized for ground-based studies. The ICARTT format can host metadata that is critical for proper use of the data, especially for in-situ measurements. However, the information that needs to be included is often in free text, meaning the information are human readable, but not machine interpretable. Furthermore, the amount and type of information provided can vary substantially between principal investigators and campaigns. To support interoperability and FAIR principles, further enhancements to the ICARTT standards are recommended. Possible recommendations include standardizing timestamps for easier data comparisons and analysis; potential use of controlled and consistent vocabulary for variable short name and certain common metadata elements; and providing guidance on variable measurement units and how they are reported.

Megan Buzanowicz↗

NASA NTRSDate not supplied

The Use of Atmospheric Composition Variable Standard Names in Airborne and Field Data Products

The number of variables measured during airborne field campaigns has increased more than tenfold over the last thirty years. With this increase in measurements, the complexity for distributed active archive centers (DAACs) to distribute the data and for data users to search for and find measurements of interest has also increased. Part of this complexity arises from the unique variable names in suborbital atmospheric composition field studies. With limited guidelines related to variable naming, variable names and structures can vary significantly, even for the same type of variable. It is common for instrument scientists to use their intended measurable quantity as the data variable name. This can make it difficult for users to locate and interact with a particular variable across multiple data sets. One effective solution to this problem, identified by the Earth Science Data System (ESDS) ICARTT Refresh Working Group [1], was to introduce variable standard names that can be used as tags for each data variable. This allows similar measurements (e.g., dew point) to be categorized and located across field campaigns, regardless of what variable name the instrument scientist has used. From this the atmospheric composition variable standard names were developed with the goal to use Findable, Accessible, Interoperable, and Reusable (FAIR) principles [2] and provide context for all users, while remaining connected to those in the subject area. These standard names have been successfully implemented in FIREX-AQ, CAMP 2EX, ACTIVATE, and DCOTSS field campaigns.

metadata↗

NASA NTRSDate not supplied

Improving “Domain-Relevant Metadata Requirements” for Supporting Open-Source Science Initiative

Implementation of the NASA Open-Source Science Initiative (OSSI) requires sharing of all relevant information to ensure “open reproducible science” [1]. However, there are several challenges in applying the OSSI to airborne field campaigns focused on atmospheric composition, which often involve a wide variety of in-situ measurements for trace gases, aerosol and cloud properties, meteorological parameters, and radiation fields. To ensure open reproducibility from airborne field campaigns, it is essential to obtain detailed measurement descriptions, which include the detection principle, sample procedure and treatment, and data processing and correction method. The challenge is that some information, e.g., sampling procedure and treatment, may be instrument-specific and campaign or platform-dependent. The data processing may also involve empirical corrections which may evolve over time. In addition, these details (especially operation- or campaign-specific ones) are often not given in journal publications. Given these issues, there is a need to leverage and improve the current “domain-relevant metadata requirements” to represent the measurement description in standardized metadata. These requirements can then facilitate systematic collection of measurement specific metadata and serve as a foundation to develop tools for making the information accessible and data more interoperable and usable or reusable. Here we show a review of existing metadata collections, use cases, and needs for new standards.

Sean Leavor↗

NASA NTRSDate not supplied

Identifying Improvements to Airborne and Field Data Stewardship from Data User and Producer Needs (Findings from the March 2022 Airborne and Field Data Workshop)

Workshop Purpose: to gather valuable feedback from data users and data producers in order to improve the acquisition and use of NASA’s airborne and field data. Workshop Goals: - Identify new ways to improve the discovery, access, and reuse of NASA’s airborne and field data - Intentionally include the airborne community in future planning to ensure the community needs are met - Understand how NASA can better help researchers realize the full value of airborne and field data - Enable Open Source Science and determine what this means for airborne and field data communities

Geoffrey Stano↗

NASA NTRSDate not supplied

ICARTT File Format Enhancements: Supporting FAIRness and Data Discovery of Suborbital Campaign Data

Suborbital campaigns aim to accomplish a wide variety of goals and can include a variety of platforms, instruments, and parameters measured. In 2004, the ICARTT (International Consortium for Atmospheric Research on Transport and Transformation) standards were developed to fulfill data management needs for the ICARTT campaign. The ICARTT file format is text-based and composed of a header with important data description information and the data section. Built on the NASA Ames and GTE data formats, the ICARTT format was created to facilitate data exchange and promote collaborations among the science teams for achieving the ICARTT campaign goals. Due to its success and adaptation for use in many other field campaigns, the ICARTT file format became a NASA standard in 2010 and was amended in January 2017. These changes provided many enhancements, including the requirement for variable standard names. Primarily designed for airborne field studies, ICARTT has been further utilized for ground-based studies. NASA has made a commitment to build an inclusive open science community over the next decade. Open-source science strives to make publicly funded scientific research transparent, inclusive, accessible, and reproducible. The ICARTT format can host metadata that is critical for proper use of the data, particularly for in-situ measurements, and can enhance data discovery and accessibility. However, the required fields are often free text, meaning that the information is human readable, but not machine interpretable. Furthermore, the amount and type of information provided can vary significantly between principal investigators and campaigns. To support FAIR principles and interoperability, enhancements to the ICARTT standards are recommended. Possible recommendations include potential use of controlled and consistent vocabulary for variable standard name and certain common metadata elements; standardizing timestamps for easier data comparisons and analysis; and providing guidance on variable measurement units and how they are reported. Enhancing ICARTT metadata can further streamline the process to make suborbital data more readily available to the data user and improve variable-level metadata. Providing more variable-level metadata can enhance data searching and discovery, supporting NASA’s Open-Source Science Initiative (OSSI).

Megan Buzanowicz↗

NASA NTRSDate not supplied

Additional Metadata Guidelines to Improve the Structure and Usability of HDF and NetCDF Files

The Hierarchical Data Format (HDF) and Network Common Data Form (NetCDF) are data file formats created to aid users in the creation or use of scientific data. These file formats are useful for handling large data volumes and hosting extensive metadata as global attributes or variables and are popular with the modeling community. HDF and NetCDF files are largely used with remote sensing data and have been used to support measurements from numerous campaigns, from satellite to aircraft or ground and mobile based measurements. The files from airborne field studies, however, vary greatly in terms of the file structure and the amount and content of metadata. Information relevant to the file that can be useful to the user such as the data producer, location where data was taken, variable descriptions, or information about the instrument might not be included in the file. This metadata might be present in another file in the dataset containing the same data using the International Consortium for Atmospheric Research on Transport and Transformation (ICARTT) format. Recently, the Aerosols, Clouds, and their Interactions for Earth System Models (MACIE) group started a grassroots effort to develop a set of requirements for the HDF and NetCDF files for field studies, aiming to make the data products more interoperable and usable. Particularly, these requirements seek to make the files more compliant to Climate and Forecast (CF) metadata conventions and to standardize the file structure and the global and variable attributes. These requirements would help to ensure that HDF and NetCDF files contain adequate metadata to better support their use for research, e.g., the modeling community, and to enhance the usability and interoperability of data for research communities at large. To be presented are the details of the MACIE requirements as well as examples of the implementation of these requirements for merge files and lidar observation data files.

Sean Leavor↗

NASA NTRSDate not supplied

Making Atmospheric Composition Observational Data Fair

Explore the source record for details and available documents.

FAIR↗

NASA NTRSDate not supplied

Atmospheric Composition Variable Standard Name Convention (ACVSNC)

Explore the source record for details and available documents.

Morgan Silverman↗

NASA NTRSDate not supplied

Facilitating Field Observations Based Atmospheric Composition Research via. SOOT

Explore the source record for details and available documents.

Gao Chen↗

NASA NTRSDate not supplied

Blast from the Past: ASDC Curation for NASA Suborbital Legacy Missions to Promote Data Discovery and Accessibility

NASA has an extensive history of conducting suborbital field campaigns to further advances in atmospheric sciences. Beginning with the Chemical Instrument Test and Evaluation (CITE) conducted in 1983-1984, NASA has completed many suborbital campaigns over the past three decades. Since the early 2010s, suborbital missions are typically assigned to a NASA Distributed Active Archive Center (DAAC) prior to the mission for long-term archival and distribution. Efforts are being made by NASA’s Earth Science Data and Information System (ESDIS) Project and the Airborne Data Management Group (ADMG) to assign legacy missions to DAACs for permanent archival and distribution, so that these valuable datasets remain to be available to the scientific community. NASA’s Atmospheric Science Data Center (ASDC) has been named the assigned DAAC for nearly 20 atmospheric composition legacy missions, including missions conducted as part of the Global Tropospheric Experiment (GTE) and expects to be named the assigned DAAC for more of these missions over the next few years. The primary goal of the ASDC is to provide access to the datasets as they are currently formatted to the broad user community and enhance their findability and accessibility. However, data reporting standards have evolved significantly since 1983 and the datasets span a wide variety of file formats, including text, Ames, GTE, and ICARTT (International Consortium for Atmospheric Research on Transport and Transformation), and the amount of metadata and relevant information included in the files also varies greatly and can not be readily extracted without subject matter knowledge. This has caused challenges for the ASDC’s suborbital metadata extraction pipeline in ensuring that accurate and necessary metadata is being provided for the missions by all the ASDC’s existing search mechanisms. To make the data more findable and accessible, the ASDC has begun researching ways to further enhance the datasets, including distributing value-added products (i.e. consistent file format such as ICARTT or netCDF), adding standard names from the ESDIS Standards Coordination Office (ESCO)-approved Atmospheric Composition Variable Standard Names Convention (ACVSNC), and creating outreach materials such as ArcGIS StoryMaps, User Guides, and Micro Articles, providing overviews of the missions and what type of data was collected during the missions. These efforts also help support NASA’s Open-Source Science by enhancing the FAIRness of the legacy data products. This presentation will review the ASDC’s ongoing efforts, progress made, and future plans for legacy missions.

Megan Buzanowicz↗

NASA NTRSDate not supplied

Extending CF Conventions to Enhance Data FAIRness for Atmospheric Composition Observations

The Hierarchical Data Format (HDF) and Network Common Data Form (NetCDF) are data file formats created to aid users in the creation or use of scientific data. These file formats are useful for handling large data volumes and hosting extensive metadata as global, group, or variable attributes and are popular with the modeling community. HDF and NetCDF files are widely used with atmospheric remote sensing data and have been used to support measurements from numerous field campaigns, from satellite to aircraft or ground and mobile based measurements. The files from airborne field studies, however, vary greatly in terms of the file structure and the amount and content of their metadata. Information relevant to the file that can be useful to the user such as the data producer, location where data was taken, variable descriptions, or information about the instrument might not be included in the file. Recently, the Measurements of Aerosols, Clouds, and their Interactions for Earth System Models (MACIE) group started a grassroots effort to develop a CF-based template for the HDF and NetCDF files for field studies, with the aim of making the data products more interoperable and usable. This template seeks to make the files more compliant to Climate and Forecast (CF) metadata conventions and to standardize the file structure and the global and variable attributes. The template would help to ensure that HDF and NetCDF files contain adequate metadata to better support their use for research, e.g., the modeling community, and to enhance the usability and interoperability of data for research communities at large. The draft template has been applied to recent field studies for various instruments and their merge files in support of the Atmosphere Observing System (AOS) project. The details of the revised template are to be presented, as well as examples of the implementation of these requirements for merge files and lidar observation data files and issues revealed during the implementation process.

Sean Leavor↗

NASA NTRSDate not supplied

ASDC’s Python-Based Metadata Extraction Pipeline for Suborbital Campaigns

The FAIRness of data products, especially findability and accessibility depend on rich metadata which, when extracted, can allow for proper curation. Over the past few years, the Atmospheric Science Data Center (ASDC) suborbital science support team has developed a metadata extraction pipeline to ensure the required metadata can be retrieved systematically, effectively, and efficiently to ensure the data can be used by a broad community. The development of a pipeline has presented many, but necessary, challenges to support archival and distribution of ASDC’s 30+ suborbital missions. Though sufficient metadata is provided by instrument scientists, the metadata may not be readily machine actionable due to different formats and templates. Further complicating metadata extraction, our team has found that the nature of metadata can be quite diverse given the difference in measurement types, instruments, and measurement platforms. A metadata extraction pipeline has been developed to provide an efficient, plugin-in based, method for adding new parsers, a configuration system that lets non-developers customize how files are processed, and a system for identifying and logging metadata quality issues to ensure they are readily found and addressed. The metadata extraction pipeline identifies critical pieces of metadata that are needed to promote data FAIRness, including location, file revision, measurement start/end datetime and can be easily modified to extract further information (such as variables). Given the wide-ranging datasets, the pipeline has been modified to accommodate multiple file formats, including multiple versions of ICARTT (International Consortium for Atmospheric Research on Transport and Transformation), HDF (Hierarchical Data Format), netCDF (network Common Data Form), and multiple versions of the Ames File Format. The pipeline also supports building metadata for file formats that cannot have metadata easily extracted from them, such as PDF (Portable Document Format) and GIF (Graphics Interchange Format). The pipeline has allowed our team to maintain a consistent flow of data and metadata to archival and distribution services, ensuring the ASDC meets the needs of the suborbital science community. This presentation will highlight the ASDC’s suborbital metadata extraction pipeline, its development, how it’s been modified to support data FAIRness, and plans for maintaining the pipeline and adding new features.

Abraham Porter↗

NASA NTRSDate not supplied

An Overview of NASA’s Airborne and Field Data Resource Center

A key recommendation from NASA’s 2022 Airborne and Field Data Workshop called for the development of a virtual Resource Center for all stakeholders across the data lifecycle of airborne and field Earth observations. The agency’s Earth Science Data and Information System (ESDIS) Project and Airborne Data Management Group (ADMG) have worked in concert to establish the newly launched NASA Airborne and Field Data Resource Center (AFDRC) to provide a single entry point for a wide assortment of information on and the effective, responsible stewardship of non-satellite observational data. The AFDRC compiles access to many existing resources, but does so in a newly organized way that integrates availability to increase efficiency and holistic understanding while simplifying users’ experience. Initially launched in fall of 2023, the AFDRC is a NASA Earthdata domain website that clarifies several previously disparate resources and provides newly updated information, including: Learning Resources: Educational resources to broaden understanding of the role airborne and field observations play in advancing understanding of our planet and NASA’s role in collecting and archiving these data. Support for data users to Find and Access Data: Advanced contextual browse/search capabilities that efficiently link researchers to data products suitable for their science objectives - this includes linking to NASA’s Catalog of Archived Suborbital Earth Science Investigations (CASEI). Working with Data: Tools specific to individual types of suborbital Earth Science data and their (inter-)disciplinary communities to provide access as well as guidance for their application. Stewardship Responsibilities: Resources for data producers with to lessen requirement burdens at the time of data transfer, and information for data stewards with consistent, authoritative guidance on best practices and agency- and/or community- specific archival procedures. This presentation will give an overview of the motivation for NASA’s AFDRC, approach for the design and content, iterative community-driven improvements, promote the use of the AFDRC, and solicit additional feedback from airborne and field data user communities.

Sara Lubkin↗