Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “system metadata”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 271 records · Page 15
All sourcesNASA NTRSDOE OSTI
NASA NTRS2002

OAI and NASA's Scientific and Technical Information

The Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) is an evolving protocol and philosophy regarding interoperability for digital libraries (DLs). Previously, "distributed searching" models were popular for DL interoperability. However, experience has shown distributed searching systems across large numbers of DLs to be difficult to maintain in an Internet environment. The OAI-PMH is a move away from distributed searching, focusing on the arguably simpler model of "metadata harvesting". We detail NASA s involvement in defining and testing the OAI-PMH and experience to date with adapting existing NASA distributed searching DLs (such as the NASA Technical Report Server) to use the OAI-PMH and metadata harvesting. We discuss some of the entirely new DL projects that the OAI-PMH has made possible, such as the Technical Report Interchange project. We explain the strategic importance of the OAI-PMH to the mission of NASA s Scientific and Technical Information Program.

Nelson, Michael L.↗
NASA NTRS2019

Quantitative Comparison of Proprietary and Open-Source Georeferencing Tools for Use with Astronaut Photography

The Crew Earth Observations (CEO) Facility within the Earth Science and Remote Sensing Unit at NASA’s Johnson Space Center supports the acquisition, analysis, and curation of astronaut photography of Earth’s surface and atmosphere. Astronauts on the International Space Station (ISS) respond to requests from CEO to acquire imagery of scientific and education targets, to include high profile targets in response to activations from the International Charter for Space & Major Disasters (also known as the International Disaster Charter, or IDC) and NASA’s Disasters Program. CEO facilitates the acquisition of astronaut photography in response to IDC events and delivers georeferenced data products to the United States Geological Survey (USGS) for distribution to the disaster community. Using GeoRef, an internal web-based tool developed in collaboration with NASA’s Ames Research Center, CEO generates data packages of georeferenced imagery, uncertainty images for assessing control and tie point accuracy, and metadata documenting raw and processed data. Operational experience with the Georef software identified vulnerabilities to internal code and server errors that can significantly increase time of data production. As such, CEO developed a backup procedure in case the GeoRef software experiences front-end or back-end errors. A system using OSGEO’s open-source QGIS software combined with a semi-automated pipeline using the object-oriented Python language and the Geospatial Abstract Library for generating metadata is quantitatively compared to GeoRef’s data package for quality and productivity. Root Mean Square Error (RMSE) provides a standard measurement of data quality as it relates to ground error. Assessing RMSE measurements generated from georeferenced astronaut photographs acquired with different obliquity and focal length offers a comprehensive accuracy assessment of the software’s transformation algorithms. This assessment will indicate the software's ability to produce data products with the least ground-error or highest data quality regarding ground accuracy. In addition, a comparison of the software’s efficiency in generating a data package that includes georeferenced images, metadata, and uncertainty images for measuring tie/ground point error was performed. Initial results, based on the comparison of three nadir-facing astronaut photographs acquired with a 95mm focal length, reveal the QGIS-based system's average RMSE is 2.36 (pixels) suggesting its georectification system produces data products that meet and perhaps improve upon Georef solution's average RMSE of 32.99 (pixels). However, the QGIS system was unable to reproduce two unique Georef data products, uncertainty images for measuring tie and control point errors and a translated unwrapped image. In addition, the Georef software is designed to accept handheld camera pose information from a hardware component (Geosens) scheduled for deployment on the ISS in late 2018; this information is intended to provide increased accuracy and auto-registration capability for astronaut photographs. Future work is expected to determine the QGIS-based georectification system’s potential as an open-source alternative (and operational backup) to Georef for georeferencing the full range of resolutions and viewing angles unique to handheld digital camera imagery in support of ISS disaster response activities.

Jagge, Amy M.↗
NASA NTRSDate not supplied

Preserving NASA Historic and Current Mission Data and Adding Value to These for Future Researchers

The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) has been actively involved in many aspects of ensuring the long-term preservation of NASA earth science data and knowledge. This involves both the recovery and preservation of early NASA meteorological and other earth observation data, as well as preserving the more recent Earth Observation System (EOS) mission data sets which continue or have reached their end of lifetime. The GES DISC adds value to these preserved data by adding metadata and making the data available online to future researchers. The early NASA meteorological and earth observation data sets from the 1960s and 70s were originally archived on magnetic tapes, and visualizations of these data were preserved on 70-mm film. As these media have aged, their contents have been at risk of permanent loss. NASA has given the task of preserving these early data sets to the GES DISC and making these data sets easily available to the public. The data from these early missions are potentially useful to climate researchers as these are some of the only global measurements made at their time. These old data on magnetic tapes and film strips do not contain easily readable metadata, and so to add value the GES DISC has added digital metadata to them so that the data are searchable and findable. The GES DISC is also involved in preserving the data and knowledge from the EOS era missions. The GES DISC follows the guidelines developed for the preservation of data as specified in the NASA EOS Data and Information System (EOSDIS) Earth Science Data Preservation Content Specification (423-SPEC-001) document. To date, the GES DISC has consulted with the data science teams from the following missions: UARS, Earth Probe TOMS, Aura HIRDLS, and SORCE, in order to properly preserve their data and accompanying documentation. The GES DISC is also currently working with the EOS science teams from TRMM, AIRS, MLS, OMI and additional missions to ensure that the relevant documents and data sets are properly archived for future researchers. A standardized procedure for mission data preservation following 423-SPEC-001 makes preservation among the many NASA EOSDIS data centers uniform, so that these could be transitioned easily to a common EOSDIS preservation repository. This presentation will give an overview of the preservation and recovery of the old NASA historical data sets archived at the GES DISC, as well as the data and documentation preservation efforts of the EOS era missions.

James Johnson↗
DOE OSTI2026

In Situ Soil Moisture and Thaw Depth Measurements Coincident with Airborne SAR Data Collections, Seward Peninsula, Alaska, 2022

The in-situ soil moisture and thaw depth measurements provided in this dataset were collected coincident with airborne overflights of L-band synthetic aperture radar (SAR) instruments at the Teller, Kougarok, and Council study sites on the Seward Peninsula, Alaska. Overflights occurred on August 19, 2022. Soil moisture data at Teller and Kougarok was collected on August 19, and at Council on August 20. Thaw depth, soil pits, and any additional measurements were recorded on August 20 and 21. Field measurements and flights were conducted during the summer of 2022 as a collaboration between the National Aeronautics and Space Administration (NASA) Arctic-Boreal Vulnerability Experiment (ABoVE) Project’s Airborne SAR Campaign and the Next-Generation Ecosystem Experiments (NGEE) Arctic Project. This dataset includes a data file (*.csv), a data dictionary (*_dd.csv) and a file-level metadata (*_flmd.csv). The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy’s Office of Biological and Environmental Research. The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska. Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy’s Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

EARTH SCIENCE > LAND SURFACE > FROZEN GROUND↗
DOE OSTI2024

Organic layer thickness and carbon concentration in burned and unburned sites, Seward Peninsula, AK, 2022

Measurements associated with organic layer samples collected from naturally burned (1971, 2002, 2015, 2019) and unburned sites at the Kougarok Fire Complex, Seward Peninsula, AK, 2022. Here, a discontinuous permafrost underlies an arctic tundra ecosystem. Measurements include elemental carbon and nitrogen concentrations and stocks, organic layer thickness, and thaw depth. There are five files in *.csv format with one data file and four data description files including data dictionary, methods, terminology, and file-level metadata. The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research.The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska.Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

UAS remote sensing (Autel EVO II platform): Red-green-blue (RGB) imagery and derived products (Level 0-2 data), Seward Peninsula, Alaska, 2023

Airborne remote sensing data collected using a optical red-green-blue (RGB) sensor installed on an Autel EVO II unoccupied aerial system (UAS). This package includes data from 4 flights flown over the NGEE-Arctic Council Mile Marker (MM) 71 and Teller MM27 sites on the Seward Peninsula, Alaska USA, in July 2023. This package provides the Level 0 (raw, unprocessed) data collected by the platform and sensors, and Level 1-2 processed products including digital surface models (DSM), photo orthomosaics, canopy height models (CHM) and digital terrain models (DTM). Ancillary aircraft data, flight mission parameters, and general flight conditions provided by the onboard flight and data collection computers are also included. Data and metadata are provided as image (*.jpg, *.tif), point cloud (*.laz), tabular (*.csv) and *.pdf formats. Raw images are compressed as tar.gz. The metadata documents contain flight campaign, platform, sensor, flight and file metadata, along with a description of the data and file naming scheme.The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a 15-year research effort (2012-2027) to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research.The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska.Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS1995

Aircraft scanner data availability via the version 0 Information Management System

As part of the Earth Observing System Data and Information System (EOSDIS) development, NASA and other government agencies have developed an operational prototype of the Information Management System (IMS). The IMS provides access to the data archived at the Distributed Active Archive Centers (DAAC's) that allows users to search through metadata describing the (image) data. Criteria based on sensor name or type, date and time, and geographic location are used to search the archive. Graphical representations of coverage and browse images are available to further refine a user's selection. previously, the EROS Data Center (EDC) DAAC had identified the Advanced SOlid-state Array Spectrometer (ASAS), Airborne Visible and infrared Imaging Spectrometer (AVIRIS), NS-001, and Thermal Infrared Multispectral Scanner (TIMS) as precursor data sets similar to those the DAAC will handle in the Earth Observing System era. Currently, the EDC DAAC staff, in cooperation with NASA, has transcribed TIMS, NS-001, and Thematic Mapper Simulation (TMS) data from Ames Research Center and also TIMS data from Stennis Space Center. During the transcription process, the IMS metadata and browse images were created to populate the inventory at the EDC DAAC. These data sets are now available in the IMS and may be requested from the any of the DAAC's via the IMS.

Mah, G. R.↗
DOE OSTI2024

UAS remote sensing (Autel EVO II platform): Red-green-blue (RGB) imagery and derived products (Level 0-2 data), Seward Peninsula, Alaska, 2022

Airborne remote sensing data collected using a optical red-green-blue (RGB) sensor installed on an Autel EVO II unoccupied aerial system (UAS). This package includes data from 15 flights flown over the NGEE-Arctic Council Mile Marker (MM) 71, Kougarok MM64, and Teller MM27 sites on the Seward Peninsula, Alaska USA, in July 2022. This package provides the Level 0 (raw, unprocessed) data collected by the platform and sensors, and Level 1-2 processed products including digital surface models (DSM), photo orthomosaics, canopy height models (CHM) and digital terrain models (DTM). Ancillary aircraft data, flight mission parameters, and general flight conditions provided by the onboard flight and data collection computers are also included. Data and metadata are provided as image (*.jpg, *.tif), point cloud (*.laz), tabular (*.csv) and *.pdf formats. Raw images are compressed as tar.gz. The metadata documents contain flight campaign, platform, sensor, flight and file metadata, along with a description of the data and file naming scheme.The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a 15-year research effort (2012-2027) to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research.The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska.Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Thaw depth, soil moisture, and vegetation height, Teller and Kougarok sites, Seward Peninsula, Alaska, 2022

Thaw depth, soil moisture, and vegetation height sampled from locations on the Teller MM27, Kougarok MM80, and Kougarok MM82 NGEE-Arctic sites, Seward Peninsula, Alaska. These data were collected in support of the ongoing NGEE-Arctic and NASA ABoVE data synthesis work. Samples were collected in July 2022, including 6 transects covering the entire Teller MM27 watershed and 4 transects covering 4 thaw ponds at Kougarok MM80 and MM82. This data package includes sample information and thaw depth, soil moisture, vegetation height data (.csv). Metadata files include data descriptions (_dd.csv) for tabular data and file level metadata (.csv). The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research. The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska. Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

NEPATEC2.0: NEPA Text Corpus v2.0

The National Environmental Policy Act of 1969, as amended (NEPA), is a major environmental law in the United States, requiring Federal agencies to consider and document potential environmental impacts before deciding on a proposed action. Modernization of NEPA and permitting processes faces significant challenges due to the lack of standardized formats and interoperable systems for organizing and sharing NEPA-related information across agencies. Much of the information gathered during NEPA reviews is written into documents such as categorical exclusions, environmental assessments, and environmental impact statements, then filed in predominately independent agency file stores that may or may not be publicly accessible. The application of metadata and data standards, such as those recommended by the Council on Environmental Quality (CEQ), to NEPA documents offers a shared vocabulary and structure for key entities like projects, processes, and documents that can streamline information exchange and enhance collaboration across systems. In this work, we publicly release NEPATEC2.0, an expanded corpus of NEPA documents with associated metadata. NEPATEC2.0 encompasses approximately 120,000 documents from 60,000 projects prepared by more than 60 different agencies. Modeled to align with CEQ metadata standards, NEPATEC2.0 promotes consistency in environmental reviews and supports the ongoing effort to modernize permitting technologies by facilitating more transparent, efficient, and data-driven decision-making. Importantly, NEPATEC2.0 demonstrates the possibilities and limitations of large language model-based prompting to extract information from NEPA documents at scale.

environmental review↗
DOE OSTI2024

Active Layer and Moisture Measurements for Intensive Site 0 and 1, Utqiagvik (Barrow), Alaska

Thaw depth, relative dielectric permittivity, and soil temperature were collected along three transects within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) near Utqiagvik (Barrow), Alaska. Measurements were acquired along the transect at 3 m intervals on 2011-09-25, 2011-09-26, 2012-07-18, 2012-09-30, 2014-07-02, 2014-08-15, and 2014-09-14 for the L2 transect (intensive site 0), which is about 470 m long. Similarly, measurements were collected every 3 m on 2012-09-30, 2013-09-30, 2014-07-02, 2014-08-14, and 2014-09-23, and every 1 m on 2013-09-30 along the AB transect (intensive site 1) which is about 500 m long. File types in the package are all csv spreadsheets, following the ESS-DIVE csv guidelines (ESS-DIVE CSV v1), organized by transect name, with accompanying data dictionary files containing the filename suffix "_dd" (also csv format). Data collection details can be found in the Methods description of the metadata record. The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research. The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska. Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS1994

JPL Physical Oceanography Distributed Active Archive Center (PO.DAAC) data availability, version 1-94

The Physical Oceanography Distributed Active Archive Center (PO.DAAC) archive at the Jet Propulsion Laboratory (JPL) includes satellite data sets for the ocean sciences and global-change research to facilitate multidisciplinary use of satellite ocean data. Parameters include sea-surface height, surface-wind vector, sea-surface temperature, atmospheric liquid water, and integrated water vapor. The JPL PO.DAAC is an element of the Earth Observing System Data and Information System (EOSDIS) and is the United States distribution site for Ocean Topography Experiment (TOPEX)/POSEIDON data and metadata.

Source record↗
NASA NTRS2011

In Situ Surface Characterization

Operation of in situ space assets, such as rovers and landers, requires operators to acquire a thorough understanding of the environment surrounding the spacecraft. The following programs help with that understanding by providing higher-level information characterizing the surface, which is not immediately obvious by just looking at the XYZ terrain data. This software suite covers three primary programs: marsuvw, marsrough, and marsslope, and two secondary programs, which together use XYZ data derived from in situ stereo imagery to characterize the surface by determining surface normal, surface roughness, and various aspects of local slope, respectively. These programs all use the Planetary Image Geometry (PIG) library to read mission-specific data files. The programs themselves are completely multimission; all mission dependencies are handled by PIG. The input data consists of images containing XYZ locations as derived by, e.g., marsxyz. The marsuvw program determines surface normals from XYZ data by gathering XYZ points from an area around each pixel and fitting a plane to those points. Outliers are rejected, and various consistency checks are applied. The result shows the orientation of the local surface at each point as a unit vector. The program can be run in two modes: standard, which is typically used for in situ arm work, and slope, which is typically used for rover mobility. The difference is primarily due to optimizations necessary for the larger patch sizes in the slope case. The marsrough program determines surface roughness in a small area around each pixel, which is defined as the maximum peak-to-peak deviation from the plane perpendicular to the surface normal at that pixel. The marsslope program takes a surface normal file as input and derives one of several slope-like outputs from it. The outputs include slope, slope rover direction (a measure of slope radially away from the rover), slope heading, slope magnitude, northerly tilt, and solar energy (compares the slope with the Sun s location at local noon). The marsuvwproj program projects a surface normal onto an arbitrary plane in space, resulting in a normalized 3D vector, which is constrained to lie in the plane. The marsuvwrot program rotates the vectors in a surface normal file, generating a new surface normal file. It also can change coordinate systems for an existing surface normal file. While the algorithms behind this suite are not particularly unique, what makes the programs useful is their integration into the larger in situ image processing system via the PIG library. They work directly with space in situ data, understanding the appropriate image metadata fields and updating them properly. The secondary programs (marsuvwproj, marsuvwrot) were originally developed to deal with anomalous situations on Opportunity and Spirit, respectively, but may have more general applicability.

Deen, Robert G.↗
NASA NTRS2005

Online Metadata Directories: A way of preserving, sharing and discovering scientific information

The Global Change Master Directory (GCMD) assists the scientific community in the discovery of and linkage to Earth Science data and provides data holders a means to advertise their data to the community through its portals, i.e. online customized subset metadata directories. These directories are effectively serving communities like the Joint Committee on Antarctic Data Management (JCADM), the Global Observing System Information Center (GOSIC), and the Global Ocean Ecosystems Dynamic Program (GLOBEC) by increasing the visibility of their data holding. The purpose of the Gulf of Maine Ocean Data Partnership (GoMODP) is to "promote and coordinate the sharing, linking, electronic dissemination, and use of data on the Gulf of Maine region". The participants have decided that a "coordinated effort is needed to enable users throughout the Gulf of Maine region and beyond to discover and put to use the vast and growing quantities of data in their respective databases". GoMODP members have invited the GCMD to discuss potential collaborations associated with this effort. The presentation will focus on the use of the GCMD s metadata directory as a powerful tool for data discovery and sharing. An overview of the directory and its metadata authoring tools will be given.

Meaux, M.↗
DOE OSTI2020

Extending the Publish/Subscribe Abstraction for High-Performance I/O and Data Management at Extreme Scale

The Adaptable I/O System (ADIOS) represents the culmination of substantial investment in Scientific Data Management, and it has demonstrated success for several important extreme-scale science cases. However, looking towards the exascale and beyond, we see the development of yet more stringent data management requirements that require new abstractions. Therefore, there is an opportunity to attempt to connect the traditional realms of HPC I/O optimization with the Database / Data Management community. As such, in this paper we offer some specific examples from our ongoing work in managing data structures, services, and performance at the extreme scale for scientific computing. Using the publish/subscribe model afforded by ADIOS, we demonstrate a set of services that connect data format, metadata, queries, data reduction, and high-performance delivery. The resulting publish/subscribe framework facilitates connection to on-line workflow systems to enable the dynamic capabilities that will be required for exascale science.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2025

dCache project status and update

The dCache project delivers an open-source, massively scalable, distributed storage system deployed internationally to satisfy today’s scientists’ ever-demanding storage requirements. Its multifaceted approach supports different use cases with the same storage, from high throughput data ingest, data sharing over wide area networks, efficient access from HPC clusters, and longterm data persistence on tertiary storage. Even though dCache was initially developed for HEP experiments, today, it is used by various scientific communities, including astrophysics, biomed, and life science, each with their specific requirements. To match the needs of these new communities and keep up with the scaling demands of existing experiments, dCache is permanently evolving. With this contribution, we would like to highlight the recent developments in dCache regarding integration with CERN Tape Archive (CTA), advanced metadata handling, token-based authorization support, bulk API for QoS transitions, REST API to control interaction with the tape system, and future development directions.

Mkrtchyan, Tigran [DESY]↗
NASA NTRSDate not supplied

WIS and WIGOS Metadata as the Foundation for a Sustainable Framework for Global Greenhouse Gas Watch Data Exchange

Metadata (data about data) is a critical component of data discovery, description, evaluation, documentation, and preservation. Developing and propagating metadata standards has been a longstanding area of activity in WMO and beyond. The WIS2 and WIGOS metadata models are being actively developed and maintained by dedicated task teams, established under the WMO Expert Team on Metadata. The metadata representations and vocabularies are governed by well-established processes within WMO. These standards are being used in a number of metadata/data exchange activities (e.g., WMO Information System 2.0 (WIS2), WIGOS (WMDR), Climate Data Management Systems (CMDS), etc.). It should also be noted that the application of the WIS2 and WIGOS standards fully support the WMO Unified Data Policy and open data policy as well as greatly enhance the value of observations by fostering data F.A.I.R.ness. Furthermore, the WMO metadata standards can serve as the foundation for a framework that will facilitate metadata mapping between the existing schemas used in well-established data centres, e.g., WMO WDCGG (World Data Centre for Greenhouse Gases) and NOAA ObsPack (Observation Package Data Products) and to automate metadata exchange between data centres as well as with WMO. These activities will play a central role in integrating measurements sponsored by various member countries and organizations to provide a more comprehensive characterization of the temporal and spatial distribution of the greenhouse gases. At the same time, this metadata exchange can lead to member countries and partner organizations improving their current metadata collection process for data discoverability, interoperability, and (re)usability. This presentation will describe metadata activities in the context of WIS2 and WIGOS and how they apply to GGGW data integration via metadata mapping and exchange.

Gao Chen↗
DOE OSTI2021

Enabling modern data discovery for atmospheric measurements

The Atmospheric Radiation Measurement (ARM) user facility is a US Department of Energy Office of Science user facility that is managed and operated through a collaborative effort led by nine US Department of Energy national laboratories. The ARM Data Center, located at Oak Ridge National Laboratory, is responsible for the timely collection, processing, and delivery of data products to the scientific community. The ARM Data Center holds more than 11,000 data products, including metadata collected from field campaigns, instruments, value-added products, and principal investigator–contributed data. These data sets are checked for successful transfer (for most data, this transfer is carried out automatically via the network; however, some of the largest data sets and some of the most remote sites require manual shipping of hard disks) and both the data and metadata are processed to a standard format, which is an ARM-standardized structure, via the Network Common Data Form. The Network Common Data Form is a self-describing binary format with many compatible software tools. Once processed, the data are cataloged, stored in the ARM Data Archive, and made discoverable through association with an array of metadata-characterizing information, such as location and measurement classification. These metadata enable powerful search capabilities through the ARM Data Center Data Discovery interface. This paper discusses the workflow of how the new discovery system has been redesigned from user requirements and how the data are distributed to the scientific community.

54 ENVIRONMENTAL SCIENCES↗