Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “system metadata”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 325 records · Page 18
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

ESS-DIVE Reporting Format for Comma-separated Values (CSV) File Structure

The ESS-DIVE reporting format for Comma-separated Values (CSV) file structure is based on a combination of existing guidelines and recommendations including some found within the Earth Science Community with valuable input from the Environmental Systems Science (ESS) Community. The CSV reporting format is designed to promote interoperability and machine-readability of CSV data files while also facilitating the collection of some file-level metadata content. Tabular data in the form of rows and columns should be archived in its simplest form, and we recommend submitting these tabular data following the ESS-DIVE reporting format for generic comma-separated values (CSV) text format files. In general, the CSV file format is more likely accessible by future systems when compared to a proprietary format and CSV files are preferred because this format is easier to exchange between different programs increasing the interoperability of a data file. By defining the reporting format and providing guidelines for how to structure CSV files and some field content within, this can increase the machine-readability of the data file for extracting, compiling, and comparing the data across files and systems.Data package files are in .csv, .png, and .md. Open the .csv with e.g. Microsoft Excel, LibreOffice, or Google Sheets. Open the .md files by downloading and using a text editor (e.g., notepad or TextEdit). Open the .png in e.g. a web browser, photo viewer/editor, or Google Drive.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Stomatal response of three Arctic plant species, Seward Peninsula, Alaska, 2022

Stomatal response to changing irradiance measurements on three common tussock tundra shrub species. Data were collected in 2022 from two sites on the Seward Peninsula, Kougarok Mile 64 and Teller Mile 27. Measurements were made with LI-COR LI-6800 gas exchange systems using a slow curve light response method. Light curve observations enable the estimation of stomatal slope and intercept parameters, g1 and g0. The data package files include a data file, metadata files and the complete instrument output for all measurements in .csv format, and a pdf describing the experimental protocol. See the related data package NGA315 for foliar trait data (leaf mass per area, leaf nitrogen concentration) and NGA209 for co-located thaw depth and soil moisture. The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research.The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska.Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Surface elevation, snow depth, vegetation height, and color imagery from multiple UAV surveys from 2018 to 2023 across a watershed near Teller road mile marker 27, Seward Peninsula, Alaska

Multiple photogrammetric surveys using an unoccupied aerial system (UAS) were performed across a watershed along Teller road on the Seward Peninsula (Alaska) to investigate terrain, vegetation and snowpack properties. This work was led by the environmental geophysics team from Lawrence Berkeley National Laboratory as part of the Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic). The multiple photogrammetric surveys used mostly a DJI Matrice 210 UAS with a DJI X5S camera, Ground Control Points (GCPs) surveyed with a Real Time Kinematic (RTK)-GPS, and a structure from motion (SfM) technique for photogrammetric reconstruction. Digital Surface elevation Models (DSMs) were inferred at various times of year, including near peak in plant and leaf density (July 19 2017), close to peak in snow depth (April 1 2019 and April 11 2022), and at low plant and leaf density of tall after the first bare-ground date (June 9 2019). Snow and canopy height were obtained by subtracting DSMs from a Digital Terrain Model (DTM) proxy inferred from the June 9 2019 survey. The derived products include ortho-mosaiced RGB map, DSMs, a DTM proxy, snow depth (April 2019 and 2022) and canopy height (July 2017). Unprocessed and processed data products are included in this package (processing levels 0-2). Data and metadata are provided as text (*.csv), image (*.jpg), and GeoTiff (*.tif) formats. This metadata document contains a description of the survey and processing steps and the inferred products.The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic) was a 15-year research effort (2012-2027) to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy’s Office of Biological and Environmental Research. The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska. Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy’s Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2012

The Re-Analysis of Ozone Profile Data from a 41-Year Series of SBUV Instruments

In this study we present the validation of ozone profiles from a number of Solar Back Scattered Ultra Violet (SBUV) and SBUV/2 instruments that were recently reprocessed using an updated (Version 8.6) algorithm. The SBUV dataset provides the longest available record of global ozone profiles, spanning a 41-year period from 1970 to 2011 (except a 5-year gap in the 1970s) and includes ozone profile records obtained from the Nimbus-4 BUV and Nimbus-7 SBUV instruments, and a series of SBUV(/2) instruments launched on NOAA operational satellites (NOAA 09, 11, 14, 16, 17, 18, 19). Although modifications in instrument design were made in the evolution from the BUV instrument to the modern SBUV(/2) model, the basic principles of the measurement technique and retrieval algorithm remain the same. The long term SBUV data record allows us to create a consistent, calibrated dataset of ozone profiles that can be used for climate studies and trend analyses. In particular, we focus on estimating the various sources of error in the SBUV profile ozone retrievals using independent observations and analysis of the algorithm itself. For the first time we include in the metadata a quantitative estimate of the smoothing error, defined as the error due to profile variability that the SBUV observing system cannot inherently measure. The magnitude of the smoothing error varies with altitude, latitude, season and solar zenith angle. Between 10 and 1 hPa the smoothing errors for the SBUV monthly zonal mean retrievals are of the order of 1 %, but start to increase above and below this layer. The largest smoothing errors, as large as 15-20%, were detected in in the troposphere. The SBUV averaging kernels, provided with the ozone profiles in version 8.6, help to eliminate the smoothing effect when comparing the SBUV profiles with high vertical resolution measurements, and make it convenient to use the SBUV ozone profiles for data assimilation and model validation purposes. The smoothing error can also be minimized by combining layers of data, and we will discuss recommendations for this approach as well. The SBUV ozone profiles have been intensively validated against satellite profile measurements obtained from the Microwave Limb Sounders (MLS) (on board the UARS and AURA satellites), Stratospheric Aerosol and Gas Experiment (SAGE) and Michelson Interferometer for Passive Atmospheric Sounding (MIPAS). Also, we compare coincident and collocated SBUV ozone retrievals with observations made by ground-based instruments, such as microwave spectrometers, lidars, Umkehr instruments and balloon-borne ozonosondes. Finally, we compare the SBUV ozone profiles with output from the NASA GSFC GEOS-CCM model. In the stratosphere between 25 and 1 hPa the mean biases and standard deviations are within 5% for monthly mean ozone profiles. Above and below this layer the vertical resolution of the SBUV algorithm decreases and the effects of vertical smoothing should be taken into account. Though the SBUV algorithm has a coarser vertical resolution in the lower stratosphere and troposphere, it is capable of precisely estimating the integrated ozone column between the surface and 25 hPa. The time series of the tropospheric - lower stratospheric ozone column derived from SBUV agrees within 5% with the corresponding values observed by an ensemble of ozone sonde stations in North Hemisphere. Drift of the ozone time series obtained from each SBUV(/2) instrument relative to ground based and satellite measurements are evaluated and some features of individual SBUV(l2) instruments are discussed. In addition to evaluating individual instruments against independent observations, we also focus on the instrument to instrument consistency in the series. Overall, Version 8.6 ozone profiles obtained from two different SBUV(l2) instruments compare within a couple of percent during overlap periods and are consistently varying in time, with some exceptions. Some of the noted discrepancies might bssociated with ozone diurnal variations, since the difference in the local time of the observations for a pair of SBUV(l2) instruments could be several hours. Other issues include the potential short-term drift in measurements as the instrument orbit drifts, and measurements are obtained at high solar zenith angles (>85 ). Based on the results of the validation, a consistent, calibrated dataset of SBUV ozone profiles has been created based on internal calibration only.

Kramarova, Natalya↗
DOE OSTI2025

Topography, surface water distribution and subsurface structure in 2023 across an Arctic coastal tundra site near Utqiagvik, Alaska

Subsurface electrical resistivity tomography (ERT), active layer thickness measurements, photogrammetry, and topographic data were collected in September 2023 along a 475 m long, 20 m wide corridor that traverses various polygon types within the Barrow Environmental Observatory (BEO) on the Alaskan Arctic Coastal Plain, approximately 4 miles from the Beaufort Sea near Utqiaġvik, Alaska. These measurements were designed to assess decadal changes in surface water distribution, topography, and subsurface structure across this dynamic landscape. This archive contains the datasets acquired in 2023 and references to the datasets acquired previously at the same location. The ERT survey was conducted along the 475 m transect using 0.5 m electrode spacing and a roll-along acquisition strategy. Thaw layer thicknesses were measured with a tile probe along the same transect. Photogrammetry data were acquired using an unoccupied aerial vehicle (UAV) and were used to generate a digital elevation model and an RGB mosaic. A real-time kinematic (RTK) GPS was used to survey the ERT electrodes and the ground control points for the aerial imagery. The dataset contains 5 *.csv data files, 6 *.csv metadata files, and 6 *.tif files.The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research. The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska. Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS1993

NELS 2.0 - A general system for enterprise wide information management

NELS, the NASA Electronic Library System, is an information management tool for creating distributed repositories of documents, drawings, and code for use and reuse by the aerospace community. The NELS retrieval engine can load metadata and source files of full text objects, perform natural language queries to retrieve ranked objects, and create links to connect user interfaces. For flexibility, the NELS architecture has layered interfaces between the application program and the stored library information. The session manager provides the interface functions for development of NELS applications. The data manager is an interface between session manager and the structured data system. The center of the structured data system is the Wide Area Information Server. This system architecture provides access to information across heterogeneous platforms in a distributed environment. There are presently three user interfaces that connect to the NELS engine; an X-Windows interface, and ASCII interface and the Spatial Data Management System. This paper describes the design and operation of NELS as an information management tool and repository.

Smith, Stephanie L.↗
DOE OSTI2024

Shrub Seedling Experiment: Environmental Conditions, Vegetation Composition, and Seedling Recruitment, Seward Peninsula, Alaska, 2018-2019

In June 2018, a manipulative seed-addition experiment was established at the Council Mile Marker 71 (CN_MM71), Quartz Creek Burn Area (Kougarok Mile Marker 86), and Quartz Creek Unburned (Kougarok Mile Marker 80) sites on the Seward Peninsula, Alaska. At each site, 5 replicate blocks were established in disturbed and intact upland tussock tundra (a total of 40 plots). Disturbed plots at Council were located on thermo-erosional (thermokarst) slopes, while disturbed plots in Quartz Creek were in burned areas affected by the Mingvk Lake fire in 2015. Using a split-plot design, an additional manual disturbance treatment (scraping to remove the surface bryophyte layer) was implemented across all blocks in June 2018. Seeds of five erect, deciduous shrubs were broadcast seeded into each plot in June 2018, and seedlings were censused in August 2018 and August 2019. The dataset includes ancillary measurements of plot-level environmental conditions (soil temperature, moisture, chemistry, thaw depth, microtopography, substrate thickness) and vegetation composition (NDVI and estimated percent cover by species and plant functional type). Dataset contains 10 csv data files (five data files with corresponding data dictionaries and one file-level metadata file) and one pdf user guide. The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research. The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska. Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Canopy spectral reflectance, Kougarok and Teller sites, Seward Peninsula, Alaska, 2016

Measurements of full-spectrum (350-2500 nm) canopy spectral reflectance of Arctic plant species at the Teller and Kougarok NGEE-Arctic sites, Seward Peninsula, Alaska. Spectra were collected in July 2016 using an SVC HR-2014i spectroradiometer together with a Spectralon white plate to calibrate each measurement under variable illumination conditions. The locations of the 43 measurement targets are provided as latitude and longitude recorded by the spectroradiometer internal GPS. This data package comprises .csv data and metadata files, and the SVC instrument output (.sig in .zip). The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research. The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska. Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2015

Taming Big Data Variety in the Earth Observing System Data and Information System

Although the volume of the remote sensing data managed by the Earth Observing System Data and Information System is formidable, an oft-overlooked challenge is the variety of data. The diversity in satellite instruments, science disciplines and user communities drives cost as much or more as the data volume. Several strategies are used to tame this variety: data allocation to distinct centers of expertise; a common metadata repository for discovery, data format standards and conventions; and services that further abstract the variations in data.

Information Systems↗
NASA NTRSDate not supplied

Additional Metadata Guidelines to Improve the Structure and Usability of HDF and NetCDF Files

The Hierarchical Data Format (HDF) and Network Common Data Form (NetCDF) are data file formats created to aid users in the creation or use of scientific data. These file formats are useful for handling large data volumes and hosting extensive metadata as global attributes or variables and are popular with the modeling community. HDF and NetCDF files are largely used with remote sensing data and have been used to support measurements from numerous campaigns, from satellite to aircraft or ground and mobile based measurements. The files from airborne field studies, however, vary greatly in terms of the file structure and the amount and content of metadata. Information relevant to the file that can be useful to the user such as the data producer, location where data was taken, variable descriptions, or information about the instrument might not be included in the file. This metadata might be present in another file in the dataset containing the same data using the International Consortium for Atmospheric Research on Transport and Transformation (ICARTT) format. Recently, the Aerosols, Clouds, and their Interactions for Earth System Models (MACIE) group started a grassroots effort to develop a set of requirements for the HDF and NetCDF files for field studies, aiming to make the data products more interoperable and usable. Particularly, these requirements seek to make the files more compliant to Climate and Forecast (CF) metadata conventions and to standardize the file structure and the global and variable attributes. These requirements would help to ensure that HDF and NetCDF files contain adequate metadata to better support their use for research, e.g., the modeling community, and to enhance the usability and interoperability of data for research communities at large. To be presented are the details of the MACIE requirements as well as examples of the implementation of these requirements for merge files and lidar observation data files.

Sean Leavor↗
NASA NTRS2023

Effects of Forcing Differences and Initial Conditions on Inter-Model Agreement in the VolMIP Volc-Pinatubo-Full Experiment

This paper provides initial results from a multi-model ensemble analysis based on the volc-pinatubo-full experiment performed within the Model Intercomparison Project on the climatic response to Volcanic forcing (VolMIP) as part of the sixth phase of the Coupled Model Intercomparison Project (CMIP6). The volc-pinatubo-full experiment is based on an ensemble of volcanic forcing-only climate simulations with the same volcanic aerosol dataset across the participating models (the 1991–1993 Pinatubo period from the CMIP6-GloSSAC dataset). The simulations are conducted within an idealized experimental design where initial states are sampled consistently across models from the CMIP6-piControl simulation providing unperturbed preindustrial background conditions. The multi-model ensemble includes output from an initial set of six participating Earth system models (CanESM5, GISS-E2.1-G, IPSL-CM6A-LR, MIROC-E2SL, MPI-ESM1.2-LR and UKESM1). The results show overall good agreement between the different models on the global and hemispheric scales concerning the surface climate responses, thus demonstrating the overall effectiveness of VolMIP's experimental design. However, small yet significant inter-model discrepancies are found in radiative fluxes, especially in the tropics, that preliminary analyses link with minor differences in forcing implementation; model physics, notably aerosol–radiation interactions; the simulation and sampling of El Niño–Southern Oscillation (ENSO); and, possibly, the simulation of climate feedbacks operating in the tropics. We discuss the volc-pinatubo-full protocol and highlight the advantages of volcanic forcing experiments defined within a carefully designed protocol with respect to emerging modelling approaches based on large ensemble transient simulations. We identify how the VolMIP strategy could be improved in future phases of the initiative to ensure a cleaner sampling protocol with greater focus on the evolving state of ENSO in the pre-eruption period.

metadata↗
DOE OSTI2024

Leaf area index (LAI), Teller site, Seward Peninsula, Alaska, 2022

Leaf area index (LAI) sampled from locations on the Teller MM27 NGEE-Arctic site, Seward Peninsula, Alaska. These data were collected in support of the ongoing NGEE-Arctic and NASA ABoVE data synthesis work. Samples were collected in July 2022 at 100 locations that cover the down slope half of Teller MM27 hillslope. This data package includes sample information and LAI data (.csv), an ESRI shape file (.shp) and Keyhole Markup Language (.kml) that define the location and area of each LAI measurement. Metadata files include data descriptions (_dd.csv) for tabular data, file level metadata (.csv), and methods and the instrument manual (.pdf). The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research. The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska. Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS1995

Interactive access and management for four-dimensional environmental data sets using McIDAS

This grant has fundamentally changed the way that meteorologists look at the output of their atmospheric models, through the development and wide distribution of the Vis5D system. The Vis5D system is also gaining acceptance among oceanographers and atmospheric chemists. Vis5D gives these scientists an interactive three-dimensional movie of their very large data sets that they can use to understand physical mechanisms and to trace problems to their sources. This grant has also helped to define the future direction of scientific visualization through the development of the VisAD system and its lattice data model. The VisAD system can be used to interactively steer and visualize scientific computations. A key element of this capability is the flexibility of the system's data model to adapt to a wide variety of scientific data, including the integration of several forms of scientific metadata.

Hibbard, William L.↗
DOE OSTI2024

Plant physiology, shrub size, thaw depth and soil water content, Seward Peninsula, Alaska, 2023

Stomatal response to changing irradiance measurements (m curves), CO2 response (ACi) and light response (AQ) curves on common tussock tundra shrub, graminoid and forb species. Data were collected in July 2023 from three sites on the Seward Peninsula, Kougarok Mile 64, Teller Mile 27 and Council. Measurements were made with LI-COR LI-6800 gas exchange systems on 18 species. For samples from Teller and Kougarok, thaw depth, soil moisture content and dGPS locations are also provided for most samples. Canopy height and canopy diameter are provided for shrubs measured at the Teller site. The data package files include data files, metadata files and the complete instrument output for all gas exchange measurements in .csv format, and a pdf describing the experimental protocols. See the related data package NGA508 for foliar trait data (leaf mass per area, leaf nitrogen concentration).The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a 15-year research effort (2012-2027) to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research.The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska.Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

BULKI-Store v0.3.2

BULKI-Store is a distributed object storage system optimized for high-performance computing environments. Built with a Rust core and Python bindings, it efficiently manages scientific and machine learning datasets across HPC clusters. The system employs a client-server architecture with MPI integration, enabling seamless scaling on supercomputers like Perlmutter. BULKI-Store's object-oriented approach provides intuitive data organization with rich metadata support, contrasting with traditional file-based solutions. Key optimizations include selective checkpoint loading, unified checkpoint files, and object chunking for large data transfers. For machine learning workloads, BULKI-Store offers advantages through fine-grained access patterns, dynamic data sharing between training instances, and reduced memory pressure. Memory management features include strategic Python GC calls, minimized data copies, and batch processing capabilities. The system leverages Rayon's thread pool for asynchronous data prefetching and supports multiple CPU architectures (ARM64, x86, AMD, RISC-V). By combining performance optimizations with developer-friendly APIs, BULKI-Store addresses the complex data management challenges of modern HPC applications while maintaining compatibility across heterogeneous computing environments.

Zhang, Wei [Lawrence Berkeley National Laboratory ↗
NASA NTRS1993

The Database Query Support Processor (QSP)

The number and diversity of databases available to users continues to increase dramatically. Currently, the trend is towards decentralized, client server architectures that (on the surface) are less expensive to acquire, operate, and maintain than information architectures based on centralized, monolithic mainframes. The database query support processor (QSP) effort evaluates the performance of a network level, heterogeneous database access capability. Air Force Material Command's Rome Laboratory has developed an approach, based on ANSI standard X3.138 - 1988, 'The Information Resource Dictionary System (IRDS)' to seamless access to heterogeneous databases based on extensions to data dictionary technology. To successfully query a decentralized information system, users must know what data are available from which source, or have the knowledge and system privileges necessary to find out this information. Privacy and security considerations prohibit free and open access to every information system in every network. Even in completely open systems, time required to locate relevant data (in systems of any appreciable size) would be better spent analyzing the data, assuming the original question was not forgotten. Extensions to data dictionary technology have the potential to more fully automate the search and retrieval for relevant data in a decentralized environment. Substantial amounts of time and money could be saved by not having to teach users what data resides in which systems and how to access each of those systems. Information describing data and how to get it could be removed from the application and placed in a dedicated repository where it belongs. The result simplified applications that are less brittle and less expensive to build and maintain. Software technology providing the required functionality is off the shelf. The key difficulty is in defining the metadata required to support the process. The database query support processor effort will provide quantitative data on the amount of effort required to implement an extended data dictionary at the network level, add new systems, adapt to changing user needs, and provide sound estimates on operations and maintenance costs and savings.

Source record↗
NASA NTRSDate not supplied

NASA Life Sciences Portal (NLSP): Supporting Scientific Transparency and Reproducibility

NASA’s Life Sciences Ports (NLSP) serves the scientific community by providing curated data from space life science experiment. The Human Research Program (HRP) with the help of NLSP is currently transforming their life sciences data archive systems and processes to improve compliance with the FAIR principles [1]. Some of these improvements will at the same time support the twin pillars of Open Science [2]: transparency of methods and reproducibility of results. Scientific transparency is marked by the easily intelligible communication of what has been investigated: what were the procedures for collecting sample and the characteristics of samples collected? what kinds of measurements were made, what were the environmental conditions of the measurements? What were the analysis techniques of the collected data? Reproducibility of the results and findings from the investigation requires a high level of transparency for all but the simplest investigations; the slightest deviation in communicating and replicating complex experimental procedures or data analyses can often yield quite different data and even findings, thwarting their validation. One of the ways the NLSP is aiming to improve the communication of scientific information is through the use of ontology-driven metadata. Ontologies are powerful, graph-based knowledge representation structures, which can be leveraged to increase data interoperability, the area of the FAIR principles in which many data systems most lack compliance. Over the past decade, there has been a concerted effort in the biomedical community to develop modular and narrowly focused domain and application-specific ontologies in a common, open-source framework, the Open Biological and Biomedical Ontology (OBO) Foundry [3]. The open sharing and modular nature of this effort promises huge increases in harmonized data sharing for systems that leverage these models. Which is in line with the FAIR Data Principles of Findability, Accessibility, Interoperability, and Reuse for scientific data management and stewardship.

Life Sciences data↗
NASA NTRSDate not supplied

Extending CF Conventions to Enhance Data FAIRness for Atmospheric Composition Observations

The Hierarchical Data Format (HDF) and Network Common Data Form (NetCDF) are data file formats created to aid users in the creation or use of scientific data. These file formats are useful for handling large data volumes and hosting extensive metadata as global, group, or variable attributes and are popular with the modeling community. HDF and NetCDF files are widely used with atmospheric remote sensing data and have been used to support measurements from numerous field campaigns, from satellite to aircraft or ground and mobile based measurements. The files from airborne field studies, however, vary greatly in terms of the file structure and the amount and content of their metadata. Information relevant to the file that can be useful to the user such as the data producer, location where data was taken, variable descriptions, or information about the instrument might not be included in the file. Recently, the Measurements of Aerosols, Clouds, and their Interactions for Earth System Models (MACIE) group started a grassroots effort to develop a CF-based template for the HDF and NetCDF files for field studies, with the aim of making the data products more interoperable and usable. This template seeks to make the files more compliant to Climate and Forecast (CF) metadata conventions and to standardize the file structure and the global and variable attributes. The template would help to ensure that HDF and NetCDF files contain adequate metadata to better support their use for research, e.g., the modeling community, and to enhance the usability and interoperability of data for research communities at large. The draft template has been applied to recent field studies for various instruments and their merge files in support of the Atmosphere Observing System (AOS) project. The details of the revised template are to be presented, as well as examples of the implementation of these requirements for merge files and lidar observation data files and issues revealed during the implementation process.

Sean Leavor↗