Materials and radiation interaction data library
A library and APIs for fast access to binary reference data compiled for X-ray and neutron interaction with materials.
Vardar-Irrgang, MichaelE [Lawrence Livermore Natio↗
SEARCH · Engineering Papers
Results for “Data access”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Optimizing tertiary storage organization and access for spatio-temporal datasets
We address in this paper data management techniques for efficiently retrieving requested subsets of large datasets stored on mass storage devices. This problem represents a major bottleneck that can negate the benefits of fast networks, because the time to access a subset from a large dataset stored on a mass storage system is much greater that the time to transmit that subset over a network. This paper focuses on very large spatial and temporal datasets generated by simulation programs in the area of climate modeling, but the techniques developed can be applied to other applications that deal with large multidimensional datasets. The main requirement we have addressed is the efficient access of subsets of information contained within much larger datasets, for the purpose of analysis and interactive visualization. We have developed data partitioning techniques that partition datasets into 'clusters' based on analysis of data access patterns and storage device characteristics. The goal is to minimize the number of clusters read from mass storage systems when subsets are requested. We emphasize in this paper proposed enhancements to current storage server protocols to permit control over physical placement of data on storage devices. We also discuss in some detail the aspects of the interface between the application programs and the mass storage system, as well as a workbench to help scientists to design the best reorganization of a dataset for anticipated access patterns.
Chen, Ling Tony↗
Constraint based scheduling for the Goddard Space Flight Center distributed Active Archive Center's data archive and distribution system
The Goddard Space Flight Center (GSFC) Distributed Active Archive Center (DAAC) has been operational since October 1, 1993. Its mission is to support the Earth Observing System (EOS) by providing rapid access to EOS data and analysis products, and to test Earth Observing System Data and Information System (EOSDIS) design concepts. One of the challenges is to ensure quick and easy retrieval of any data archived within the DAAC's Data Archive and Distributed System (DADS). Over the 15-year life of EOS project, an estimated several Petabytes (10(exp 15)) of data will be permanently stored. Accessing that amount of information is a formidable task that will require innovative approaches. As a precursor of the full EOS system, the GSFC DAAC with a few Terabits of storage, has implemented a prototype of a constraint-based task and resource scheduler to improve the performance of the DADS. This Honeywell Task and Resource Scheduler (HTRS), developed by Honeywell Technology Center in cooperation the Information Science and Technology Branch/935, the Code X Operations Technology Program, and the GSFC DAAC, makes better use of limited resources, prevents backlog of data, provides information about resources bottlenecks and performance characteristics. The prototype which is developed concurrently with the GSFC Version 0 (V0) DADS, models DADS activities such as ingestion and distribution with priority, precedence, resource requirements (disk and network bandwidth) and temporal constraints. HTRS supports schedule updates, insertions, and retrieval of task information via an Application Program Interface (API). The prototype has demonstrated with a few examples, the substantial advantages of using HTRS over scheduling algorithms such as a First In First Out (FIFO) queue. The kernel scheduling engine for HTRS, called Kronos, has been successfully applied to several other domains such as space shuttle mission scheduling, demand flow manufacturing, and avionics communications scheduling.
Short, Nick, Jr.↗
Cloud-Based Time Series Analysis of Extremes: Use Cases and Applications
"Extreme weather events, such as hurricanes, tornadoes, floods, droughts, heatwaves, and blizzards, can cause widespread damage, disrupting ecosystems, agricultural production, and economies. The frequency and intensity of these events have been increasing, likely due to climate change, raising concerns and the need for more accurate analysis and predictions. NASA's Goddard Earth Sciences Data and Information Services Center (GES DISC) has migrated its long-term historical datasets, including precipitation data from MERRA-2 reanalysis, GLDAS land data assimilation, and IMERG satellite observations, to the cloud. This cloud-stored data enables scientists and researchers to utilize cloud computing for advanced modeling and forecasting of extreme weather events, eliminating the need to download large datasets. In this presentation, we will provide an overview of the cloud-based data and services managed by GES DISC; demonstrate methods for accessing and analyzing time series data stored in the cloud; and compare results across various datasets to address critical questions related to extreme precipitation. We will present use cases including: 1. Determining the average total precipitation in California during January and February from 2000 to 2024, and identifying anomalous precipitation in 2021. 2. Calculating the 10, 20, 50, and 100-year return periods for maximum daily rainfall based on 25 years of historical precipitation data (2000-2024) for Maryland.
time series↗
A Vision for Coupling Operation of US Fusion Facilities with HPC Systems and the Implications for Workflows and Data Management
The operation of large US Department of Energy (DOE) research facilities, like the DIII-D National Fusion Facility, results in the collection of complex multi-dimensional scientific datasets, both experimental and model-generated. In the future, it is envisioned that integrated data analysis coupled with large-scale high performance computing (HPC) simulations will be used to improve experimental planning and operation. Practically, massive data sets from these simulations provide the physics basis for generation of both reduced semi-analytic and machine-learning-based models. Storage of both HPC simulation datasets (generated from US DOE leadership computing facilities) and experimental datasets presents significant challenges. In this paper, we present a vision for a DOE-wide data management workflow that integrates US DOE fusion facilities with leadership computing facilities. Data persistence and long-term availability beyond the length of allocated projects is essential, particularly for verification and recalibration of artificial intelligence and machine learning (AI/ML) models. Because these data sets are often generated and shared among hundreds of users across multiple leadership computing facility centers, they would benefit from cross-platform accessibility, persistent identifiers (e.g. DOI, or digital object identifier), and provenance tracking. Here, the ability to handle different data access patterns suggests that a combination of low cost, high latency (e.g. for storing ML training sets) and high cost, low latency systems (e.g. for real-time, integrated machine control feedback) may be needed.
70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
A Data Deposition Platform for Sharing Nuclear Magnetic Resonance Data
Nuclear magnetic resonance (NMR) data are rarely deposited in open databases, leading to loss of critical scientific knowledge. Existing data reporting methods (images, tables, lists of values) contain less information than raw data, and are poorly standardized. Together, these issues limit FAIR (findable, accessible, interoperable, reusable) access to these data, which in turn creates barriers for compound dereplication and the development of new data-driven discovery tools. Existing NMR databases are either not designed for natural products data, or employ complex deposition interfaces that disincentivize deposition. Journals, including the Journal of Natural Products (JNP), are now requiring data submission as part of the publication process, creating the need for a streamlined, user-friendly mechanism to deposit and distribute NMR data. Recently, our team reported the development of the Natural Products Magnetic Resonance Database (NP-MRD; www.np-mrd.org). Here in this paper we present a new data deposition platform for the NP-MRD project that is designed to enable users to deposit NMR data for published or submitted manuscripts in under five minutes. This platform includes a suite of automated data extraction and standardization tools, together with a simple-to-use web-based interface and detailed error reporting to simplify the data deposition process and is available at www.np-mrd.org/submissions.
96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
Mic-hackathon 2024: hackathon on machine learning for electron and scanning probe microscopy
Microscopy is one of the primary sources of information on materials structure and functionality at the nanometer and atomic scales. The data generated through microscopy is often contained in well-structured datasets, enriched with extensive metadata and sample histories, although not always with the same level of detail or storage format. The broad incorporation of data management plans by major funding agencies ensures the preservation and accessibility of this data. However, deriving insights from these rich datasets remains challenging due to the lack of established code ecosystems, standardized benchmarks, and integration strategies. Correspondingly, the efficiency of data usage is very low, and time expenditures at the analysis stage are enormous. In addition to post-acquisition data analysis, the emergence of application programming interfaces by major microscope manufacturers now creates opportunities for real-time ML-based data analytics to enable automated decision making, and particularly ML-agent controlled real-time microscope operation. Despite these opportunities, there is a significant gap in integrating the ML community with the broader microscopy community, limiting the value that these methods bring to physics and materials discovery and materials optimization. Hackathons address these challenges by fostering collaboration between ML experts and microscopy professionals, encouraging the development of innovative solutions that leverage ML for microscopy and preparing the workforce of the future both for microscopy-intensive domains areas, instrument manufacturers, and ML scientists interested in real world applications for fundamental research, materials optimization, and manufacturing. The hackathon generated benchmark datasets and digital twins of microscopes that further contribute to the development of the field and establish data analysis ecosystems. All the codes can be found at GitHub(https://github.com/KalininGroup/Mic-hackathon-2024-codes-publication/tree/1.0.0.1) and Zenodo (https://zenodo.org/records/15579940).
97 MATHEMATICS AND COMPUTING↗
Battery Lifecycle Framework
The Battery Lifecycle (BLC) Framework is an open-source platform that provides tools to visualize and share battery data from material characterization, cell testing, manufacturing, and field testing through the technology development cycle. BLC has three components: data importers, a front-end for querying the data and creating visualizations, and an application programming interface to provide access to the data from Python. BLC has been deployed for tracking the development of a battery from the lab to a manufacturing line and systems installed in the field and for comparing studies of multiple cells of the same battery chemistry and configuration. The code was developed around Redash, a robust open-source extract-transform-load engine. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525. SAND2021-4546 O
De Angelis, Valerio↗
Collation of earth resources data collected by ERIM airborne sensors
Earth resources imagery from nine years of data collection with developmental airborne sensors is cataloged for reference. The imaging sensors include single and multiband line scanners and side-looking radars. The operating wavelengths of the sensors include ultraviolet, visible and infrared band scanners, and X- and L-band radar. Imagery from all bands (radar and scanner) were collected at some sites and many sites had repeated coverage. The multiband scanner data was radiometrically calibrated. Illustrations show how the data can be used in earth resource investigations. References are made to published reports which have made use of the data in completed investigations. Data collection sponsors are identified and a procedure described for gaining access to the data.
Hasell, P. G., Jr.↗
Systems aspects of COBE science data compression
A general approach to compression of diverse data from large scientific projects has been developed and this paper addresses the appropriate system and scientific constraints together with the algorithm development and test strategy. This framework has been implemented for the COsmic Background Explorer spacecraft (COBE) by retrofitting the existing VAS-based data management system with high-performance compression software permitting random access to the data. Algorithms which incorporate scientific knowledge and consume relatively few system resources are preferred over ad hoc methods. COBE exceeded its planned storage by a large and growing factor and the retrieval of data significantly affects the processing, delaying the availability of data for scientific usage and software test. Embedded compression software is planned to make the project tractable by reducing the data storage volume to an acceptable level during normal processing.
Freedman, I.↗
A data base of ASAS digital imagery
The Advanced Solid-State Array Spectroradiometer (ASAS) is an airborne, off-nadir tilting, imaging spectroradiometer that acquires digital image data for 29 spectral bands in the visible and near-infrared. The sensor is used principally for studies of the bidirectional distribution of solar radiation scattered by terrestial surfaces. ASAS has acquired data for a number of terrestial ecosystem field experiments and investigators have received over 170 radiometrically corrected, multiangle, digital image data sets. A database of ASAS digital imagery has been established in the Pilot Land Data System (PLDS) at the NASA/Goddard Space Flight Center to provide access to these data by the scientific community. ASAS, its processed data, and the PLDS are described, together with recent improvements to the sensor system.
Irons, James R.↗
Life Sciences Data Archive Scientific Development
The Life Sciences Data Archive will provide scientists, managers and the general public with access to biomedical data collected before, during and after spaceflight. These data are often irreplaceable and represent a major resource from the space program. For these data to be useful, however, they must be presented with enough supporting information, description and detail so that an interested scientist can understand how, when and why the data were collected. The goal of this contract was to provide a scientific consultant to the archival effort at the NASA-Johnson Space Center. This consultant (Jay C. Buckey, Jr., M.D.) is a scientist, who was a co-investigator on both the Spacelab Life Sciences-1 and Spacelab Life Sciences-2 flights. In addition he was an alternate payload specialist for the Spacelab Life Sciences-2 flight. In this role he trained on all the experiments on the flight and so was familiar with the protocols, hardware and goals of all the experiments on the flight. Many of these experiments were flown on both SLS-1 and SLS-2. This background was useful for the archive, since the first mission to be archived was Spacelab Life Sciences-1. Dr. Buckey worked directly with the archive effort to ensure that the parameters, scientific descriptions, protocols and data sets were accurate and useful.
Buckey, Jay C., Jr.↗
Linking NASA Environmental Data with a National Public Health Cohort Study and a CDC On-Line System to Enhance Public Health Decision Making
The overall goal of this study is to address issues of environmental health and enhance public health decision making by utilizing NASA remotely-sensed data and products. This study is a collaboration between NASA Marshall Space Flight Center, Universities Space Research Association (USRA), the University of Alabama at Birmingham (UAB) School of Public Health and the Centers for Disease Control and Prevention (CDC) National Center for Public Health Informatics. The objectives of this study are to develop high-quality spatial data sets of environmental variables, link these with public health data from a national cohort study, and deliver the linked data sets and associated analyses to local, state and federal end-user groups. Three daily environmental data sets were developed for the conterminous U.S. on different spatial resolutions for the period 2003-2008: (1) spatial surfaces of estimated fine particulate matter (PM2.5) exposures on a 10-km grid utilizing the US Environmental Protection Agency (EPA) ground observations and NASA s MODerate-resolution Imaging Spectroradiometer (MODIS) data; (2) a 1-km grid of Land Surface Temperature (LST) using MODIS data; and (3) a 12-km grid of daily Solar Insolation (SI) and maximum and minimum air temperature using the North American Land Data Assimilation System (NLDAS) forcing data. These environmental datasets were linked with public health data from the UAB REasons for Geographic and Racial Differences in Stroke (REGARDS) national cohort study to determine whether exposures to these environmental risk factors are related to cognitive decline and other health outcomes. These environmental national datasets will also be made available to public health professionals, researchers and the general public via the CDC Wide-ranging Online Data for Epidemiologic Research (WONDER) system, where they can be aggregated to the county, state or regional level as per users need and downloaded in tabular, graphical, and map formats. The linkage of these data provides a useful addition to CDC WONDER, allowing public health researchers and policy makers to better include environmental exposure data in the context of other health data available in this online system. It also substantially expands public access to NASA data, making their use by a wide range of decision makers feasible.
Al-Hamdan, Mohammad↗
GES DISC Datalist Enables Easy Data Selection For Natural Phenomena Studies
In order to investigate and assess natural hazards such as tropical storms, winter storms, volcanic eruptions, floods, and drought in a timely manner, the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) has been developing an efficient data search and access service. Called "Datalist," this service enables users to acquire their data of interest "all at once," with minimum effort. A Datalist is a virtual collection of predefined or user-defined data variables from one or more archived data sets. Datalists are more than just data. Datalists effectively provide users with a sophisticated integrated data and services package, including metadata, citation, documentation, visualization, and data-specific services (e.g., subset and OPeNDAP), all available from one-stop shopping. The predefined Datalists, created by the experienced GES DISC science support team, should save a significant amount of time that users would otherwise have to spend. The Datalist service is an extension of the new GES DISC website, which is completely data-driven. A Datalist, also known as "data bundle," is treated just as any other data set. Being a virtual collection, a Datalist requires no extra storage space.
natural hazards↗
GES DISC Datalist Enables Easy Data Selection for Natural Phenomena Studies
In order to investigate and assess natural hazards such as tropical storms, winter storms, volcanic eruptions, floods, and drought in a timely manner, the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) has been developing an efficient data search and access service. Called Datalist, this service enables users to acquire their data of interest all at once, with minimum effort. A Datalistis a virtual collection of predefined or user-defined data variables from one or more archived data sets. Datalistsare more than just data. Datalistseffectively provide users with a sophisticated integrated data and services package, including metadata, citation, documentation, visualization, and data-specific services (e.g., subset and OPeNDAP), all available from one-stop shopping. The predefined Datalists, created by the experienced GES DISC science support team, should save a significant amount of time that users would otherwise have to spend. The Datalistservice is an extension of the new GES DISC website, which is completely data-driven. A Datalist, also known as data bundle, is treated just as any other data set. Being a virtual collection, a Datalistrequires no extra storage space.
Earth events↗
Pilot climate data system user's guide
Instructions for using the Pilot Climate Data System (PCDS), an interactive, scientific data management system for locating, obtaining, manipulating, and displaying climate-research data are presented. The PCDS currently provides this supoort for approximately twenty data sets. Figures that illustrate the terminal displays which a user sees when he/she runs the PCDS and some examples of the output from this system are included. The capabilities which are described in detail allow a user to perform the following: (1) obtain comprehensive descriptions of a number of climate parameter data sets and the associated sensor measurements from which they were derived; (2) obtain detailed information about the temporal coverage and data volume of data sets which are readily accessible via the PCDS; (3) extract portions of a data set using criteria such as time range and geographic location, and output the data to tape, user terminal, system printer, or online disk files in a special data-set-independent format; (4) access and manipulate the data in these data-set-independent files, performing such functions as combining the data, subsetting the data, and averaging the data; and (5) create various graphical representations of the data stored in the data-set-independent files.
Reph, M. G.↗
A Virtual Bioinformatics Knowledge Environment for Early Cancer Detection
Discovery of disease biomarkers for cancer is a leading focus of early detection. The National Cancer Institute created a network of collaborating institutions focused on the discovery and validation of cancer biomarkers called the Early Detection Research Network (EDRN). Informatics plays a key role in enabling a virtual knowledge environment that provides scientists real time access to distributed data sets located at research institutions across the nation. The distributed and heterogeneous nature of the collaboration makes data sharing across institutions very difficult. EDRN has developed a comprehensive informatics effort focused on developing a national infrastructure enabling seamless access, sharing and discovery of science data resources across all EDRN sites. This paper will discuss the EDRN knowledge system architecture, its objectives and its accomplishments.
knowledge systems↗
DUNE Database Development
The DUNE experiment will produce vast amounts of metadata, which describe the data coming from the read-out of the primary DUNE detectors. Various databases will make up the overall DB architecture for this metadata. ProtoDUNE at CERN is the largest existing prototype for DUNE and serves as a testing ground for - among other things - possible database solutions for DUNE. The subset of all metadata that is accessed during offline data reconstruction and analysis is referred to as ‘conditions data’ and it is stored in a dedicated database. As offline data reconstruction and analysis will be deployed on HTC and HPC resources, conditions data is expected to be accessed at very high rates. It is therefore crucial to store it in a granularity that matches the expected access patterns allowing for extensive caching. This requires a good understanding of the sources and use cases of conditions data. This contribution will briefly summarize the database architecture deployed at ProtoDUNE and explain the various sources of conditions data. We will present how the conditions data is retrieved and streamed from the databases and how it is handled to match expected access patterns.
Vizcaya Hernandez, Ana Paula↗