Interactive Data Analysis of Multi-Run Performance Data
Explore the source record for details and available documents.
Computer science↗
SEARCH · Engineering Papers
Results for “data and data science”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Closing the Gap between FAIR Data Repositories and Hierarchical Data Formats
Many in the scientific community, particularly in publicly funded research, are pushing to adhere to more accessible data standards to maximize the findability, accessibility, interoperability, and reusability (FAIR) of scientific data, especially with the growing prevalence of machine learning augmented research. Online FAIR data repositories, such as the Open Science Framework (OSF), help facilitate the adoption of these standards by providing frameworks for storage, access, search, APIs, and other features that create organized hubs of scientific data. However, the wider acceptance of such repositories is hindered by the lack of support of hierarchical data formats, such as Technical Data Management Streaming (TDMS) and Hierarchical Data Format 5 (HDF5), that many researchers rely on to organize their datasets. Various tools and strategies should be used to allow hierarchical data formats, FAIR data repositories, and scientific organizations to work more seamlessly together. A pilot project at Los Alamos National Laboratory (LANL) addresses the disconnect between them by integrating the OSF FAIR data repository with hierarchical data renderers, extending support for additional file types in their framework. The multifaceted interactive renderer displays a tree of metadata alongside a table and plot of the data channels in the file. This allows users to quickly and efficiently load large and complex data files directly in the OSF webapp. Users who are browsing files can quickly and intuitively see the files in the way they or their colleagues structured the hierarchical form and immediately grasp their contents. This solution helps bridge the gap between hierarchical data storage techniques and FAIR data repositories, making both of them more viable options for scientific institutions like LANL which have been put off by the lack of integration between them.
97 MATHEMATICS AND COMPUTING↗
Automated, reliable, and efficient continental-scale replication of 7.3 petabytes of computational simulation data: A case study
We report on our experiences replicating 7.3 petabytes (PB) of Earth System Grid Federation (ESGF) computational simulation data from Lawrence Livermore National Laboratory (LLNL) in California to Argonne National Laboratory (ANL) in Illinois and Oak Ridge National Laboratory (ORNL) in Tennessee—a task motivated by a need for increased reliability, capacity, and performance. This task presented significant challenges: the need to move 29 million files twice under time pressure from aging storage hardware; a source file system bottleneck limiting throughput to 1.5 GB/s; frequent site maintenance windows; and the need for complete reliability at scale. We addressed these challenges using a simple replication tool that invoked Globus to transfer large bundles of files while tracking progress in a database, dynamically rerouting transfers to work around maintenance periods and file system limitations. Under the covers, Globus organized transfers to make efficient use of the high-speed Energy Sciences network (ESnet) and the data transfer nodes deployed at participating sites, and also addressed security, integrity checking, and recovery from a variety of transient failures. This success demonstrates the considerable benefits that can accrue from the adoption of performant data replication infrastructure. The replication tool is available at https://github.com/esgf2-us/data-replication-tools.
Globus↗
Tencoder: tensor-product encoder-decoder architecture for predicting solutions of PDEs with variable boundary data
It is widely hoped that artificial intelligence will boost data-driven surrogate models in science and engineering. However, fundamental spatial aspects of AI surrogate models remain under-studied. We investigate the ability of neural-network surrogate models to predict solutions to PDEs under variable boundary values. We do not wish to retrain the model when the boundary values change but to make them inputs to the model and infer the solution of the PDE under those boundary conditions. Such a capability is essential to making AI-based surrogate models practically useful. While simple feedforward networks are used for one-dimensional (1D) Poisson equation, an encoder-decoder architecture with a tensor-product layer is developed for the two-dimensional Poisson equation posed on a rectangular domain. We show that it is indeed possible to infer solutions to PDEs from variable boundary data using neural networks in this relatively simple setting, and point to future directions.
Kashi, Aditya↗
Position Papers for the ASCR Workshop on the Management and Storage of Scientific Data
The purpose of this workshop is to identify priority research directions in the area of data management for high-performance and scientific computing above and beyond HPC’s traditional "the parallel file system is the data-management system" model. Supporting the breadth of the DOE mission, including the explosion of AI uses and the growing needs of experimental and observational science, motivates revisiting our assumptions about data management. There are many facets of this topic to explore including: (1) Interfaces for accessing data that resides on traditional persistent storage as well as memory devices; (2) Storage-system architecture design that supports scientific workflows on varied hierarchical storage and networking devices; (3) Devising metadata management infrastructure to support FAIR principles (Findability, Accessibility, Interoperability, and Reusability); (4) Capturing provenance information about scientific data; (5) Utilizing AI to learn I/O patterns of emerging workloads for efficient data management; (6) Providing data management support for AI and complex workflows; and (7) Understanding the overlap between traditional storage systems and I/O (SSIO) efforts and data management. While the program committee has identified these topics as important areas for discussion, we welcome position papers from the community that propose additional topics of interest for discussion at the workshop. The workshop agenda will include breakout sessions for discussing these and selected topic areas to inform priority research directions for data management for high-performance and scientific computing.
97 MATHEMATICS AND COMPUTING↗
Report for the ASCR Workshop on the Management and Storage of Scientific Data
The purpose of this workshop is to identify priority research directions in the area of data management for high-performance and scientific computing above and beyond HPC’s traditional "the parallel file system is the data-management system" model. Supporting the breadth of the DOE mission, including the explosion of AI uses and the growing needs of experimental and observational science, motivates revisiting our assumptions about data management. There are many facets of this topic to explore including: (1) Interfaces for accessing data that resides on traditional persistent storage as well as memory devices; (2) Storage-system architecture design that supports scientific workflows on varied hierarchical storage and networking devices; (3) Devising metadata management infrastructure to support FAIR principles (Findability, Accessibility, Interoperability, and Reusability); (4) Capturing provenance information about scientific data; (5) Utilizing AI to learn I/O patterns of emerging workloads for efficient data management; (6) Providing data management support for AI and complex workflows; and (7) Understanding the overlap between traditional storage systems and I/O (SSIO) efforts and data management. While the program committee has identified these topics as important areas for discussion, we welcome position papers from the community that propose additional topics of interest for discussion at the workshop. The workshop agenda will include breakout sessions for discussing these and selected topic areas to inform priority research directions for data management for high-performance and scientific computing.
97 MATHEMATICS AND COMPUTING↗
Reduction of detection limit and quantification uncertainty due to interferent by neural classification with abstention
Many measurements in the physical sciences can be cast as counting experiments, where the number of occurrences of a physical phenomenon informs the prevalence of the phenomenon's source. Often, detection of the physical phenomenon (termed signal) is difficult to distinguish from naturally occurring phenomena (termed background). In this case, the discrimination of signal events from background can be performed using classifiers, and they may range from simple, threshold-based classifiers to sophisticated neural networks. These classifiers are often trained and validated to obtain optimal accuracy, however we show that the optimal accuracy classifier does not generally coincide with a classifier that provides the lowest detection limit, nor the lowest quantification uncertainty. Here, we present a derivation of the detection limit and quantification uncertainty in the classifier-based counting experiment case. We also present a novel abstention mechanism to minimize the detection limit or quantification uncertainty a posteriori. We illustrate the method on two data sets from the physical sciences, discriminating Ar-37 and Ar-39 radioactive decay from non-radioactive events in a gas proportional counter, and discriminating neutrons from gammas in an inorganic scintillator and report results therefrom.
46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
The Geothermal Data Repository: Ten Years of Supporting the Geothermal Industry with Open Access to Geothermal Data: Preprint
The Department of Energy's (DOE) Geothermal Data Repository (GDR) is celebrating its tenth anniversary! Over the last decade it has grown from the simple idea of storing public data in a centralized location to a valuable tool at the center of the US geothermal scientific community and an integral part of the DOE Geothermal Technologies Office (DOE GTO) project management strategy. Researchers funded by the DOE GTO have contributed over 1,300 data submissions to the GDR. These data have been used to further advancements in geothermal science, economic analysis, exploration, research, development, and operational efficiency. The adoption of open data methodologies and a data management strategy that prioritizes universal open access and standardized, interoperable data have further increased the value of GDR data, making them available across a distributed network of data sharing partners and improving their utility to other industries and related fields, including material science and space exploration. Incorporating feedback from users has been critical to the GDRs success, allowing it to grow over the years to meet the evolving needs of the geothermal community. This paper will explore some of many changes that occurred throughout the GDRs tenure and the lessons learned along the way, as well as highlight some of the new features and recent improvements that been implemented to support innovation, reduce duplication of effort, and advance the geothermal industry as a whole.
accessibility↗
The Geothermal Data Repository: Ten Years of Supporting the Geothermal Industry with Open Access to Geothermal Data
The Department of Energy's (DOE) Geothermal Data Repository (GDR) is celebrating its tenth anniversary! Over the last decade it has grown from the simple idea of storing public data in a centralized location to a valuable tool at the center of the US geothermal scientific community and an integral part of the DOE Geothermal Technologies Office (DOE GTO) project management strategy. Researchers funded by the DOE GTO have contributed over 1,300 data submissions to the GDR. These data have been used to further advancements in geothermal science, economic analysis, exploration, research, development, and operational efficiency. The adoption of open data methodologies and a data management strategy that prioritizes universal open access and standardized, interoperable data have further increased the value of GDR data, making them available across a distributed network of data sharing partners and improving their utility to other industries and related fields, including material science and space exploration. Incorporating feedback from users has been critical to the GDR's success, allowing it to grow over the years to meet the evolving needs of the geothermal community. This paper will explore some of many changes that occurred throughout the GDRs tenure and the lessons learned along the way, as well as highlight some of the new features and recent improvements that been implemented to support innovation, reduce duplication of effort, and advance the geothermal industry as a whole.
access↗
Mitigation of LEO Satellite Brightness and Trail Effects on the Rubin Observatory LSST
In this work, we report studies on the mitigation of optical effects of bright low-Earth-orbit (LEO) satellites on Vera C. Rubin Observatory and its Legacy Survey of Space and Time (LSST). These include options for pointing the telescope to avoid satellites, laboratory investigations of bright trails on the Rubin Observatory LSST camera sensors, algorithms for correcting image artifacts caused by bright trails, experiments on darkening SpaceX Starlink satellites, and ground-based follow-up observations. The original Starlink v0.9 satellites are g ~ 4.5 mag, and the initial experiment "DarkSat" is g ~ 6.1 mag. Future Starlink darkening plans may reach g ~ 7 mag, a brightness level that enables nonlinear image artifact correction to well below background noise. However, the satellite trails will still exist at a signal-to-noise ratio ~ 100, generating systematic errors that may impact data analysis and limit some science. For the Rubin Observatory 8.4 m mirror and a satellite at 550 km, the full width at half maximum of the trail is about 3'' as the result of an out-of-focus effect, which helps avoid saturation by decreasing the peak surface brightness of the trail. For 48,000 LEOsats of apparent magnitude 4.5, about 1% of pixels in LSST nautical twilight images would need to be masked.
79 ASTRONOMY AND ASTROPHYSICS↗
TNet: A Model-Constrained Tikhonov Network Approach for Inverse Problems
Deep learning (DL), in particular deep neural networks, by default is purely data-driven and in general does not require physics. This is the strength of DL but also one of its key limitations when applied to science and engineering problems in which underlying physical properties—such as stability, conservation, and positivity—and accuracy are required. DL methods in their original forms are often not capable of respecting the underlying mathematical models or achieving desired accuracy even in big-data regimes. On the other hand, many data-driven science and engineering problems, such as inverse problems, typically have limited experimental or observational data, and DL would overfit the data in this case. Leveraging information encoded in the underlying mathematical models, we argue, not only compensates for missing information in low data regimes but also provides opportunities to equip DL methods with the underlying physics, hence promoting better generalization. This paper develops a model-constrained DL approach and its variant TNet—a Tikhonov neural network—which are capable of learning not only information hidden in the training data but also in the underlying mathematical models to solve inverse problems governed by partial differential equations in low data regimes. We provide the constructions and some theoretical results for the proposed approaches for both linear and nonlinear inverse problems. Since TNet is designed to learn inverse solutions with Tikhonov regularization, it is interpretable: in fact it recovers Tikhonov solutions for linear cases while potentially approximating Tikhonov solutions for nonlinear inverse problems. We also prove that data randomization can enhance not only the smoothness of the networks but also their generalizations. Comprehensive numerical results confirm the theoretical findings and show that with even as little as 1 training data sample for one-dimensional (1D) deconvolution, 5 for an inverse 2D heat conductivity problem, 100 for inverse initial conditions for a time-dependent 2D Burgers’s equation, and 50 for inverse initial conditions for 2D Navier–Stokes equations, TNet solutions can be as accurate as Tikhonov solutions while being several orders of magnitude faster. Furthermore, this is possible owing to the model-constrained term, replications, and randomization.
97 MATHEMATICS AND COMPUTING↗
Improved Models of Long-Term Creep Behavior of High Performance Structural Alloys
In this program, QuesTek Innovations LLC, a leader in the field of computational materials design, proposes to develop a robust creep-modeling toolkit that expands its computational Materials by Design® technology, in order to predict the long-term creep performance of materials for base alloys and weldments in fossil energy systems under wide thermal and mechanical conditions. The Material Database Group of NIMS (National Institute for Material Science, JAPAN) provides reliable public data for long-term creep behavior over 300,000 hours (> 34 years) for a variety of materials including high Cr steels along with seminal work performed by EPRI (Electric Power Research Institute) on creep failure of these steels. QuesTek would rely on these premier institutions for experimental data needed to calibrate models. The developed tool has wider applications than the state-of-the-art modeling tools. It includes the effects of chemistry variations, microstructure variation (weldments) etc .which is lacking in many available models requiring the model to be calibrated to large number of datasets for accurate predictions. A model based on fundamental mechanism has promises in application to different materials, other than the demonstrated material (Grade 91 steels) in the current program. As a result of such an effort, we envision greatly advancing the state-of-the-art modeling tools for creep life assessments of different materials.
20 FOSSIL-FUELED POWER PLANTS↗
Prediction of Distributed River Sediment Respiration Rates Using Community-Generated Data and Machine Learning
River sediment microbial respiration is a key indicator of ecosystem functioning and the biogeochemical fluxes across this critical zone link surface and subsurface waters. As such, there is tremendous interest in measuring and mapping these respiration rates. Respiration observations are expensive and labor intensive; there is limited data available to the community. An open science, collaborative initiative is collecting samples for respiration rate analysis and multi-scale metadata; this evolving data set is being used for making machine learning (ML) predictions at unsampled sites to help inform continued community engagement. However, it is a challenge to find an optimum configuration for ML models to work with this feature-rich (i.e., 100+ possible input variables) data set. Here, we present results from a two-tiered approach to managing the analysis of this complex data set: (a) a stacked ensemble of models that automatically optimizes hyperparameters and manages the training of many models and (b) feature permutation importance to detect the most important features in the models. The major elements of this workflow are modular, portable, open, and cloud-based thus making this implementation a potential template for other applications. The models developed here predict that sediment organic matter chemistry is one of the most important features for predicting sediment respiration rate. Other larger-scale, important features fall into the categories of climatic, ecological, geological, and fluvial settings. Leveraging these larger-scale features to generate data-driven estimates of river sediment respiration rates reveals spatially consistent but heterogeneous patterns across the river network of the Columbia River Basin.
54 ENVIRONMENTAL SCIENCES↗
Data Federation Challenges in Remote Near-Real-Time Fusion Experiment Data Processing
Fusion energy experiments and simulations provide critical information needed to plan future fusion reactors. As next-generation devices like ITER move toward long-pulse experiments, analyses, including AI and ML, should be performed in a wide range of time and computing constraints, from near-real-time constraints, between-shot analysis, and to campaign-wide long-term analysis. However, the data volume, velocity, and variety make it extremely challenging for analyses using only local computational resources. Researchers need the ability to compose and execute workflows spanning edge resources to large-scale high-performance computing facilities.We present Delta, a system to address data analysis challenges, including AI/ML, in fusion science, by leveraging the ADIOS I/O library and middleware, to support executing science workflows over the wide area network for near-real-time streaming. We discuss the data federation challenges in performing remote workflows, focusing on on-going research work in (1) managing, reducing, and streaming data to minimize I/O and data movement overheads, (2) decompressing and reorganizing data for analysis, and (3) executing workflows for automated data analysis. We introduce examples for deep-learning based data analysis for the fusion domain and demonstrate how we use Delta to construct end-to-end workflows for a fusion device in Korea, connecting a remote DOE facility in the USA. The capability demonstrated by this project is the basis for improving the state of the art for near-real-time data federation amongst remote facilities.
Choi, Jong Youl↗
University Data Management Pilot Utilizing the Nuclear Research Data System
Background In 2022, the Office of Science and Technology Policy (OSTP) issued a memo that significantly reshaped the landscape of access to federally funded research. The memo mandated that all taxpayer-funded research be made available to the public without delay upon publication, without an embargo period, superseding the 2013 OSTP public access policy. This public access policy promotes transparency and the democratization of knowledge, ensuring that the fruits of scientific endeavors funded by federal agencies could be immediately accessed and built upon by scientists, educators, students, and the public at large. To implement the requirements of the OSTP guidance and DOE Public Access Plan, the Office of Nuclear Energy (NE) has implemented public access plan guidance and has identified several areas where better data management practices would further expand public access to important nuclear energy related scientific data, reports, and other technical products. Significant NE supported efforts are already underway for data management and public access to important nuclear energy related data.1 2 To address gaps in data management practices, and improve retention and accessibility of data, NE is actively exploring enhanced data management options utilizing its high-performance computing resources administered by its Nuclear Scientific User Facility Program. A newly piloted system, the Nuclear Research Data System (NRDS) acts as a portal for data collection and dissemination. Nuclear Energy University Program Research and Development Portfolio According to Web of Science, NEUP has produced 2,345 journal publication that have been cited more than 61,000 times3 and countless conference proceedings. These publications are publicly available through OSTI.gov and in the open literature. Additional scientific and technical products including project milestones that are not publications and NEUP project final reports are vetted through OSTI.gov and released once reviewed and approved by DOE. Since 2009, NEUP has awarded close to 1,000 different R&D projects in technical areas across the NE research programs. As of June 2023, 512 NEUP reports are publicly available on OSTI. The underlying data for projects is still held at universities, and data transfer, co-location, and dissemination has not occurred in a systematic way. NEUP data is currently accessible through myriad university-based data repositories, or through direct requests to PIs. The program identified this patchwork of repositories, or often lack of publicly available data, as a significant barrier to an organized, accessible, and comprehensive solution to sharing data with the larger nuclear energy community. Approach The goal of this pilot project is to establish a pathway to a consolidated long-term repository for NEUP project data. To accomplish this goal, the pilot strives to accomplish the following objectives: Establish data collection standards, including a standard set of required supplementary information to contextualize and support raw data files. Work with the HPC group collect and upload information and to modify the NRDS system, as needed, to support a standardized approach. Resolve potential barriers to successful roll out of an expanded data collection strategy, including modifying data management plan guidelines and establishing a document and data release process that accounts for potential intellectual property and/or export control concerns. Results Overall, the pilot was successful in collecting 8,982 raw and processes data files, 220 reports, 56 calibration files, and 5,931 other supplementary documents. Supplementary documents included experimental plans, methods, journal publications and conference proceedings, milestone reports, and final reports. Figure 2 shows the number of data sets and supplementary project information provided by each project. Projects has significantly different input, depending on experimental data produced and completeness of the datasets provided.
Data collection↗
Constraining the impact of chlorine as a neutron absorber in next-gen fast reactor designs
The role of chlorine as a neutron poison and as a seed for producing radioactive waste in nuclear systems has driven a renewed interest to improve its nuclear data uncertainties. Additionally, basic and applied science programs that use CLYC (Cs 2 LiYCl 6 :Ce) detectors for neutron spectroscopy and monitoring are also very sensitive to any change in chlorine nuclear data for simulations of the detector response. In this work, sensitivities relevant for these different applications are addressed through simulations of the efficiency of CLYC detectors in a fast fission spectrum when applying new chlorine nuclear data as input. These simulations are validated by an experimental measurement using CLYC detectors coupled to an ionization chamber loaded with a 252 Cf spontaneous fission source. The results are then used to obtain the first reliable direct measurement of the 35 Cl(n,p 0 ) and summed Cl(n,p+n,α) fission spectrum average cross sections, found to be 54.7(32) and 105.0(98) mb, respectively. The results are within uncertainty of calculated fission spectrum averaged cross sections based on recently re-evaluated chlorine nuclear data, which confirm recent impact studies performed for the Molten Chloride Reactor Experiment. Meanwhile, there currently exists only one published criticality benchmark experiment that is sufficiently sensitive to chlorine nuclear data. Discrepancies are found with this set of criticality safety benchmarks, which are more sensitive to thermal and epithermal neutron energies than the energies, above 100 keV, tested in this current work. Hence, there is still a need to re-evaluate the chlorine nuclear data at lower energies to assess these discrepancies. Interpretation of the data from future “faster” criticality benchmarks, which are needed for next-gen fast reactor designs, benefit from the improved constraints on the chlorine nuclear data validated in this work.
73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
Volatile Organic Compounds and Meteorological Conditions in the Missouri Ozark AmeriFlux (MOFLUX) Site, 2023
This data set contains measurements of atmospheric components and meteorological conditions of central Missouri, United States during the summer of 2023. Data serve to examine the impact of meteorological conditions relevant to future climate on the emission and transformation of volatile organic compounds (VOCs). During the field campaign, researchers were also able to incorporate opportunistic analyses of the long-range transport of smoke plumes generated from extreme forest fire activities in Canada. VOC measurements were conducted at the Missouri Ozark AmeriFlux (MOFLUX) site (latitude 38.7441, longitude −92.2000) using a proton transfer reaction time of flight mass spectrometer (PTR-ToF-MS 6000 X2). The sampling campaign was conducted during the summer of 2023 (2023-06-25 to 2023-08-12) with measurements being taken at high temporal resolution (1 hour). Particular VOCs analyzed include: Methanol, Acetonitrile, Acetone, Isoprene, Methylvinyl ketone (MVK) and methacrolein (MACr), Benzene, Toluene, Catechol, and Monoterpene. Meteorological parameters included in this dataset were collected from a nearby Columbia Regional Airport (~10 km). Global solar radiation data were measured at a weather site in Ashland, MO, 5.22 km from the MOFLUX tower. The data were accessed using the MesoWest online website (https://mesowest.utah.edu/) provided by the Department of Atmospheric Sciences, University of Utah. Discussion of the data providers, database, and dissemination were highlighted in prior studies (Horel et al., 2002a; Horel et al., 2002b) . Smoke mixing ratios (in mg m−3) were estimated from the High-Resolution Rapid Refresh (HRRR) 3 km weather model for Missouri at 6-hour intervals (Dowell et al., 2022). This dataset contains three data files in comma separate (*.csv) format. Additional metadata are provided: three data dictionaries and a file-level metadata file in comma separate (*.csv) format and a user guide in PDF (*.pdf) format.
Acetone↗
A representation-independent electronic charge density database for crystalline materials
In addition to being the core quantity in density functional theory, the charge density can be used in many tertiary analyses in materials sciences from bonding to assigning charge to specific atoms. The charge density is data-rich since it contains information about all the electrons in the system. With increasing utilization of machine-learning tools in materials sciences, a data-rich object like the charge density can be utilized in a wide range of applications. The database presented here provides a modern and user-friendly interface for a large and continuously updated collection of charge densities as part of the Materials Project. In addition to the charge density data, we provide the theory and code for changing the representation of the charge density which should enable more advanced machine-learning studies for the broader community.
36 MATERIALS SCIENCE↗