Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data science workflows”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10

Earth Science Data Processing With Nextflow

Earth science data processing tasks present many challenges. These tasks often process large input datasets and require scores of CPU-hours to generate results. All but the simplest tasks will be decomposed into a series of computational or data manipulation steps, also known as a scientific workflow. In order to reduce the burden of orchestrating and running the dependent processing steps, a workflow execution engine is required. This poster describes the lessons learned by the CLARREO Pathfinder (CPF) team while developing multiple scientific workflows and utilizing the open-source Nextflow engine to execute them in a cloud computing environment. The Nextflow engine is designed with the following stated goals: first, the engine does not dictate how individual steps in the task are implemented (i.e. it is language and interface agnostic); second, the engine supports easy configuration and modularity at the workflow level so that others can easily execute our workflows to reproduce results; lastly, the engine eases development by transparently scaling execution from local to remote environments. Nextflow was developed for the bioinformatics domain but is a good fit for other scientific workflows where the overall task is well-described by a dataflow diagram. The CPF team has developed Nextflow pipelines (i.e. scientific workflows) to simulate CLARREO radiance, generate large look-up tables for inter-calibration algorithms, and generate L4 intercalibration data products. These pipelines consume from single-digits to hundreds of thousands of CPU-hours. In the development and evolution of these pipelines we have discovered many design patterns, pitfalls, and solutions to common problems. Our goal is to demonstrate important aspects of how to design, implement, run, and ultimately share Nextflow pipelines in the domain of Earth science.

Aron D Bartle↗

Materials characterization: Can artificial intelligence be used to address reproducibility challenges?

Material characterization techniques are widely used to characterize the physical and chemical properties of materials at the nanoscale and, thus, play central roles in material scientific discoveries. However, the large and complex datasets generated by these techniques often require significant human effort to interpret and extract meaningful physicochemical insights. Artificial intelligence (AI) techniques such as machine learning (ML) have the potential to improve the efficiency and accuracy of surface analysis by automating data analysis and interpretation. In this perspective paper, we review the current role of AI in surface analysis and discuss its future potential to accelerate discoveries in surface science, materials science, and interface science. We highlight several applications where AI has already been used to analyze surface analysis data, including the identification of crystal structures from XRD data, analysis of XPS spectra for surface composition, and the interpretation of TEM and SEM images for particle morphology and size. We also discuss the challenges and opportunities associated with the integration of AI into surface analysis workflows. These include the need for large and diverse datasets for training ML models, the importance of feature selection and representation, and the potential for ML to enable new insights and discoveries by identifying patterns and relationships in complex datasets. Most importantly, AI analyzed data must not just find the best mathematical description of the data, but it must find the most physical and chemically meaningful results. In addition, the need for reproducibility in scientific research has become increasingly important in recent years. The advancement of AI, including both conventional and the increasing popular deep learning, is showing promise in addressing those challenges by enabling the execution and verification of scientific progress. By training models on large experimental datasets and providing automated analysis and data interpretation, AI can help to ensure that scientific results are reproducible and reliable. Although integration of knowledge and AI models must be considered for the transparency and interpretability of models, the incorporation of AI into the data collection and processing workflow will significantly enhance the efficiency and accuracy of various surface analysis techniques and deepen our understanding at an accelerated pace.

Materials Science↗

Cross-Cutting Software Solutions in Support of Experimental Analysis Challenges at National Scattering Facilities

Rapid improvements in instrumentation at neutron and light source combined with new capabilities in computing, data analytics, and machine learning offer unprecedented scientific capabilities to address fundamental and applied science questions in a wide area of subjects from quantum materials to biological systems. In order to make these capabilities accessible to the broader scientific community, significant developments are needed related to workflow spanning from experimental control to advanced materials simulations as well as the underpinning scientific software. In this chapter, we explore these needs and recent developments at the Spallation Neutron Source (SNS) at Oak Ridge National Laboratory. A central part of the workflow at the SNS is the data reduction framework Mantid which is developed as a partnership between European and US neutron-scattering facilities.

Proffen, Thomas↗

Where are the Data? Automating a Workflow for Carbon Storage Data Gap Analyses

This presentation demonstrates a spatial analysis workflow to assess data availability for the many components of geologic carbon storage technical viability. The workflow relies upon a knowledge-data framework that links the different components of GCS technical viability to the data types needed for evaluation. Using this contextual information, a combination of data science methods (e.g., natural language processing) and spatial analyses are applied to identify areas where sufficient data exists for a given component. The results are aggregated into maps illustrating data density and spatial gaps across all technical viability factors and data categories, as well as the individual component and category level for a more nuanced understanding. Presented at the FECM NETL Carbon Management Program Review Meeting 2024.

Creason, Christopher↗

Where are the Data? Automating a Workflow for Carbon Storage Data Gap Analyses

This presentation demonstrates a spatial analysis workflow to assess data availability for the many components of geologic carbon storage technical viability. The workflow relies upon a knowledge-data framework that links the different components of GCS technical viability to the data types needed for evaluation. Using this contextual information, a combination of data science methods (e.g., natural language processing) and spatial analyses are applied to identify areas where sufficient data exists for a given component. The results are aggregated into maps illustrating data density and spatial gaps across all technical viability factors and data categories, as well as the individual component and category level for a more nuanced understanding. Presented at the Geological Society of America Connects 2024 Annual Meeting in Anaheim, California, 22-25 September 2024.

Creason, Christopher↗

A large-scale benchmarking of deterministic and stochastic derivative-free optimization algorithms

This presentation summarizes our work in the PrOMMiS project on benchmarking of data-driven optimization algorithms and their applications in self-driving laboratories. This work supports the broader project goal of accelerating the identification of promising separation methods and operating conditions for critical minerals separation processes. We present a systematic benchmarking study of 42 data-driven optimization algorithms on a broad collection of 502 test problems. The results identify BAM, GLCCLUSTER, and MULTIMIN as the most effective optimization solvers, with BAM showing the highest overall performance and solving more than 80% of the benchmark problems. The study also shows that no single solver consistently outperforms the others across all problem types, indicating that our future laboratory applications may benefit from using a small set of strong solvers rather than relying on a single method. The presentation also illustrates an in-silico chemical reactor case study showing that data-driven optimization methods can guide autonomous experimentation in a self-driving laboratory and identify optimal operating conditions within a small number of experiments. Overall, the results provide a basis for selecting efficient optimization methods and demonstrate the practical use of data-driven optimization in self-driving laboratory workflows.

36 MATERIALS SCIENCE↗

Algorithm-guided experimentation for autonomous AI systems in self-driving laboratories

This presentation summarizes our work in the PrOMMiS project on benchmarking of data-driven optimization algorithms and their applications in self-driving laboratories. This work supports the broader project goal of accelerating the identification of promising separation methods and operating conditions for critical minerals separation processes. We present a systematic benchmarking study of 42 data-driven optimization algorithms on a broad collection of 502 test problems. The results identify BAM, GLCCLUSTER, and MULTIMIN as the most effective optimization solvers, with BAM showing the highest overall performance and solving more than 80% of the benchmark problems. The study also shows that no single solver consistently outperforms the others across all problem types, indicating that our future laboratory applications may benefit from using a small set of strong solvers rather than relying on a single method. The presentation also illustrates an in-silico chemical reactor case study showing that data-driven optimization methods can guide autonomous experimentation in a self-driving laboratory and identify optimal operating conditions within a small number of experiments. Overall, the results provide a basis for selecting efficient optimization methods and demonstrate the practical use of data-driven optimization in self-driving laboratory workflows.

36 MATERIALS SCIENCE↗

An Integrated Data Analytics Platform

An Integrated Science Data Analytics Platform is an environment that enables the confluence of resources for scientific investigation. It harmonizes data, tools and computational resources which subsequently enable the research community to focus on the investigation rather than spending time on security, data preparation, management, etc. OceanWorks is a NASA technology integration project to establish a cloud-based Integrated Ocean Science Data Analytics Platform at NASA’s Physical Oceanography Distributed Active Archive Center (PO.DAAC) for big ocean science. It focuses on advancement and maturity by bringing together several NASA open-source, big data projects for parallel analytics, anomaly detection, in-situ to satellite data matchup, quality-screened data subsetting, search relevancy, and data discovery. Our communities are relying on data distributed through data centers such as the PO.DAAC, COAPS, NCAR, and many others to conduct their research. In typical investigations, scientists would engage in: search for data, evaluate the relevance of that data, download it, and then apply algorithms to identify trends. Such workflow cannot scale if the research involves a massive amount of data or multi-variate measurements. NASA’s Surface Water and Ocean Topography (SWOT) mission is expected to produce massive amount of observational data during its 3-year nominal mission. Collections like SWOT challenges all existing Earth Science data archival, distribution and analysis paradigms. In this paper, we will discuss how OceanWorks enhances the analysis of physical ocean data where the computation is done on an elastic cloud platform next to the archive to deliver fast, web-accessible services for working with oceanographic measurements.

Yang, Chaowei↗

ExaFEL: extreme-scale real-time data processing for X-ray free electron laser science

ExaFEL is an HPC-capable X-ray Free Electron Laser (XFEL) data analysis software suite for both Serial Femtosecond Crystallography (SFX) and Single Particle Imaging (SPI) developed in collaboration with the Linac Coherent Lightsource (LCLS), Lawrence Berkeley National Laboratory (LBNL) and Los Alamos National Laboratory. ExaFEL supports real-time data analysis via a cross-facility workflow spanning LCLS and HPC centers such as NERSC and OLCF. Our work therefore constitutes initial path-finding for the US Department of Energy's (DOE) Integrated Research Infrastructure (IRI) program. We present the ExaFEL team's 7 years of experience in developing real-time XFEL data analysis software for the DOE's exascale supercomputers. We present our experiences and lessons learned with the Perlmutter and Frontier supercomputers. Furthermore we outline essential data center services (and the implications for institutional policy) required for real-time data analysis. Finally we summarize our software and performance engineering approaches and our experiences with NERSC's Perlmutter and OLCF's Frontier systems. This work is intended to be a practical blueprint for similar efforts in integrating exascale compute resources into other cross-facility workflows.

59 BASIC BIOLOGICAL SCIENCES↗

A machine learning approach for particle accelerator errant beam prediction using spatial phase deviation

Particle accelerators are extremely complex systems that are expected to operate on high availability. Predicting impending failures only by utilizing data collected from diagnostic equipment already on board can help operators to avoid installing expensive sensors, unscheduled downtime and associated costs. For this purpose we explore the predictive power of Machine Learning algorithms to detect faulty beams prior to the failure. In this study, we propose a Machine Learning approach to model mapping from a pair of sensors located across the accelerator. While the model is trained to represent normal operation, we evaluate the predictive performance on known faulty beam pulses. We also investigate the model performance on unseen data through k-fold cross-validation. Then we recap the analysis with a neural architecture search and hyperparameter optimization study to fine tune our initial model. In conclusion, this paper will also introduce a sustainable framework that can standardize Machine Learning workflow applied to particle accelerators.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Cultivating an Emergent Earth Observation Analytics Ecosystem in the Cloud

A diverse set of data analytics systems for Earth Observations are sprouting up in the Earth Science community, with a wealth of processing algorithms and analysis methods. There is a similar wealth of data resources available via myriad data providers and clearinghouses, including large institutional systems like the Earth Observing System Data and Information System, Comprehensive Large Scale Array-data Stewardship System, and Federated Earth Observation Missions gateway. With Earth system science driving a need to work with more datasets together, and the community developing more analysis tools (some of them dataset-specific), how can we develop analysis workflows that incorporate far-flung datasets and leverage analysis resources from multiple organizations? Cloud computing points the way toward a solution in two different respects. Firstly, the access to and abstraction of virtually unlimited storage and computing power provides an environment that enables more straightforward means of pulling datasets and analysis resources together. Just as importantly, however, cloud computing serves as an example of an "ecosystem" of interoperating services, since the essence of cloud computing is the presentation of all resources as a service, from hardware to infrastructure to platform to software. This enables the combination of off-the-shelf, diverse services to construct entire systems that emerge out of an equally diverse community of architects and developers. This approach can be similarly applied to the data and analysis resources in the Earth Observation community. By exposing these resources via well understood services, and consuming resources in the same way, different organizations can construct bespoke analysis workflows and systems for their own purposes. The key leap the community needs to make is to develop analysis systems in components that interact with other components via services. The result would be a rich ecosystem of analytics components that can be combined to analyze datasets at scale and in conjunction with other datasets from other sources.

chaos↗

Transitioning from File-Based HPC Workflows to Streaming Data Pipelines with openPMD and ADIOS2

This paper aims to create a transition path from file-based IO to streaming-based workflows for scientific applications in an HPC environment. By using the openPMP-api, traditional workflows limited by filesystem bottlenecks can be overcome and flexibly extended for in situ analysis. The openPMD-api is a library for the description of scientific data according to the Open Standard for Particle-Mesh Data (openPMD). Its approach towards recent challenges posed by hardware heterogeneity lies in the decoupling of data description in domain sciences, such as plasma physics simulations, from concrete implementations in hardware and IO. The streaming backend is provided by the ADIOS2 framework, developed at Oak Ridge National Laboratory. This paper surveys two openPMD-based loosely-coupled setups to demonstrate flexible applicability and to evaluate performance. In loose coupling, as opposed to tight coupling, two (or more) applications are executed separately, e.g. in individual MPI contexts, yet cooperate by exchanging data. This way, a streaming-based workflow allows for standalone codes instead of tightly-coupled plugins, using a unified streaming-aware API and leveraging high-speed communication infrastructure available in modern compute clusters for massive data exchange. We determine new challenges in resource allocation and in the need of strategies for a flexible data distribution, demonstrating their influence on efficiency and scaling on the Summit compute system. The presented setups show the potential for a more flexible use of compute resources brought by streaming IO as well as the ability to increase throughput by avoiding filesystem bottlenecks.

Poeschel, Franz↗

Image-Based Methods to Score Fungal Pathogen Symptom Progression and Severity in Excised Arabidopsis Leaves

Image-based symptom scoring of plant diseases is a powerful tool for associating disease resistance with plant genotypes. Advancements in technology have enabled new imaging and image processing strategies for statistical analysis of time-course experiments. There are several tools available for analyzing symptoms on leaves and fruits of crop plants, but only a few are available for the model plant Arabidopsis thaliana (Arabidopsis). Arabidopsis and the model fungus Botrytis cinerea (Botrytis) comprise a potent model pathosystem for the identification of signaling pathways conferring immunity against this broad host-range necrotrophic fungus. Here, we present two strategies to assess severity and symptom progression of Botrytis infection over time in Arabidopsis leaves. Thus, a pixel classification strategy using color hue values from red-green-blue (RGB) images and a random forest algorithm was used to establish necrotic, chlorotic, and healthy leaf areas. Secondly, using chlorophyll fluorescence (ChlFl) imaging, the maximum quantum yield of photosystem II (Fv/Fm) was determined to define diseased areas and their proportion per total leaf area. Both RGB and ChlFl imaging strategies were employed to track disease progression over time. This has provided a robust and sensitive method for detecting sensitive or resistant genetic backgrounds. A full methodological workflow, from plant culture to data analysis, is described.

59 BASIC BIOLOGICAL SCIENCES↗

Automated Resonance Fitting for Nuclear Data Evaluation

Global and national efforts to deliver high-quality nuclear data to users have a wide-ranging impact, affecting applications in national security, reactor operations, basic science, medicine, and more. Cross section evaluation is a major part of this effort, combining theory and experimentation to produce recommended values and uncertainties for reaction probabilities. Resonance region evaluation is a specialized type of nuclear data evaluation that can require significant manual effort and months of time from expert scientists. In this article, non-convex non-linear optimization methods are combined with concepts of inferential statistics to infer a resonance model from experimental data in an automated manner that is not dependent on prior evaluation(s). This methodology aims to enhance the workflow of a resonance evaluator by minimizing time, effort, and the potential for bias from prior assumptions, while enhancing reproducibility and documentation, thereby addressing well-known challenges in the field.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Pypromice: A Python Package for Processing Automated Weather Station Data

The pypromice Python package is for processing and handling observation datasets from automated weather stations (AWS). It is primarily aimed at users of AWS data from the Geological Survey of Denmark and Greenland (GEUS), which collects and distributes in situ weather station observations to the cryospheric science research community. Functionality in pypromice is primarily handled using two key open-source Python packages, xarray (Hoyer & Hamman, 2017) and pandas (The pandas development team, 2020). A defined processing workflow is included in pypromice for transforming original AWS observations (Level 0, L0) to a usable, CF-convention-compliant dataset (Level 3, L3) (Figure 1). Intermediary processing levels (L1,L2) refer to key stages in the workflow, namely the conversion of variables to physical measurements and variable filtering (L1), cross-variable corrections and user-defined data flagging and fixing (L2), and derived variables (L3). Information regarding the station configuration is needed to perform the processing, such as instrument calibration coefficients and station type (one-boom tripod or two-boom mast station design, for example), which are held in a toml configuration file. Two example configuration files are provided with pypromice , which are also used in the package’s unit tests. More detailed documentation of the AWS design, instrumentation, and processing steps are described in Fausto et al. (2021).

pypromice↗

leapR: An R Package for Multiomic Pathway Analysis

A generalized goal of many high-throughput data studies is to identify functional mecha-nisms that underlie observed biological phenomena, whether disease outcomes or metabolic out-put. Increasingly, studies that rely on multiple sources of high-throughput data (genomic, tran-scriptomic, proteomic, metabolomic) are faced with a challenge of utilizing the data in a way that maximizes utility. However, methods for integration of multiple forms of molecular data into a biolog-ically coherent frameworks are needed. Furthermore, we have developed a framework to assess biological pathway activity that relates to phenotypic outcome using multi-source data. Availability and implementation: The leapR package with user manual and example workflow is available for download from GitHub (https://github.com/biodataganache/leapR).

59 BASIC BIOLOGICAL SCIENCES↗

Science-informed Machine Learning to Increase Recovery Efficiency in Unconventional Reservoirs

Production of hydrocarbons from fractured, unconventional reservoirs is inherently inefficient. But machine learning offers a pathway both to increasing recovery efficiency at a site and to improving forecasts of production, thereby improving the economics of operations in unconventional reservoirs. Los Alamos—in partnership with DOE, NETL, and WVU—has been developing a science-informed workflow and platform for optimizing pressure-drawdown at a site, which will allow an operator to make reservoir-management decisions that optimize recovery in consideration of future production. This work relies on a hybridization of physics-based prediction and machine learning, whereby accurate synthetic data (in combination with available site data) can enable the application of machine learning methods for rapid forecasting and optimization. The physics-based prediction is built upon experimental and theoretical work to determine transport characteristics in shale at various scales, with an emphasis on materials from MSEEL-I; this fundamental shale R&D was conducted in partnership with DOE, NETL, and several other national labs. This work has resulted from a coordinated leveraging of developments across several projects within DOE FE30, along with internal investments from Los Alamos via LDRD. The development has utilized data from the MSEEL–I site for calibration and demonstration; however, the workflow and platform are readily extendable to operations at other sites, plays, and basins. This machine-learning method can aid operators to improve both recovery efficiency and competitiveness; to this end, future work would quantify processes for other plays/basins and integrate production details with economics.

04 OIL SHALES AND TAR SANDS↗

Computationally Accelerated Discovery and Experimental Demonstration of High-Performance Materials for Advanced Solar Thermochemical Hydrogen Production

This project achieved its overarching goal of accelerating the discovery and validation of solar thermochemical hydrogen (STCH) materials through a tightly integrated approach that combined high-throughput computational screening, advanced machine learning (ML), and experimental testing. Guided by the objectives outlined in the Statement of Project Objectives (SOPO), our work fulfilled all major milestones across four technical tasks and delivered scientific breakthroughs and practical tools that significantly exceeded the original scope of the project. We began by addressing the challenge of predicting material phase stability through machine learning. A novel Python module was developed to generate thousands of meaningful features from composition, structure, and electronic properties, enabling rapid and reproducible ML model development. Using these tools, we trained a model to predict temperature-dependent Gibbs energies (G(T)) for inorganic crystalline materials with near-chemical accuracy—roughly 40 meV/atom—marking the first such descriptor of its kind. We also introduced a new machine-learned tolerance factor, τ, that accurately predicted perovskite formability with over 90% success, outperforming traditional heuristic models, such as the Goldschmidt tolerance factor. These capabilities allowed for rapid and accurate predictions of phase stability across a vast oxide composition space, setting the stage for high-throughput thermodynamic screening. Building on this foundation, we conducted an extensive computational screening of candidate STCH oxide materials. Over 1.1 million perovskite compositions were evaluated using the τ descriptor, leading to the identification of more than 27,000 predicted stable structures. Using density functional theory (DFT), we refined over 68,000 multinary perovskite structures and computed oxygen vacancy formation energies for over 1,300 ternary and double perovskites. These calculations enabled us to isolate compounds with redox behavior consistent with STCH requirements and resulted in a public dataset now hosted on the Materials Project. Recognizing that thermodynamic screening alone is insufficient, we addressed kinetic limitations by developing a suite of tools to estimate transition state (TS) energies for key redox reactions. We implemented a novel bounding approach that provides lower and upper estimates of TS energies with dramatically reduced computational cost, requiring less than 10% of the CPU time of a full nudged elastic band (NEB) calculation while maintaining high accuracy. This enabled rapid evaluation of over 200 reaction pathways across 90 materials. To further accelerate screening, we developed a SISSO-based ML model to predict diffusion barriers with a 96.7% success rate in classifying fast vs. slow materials, supporting a robust, data-driven framework for assessing redox kinetics. Experimental validation was critical to confirming the predictive power of our models. We synthesized and tested a wide array of candidate materials, including Mn-doped hercynite and several Gd- and La-based perovskites. Notably, Sr 0.4 Gd 0.6 Mn 0.6 Al 0.4 O 3 (SGMA) and Gd 0.5 La 0.5 Co 0.5 Fe 0.5 O 3 (GLCF) emerged as leading STCH materials, exhibiting robust redox cycling and high hydrogen yields exceeding 150 µmol H 2 /g per cycle. These materials also retained over 50% of their hydrogen productivity under high-conversion conditions (H 2 O:H 2 = 1333:1), demonstrating strong thermodynamic favorability and promising performance under industrially relevant scenarios. Additional candidates, such as La 2 MnNiO 6 (L2MN), were found to produce even higher yields than ceria under standard STCH conditions. Our collaborators at Sandia National Laboratories confirmed these findings using high-temperature X-ray diffraction and thermogravimetric analysis, observing stable phase evolution and reversible redox activity. In several respects, the project went beyond the goals initially outlined in the SOPO. We published 17 peer-reviewed articles, including a large dataset of over 66,000 theoretical perovskites and a new structure prediction method (SPuDS-DFT) that accurately identifies ground-state structures at a fraction of the cost of traditional DFT. We demonstrated that our machine-learned G(T) model offers accuracy rivaling quasiharmonic calculations while being orders of magnitude faster. In partnership with the Materials Project, we made our datasets openly available, providing a powerful new resource for the broader materials science community. The combined computational and experimental advances of this project represent a significant advance in STCH materials discovery. By creating a robust, generalizable, and open workflow for thermodynamic and kinetic screening, and validating key findings through synthesis and reactor testing, we have provided a practical and scalable pathway for the rapid identification of new redox-active materials. The tools, data, and materials developed under this project are already supporting ongoing research and have laid the groundwork for the next generation of solar fuel technologies.

08 HYDROGEN↗