Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data acquisition workflows”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Learning continuous scattering length density profiles from neutron reflectivities using convolutional neural networks

Interpreting neutron reflectivity (NR) data using ad hoc multi-layer models and physics-based models provides information about spatially resolved neutron scattering length density (NSLD) profiles. Recent improvements in data acquisition systems have allowed acquiring thousands of NR curves in a couple of hours, which has led to a need for automated data analysis tools to interpret NR measurements in real-time. Here, we present a machine learning analysis workflow that uses a series of models, based on a convolutional neural network (CNN), to learn the relation between the NSLDs and the NRs, and subsequently produce continuous NSLD profiles directly from NRs. The usefulness of our CNN-based models is demonstrated by constructing NSLDs from NRs of several films containing homopolymer polyzwitterions and diblock copolymers mixed with different types of salts. Comparisons of the NSLDs with those constructed using ad hoc multi-layer models reveal a very good agreement, suggesting the potential of CNN-based models for real-time automated data analysis of NRs.

36 MATERIALS SCIENCE↗
DOE OSTI2023

Preliminary report on applications of machine learning techniques to the Nevada play fairway analysis

We are applying machine learning (ML) techniques, including training set augmentation and artificial neural networks, to mitigate key challenges in the Nevada play fairway project. The study area includes ~85 active geothermal systems as potential training sites and >12 geologic, geophysical, and geochemical features. The main goal is to develop an algorithmic approach to identify new geothermal systems in the Great Basin region. Major objectives include: 1) integrate ML techniques into the geothermal community; 2) develop open community datasets, whereby all play fairway and ML datasets and algorithms are publicly released and available for modification by various user groups; 3) identify data acquisition targets with high value for future work; 4) identify new signatures to detect blind geothermal systems; and 5) foster new capabilities for characterizing subsurface temperature and permeability. Initially, ML techniques are being applied to the same play fairway datasets and workflow. ML will then be applied to both enhanced and additional datasets, with modification of the PFA workflow to incorporate the new datasets. Finally, ML will be applied to define new workflows using the enhanced and additional datasets. An algorithmic approach that empirically learns to estimate weights of influence for diverse parameters can potentially scale and perform better than the play fairway analysis. Initial work on this project has involved 1) evaluating potential positive and negative training sites, 2) transformation of datasets into formats suitable for ML, and 3) initial development and testing of ML techniques.

58 GEOSCIENCES↗
DOE OSTI2023

Machine learning for automated experimentation in scanning transmission electron microscopy

Abstract Machine learning (ML) has become critical for post-acquisition data analysis in (scanning) transmission electron microscopy, (S)TEM, imaging and spectroscopy. An emerging trend is the transition to real-time analysis and closed-loop microscope operation. The effective use of ML in electron microscopy now requires the development of strategies for microscopy-centric experiment workflow design and optimization. Here, we discuss the associated challenges with the transition to active ML, including sequential data analysis and out-of-distribution drift effects, the requirements for edge operation, local and cloud data storage, and theory in the loop operations. Specifically, we discuss the relative contributions of human scientists and ML agents in the ideation, orchestration, and execution of experimental workflows, as well as the need to develop universal hyper languages that can apply across multiple platforms. These considerations will collectively inform the operationalization of ML in next-generation experimentation.

36 MATERIALS SCIENCE↗
DOE OSTI2021

Integrating Models with Real-time Field Data for Extreme Events: From Field Sensors to Models and Back with AI in the Loop

Focal Area(s): This whitepaper is responsive to focal area (1) Data acquisition and assimilation enabled by machine learning, AI, and advanced methods including experimental/network design/optimization, unsupervised learning (including deep learning), and hardware-related efforts involving AI (e.g., edge computing). We discuss Artificial Intelligence and Machine Learning (AI/ML) enabled integration of real-time data into the extreme event modeling workflow to improve the predictive capabilities of these models, and deliver real-time feedback to remote sensors, including software and data engineering challenges.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Predicting Trends in VOC Through Rapid, Multimodal Characterization of State-of-the-Art p-i-n Perovskite Devices

Perovskite photovoltaic technologies are approaching commercial deployment, yet single junction and tandem architectures both still have significant room to improve power conversion efficiency and stability. The ability to perform rapid screening of material quality after altering processing conditions is critical to accelerating the optimization and commercialization of perovskite-based technologies. Currently, researchers utilize a wide range of stand-alone metrology tools to isolate sources of power loss throughout a device stack, which can be slow and labor intensive. Here, we demonstrate the use of a multimodal metrology approach to rapidly determine the maximum achievable and predicted open circuit voltages of >100 perovskite devices during fabrication. Acquisition of these different data is facilitated by combining them into a single integrated measurement platform. We show that these data and automated analysis can be used to rapidly understand and ultimately predict quantitative trends in open circuit voltages of state-of-the-art device architectures. The data and automated analysis workflow presented provides a reliable approach to quickly identify absorber and charge transport layer combinations that can lead to improved open circuit voltages.

14 SOLAR ENERGY↗
DOE OSTI2023

Streaming Readout and Remote Compute

Streaming Readout Data Acquisition systems coupled with distributed resources spread over vast geographic distances present new challenges to the next generation of experiments. High bandwidth modern network connectivity opens the possibility to utilize large, general-use, HTC systems that are not necessarily located close to the experiment. Near real-time response rates and workflow colocation can provide high reliability solutions to ensure efficient use of beam time. This talk will focus on a few technologies currently being developed at Jefferson Lab and in collaboration with ESnet to support fully streaming DAQ systems.

Lawrence, David↗
DOE OSTI2025

Evolution of the ATLAS TDAQ online software framework towards Phase-II upgrade: Use of Kubernetes as an orchestrator of the ATLAS Event Filter computing farm

The ATLAS experiment at the LHC at CERN continuously evolves its TDAQ system to meet the challenges of new physics goals and technological advancements. As ATLAS prepares for the Phase-II Run 4 of the LHC, significant enhancements in the TDAQ Controls and Configuration (TDAQ-CC) tools have been designed to ensure efficient data collection, processing, and management. This abstract presents the evolution of ATLAS TDAQ-CC system leading up to Phase-II Run 4. As part of the evolution towards Phase-II, Kubernetes has been chosen to orchestrate the Event Filter (EF) farm. By leveraging Kubernetes, ATLAS can dynamically allocate computing resources, scale processing capacity in response to changing data taking conditions and ensure high availability of data processing services. The integration of the Kubernetes with the TDAQ Run Control framework enables perfect synchronisation between the experiment’s data acquisition components and the computing infrastructure. We will discuss the architectural considerations and implementation challenges involved in Kubernetes integration with the ATLAS TDAQ-CC system. We will highlight the benefits of using Kubernetes as an EF farm orchestrator, including improved resource utilization, enhanced fault tolerance, and simplified deployment and management of data processing workflows. In addition, we will report on the extensive testing of Kubernetes that was conducted using a farm of 2500 servers within the experiment data taking environment, demonstrating its scalability and robustness in handling the demands of the ATLAS TDAQ system for Phase-II. The adoption of Kubernetes represents a significant step forward in the evolution of ATLAS TDAQ-CC system, aligning with industry best practices in container orchestration.

Corso Radu, Alina [Univ. of California, Irvine, CA↗
DOE OSTI2023

Auto-HPCnet: An Automatic Framework to Build Neural Network-based Surrogate for High-Performance Computing Applications

High-performance computing communities are increasingly adopt- ing Neural Networks (NN) as surrogate models in their applications to generate scientific insights. Replacing an execution phase in the application with NN models can bring significant performance im- provement. However, there is a lack of tools that can help domain scientists automatically apply NN-based surrogate models to HPC applications. We introduce a framework, named Auto-HPCnet, to democratize the usage of NN-based surrogates. Auto-HPCnet is the first end-to-end framework that makes past proposals for the NN-based surrogate model practical and disciplined. Auto-HPCnet introduces a workflow to address unique challenges when apply- ing the approximation, such as feature acquisition and meeting the application-specific constraint on the quality of final computation outcome. We show that Auto-HPCnet can leverage NN for a set of HPC applications and achieve 5.50× speedup on average (up to 16.8× speedup and with data preparation cost included) while meeting the application-specific constraint on the final computation quality.

Dong, Wenqian↗
DOE OSTI2025

Advancing $otsdaq$ for Optimized Data Acquisition

High-energy physics (HEP) experiments demand data acquisition (DAQ) systems capable of orchestrating complex detector operations, high data throughput, and responsive, real-time feedback. Traditional systems often have steep learning curves, making onboarding difficult for new users. The Off-The-Shelf Data Acquisition $otsdaq$ framework was developed to address these issues by providing a modular and flexible interface that is easier to operate while remaining customizable enough for experimental setups. As the upcoming Mu2e experiment prepares for deployment, improving stability, usability, and performance has become increasingly critical. Our work enhances $otsdaq$ with features that streamline visualization, correct data metrics, improve debugging workflows, and stabilize the user interface.

Mohammed, Ali (ORCID:0009000860386626)↗
DOE OSTI2021

DEIMoS: An Open-Source Tool for Processing High-Dimensional Mass Spectrometry Data

We present DEIMoS: Data Extraction for Integrated Multidimensional Spectrometry, a Python application programming interface and command-line tool for high-dimensional mass spectrometry data analysis workflows, offering ease of development and access to efficient algorithmic implementations. Functionality includes feature detection, feature alignment, collision cross section calibration, isotope detection, and MS/MS spectral deconvolution, with the output comprising detected features aligned across study samples and characterized by mass, CCS, tandem mass spectra, and isotopic signature. Notably, DEIMoS operates on N-dimensional data, largely agnostic to acquisition instrumentation: algorithm implementations utilize all dimensions simultaneously to (i) offer greater separation between features, improving detection sensitivity, (ii) increase alignment/feature matching confidence among datasets, and (iii) mitigate convolution artifacts in tandem mass spectra. We demonstrate DEIMoS with LC-IMS-MS/MS data, demonstrating the advantages of a multidimensional approach in each data processing step.

Colby, Sean M↗
DOE OSTI2020

Subtask 1.5 – CO2 Injection Monitoring with an Optimized Scalable, Automated, Semipermanent Seismic Array

The scalable, automated, semipermanent seismic array (SASSA) method is a flexible and relatively cost-effective surface geophysical method for regular time-lapse monitoring of the movement of injected carbon dioxide (CO2) in a reservoir for CO2 enhanced oil recovery (EOR) or geologic CO2 storage operations. It has the advantages of a low-environmental-footprint while monitoring regions of a reservoir from the surface without the need for a regular grid distribution of receivers. Automated data collection is possible. As only time-lapse amplitude changes at the reservoir level due to CO2 movement within the reservoir are monitored, the turnaround time to deliver results from the SASSA method can be short, without the need for long, time-consuming data-processing workflows. As data is collected and processed, incremental information can be provided to the field operator. The Energy & Environmental Research Center (EERC) conducted a SASSA field test from September 2018 to November 2020 in a portion of the Bell Creek Field in Montana, which implemented new CO2 EOR field activities during the study period. Lessons learned from a proof-of-concept study were incorporated to improve the data quality of the SASSA method and demonstrate the viability of the technology. The EERC implemented several enhancements to improve data quality, including 1) an iterative survey design, which allowed placing the receivers in strategic locations where the movement of the CO2 in the reservoir could be tracked with minimum interference by the cultural noise in the study area; 2) the use of powerful seismic sources in the form of surface orbital vibrators, and 3) data acquisition during optimal periods. History-matched reservoir simulation was performed to predict gas saturation and pressure response induced by CO2 injection in the study area. The results were compared with the SASSA-measured responses to CO2 injection as a partial validation technique. A match between the two methods was observed for most of the SASSA points predicted to have intersected a CO2 saturation change. The validated results provide confidence that the SASSA method can be used independently as a CO2 saturation monitoring technique. As data are collected and processed, incremental information can be provided to the field operator. The critical components of the SASSA workflow for a successful application of the method are the following: Iterative survey design with information about CO2 injection activities from the oilfield operator. A detailed CO2 injection plan is the key driver to select the strategic monitoring location of the SASSA sensors. After this information is incorporated in the initial distribution of sources and receivers in the study area, high-resolution satellite images are used to identify ground locations not affected by cultural noise sources, such as power lines, pipelines/flow lines, or roadways. In the next iteration of the survey design, a scouting trip to the study area is needed to understand more details of the noise sources identified in the previous step and the intensity of the field activities that can also generate noise during the monitoring. Integrating the information from the scouting trip into the survey design to select the optimum source and receiver locations is the final step. Noise attenuation. The variety of noise types during seismic monitoring of an oil field is enormous. Tailored noise characterization and processing at a node-by-node level can enhance the performance and sensitivity of the SASSA technique. Future advancements that could improve the efficiency and application of the SASSA technology include: Gaining a better understanding of the noise field produced by the seismic source to aid the choice of receiver location. Surface noise from the source can overwhelm the small signal changes due to CO2 that the SASSA method measures. Improved data-processing workflow to automatically analyze and adapt to dynamic noise conditions associated with industrial settings. This subtask was funded through the EERC–DOE Joint Program on Research and Development for Fossil Energy-Related Resources Cooperative Agreement No. DE- FE0024233.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2023

An Edge Alignment-Based Orientation Selection Method for Neutron Tomography

Neutron computed tomography (nCT) is a 3D char-acterization technique used to image the internal morphology or chemical composition of samples in biology and materials sciences. A typical workflow involves placing the sample in the path of a neutron beam, acquiring projection data at a predefined set of orientations, and processing the resulting data using an analytic reconstruction algorithm. Typical nCT scans require hours to days to complete and are then processed using conventional filtered back-projection (FBP), which performs poorly with sparse views or noisy data. Hence, the main methods in order to reduce overall acquisition time are the use of an improved sampling strategy combined with the use of advanced reconstruction methods such as model-based iterative reconstruction (MBIR). In this paper, we propose an adaptive orientation selection method in which an MBIR reconstruction on previously-acquired measurements is used to define an objective function on orientations that balances a data-fitting term promoting edge alignment and a regularization term promoting orientation diversity. Using simulated and experimental data, we demonstrate that our method produces high-quality reconstructions using significantly fewer total measurements than the conventional approach.

Yang, Diyu↗
DOE OSTI2021

A data-independent acquisition-based global phosphoproteomics system enables deep profiling

Phosphoproteomics can provide insights into cellular signaling dynamics. To achieve deep and robust quantitative phosphoproteomics profiling for minute amounts of sample, we here develop a global phosphoproteomics strategy based on data-independent acquisition (DIA) mass spectrometry and hybrid spectral libraries derived from data-dependent acquisition (DDA) and DIA data. Benchmarking the method using 166 synthetic phosphopeptides shows high sensitivity (<0.1 ng), accurate site localization and reproducible quantification (~5% median coefficient of variation). As a proof-of-concept, we use lung cancer cell lines and patient-derived tissue to construct a hybrid phosphoproteome spectral library covering 159,524 phosphopeptides (88,107 phosphosites). Based on this library, our single-shot streamlined DIA workflow quantifies 36,350 phosphosites (19,755 class 1) in cell line samples within two hours. Application to drug-resistant cells and patient-derived lung cancer tissues delineates site-specific phosphorylation events associated with resistance and tumor progression, showing that our workflow enables the characterization of phosphorylation signaling with deep coverage, high sensitivity and low between-run missing values.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2024

High-Throughput Data Processing at FRIB Using ESnet

Real-time or nearly real-time (nearline) data processing methods are critical tools as detector technologies and data acquisition (DAQ) systems allow for higher data rates and volumes. The introduction of the energy sciences network (ESnet), a U.S. Department of Energy (DOE) supported high-speed network for scientific research, creates opportunities to leverage the computing power of DOE facilities like the National Energy Research Scientific Computing Center (NERSC). As a first step toward realizing a DOE Office of Science Integrated Research Infrastructure (IRI) pattern, an automated workflow was developed to remotely process data obtained from a nuclear physics experiment at the Facility for Rare Isotope Beams (FRIB) at NERSC with data transferred between FRIB and NERSC over ESnet. The workflow demonstrated the ability to process one week’s worth of experimental data in approximately 90 min and was used successfully for nearline analysis during a recently completed FRIB experiment. Here, a summary of the workflow development and results of recent demonstrations will be presented.

Data processing↗
DOE OSTI2022

Dimensional Control over Metal Halide Perovskite Crystallization Guided by Active Learning

Metal halide perovskite (MHP) derivatives, a promising class of optoelectronic materials, have been synthesized with a range of dimensionalities that govern their optoelectronic properties and determine their applications. We demonstrate a data-driven approach combining active learning and high-throughput experimentation to discover, control, and understand the formation of phases with different dimensionalities in the morpholinium (morph) lead iodide system. Using a robot-assisted workflow, we synthesized and characterized two novel MHP derivatives that have distinct optical properties: a one-dimensional (1D) morphPbI 3 phase ([C 4 H 10 NO][PbI 3 ]) and a two-dimensional (2D) (morph) 2 PbI 4 phase ([C 4 H 10 NO] 2 [PbI 4 ]). To efficiently acquire the data needed to construct a machine learning (ML) model of the reaction conditions where the 1D and 2D phases are formed, data acquisition was guided by a diverse-mini-batch-sampling active learning algorithm, using prediction confidence as a stopping criterion. Querying the ML model uncovered the reaction parameters that have the most significant effects on dimensionality control. Based on these insights, we discuss possible reaction schemes that may selectively promote the formation of morph-Pb-I phases with different dimensionalities. The data-driven approach presented here, including the use of additives to manipulate dimensionality, will be valuable for controlling the crystallization of a range of materials over large reaction-composition spaces.

36 MATERIALS SCIENCE↗
DOE OSTI2022

Small experiment, Big Data: the data production of the Muon $g-2$ Experiment

The Muon $g-2$ Experiment at Fermilab aims to measure the muon anomalous magnetic moment with the unprecedented precision of 140 parts per billion (ppb). In April 2021 the collaboration published the first measurement, based on the first year of data taking. The result confirmed the previous experiment at Brookhaven National Laboratory (BNL), and increased the long-standing tension with the Standard Model prediction to 4.2 $\sigma$. The experiment is now running the sixth year of data acquisition of positive muon data, having accumulated a total of $\sim$19 times the statistics of the BNL experiment. A collaboration-wide effort is now in place to help produce the multi-petabyte-sized data sets, a challenge typically faced by much bigger experiments. Having a quick production turnaround time is of critical importance in order to achieve a timely analysis and publication schedule. In this paper I will describe the production workflow, the former and current challenges, the resources, tools, and the future prospects of the Muon $g-2$ Experiment.

Girotti, Paolo↗
DOE OSTI2026

CHESS 2025: Discrete-return LiDAR point clouds from NEON AOP surveys

This dataset provides Level 1 (L1) discrete-return light detection and ranging (LiDAR) point cloud data collected for the 2025 Colorado Headwaters Ecological Spectroscopy Study (CHESS). These data were acquired to enable characterization of vegetation structure and other three-dimensional features of the land surface, and to evaluate structural changes that may have occurred between a prior LiDAR acquisition in 2018 and the 2025 overflight. The data were acquired over three study domains in the Upper Gunnison river basin: the upper East River watershed (CRBU); Almont Triangle and Taylor Canyon (ALMO); and Upper Taylor River watershed (UPTA) between 2025-06-13 and 2025-07-15. LiDAR data were acquired using the Optech Galaxy Prime Airborne LiDAR Terrain Mapper onboard the National Ecological Observatory Network (NEON) Airborne Observation Platform (AOP). These are the primary unclassified discrete-return LiDAR data delivered by NEON and are provided per flightline as LASzip (LAZ) 1.4 Format 6 files. Data were processed following the workflow described in the NEON L0-to-L1 Discrete Return LiDAR Algorithm Theoretical Basis Document (Krause and Goulden 2022). Each record in the unclassified point clouds represents a geolocated laser target/return recorded by the LiDAR system, with values for X, Y, Z position and return intensity. All point coordinates are provided in meters. Horizontal coordinates are referenced in Universal Transverse Mercator (UTM) zone 13N and the World Geodetic System (WGS) 1984 ensemble datum. Elevations are referenced to Geoid12A. Flight metadata describing flightline boundaries and positional uncertainty by point are also included. CHESS Project Description: The Colorado Headwaters Ecological Spectroscopy Study (CHESS) comprised a multi-week airborne remote sensing and field observation campaign in the Upper Gunnison Basin, Colorado, conducted in June and July of 2025. Airborne remote sensing was conducted by the National Ecological Observatory Network Airborne Observation Platform (NEON AOP), concurrent with a field campaign run by the Rocky Mountain Biological Laboratory (RMBL), the Lawrence Berkeley National Laboratory (LBNL) and SLAC National Accelerator Laboratory Watershed Function Science Focus Area (SFA), and NASA-JPL (Jet Propulsion Laboratory) Earth Surface Mineral Dust Source Investigation (EMIT) program. Between June 10 and July 18, 2025, the NEON AOP flight team collected high-resolution aerial imaging spectroscopy and Light Detection and Ranging (LiDAR) data over three domains: the Upper East River (CRBU), Almont Triangle (ALMO), and the Upper Taylor Basin (UPTA). In coordination with the flights, a field campaign acquired ground-truth observations, including observations of vegetation composition, foliar traits, forest demography, and subsurface properties in 18 core sampling areas within the domains. Additional surface water observations were taken at over 380 point locations. All CHESS campaign datasets can be found within the CHESS ESS-DIVE data portal: https://data.ess-dive.lbl.gov/portals/chess. Funding Acknowledgement: Field and remote-sensing data acquisition was performed under a grant from the National Aeronautics and Space Administration (80NSSC24K1005). This work was also supported by the Watershed Function Science Focus Area at Lawrence Berkeley National Laboratory funded by the US Department of Energy, Office of Science, Biological and Environmental Research under Contract No. DE-AC02-05CH11231.

2018 NEON and 2025 CHESS Campaigns↗
DOE OSTI2020

Policy Considerations When Federating Facilities for Experimental and Observational Data Analysis

Today’s computational, experimental, and observational facilities afford us tremendous opportunities to couple theory and experiment at increasingly large scales. Empirical sensing capabilities are growing dramatically with beam line and detector improvements, and with advances in our ability to deploy large-scale data gathering observations of the natural world. The coupling of computational simulations and analysis to process the data from experimental and observational facilities is giving rise to cross-facility workflows. Such federations of facilities are in fact becoming an explicit requirement for large-scale scientific discovery. As we scale up these pipelines of scientific discovery, each participating facility needs to establish and align policies so that the federation can work seamlessly in an end-to-end manner. This chapter outlines specific policy considerations in enabling the federation of facilities for data analysis. Design choices and vital policy decisions cover the areas of data acquisition and storage, data transfer, computational resource allocation and co-scheduling, seamless federated user access, and cross-cutting governance. By highlighting the explicit and implicit interdependencies between facilities, we aim to provide facility designers and policymakers the information on policy issues to address early in a facility’s operations, thus enabling successful cross-facility federation and improved experimental and observational data analysis outcomes.

Shankar, Mallikarjun (Arjun)↗