Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data pipeline”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

Modeling performance of data collection systems for high-energy physics

Exponential increases in scientific experimental data are outpacing silicon technology progress, necessitating heterogeneous computing systems—particularly those utilizing machine learning (ML)—to meet future scientific computing demands. The growing importance and complexity of heterogeneous computing systems require systematic modeling to understand and predict the effective roles for ML. We present a model that addresses this need by framing the key aspects of data collection pipelines and constraints and combining them with the important vectors of technology that shape alternatives, computing metrics that allow complex alternatives to be compared. For instance, a data collection pipeline may be characterized by parameters such as sensor sampling rates and the overall relevancy of retrieved samples. Alternatives to this pipeline are enabled by development vectors including ML, parallelization, advancing CMOS, and neuromorphic computing. By calculating metrics for each alternative such as overall F1 score, power, hardware cost, and energy expended per relevant sample, our model allows alternative data collection systems to be rigorously compared. We apply this model to the Compact Muon Solenoid experiment and its planned high luminosity-large hadron collider upgrade, evaluating novel technologies for the data acquisition system (DAQ), including ML-based filtering and parallelized software. The results demonstrate that improvements to early DAQ stages significantly reduce resources required later, with a power reduction of 60% and increased relevant data retrieval per unit power (from 0.065 to 0.31 samples/kJ). However, we predict that further advances will be required in order to meet overall power and cost constraints for the DAQ.

Olin-Ammentorp, Wilkie (ORCID:0000000224729862)↗

Analytical methods for online data quality assessment

This chapter provides a comprehensive overview of the main steps for algorithmic sensor signal quality assessment, which can enhance the decision-making process for water resource recovery facility (WRRF) operation and optimization. It introduces the concept of redundancy as the basis for data quality assessment. It also explains the typical data processing pipeline, which consists of preliminary analysis, data pre-processing, and specific algorithmic approaches. Each of these processes is presented and discussed in three separate sections. Importantly, this chapter introduces the main approaches for data quality assessment, provides guidelines for selecting the most suitable one and the key performance indicators to evaluate them and explains how to collect metadata through such an algorithmic approach.

Aguado, Daniel↗

HydroForecast Long-term: Improving hydropower’s resilience to climate change through accurate climate-scale

With hydrologic patterns and water availability across the globe shifting due to climate change, advancements in hydrologic prediction systems can help significantly reduce the uncertainties that utilities and water supply entities have in their decision making. Understanding and estimating hydrology at the climate scale is critical for managing water resources under changing climate scenarios. This project focuses on integrating state-of-the-art neural network modeling with downscaled climate projections to deliver the reliable water supply projections decades into the future to meet an urgent need from hydropower operators and water utilities. In this Phase 1 DOE SBIR proposal, we developed and validated a theory-guided neural network model, HydroForecast Long-term, for climate-scale hydrology and implemented the model within existing HydroForecast infrastructure. HydroForecast Long-term combines the most accurate streamflow modeling system with a flexible and scalable data architecture to generate water supply projections out to the year 2100. This report illustrates that we have achieved our four objectives: 1) create a prototype of HydroForecast Long-term, building the neural network prediction model, 2) build an automated data input pipeline that processes large amounts of data from the latest global temperature and precipitation climate models; 3) benchmark the accuracy of the hydrologic model over the recent two decades over a large set of diverse basins, and 4) create a set of output visuals and summary metrics informed by customer feedback that connect the data to critical decision points. This work empowers water users to make data-informed decisions supporting a resilient, renewable-powered grid and water system. The results advance the Department of Energy’s mission by addressing critical gaps in water supply planning under climate change.

13 HYDRO ENERGY↗

Data for An End-to-End Pipeline for Succinic Acid Production at an Industrially Relevant Scale Using Issatchenkia orientalis

Microbial production of succinic acid (SA) at an industrially relevant scale has been hindered by high downstream processing costs arising from neutral pH fermentation for over three decades. Here, we metabolically engineer the acid-tolerant yeast Issatchenkia orientalis for SA production, attaining the highest titers in sugar-based media at low pH (pH 3) in fed-batch fermentations, i.e. 109.5 g/L in minimal medium and 104.6 g/L in sugarcane juice medium. We further perform batch fermentation using sugarcane juice medium in a pilot-scale fermenter (300×) and achieve 63.1 g/L of SA, which can be directly crystallized with a yield of 64.0%. Finally, we simulate an end-to-end low-pH SA production pipeline, and techno-economic analysis and life cycle assessment indicate our process is financially viable and can reduce greenhouse gas emissions by 34–90% relative to fossil-based production processes. We expect I. orientalis can serve as a general industrial platform for production of organic acids.

Metabolomics↗

Curation and Dissemination of Complex Multi-Modal Datasets for Radiation Detection, Localization, and Tracking

The PANDAWN sensor network in Chicago, IL, is a state-of-the-art testbed for networked, multi-modal sensing. It integrates AI/data science methods into its operation, from data acquisition to automated data labeling and curation workflows. The curation and dissemination of diverse multi-modal datasets will enable the development of new radiological/nuclear (R/N) detection, localization, and tracking algorithms and methods relevant across the nonproliferation mission space. This article first introduces the PANDAWN sensor network and the features that make it stand out from previous multi-modal data acquisition efforts. We then review the various data streams acquired on the PANDAWN nodes and present the implementation of an automated data curation pipeline that includes the labeling of radiation and contextual data streams. Here, we finally provide a short overview of different studies that leveraged the curated datasets.

Data curation↗

Comparability of Liquid Chromatography Tandem Mass Spectrometry Analysis of Dissolved Organic Matter across Laboratories

Non-targeted liquid chromatography tandem highresolution mass spectrometry (LC−MS/MS) is increasingly applied for the structure-resolved chemical analysis of dissolved organic matter (DOM). With new developments in MS instrumentation and analysis software, the approach has gained substantial momentum over the past decade. However, achieving high-quality analytical data that is reproducible and comparable across laboratories can be a bottleneck in non-targeted metabolomics and organic matter chemical analysis, especially for data reuse in repository-scale analyses. Understanding the capabilities as well as challenges of comparing LC−MS/MS data from different laboratories is necessary for inferring global trends from public data sets. To illuminate instrumentation factors that drive differences and variability, we used a standardized data analysis pipeline, including classical (CMN) and featurebased molecular networking (FBMN), to analyze data from a ring trial by 24 laboratories on identical sample sets of algal and DOM extracts that were mixed in predefined concentrations and spiked with standards. Our results showed that data sets from similar mass spectrometer types with unified instrument parameters were qualitatively comparable, resolving the same general trends and shared mass spectral features. Interlaboratory comparability was best for high-intensity features, while low-intensity features showed greater detection variability. Our analysis also highlights challenges when comparing data from instruments with different acquisition rates or operating with less standardized methods. Lastly, we provide recommendations for data integration, public data sharing, standardization, and best practices for standardized LC−MS/MS data acquisition, which will be critical for long-term time series and intercomparability of DOM chemical analyses.

DOM↗

Reinforcement Learning for In-Spill Optimization of the Mu2e Resonant Extraction: Compensating Non-Stationarity

We present design considerations and challenges for the fast machine learning component of a third-order resonant beam extraction regulation system being commissioned to deliver steady beam rates to the mu2e experiment at Fermilab. Dedicated quadrupoles drive the tune toward the 29/3 resonance each spill, extracting beam at kV multiwire septa. The overall Spill Regulation System consists of (1) a “slow” process using ~100-spill averages to adjust the base quad ramp infrequently, (2) a feedforward harmonic content compensator, and (3) the “fast” ML agent reacting during each ongoing spill with on-the-fly additive corrections to the sum of (1) and (2). We have demonstrated improved beam-rate steadying for a fast ML agent compared to a PID controller using a quasi-physical spill simulation, and demonstrated distillation of that simulation into a predictive surrogate model. Current work includes a data-and-training pipeline to generate data-aware surrogates with real-world dynamics, even as the dynamics shift unpredictably. The surrogates are to act as RL environments against which to train our fast ML control agents before deploying them on FPGA in the live system. Further current efforts focus on modeling and controlling beam loss around the storage ring, understanding additional available hardware inputs to the model, and the interplay of these with beam-steadying performance.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

TomoPyUI : a user-friendly tool for rapid tomography alignment and reconstruction

The management and processing of synchrotron and neutron computed tomography data can be a complex, labor-intensive and unstructured process. Users devote substantial time to both manually processing their data ( i.e. organizing data/metadata, applying image filters etc. ) and waiting for the computation of iterative alignment and reconstruction algorithms to finish. In this work, we present a solution to these problems: TomoPyUI , a user interface for the well known tomography data processing package TomoPy . This highly visual Python software package guides the user through the tomography processing pipeline from data import, preprocessing, alignment and finally to 3D volume reconstruction. The TomoPyUI systematic intermediate data and metadata storage system improves organization, and the inspection and manipulation tools (built within the application) help to avoid interrupted workflows. Notably, TomoPyUI operates entirely within a Jupyter environment. Herein, we provide a summary of these key features of TomoPyUI , along with an overview of the tomography processing pipeline, a discussion of the landscape of existing tomography processing software and the purpose of TomoPyUI , and a demonstration of its capabilities for real tomography data collected at SSRL beamline 6-2c.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Carbon dioxide pipeline network transportation cost model: evaluating economic and geographic factors for efficient carbon capture, storage, and utilization

This study presents a comprehensive pipeline network modeling framework to estimate the CO 2 delivery cost for CO 2 utilization and geologic CO 2 storage across the United States. We developed a Python-based CO 2 pipeline transportation cost model leveraging Argonne National Laboratory’s pipeline engineering expertise and detailed natural gas transmission pipeline cost data across U.S. regions. Using existing road corridors as practical routing guides, the model designs pipeline networks that aggregate CO 2 from one or multiple sources and deliver it to selected destinations. It then minimizes the total transportation cost by optimizing pipeline diameters and incorporating booster pumps. A key contribution is the incorporation of up-to-date, region-specific cost factors with itemized components for materials, labor, miscellaneous construction expenses, and right-of-way acquisition. Results emphasize that regional variation and economies of scale associated with CO 2 pipeline costs are significant and should be explicitly accounted for in screening and planning studies. By combining realistic routing constraints with regionalized cost inputs, the model provides transparent design methodology and location-specific insights into source–destination delivery costs, including the effects of routing complexity along existing road networks. We demonstrate the model with two illustrative case studies – one for CO 2 storage and one for CO 2 utilization – in which the model designs pipeline networks spanning hundreds of miles across the states, collecting CO 2 from multiple sources and delivering it to designated endpoints while minimizing levelized cost of delivery via diameter and compression optimization. The model offers a practical, scalable approach for alternative design option screening and early-stage CO 2 transportation planning.

CCS↗

The DECADE cosmic shear project I: A new weak lensing shape catalog of 107 million galaxies

We present the Dark Energy Camera All Data Everywhere (DECADE) weak lensing dataset: a catalog of 107 million galaxies observed by the Dark Energy Camera (DECam) in the northern Galactic cap. This catalog was assembled from public DECam data including survey and standard observing programs. These data were consistently processed with the Dark Energy Survey Data Management pipeline as part of the DECADE campaign and serve as the basis of the DECam Local Volume Exploration survey (DELVE) Early Data Release 3 (EDR3). We apply the Metacalibration measurement algorithm to generate and calibrate galaxy shapes. After cuts, the resulting cosmology-ready galaxy shape catalog covers a region of $5,\!412 \,\,{\rm deg}^2$ with an effective number density of $4.59\,\, {\rm arcmin}^{-2}$. The coadd images used to derive this data have a median limiting magnitude of $r = 23.6$, $i = 23.2$, and $z = 22.6$, estimated at ${\rm S/N} = 10$ in a 2 arcsecond aperture. We present a suite of detailed studies to characterize the catalog, measure any residual systematic biases, and verify that the catalog is suitable for cosmology analyses. In parallel, we build an image simulation pipeline to characterize the remaining multiplicative shear bias in this catalog, which we measure to be $m = (-2.454 \pm 0.124) \times10^{-2}$ for the full sample. Despite the significantly inhomogeneous nature of the data set, due to it being an amalgamation of various observing programs, we find the resulting catalog has sufficient quality to yield competitive cosmological constraints.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Scalable GPS Data Logging To Support Advanced Fleet Analysis

This highlight details the key takeaways from a project that utilized NLR's Fleet Research, Energy Data, and Insights (FleetREDI) data analysis pipeline. National Laboratory of the Rockies researchers developed and demonstrated low-cost, open-source Arduino data loggers with 3D-printed cases that are compatible with global navigational systems and built with components available ubiquitously worldwide, enabling cost-effective collection and analysis of fleet operational data. Validated on an overseas transit bus fleet, NLR analysis showed that, with sufficient charging opportunities, 90% of observed duty cycles could be accomplished by electric buses with no modifications to operations.

33 ADVANCED PROPULSION SYSTEMS↗

Estimating Electrification Potential for Class 8 Regional-Haul Trucks

This one-page highlight details the key takeaways from a project that utilized NREL's Fleet Research, Energy Data, and Insights (FleetREDI) data analysis pipeline. As part of the North American Council for Freight Efficiency's (NACFE's) Run on Less Depot data workshop, NREL sought to understand how Tesla semi-trucks would perform in real-world regional haul applications. Analysis reveals that the modeled Tesla trucks, with an average efficiency of 1.78 kWh/mi, struggle to achieve full operational coverage using current battery and charging configurations assuming operations remain unchanged. However, in an extreme case where ubiquitous charging exists, 100% EV coverage is possible for the given drive cycles. These findings highlight the trade-off between battery size and charge rate in electrification potential and emphasize the necessity for advancements in charging infrastructure to enable electric trucks for regional haul operations.

ADVANCED PROPULSION SYSTEMS↗

The FRB-searching Pipeline of the Tianlai Cylinder Pathfinder Array

This paper presents the design, calibration, and survey strategy of the Fast Radio Burst (FRB) digital backend and its real-time data processing pipeline employed in the Tianlai Cylinder Pathfinder Array. The array, consisting of three parallel cylindrical reflectors and equipped with 96 dual-polarization feeds, is a radio interferometer array designed for conducting drift scans of the northern celestial semi-sphere. The FRB digital backend enables the formation of 96 digital beams, effectively covering an area of approximately 40 square degrees with the 3 dB beam. Our pipeline demonstrates the capability to conduct an automatic search of FRBs, detecting at quasi-real-time and classifying FRB candidates automatically. The current FRB searching pipeline has an overall recall rate of 88%. During the commissioning phase, we successfully detected signals emitted by four well-known pulsars: PSR B0329+54, B2021+51, B0823+26, and B2020+28. We report the first discovery of an FRB by our array, designated as FRB 20220414A. We also investigate the optimal arrangement for the digitally formed beams to achieve maximum detection rate by numerical simulation.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

RTN-117: Image Calibration and Instrument Signal Removal for the First Year of the LSST

The NSF-DOE Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST) requires calibration products that provide uniform, stable, and accurate photometric and astrometric performance across the 3.2 gigapixel focal plane and throughout the 10-year survey. This paper details the algorithms and workflows used to produce instrument calibrations and remove instrumental artifacts for Data Preview 2 (DP2)---the first end-to-end processing demonstration using on-sky data with the LSST Camera (LSSTCam). We describe the verification, acceptance, and certification framework used to assess calibration quality and quantify residual systematics. We show baseline metrics on calibrated science images to evaluate the robustness of the calibration and instrument signature removal (ISR) data processing pipelines for DP2. Finally, we summarize the known limitations observed in DP2 production and outline expected algorithmic improvements for the first public LSST data release (Data Release~1, DR1).

79 ASTRONOMY AND ASTROPHYSICS↗

Electrification Analysis: All Aboard America!

This one-page highlight details the key takeaways from a project that utilized NREL's Fleet Research, Energy Data, and Insights (FleetREDI) data analysis pipeline, an electrification analysis for the Bustang motorcoach fleet operated by All Aboard America! Holdings Inc. (AAA). NREL installed logging devices and collected operational data on nine 40-foot Bustang motorcoaches operating on fixed routes from May 2022 through August 2022. The analysis determined that partial fleet electrification may be feasible with electrified motorcoach options currently on the market. While this fleet faces significant challenges to electrification given current market options due to demanding range requirements and relatively limited charging opportunities, vehicles operating on the shorter, lower-grade routes along the I-25 corridor show more immediately available electrification potential. Increases in available battery capacity and the availability of fast-charging locations along I-70 routes are likely critical for electrification of the full fleet.

AAA↗

Trust Not Verify? The Critical Need for Data Curation Standards in Materials Informatics

The importance of data curation has been recognized in multiple areas of research; however, the discussion of this important issue is only beginning to emerge in materials science. In this Perspective, we highlight the benefits of using the standardized data curation protocols in materials science and discuss current gaps in accurate and reproducible data reporting using case studies drawn from high-impact materials science papers and well-known databases such as the Crystallography Open Database (COD) and the Cambridge Structural Database (CSD). We argue that both experimental and computational materials scientists need to embrace a culture of rigorous data curation as part of modern research data management. We propose a sample data curation pipeline for materials chemistry and illustrate its use by creating two new materials chemistry databases. Here, we hope that this perspective will serve to catalyze further discussion and promote the continuous development of rigorous data curation practices within the materials science research community. We posit that adherence to best practices of data curation will promote and enhance the reliability, reproducibility, and integrity of materials research and enable the development of reliable AI and machine learning models that critically depend on the use of quality data.

Chemical structure↗

Validating automated resonance evaluation with synthetic data

The integrity and precision of nuclear data are crucial for a broad spectrum of applications, from national security and nuclear reactor design to medical diagnostics, where the associated uncertainties can significantly impact outcomes. A substantial portion of uncertainty in nuclear data originates from the subjective biases in the evaluation process, a crucial phase in the nuclear data production pipeline. Recent advancements indicate that automation of certain routines can mitigate these biases, thereby standardizing the evaluation process and enhancing reproducibility. This research aims to provide a methodology, framework, and metrics for the validation of automated nuclear data evaluation software leveraging high-quality synthetic data that closely mimic real experimental observables. An introduced error metric provides a scale and intuitive measure of the evaluation quality by quantifying the estimate’s accuracy and performance across the specified energy range. Synthetic data provides access to experimental observables and underlying resonance parameters, enabling comparison of different evaluations. The methodology is demonstrated using Ta-181 isotope data in the resolved resonance region. The Automated Resonance Identification Subroutine (ARIS), which operates without prior resonance information, was used to test and showcase the framework’s capabilities utilizing the proposed error metrics. The results demonstrate the effectiveness of the proposed approach and framework for optimizing software parameters and testing hypotheses through “what-if” controlled experiments, such as modifying assumptions about experimental conditions or average resonance parameters.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Characterization of contaminants in the Lyman-alpha forest auto-correlation with DESI

Baryon Acoustic Oscillations can be measured with sub-percent precision above redshift two with the Lyman-α (Lyα) forest auto-correlation and its cross-correlation with quasar positions. This is one of the key goals of the Dark Energy Spectroscopic Instrument (DESI) which started its main survey in May 2021. We present in this paper a study of the contaminants to the Lyα forest which are mainly caused by correlated signals introduced by the spectroscopic data processing pipeline as well as astrophysical contaminants due to foreground absorption in the intergalactic medium. Notably, an excess signal caused by the sky background subtraction noise is present in the Lyα auto-correlation in the first line-of-sight separation bin. We use synthetic data to isolate this contribution, we also characterize the effect of spectro-photometric calibration noise, and propose a simple model to account for both effects in the analysis of the Lyα forest. We then measure the auto-correlation of the quasar flux transmission fraction of low redshift quasars, where there is no Lyα forest absorption but only its contaminants. We demonstrate that we can interpret the data with a two-component model: data processing noise and triply ionized Silicon and Carbon auto-correlations. This result can be used to improve the modeling of the Lyα auto-correlation function measured with DESI.

79 ASTRONOMY AND ASTROPHYSICS↗