Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Software pipelines”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 217 records · Page 12
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Towards Interactive, Reproducible Analytics at Scale on HPC Systems

The growth in scientific data volumes has resulted in a need to scale up processing and analysis pipelines using High Performance Computing (HPC) systems. These workflows need interactive, reproducible analytics at scale. The Jupyter platform provides core capabilities for interactivity but was not designed for HPC systems. In this paper, we outline our efforts that bring together core technologies based on the Jupyter Platform to create interactive, reproducible analytics at scale on HPC systems. Our work is grounded in a real world science use case-applying geophysical simulations and inversions for imaging the subsurface. Our core platform addresses three key areas of the scientific analysis workflow-reproducibility, scalability, and interactivity. We describe our implemention of a system, using Binder, Science Capsule, and Dask software. We demonstrate the use of this software to run our use case and interactively visualize real-Time streams of HDF5 data.

containers↗
DOE OSTI2021

Development of Automated Pipeline for Time-Resolved Link-Wise Vehicular Energy Consumption in the Chattanooga, TN Road Network

The Department of Energy (DOE) has shown strong interest in detecting energy inefficiencies in regional road networks, so as to derive energy consumed at a high spatial temporal resolution. We have developed a workflow to automate the estimation of time-resolved vehicular energy consumption over each link in a road network of interest. The road network used in the current work is centered around the city of Chattanooga, Tennessee and its bordering regions. Utilizing the most mature road network for the Chattanooga, TN region, vehicle speed & count data from TomTom in conjunction with machine learning methods, we have developed an automated pipeline to estimate energy consumption for every link in the network. The first step in the pipeline is ingesting vehicle probe counts and speed estimates from TomTom API. In the next step, the probe counts, speed profiles and other exogenous data (i.e. road types, weather data, ground-truth volume counts and more) were used as input to a supervised learning algorithm to estimate the number of vehicles throughout the entire region for each road segment. These volume estimates were then mapped to a unified road network that contained additional important information such as percentage change in gradient across a link, number of lanes and link lengths that are features in pre-trained single vehicle energy-consumption models available with the RouteE software developed at NREL. The per vehicle energy consumption on each road link predicted using appropriate RouteE vehicular models were multiplied by the volume estimate for the corresponding link over a given time period to predict energy consumed per link for the time interval of interest. Currently, work is underway to improve both the RouteE per vehicle energy estimate and the volume estimates derived from TomTom probe counts. We have also explored the correlation of the link-wise energy estimates with the features of the pre-trained RouteE machine learning model in order to gain insight into what factors contribute most to the link-wise energy consumption.

ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATION,↗
DOE OSTI2022

A Search of the Full Six Years of the Dark Energy Survey for Outer Solar System Objects

We present a search for outer solar system objects in the 6 yr of data from the Dark Energy Survey (DES). The DES covered a contiguous 5000 deg 2 of the southern sky with ≈80,000 3 deg 2 exposures in the grizY filters between 2013 and 2019. This search yielded 812 trans-Neptunian objects (TNOs), one Centaur and one Oort cloud comet, 458 reported here for the first time. We present methodology that builds upon our previous search on the first 4 yr of data. All images were reprocessed with an optimized detection pipeline that leads to an average completeness gain of 0.47 mag per exposure, as well as improved transient catalog production and algorithms for linkage of detections into orbits. All objects were verified by visual inspection and by the “sub-threshold significance,” the signal-to-noise ratio in the stack of images in which its presence is indicated by the orbit, but no detection was reported. This yields a pure catalog complete to r ≈ 23.8 mag and distances 29 < d < 2500 au. The TNOs have minimum (median) of 7 (12) nights’ detections and arcs of 1.1 (4.2) yr, and will have grizY magnitudes available in a further publication. We present software for simulating our observational biases for comparisons of models to our detections. Initial inferences demonstrating the catalog’s statistical power are: the data are inconsistent with the CFEPS-L7 model for the classical Kuiper Belt; the 16 “extreme” TNOs (a > 150 au, q > 30 au) are consistent with the null hypothesis of azimuthal isotropy; and nonresonant TNOs with q > 38 au, a > 50 au show a significant tendency to be sunward of major mean-motion resonances.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2026

Opportunities in AI/ML for the Rubin LSST Dark Energy Science Collaboration

The Vera C. Rubin Observatory's Legacy Survey of Space and Time (LSST) will produce unprecedented volumes of heterogeneous astronomical data (images, catalogs, and alerts) that challenge traditional analysis pipelines. The LSST Dark Energy Science Collaboration (DESC) aims to derive robust constraints on dark energy and dark matter from these data, requiring methods that are statistically powerful, scalable, and operationally reliable. Artificial intelligence and machine learning (AI/ML) are already embedded across DESC science workflows, from photometric redshifts and transient classification to weak lensing inference and cosmological simulations. Yet their utility for precision cosmology hinges on trustworthy uncertainty quantification, robustness to covariate shift and model misspecification, and reproducible integration within scientific pipelines. This white paper surveys the current landscape of AI/ML across DESC's primary cosmological probes and cross-cutting analyses, revealing that the same core methodologies and fundamental challenges recur across disparate science cases. Since progress on these cross-cutting challenges would benefit multiple probes simultaneously, we identify key methodological research priorities, including Bayesian inference at scale, physics-informed methods, validation frameworks, and active learning for discovery. With an eye on emerging techniques, we also explore the potential of the latest foundation model methodologies and LLM-driven agentic AI systems to reshape DESC workflows, provided their deployment is coupled with rigorous evaluation and governance. Finally, we discuss critical software, computing, data infrastructure, and human capital requirements for the successful deployment of these new methodologies, and consider associated risks and opportunities for broader coordination with external actors.

Aubourg, Eric [APC, Paris] (ORCID:000000025592023X↗
DOE OSTI2022

High–throughput measurement of plant fitness traits with an object detection method using Faster R–CNN

Revealing the contributions of genes to plant phenotype is frequently challenging because loss-of-function effects may be subtle or masked by varying degrees of genetic redundancy. Such effects can potentially be detected by measuring plant fitness, which reflects the cumulative effects of genetic changes over the lifetime of a plant. However, fitness is challenging to measure accurately, particularly in species with high fecundity and relatively small propagule sizes such as Arabidopsis thaliana. An image segmentation-based method using the software ImageJ and an object detection-based method using the Faster Region-based Convolutional Neural Network (R-CNN) algorithm were used for measuring two Arabidopsis fitness traits: seed and fruit counts. The segmentation-based method was error-prone (correlation between true and predicted seed counts, r 2 = 0.849) because seeds touching each other were undercounted. By contrast, the object detection-based algorithm yielded near perfect seed counts (r 2 = 0.9996) and highly accurate fruit counts (r 2 = 0.980). Comparing seed counts for wild-type and 12 mutant lines revealed fitness effects for three genes; fruit counts revealed the same effects for two genes. Our study provides analysis pipelines and models to facilitate the investigation of Arabidopsis fitness traits and demonstrates the importance of examining fitness traits when studying gene functions.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

AmeriFlux FLUXNET-1F US-xTL NEON Toolik (TOOL)

This is the AmeriFlux Management Project (AMP) created FLUXNET-1F version of the carbon flux data for the site US-xTL NEON Toolik (TOOL). This is the FLUXNET version of the carbon flux data for the site US-xTL NEON Toolik (TOOL) produced by applying the standard ONEFlux (1F) software. Site Description - The Toolik field site is located in a remote wilderness area at the Toolik Field Station which is on land managed by the Bureau of Land Management. The area is underlain by continuous permafrost, which exerts a major influence on hydrology and the distribution, structure, and function of terrestrial and aquatic ecosystems. Because of its location between the Brooks Range and the coastal plain, the vegetation and soils at TOOL are representative of much of the Alaskan foothills tundra. This terrestrial field site is collocated with NEON’s Toolik Lake aquatic site. The closest city, Fairbanks, Alaska is 400 miles away. The Dalton Highway (also known as the Haul Road or Pipeline Road), which traverses the middle of the state from Fairbanks to Prudhoe Bay, provides access to the area.

Network), NEON (National Ecological Observatory↗
DOE OSTI2022

PDBx/mmCIF Ecosystem: Foundational Semantic Tools for Structural Biology

PDBx/mmCIF, Protein Data Bank Exchange (PDBx) macromolecular Crystallographic Information Framework (mmCIF), has become the data standard for structural biology. With its early roots in the domain of small-molecule crystallography, PDBx/mmCIF provides an extensible data representation that is used for deposition, archiving, remediation, and public dissemination of experimentally determined three-dimensional (3D) structures of biological macromolecules by the Worldwide Protein Data Bank (wwPDB, wwpdb.org). Extensions of PDBx/mmCIF are similarly used for computed structure models by ModelArchive (modelarchive.org), integrative/hybrid structures by PDB-Dev (pdb-dev.wwpdb.org), small angle scattering data by Small Angle Scattering Biological Data Bank SASBDB (sasbdb.org), and for models computed generated with the AlphaFold 2.0 deep learning software suite (alphafold.ebi.ac.uk). Community-driven development of PDBx/mmCIF spans three decades, involving contributions from researchers, software and methods developers in structural sciences, data repository providers, scientific publishers, and professional societies. Having a semantically rich and extensible data framework for representing a wide range of structural biology experimental and computational results, combined with expertly curated 3D biostructure data sets in public repositories, accelerates the pace of scientific discovery. Herein, we describe the architecture of the PDBx/mmCIF data standard, tools used to maintain representations of the data standard, governance, and processes by which data content standards are extended, plus community tools/software libraries available for processing and checking the integrity of PDBx/mmCIF data. Use cases exemplify how the members of the Worldwide Protein Data Bank have used PDBx/mmCIF as the foundation for its pipeline for delivering Findable, Accessible, Interoperable, and Reusable (FAIR) data to many millions of users worldwide.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2021

CLMM : a LSST-DESC cluster weak lensing mass modeling library for cosmology

ABSTRACT We present the v1.0 release of CLMM, an open source python library for the estimation of the weak lensing masses of clusters of galaxies. CLMM is designed as a stand-alone toolkit of building blocks to enable end-to-end analysis pipeline validation for upcoming cluster cosmology analyses such as the ones that will be performed by the Vera C. Rubin Legacy Survey of Space and Time-Dark Energy Science Collaboration (LSST-DESC). Its purpose is to serve as a flexible, easy-to-install, and easy-to-use interface for both weak lensing simulators and observers and can be applied to real and mock data to study the systematics affecting weak lensing mass reconstruction. At the core of CLMM are routines to model the weak lensing shear signal given the underlying mass distribution of galaxy clusters and a set of data operations to prepare the corresponding data vectors. The theoretical predictions rely on existing software, used as backends in the code, that have been thoroughly tested and cross-checked. Combined theoretical predictions and data can be used to constrain the mass distribution of galaxy clusters as demonstrated in a suite of example Jupyter Notebooks shipped with the software and also available in the extensive online documentation.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2020

Dragonstone Strategy Kickoff Report [Slides]

The oil & natural gas (ONG) system like any other heavy industrial process, relies on a complex system of information technology (IT) and operational technology (OT) devices. Cyber-preparedness varies across the ONG industry from organization to organization. Common challenges include remote locations, long-lived field assets, and insufficient capabilities to find and track malware. ONG companies tend to be concerned with lack of cyber-awareness from employees, risk stemming from remote access for operations & maintenance, and software vulnerabilities within third-party equipment. For a variety of reasons, the general consensus from stakeholders interviewed by LLNL is that the state of cybersecurity within the electric grid is currently outpacing its ONG cousin. Existing strategies for the resilience and cybersecurity of the electric grid can and should be leveraged to provide immediate benefits to the ONG system. In LLNL’s view, there are two key factors currently limiting the development of necessary cyber-practices within the ONG industry: regulatory overlap and inefficiencies, and lack of industry awareness of potentially useful technologies for solving known issues. In the months following this report, LLNL will develop a cyber-resilience roadmap for the oil & gas pipeline sector, coupling threat intelligence, industry knowledge base, and ongoing outreach

02 PETROLEUM↗
DOE OSTI2022

Towards Low-Overhead Resilience for Data Parallel Deep Learning

Data parallel techniques have been widely adopted both in academia and industry as a tool to enable scalable training of deep learning models. At scale, DL training jobs can fail due to software or hardware bugs, may need to be preempted or terminated due to unexpected events, or may perform suboptimally because they were misconfigured. Under such circumstances, there is a need to recover and/or reconfigure data-parallel DL training jobs on-the-fly, while minimizing the impact on the accuracy of the DNN model and the runtime overhead. In this regard, state-of-art techniques adopted by the HPC community mostly rely on checkpoint-restart, which inevitably leads to loss of progress, thus increasing the runtime overhead. In this paper we explore alternative techniques that exploit the properties of modern deep learning frameworks (overlapping of gradient averaging and weight updates with local gradient computations through pipeline parallelism) to reduce the overhead of resilience/elasticity. To this end we introduce a failure simulation framework and two resilience strategies (immediate mini-batch rollback and lossy forward recovery), which we study compared with checkpoint-restart approaches in a variety of settings in order to understand the trade-offs between the accuracy loss of the DNN model and the runtime overhead.

data-parallel training↗
DOE OSTI2026

pathSQE : an automated workflow for single-crystal inelastic neutron scattering data processing and analysis

Inelastic neutron scattering (INS) experiments utilizing modern time-of-flight spectrometers enable the comprehensive mapping of the energy (E)- and momentum (Q)-resolved dynamical structure factor of single crystals, probing both the lattice and magnetic excitations. Yet, the large size and complexity of four-dimensional INS data are challenging current analysis workflows, often resulting in an underutilization of the measured information. To help address this issue, this paper introduces new software interfaced with the Mantid framework, pathSQE, designed to streamline the processing, analysis and interpretation of 4D single-crystal INS data. By automating key tasks such as 1D/2D slicing, symmetrization, Brillouin zone folding, data visualization, prioritization and filtering, and comparisons with simulations, pathSQE facilitates and accelerates INS data analysis workflows. Here, this paper outlines the features and implementation and provides several illustrations of the use of pathSQE on data collected on single crystals using direct-geometry time-of-flight spectrometers at the Spallation Neutron Source, including Ge, FeSi, MnO and SnS single-crystal measurements on the ARCS, HYSPEC and CNCS neutron spectrometers. Beyond streamlining post-experiment data processing, pathSQE establishes an automated and modular processing pipeline that could support future real-time experiment steering.

36 MATERIALS SCIENCE↗
DOE OSTI2020

The Mertens Unrolled Network (MU-Net): A High Dynamic Range Fusion Neural Network for Through the Windshield Driver Recognition

Face recognition of vehicle occupants through windshields in unconstrained environments poses a number of unique challenges ranging from glare, poor illumination, driver pose and motion blur. In this paper, we further develop the hardware and software components of a custom vehicle imaging system to better overcome these challenges. After the build out of a physical prototype system that performs High Dynamic Range (HDR) imaging, we collect a small dataset of through-windshield image captures of known drivers. We then reformulate the classical Mertens-Kautz-Van Reeth HDR fusion algorithm as a pre-initialized neural network, which we name the Mertens Unrolled Network (MU-Net), for the purpose of fine-tuning the HDR output of through-windshield images. Reconstructed faces from this novel HDR method are then evaluated and compared against other traditional and experimental HDR methods in a pre-trained state-of-the-art (SOTA) facial recognition pipeline, verifying the efficacy of our approach.

Ruby, Max↗
DOE OSTI2025

SITCOMTN-149: An Interim Report on the LSSTComCam On-Sky Campaign

From 24 October to 11 December 2024, the NSF-DOE Vera C. Rubin Observatory conducted an on-sky campaign using the engineering LSST Commissioning Camera (LSSTComCam) to test the end-to-end functionality of hardware and software, as well as operational procedures. This interim report provides a preliminary technical overview of our understanding of the integrated system performance based tests and analyses conducted during the LSSTComCam on-sky campaign. The objectives are to synthesize what we have learned about the system in a timely way to inform on-going commissioning efforts, and to inform the Rubin science community on the progress of the LSSTComCam on-sky campaign. The report is organized into sections to describe major activities during the campaign, as well as multiple aspects of the demonstrated system and science performance. All of the results presented here are to be understood as work in progress using engineering data and the initial versions of the data processing pipelines; the report is a living document that will be updated as analyses are refined.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2026

OpenSAMPL: An Open Source Library for Timing and Synchronization Measurements and Analytics

Today's power grid operators are implementing timing and synchronization solutions that provide resilience to Global Navigation Satellite System (GNSS) vulnerabilities. These vendor-specific solutions often come with additional software applications that are designed to monitor that vendor's synchronization performance data. However, resilient timing architectures often resulting in multi-vendor solutions, including approaches that blend terrestrial clocks with space-based subscription services. In such an environment, collecting, analyzing, and visualizing data from a variety of sources within a single platform was heretofore not possible. To address this need, the US Department of Energy's Center for Alternative Synchronization and Timing (CAST) developed OpenSAMPL, the Open Synchronized Analytics and Monitoring Platform, an open-source Python framework for processing, loading, and observing clock measurement data from distributed devices. OpenSAMPL enables the ingestion of diverse clock-probe sources into a scalable time-series database and applies robust analytics. OpenSAMPL currently supports two vendor data pipelines, and will be extended to more in the near future, enabling seamless monitoring of a variety of timing and synchronization devices in a common environment.

Grant, Josh [ORNL] (ORCID:0000000163475060)↗
DOE OSTI2024

Synthetic Biology PacBio/JAWS QC Analysis (PBJ) v3.0

This software was designed as a sequence validation tool for the assembly of synthetic constructs. It analyzes FASTQ files against a list of reference sequences, combining the results from eight sequencing libraries to generate a summary, and the files needed to view the results in the Integrative Genomics Viewer (IGV) application for manual verification. This was developed for FASTQ files generated by PacBio sequencing, but could be used on any FASTQ files that do not have paired end reads. It can be used to analyze one - eight libraries at a time, and assumes that each construct sequence in the reference will be in each pool, however, this is not a requirement. This is used to identify which libraries of pooled sequences contains a perfect match, or fixable match to the reference file. This pipeline uses many freely available open source libraries, the value added is that in our application the steps of the pipeline are defined in Workflow Description Language (WDL) and run through the Cromwell workflow engine in Docker containers, for easy distribution and set up, as well as the user friendly html summary that is generated.

Simirenko, Lisa↗
DOE OSTI2026

ObstacleSense: Low-Power Neuromorphic Vision for Corridor Obstacle Awareness in Low-Level ADAS

The automotive industry’s pursuit of Level 5 autonomy is constrained by substantial perception-compute power requirements, often reaching 1, 000 + watts in full autonomy stacks. Reducing this energy burden requires rethinking perception not only at the high-end autonomy level, but also at the foundational Advanced Driver Assistance Systems (ADAS) level where low-power, safety-critical sensing can have broad impact. Neuromorphic vision provides a promising starting point: HD Dynamic Vision Sensors (DVS) can operate below 100 mW at the sensor level by reporting only asynchronous brightness changes. However, low-power sensing alone is insufficient if downstream perception reintroduces dense, energy-intensive computation. In particular, many event-driven object-detection pipelines still rely on CNN backbones, while purely spiking alternatives often trade away accuracy or ignore deployment constraints. We introduce ObstacleSense, a highly compact, CNN-free hybrid ANN–SNN framework for Level 0–1 forward-corridor obstacle awareness. Instead of performing full-scene object detection with a convolutional feature backbone, ObstacleSense targets the safety-critical question of whether the ego corridor is occupied and how far the nearest obstacle is. The architecture combines polarity-conditioned event encoding, lightweight temporal spiking dynamics, axial spatial mixing, and coarse-to-fine range estimation within a regular fixed-grid compute pattern. This design avoids the dense CNN backbone commonly used in event-based detection while maintaining a small state footprint suitable for eventual small-FPGA deployment. Before hardware mapping, we evaluate the software implementation using a model-side power proxy derived from MACs, weight and activation traffic, and spiking state updates under shared FP16 assumptions. On simulated CARLA event corpora, the deployment-oriented model achieves 0.9464 objectness F1, 0.9978 grid-level mAP, and 0.8987 m distance Mean Absolute Error at an estimated 1.92 mW proxy cost, while maintaining performance on unseen generalization test sequences.

Johnson-Scott, Zac [ORNL]↗
DOE OSTI2022

Constraining galaxy–halo connection with high-order statistics

ABSTRACT We investigate using three-point statistics in constraining the galaxy–halo connection. We show that for some galaxy samples, the constraints on the halo occupation distribution parameters are dominated by the three-point function signal (over its two-point counterpart). We demonstrate this on mock catalogues corresponding to the Luminous red galaxies (LRGs), Emission-line galaxies (ELGs), and quasars (QSOs) targeted by the Dark Energy Spectroscopic Instrument (DESI) Survey. The projected three-point function for triangle sides less up to 20 h−1 Mpc measured from a cubic Gpc of data can constrain the characteristic minimum mass of the LRGs with a preci sion of 0.46 per cent. For comparison, similar constraints from the projected two-point function are 1.55 per cent. The improvements for the ELGs and QSOs targets are more modest. In the case of the QSOs, it is caused by the high shot-noise of the sample, and in the case of the ELGs, it is caused by the range of halo masses of the host haloes. The most time-consuming part of our pipeline is the measurement of the three-point functions. We adopt a tabulation method, proposed in earlier works for the two-point function, to significantly reduce the required compute time for the three-point analysis.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2026

NanoPSD: A software for automatic detection of Nano-Particle Shape Distribution in electron microscopy images

Accurate quantification of the size and morphology of nanoparticles from electron microscopy (EM) images is essential to understand growth mechanisms, surface reactivity, and functional behavior in nanoscale materials. Manual analysis remains slow, subjective, and difficult to reproduce in large datasets. We introduce NanoPSD (Nano-Particle Shape Distribution), an open-source and fully automated framework for quantitative particle detection and morphology analysis from EM images. NanoPSD integrates adaptive contrast enhancement, polarity-agnostic scale-bar detection, Optical Character Recognition (OCR)-based calibration, and classical segmentation via Otsu thresholding with morphological refinement. Particle contours are used to extract geometric descriptors, including equivalent circular diameter, aspect ratio, circularity, and solidity, enabling automated classification into spherical, rod-like, and aggregate morphologies. The framework supports both single-image and batch processing, generating publication-quality visualizations, LaTeX-ready tables, and structured comma-separated values (CSV) datasets. As a demonstration, we applied NanoPSD to plasma-synthesized nanoparticle samples diagnosed via transmission electron microscopy (TEM). The code produced statistically robust size and morphology distributions spanning a few to tens of nanometers with minimal user supervision. The pipeline demonstrates high reproducibility and scalability, processing large image collections with consistent calibration and output formatting. Its modular design enables seamless integration of future deep-learning-based segmentation models, providing a pathway toward intelligent, data-driven electron microscopy analysis.

36 MATERIALS SCIENCE↗