SEARCH · Engineering Papers

Results for “Software pipelines”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10

All sources NASA NTRS DOE OSTI

DOE OSTI2022

A multi-modal scanning system to digitize CBRNE emergency response scenes

A handheld system developed to digitize a contextual understanding of the scene at a chemical, biological, radiological, nuclear and/or explosives (CBRNE) events is described. The system uses LiDAR and cameras to create a colorized 3D model of the environment, which helps domain experts that are supporting responders in the field. To generate the digitized model, a responder scans any suspicious objects and the surroundings by carrying the system through the scene. The scanning system provides a real-time user interface to inform the user about scanning progress and to indicate any areas that may have been missed either by the LiDAR sensors or the cameras. Currently, the collected data are post-processed on a different device, building a colorized triangular mesh of the encountered scene, with the intention of moving this pipeline to the scanner at a later point. The mesh is sufficiently compressed to be sent over a reduced bandwidth connection to a remote analyst. Furthermore, the system tracks fiducial markers attached to diagnostic equipment that is placed around the suspicious object. The resulting tracking information can be transmitted to remote analysts to further facilitate their supporting efforts. The paper will discuss the system's design, software components, the user interface used for scanning a scene, the necessary procedures for calibration of the sensors, and the processing steps of the resulting data. The discussion will close by evaluating the system's performance on 11 scenes.

Salathe, Marco↗

DOE OSTI2023

Visualizing Comparisons of Bill of Materials

Protecting critical infrastructure from cyber attacks, natural disasters, and other disruptions is a priority of the U.S. Government. Critical infrastructure includes providing electricity to homes and businesses, supplying natural gas for heating, and producing renewable energy sources. A loss of these services, as seen in the Solarwinds supply chain attack in 2020 , Texas snowstorm of 2021, the Colonial Pipeline cyber incident of 2021, and the Washington power substation attacks in 2022 result in high costs to consumers, disruption of everyday life, and even death. To protect the infrastructure, we first have to know what equipment we are protecting. The complexity of distributed manufacturing and development coupled with the increasing prevalence of cyber and supply chain attacks necessitates a greater understanding of the hardware and software components that comprise equipment in critical infrastructure. When a vulnerability in a single software library can have disastrous consequences, it is vital to understand critical equipment and systems at a granular level. This need has led to increased energy around the development and incorporation of bill-of-materials (BOM) into existing asset management practices to aid in mitigating, and responding to future attacks \cite{noauthor_software_nodate}. While much of the current research is devoted to creating BOMs, it is equally important to develop methodologies for leveraging BOMs to answer questions, such as: How has my software changed? Are two pieces of equipment equivalent? Does this piece of equipment that just arrived match my historical information? In this work, we demonstrate how BOMs can be represented by graph structures. We then describe how these structures can be fed into a graph comparison algorithm to produce a novel interactive visualization that allows us to not only identify differences in BOMs, but show exactly where they are in the product.

Jones, Rebecca D.↗

DOE OSTI2026

Persistent DynAMICS

Persistent DynAMICS is a lightweight software platform that enables distributed sensing systems to operate as a unified, responsive network. It integrates with existing sensors, data systems, and processing tools as an overlay, without requiring hardware replacement or increasing user burden. As data is collected, the system identifies situations that require additional attention and directs selected sensors to gather more information. For example, if a sensor detects an unusual condition along a pipeline, the system can task nearby sensors or different sensing modalities to confirm the event and focus data collection where it is needed. To support this, the platform forms temporary groups of sensors that work together to address specific objectives. Once complete, these groups are dissolved and resources are reassigned as needed. Communication is handled through an integrated messaging framework with client and server components, enabling reliable operation and data continuity even in low-bandwidth or intermittent connectivity environments.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

A multi-modal scanning system to digitize CBRNE emergency response scenes

Prins, Nicholas↗

DOE OSTI2024

Poplar

SAND2025-00683O Poplar is a software tool that generates a phylogenetic tree from input gene and genome sequences. It integrates established tools to identify genes within genomes, group sequences, construct gene trees, and infer a species tree. Poplar processes nucleotide sequences, identifies similar sequences using Nucleotide BLAST, groups them with DBSCAN, aligns sequences with MAFFT, constructs gene trees with RAxML-NG, and infers a species tree using ASTRAL-Pro3. This pipeline provides a structured approach to phylogenetic analysis, facilitating the study of evolutionary relationships among species. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Krishnakumar, Raga↗

DOE OSTI2024

From Sim to Real: A Pipeline for Training and Deploying Traffic Smoothing Cruise Controllers

Designing and validating controllers for connected and automated vehicles to enhance traffic flow presents significant challenges, from the complexity of replicating real-world stop-and-go traffic dynamics in simulation, to the intricacies involved in transitioning from simulation to actual deployment. In this work, we present a full pipeline from data collection to controller deployment. Specifically, we collect 772 km of driving data from the I-24 in Tennessee, and use it to build a one-lane simulator, placing simulated vehicles behind real-world trajectories. Using policy-gradient methods with an asymmetric critic, we improve fuel efficiency by over 10% when simulating congested scenarios. Our comprehensive approach includes reinforcement learning for controller training, software verification, hardware validation and setup, and navigating various sim-to-real challenges. Furthermore, we analyze the controller's behavior and wave-smoothing properties, and deploy it on four Toyota Rav4’s in a real-world validation experiment on the I-24. Lastly, we release the driving dataset, the simulator and the trained controller, to enable future benchmarking and controller design.

42 ENGINEERING↗

DOE OSTI2025

EV-ELM (Electric Vehicle Policies with the Energy Language Model) [SWR-25-156]

Electric Vehicle Policies with the Energy Language Model (EV-ELM) leverages previous work using Large Language Models (LLMs) to find, download, and parse policy information related to energy infrastructure. In this application, we use LLMs to find policy documents related to the permitting and installation of electric vehicle charging infrastructure. This software contains the code to find, download, and parse these documents, while a related data record in the Open Energy Data Initiative (OEDI) will include the resulting output dataset that can be used for downstream analysis. The EV-ELM repository contains code for the EV-ELM project, which focuses on retrieving and processing EV permitting processes using large language models. The project is composed of two pipelines: (1) a web scraping pipeline for discovering and downloading EV permitting documents, and (2) a document parsing and extraction pipeline that processes the downloaded files to produce structured data. The web scraping pipeline is designed to extract relevant information from various websites, while the document parsing pipeline processes and analyzes the extracted documents to derive meaningful insights. Both pipelines depend on the NLR elm repository, which provides essential tools and functionalities for handling and processing the data. The web scraping pipeline is a modified version of the ordinance_gpt example within the elm repository. It has been adapted to fit the specific requirements of the EV-ELM project, ensuring that it effectively captures and processes the necessary information related to EV permitting.

Olson, Reid [National Laboratory of the Rockies (N↗

DOE OSTI2022

SNEWPY: A Data Pipeline from Supernova Simulations to Neutrino Signals

Current neutrino detectors will observe hundreds to thousands of neutrinos from Galactic supernovae, and future detectors will increase this yield by an order of magnitude or more. With such a data set comes the potential for a huge increase in our understanding of the explosions of massive stars, nuclear physics under extreme conditions, and the properties of the neutrino. However, there is currently a large gap between supernova simulations and the corresponding signals in neutrino detectors, which will make any comparison between theory and observation very difficult. SNEWPY is an open-source software package that bridges this gap. The SNEWPY code can interface with supernova simulation data to generate from the model either a time series of neutrino spectral fluences at Earth, or the total time-integrated spectral fluence. Data from several hundred simulations of core-collapse, thermonuclear, and pair-instability supernovae is included in the package. This output may then be used by an event generator such as sntools or an event rate calculator such as the SuperNova Observatories with General Long Baseline Experiment Simulator (SNOwGLoBES). Additional routines in the SNEWPY package automate the processing of the generated data through the SNOwGLoBES software and collate its output into the observable channels of each detector. In this paper we describe the contents of the package, the physics behind SNEWPY, the organization of the code, and provide examples of how to make use of its capabilities.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2021

SNEWPY: A Data Pipeline from Supernova Simulations to Neutrino Signals

Current neutrino detectors will observe hundreds to thousands of neutrinos from a Galactic supernovae, and future detectors will increase this yield by an order of magnitude or more. With such a data set comes the potential for a huge increase in our understanding of the explosions of massive stars, nuclear physics under extreme conditions, and the properties of the neutrino. However, there is currently a large gap between supernova simulations and the corresponding signals in neutrino detectors, which will make any comparison between theory and observation very difficult. SNEWPY is an open-source software package which bridges this gap. The SNEWPY code can interface with supernova simulation data to generate from the model either a time series of neutrino spectral fluences at Earth, or the total time-integrated spectral fluence. Data from several hundred simulations of core-collapse, thermonuclear, and pair-instability supernovae is included in the package. This output may then be used by an event generator such as sntools or an event rate calculator such as SNOwGLoBES. Additional routines in the SNEWPY package automate the processing of the generated data through the SNOwGLoBES software and collate its output into the observable channels of each detector. In this paper we describe the contents of the package, the physics behind SNEWPY, the organization of the code, and provide examples of how to make use of its capabilities.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2021

efam: an e xpanded, metaproteome-supported HMM profile database of viral protein fam ilies

Viruses infect, reprogram and kill microbes, leading to profound ecosystem consequences, from elemental cycling in oceans and soils to microbiome-modulated diseases in plants and animals. Although metagenomic datasets are increasingly available, identifying viruses in them is challenging due to poor representation and annotation of viral sequences in databases. Here, we establish efam, an expanded collection of Hidden Markov Model (HMM) profiles that represent viral protein families conservatively identified from the Global Ocean Virome 2.0 dataset. This resulted in 240 311 HMM profiles, each with at least 2 protein sequences, making efam >7-fold larger than the next largest, pan-ecosystem viral HMM profile database. Adjusting the criteria for viral contig confidence from ‘conservative’ to ‘eXtremely Conservative’ resulted in 37 841 HMM profiles in our efam-XC database. To assess the value of this resource, we integrated efam-XC into VirSorter viral discovery software to discover viruses from less-studied, ecologically distinct oxygen minimum zone (OMZ) marine habitats. This expanded database led to an increase in viruses recovered from every tested OMZ virome by ~24% on average (up to ~42%) and especially improved the recovery of often-missed shorter contigs (<5 kb). Additionally, to help elucidate lesser-known viral protein functions, we annotated the profiles using multiple databases from the DRAM pipeline and virion-associated metaproteomic data, which doubled the number of annotations obtainable by standard, single-database annotation approaches. Together, these marine resources (efam and efam-XC) are provided as searchable, compressed HMM databases that will be updated bi-annually to help maximize viral sequence discovery and study from any ecosystem.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2024

Results from a multi-laboratory ocean metaproteomic intercomparison: effects of LC-MS acquisition and data analysis procedures

Metaproteomics is an increasingly popular methodology that provides information regarding the metabolic functions of specific microbial taxa and has potential for contributing to ocean ecology and biogeochemical studies. A blinded multi-laboratory intercomparison was conducted to assess comparability and reproducibility of taxonomic and functional results and their sensitivity to methodological variables. Euphotic zone samples from the Bermuda Atlantic Time-series Study (BATS) in the North Atlantic Ocean collected by in situ pumps and the autonomous underwater vehicle (AUV) Clio were distributed with a paired metagenome, and one-dimensional (1D) liquid chromatographic data-dependent acquisition mass spectrometry analysis was stipulated. Analysis of mass spectra from seven laboratories through a common bioinformatic pipeline identified a shared set of 1056 proteins from 1395 shared peptide constituents. Quantitative analyses showed good reproducibility: pairwise regressions of spectral counts between laboratories yielded R 2 values averaged 0.62±0.11, and a Sørensen similarity analysis of the top 1000 proteins revealed 70 %–80 % similarity between laboratory groups. Taxonomic and functional assignments showed good coherence between technical replicates and different laboratories. A bioinformatic intercomparison study, involving 10 laboratories using eight software packages, successfully identified thousands of peptides within the complex metaproteomic datasets, demonstrating the utility of these software tools for ocean metaproteomic research. Lessons learned and potential improvements in methods were described. Future efforts could examine reproducibility in deeper metaproteomes, examine accuracy in targeted absolute quantitation analyses, and develop standards for data output formats to improve data interoperability. Together, these results demonstrate the reproducibility of metaproteomic analyses and their suitability for microbial oceanography research, including integration into global-scale ocean surveys and ocean biogeochemical models.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2021

py4DSTEM: A Software Package for Four-Dimensional Scanning Transmission Electron Microscopy Data Analysis

Scanning transmission electron microscopy (STEM) allows for imaging, diffraction, and spectroscopy of materials on length scales ranging from microns to atoms. By using a high-speed, direct electron detector, it is now possible to record a full two-dimensional (2D) image of the diffracted electron beam at each probe position, typically a 2D grid of probe positions. These 4D-STEM datasets are rich in information, including signatures of the local structure, orientation, deformation, electromagnetic fields, and other sample-dependent properties. However, extracting this information requires complex analysis pipelines that include data wrangling, calibration, analysis, and visualization, all while maintaining robustness against imaging distortions and artifacts. In this paper, we present py4DSTEM, an analysis toolkit for measuring material properties from 4D-STEM datasets, written in the Python language and released with an open-source license. We describe the algorithmic steps for dataset calibration and various 4D-STEM property measurements in detail and present results from several experimental datasets. We also implement a simple and universal file format appropriate for electron microscopy data in py4DSTEM, which uses the open-source HDF5 standard. We hope this tool will benefit the research community and help improve the standards for data and computational methods in electron microscopy, and we invite the community to contribute to this ongoing project.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

DOE OSTI2023

MLIR loop optimizations for High-Level Synthesis: a case study

High-Level Synthesis (HLS) tools simplify the design of hardware accelerators by automatically generating Verilog/VHDL code starting from a general purpose software programming language. They include a wide range of optimization techniques in the process, most of them performed on a low-level intermediate representation (IR) of the code. Introducing optimizations on a higher level of abstraction could significantly contribute to the automated design process results; for example, polyhedral techniques for the manipulation of loops could have a significant impact on the generated accelerators when applied on a specialized IR. We use loop pipelining as a case study to explore the introduction of compiler-based transformations on top of an existing HLS process. We leverage the Multi-Level Intermediate Representation (MLIR) framework and an external scheduler to implement the required transformations, and couple them with existing HLS tools to evaluate the improvements that loop pipelining brings to the performance of generated accelerators. The proposed approach can be integrated with other high-level transformations on the MLIR representation, combining different techniques to obtain pre-optimized inputs for HLS that do not have to rely on a specific backend tool.

Curzel, Serena↗

DOE OSTI2022

A reusable neural network pipeline for unidirectional fiber segmentation

Abstract Fiber-reinforced ceramic-matrix composites are advanced, temperature resistant materials with applications in aerospace engineering. Their analysis involves the detection and separation of fibers, embedded in a fiber bed, from an imaged sample. Currently, this is mostly done using semi-supervised techniques. Here, we present an open, automated computational pipeline to detect fibers from a tomographically reconstructed X-ray volume. We apply our pipeline to a non-trivial dataset by Larson et al . To separate the fibers in these samples, we tested four different architectures of convolutional neural networks. When comparing our neural network approach to a semi-supervised one, we obtained Dice and Matthews coefficients reaching up to 98%, showing that these automated approaches can match human-supervised methods, in some cases separating fibers that human-curated algorithms could not find. The software written for this project is open source, released under a permissive license, and can be freely adapted and re-used in other domains.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2025

HyRAM+ (Hydrogen Plus Other Alternative Fuels Risk Assessment Models) v.6.1

SAND2025-11565O HyRAM+ (Hydrogen Plus Other Alternative Fuels Risk Assessment Models) is a tool for conducting quantitative risk assessment (QRA) in transportation systems. HyRAM+ contains validated, simplified release behavior models, engineering models, and generic data relevant to hydrogen installations. HyRAM+’s platform integrates models and data to conduct QRA on user-defined hydrogen or other alternative fuel systems. The software will enable the international safety research community to add validated models to the HyRAM+ platform for use in QRAs. HyRAM (hydrogen-only) versions 1.0 to 3.1 were developed by Sandia for the Department of Energy (DOE) Hydrogen and Fuel Cell Technologies Office. The following agencies contributed to the development of HyRAM+ version 4.0 regarding the addition of methane (natural gas) and propane models: the DOE Vehicle Technologies Office and the Department of Transportation Pipeline and Hazardous Material Safety Administration. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Groth, Katrina [Sandia National Lab. (SNL-NM), Alb↗

DOE OSTI2021

Large-scale Gravitational Lens Modeling with Bayesian Neural Networks for Accurate and Precise Inference of the Hubble Constant

We investigate the use of approximate Bayesian neural networks (BNNs) in modeling hundreds of time delay gravitational lenses for Hubble constant (H 0 ) determination. Here, our BNN was trained on synthetic Hubble Space Telescope quality images of strongly lensed active galactic nuclei with lens galaxy light included. The BNN can accurately characterize the posterior probability density functions (PDFs) of model parameters governing the elliptical power-law mass profile in an external shear field. We then propagate the BNN-inferred posterior PDFs into an ensemble H 0 inference, using simulated time delay measurements from a plausible dedicated monitoring campaign. Assuming well-measured time delays and a reasonable set of priors on the environment of the lens, we achieve a median precision of 9.3% per lens in the inferred H 0 . A simple combination of a set of 200 test lenses results in a precision of 0.5 km s –1 Mpc –1 (0.7%), with no detectable bias in this H 0 recovery test. The computation time for the entire pipeline—including the generation of the training set, BNN training and H0 inference—translates to 9 minutes per lens on average for 200 lenses and converges to 6 minutes per lens as the sample size is increased. Being fully automated and efficient, our pipeline is a promising tool for exploring ensemble-level systematics in lens modeling for H 0 inference.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2025

Hypermut 3: identifying specific mutational patterns in a defined nucleotide context that allows multistate characters

Abstract Motivation The detection of APOBEC3F- and APOBEC3G-induced mutations in virus sequences is useful for identifying hypermutated sequences. These sequences are not representative of viral evolution and can therefore alter the results of downstream sequence analyses if included. We previously published the software Hypermut, which detects hypermutation events in sequences relative to a reference. Two versions of this method are available as a webtool. Neither of these methods consider multistate characters or gaps in the sequence alignment. Results Here, we present an updated, user-friendly web and command-line version of Hypermut with functionality to handle multistate characters and gaps in the sequence alignment. This tool allows for straightforward integration of hypermutation detection into sequence analysis pipelines. As with the previous tool, while the main purpose is to identify G to A hypermutation events, any mutational pattern and context can be specified. Availability and implementation Hypermut 3 is written in Python 3. It is available as a command-line tool at https://github.com/MolEvolEpid/hypermut3 and as a webtool at https://www.hiv.lanl.gov/content/sequence/HYPERMUT/hypermutv3.html.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2026

Differential Privacy in Grid Kitchen: Implementation & Software Documentation

Sharing of power grid feeder models faces significant challenges due to the potential risk of exposing sensitive operational information. Traditional anonymization techniques have shown notable limitations in other sensitive domains, as evidenced by documented re-identification attacks that combine supposedly anonymized datasets with auxiliary information, raising concerns that similar vulnerabilities could affect power grid data. Consequently, there is a pressing need for a more rigorous privacy protection strategy that not only delivers formal mathematical guarantees but also preserves the analytical value of the shared models. To address this challenge, we have enhanced the Grid Kitchen framework by implementing differential privacy mechanisms within the distribution model dehydration pipeline. This implementation carefully calibrates and applies noise to sensitive attributes in feeder models according to configurable privacy levels—low, moderate, and high—each offering different balances between data utility and privacy protection. Our approach uses established noise functions (Gaussian for continuous data and Discrete Laplace for integer values) with parameters carefully calibrated so that the impact of individual data points is effectively masked in the final output. The integration leverages our Noise Catalog, which we developed to categorize feeder model properties by component type, data type, and sensitivity. This catalog guides the application of appropriate noise functions and privacy parameters ($\varepsilon$ and $\delta$) to each attribute, ensuring consistent privacy protection across the model while maintaining its structural integrity and analytical usefulness. This implementation also includes evaluation tools that allow model owners to assess the impact of privacy-preserving transformations before sharing data with external parties. This report provides documentation for the differential privacy capabilities added to the Grid Kitchen project. It includes a primer on differential privacy concepts and their importance in modern data sharing, details the architecture of our implementation, explains the privacy modes and parameter configurations, and offers practical guidance on using the code for applying differential privacy to grid feeder models. Through examples and code snippets, we demonstrate the effective application of these privacy-enhancing technologies, enabling utility operators and researchers to confidently share grid data while protecting sensitive information.

24 POWER TRANSMISSION AND DISTRIBUTION↗