Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Software pipelines”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Transcript and blood-microbiome analysis towards a blood diagnostic tool for goats affected by Haemonchus contortus

The Alpine goat (Capra aegagrus hircus) is parasitized by the barber pole worm (Haemonchus contortus). Hematological parameters from transcript and metagenome analysis in the host are reflective of infestation. We explored comparisons between blood samples of control, infected, infected zoledronic acid-treated, and infected antibody (anti-γδ T cells) treated wethers under controlled conditions. Seven days post-inoculation (dpi), we identified 7,627 transcripts associated with the different treatment types. Microbiome measurements at 7 dpi revealed fewer raw read counts across all treatments and a less diverse microbial flora than at 21 dpi. This study identifies treatment specific transcripts and an increase in microflora abundance and diversity as wethers age. Further, F/B ratio reflect health, based on depression or elevation above thresholds defined by the baseline of non-infected controls. Forty Alpine wethers were studied where blood samples were collected from five goats in four treatment groups on 7 dpi and 21 dpi. Transcript and microbiome profiles were obtained using the Partek Flow (St. Louis, Missouri, USA) software suites pipelines. Inflammation comparisons were based on the Firmicutes/Bacteriodetes ratios that are calculated as well as the reduction of microbial diversity.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2025

POWER DATA PIPELINE

SF-25-081 Utility software for creating high-performance data pipelines to extract, load, and transform raw electric power systems measurements. For use with anomaly detection models training workflows. The software supports the project: Adaptive Cybersecurity for DER: A Game-Theoretic and Machine Learning approach for Real-Time Threat Detection and Mitigation

Plathottam, Silby Jose [Argonne National Laborator↗
DOE OSTI2020

Data analysis and modeling pipelines for controlled networked social science experiments

There is large interest in networked social science experiments for understanding human behavior at-scale. Significant effort is required to perform data analytics on experimental outputs and for computational modeling of custom experiments. Moreover, experiments and modeling are often performed in a cycle, enabling iterative experimental refinement and data modeling to uncover interesting insights and to generate/refute hypotheses about social behaviors. The current practice for social analysts is to develop tailor-made computer programs and analytical scripts for experiments and modeling. This often leads to inefficiencies and duplication of effort. In this work, we propose a pipeline framework to take a significant step towards overcoming these challenges. Our contribution is to describe the design and implementation of a software system to automate many of the steps involved in analyzing social science experimental data, building models to capture the behavior of human subjects, and providing data to test hypotheses. The proposed pipeline framework consists of formal models, formal algorithms, and theoretical models as the basis for the design and implementation. We propose a formal data model, such that if an experiment can be described in terms of this model, then our pipeline software can be used to analyze data efficiently. The merits of the proposed pipeline framework is elaborated by several case studies of networked social science experiments.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

RTN-056: Study of the Photon Transfer Curve in the CCD detectors of the Vera C. Rubin Observatory

The RECA internship program provides Colombian students with an opportunity to enhance their research skills in Astronomy, Astrophysics, and Cosmology. During this three-month program, our main objective was to study the Photon Transfer Curves (PTC) of the Vera C. Rubin Observatory, specifically the gain, and to compare it with the gain obtained through pairs of flats. Overall, the study of PTCs is crucial in understanding the performance of detectors and instruments used in Astronomy. The Vera C. Rubin Observatory is an important facility that will enable researchers to carry out a wide range of studies in this field, making it essential to investigate its gain performance. We used run 13144 to construct the PTCs and 13186 to analyze the crosstalk. We employed the LSST Science Pipelines (also known as the DM stack), a software under development for this observatory, which performs all the necessary reductions for the construction of the PTCs. We also used simulations to replicate the observed effects. Initially, we found a 5% difference between the gain calculated by PTC and pairs of flats for a flow range between 5000 and 10000 ADU. Simulations showed that this difference was due to the handling of statistics and the assumption that the distribution following the Lupton equation is Gaussian. We found an error interval for this flow region based on the vendor, with (1.8 ± 0.7, 4.1 ± 0.9) % for E2V and (0.85 ± 0.7, 2.2 ± 0.9) % for ITL. From the PTC, we also obtained the average Full Well Capacity of LSSTCam as 130000 ± 10000$ electrons. We identified a list of segments where we found differences with the results obtained by SLAC National Acceleration Laboratory in PTC parameters, low saturation level, or other defects. We detected and corrected the effect of statistics in the gain calculation using pairs of flats and proposed a code change, which was implemented in the pipeline software. We do not recommend correcting for crosstalk as it does not significantly affect the parameters and does not change the shape of the PTC. However, the opposite is true for the nonlinearity correction.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

Soil Incubation, Methane Pool Dilution, Edgewater, MD, USA, July 2022 v1.0.1

Isotopic pool dilution is a powerful approach to quantify gross biogeochemical transformation rates, but remains seldom used despite its potential. To facilitate broader implementation of pool dilution methods, we developed a user-friendly R package that optimizes gross production and consumption rates (and optionally fractionation constants as well) based on standard pool dilution time series data. This data package is the soil methane pool dilution data and code for building and using this software, including source code (R) and raw data (.csv). This package features extensive documentation and example analyses, and is easily integrated into analytical pipelines as R software is free and open source. Operation of this package requires R version 2.10 or newer, and we recommend its use in conjunction with packages testthat (3.0.0), knitr, rmarkdown, ggploto2, tibble, and tidyr.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

SynBio QC-Library QC (LibQC) v2.0.0

This software is a pipeline used to identify the distribution of user-specified sequences in oligo pools that have been sequenced on the MiSeq platform for quality control purposes. The software counts instances of user-specified sequences (inserts or spacers) identified in sequencing data, returning the distribution of sequences in each pool and the length distribution of all the sequences. Among other types similar types of projects, this can be used to assess the quality of a guideRNA library once it has been inserted into the vector of choice to ensure that there is an even distribution of all the individual protospacers in the pool. This pipeline uses many freely available open source libraries, the value added is that in our application the steps of the pipeline are defined in Workflow Description Language (WDL) and run through the Cromwell workflow engine in Docker containers, for easy distribution and set up.

Simirenko, Lisa↗
DOE OSTI2026

Detector Interface for Streaming, Control, and Open-source integration (DISCO) v1.0.0

This suite consists of a multi-package ecosystem featuring detector emulators, EPICS areaDetector drivers, and remote server frameworks designed for the Advanced Light Source (ALS). Engineered for high-bandwidth devices—including VFCCD, Timepix3, Timepix4, and related pixel detectors—the software simulates hardware, wraps vendor SDKs into remote-callable servers, and integrates with open-source control systems. Key Capabilities: Distributed SDK Architecture: Server packages wrap hardware-specific SDKs, allowing areaDetector drivers to execute remote framework calls. This isolates proprietary libraries from the EPICS IOC, enhancing stability and enabling distributed computing across beamline networks. Device Support: Custom drivers for VFCCD, the Timepix family, and similar sensors optimize the data path from hardware control to high-speed transport. Full-Stack Emulation: Sophisticated emulator packages allow end-to-end pipeline testing and software development without requiring physical hardware or beam time. Integrated Workflows: Supports high-bandwidth streaming for real-time analysis and robust, metadata-rich file-based workflows (e.g., HDF5/NeXus). By standardizing interfaces across heterogeneous hardware, this suite reduces technical debt. It provides the ALS with a scalable, open-source solution to manage massive data rates within a unified control environment.

Mahl, Johannes [Lawrence Berkeley National Laborat↗
DOE OSTI2026

NLR Data Processing Pipeline for MADIS [SWR-26-050]

The NLR Data Processing Pipeline for MADIS software package is for downloading, processing, and performing QA/QC on MADIS data. Designed to handle the following steps: 1) Download all MADIS data as compressed netcdf files for a given time period. 2) Unpack netcdf files into timeseries csvs for each coordinate within the given bounding box. 3) Process the csvs to filter according to quality control checks and convert variables to correct units. 4) Write processed csvs to a single nc file.

Benton, Brandon [National Laboratory of the Rockie↗
DOE OSTI2022

Accelerating the Inference of the Exa.TrkX Pipeline

Recently, graph neural networks (GNNs) have been successfully used for a variety of particle reconstruction problems in high energy physics, including particle tracking. The Exa.TrkX pipeline based on GNNs demonstrated promising performance in reconstructing particle tracks in dense environments. It includes five discrete steps: data encoding, graph building, edge filtering, GNN, and track labeling. All steps were written in Python and run on both GPUs and CPUs. In this work, we accelerate the Python implementation of the pipeline through customized and commercial GPU-enabled software libraries, and develop a C++ implementation for inferencing the pipeline. The implementation features an improved, CUDA-enabled fixed-radius nearest neighbor search for graph building and a weakly connected component graph algorithm for track labeling. GNNs and other trained deep learning models are converted to ONNX and inferenced via the ONNX Runtime C++ API. The complete C++ implementation of the pipeline allows integration with existing tracking software. We report the memory usage and average event latency tracking performance of our implementation applied to the TrackML benchmark dataset.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2023

VDiSC: An Open Source Framework for Distributed Smart City Vision and Biometric Surveillance Networks

Recent global growth in the interest of smart cities has led to trillions of dollars of investment toward research and development. These connected cities have the potential to create a symbiosis of technology and society and revolutionize the cost of living, safety, ecological sustainability, and quality of life of societies on a world-wide scale. Some key components of the smart city construct are connected smart grids, self-driving cars, federated learning systems, smart utilities, large-scale public transit, and proactive surveillance systems. While exciting in prospect, these technologies and their subsequent integration cannot be attempted without addressing the potential societal impacts of such a high degree of automation and data sharing. Additionally, the feasibility of coordinating so many disparate tasks will require a fast, extensible, unifying framework. To that end, we propose the Distributed Smart City framework for Vision, or VDiSC. VDiSC serves as a unified biometric API harness that allows for seamless evaluation, deployment, and simple pipeline creation for heterogeneous biometric software. VDiSC additionally provides a fully declarative capability for defining and coordinating custom machine learning and sensor pipelines, allowing the distribution of processes across otherwise incompatible hardware and networks. VDiSC ultimately provides a way to quickly configure, hot-swap, and expand large coordinated or federated systems online without interruptions for maintenance. Because much of the data collected in a smart city contains Personally Identifying Information (PII), VDiSC also provides built-in tools and layers to ensure secure and encrypted streaming, storage, and access of PII data across distributed systems.

Brogan, Joel↗
DOE OSTI2025

ADEPT: A Pedagogical Framework for Integrating Agentic AI with Deterministic Scientific Workflows

The integration of Large Language Models (LLMs) into scientific research promises to accelerate discovery, yet a significant gap remains between the dynamic reasoning of Artificial Intelligence (AI) agents and the static, deterministic nature of canonical scientific workflows. This paper introduces ADEPT (Agentic Discovery and Exploration Platform for Tools), a reference architecture and pedagogical framework explicitly designed to bridge this gap. ADEPT's primary mission is to provide a transparent, "glass-box" environment where researchers and engineers can learn to effectively wrap established scientific software (e.g., BLAST, Nextflow pipelines) and compose it into reliable, agent-driven workflows. We describe its modular, multi-server architecture, which leverages the Model Context Protocol (MCP) for tool serving, LangGraph for robust agentic orchestration, and a secure nsjail-based sandbox for safe code execution. By prioritizing architectural clarity, safety, and modularity, ADEPT serves as an extensible blueprint for building trustworthy AI-augmented systems and fosters the collaborative development necessary to responsibly employ agentic AI for science. We provide practical examples of how to adapt and extend this framework, highlighting its utility in workforce development and AI-readiness capabilities across research and development projects.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Automated pipeline processing X-ray diffraction data from dynamic compression experiments on the Extreme Conditions Beamline of PETRA III

Presented and discussed here is the implementation of a software solution that provides prompt X-ray diffraction data analysis during fast dynamic compression experiments conducted within the dynamic diamond anvil cell technique. It includes efficient data collection, streaming of data and metadata to a high-performance cluster (HPC), fast azimuthal data integration on the cluster, and tools for controlling the data processing steps and visualizing the data using the DIOPTAS software package. This data processing pipeline is invaluable for a great number of studies. The potential of the pipeline is illustrated with two examples of data collected on ammonia–water mixtures and multiphase mineral assemblies under high pressure. The pipeline is designed to be generic in nature and could be readily adapted to provide rapid feedback for many other X-ray diffraction techniques, e.g. large-volume press studies, in situ stress/strain studies, phase transformation studies, chemical reactions studied with high-resolution diffraction etc.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

PERCEPTIVE: an R shiny $\underline{p}$ipelin$\underline{e}$ for the p$\underline{r}$edi$\underline{c}$tion of $\underline{ep}$igenetic modula$\underline{t}$ors $\underline{i}$n no$\underline{v}$el sp$\underline{e}$cies

Epigenetic processes are central to regulating gene expression, genome stability, and metabolic function across the tree of life; yet, their roles remain underexplored in microalgae, especially as new species continue to be identified and characterized. This is likely due to the cumbersome nature and species-dependent attributes of epigenetic wet-lab methodologies, which preclude the rapid identification of epigenetic modifications and modulators. However, there is high conservation of epigenetic processes from budding yeast to humans; in many cases, one may infer how behavior and function are epigenetically regulated in novel species by identifying epigenetic modulators, or the proteins responsible for conferring epigenetic modifications. Here, to this end, we have developed a graphical software package, titled PERCEPTIVE (pipeline for the prediction of epigenetic modulators in novel species). This platform solely uses the genomic sequence of an algal species, and preexisting information from other model organisms, to predict the epigenetic modulators and associated modifications in algae. Predictions are presented to the user in a graphical interface, which provides literature-based interpretation of results, enabling users to quickly understand potential epigenetic processes in their algal species of interest and plan follow-up experiments. To test PERCEPTIVE, we predicted epigenetic modulators in several feedstock candidate algae species. To validate these predictions, wet-lab studies were performed, including mass spectrometry; these results underscore the high accuracy of PERCEPTIVE predictions. Overall, PERCEPTIVE represents a powerful in silico tool for the research and manipulation of algal species, which does not require a priori knowledge of epigenetics and is accessible to a broad set of investigators.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

WizEM

The software provides a flexible pipeline for artificial intelligence (AI) assisted analysis of data collected from an electron microscope. This tool distinctly provides near real-time image quantification results during data collection, incorporating feedback from a user to tune performance.

Akers, Sarah [Pacific Northwest National Laborator↗
DOE OSTI2020

The updated DESGW processing pipeline for the third LIGO/VIRGO observing run

The DESGW group seeks to identify electromagnetic counterpartsof gravitational wave events seen by the LIGO-VIRGO network, such as thoseexpected from binary neutron star mergers or neutron star- black hole mergers.DESGW was active throughout the first two LIGO observing seasons, followingup several binary black hole mergers and the first binary neutron star merger,GW170817. We describe the modifications to the observing strategy generationand image processing pipeline between the second (ending in August 2017)and third (beginning in April 2019) LIGO observing seasons. The modifica-tions include a more robust observing strategy generator, further parallelizationof the image reduction software and dierence imaging processing pipeline,data transfer streamlining, and a web page listing identified counterpart candi-dates that updates in real time. Taken together, the additional parallelizationsteps enable us to identify potential electromagnetic counterparts within fullycalibrated search images in less than one hour, compared to the 3-5 hours itwould typically take during the first two seasons. These performance improve-ments are critical to the entire EM followup community, as rapid identification(or rejection) of candidates enables detailed spectroscopic followup by multipleinstruments as soon as possible, leading to more information about the environ-ment immediately following such gravitational wave events.

Herner, Kenneth R.↗
DOE OSTI2023

TDCOSMO. X. Automated modeling of nine strongly lensed quasars and comparison between lens-modeling software

When strong gravitational lenses are to be used as an astrophysical or cosmological probe, models of their mass distributions are often needed. We present a new, time-efficient automation code for the uniform modeling of strongly lensed quasars with GLEE, a lens-modeling software for multiband data. By using the observed positions of the lensed quasars and the spatially extended surface brightness distribution of the host galaxy of the lensed quasar, we obtain a model of the mass distribution of the lens galaxy. We applied this uniform modeling pipeline to a sample of nine strongly lensed quasars for which images were obtained with the Wide Field Camera 3 of the Hubble Space Telescope. The models show well-reconstructed light components and a good alignment between mass and light centroids in most cases. We find that the automated modeling code significantly reduces the input time during the modeling process for the user. The time for preparing the required input files is reduced by a factor of 3 from ~3 h to about one hour. The active input time during the modeling process for the user is reduced by a factor of 10 from ~ 10 h to about one hour per lens system. This automated uniform modeling pipeline can efficiently produce uniform models of extensive lens-system samples that can be used for further cosmological analysis. A blind test that compared our results with those of an independent automated modeling pipeline based on the modeling software Lenstronomy revealed important lessons. Quantities such as Einstein radius, astrometry, mass flattening, and position angle are generally robustly determined. Other quantities, such as the radial slope of the mass density profile and predicted time delays, depend crucially on the quality of the data and on the accuracy with which the point spread function is reconstructed. Better data and/or a more detailed analysis are necessary to elevate our automated models to cosmography grade. Nevertheless, our pipeline enables the quick selection of lenses for follow-up and further modeling, which significantly speeds up the construction of cosmography-grade models. This important step forward will help us to take advantage of the increase in the number of lenses that is expected in the coming decade, which is an increase of several orders of magnitude.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

TomoPyUI : a user-friendly tool for rapid tomography alignment and reconstruction

The management and processing of synchrotron and neutron computed tomography data can be a complex, labor-intensive and unstructured process. Users devote substantial time to both manually processing their data ( i.e. organizing data/metadata, applying image filters etc. ) and waiting for the computation of iterative alignment and reconstruction algorithms to finish. In this work, we present a solution to these problems: TomoPyUI , a user interface for the well known tomography data processing package TomoPy . This highly visual Python software package guides the user through the tomography processing pipeline from data import, preprocessing, alignment and finally to 3D volume reconstruction. The TomoPyUI systematic intermediate data and metadata storage system improves organization, and the inspection and manipulation tools (built within the application) help to avoid interrupted workflows. Notably, TomoPyUI operates entirely within a Jupyter environment. Herein, we provide a summary of these key features of TomoPyUI , along with an overview of the tomography processing pipeline, a discussion of the landscape of existing tomography processing software and the purpose of TomoPyUI , and a demonstration of its capabilities for real tomography data collected at SSRL beamline 6-2c.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗