Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “pipeline”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 325 records · Page 18
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

A NWB-based dataset and processing pipeline of human single-neuron activity during a declarative memory task

A challenge for data sharing in systems neuroscience is the multitude of different data formats used. Neurodata Without Borders: Neurophysiology 2.0 (NWB:N) has emerged as a standardized data format for the storage of cellular-level data together with meta-data, stimulus information, and behavior. A key next step to facilitate NWB:N adoption is to provide easy to use processing pipelines to import/export data from/to NWB:N. Here, we present a NWB-formatted dataset of 1863 single neurons recorded from the medial temporal lobes of 59 human subjects undergoing intracranial monitoring while they performed a recognition memory task. We provide code to analyze and export/import stimuli, behavior, and electrophysiological recordings to/from NWB in both MATLAB and Python. The data files are NWB:N compliant, which affords interoperability between programming languages and operating systems. This combined data and code release is a case study for how to utilize NWB:N for human single-neuron recordings and enables easy re-use of this hard-to-obtain data for both teaching and research on the mechanisms of human memory.

Science & Technology - Other Topics↗
DOE OSTI2024

Enabling high-throughput enzyme discovery and engineering with a low-cost, robot-assisted pipeline

Abstract As genomic databases expand and artificial intelligence tools advance, there is a growing demand for efficient characterization of large numbers of proteins. To this end, here we describe a generalizable pipeline for high-throughput protein purification using small-scale expression in E. coli and an affordable liquid-handling robot. This low-cost platform enables the purification of 96 proteins in parallel with minimal waste and is scalable for processing hundreds of proteins weekly per user. We demonstrate the performance of this method with the expression and purification of the leading poly(ethylene terephthalate) hydrolases reported in the literature. Replicate experiments demonstrated reproducibility and enzyme purity and yields (up to 400 µg) sufficient for comprehensive analyses of both thermostability and activity, generating a standardized benchmark dataset for comparing these plastic-degrading enzymes. The cost-effectiveness and ease of implementation of this platform render it broadly applicable to diverse protein characterization challenges in the biological sciences.

36 MATERIALS SCIENCE↗
DOE OSTI2020

SYMBA: An end-to-end VLBI synthetic data generation pipeline

Realistic synthetic observations of theoretical source models are essential for our understanding of real observational data. In using synthetic data, one can verify the extent to which source parameters can be recovered and evaluate how various data corruption effects can be calibrated. These studies are the most important when proposing observations of new sources, in the characterization of the capabilities of new or upgraded instruments, and when verifying model-based theoretical predictions in a direct comparison with observational data. We present the SYnthetic Measurement creator for long Baseline Arrays (SYMBA), a novel synthetic data generation pipeline for Very Long Baseline Interferometry (VLBI) observations. SYMBA takes into account several realistic atmospheric, instrumental, and calibration effects. Here, we used SYMBA to create synthetic observations for the Event Horizon Telescope (EHT), a millimetre VLBI array, which has recently captured the first image of a black hole shadow. After testing SYMBA with simple source and corruption models, we study the importance of including all corruption and calibration effects, compared to the addition of thermal noise only. Using synthetic data based on two example general relativistic magnetohydrodynamics (GRMHD) model images of M 87, we performed case studies to assess the image quality that can be obtained with the current and future EHT array for different weather conditions. Our synthetic observations show that the effects of atmospheric and instrumental corruptions on the measured visibilities are significant. Despite these effects, we demonstrate how the overall structure of our GRMHD source models can be recovered robustly with the EHT2017 array after performing calibration steps, which include fringe fitting, a priori amplitude and network calibration, and self-calibration. With the planned addition of new stations to the EHT array in the coming years, images could be reconstructed with higher angular resolution and dynamic range. In our case study, these improvements allowed for a distinction between a thermal and a non-thermal GRMHD model based on salient features in reconstructed images.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

The Spectroscopic Classification of Astronomical Transients (SCAT) Survey: Overview, Pipeline Description, Initial Results, and Future Plans

Here, we present the Spectroscopic Classification of Astronomical Transients (SCAT) survey, which is dedicated to spectrophotometric observations of transient objects such as supernovae and tidal disruption events. SCAT uses the SuperNova Integral-Field Spectrograph (SNIFS) on the University of Hawai’i 2.2 m (UH2.2m) telescope. SNIFS was designed specifically for accurate transient spectrophotometry, including absolute flux calibration and host-galaxy removal. We describe the data reduction and calibration pipeline including spectral extraction, telluric correction, atmospheric characterization, nightly photometricity, and spectrophotometric precision. We achieve ≲5% spectrophotometry across the full optical wavelength range (3500–9000 Å) under photometric conditions. The inclusion of photometry from the SNIFS multi-filter mosaic imager allows for decent spectrophotometric calibration (10%–20%) even under unfavorable weather/atmospheric conditions. SCAT obtained ≈640 spectra of transients over the first 3 yr of operations, including supernovae of all types, active galactic nuclei, cataclysmic variables, and rare transients such as superluminous supernovae and tidal disruption events. These observations will provide the community with benchmark spectrophotometry to constrain the next generation of hydrodynamic and radiative transfer models.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2026

Crystal generation using the fully differentiable pipeline and latent space optimization

We present a materials generation framework that couples a symmetry-conditioned variational autoencoder with a differentiable SO(3) power spectrum objective to steer candidates toward a specified local environment under the crystallographic constraints. In particular, we implement a fully differentiable pipeline that performs batch-wise optimization on both direct and latent crystallographic representations. Using the GPU acceleration, the implementation achieves about fivefold speed compared to our previous CPU workflow, while yielding comparable outcomes. In addition, we introduce the optimization strategy that alternatively performs optimization on the direct and latent crystal representations. This dual-level relaxation approach can effectively escape local minima defined by different objective gradients, thus increasing the success rate of generating complex structures satisfying the target local environments. This framework can be extended to systems consisting of multi-components and multi-environments, providing a scalable route to generate material structures with the target local environment.

conditional VAE↗
DOE OSTI2026

nf-core/proteinfamilies: a scalable pipeline for the generation of protein families

The growth of metagenomics-derived amino acid sequence data has transformed our understanding of protein function, microbial diversity, and evolutionary relationships. However, the vast majority of these proteins remain functionally uncharacterized. Grouping the millions of such uncharacterized sequences with the few experimentally characterized ones allows the transfer of annotations, while the inspection of conserved residues with multiple sequence alignments can provide clues to function, even in the absence of existing functional information. To address the challenges associated with this data surge and the need to group sequences, we present a scalable, open-source, parametrizable Nextflow pipeline (nf-core/proteinfamilies) that generates nascent protein families or assigns new proteins to existing families. The computational benchmarks demonstrated that resource usage scales approximately linearly with input size, and the biological benchmarks showed that the generated protein families closely resemble manually curated families in widely used databases.

Nextflow↗
DOE OSTI2021

Characterizing the target selection pipeline for the Dark Energy Spectroscopic Instrument Bright Galaxy Survey

ABSTRACT We present the steps taken to produce a reliable and complete input galaxy catalogue for the Dark Energy Spectroscopic Instrument (DESI) Bright Galaxy Survey (BGS) using the photometric Legacy Survey DR8 DECam. We analyse some of the main issues faced in the selection of targets for the DESI BGS, such as star–galaxy separation, contamination by fragmented stars and bright galaxies. Our pipeline utilizes a new way to select BGS galaxies using Gaia photometry and we implement geometrical and photometric masks that reduce the number of spurious objects. The resulting catalogue is cross-matched with the Galaxy And Mass Assembly (GAMA) survey to assess the completeness of the galaxy catalogue and the performance of the target selection. We also validate the clustering of the sources in our BGS catalogue by comparing with mock catalogues and the Sloan Digital Sky Survey (SDSS) data. Finally, the robustness of the BGS selection criteria is assessed by quantifying the dependence of the target galaxy density on imaging and other properties. The largest systematic correlation we find is a 7 per cent suppression of the target density in regions of high stellar density.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2026

Toward an AI-Powered Software Pipeline for Real-Time Tracking and Analysis of Wildfire and Smoke

Real-time tracking of wildfires and smoke is crucial for effective response, minimizing damage, protecting lives, and efficiently managing resources during fire emergencies. We develop a web-based AI-powered pipeline that detects wildfires in aerial video and estimates deployment-relevant behavior metrics, including cumulative burned area, burned-area growth rate, fire spread direction, and smoke dispersion. The system combines a YOLO-based detector with YCbCr-based fire segmentation, HSV-based smoke segmentation, Farneback optical flow, and centroid-based spatiotemporal tracking. Using ground sampling distance (GSD), pixel-level fire masks are converted to physical burned-area measurements by correlating fire pixel counts with camera altitude and tilt angle. We benchmark YOLO variants and non-YOLO baselines (GoogLeNet, CNN, DBN, Autoencoder, U-Net, and AlexNet) on the IEEE FLAME dataset and a newly created aerial frame dataset, Wildfire-DB. Cross-dataset evaluation uses a strict threshold-transfer protocol: decision thresholds are selected on FLAME validation and transferred unchanged to Wildfire-DB to quantify generalization under domain shift. YOLOv6 achieves the strongest cross-dataset frame-level fire detection on Wildfire-DB (ROC-AUC 0.8200, PR-AUC 0.8044, and transferred-threshold F1 0.7596). For tracking-oriented deployment requiring oriented localization, YOLO11-OBB provides the most reliable cross-dataset behavior among OBB-capable models while remaining computationally feasible. To analyze the feasibility of UAV deployment, we further measure inference efficiency using synchronized GPU and CPU power logs on a fixed workload of 1569 frames. YOLO-family models process the video in 5.73–12.47 seconds with net energy of 1247.28–1775.39 J, substantially lower latency and energy than heavier classification and reconstruction baselines. Overall, model optimality depends on operational objectives: YOLOv6 is best for cross-dataset detection robustness, whereas YOL...

Color segmentation↗
DOE OSTI2022

DRIPS: Dynamic Rebalancing of Pipelined Streaming Applications on CGRAs

Coarse-grained reconfigurable arrays (CGRAs) provide higher flexibility than application-specific integrated circuits (ASICs) and higher efficiency than fine-grained reconfigurable devices such as Field Programmable Gate Arrays (FPGAs). However, CGRAs are generally designed to support offloading of a single kernel. While their design, based on communicating functional units, appears to naturally suit data streaming applications composed of multiple cooperating kernels, current approaches only statically partition the resources across application kernels. However, emerging streaming applications at the edge (scientific instruments, sensor networks, network processing) perform much more than digital signal processing and often are data and input dependent. This leads to extremely variable kernel execution times, severely impacting the throughput of the entire pipeline if resources are only statically allocated. Therefore, in this paper, we propose DRIPS — a coarse-grained, dynamically, and partially reconfigurable array for data-dependent streaming applications. We present a unified compiler framework to facilitate the mapping of a given streaming application onto the DRIPS CGRA architecture. The experimental results show that DRIPS achieves an average throughput improvement of 1.46$\times$ across a set of representative applications over a statically partitioned solution. The additional area overhead to enable dynamic rebalancing consumes 16.34% of the entire area for a 5x5 CGRA prototype.

Tan, Cheng↗
DOE OSTI2020

Supercomputing Pipelines Search for Therapeutics Against COVID-19

The urgent search for drugs to combat SARS-CoV-2 has included the use of supercomputers. The use of general-purpose graphical processing units (GPUs), massive parallelism, and new software for high-performance computing (HPC) has allowed researchers to search the vast chemical space of potential drugs faster than ever before. We developed a new drug discovery pipeline using the Summit supercomputer at Oak Ridge National Laboratory to help pioneer this effort, with new platforms that incorporate GPU-accelerated simulation and allow for the virtual screening of billions of potential drug compounds in days compared to weeks or months for their ability to inhibit SARS-COV-2 proteins. Here, this effort will accelerate the process of developing drugs to combat the current COVID-19 pandemic and other diseases.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2023

Alerga: Alert Aggregation and Reasoning in GOOSE Simulation Pipeline

IEC 61850 specifies the Generic Object Oriented Substation Event (GOOSE) protocol as one option for low latency communication of substation-related events. Due to its strict timing requirements, GOOSE lacks any form of encryption or authentication and has only minimal integrity guarantees. These absences render the protocol vulnerable to a variety of communication anomalies, including adversarial action. In particular, an adversary with access to the substation network can launch man in the middle (MITM) attacks. We propose Alerga, a set of tools to allow operators to mitigate some of the risks of the protocol while retaining its strengths. To that end, we have developed first a GOOSE simulation pipeline including data generation, anomaly detection, alert handling, causal reasoning and data visualization components. The simulator is designed to be modular, allowing operators to swap components to better fit their network capabilities. The volume of alert traffic on a substation network threatens operators with alert fatigue. In order to combat this, we secondly present a novel form of alert aggregation and processing, offering operators a condensed view of any threats to the system. Thirdly, to facilitate the handling of these threats, our causal reasoning system traces the alerts back to their most likely cause, generating an initial hypothesis for operators to investigate.

alert aggregation↗
DOE OSTI2024

From Sim to Real: A Pipeline for Training and Deploying Traffic Smoothing Cruise Controllers

Designing and validating controllers for connected and automated vehicles to enhance traffic flow presents significant challenges, from the complexity of replicating real-world stop-and-go traffic dynamics in simulation, to the intricacies involved in transitioning from simulation to actual deployment. In this work, we present a full pipeline from data collection to controller deployment. Specifically, we collect 772 km of driving data from the I-24 in Tennessee, and use it to build a one-lane simulator, placing simulated vehicles behind real-world trajectories. Using policy-gradient methods with an asymmetric critic, we improve fuel efficiency by over 10% when simulating congested scenarios. Our comprehensive approach includes reinforcement learning for controller training, software verification, hardware validation and setup, and navigating various sim-to-real challenges. Furthermore, we analyze the controller's behavior and wave-smoothing properties, and deploy it on four Toyota Rav4’s in a real-world validation experiment on the I-24. Lastly, we release the driving dataset, the simulator and the trained controller, to enable future benchmarking and controller design.

42 ENGINEERING↗
DOE OSTI2021

CAN-D: A Modular Four-Step Pipeline for Comprehensively Decoding Controller Area Network Data

Controller area networks (CANs) are a broadcast protocol for real-time communication of critical vehicle subsystems. Original equipment manufacturers of passenger vehicles hold secret their mappings of CAN data to vehicle signals, and these definitions vary according to make, model, and year. Without these mappings, the wealth of real-time vehicle information hidden in the CAN packets is uninterpretable, severely impeding vehicle-related research, including CAN cybersecurity and privacy studies, aftermarket tuning, efficiency and performance monitoring, and fault diagnosis to name a few. Guided by the four-part CAN signal definition, we present CAN-D (CAN-Decoder), a modular, four-step pipeline for identifying each signal's boundaries (start bit and length), endianness (byte ordering), signedness (bit-to-integer encoding), and by leveraging diagnostic standards, augmenting a subset of the extracted signals with meaningful, physical interpretation. En route to CAN-D, we provide a comprehensive review of the CAN signal reverse engineering research. All previous methods ignore endianness and signedness, rendering them incapable of decoding many standard CAN signal definitions. Incorporating endianness grows the search space from 128 to 4.72E21 signal tokenizations and introduces a web of changing dependencies. In response, we formulate, formally analyze, and provide an efficient solution to an optimization problem, allowing identification of the optimal set of signal boundaries and byte orderings. In addition, we provide two novel, state-of-the-art signal boundary classifiers—both of which are superior to previous approaches in precision and recall in three different test scenarios—and the first signedness classification algorithm, which exhibits a $>$ 97% F-score. Altogether, CAN-D is the only solution with the potential to extract any CAN signal that is also the state of the art. In evaluation on 10 vehicles of different makes, CAN-D's average $\ell ^1$ error is five times better (81% less) than all previous methods and exhibits lower average error, even when considering only signals that meet prior methods’ assumptions. Finally, CAN-D is implemented in lightweight hardware, allowing for an on-board diagnostic (OBD-II) plugin for real-time in-vehicle CAN decoding.

42 ENGINEERING↗
DOE OSTI2024

Investigating In-Situ Fracture Behaviors of Polymer Pipeline Materials in Hydrogen and Hydrogen-Methane Blended Gas Environments

To reduce carbon emissions, the US natural gas infrastructure is seen as a primary solution for efficiently transporting hydrogen gas. Blending hydrogen gas with natural gas and transporting it across a national infrastructure could save significant infrastructure costs. To properly operate the infrastructure under the new gas system, it is critical to understand material compatibility with hydrogen under various conditions. The Blended Gas CRADA, a Hyblend project, is established to determine the material compatibility of existing natural gas pipes with hydrogen gas. In this study, we investigate the in-plane fracture behaviors of MDPEMarlex and HDPEGDB exposed to hydrogen and hydrogen-methane blended gas. Single-edge notch bending geometry is used. All tests are executed in-situ with the gas environment. The experimental results show a significant effect of the gas environment on HDPEGDB specimens, reducing 5% (H2) to 42% (Blended gas) of specific fracture energy compared to non-aged specimens. For the MDPEMarlex, the effects of the gas environment have increased the specific fracture energy by 10% (H2) to 15% (Blended gas). Fracture surfaces of the tested samples are observed using an electronic microscope. The in-plane fracture surface of HDPEGDB shows a pronounced dimple fracture pattern after exposure to hydrogen and blended gas. The expanded fracture pattern contributes to lower the specific fracture energy. These observations provide critical information for validating polymer pipeline materials when interact with hydrogen and hydrogen-blend gas.

Ko, Seunghyun↗
DOE OSTI2021

MetFish: a Metabolomics Pipeline for Studying Microbial Communities in Chemically Extreme Environments

Metabolites have essential roles in microbial communities, including as mediators of nutrient and energy exchange, cell-to-cell communication, and antibiosis. However, detecting and quantifying metabolites and other chemicals in samples having extremes in salt or mineral content using liquid chromatography-mass spectrometry (LC-MS)-based methods remains a significant challenge. Here, we report a facile method based on in situ chemical derivatization followed by extraction for analysis of metabolites and other chemicals in hypersaline samples, enabling for the first time direct LC-MS-based exometabolomics analysis in sample matrices containing up to 2 M total dissolved salts. The method, MetFish, is applicable to molecules containing amine, carboxylic acid, carbonyl, or hydroxyl functional groups, and it can be integrated into either targeted or untargeted analysis pipelines. In targeted analyses, MetFish provided limits of quantification as low as 1 nM, broad linear dynamic ranges (up to 5 to 6 orders of magnitude) with excellent linearity, and low median interday reproducibility (e.g., 2.6%). MetFish was successfully applied in targeted and untargeted exometabolomics analyses of microbial consortia, quantifying amino acid dynamics in the exometabolome during community succession; in situ in a native prairie soil, whose exometabolome was isolated using a hypersaline extraction; and in input and produced fluids from a hydraulically fractured well, identifying dramatic changes in the exometabolome over time in the well.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

A Length Adaptive Algorithm-Hardware Co-design of Transformer on FPGA Through Sparse Attention and Dynamic Pipelining

Transformers are considered one of the most important deep learning models since 2018, in part because it establishes state-of-the-art (SOTA) records and could potentially replace existing Deep Neural Networks (DNNs). Despite the remarkable triumphs, the prolonged turnaround time of Transformer models is a widely recognized roadblock. The variety of sequence lengths imposes additional computing overhead where inputs need to be zero-padded to the maximum sentence length in the batch to accommodate the parallel computing platforms. This paper targets the field-programmable gate array (FPGA) and proposes a coherent sequence length adaptive algorithm–hardware co-design for Transformer acceleration. Particularly, we develop a hardware-friendly sparse attention operator and a length-aware hardware resource scheduling algorithm. The proposed sparse attention operator brings the complexity of attention-based models down to linear complexity and alleviates the off-chip memory traffic. The proposed length-aware resource hardware scheduling algorithm dynamically allocates the hardware resources to fill up the pipeline slots and eliminates bubbles for NLP tasks. Experiments show that our design has very small accuracy loss and has 80.2 × and 2.6 × speedup compared to CPU and GPU implementation, and 4 × higher energy efficiency than state-of-the-art GPU accelerator optimized via CUBLAS GEMM.

Peng, Hongwu↗