Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data transfer pipeline”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Moving small files in a networked environment

Globally distributed computing infrastructures, such as clouds and supercomputers, are currently used to manage data that is generated with an unprecedented speed from a variety of resources. Coping with this trend, the volume of data exchanged across distant sites increases substantially. To accelerate data transfer, high-speed networks are provided to connect remote sites. Most existing data movement solutions are optimized for moving large files. However, it is still challenging to transfer a large number of small files across networks. This disadvantage not only lowers data transfer performance, but also decreases overall system utilization. Here, we identify that moving small files is mainly constrained by degraded file system throughput, not just network performance as might be suspected. We have built a data transfer pipeline model to analyze the impact of small network I/O and storage I/O on data movement. Extending one of the widely used open source data movement solutions, GridFTP, we demonstrate several appropriate engineering approaches that mitigate the bottleneck and increase data transfer efficiency. We show optimizations that improve data transfer performance more than 5 times. In comparison to existing solutions, our approaches can save a significant amount of system resources for moving lots of small files.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2010

First Results of Venus Express Spacecraft Observations with Wettzell

The ESA Venus Express spacecraft was observed at X-band with the Wettzell radio telescope in October-December 2009 in the framework of an assessment study of the possible contribution of the European VLBI Network to the upcoming ESA deep space missions. A major goal of these observations was to develop and test the scheduling, data capture, transfer, processing, and analysis pipeline. Recorded data were transferred from Wettzell to Metsahovi for processing, and the processed data were sent from Mets ahovi to JIVE for analysis. A turnover time of 24 hours from observations to analysis results was achieved. The high dynamic range of the detections allowed us to achieve a milliHz level of spectral resolution accuracy and to extract the phase of the spacecraft signal carrier line. Several physical parameters can be determined from these observational results with more observational data collected. Among other important results, the measured phase fluctuations of the carrier line at different time scales can be used to determine the influence of the solar wind plasma density fluctuations on the accuracy of the astrometric VLBI observations.

Calves, Guifre Molera↗
NASA NTRS2012

Rapid and Reliable Damage Proxy Map from InSAR Coherence

Future radar satellites will visit SoCal within a day after a disaster event. Data acquisition latency in 2015-2020 is 8 to approx. 15 hours. Data transfer latency that often involves human/agency intervention far exceeds the data acquisition latency. Need interagency cooperation to establish automatic pipeline for data transfer. The algorithm is tested with ALOS PALSAR data of Pasadena, California. Quantitative quality assessment is being pursued: Meeting with Pasadena City Hall computer engineers for a complete list of demolition/construction project 1. Estimate the probability of detection and probability of false alarm 2. Estimate the optimal threshold value.

InSAR↗
DOE OSTI2026

ML–Enabled FPGA Framework for Fast Quantum State Discrimination in Mid-Circuit Measurement Regimes

Accurate and low-latency quantum state discrimination is essential for protocols involving mid-circuit measurement (MCM) and conditional feed-forward. In superconducting quantum systems, conventional readout pipelines transfer measurement data to host processors for post-processing, introducing millisecond-scale delays that far exceed qubit coherence times. To overcome this bottleneck, we present an in-situ machine learning (ML) inference engine implemented on an FPGA for real-time quantum state discrimination. Our design performs inference directly on digitized readout signals with 40 ns latency, supports both qubit and qutrit readout, and enables conditional operations without host-side intervention. This capability is critical for MCM and for feedback-driven protocols such as quantum error correction. We validate the system on superconducting transmon hardware, demonstrating robust discrimination fidelity across multiple qubit and qutrit channels. We further demonstrate conditional qutrit logic driven by FPGA-resident classification, highlighting the potential of low-latency ML-on-FPGA control for NISQ applications and scalable fault-tolerant quantum computing.

Vora, Neel [Lawrence Berkeley National Laboratory ↗
DOE OSTI2025

DoCeph: DPU-Offloaded Messaging in Ceph for Reduced Host CPU Utilization

Ceph is a widely used distributed object store, but its messenger layer imposes substantial CPU overhead on the host. To address this limitation, we propose DoCeph, a DPU-offloaded storage architecture for Ceph that disaggregates the system by offloading the communication-intensive messaging component to the DPU while retaining the storage backend on the host. The DPU efficiently manages communication, using lightweight RPC for metadata operations and DMA for data transfer. Moreover, DoCeph introduces a pipelining technique that overlaps data transmission with buffer preparation, mitigating hardware-imposed transfer size limitations. We implemented DoCeph on a Ceph cluster with NVIDIA BlueField-3 DPUs. Evaluation results indicate that DoCeph cuts host CPU usage by up to 92% while sustaining stable throughput and providing larger performance benefits for object writes over 1 MB.

Park, Kuri [Sogang University]↗
NASA NTRS2003

Thermal Performance Testing Of Cryogenic Piping Systems

Thermal performance measurement of piping systems under actual field conditions is important for space launch development and commercial industry. Knowledge of the true insulating effectiveness is needed in system design, development, and research activities. A new 18-meter-long test apparatus for cryogenic pipelines has been developed. Three different pipelines, rigid or flexible, can be tested simultaneously. Critical factors in heat leak measurements include eliminating heat transfer at end connections and obtaining proper liquid saturation condition. Effects due to variations in the external ambient conditions like wind, humidity, and solar radiation must be minimized. The static method of liquid nitrogen evaporation has been demonstrated, but the apparatus can be adapted for dynamic testing with cryogens, chilled water, or other working fluids. This technology is suited for the development of an industry standard test apparatus and method. Examples of the heat transfer data from testing commercially available pipelines are given. Prototype pipelines are currently being tested and evaluated at the Cryogenics Test Laboratory of NASA Kennedy Space Center.

Fesmire, J. E.↗
DOE OSTI2024

An Intermediate Level Trigger Supernova Pointing Trigger for DUNE Using In-Storage AI

We propose a near-data approach to reduce the buffered supernova data on the DUNE DAQ front-end computers using ML techniques. Based on simulations, we are able to reduce the data by 5 orders of magnitude to 0(100)MB which can then be transferred quickly over ethernet to a single server that executes a full reconstruction and pointing analysis to determine the SN direction. The entire process of reducing the data and executing reconstruction and pointing analysis pipeline takes less time than that required to transfer all the data back to Fermilab before performing more processing to determine the direction of the supernova, which was the baseline plan for DUNE.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
NASA NTRSDate not supplied

Improving Sim-to-Real Transfer in Vision-Based Robot Navigation Via Instance-Level GAN-Based Data Augmentation

Achieving robust vision-based robotic tasks requires large amounts of data, which are often difficult to obtain in real-world scenarios. Simulators and synthetic data offer a cost-effective alternative, but the visual gap between simulation and reality hinders the performance of models when deployed in real-world environments. In this paper, we present a data augmentation pipeline that integrates a foundation model (Segment Anything Model) with an unsupervised image-to-image translation model (CycleGAN) for instance-level domain transfer from simulation to reality. This pipeline enables the generation of realistic labeled data from synthetic images for training supervised machine learning models in vision-based navigation tasks. We evaluate our approach on real-world data for ego-vehicle pose estimation, a critical autonomous navigation task involving the prediction of cross-track position and heading angle relative to road center line markings. The results of our tests show that our GAN-based data augmentation pipeline significantly outperforms models trained solely on simulation data or on data processed with standard image augmentation methods for sim-to-real transfer, enhancing model robustness and generalizability in real-world scenarios. Our method provides a scalable and flexible data augmentation tool for leveraging large synthetic datasets to enhance vision-based robotic navigation tasks.

artificial intelligence↗
NASA NTRS2013

The Hopkins Ultraviolet Telescope: The Final Archive

The Hopkins Ultraviolet Telescope (HUT) was a 0.9 m telescope and moderate-resolution (Delta)lambda equals 3 A) far-ultraviolet (820-1850 Å) spectrograph that flew twice on the space shuttle, in 1990 December (Astro-1, STS-35) and 1995 March (Astro-2, STS-67). The resulting spectra were originally archived in a nonstandard format that lacked important descriptive metadata. To increase their utility, we have modified the original datareduction software to produce a new and more user-friendly data product, a time-tagged photon list similar in format to the Intermediate Data Files (IDFs) produced by the Far Ultraviolet Spectroscopic Explorer calibration pipeline. We have transferred all relevant pointing and instrument-status information from locally-archived science and engineering databases into new FITS header keywords for each data set. Using this new pipeline, we have reprocessed the entire HUT archive from both missions, producing a new set of calibrated spectral products in a modern FITS format that is fully compliant with Virtual Observatory requirements. For each exposure, we have generated quicklook plots of the fully-calibrated spectrum and associated pointing history information. Finally, we have retrieved from our archives HUT TV guider images, which provide information on aperture positioning relative to guide stars, and converted them into FITS-format image files. All of these new data products are available in the new HUT section of the Mikulski Archive for Space Telescopes (MAST), along with historical and reference documents from both missions. In this article, we document the improved data-processing steps applied to the data and show examples of the new data products.

Hopkins↗
DOE OSTI2025

A machine-learning-driven data labeling pipeline for scientific analysis in MLExchange

This study introduces a novel labeling pipeline to accelerate the labeling process of scientific data sets by using artificial intelligence (AI)-guided tagging techniques. This pipeline includes a set of interconnected web-based graphical user interfaces (GUIs), where Data Clinic and MLCoach enable the preparation of machine learning (ML) models for data reduction and classification, respectively, while Label Maker is used for label assignment. Throughout this pipeline, data can be accessed through a direct connection to a file system or through Tiled for access through Hypertext Transfer Protocol (HTTP). Our experimental results present three use cases where this labeling pipeline has been instrumental for the study of large X-ray scattering data sets in the area of pattern recognition, the remote analysis of resonant soft X-ray scattering data and the fine-tuning process of foundation models. These use cases highlight the labeling capabilities of this pipeline, including the ability to label large data sets in a short period of time, to perform remote data analysis while minimizing data movement and to enhance the fine-tuning process of complex ML models with human involvement.

Chavez, Tanny (ORCID:0000000193172896)↗
DOE OSTI2022

Deploying Traffic Smoothing Cruise Controllers Learned from Trajectory Data

Autonomous vehicle-based traffic smoothing con- trollers are often not transferred to real-world use due to challenges in calibrating many-agent traffic simulators. We show a pipeline to sidestep such calibration issues by collecting trajectory data and learning controllers directly from trajectory data that are then deployed zero-shot onto the highway. We construct a dataset of 772.3 kilometers of recorded drives on the I-24. We then construct a simple simulator using the recorded drives as the lead vehicle in front of a simulated platoon consisting of one autonomous vehicle and five human followers. Using policy-gradient methods with an asymmetric critic to learn the controller, we show that we are able to improve average MPG by 11% in simulation on congested trajectories. We deploy this controller to a mixed platoon of 4 autonomous Toyota RAV-4’s and 7 human drivers in a validation experiment and demonstrate that the expected time-gap of the controller is maintained in the real world test. Finally, we release the driving dataset [1], the simulator, and the trained controller at https://github.com/nathanlct/trajectory-training-icra.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
NASA NTRS1990

A bit-serial VLSI array processing chip for image processing

An array processing chip integrating 128 bit-serial processing elements (PEs) on a single die is discussed. Each PE has a 16-function logic unit, a single-bit adder, a 32-b variable-length shift register, and 1 kb of local RAM. Logic in each PE provides the capability to mask PEs individually. A modified grid interconnection scheme allows each PE to communicate with each of its eight nearest neighbors. A 32-b bus is used to transfer data to and from the array in a single cycle. Instruction execution is pipelined, enabling all instructions to be executed in a single cycle. The 1-micron CMOS design contains over 1.1 x 10 to the 6th transistors on an 11.0 x 11.7-mm die.

Heaton, Robert↗
DOE OSTI2026

HydraGNN_GFM_FineTuning4Materials v1.0

This repository enables fine-tuning of the HydraGNN Predictive GFM 2026 — an open-source ensemble of pre-trained graph foundation models for atomistic materials modeling, developed at Oak Ridge National Laboratory. The GFM 2026 is freely available and downloadable via Globus from the OLCF Data Constellation (DOI: 10.13139/OLCF/2562660). Starting from these pre-trained weights, this repository provides a complete transfer learning pipeline for adapting the GFM ensemble to domain-specific molecular and materials property prediction tasks. It includes: 1) Utilities for ensemble fine-tuning with task-specific output heads 2) Example pipelines for eight widely-used materials and molecular datasets 3) Tools for model adaptation and head configuration 4) Data preprocessing utilities for each supported dataset 5) Benchmarking and evaluation scripts

Ungerboeck, Linda↗
DOE OSTI2024

Efficient Parallelization of Irregular Applications on GPU Architectures

With the enlarging computation capacity of general Graphics Processing Units (GPUs), leveraging GPUs to accelerate parallel applications has become a critical topic in academia and industry. However, a wide range of irregular applications with the computation-/memory-intensive nature cannot easily achieve high GPU utilization. The challenges mainly involve the following aspects: first, data dependence leads to coarse-grained kernel and inefficient parallelism; second, heavy GPU memory usage may cause frequent memory evictions and extra overhead of I/O; third, specific computation patterns produce memory redundancies; last, workload balance and data reusability conjunctly benefit the overall performance, but there may exist a dynamic trade-off between them. Targeting these challenges, this dissertation proposes multiple optimizations to accelerate two real-world applications: many-body correlation functions to simulate nuclear physics in a large-scale scientific system; the other is the eALS-based matrix factorization recommendation system. To accelerate the calculations of many-body correlation functions, this dissertation presents three frameworks in GPU memory management and multi-GPU scheduling. Firstly, an optimized systematic GPU memory management framework, MemHC, utilizes a series of new memory reduction designs in GPU memory allocation, CPU/GPU communications, and GPU memory oversubscription. Secondly, an enhanced multi-GPU scheduling framework, MICCO, particularly by taking both data dimension (e.g., data reuse and data eviction) and computation dimension into account. MICCO designs a heuristic scheduling algorithm and a machine learning-based regression model to generate the optimal settings of a proposed new concept to manage the trade-off. Thirdly, a locality-aware multi-GPU scheduling framework. This scheduler leverages pipeline batch generation with a looking-ahead strategy by building local dependency graphs for memory transfer reduction and better data reuse, achieving up to 79.92% memory cost reduction and 1.67x speedup. To parallelize the eALS-based recommendation system, this dissertation proposes an efficient CPU/GPU heterogeneous recommendation system, HEALS. HEALS employs newly designed architecture-adaptive data formats to achieve load balance and good data locality on CPU and GPU. To mitigate the data dependence, HEALS presents a CPU/GPU collaboration model for both task parallelism and data parallelism with multiple kernel computation optimizations. In summary, this dissertation efficiently accelerates two typical irregular applications on GPUs by building four frameworks, including CPU/GPU collaboration, GPU memory management, and multi-GPU scheduling.

Wang, Qihan↗
DOE OSTI2026

Adaptive Cybersecurity for Distributed Energy Resources (AdCyDER): Online Reinforcement Learning with Stackelberg-Optimized Defenses — Pipeline Architecture, Evaluation Methodology, and Findings from a Synthetic-Data Evaluation

This report documents the design and evaluation of an integrated online-learning pipeline developed within the AdCyDER project for Distributed Energy Resource (DER) cybersecurity. The pipeline couples a Reinforcement Learning (RL) attack classifier — which produces an attack-type probability distribution — with a Stackelberg game-theoretic (GT) defense selector that consumes those distributions alongside SME-encoded priors over (defense, attack) effectiveness pairings and perdefense costs to choose grid-health-preserving defenses. The objective is not attack classification per se but production of distributions that drive effective defense selection through the Stackelberg layer, learned from delayed grid-health feedback rather than labeled attack data. AdCyDER as a whole is broader than the work presented here; this report covers the specific RL/GT loop integration and its evaluation. We present the integrated pipeline (SCADA telemetry with Fronius inverter physics, Suricata IDS, time-windowed aggregation, per-facility LSTM classifier, Stackelberg optimizer, OpenC2 actuators), an experimental campaign of 28 eight-hour iterations across three baseline modes, and a pipeline-ordered diagnostic protocol. The protocol identifies two distinct failure modes within the loop: paired supervised ceilings on the same features establish that the deployed online RL classifier (macro F1 ≈ 0.07) sits at least 4.7× below a same-architecture supervised LSTM (≈ 0.34) and 10–11× below a linear feature-signal ceiling (≈ 0.70–0.79 depending on per-facility isolation), localizing the dominant failure to the training procedure; and the reward signal driving online updates carries weak directional coupling with classifier correctness in the methodology-expected direction (multi-lens convergent: top-decile P(true) records produce more frequent state changes and slightly larger improvements, top-vs-bot Cohen’s 𝑑 ≈ −0.19), but at effect magnitudes too small to drive gradient-based learning at the campaign sample size. The original learning hypothesis is not supported by the data. The primary contributions are the diagnostic methodology — proposed as a transferable falsification protocol for online RL/GT defense pipelines learning from delayed environmental reward — and the open, reproducible experimental infrastructure. We outline reward reformulation as the highest-priority aspirational next step given the underpowered-but-aligned Q6 reading, with hardware-in-the-loop evaluation as the broadest scope-expansion option.

Blakely, Benjamin [Argonne National Laboratory (AN↗
DOE OSTI2026

A Real2Sim Digital Twin Pipeline for Photorealistic Robot Simulation: Evaluating VLA Policy Deployment on a Bimanual Mobile Robot

Digital twins that are automatically constructed from robot sensor data offer a promising pathway for scalable Real2Sim and Sim2Real transfer. However, it remains an open question whether photorealistic reconstruction alone is sufficient to support reliable deployment of vision-language-action (VLA) policies. We present a generative-AI-assisted Real2Sim pipeline that generates simulation-ready digital twins from real-world RGB observations with minimal manual intervention. The pipeline uses prompted segmentation to isolate scene components and a generative 3D model to directly produce simulation assets, eliminating the need for traditional multi-view reconstruction or manual 3D modeling.\r\nTo evaluate simulation fidelity, we deploy and compare policies from two VLA models in both the real robot and the reconstructed\r\nsimulation under identical tasks and initial conditions. We compare joint-level action trajectories and analyze how divergence evolves over time in closed-loop execution. Although the reconstructed environments are visually accurate, we observe increasing trajectory divergence during closedloop operation. These results indicate that photorealistic reconstruction alone is insufficient to preserve closed-loop control behavior\r\nin VLA policies, particularly in contact-rich manipulation settings where small perceptual errors compound over time.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Machine Learning Digital Twin for Lithium Ion Battery State of Health Predictions

A digital twin system has been established to model the long term degradation of the state of health of lithium ion batteries. Two data streams result from the computational model of the system and from the physical experiment measurements. A machine learning pipeline has been developed at the nexus of these data streams. Leveraging the unique data sources in multiple transfer learning approaches has lead to the development of multiple cell specific machine learning digital twins. Our discussion on this first of its kind technology will cover challenges in data ingestion, scalability, architecture and orchestration, and research findings.

25 - ENERGY STORAGE↗
DOE OSTI2025

CO2 hydrate crystal thickening, morphology, and Raman spectroscopy in a microfluidic device

Gas hydrates are a solid, crystalline form of water that often form at low temperatures and high pressures. Carbon dioxide (CO2) hydrates may form during carbon dioxide capture and storage (CCS) processes. These solid compounds may form in CO2 pipelines, potentially leading to a full blockage and process shutdown for plug removal. On the other hand, formation of CO2 hydrates may be desired for CO2 capture and separation. In either case, understanding the growth behavior and nature of the hydrates is vital to managing these CCS processes. Using a high-pressure, transparent microfluidic reactor, the crystalline film thickening of CO2 hydrates was observed and measured through visual microscopy and Raman spectroscopy. The impact of subcooling, pressure, and CO2 flow rate was investigated, and only CO2 flow rate was found to have a significant impact on the overall thickness of the film. Visual observations and Raman spectroscopy measurements confirmed that two distinct hydrate layers formed during thickening, one which was more porous than the other. The capillary-like channels in the porous layer indicated a mechanism for mass transfer of water through the hydrate layer. A model was developed based on this observation, and it was fit to the thickening data in order to obtain mass transfer coefficients. Results of this study can be applied to CO2 hydrate formation in pipelines and near porous media used for CO2 capture.

Wadsworth, Lindsey [Colorado School of Mines, Gold↗