SEARCH · Engineering Papers

Results for “hybrid workflows”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

All sources NASA NTRS DOE OSTI

DOE OSTI2025

mzPeak: Designing a Scalable, Interoperable, and Future-Ready Mass Spectrometry Data Format

Advances in mass spectrometry (MS) instrumentation, such as higher resolution, faster scan speeds, and improved sensitivity, have significantly increased the volume and complexity of data. The growing adoption of imaging and ion mobility further amplifies these challenges across MS-based omics fields, including proteomics, metabolomics, and lipidomics. While these technologies unlock new possibilities, they also present significant challenges in data management, storage, and accessibility. Existing open formats, such as the XML-based community standards mzML and imzML, struggle to meet the demands of modern MS workflows due to their large file sizes, slow data access, and limited metadata support. Vendor-specific formats, while optimized for proprietary instruments, lack interoperability, comprehensive metadata support and long-term archival reliability. This white paper lays the groundwork for mzPeak, a next-generation community data format designed to address these challenges and support high-throughput, multi-dimensional MS workflows. By adopting a hybrid model that combines efficient binary storage for numerical data and both human and machine-readable metadata storage, mzPeak will reduce file sizes, accelerate data access, and offer a scalable, adaptable solution for evolving MS technologies. For researchers, mzPeak will enable enhanced interoperability across platforms, seamless support for complex workflows including ion mobility and MS imaging, and faster data access compared to existing community formats such as mzML. Its design will ensure data is managed in compliance with regulatory standards, essential for applications such as precision medicine and chemical safety, where long-term data integrity and accessibility are critical. For vendors, mzPeak provides a streamlined, open alternative to proprietary formats, reducing the burden of regulatory compliance while aligning with the industry's push for transparency and standardization. By offering a high-performance, interoperable solution, mzPeak positions vendors to meet customer demands for sustainable data management tools which will be able to handle emerging and future data types and workflows. mzPeak aspires to become the cornerstone of MS data management, empowering researchers, vendors, and developers to innovate and collaborate more effectively.

data formats↗

DOE OSTI2021

A MLIR Dialect for Quantum Assembly Languages

We demonstrate the utility of the Multi-Level Intermediate Representation (MLIR) for quantum computing. Specifically, we extend MLIR with a new quantum dialect that enables the expression and compilation of common quantum assembly languages. The true utility of this dialect is in its ability to be lowered to the LLVM intermediate representation (IR) in a manner that is adherent to the quantum intermediate representation (QIR) specification recently proposed by Microsoft. We leverage a qcor-enabled implementation of the QIR quantum runtime API to enable a retargetable (quantum hardware agnostic) compiler workflow mapping quantum languages to hybrid quantum-classical binary executables and object code. We evaluate and demonstrate this novel compiler workflow with quantum programs written in OpenQASM 2.0. We provide concrete examples detailing the generation of MLIR from OpenQASM source files, the lowering process from MLIR to LLVM IR, and ultimately the generation of executable binaries targeting available quantum processors.

Mccaskey, Alex↗

DOE OSTI2022

Joint physics-based and data-driven time-lapse seismic inversion: Mitigating data scarcity

In carbon capture and sequestration (CCS), developing rapid and effective imaging techniques is crucial for real-time monitoring of the spatial and temporal dynamics of CO 2 propagation during/after injection. With continuing improvements in computational power and data storage, data-driven techniques based on machine learning (ML) have been effectively applied to seismic inverse problems. In particular, ML helps alleviate the ill-posedness and high computational cost of full-waveform inversion (FWI). However, such data-driven inversion techniques require massive high-quality training data sets to ensure prediction accuracy, which hinders their application to time-lapse monitoring of CO 2 sequestration. We propose an efficient “hybrid” time-lapse workflow that combines physics-based FWI and data-driven ML inversion. The scarcity of the available training data is addressed by developing a new data-generation technique with physics constraints. The method is vali dated on a synthetic CO 2 -sequestration model based on the Kimberlina storage reservoir in California. The proposed approach is shown to synthesize a large volume of high-quality, physically realistic training data, which is critically important in accurately characterizing the CO 2 movement in the reservoir. In conclusion, the developed hybrid methodology can also simultaneously predict the variations in velocity and saturation and achieve high spatial resolution in the presence of realistic noise in the data.

58 GEOSCIENCES↗

DOE OSTI2020

A scoping review of non-destructive testing (NDT) techniques in building performance diagnostic inspections

Understanding building envelope thermodynamics is an essential foundation of building sciences, mainly due to the envelope’s role as a boundary layer for exterior environments, as well as a container and regulator of internal microclimates. This paper presents a scoping literature review of select Non-destructive Testing (NDT) techniques for building envelope scanning and surveying for thermodynamic diagnostics. The investigation focuses specifically on reviewing six NDT techniques: Ground Penetrating Radar (GPR), Light Detection and Ranging (LiDAR)/Laser Scanning, Thermography, Ultrasound, Close-Range Photogrammetry and Through Wall Imaging Radar (TWIR). The aim is to identify knowledge gaps in terms of their use in accurately characterizing envelope compositions for further integration in Building Energy Modeling (BEM). Each technique was evaluated according to set categories imbibed from the American Society of Heating, Refrigerating, and Air Conditioning Engineering (ASHRAE) Standard 211P that showcase the technique’s ability to extract various relevant information. A framework is then developed to inform users on how to use hybrid NDT-based workflows applied in building envelope energy audits. The study concludes by discussing possibilities of utilizing NDT in large-scale audit automation, BEM integration, and developing built environment policies focusing on increasing existing building performance through retrofitting design.

36 MATERIALS SCIENCE↗

DOE OSTI2025

Integrating and Characterizing HPC Task Runtime Systems for hybrid AI-HPC workloads

Scientific workflows increasingly involve both HPC and machine-learning tasks, combining MPI-based simulations, training, and inference in a single execution. Launchers such as Slurm’s srun constrain concurrency and throughput, making them unsuitable for dynamic and heterogeneous workloads. We present a performance study of RADICAL-Pilot (RP) integrated with Flux and Dragon, two complementary runtime systems that enable hierarchical resource management and high-throughput function execution. Using synthetic and production-scale workloads on Frontier, we characterize the task execution properties of RP across runtime configurations. RP+Flux sustains up to 930 tasks/s, and RP+Flux+Dragon exceeds 1,500 tasks/s with over 99.6% utilization. In contrast, srun peaks at 152 tasks/s and degrades with scale, with utilization below 50%. For IMPECCABLE.v2 drug discovery campaign, RP+Flux reduces makespan by 30–60% relative to srun/Slurm and increases throughput more than four times on up to 1,024. These results demonstrate hybrid runtime integration in RP as a scalable approach for hybrid AI-HPC workloads.

HPC-AI↗

DOE OSTI2020

CoREC: Scalable and Resilient In-memory Data Staging for In-situ Workflows

The dramatic increase in the scale of current and planned high-end HPC systems is leading new challenges, such as the growing costs of data movement and IO, and the reduced mean time between failures (MTBF) of system components. In-situ workflows, i.e., executing the entire application workflows on the HPC system, have emerged as an attractive approach to address data-related challenges by moving computations closer to the data, and staging-based frameworks have been effectively used to support in-situ workflows at scale. However, the resilience of these staging-based solutions has not been addressed, and they remain susceptible to expensive data failures. Furthermore, naive use of data resilience techniques such as n-way replication and erasure codes can impact latency and/or result in significant storage overheads. In this article, we present CoREC, a scalable and resilient in-memory data staging runtime for large-scale in-situ workflows. CoREC uses a novel hybrid approach that combines dynamic replication with erasure coding based on data access patterns. It also leverages multiple levels of replications and erasure coding to support diverse data resiliency requirements. Furthermore, the article presents optimizations for load balancing and conflict-avoiding encoding, and a low overhead, lazy data recovery scheme. We have implemented the CoREC runtime and have deployed with the DataSpaces staging service on leadership class computing machines and present an experimental evaluation in the article. Here, the experiments demonstrate that CoREC can tolerate in-memory data failures while maintaining low latency and sustaining high overall storage efficiency at large scales.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

A methodology for decay heat characterization in molten salt reactors

Accurate decay heat prediction in molten salt reactors (MSRs) faces dual challenges: complex operational uncertainties and the need for interpretable models compatible with engineering workflows. This work presents a hybrid machine learning and segmented polynomial methodology that addresses both requirements through three key innovations. First, a modular data architecture encodes MSR-specific operational parameters (power density: 1-100 W cm -3 , humidity: 0-0.1 wt %, air ingress: 0-0.1 mol %) with uncertainty-aware temporal discretization spanning 15 orders of magnitude. Second, region-optimized machine learning models achieve 92.3 % root mean square error (RMSE) reduction over conventional polynomials while maintaining physical interpretability through automated piecewise equation generation. Third, dual front-end interfaces accelerate safety analyses — a Jupyter environment enables researchers to explore 10,000+ parameter combinations via interactive widgets, while a Streamlit web application reduces design iteration cycles through production-grade visualization tools. Operational deployment demonstrates prediction times of only a couple hundred milliseconds for 10 4 years decay profiles, enabling real-time optimization of spent fuel container designs.

42 - ENGINEERING↗

DOE OSTI2023

Discovering Dinuclear Dioxygen-Bridged Cobalt(III) Complexes for Selective Binding of O 2 from Air

The design and development of dioxygen activation in porous crystalline materials is a useful avenue for exploring selective adsorption of O 2 that shows significant potential to enable separations of O 2 and N 2 from air. Porous materials featuring redox-active metal centers have received attention regarding selective O 2 adsorption via chemisorption. Drawing inspiration from a dinuclear cobalt material ([(Co(III) 2 (bpbp)O 2 ) 2 bdc](PF 6 ) 4 (CSD code: GAMVIB; bpbp – = 2,6-bis(N,N-bis(2-pyridylmethyl)aminomethyl)-4-tert-butylphenolato; bdc 2– = 1,4-benzenedicarboxylato)) that displays reversible and selective O 2 adsorption, we focus on searching for potential O 2 -selective materials with dinuclear cobalt clusters that have dioxygen-bridged Co(III) complexes. We combine structure screening with a high-level hybrid periodic density functional theory (DFT) workflow to investigate O 2 and N 2 adsorption in materials from validated crystal structure databases (e.g., the CSD database). These calculations identify multiple materials that are predicted to have superior O 2 binding capability relative to GAMVIB. Grand Canonical Monte Carlo (GCMC) simulations based on DFT-developed force fields were performed for selected candidates to estimate the adsorption performance for O 2 /N 2 mixtures.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2026

Interpretable Models for Workflow Differentiation in High-Performance Scientific Networks

Scientific workflows in high-performance networks spawn hundreds of interdependent flows that must be managed collectively—yet existing network classifiers treat each flow in isolation, leading to fragmented QoS decisions and missed interflow patterns. We present a novel traffic classification solution that operates at the workflow level, distinguishing entire filetransfer operations from streaming analytics by capturing how concurrent flows interact and burst together. We introduce a workflow identification window (WIW) that ingests raw packet headers from parallel flows into unified tensors, preserving the spatial-temporal patterns that differentiate scientific workflows. This approach achieves 98.7% accuracy using CNN, LSTM, and hybrid architectures, while maintaining 84% accuracy on production traffic collected a week later—demonstrating robustness to temporal drift. By integrating SHAP and GradCAM explainability, we reveal that early-packet timing patterns and cross-flow correlations drive classification decisions, providing operators with interpretable insights. Our system enables coherent workflow-level QoS enforcement and dynamic bandwidth allocation in scientific networks, eliminating manual per-flow configuration while maintaining classification latency at millisecond level.

Giannakou, Anna [LBL, Berkeley]↗

DOE OSTI2025

An Efficient Storage-Driven Machine Learning Model for Performance in the Era of Multimodal Scientific Data

Scientific workflows are increasingly relying on machine learning (ML), simulation, and hybrid techniques to predict, understand, and optimize the behavior of complex experiments. High-performance computing has greatly improved researchers’ ability to acquire diverse data modalities in these workflows. Recent studies suggest that the performance of machine learning models can be improved by integrating data from various sources. Unfortunately, these workloads pose unprecedent pressure on the network storage to meet the demands associated with accessing these multimodal data. To mitigate the impact of intensive IO, we propose a solution that utilizes a multi-tier High-Performance Computing (HPC) distributed storage and data processing framework, placing computation where the data resides for better performance. By adopting this project, the scientific community will gain new opportunities to explore multimodal storage-driven possibilities, integrating multiple scientific data sources with advanced streaming frameworks. Additionally, our framework effectively utilizes computing resources and bridges the gaps identified by HPC experts. Our proposed approach tackles scalability and persistence challenges by leveraging native persistency, which has posed difficulties in traditional approaches. Furthermore, we seek to enhance fault-tolerance and load-balance of computations by leveraging real-time streaming in diverse scientific computing environments, thereby propelling advanced scientific computing research into the next generation.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

ClimSim: A large multi-scale dataset for hybrid physics-ML climate emulation

Modern climate projections lack adequate spatial and temporal resolution due to computational constraints. A consequence is inaccurate and imprecise predictions of critical processes such as storms. Hybrid methods that combine physics with machine learning (ML) have introduced a new generation of higher fidelity climate simulators that can sidestep Moore’s Law by outsourcing compute-hungry, short, high-resolution simulations to ML emulators. However, this hybrid ML-physics simulation approach requires domain-specific treatment and has been inaccessible to MLexperts because of lack of training data and relevant, easy-to-use workflows. Wepresent ClimSim, the largest-ever dataset designed for hybrid ML-physics research. It comprises multi-scale climate simulations, developed by a consortium of climate scientists and ML researchers. It consists of 5.7 billion pairs of multivariate input and output vectors that isolate the influence of locally-nested, high-resolution, high-fidelity physics on a host climate simulator’s macro-scale physical state. The dataset is global in coverage, spans multiple years at high sampling frequency, and is designed such that resulting emulators are compatible with downstream coupling into operational climate simulators. We implement a range of deterministic and stochastic regression baselines to highlight the ML challenges and their scoring. The data (https://huggingface.co/datasets/LEAP/ClimSim_high-res2) and code(https://leap-stc.github.io/ClimSim)arereleasedopenlytosupport the development of hybrid ML-physics and high-fidelity climate simulations for the benefit of science and society.

artificial intelligence, machine learning↗

DOE OSTI2022

NREL Stratus - Enabling Workflows to Fuse Data Streams, Modeling, Simulation, and Machine Learning

Integrating cloud services into advanced computing facilities provides significant new capabilities over focusing solely on traditional high performance computing (HPC) workloads. This brings complementary capabilities as well as enabling new focused roles for HPC. They are especially potent for workflows that fuse data streams, modeling and simulation ('modsim') and machine learning. A key challenge to adopting a hybrid edge-cloud-HPC model is to align optimal capability, data, and user intent on the right resources for each step in a workflow.?The NREL Stratus service provides a basis for this: Stratus layers capabilities needed to make?cloud services accessible to a lab-based scientific community on commercial offerings, and; currently supports upwards of 200 projects ranging from IOT integration to traditional modeling and simulation. This provides a real-world inventory of scientific workflow elements. A growing knowledge base enables placing these elements appropriately between the edge, cloud, and traditional HPC. This paper outlines a vision via reference architecture and the application of that architecture in a typical workflow highlighting multiple components: sensor data intake, cleaning and transforming (edge/cloud suitable); generation of synthetic data through modsim, computationally heavy ML training and hyperparameter optimization (HPC suitable), and; inference and deployment (cloud ideal). Every step in such a workflow involves a cost-benefit analysis regarding the data movement, computational efficiency, availability, latency, and resource capabilities. The reference architecture and examples outlined allow for understanding new opportunities in the context of emerging workflows that combine IOT, cloud, and HPC to bolster scientific productivity.

AI↗

DOE OSTI2022

28 NREL Stratus - Enabling Workflows to Fuse Data Streams, Modeling, Simulation, and Machine Learning: Preprint

AI↗

DOE OSTI2025

Defining quantum-ready primitives for hybrid HPC-QC supercomputing: a case study in Hamiltonian simulation

As computational demands in scientific applications continue to rise, hybrid high-performance computing (HPC) systems integrating classical and quantum computers (HPC-QC) are emerging as a promising approach to tackling complex computational challenges. One critical area of application is Hamiltonian simulation, a fundamental task in quantum physics and other large-scale scientific domains. This paper investigates strategies for quantum-classical integration to enhance Hamiltonian simulation within hybrid supercomputing environments. By analyzing computational primitives in HPC allocations dedicated to these tasks, we identify key components in Hamiltonian simulation workflows that stand to benefit from quantum acceleration. To this end, we systematically break down the Hamiltonian simulation process into discrete computational phases, highlighting specific primitives that could be effectively offloaded to quantum processors for improved efficiency. Our empirical findings provide insights into system integration, potential offloading techniques, and the challenges of achieving seamless quantum-classical interoperability. We assess the feasibility of quantum-ready primitives within HPC workflows and discuss key barriers such as synchronization, data transfer latency, and algorithmic adaptability. These results contribute to the ongoing development of optimized hybrid solutions, advancing the role of quantum-enhanced computing in scientific research.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

The Case for Strong Scaling in Deep Learning: Training Large 3D CNNs with Hybrid Parallelism

Here, we present scalable hybrid-parallel algorithms for training large-scale 3D convolutional neural networks. Deep learning-based emerging scientific workflows often require model training with large, high-dimensional samples, which can make training much more costly and even infeasible due to excessive memory usage. We solve these challenges by extensively applying hybrid parallelism throughout the end-to-end training pipeline, including both computations and I/O. Our hybrid-parallel algorithm extends the standard data parallelism with spatial parallelism, which partitions a single sample in the spatial domain, realizing strong scaling beyond the mini-batch dimension with a larger aggregated memory capacity. We evaluate our proposed training algorithms with two challenging 3D CNNs, CosmoFlow and 3D U-Net. Our comprehensive performance studies show that good weak and strong scaling can be achieved for both networks using up to 2K GPUs. More importantly, we enable training of CosmoFlow with much larger samples than previously possible, realizing an order-of-magnitude improvement in prediction accuracy.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

ESMs Latent Space Exploration for Uncertainty Quantification and Spatiotemporal Downscaling

This final report for DOE Award DE-SC0023044 presents advances in two key areas of climate modeling: (1) representative climate model selection and (2) Earth System Model (ESM) downscaling using hybrid AI methods. The first section introduces a reordered, three-stage workflow to select representative GCM runs that more effectively balance historical skill with ensemble spread, validated across Texas, Bihar, and New York. The second section introduces two novel super-resolution frameworks, ViSIR and ViFOR, that integrate Vision Transformers with sinusoidal and Fourier-based implicit neural representations. These models achieve state-of-the-art reconstruction accuracy for ESM variables including surface temperature and heat fluxes. The report includes detailed methodology, benchmarks, and results, demonstrating significant gains in uncertainty quantification, spatial fidelity, and scalability for climate-impact studies.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2025

White Paper: Scalable Digital Twin Capabilities for Aging and Surveillance of Engineered Systems

This white paper presents a multi-year initiative to develop practical, secure, and scalable digital twin capabilities for engineered systems in aging and surveillance contexts—an approach pioneered at the National Nuclear Security Administration (NNSA) Lawrence Livermore National Laboratory (LLNL) that maps directly onto the needs and ambitions of the Navy for ship- and fleet-level digital twins. LLNL’s work in building part- and process-level digital twins for advanced manufacturing, with a vision to scale up to entire factory floors and, ultimately, enterprise-wide digital twins, offers an adaptable pathway for the Navy as it seeks to modernize lifecycle management, readiness, and predictive maintenance across ships and fleets. For our application, we integrate physics-based modeling with automated data ingestion, processing, and AI-driven calibration, creating hybrid models that are both interpretable and data responsive. We modernized legacy workflows, established centralized data infrastructure, automated experimental pipelines, and demonstrated end-to-end coupling of accelerated aging data with finite element simulations via optimization and surrogate modeling. The result is a generalizable framework that supports part-level digital twins today and lays the groundwork for future system-level twins suitable for Navy applications.

36 MATERIALS SCIENCE↗

DOE OSTI2024

HybridSystemsSimulations.jl - Solving the Merchant Collocated Facilities with JuMP

The development of new clean-generation technologies also leads to new plant-level architectures that combine several generation and storage assets behind the point of connection. These co-located generation resources (Hybrid Systems) primarily operate as merchant assets that employ automated market bidding models and internal Energy Management Systems (EMS) to comply with the operator's signals. Formulating an optimal bidding model requires embedding the EMS control model into the bidding algorithm, resulting in a bi-level optimization problem. In this presentation, we first showcase using JuMP to formulate and solve this problem effectively for multiple merchant systems and the bidding outcomes considering different model formulations. Second, the bidding outcomes are later integrated into a PowerSimulations.jl (also built with JuMP) simulation to study the system-level effects of the various merchant bidding and the interactions between market-clearing models and the embedded EMS model. We will showcase simulations conducted in the RTS system considering different levels of merchant hybrid systems participation. The presentation provides the following specific insights on JuMP usage: 1) the Formulation of specialized bi-level problems with custom cuts to solve the merchant hybrid system bidding problem; 2) the integration of a modular model within a complex simulation workflow supported by JuMP in PowerSimulations.jl; 3) Accelerating the solution of power systems operations simulation that employ agent optimization problems using JuMP.

energy markets↗