SEARCH · Engineering Papers

Results for “compute workflow”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 379 records · Page 21

All sources NASA NTRS DOE OSTI

DOE OSTI2021

Position Papers for the ASCR Workshop on Cybersecurity and Privacy for Scientific Computing Ecosystems

At the request of the Department of Energy's (DOE) Office of Advanced Scientific Computing Research (ASCR), this program committee has been tasked with organizing a workshop to identify basic research needs in cybersecurity and privacy to better support DOE's science and energy mission. As part of the process, the program committee is soliciting community input in the form of position papers to help identify significant use cases, facility issues, and other barriers to enabling verifiably trustworthy computational science while preserving data confidentiality as appropriate for scientific workflows of interest to DOE. The program committee will review these position papers and based on the fit of their area of expertise and interest, selected contributors will have the opportunity to participate in the workshop currently planned as a virtual event November 3-5th, 2021. The thrust areas that will be explored by this workshop are the following: (1) Algorithms for secure, scalable, privacy-enhancing technologies and frameworks, including: Federated AI/ML, Differential privacy, Randomized algorithms, Adversarial modeling & simulation, Graph algorithms, and Formal methods; (2) Platforms to support the entire scientific-computing ecosystem, including edge computing for large-scale experiments, focusing on heterogeneous systems and distributed systems, including: Heterogeneous computing systems, Distributed computing systems, and Secure data architectures; and (3) Data workflows to allow agile use of data while preserving integrity and privacy, making the important properties verifiable either at runtime or post-computation, including: Integrity and provenance and Data management infrastructure. Topics that are out-of-scope for the workshop include discussing specific proposed solutions or areas that are clearly out of DOE's fundamental and applied-sciences mission scope, e.g., cryptography, enterprise security, and general-operations technology.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

High Energy Physics Network Requirements Review: Final Report, July 2024–December 2024

The world-class research infrastructure at the US Department of Energy (DOE) Office of Science (SC) provides the research community with premier observational, experimental, computational, and network capabilities. Each user facility is designed to provide unique capabilities to advance the core DOE mission in science and technology for its SC program to stimulate rich scientific discoveries and enhance its innovation ecosystem. Research communities gather and flourish around each user facility, bringing together new and enhanced perspectives. The continual reinvention of the practice of science — as users and staff forge novel approaches expressed in research workflows — unlocks new discoveries and propels scientific progress. Within this research ecosystem, the high-performance computing (HPC) and networking user facilities stewarded by the SC’s Advanced Scientific Computing Research (ASCR) program play a dynamic cross-cutting role, enabling complex workflows demanding high-performance data, networking, and computing solutions. The ASCR facilities enterprise seeks to understand and meet the needs and requirements across SC and DOE domain science programs and priority efforts, highlighted by the formal requirements review methodology. Between July and December 2024, the Energy Sciences Network (ESnet) and the Office of High Energy Physics (HEP) of the DOE-SC organized an ESnet requirements review of HEP-supported program activities. Preparation for these events included identification of key stakeholders: program and facility management, research groups, and technology providers. Each stakeholder group was asked to prepare formal case study documents about its relationship to the HEP program to build a complete understanding of the current, near-term, and long-term status, expectations, and processes that will support the science going forward.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2025

SWARM: Reimagining scientific workflow management systems in a distributed world

Modern scientific workflows process massive amounts of data from diverse instruments and sensors, leveraging geographically distributed, heterogeneous compute and storage resources—from leadership-class systems to edge devices—connected by high-performance networks. The diversity of resources introduces challenges in harnessing their full potential, with resilience issues arising across applications, system software, networks, storage, and hardware. Today, workflow management systems (WMS) coordinate the execution of computation and data management tasks across target resources. However, WMS’s centralized nature makes them vulnerable to faults and scalability issues that may result in failures of entire computational campaigns. In conclusion, this paper introduces a novel agentic framework for workflow management, fully distributing and decentralizing the WMS functions and modeling them as swarm intelligence agents infused with advanced artificial intelligence solutions and traditional distributed computing algorithms that can make coordinated decisions in the presence of failures of the underlying cyberinfrastructure.

Swarm intelligence↗

DOE OSTI2025

Bridging paradigms: Designing for HPC-Quantum convergence

Here, this paper presents a comprehensive software stack architecture for integrating quantum computing (QC) capabilities with High-Performance Computing (HPC) environments. While quantum computers show promise as specialized accelerators for scientific computing, their effective integration with classical HPC systems presents significant technical challenges. We propose a hardware-agnostic software framework that supports both current noisy intermediate-scale quantum devices and future fault-tolerant quantum computers, while maintaining compatibility with existing HPC workflows. The architecture includes a quantum gateway interface, standardized APIs for resource management, and robust scheduling mechanisms to handle both simultaneous and interleaved quantum–classical workloads. Key innovations include: (1) a unified resource management system that efficiently coordinates quantum and classical resources, (2) a flexible quantum programming interface that abstracts hardware-specific details, (3) A Quantum Platform Manager API that simplifies the integration of various quantum hardware systems, and (4) a comprehensive tool chain for quantum circuit optimization and execution. We demonstrate our architecture through implementation of quantum–classical algorithms, including the variational quantum linear solver, showcasing the framework’s ability to handle complex hybrid workflows while maximizing resource utilization. This work provides a foundational blueprint for integrating QC capabilities into existing HPC infrastructures, addressing critical challenges in resource management, job scheduling, and efficient data movement between classical and quantum resources.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

Performance Characteristics of the BlueField-2 SmartNIC

High-performance computing (HPC) researchers have long envisioned scenarios where application workflows could be improved through the use of programmable processing elements embedded in the network fabric. Recently, vendors have introduced programmable Smart Network Interface Cards (SmartNICs) that enable computations to be offloaded to the edge of the network. There is great interest in both the HPC and high-performance data analytics (HPDA) communities in understanding the roles these devices may play in the data paths of upcoming systems. This paper focuses on characterizing both the networking and computing aspects of NVIDIA’s new BlueField-2 SmartNIC when used in a 100Gb/s Ethernet environment. For the networking evaluation we conducted multiple transfer experiments between processors located at the host, the SmartNIC, and a remote host. These tests illuminate how much effort is required to saturate the network and help estimate the processing headroom available on the SmartNIC during transfers. For the computing evaluation we used the stress-ng benchmark to compare the BlueField-2 to other servers and place realistic bounds on the types of offload operations that are appropriate for the hardware. Our findings from this work indicate that while the BlueField-2 provides a flexible means of processing data at the network’s edge, great care must be taken to not overwhelm the hardware. While the host can easily saturate the network link, the SmartNIC’s embedded processors may not have enough computing resources to sustain more than half the expected bandwidth when using kernel-space packet processing. From a computational perspective, encryption operations, memory operations under contention, and on-card IPC operations on the SmartNIC perform significantly better than the general-purpose servers used for comparisons in our experiments. Therefore, applications that mainly focus on these operations may be good candidates for offloading to the SmartNIC.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

Modeling supercritical CO2 injection induced rupture of a minor fault embedded in a poroelastic layered reservoir-caprock system

CO2 injection for geologic carbon sequestration involves hydromechanical processes that lead to changes in fluid pressure and stresses that can activate existing faults. This paper presents a new method and workflow of modeling fault activation considering more complex three-dimensional geometry of natural faults using the TOUGH-FLAC multiphase fluid flow and geomechanical simulator. In this method and workflow, FLAC3D mechanical interfaces and TOUGH3 finite volume elements are discretized using computer aided design and gridding software along with a tailored mesh translation routine. The method and workflow are demonstrated with a model of a curved minor fault embedded in a poro-elastic layered reservoir-caprock system. The model is used for a comprehensive sensitivity analysis of fault responses to fault length, injection mass rate, injection schedule, well-fault distance, and well locations versus fault location. Four metrics (CO2 plume, shear state of fault, pressure and stress path at fault monitoring points) are selected to assess CO2 migration, pressure change, and the reactivation of faults. The results reveal that CO2 can bypass around the tip of the minor impermeable fault, building up pressure and poro-elastic stress on both sides that tends to impede fault rupture. Our study shows the benefit of carefully designing the injection to achieve the targeted final storage volume, starting at a relatively low rate for considerable time, and then ramping up the injection rate to the full rate of injection. The initial low injection has two distinct benefits: (1) it allows for the formation of an extensive CO2 plume with a much higher mobility through a low viscosity that will result in a lower pressure for a given injection rate, and (2) it allows for gradual build-up of horizontal poro-elastic stress within the reservoir that will tend to impede activation of steeply dipping faults. The injection scenario starting at a low injection rate, denoted here as conservative injection, can significantly reduce the risk of fault activation as high fluid mobility and reservoir strengthening poro-elastic stress has been established long before reaching the peak injection rates. Moreover, simultaneous injection in two injection wells on both sides of fault can provide further reservoir strengthening through poro-elastic stress buildup acting on a fault under normal faulting stress regime. The findings presented in the paper can provide practical and effective guidance on long-term, safe, and reliable geological CO2 storage.

Cao, Meng↗

DOE OSTI2025

OLCF’s Advanced Computing Ecosystem (ACE): FY25 Update for Ongoing Efforts

The advent of widespread use of artificial intelligence (AI) and machine learning (ML) models in science, coupled with fast data production rates of scientific instruments strain the traditional batch-oriented high-performance computing (HPC) environment. As scientific exploration continues to require more data and faster processing and analysis, new emerging technologies and capabilities to enable cross-facility and time-sensitive workflows are required for seamless integration of HPC and experimental facilities. The Advanced Computing Ecosystem (ACE) is a strategic initiative within the Oak Ridge Leadership Computing Facility (OLCF) established in 2024 to support the development of cutting-edge technologies to advance computational research and infrastructure at OLCF and across the Department of Energy (DOE). Several DOE initiatives are spearheading the evolution of the scientific landscape by blurring facility boundaries and connecting the user facilities to advance scientific capabilities and ensure energy dominance. The DOE Integrated Research Infrastructure (IRI) program is one example that is laying a foundation to support complex cross-facility workflows. The IRI program aims to integrate diverse computational resources, data infrastructures, and scientific instruments to facilitate collaboration and accelerate scientific discovery. The Interconnected Science Ecosystem (INTERSECT) initiative at Oak Ridge National Laboratory (ORNL) is another example that aims to revolutionize scientific research through AI-driven, interconnected autonomous laboratories and research facilities. Finally, the American Science Cloud (AmSC), recently announced in the “One Big Beautiful Bill”, aims to leverage prior infrastructure efforts of the IRI and automation and AI efforts of INTERSECT (and others) to build a federated, AI-augmented AmSC platform to unify the DOE’s computing, experimental, and data resources to catalyze scientific innovation.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

Optimizing Management of Persistent Data Structures in High-Performance Analytics

Large-scale data analytics workflows ingest massive input data into various data structures, including graphs and key-value datastores. These data structures undergo multiple transformations and computations and are typically reused in incremental and iterative analytics workflows. Persisting in-memory views of these data structures enables reusing them beyond the scope of a single program run while avoiding repetitive raw data ingestion overheads. Memory-mapped I/O enables persisting in-memory data structures without data serialization and deserialization overheads. However, memory-mapped I/O lacks the key feature of persisting consistent snapshots of these data structures for incremental ingestion and processing. The obstacles to efficient virtual memory snapshots using memory-mapped I/O include background writebacks outside the application’s control, and the significantly high storage footprint of such snapshots. To address these limitations, we present Privateer, a memory and storage management tool that enables storage-efficient virtual memory snapshotting while also optimizing snapshot I/O performance. Here, we integrated Privateer into Metall, a state-of-the-art persistent memory allocator for C++, and the Lightning Memory-Mapped Database (LMDB), a widely-used key-value datastore in data analytics and machine learning. Privateer optimized application performance by 1.22× when storing data structure snapshots to node-local storage, and up to 16.7× when storing snapshots to a parallel file system. Privateer also optimizes storage efficiency of incremental data structure snapshots by up to 11× using data deduplication and compression.

Computer science↗

DOE OSTI2024

Physics-based reward driven image analysis in microscopy

The rise of electron microscopy has expanded our ability to acquire nanometer and atomically resolved images of complex materials. The resulting vast datasets are typically analyzed by human operators, an intrinsically challenging process due to the multiple possible analysis steps and the corresponding need to build and optimize complex analysis workflows. We present a methodology based on the concept of a Reward Function coupled with Bayesian Optimization, to optimize image analysis workflows dynamically. The Reward Function is engineered to closely align with the experimental objectives and broader context and is quantifiable upon completion of the analysis. Here, cross-section, high-angle annular dark field (HAADF) images of ion-irradiated (Y, Dy)Ba 2 Cu 3 O 7–δ thin-films were used as a model system. The reward functions were formed based on the expected materials density and atomic spacings and used to drive multi-objective optimization of the classical Laplacian-of-Gaussian (LoG) method. These results can be benchmarked against the DCNN segmentation. This optimized LoG* compares favorably against DCNN in the presence of the additional noise. We further extend the reward function approach towards the identification of partially-disordered regions, creating a physics-driven reward function and action space of high-dimensional clustering. We pose that with correct definition, the reward function approach allows real-time optimization of complex analysis workflows at much higher speeds and lower computational costs than classical DCNN-based inference, ensuring the attainment of results that are both precise and aligned with the human-defined objectives.

47 OTHER INSTRUMENTATION↗

DOE OSTI2025

From Edge to HPC: Investigating Cross-Facility Data Streaming Architectures

In this paper, we investigate three cross-facility data streaming architectures, Direct Streaming (DTS), Proxied Streaming (PRS), and Managed Service Streaming (MSS). We examine their architectural variations in data flow paths and deployment feasibility, and detail their implementation using the Data Streaming to HPC (DS2HPC) architectural framework and the SciStream memory-to-memory streaming toolkit on the production-grade Advanced Computing Ecosystem (ACE) infrastructure at Oak Ridge Leadership Computing Facility (OLCF). We present a workflow-specific evaluation of these architectures using three synthetic workloads derived from the streaming characteristics of scientific workflows. Through simulated experiments, we measure streaming throughput, round-trip time, and overhead under work sharing, work sharing with feedback, and broadcast and gather messaging patterns commonly found in AI-HPC communication motifs. Our study shows that DTS offers a minimal-hop path, resulting in higher throughput and lower latency, whereas MSS provides greater deployment feasibility and scalability across multiple users but incurs significant overhead. PRS lies in between, offering a scalable architecture whose performance matches DTS in most cases.

George, Anjus [ORNL] (ORCID:0000000179737061)↗

DOE OSTI2026

Integrating DOE ASCR Computing into HEPCloud through GlideinWMS

Fermilab's HEPCloud facility expands the laboratory's computing capacity by provisioning resources beyond the local grid, using GlideinWMS to deliver pilots to where experiments such as CMS and DUNE run. The High-Performance Computing (HPC) facilities of the DOE Office of Advanced Scientific Computing Research (ASCR) are a growing part of that pool. HEPCloud currently provisions NERSC over SSH, but NERSC is moving away from that path as it adopts multi-factor authentication and directs automated access to its Superfacility API and the DOE Integrated Research Infrastructure (IRI) APIs. Maintaining and extending access across the ASCR ecosystem now requires provisioning through these interfaces. This work adds new pilot submission paths to GlideinWMS for the NERSC Superfacility API, IRI, and Globus Compute. Each uses the provisioning model GlideinWMS already applies to batch resources, so experiments can run on ASCR computing resources without changes to their existing workflows. This work finally presents a comparison of the paths to guide which interfaces are best suited for different workflows.

Majumder, Meghanto [U. Houston (main)]↗

DOE OSTI2026

Accelerating lattice gauge theory studies with Agentic AI

Lattice gauge theory research, with its computationally intensive simulations and complex multi‑stage workflows, is well positioned to benefit from agentic AI systems. We demonstrate how such tools can support key components of lattice gauge theory research, including novel simulation code development using standard LQCD frameworks, HPC job orchestration, simulation data analysis, and expert‑guided tuning of algorithmic parameters such as Hasenbusch mass preconditioning and multigrid solvers. Our results show that agentic AI can reduce manual effort, improve productivity, and accelerate the research cycle while maintaining essential human oversight.

Ayyar, Venkitesh [Fermilab]↗

NASA NTRS2020

Sketch-to-Solution: A Case Study in RCS Aerodynamic Interaction

Thanks to recent advances in the fields of anisotropic grid adaptation, error estimation, and geometry modeling, a sketch-to-solution work flow is now possible for viscous computational fluid dynamic (CFD) simulations. With this workflow, a CFD application engineer provides geometry, boundary conditions, and flow parameters; and the sketch-to-solution process yields a CFD simulation through automatic, error-based, grid adaptation. To explore the benefits of this nascent capability, a conventional manual grid generation work flow is compared to this new automatic grid generation work flow for a given engineering question: What are the aerodynamic interactions caused by the reaction control system (RCS) on an entry vehicle? This case study indicates that while the automatic grid generation sketch-to- solution process is not yet mature, it is preferred over a manual grid generation work flow because it greatly reduces manual labor, eliminates many opportunities for human error, and provides grid sensitivity information.

Bill Kleb↗

NASA NTRSDate not supplied

Modeling and Simulation Efforts to Support Improved Comfort in ARGOS

BACKGROUND: The Active Response Gravity Offload System (ARGOS) provides an analog environment for extravehicular activity (EVA) testing and training. Discomfort has been observed during longer suited test sessions. While the subject’s core is offloaded during surface EVA evaluations, his/her arms experience full Earth gravity and can become overly fatigued, especially during suited tests which involve reaching and prolonged arm extensions. A device (ARGOS Negation of Gravitational Effects on the Limbs: ANGEL) to offload the weight of the arms and suit sleeves is being developed by JSC’s Flight Systems Branch, and here we present preliminary modeling of that device using the open-source biomechanical tool OpenSim [1,2] with an in-house developed plugin. We analyze a series of motions performed by a single shirt-sleeved subject with goals of characterizing the device, validating the model, and predicting whether reduced gravity conditions (i.e., lunar gravity (Lg) or Martian gravity (Mg)) can be accurately simulated with the device, as well as providing comfort to the ARGOS user. METHODS AND RESULTS: To model the offload device, we augment the OpenSim human model topology with the offload mechanism components and joints, using CAD models to represent the mechanism graphically. The joint angles of the device are either obtained from (1) inverse kinematics (IK) using motion capture markers on the various components of the device or (2) calculated in the OpenSim plugin by modeling how the components configure themselves under the offloading spring tension given a particular IK-derived arm position. Given the joint angles of the device, the resulting force on the arm is computed by the plugin and applied as an external load in inverse dynamics (ID) in order to enable study of overall shoulder joint torques as well as offload achieved. We verify the calculated joint angles by using the inverse kinematic data and the forces from manual measurements of the spring both independently and integrated within the device. We found that calculated joint angles generally represent the angles measured and computed with IK, supporting a possible analysis workflow inputting human motion data and observing system behavior under varied design parameters. In two different device configurations in which the maximum applied force was 131 N, our current model accurately captured force with a difference of 2-3 N from measured loads. Though our initial test was performed with a shirt-sleeve subject, arm weights were added to emulate the weight of the suit sleeve and the subject was positioned in a test stand with a Mark-III Hard Upper Torso (HUT) and Portable Life Support System (PLSS) mockup. Arm range of motion tasks were performed outside of the HUT, inside the HUT, and inside the HUT while using the device. A variety of other upper body tasks were completed as well. In summary, we have developed a model to investigate and verify an upper limb offload device currently in development. We believe this model will be a valuable tool not only for device characterization but also to predict proper configurations to simulate Lg or Mg conditions, investigate range of motion concerns, predict limitations such as internal collisions and contacts, and inform future design improvements.

L B Nilsson↗

DOE OSTI2024

Using scalable computer vision to automate high-throughput semiconductor characterization

Abstract High-throughput materials synthesis methods, crucial for discovering novel functional materials, face a bottleneck in property characterization. These high-throughput synthesis tools produce 10 4 samples per hour using ink-based deposition while most characterization methods are either slow (conventional rates of 10 1 samples per hour) or rigid (e.g., designed for standard thin films), resulting in a bottleneck. To address this, we propose automated characterization (autocharacterization) tools that leverage adaptive computer vision for an 85x faster throughput compared to non-automated workflows. Our tools include a generalizable composition mapping tool and two scalable autocharacterization algorithms that: (1) autonomously compute the band gaps of 200 compositions in 6 minutes, and (2) autonomously compute the environmental stability of 200 compositions in 20 minutes, achieving 98.5% and 96.9% accuracy, respectively, when benchmarked against domain expert manual evaluation. These tools, demonstrated on the formamidinium (FA) and methylammonium (MA) mixed-cation perovskite system FA 1−x MA x PbI 3 , 0 ≤ x ≤ 1, significantly accelerate the characterization process, synchronizing it closer to the rate of high-throughput synthesis.

Science & Technology - Other Topics↗

DOE OSTI2024

Operational experience and R&D results using the Google Cloud for High-Energy Physics in the ATLAS experiment

The ATLAS experiment at CERN relies on a Worldwide Distributed Computing Grid infrastructure to support its physics program at the Large Hadron Collider. ATLAS has integrated cloud computing resources to complement its Grid infrastructure and conducted an R&D program on Google Cloud Platform. These initiatives leverage key features of commercial cloud providers: lightweight configuration and operation, elasticity and availability of diverse infrastructures. Here this paper examines the seamless integration of cloud computing services as a conventional Grid site within the ATLAS workflow management and data management systems, while also offering new setups for interactive, parallel analysis. It underscores pivotal results that enhance the on-site computing model and outlines several R&D projects that have benefited from large-scale, elastic resource provisioning models. Furthermore, this study discusses the impact of cloud-enabled R&D projects in three domains: accelerators and AI/ML, ARM CPUs and columnar data analysis techniques.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2023

Deep learning at the edge enables real-time streaming ptychographic imaging

Abstract Coherent imaging techniques provide an unparalleled multi-scale view of materials across scientific and technological fields, from structural materials to quantum devices, from integrated circuits to biological cells. Driven by the construction of brighter sources and high-rate detectors, coherent imaging methods like ptychography are poised to revolutionize nanoscale materials characterization. However, these advancements are accompanied by significant increase in data and compute needs, which precludes real-time imaging, feedback and decision-making capabilities with conventional approaches. Here, we demonstrate a workflow that leverages artificial intelligence at the edge and high-performance computing to enable real-time inversion on X-ray ptychography data streamed directly from a detector at up to 2 kHz. The proposed AI-enabled workflow eliminates the oversampling constraints, allowing low-dose imaging using orders of magnitude less data than required by traditional methods.

36 MATERIALS SCIENCE↗

DOE OSTI2021

Extending C++ for Heterogeneous Quantum-Classical Computing

In this report we present qcor - a language extension to C++ and compiler implementation that enables heterogeneous quantum-classical programming, compilation, and execution in a single-source context. Our work provides a first-of-its-kind C++ compiler enabling high-level quantum kernel (function) expression in a quantum-language agnostic manner, as well as a hardware-agnostic, retargetable compiler workflow targeting a number of physical and virtual quantum computing backends. qcor leverages novel Clang plugin interfaces and builds upon the XACC system-level quantum programming framework to provide a state-of-the-art integration mechanism for quantum-classical compilation that leverages the best from the community at-large. qcor translates quantum kernels ultimately to the XACC intermediate representation, and provides user-extensible hooks for quantum compilation routines like circuit optimization, analysis, and placement. This work details the overall architecture and compiler workflow for qcor, and provides a number of illuminating programming examples demonstrating its utility for near-term variational tasks, quantum algorithm expression, and feed-forward error correction schemes.

97 MATHEMATICS AND COMPUTING↗