Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Workflow”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 253 records · Page 14
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Generalizable coordination of large multiscale workflows: challenges and learnings at scale

The advancement of machine learning techniques and the heterogeneous architectures of most current supercomputers are propelling the demand for large multiscale simulations that can automatically and autonomously couple diverse components and map them to relevant resources to solve complex problems at multiple scales. Nevertheless, despite the recent progress in workflow technologies, current capabilities are limited to coupling two scales. In the first-ever demonstration of using three scales of resolution, we present a scalable and generalizable framework that couples pairs of models using machine learning and in situ feedback. We expand upon the massively parallel Multiscale Machine-Learned Modeling Infrastructure (MuMMI), a recent, award-winning workflow, and generalize the framework beyond its original design. We discuss the challenges and learnings in executing a massive multiscale simulation campaign that utilized over 600,000 node hours on Summit and achieved more than 98% GPU occupancy for more than 83% of the time. We present innovations to enable several orders of magnitude scaling, including simultaneously coordinating 24,000 jobs, and managing several TBs of new data per day and over a billion files in total. Finally, we describe the generalizability of our framework and, with an upcoming open-source release, discuss how the presented framework may be used for new applications.

Bhatia, Harsh↗
DOE OSTI2023

Julia as a unifying end-to-end workflow language on the Frontier exascale system

We evaluate Julia as a single language and ecosystem paradigm powered by LLVM to develop workflow components for high-performance computing. We run a Gray-Scott, 2-variable diffusion-reaction application using a memory-bound, 7-point stencil kernel on Frontier, the US Department of Energy’s first exascale supercomputer. We evaluate the performance, scaling, and trade-offs of (i) the computational kernel on AMD’s MI250x GPUs, (ii) weak scaling up to 4,096 MPI processes/GPUs or 512 nodes, (iii) parallel I/O writes using the ADIOS2 library bindings, and (iv) Jupyter Notebooks for interactive analysis. Results suggest that although Julia generates a reasonable LLVM-IR, a nearly 50% performance difference exists vs. native AMD HIP stencil codes when running on the GPUs. As expected, we observed near-zero overhead when using MPI and parallel I/O bindings for system-wide installed implementations. Consequently, Julia emerges as a compelling high-performance and high-productivity workflow composition language, as measured on the fastest supercomputer in the world.

Godoy, William↗
DOE OSTI2024

Towards Cross-Facility Workflows Orchestration through Distributed Automation

Modern science relies on end-to-end workflows that incorporate experimental instruments and utilize edge, cloud, or high-performance computing and storage resources. These components are geographically dispersed across various user facilities and interconnected through high-speed networks. In this paper, we present Zambeze, an automated distributed framework designed to facilitate this new class of cross-facility workflows. Utilizing swarm intelligence principles, Zambeze orchestrates science campaigns by managing distributed autonomous agents. These agents can offer a suite of services, including computing, storage, and data management. We demonstrate the feasibility of Zambeze through a real-world application involving electron microscopy, enhanced with Artificial Intelligence capabilities.

Skluzacek, Tyler↗
DOE OSTI2025

Accelerating Advanced Light Source Science Through Multi-Facility HPC Workflows

Synchrotron light sources support a wide array of techniques to investigate materials, often producing complex, high-volume data that challenge traditional workflows. At the Advanced Light Source (ALS), we developed infrastructure to move microtomography data over ESnet to ALCF and NERSC, where CPU- and GPU-based algorithms generate 3D reconstructed volumes of experimental samples. We employ two data movement and reconstruction models: real-time processing as data streams directly to NERSC compute nodes, and automated file transfer to NERSC and ALCF file systems. The streaming pipeline provides users with feedback in under ten seconds, while the file-based workflow produces high-quality reconstructions suitable for deeper analysis in 20-30 minutes. This infrastructure enables users to utilize HPC resources without direct access to backend systems. We plan to extend this architecture to more endstations, supporting our beamline scientists and users.

Abramov, David↗
DOE OSTI2023

UWLi (Universal Workflow Language Interface) [SWR-24-12]

UWLi is a user interface for modifying Universal Workflow Language (UWL) files. UWL is a data format used to represent high fidelity scientific procedures in a generalized, field agnostic workflow format. See related publication: https://arxiv.org/pdf/2409.05899

Epps, Robert↗
DOE OSTI2026

SPAROW: Stochastic Programming and Related Optimization Workflows

SAND2026-16703O SPAROW: Stochastic Programming and Related Optimization Workflows is a Python library tool that facilitates the development and solution of stochastic programming problems. It provides a user-friendly class structure for defining stochastic programs through scenario-based representations of uncertainties. SPAROW incorporates multiple optimization strategies, including integer programming with all scenarios, progressive hedging, Benders decomposition, and Snoglode, a novel technique developed by Carnegie Mellon University. It also features interfaces to external solvers and functions that are commonly used in analysis workflows, making it applicable to a wide range of scientific and engineering design challenges, particularly in power grid planning. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Hart, William [Sandia National Lab. (SNL-NM), Albu↗
DOE OSTI2024

Regional-scale fault-to-structure earthquake simulations with the EQSIM framework: Workflow maturation and computational performance on GPU-accelerated exascale platforms

Continuous advancements in scientific and engineering understanding of earthquake phenomena, combined with the associated development of representative physics-based models, is providing a foundation for high-performance, fault-to-structure earthquake simulations. However, regional-scale applications of high-performance models have been challenged by the computational requirements at the resolutions required for engineering risk assessments. The EarthQuake SIMulation (EQSIM) framework, a software application development under the US Department of Energy (DOE) Exascale Computing Project, is focused on overcoming the existing computational barriers and enabling routine regional-scale simulations at resolutions relevant to a breadth of engineered systems. This multidisciplinary software development—drawing upon expertise in geophysics, engineering, applied math and computer science—is preparing the advanced computational workflow necessary to fully exploit the DOE’s exaflop computer platforms coming online in the 2023 to 2024 timeframe. Achievement of the computational performance required for high-resolution regional models containing upward of hundreds of billions to trillions of model grid points requires numerical efficiency in every phase of a regional simulation. This includes run time start-up and regional model generation, effective distribution of the computational workload across thousands of computer nodes, efficient coupling of regional geophysics and local engineering models, and application-tailored highly efficient transfer, storage, and interrogation of very large volumes of simulation data. This article summarizes the most recent advancements and refinements incorporated in the workflow design for the EQSIM integrated fault-to-structure framework, which are based on extensive numerical testing across multiple graphics processing unit (GPU)-accelerated platforms, and demonstrates the computational performance achieved on the world’s first exaflop computer platform through representative regional-scale earthquake simulations for the San Francisco Bay Area in California, USA.

58 GEOSCIENCES↗
DOE OSTI2023

Machine learning-based inversion for acoustic impedance with large synthetic training data: Workflow and data characterization

Where wells are sparse or training data are difficult to label with high-quality wireline-derived impedance logs, machine learning (ML)-based inversion of acoustic impedance typically depends on small training data sets, leading to biased prediction. We have advanced a novel workflow that applies large synthetic seismic training data to reduce facies-related bias. Using a geologically realistic model as the truth model, we randomly select sparse seed wells to perform sequential Gaussian simulation (SGS) for impedance models of the same geometry and simulate facies variability. We implement random forest regression on 30 features extracted from the synthetic volume. We observe that more seed wells tend to reduce facies-induced bias by sampling more types of facies, resulting in a better prediction. We then focus on the responses of SGS models to facies changes, the number of seed wells necessary for a useful synthetic model, and how much a synthetic model can help ML-based inversion. Here, we observe that the SGS synthetic training model outperforms well-direct training in general. For modeled clastic shore-zone systems in Miocene Gulf of Mexico, two or more seed wells are necessary for a significant reduction of root-mean-square error and outliners, and improvement of facies imaging. In a field-data test, we apply a similar workflow to quantitatively predict acoustic impedance, which is then converted to a sand-volume map at a high-frequency sequence (10–100 m), revealing detailed facies and sandstone patterns. Such results are valuable in many geologic and engineering applications, such as hydrocarbon and CO 2 reservoir prospecting, reserve estimation, simulation, etc.

3D seismic↗
DOE OSTI2024

Manuscript Workflows from and Processed Organic Matter Composition of Experimentally Burned Open Air and Muffle Furnace Vegetation Chars across Differing Burn Severity and Feedstock Types from Pacific Northwest, USA (v3)

This dataset includes processed organic matter chemistry data from an experimental study designed to compare how the chemical composition of organic matter changes across different burn conditions and vegetation materials representative of major land cover types of the Pacific Northwest, USA. Chars were created in a closed muffle furnace or on an open burn table from four different feedstock species representing vegetation commonly impacted by fire regimes across the Pacific Northwest, USA. Source data and associated metadata (including methods and geospatial information) can be found at https://data.ess-dive.lbl.gov/datasets/doi:10.15485/1894135 (Grieger et al. 2022). This dataset provides processing scripts and processed data for both solid and dissolved phase organic matter characterization data from experimentally generated chars. These processed data can be used to compare how different burn conditions may influence resultant organic matter chemistry and help further our understanding of potential biogeochemical impacts on river corridors post-fire. The processed data were subsequently analyzed; and the results and ecological implications of the findings were published in peer-reviewed manuscripts. The scripts and workflows used to develop the manuscripts are also included in this data package.This data package was originally published June 2024. It was updated September 2024 (new and modified files) and in January 2025 (modified files). See the change history section in the readme for more details.This dataset is comprised of one data package readme, one data dictionary (dd), one file level metadata (flmd), and folders containing (A) processed data; (B) general processing scripts; and (C) additional folders with specific manuscript analysis scripts and processed data. Step-by-step instructions to assist the user in recreating the workflow used to generate the results in the manuscripts is also provided. The processed data folder includes (1) a folder of processed Parallel Factor Analysis (PARAFAC) and spectra indices outputs from excitation emissions matrix (EEM) fluorescence and absorbance data; (2) a folder of processed solid state carbon-13 (13-C NMR) integrals; (3) folder of high resolution characterization of organic matter via 21 Tesla Fourier transform ion cyclotron resonance mass spectrometry (FTICR-MS) generated through the Environmental Molecular Sciences Laboratory (EMSL; https://www.pnnl.gov/environmental-molecular-sciences-laboratory) processed data outputs from Formultitude (https://github.com/PNNL-Comp-Mass-Spec/Formultitude), blank corrections and data aggregation, and calculated molecular indices. All files are .pdf, .csv, .html, .Rmd, .R, or .RData.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Usable Data Abstractions for Next-Generation Scientific Workflows

Data- and computationally-intensive scientific research, such as numerical simulations and inversions or the training of large neural networks in machine learning applications, that are well suited for HPC environments also often require expert insight and evaluation throughout the computation which can be greatly facilitated with the use of interactive computing tools, such as those in the Jupyter ecosystem. HPC workflows and interactive workflows are typically treated as orthogonal, however, the next generation of research will require both. The first challenge we face in this project is thus designing the right level of abstractions to allow interactive capabilities in the JupyterLab environment to allow the working scientist to flexibly explore and query their data at multiple levels, with a minimal amount of customization required of the underlying optimized codes. In addition to these questions regarding the high-level representation of data for interactive use in HPC, we tackled two additional issues that are part of the entire lifecycle of research and that become particularly acute in HPC contexts: how to improve the experience of interfacing with the HPC system's scheduling environment for a scientist focused on exploratory questions, and how can that scientist then best share the results of their work with others in a self-contained, reproducible manner.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Preliminary design analysis workflow for Division 5 HHA-3200 requirements for graphite core components

This report presents a design analysis workflow for graphite core components and assemblies, based on the design rules of ASME Boiler Pressure and Vessel Code, Section III, Division 5, Article HHA-3000. The workflow contains three stages: developing the design of the graphite core component, modeling the component with the finite element software MOOSE, and assessing if the component passes/fails the criteria of the HHA-3000 design rules. Since the design rules use probabilistic metrics specifically established to evaluate brittle materials, we developed a python library that performs all the statistical calculations necessary for the evaluations of the HHA-3000 criteria.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Universal Utility Data Exchange (UUDEX) - Workflow Design - Rev 1

This workflow design document describes the process of establishing a Universal Utility Data Exchange (UUDEX) Connection between two or more UUDEX Endpoints. The existing processes required to establish a data link using Inter Control Center Communications Protocol (ICCP) are very time consuming, from both the perspectives of effort and calendar time. The intent of UUDEX is to provide a more streamlined alternative. The UUDEX Workflow is also used to establish UUDEX Connections to exchange data other than that found in traditional ICCP data exchanges such as exchanges of power system model files, security events and mitigations, disturbance reports, and market data.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Official Report on the 2021 Computational and Autonomous Workflows Workshop (CAW 2021)

This technical report documents the Computational and Autonomous Workflows (CAW) workshop held at ORNL (Oak Ridge National Laboratory) in July 2021. The theme of the workshop was "FAIR workflows". This document describes the workshop and takeaways for the Department of Energy, for ORNL, and for the Oak Ridge Leadership Computing Facility.

42 ENGINEERING↗
DOE OSTI2022

Fast Reactor Physics Model Verification Studies using ARC and PyARC Workflows

PyARC was recently developed at Argonne National Laboratory to automate many of the tasks required in the ARC (Argonne Reactor Computation) fast reactor simulation workflow, from input file generation, code execution, data transfer between ARC codes, and output postprocessing. PyARC will likely be the path forward to train new users of the ARC codes with the goal of wide adoption by the national laboratories, academia, and industry. In particular, for the ANL-JAEA collaboration under the Civil Nuclear Working Group (CNWG) project agreement NE-01, PyARC will be used to model the Joyo and EBR-II reactors for comparisons with measured data and calculated results from JAEA (Task 3: Fast Reactor Fuel and Core). As an additional avenue for verification and validation, this report investigates the use of PyARC towards a variety of existing ARC-based reactor models, in order to understand its efficacy in replicating the behavior of base ARC codes and better understand any limitations within modeling realistic fast reactor problems. To this end, PyARC was used to model the Joyo MKI, RBEC Benchmark-M, PRISM Mod-B, and EBR-II Run 138B cores, and its results were compared to those from existing ARC-based models. It was found that for hexagonal-based geometries PyARC was able to replicate the behavior of ARC codes to within 10 pcm for small reactor cores, and ~150pcm difference in eigenvalue for larger cores. These discrepancies are attributed primarily to differences in local mesh refinement options between ARC and PyARC, which currently cannot be resolved with PyARC’s latest version (1.6.0). In some of these cases, PyARC was used to model steady-state problems with initial core compositions originating from a prior REBUS depletion calculation. While PyARC was not designed to support such steady-state calculations, workarounds were applied to replicate the behavior of ARC-based calculations as closely as possible. Thus, these results demonstrate the wide extent to which they can be applied to fast reactor problems while still providing immense benefit to the user in terms of automating and standardizing common routines within the fast reactor analysis workflow. This study concluded that PyARC will be suitable for modeling the steady-state conditions of the EBR-II and Joyo fast reactors as part of the CNWG project agreement.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2023

Results of the Micromorphic Upscaling Workflow for the PSAAP III Year 3 Report

Predicting the mechanical response and failure of heterogeneous materials has proven difficult. Multiscale numerical methods based in higher order continuum theories attempt to bridge the gap between microscale and macroscale structural behavior. Micromorphic continuum theories have shown promise. The Tardigrade software package is an implementation of Eringen’s micromorphic continuum theory with capabilities to support multiscale material modeling workflows. These include homogenization through the Micromorphic Filter, calibration of micromorphic material models, and macroscale simulation in Tardigrade-MOOSE. This work discusses micromorphic upscaling efforts of the University of Colorado Boulder PSAAP III multidisciplinary simulation center (MSC). Verification studies are presented that compare the accuracy of the upscaling workflow with analytical solutions for a trivial stress state and homogeneous material using direct numerical simulations (DNS) conducted in the Ratel finite element method (FEM) and GEOS material point method (MPM) codes. These verification studies consider upscaling using the Micromorphic Filter for a “single filter domain”. Finally, DNS of a heterogeneous composite material is upscaled using a “multiple filter domain” method.

36 MATERIALS SCIENCE↗
DOE OSTI2024

Algorithms and file structures to enhance software workflows for ion mobility mass spectrometry (IM-MS)

Support customizations of algorithms and raw data file structures to enhance software workflows for liquid chromatography (LC), mass spectrometry (MS) and ion mobility mass spectrometry (IM-MS)-based metabolite characterization. Evaluate and improve the integration of ion mobility to existing MS analysis methods of the Mass Profiler Professional workflow (Mass Profiler, ID Browser and Mass Profiler Professional).

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2024

The SciDAC QuantOm Framework: A composable Workflow

As part of the Scientific Discovery through Advanced Computing (SciDAC) program, the Quantum Chromodynamics Nuclear Tomography (QuantOM) project aims to analyze data from Deep Inelastic Scattering (DIS) experiments conducted at Jefferson Lab and the upcoming Electron Ion Collider. The DIS data analysis is performed on an event-level by leveraging nuclear theory models and accounting for experimental conditions. In order to efficiently run multiple analyses under varying conditions, a composable workflow was designed where each section (theory, experiment, objective minimization, etc.) has its own dedicated module. The optimization, i.e. the fit of theory to experimental data is carried out by deep learning techniques, such as Generative Adversarial Networks (GANs) or Reinforcement Learning (RL). This presentation gives an overview of the current status of the workflow, highlights present and future challenges, and highlights possible extensions to other projects with similar requirements.

Lersch, Daniel↗