Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Workflow management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
DOE OSTI2018

MaDaTS: Managing Data on Tiered Storage for Scientific Workflows

Scientific workflows are processing large amounts of data through complex simulation and analysis tasks. Meanwhile, the need to minimize I/O costs on next generation systems and the evolution of new technologies (NVRAMs, SSDs etc.) is resulting in deeper storage hierarchies on High Performance Computing (HPC) systems. A multi-tiered storage hierarchy introduces complexities in workflow and data management. There is need for simple and flexible data abstractions that can allow users to seamlessly manage workflow data and tasks on HPC systems with multiple storage tiers. MaDaTS (Managing Data on Tiered Storage for Scientific Workflows) provides an API and a command-line tool that allows users to manage their workflows and data on tiered storage (Ghoshal & Ramakrishnan (2017)).

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

MPI jobs within MPI jobs: A practical way of enabling task-level fault-tolerance in HPC workflows

While the use of workflows for HPC is growing, MPI interoperability remains a challenge for workflow management systems. The MPI standard and/or its implementations provide a number of ways to build multiple-programs-multiple-data (MPMD) applications. These methods present limitations related to fault tolerance, and are not easy to use. In this paper, we advocate for a novel MPI_Comm_launch function acting as the parallel counterpart of a system(3) call. MPI_Comm_launch allows a child MPI application to be launched inside the resources originally held by processes of a parent MPI application. Two important aspects of MPI_Comm_launch is that it pauses the calling process, and runs the child processes on the parent’s CPU cores, but in an isolated manner with respect to memory. This function makes it easier to build MPMD applications with well-decoupled subtasks. We show how this feature can provide better flexibility and better fault tolerance in ensemble simulations and HPC workflows. Here, we report results showing 2x throughput improvement for application workflows with faults, and scaling results for challenging workloads up to 256 nodes.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Data for Yield from Iowa’s first commercial miscanthus fields: implications of spatial variability for productivity and sustainability beyond research plots

This dataset contains biomass yield measurements and associated vegetation index data collected from commercial Miscanthus × giganteus fields in eastern Iowa during the 2022–2023 growing seasons. The data support the analyses presented in the article: “Yield From Iowa's First Commercial Miscanthus Fields: Implications of Spatial Variability for Productivity and Sustainability Beyond Research Plots.” We collected 105 ground-truth biomass samples from four mature commercial fields (>4 years old) covering 92.81 ha. Samples were taken from 3 m² quadrats that were hand-harvested in alignment with commercial harvest timing. Stem biomass (excluding leaves) was weighed, moisture-corrected, and converted to dry-matter yield expressed in Mg DM ha⁻¹. Sampling locations were selected to capture spatial variability visible in aerial imagery and were recorded using RTK GPS. Each biomass observation was paired with vegetation indices derived from high-resolution PlanetScope satellite imagery (3 m resolution). Images were acquired throughout the growing season, and indices were calculated to evaluate their ability to predict end-of-season biomass yield. Statistical and machine learning approaches were used to identify key predictors, and a linear regression model based on end-of-July Green Normalized Difference Vegetation Index (GNDVI) was developed and evaluated. This repository includes the data used in that modeling workflow. Management practices, economic data, full imagery time series, and additional methodological details are described in the associated publication and are not included here. The dataset consists of three comma-separated value (CSV) files: 1. Combine_Groundtruth_Yield_VI_22_23.csv This file contains ground-truth biomass yield measurements and associated key vegetation index values collected during the 2022 and 2023 growing seasons. Rows: 105 observations Columns: Year — Year of observation (2022 or 2023) Field — Field location identifier Sample_number — Unique sample identifier GNDVI_End_Jul — Green Normalized Difference Vegetation Index calculated at end of July GNDVI_End_Aug — Green Normalized Difference Vegetation Index calculated at end of August NDRE_End_Aug — Normalized Difference Red Edge index calculated at end of August Biomass_Stem_Yield_MgDM/ha — Measured stem biomass yield (megagrams dry matter per hectare) 2. trainData_GNDVI.csv This file contains the subset of observations used to train the predictive relationship between July GNDVI and biomass yield. Rows: 76 observations Columns: Unnamed: 0 — Row index retained from the original data processing workflow GNDVI_End_Jul — GNDVI at end of July Stem_Yield_MgDM/ha — Observed stem biomass yield (Mg DM ha⁻¹) 3. testData_GNDVI.csv This file contains the test dataset used to evaluate model performance. Rows: 29 observations Columns: Unnamed: 0 — Row index retained from the original data processing workflow GNDVI_End_Jul — GNDVI at end of July Predicted_Yield_MgDM/ha — Model-predicted stem biomass yield (Mg DM ha⁻¹) Observed_Yield_MgDM/ha — Measured stem biomass yield (Mg DM ha⁻¹)

Potential yield, yield gap, in-field management, y↗
DOE OSTI2018

Case Study: Nuclear-Renewable-Water Integration in Arizona

This document reports the application of the N-R HES software framework to a case study for Arizona Public Service (APS); the manager and part owner of the Palo Verde (PV) nuclear power plant. The case study is a work in progress of which this report presents a detailed description of the current model input data, assumptions and the corresponding results produced by the developed software framework. The goal of the report is, that with this information APS (together with INL) should be able to identify where more detailed data and more accurate model assumptions are necessary to reproduce all driving physical and economic phenomena in order to capture enough complexity of the reality to allow APS to use the results for their strategic decisions. The N-R HES software framework started being developed at INL two years ago [ref]. The framework has reached some level of maturity so that it can be applied to more than simple demonstration cases, e.g. real industry problems. Nevertheless, more capabilities are constantly added to accommodate the special needs of these challenging real life problems. The N-R HES framework is built on top of the RAVEN code [ref] and uses it as driver and workflow manager for all calculations. The framework has specifically been developed for the economic assessment of N-R HES systems. There are four main cornerstones of the N-R HES simulation framework: 1) generation of stochastic time series, 2) a probabilistic analysis and optimization set of algorithms available in RAVEN, 3) a set of models for representation of the physical behavior of N-R HES, and 4) a RAVEN plug-in that maps physical performance into economic performance called CashFlow. Within this framework, a broad spectrum of questions related to N-R HES can be addressed. One of the challenges currently of high interest is that the increasing penetration of variable renewables is altering the profile of the net demand (demand after removing all non curtailable renewable energy sources), with which the other generators on the grid have to cope. The N-R HES software framework is capable to analyze the potential feasibility of mitigating the resultant volatility in the net electricity demand. One possible solution to that problem currently intensively studied by the energy industry is adding stabilizing loads to the grid. These loads can be external industrial processes that are able to ramp production up and down quickly or adding variable internal loads that will lift the base load of the power plant. The latter is a possible solution currently studied by APS. APS if currently anticipating several challenges that will likely arise in the near future. One of them is, as mentioned, coping with the rapid growth of variable renewable energy (VRE) sources put on the grid. Although APS can sell its electricity for a fixed retail price to cover the local demand first, it also sells electricity in excess of the local demand at the Palo Verde energy hub. Due to the growing VRE penetration, the Palo Verde hub electricity spot price is negative more and more frequently. To mitigate this price volatility, APS is seeking to add more base load, so that the quantity of excess energy to be sold for prices potentially less than the internal retail price or even negative prices at the hub is reduced. A second challenge APS is facing is that their cooling water acquisition contract with the Sub Regional Operating Group (SROG) will expire soon and a renewal can only be done for a significantly higher price of the water. Therefore, APS is also seeking for alternative sources for their cooling water. An opportunity currently under investigation is to pump a limited quantity of brackish water from the regional ground water. Although much cheaper than the water from the new SROG contract, the salinity of the brackish water is so high that a blend of brackish and SROG water would need additional treatment to improve its quality for use in the cooling towers. An on-site reverse osmosis (RO) desalination plant is envisaged to reduce the salinity to an acceptable level. This report investigates the economic impact of such an RO plant. As one can see, the RO plant would help with both problems APS is facing: First, it would rise the APS base load helping to mitigate VRE induced hub price volatility. From this perspective, the RO plant can be seen as a stabilizing load on the grid as would be any other industrial process. Second, the RO’s goods, e.g. the clean water produced can be used by APS itself for a (hopefully) lower cost than 100% SROG water would be. The report will first describe the APS water procurement strategy in detail and lay out the possible alternative scenarios studied in this document. This is covered in Chapter 2. Next, in Chapter 3, the APS case model developed within the N-R HES software framework is described. This chapter includes also a description of the needed framework developments to accommodate the APS study.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Machine learning in nuclear materials research

Nuclear materials are often demanded to function for extended time in extreme environments, including high radiation fluxes with associated transmutations, high temperature and temperature gradients, mechanical stresses, and corrosive coolants. They also have a wide range of microstructural and chemical makeups, resulting in multifaceted and often out-of-equilibrium interactions. Machine learning (ML) is increasingly being used to tackle these complex time-dependent interactions and aid researchers in developing models and making predictions, sometimes with better accuracy than traditional modeling that focuses on one or two parameters at a time. Conventional practices of acquiring new experimental data in nuclear materials research are often slow and expensive, limiting the opportunity for data-centric ML, but new methods are changing that paradigm. Here we review high-throughput computational and experimental data approaches, especially robotic experimentation and active learning that is based on Gaussian process and Bayesian optimization. We show ML examples in structural materials (e.g., reactor pressure vessel (RPV) alloys and radiation detecting scintillating materials) and highlight new techniques of high-throughput sample preparation and characterizations, and automated radiation/environmental exposures and real-time online diagnostics. Herein, this review suggests that ML models of material constitutive relations in plasticity, damage, and even electronic and optical responses to radiation are likely to become powerful tools as they develop. Finally, we speculate on how the recent trends of using natural language processing (NLP) to aid the collection and analysis of literature data, interpretable artificial intelligence (AI), and the use of streamlined scripting, database, workflow management, and cloud computing platforms that will soon make the utilization of ML techniques as commonplace as the spreadsheet curve-fitting practices of today.

36 MATERIALS SCIENCE↗
DOE OSTI2019

Implementation of ATLAS Distributed Computing monitoring dashboards using InfluxDB and Grafana

For the last 10 years, the ATLAS Distributed Computing project has based its monitoring infrastructure on a set of custom designed dashboards provided by CERN. This system functioned very well for LHC Runs 1 and 2, but its maintenance has progressively become more difficult and the conditions for Run 3, starting in 2021, will be even more demanding; hence a more standard code base and more automatic operations are needed. A new infrastructure has been provided by CERN, based on InfluxDB as the data store and Grafana as the display environment. ATLAS has adapted and further developed its monitoring tools to use this infrastructure for data and workflow management monitoring and accounting dashboards, expanding the range of previous possibilities with the aim to achieve a single, simpler, environment for all monitoring applications. This document describes these tools and the data flows for monitoring and accounting.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

dCache - Keeping up With the Evolution of Science

The dCache project provides open-source software deployed internationally to satisfy ever more demanding storage requirements of various scientific communities. Its multifaceted approach provides an integrated way of supporting different use-cases with the same storage, from high throughput data ingest, through wide access and easy integration with existing systems, including event driven workflow management. With this presentation, we will show some of the recent developments that optimize data management and access to maximise the gain from stored data.

Mkrtchyan, Tigran↗
DOE OSTI2025

Evolution of DUNE’s Production System

The DUNE experiment will start running in 2029 and record 30 PB/year of raw waveforms from Liquid Argon TPCs and photon detectors. The size of individual readouts can range from 100 MB to a typical 8 GB full readout of the detector, and even 100 TB for extended readouts from supernova candidates. These data then need to be cataloged, stored and distributed for processing worldwide. This massive amount of data and a heterogeneous computing environment necessitates a powerful and robust distributed computing infrastructure. In the process of building up that infrastructure, DUNE’s production system has recently undergone an overhaul, in which it has integrated 1) a new workflow management system (justIN) 2) a new data catalog (MetaCat) and 3) a state-of-the-art data management system (Rucio). Simulations of DUNE’s Far Detector and its prototypes ProtoDUNE Horizontal Drift (ProtoDUNE-HD) and ProtoDUNE Vertical Drift (ProtoDUNE-VD), as well as data from ProtoDUNE-HD serve as the first tests of this infrastructure.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2021

Advanced calculations of X-ray spectroscopies with FEFF10 and Corvus

The real-space Green's function code FEFF has been extensively developed and used for calculations of X-ray and related spectra, including X-ray absorption (XAS), X-ray emission (XES), inelastic X-ray scattering, and electron energy-loss spectra. The code is particularly useful for the analysis and interpretation of the XAS fine-structure (EXAFS) and the near-edge structure (XANES) in materials throughout the periodic table. Nevertheless, many applications, such as non-equilibrium systems, and the analysis of ultra-fast pump–probe experiments, require extensions of the code including finite-temperature and auxiliary calculations of structure and vibrational properties. To enable these extensions, we have developed in tandem a new version FEFF10 and new FEFF -based workflows for the Corvus workflow manager, which allow users to easily augment the capabilities of FEFF10 via auxiliary codes. This coupling facilitates simplified input and automated calculations of spectra based on advanced theoretical techniques. The approach is illustrated with examples of high-temperature behavior, vibrational properties, many-body excitations in XAS, super-heavy materials, and fits of calculated spectra to experiment.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2024

Operational experience and R&D results using the Google Cloud for High-Energy Physics in the ATLAS experiment

The ATLAS experiment at CERN relies on a Worldwide Distributed Computing Grid infrastructure to support its physics program at the Large Hadron Collider. ATLAS has integrated cloud computing resources to complement its Grid infrastructure and conducted an R&D program on Google Cloud Platform. These initiatives leverage key features of commercial cloud providers: lightweight configuration and operation, elasticity and availability of diverse infrastructures. Here this paper examines the seamless integration of cloud computing services as a conventional Grid site within the ATLAS workflow management and data management systems, while also offering new setups for interactive, parallel analysis. It underscores pivotal results that enhance the on-site computing model and outlines several R&D projects that have benefited from large-scale, elastic resource provisioning models. Furthermore, this study discusses the impact of cloud-enabled R&D projects in three domains: accelerators and AI/ML, ARM CPUs and columnar data analysis techniques.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2019

Laboratory Review System/Scientific OSTI Release Tool

Software is a workflow management system designed to capture documents and route for compliance and peer reviews. Software does not in itself contain any technical information but is used as a business system to manage and release information from a research organization.

Crapo, MichaelJ↗
DOE OSTI2022

JADE

JADE provides a simple, flexible interface to submit jobs to an HPC. JADE is an HPC workflow management system that enables researchers to run simulations efficiently. It provides a simple, flexible interface to define jobs, as well as dependencies between jobs, and then dispatch them to an HPC with maximized parallelization. It also provides detailed error reporting and resource utilization statistics to aid in debugging problems.

Thom, Daniel↗
DOE OSTI2025

aiida-flux-scheduler

AiiDA is a workflow management software that is capable of accelerating simulations on HPC machines. Currently, there is no scheduler plugin for flux. The current code that is being submitted to be released is the initial alpha version. The code will be hosted on the external LLNL github group.

Keilbart, Nathan [Lawrence Livermore National Labo↗
DOE OSTI2025

Multiscale Machine-Learned Modeling Infrastructure

The Multiscale Machine-Learned Modeling Infrastructure (MuMMI) is a multiscale workflow management infrastructure that can concurrently orchestrate thousands of molecular dynamics (MD) simulations operating at different time and/or length scales, spanning nanoseconds to seconds and nanometers to micrometers. MuMMI uses machine learning (backed by biology experiments) to guide a massive ensemble of MD simulations that capture biologically relevant time and length scales with unprecedented resolution. MuMMI supports multiple MD codes such as GROMACS and ddcMD and can be fully deployed using the HPC package manager Spack. MuMMI has been used in many publications to run hundreds of thousands simulations, leading to significant biology breakthroughs.

Di Natale, Francesco [Lawrence Livermore National ↗
DOE OSTI2026

Scientific Core Library Stack (SCLS) v2026

SCLS (Scientific Core Library Stack) is an opinionated build and packaging system for scientific computing libraries developed at Lawrence Berkeley National Laboratory. It produces a coherent, reproducible stack of numerical libraries — including BLAS/LAPACK, MPI, sparse direct and iterative solvers, graph partitioners, and parallel I/O libraries (e.g., PETSc, SLEPc, HDF5, NetCDF, MUMPS, OpenBLAS) — that work together without manual repair by downstream scientific software. From a single recipe-and-flavor model, SCLS produces native RPM packages for RHEL-family Linux, DEB packages for Debian/Ubuntu, direct Unix-style prefix installs for HPC and locked-down environments, and native macOS builds. Multiple build "flavors" (e.g., GCC+OpenBLAS, GCC+MKL, Intel+MKL, debug) coexist in distinct prefixes on the same host. Compared to general-purpose meta-build frameworks, SCLS is deliberately curated rather than infinitely configurable. It enforces deterministic, audit-friendly behavior: explicit build dependencies, no silent feature autodetection, a clear open-source license policy, and rpath-based runtime linkage so installs integrate cleanly with standard package-manager workflows.

Messe, Christian [Lawrence Berkeley National Labor↗
DOE OSTI2025

Establishing model credibility for process-microstructure-property relationships in additive manufacturing using exascale computing

Additive Manufacturing (AM) of alloys holds significant promise as a disruptive technology in various industries, yet its adoption is often hindered by challenges in achieving consistent part quality. These issues are primarily due to the complex process-microstructure-property (PSP) relationships inherent to AM. Computational models can greatly aid in understanding these relationships, but their widespread impact and adoption has been limited by a lack of validated, open-source, and computationally efficient PSP modeling frameworks and hardware limitations. Here, this study leverages the ExaAM software suite and data from the AMBench-2018 series of laser powder bed fusion (LPBF) benchmark experiments to perform a comprehensive model assessment, including verification, validation, sensitivity analysis, and uncertainty quantification. The RADICAL-EnTK workflow manager was used to perform an ensemble of heat transport, solidification, and mechanical response simulations on the exascale computer Frontier, considering uncertainties in critical model inputs such as laser spot size and nucleation parameters, and consisting of 125 explicit grain structure simulations and 7875 crystal plasticity simulations. For a selected location within the Inconel 625 AMBench-2018 test artifact, sensitivity analysis and uncertainty quantification were performed using the predicted distributions of grain structure and mechanical properties. Qualitative agreement was found between the predicted grain size and texture and the observed AMBench-2018 microstructure, the mean predicted yield stress was within 5% of the experimental measurement mean, and the mean predicted engineering stress at 5% strain was within 10% of the experimental measurement mean. The insights gained from development and validation of the ExaAM PSP modeling framework will help guide future directions for enhancing the credibility and reliability of PSP models in AM, thereby accelerating the adoption of AM technologies in various industries.

Additive manufacturing↗
DOE OSTI2016

Warthog: Progress on Coupling BISON and PROTEUS

The Nuclear Energy Advanced Modeling and Simulation (NEAMS) program from the Office of Nuclear Energy at the Department of Energy (DOE) provides a robust toolkit for modeling and simulation of current and future advanced nuclear reactor designs. This toolkit provides these technologies organized across product lines, with two divisions targeted at fuels and end-to-end reactor modeling, and a third for integration, coupling, and high-level workflow management. The Fuels Product Line (FPL) and the Reactor Product Line (RPL) provide advanced computational technologies that serve each respective field effectively. There is currently a lack of integration between the product lines, impeding future improvements of simulation solution fidelity. In order to mix and match tools across the product lines, a new application called Warthog was produced. Warthog is built on the Multi-physics Object-Oriented Simulation Environment (MOOSE) framework developed at Idaho National Laboratory (INL). This report details the continuing efforts to provide the Integration Product Line (IPL) with interoperability using the Warthog code. Currently, this application strives to couple the BISON fuel performance application from the FPL using the PROTEUS Core Neutronics application from the RPL. Warthog leverages as much prior work from the NEAMS program as possible, enabling interoperability between the independently developed MOOSE and SHARP frameworks, and the libMesh and MOAB mesh data formats. Previous work performed on Warthog allowed it to couple a pin cell between the two codes. However, as the temperature changed due to the BISON calculation, the cross sections were not recalculated, leading to errors as the temperature got further away from the initial conditions. XSProc from the SCALE code suite was used to calculate the cross sections as needed. The remainder of this report discusses the changes to Warthog to allow for the implementation of XSProc as an external code. It also discusses the changes made to Warthog to allow it to fit more cleanly into the MultiApp syntax of the MOOSE framework. The capabilities, design, and limitations of Warthog will be described, in addition to some of the test cases that were used to demonstrate the code. Future plans for Warthog will be discussed, including continuation of the modifications to the input and coupling to other SHARP codes such as Nek5000.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗