Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Workflow management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Full spectrum optical constant interface to the Materials Project

Optical constants characterize the interaction of materials with light and are important properties in material design. Here we present a Python-based Corvus workflow for simulations of full spectrum optical constants from the visible and ultraviolet to hard x-ray wavelengths based on the real-space Green’s function code FEFF10 and structural data from the Materials Project (MP). The Corvus workflow manager and its associated tools provide an interface to FEFF10 and the MP database. The workflow parallelizes the FEFF computations of optical constants over all absorption edges for each material in the MP database specified by a unique MP-ID. The workflow tools determine the distribution of computational resources needed for that case. Similarly, the optical constants for selected sets of materials can be computed in a single-shot. Additionally, to illustrate the approach, we present results for several elemental solids in the periodic table, as well as a sample compound, and compare our predictions with experimental results. In addition, we provide a database of calculated results for all elements for which there is a stable elemental solid at standard conditions available in the Materials Project database. As in x-ray absorption spectra, these results are interpreted in terms of an atomic-like background and fine-structure contributions.

36 MATERIALS SCIENCE↗
DOE OSTI2024

Ensemble Simulations on Leadership Computing Systems

Scientific productivity can be enhanced through workflow management tools, relieving large High Performance Computing (HPC) system users from the tedious tasks of scheduling and designing the complex computational execution of scientific applications. This paper presents a study on the usage of ensemble workflow tools to accelerate science using the Summit and Frontier supercomputing systems. The research aims to connect science domain simulations using Oak Ridge Leadership Computing Facility (OLCF) supercomputing platforms with ensemble workflow methods in order to accelerate HPC-enabled discovery and boost scientific impact. We present the coupling, porting and optimization of Radical-Cybertools on three applications: Chroma, NAMD and LAMMPS. The tools augment traditional HPC monolithic runs with a pilot scheduler. Lessons-learned are discussed for physics, biology and materials science applications. We discuss intrinsic limitations of coupling and porting ensemble workflow tools to applications that run on large HPC systems. The origins of technical challenges and their solutions developed during the implementation process are discussed. Data management strategies, OLCF’s policies for ensembles, and natively supported workflow tools are also summarized.

Georgiadou, Antigoni [ORNL] (ORCID:000000020977631↗
DOE OSTI2017

Performance Analysis of OpenSHMEM Applications with TAU Commander

The TAU Performance System® (TAU) is a powerful and highly versatile profiling and tracing tool ecosystem for performance engineering of parallel programs. Developed over the last twenty years, TAU has evolved with each new generation of HPC systems and scales efficiently to hundreds of thousands of cores. TAU’s organic growth has resulted in a loosely coupled software toolbox such that novice users first encountering TAU’s complexity and vast array of features are often intimidated and easily frustrated. To lower the barrier to entry for novice TAU users, ParaTools and the US Department of Energy have developed “TAU Commander,” a performance engineering workflow manager that facilitates a systematic approach to performance engineering, guides users through common profiling and tracing workflows, and offers constructive feedback in case of error. This work compares TAU and TAU Commander workflows for common performance engineering tasks in OpenSHMEM applications and demonstrates workflows targeting two different SHMEM implementations, Intel Xeon “Haswell” and “Knights Landing” processors, direct and indirect measurement methods, callsite, profiles, and traces.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Curifactory: A research experiment manager

Curifactory is a command line tool and framework for organizing Python experiment code, configuration parameters, and results. It is an opinionated and lightweight approach to workflow management infrastructure and is primarily intended to support researchers conducting experiments on one machine. This software was developed to support the reproducibility of results for several data science projects in the Nuclear Nonproliferation Division at Oak Ridge National Laboratory. Curifactory is intended to be a general framework and is not specific to machine learning or data science. It can aid in any field in which experiments are primarily computation-based studies and can be implemented in Python (e.g., high-energy physics, astronomy, computational chemistry). Here, the design emphasizes the automated caching of intermediate data analysis artifacts to speed up development involving computationally intensive tasks. It also allows for data provenance and experiment reproduction. Individual experiment runs are tracked through logs and their output reports, and entire copies of a run with all cached data and metadata can be exported for others to run using Curifactory on another machine. Curifactory experiments can either be integrated into a project from the beginning or can be written on top of an existing codebase without needing significant modification. A few important views of the Curifactory library can be seen in Figure 1.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

CAMEO: A Co-design Architecture for Multi-objective Energy System Optimization (Project Report)

CAMEO (Codesign Architecture for Multi-objective Energy System Optimization) is a modular workflow management framework that abstracts co-design problems as Directed Acyclic Graphs (DAG). The framework employs JSON-based workflow specifications that enable systematic decomposition of complex optimization problems into reusable, interchangeable components including data loaders, scenario generators, optimization solvers, and result summarizers.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Toward designing effective exascale scientific computing workflows: experiences and best practices

Many fields within scientific computing have embraced advances in big-data analysis and machine learning, which often requires the deployment of large, distributed and complicated workflows that may combine training neural networks, performing simulations, running inference, and performing database queries and data analysis in asynchronous, parallel and pipelined execution frameworks. Such a shift has brought into focus the need for scalable, efficient workflow management solutions with reproducibility, error and provenance handling, traceability, and checkpoint-restart capabilities, among other needs. Here, we discuss challenges and best-practices for deploying exascale-generation computational science workflows on resources at the Oak Ridge Leadership Computing Facility (OLCF). We present our experiences with large-scale deployment of distributed workflows on the Summit supercomputer, including for bioinformatics and computational biophysics, materials science, and deep learning model optimization. We also present problems and solutions created by working within a Python-centric software base on traditional HPC systems, and discuss steps that will be required before the convergence of HPC, AI, and data science can be fully realized. Our results point to a wealth of exciting new possibilities for harnessing this convergence to tackle new scientific challenges.

Coletti, Mark↗
DOE OSTI2021

Multi-objective optimization with an integrated electromagnetics and beam dynamics workflow

In particle accelerators, RF cavities are used to accelerate charged particle beams to designed high energy for physical applications. In a typical accelerator design, the optimization of RF cavities and the optimization of beam dynamics are carried out in separate studies. For a more general and unrestricted accelerator design, a coupled optimization of the RF cavities and the beam parameters is required. For this coupled optimization problem, we have developed an integrated electromagnetics and beam dynamics workflow management system. Within this system, the geometries for a set of cavity components are first adjusted; the field modes are then computed with an electromagnetics program, and imported into a beam dynamics program for beam dynamics simulation. This workflow is encapsulated into a parallel multi-objective optimizer to achieve the integrated accelerator design optimization. A multi fidelity strategy is developed to improve the speed of the optimizer. Furthermore, this integrated global optimization capability is illustrated using a photoinjector design example and yields an improved design.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2025

The (R)evolution of Scientific Workflows in the Agentic AI Era: Towards Autonomous Science

Modern scientific discovery increasingly requires coordinating distributed facilities and heterogeneous resources, forcing researchers to act as manual workflow coordinators rather than scientists. Advances in AI leading to AI agents show exciting new opportunities that can accelerate scientific discovery by providing intelligence as a component in the ecosystem. However, it is unclear how this new capability would materialize and integrate in the real world. To address this, we propose a conceptual framework where workflows evolve along two dimensions which are intelligence (from static to intelligent) and composition (from single to swarm) to chart an evolutionary path from current workflow management systems to fully autonomous scientific laboratories. With these trajectories in mind, we present an architectural blueprint that can help the community take the next steps towards harnessing the opportunities in autonomous science with the potential for 100x discovery acceleration and transformational scientific workflows.

Shin, Woong [ORNL] (ORCID:0000000172077814)↗
DOE OSTI2022

MISPR : an open-source package for high-throughput multiscale molecular simulations

Computational tools provide a unique opportunity to study and design optimal materials by enhancing our ability to comprehend the connections between their atomistic structure and functional properties. However, designing materials with tailored functionalities is complicated due to the necessity to integrate various computational-chemistry software (not necessarily compatible with one another), the heterogeneous nature of the generated data, and the need to explore vast chemical and parameter spaces. The latter is especially important to avoid bias in scattered data points-based models and derive statistical trends only accessible by systematic datasets. Here, we introduce a robust high-throughput multi-scale computational infrastructure coined MISPR (Materials Informatics for Structure–Property Relationships) that seamlessly integrates classical molecular dynamics (MD) simulations with density functional theory (DFT). By enabling high-performance data analytics and coupling between different methods and scales, MISPR addresses critical challenges arising from the needs of automated workflow management and data provenance recording. The major features of MISPR include automated DFT and MD simulations, error handling, derivation of molecular and ensemble properties, and creation of output databases that organize results from individual calculations to enable reproducibility and transparency. In this work, we describe fully automated DFT workflows implemented in MISPR to compute various properties such as nuclear magnetic resonance chemical shift, binding energy, bond dissociation energy, and redox potential with support for multiple methods such as electron transfer and proton-coupled electron transfer reactions. The infrastructure also enables the characterization of large-scale ensemble properties by providing MD workflows that calculate a wide range of structural and dynamical properties in liquid solutions. MISPR employs the methodologies of materials informatics to facilitate understanding and prediction of phenomenological structure–property relationships, which are crucial to designing novel optimal materials for numerous scientific applications and engineering technologies.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Using the ATLAS experiment software on heterogeneous resources

With the large dataset expected from 2030 onwards by the HL-LHC at CERN, the ATLAS experiment is reaching the limits of the current data processing model in terms of traditional CPU resources based on x86_64 architectures and an extensive program for software upgrades towards the HL-LHC has been set up. The ARM CPU architecture is becoming a competitive and energy efficient alternative. Accelerators like GPUs are available in any recent HPC. In the past years ATLAS has successfully ported its full data processing and simulation software framework Athena to ARM and has invested significant effort in porting parts of the reconstruction and simulation algorithms to GPUs. We report on the successful usage of the ATLAS experiment offline and online software framework Athena on ARM and GPUs through the PanDA workflow management system at various WLCG sites. Furthermore we report on performance optimizations of the builds for ARM CPUs and the GPU integration efforts. We will discuss performance comparisons of different ARM and x86_64 architectures on WLCG resources and Cloud compute providers like GCP and AWS using ATLAS productions workflows as used in the Hep-Score23 benchmark suite.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2023

Panorama 360 (Final Report)

This is the final technical report for the DOE-funded Panorama 360 project. Panorama 360 provided a resource for the collection, analysis, and sharing of performance data about end-to-end scientific workflows executing on DOE facilities. The work focused on workflows that include experimental data generation at DOE facilities. The main activities of Panorama 360 include the development of: 1. A distributed repository that stores different types of workflow execution data (e.g., point and time series performance traces at fine- and coarse-grained levels); 2. A set of open-source data capture, curation, and publishing tools fully integrated with a state-of-the-art workflow management system that automates data ingestion to the repository and enables users to discover, query, and process data from the repository; 3. A set of analysis algorithms and machine learning based tools to perform analysis and characterization of the gathered data, which can be used to detect anomalous performance or system faults; and 4. Best practices and recommendations for workflow evaluation, analysis, execution, and architectures.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Panorama 360 (Final Report)

This final technical report from the lead institution, USC grant #DE-SC0012636, serves as the final technical report for collaborative institution UNC-CH grant #DE-SC0012390. The goal was to develop a repository and associated capabilities for data collection, ingestion, and analysis for a broad class of DOE applications that span experimental and simulation science workflows. In particular, this work focuses on workflows that include experimental data generation at DOE facilities. The main activities of Panorama 360 include the development of: (1) A distributed repository that stores different types of workflow execution data (e.g., point and time series performance traces at fine- and coarse-grained levels); (2) A set of open-source data capture, curation, and publishing tools fully integrated with a state-of-the-art workflow management system that automates data ingestion to the repository and enables users to discover, query, and process data from the repository; (3) A set of analysis algorithms and machine learning based tools to perform analysis and characterization of the gathered data, which can be used to detect anomalous performance or system faults; and (4) Best practices and recommendations for workflow evaluation, analysis, execution, and architectures.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

MaDaTS: Managing Data on Tiered Storage for Scientific Workflows

Scientific workflows are processing large amounts of data through complex simulation and analysis tasks. Meanwhile, the need to minimize I/O costs on next generation systems and the evolution of new technologies (NVRAMs, SSDs etc.) is resulting in deeper storage hierarchies on High Performance Computing (HPC) systems. A multi-tiered storage hierarchy introduces complexities in workflow and data management. There is need for simple and flexible data abstractions that can allow users to seamlessly manage workflow data and tasks on HPC systems with multiple storage tiers. MaDaTS (Managing Data on Tiered Storage for Scientific Workflows) provides an API and a command-line tool that allows users to manage their workflows and data on tiered storage (Ghoshal & Ramakrishnan (2017)).

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

MPI jobs within MPI jobs: A practical way of enabling task-level fault-tolerance in HPC workflows

While the use of workflows for HPC is growing, MPI interoperability remains a challenge for workflow management systems. The MPI standard and/or its implementations provide a number of ways to build multiple-programs-multiple-data (MPMD) applications. These methods present limitations related to fault tolerance, and are not easy to use. In this paper, we advocate for a novel MPI_Comm_launch function acting as the parallel counterpart of a system(3) call. MPI_Comm_launch allows a child MPI application to be launched inside the resources originally held by processes of a parent MPI application. Two important aspects of MPI_Comm_launch is that it pauses the calling process, and runs the child processes on the parent’s CPU cores, but in an isolated manner with respect to memory. This function makes it easier to build MPMD applications with well-decoupled subtasks. We show how this feature can provide better flexibility and better fault tolerance in ensemble simulations and HPC workflows. Here, we report results showing 2x throughput improvement for application workflows with faults, and scaling results for challenging workloads up to 256 nodes.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Data for Yield from Iowa’s first commercial miscanthus fields: implications of spatial variability for productivity and sustainability beyond research plots

This dataset contains biomass yield measurements and associated vegetation index data collected from commercial Miscanthus × giganteus fields in eastern Iowa during the 2022–2023 growing seasons. The data support the analyses presented in the article: “Yield From Iowa's First Commercial Miscanthus Fields: Implications of Spatial Variability for Productivity and Sustainability Beyond Research Plots.” We collected 105 ground-truth biomass samples from four mature commercial fields (>4 years old) covering 92.81 ha. Samples were taken from 3 m² quadrats that were hand-harvested in alignment with commercial harvest timing. Stem biomass (excluding leaves) was weighed, moisture-corrected, and converted to dry-matter yield expressed in Mg DM ha⁻¹. Sampling locations were selected to capture spatial variability visible in aerial imagery and were recorded using RTK GPS. Each biomass observation was paired with vegetation indices derived from high-resolution PlanetScope satellite imagery (3 m resolution). Images were acquired throughout the growing season, and indices were calculated to evaluate their ability to predict end-of-season biomass yield. Statistical and machine learning approaches were used to identify key predictors, and a linear regression model based on end-of-July Green Normalized Difference Vegetation Index (GNDVI) was developed and evaluated. This repository includes the data used in that modeling workflow. Management practices, economic data, full imagery time series, and additional methodological details are described in the associated publication and are not included here. The dataset consists of three comma-separated value (CSV) files: 1. Combine_Groundtruth_Yield_VI_22_23.csv This file contains ground-truth biomass yield measurements and associated key vegetation index values collected during the 2022 and 2023 growing seasons. Rows: 105 observations Columns: Year — Year of observation (2022 or 2023) Field — Field location identifier Sample_number — Unique sample identifier GNDVI_End_Jul — Green Normalized Difference Vegetation Index calculated at end of July GNDVI_End_Aug — Green Normalized Difference Vegetation Index calculated at end of August NDRE_End_Aug — Normalized Difference Red Edge index calculated at end of August Biomass_Stem_Yield_MgDM/ha — Measured stem biomass yield (megagrams dry matter per hectare) 2. trainData_GNDVI.csv This file contains the subset of observations used to train the predictive relationship between July GNDVI and biomass yield. Rows: 76 observations Columns: Unnamed: 0 — Row index retained from the original data processing workflow GNDVI_End_Jul — GNDVI at end of July Stem_Yield_MgDM/ha — Observed stem biomass yield (Mg DM ha⁻¹) 3. testData_GNDVI.csv This file contains the test dataset used to evaluate model performance. Rows: 29 observations Columns: Unnamed: 0 — Row index retained from the original data processing workflow GNDVI_End_Jul — GNDVI at end of July Predicted_Yield_MgDM/ha — Model-predicted stem biomass yield (Mg DM ha⁻¹) Observed_Yield_MgDM/ha — Measured stem biomass yield (Mg DM ha⁻¹)

Potential yield, yield gap, in-field management, y↗
DOE OSTI2018

Case Study: Nuclear-Renewable-Water Integration in Arizona

This document reports the application of the N-R HES software framework to a case study for Arizona Public Service (APS); the manager and part owner of the Palo Verde (PV) nuclear power plant. The case study is a work in progress of which this report presents a detailed description of the current model input data, assumptions and the corresponding results produced by the developed software framework. The goal of the report is, that with this information APS (together with INL) should be able to identify where more detailed data and more accurate model assumptions are necessary to reproduce all driving physical and economic phenomena in order to capture enough complexity of the reality to allow APS to use the results for their strategic decisions. The N-R HES software framework started being developed at INL two years ago [ref]. The framework has reached some level of maturity so that it can be applied to more than simple demonstration cases, e.g. real industry problems. Nevertheless, more capabilities are constantly added to accommodate the special needs of these challenging real life problems. The N-R HES framework is built on top of the RAVEN code [ref] and uses it as driver and workflow manager for all calculations. The framework has specifically been developed for the economic assessment of N-R HES systems. There are four main cornerstones of the N-R HES simulation framework: 1) generation of stochastic time series, 2) a probabilistic analysis and optimization set of algorithms available in RAVEN, 3) a set of models for representation of the physical behavior of N-R HES, and 4) a RAVEN plug-in that maps physical performance into economic performance called CashFlow. Within this framework, a broad spectrum of questions related to N-R HES can be addressed. One of the challenges currently of high interest is that the increasing penetration of variable renewables is altering the profile of the net demand (demand after removing all non curtailable renewable energy sources), with which the other generators on the grid have to cope. The N-R HES software framework is capable to analyze the potential feasibility of mitigating the resultant volatility in the net electricity demand. One possible solution to that problem currently intensively studied by the energy industry is adding stabilizing loads to the grid. These loads can be external industrial processes that are able to ramp production up and down quickly or adding variable internal loads that will lift the base load of the power plant. The latter is a possible solution currently studied by APS. APS if currently anticipating several challenges that will likely arise in the near future. One of them is, as mentioned, coping with the rapid growth of variable renewable energy (VRE) sources put on the grid. Although APS can sell its electricity for a fixed retail price to cover the local demand first, it also sells electricity in excess of the local demand at the Palo Verde energy hub. Due to the growing VRE penetration, the Palo Verde hub electricity spot price is negative more and more frequently. To mitigate this price volatility, APS is seeking to add more base load, so that the quantity of excess energy to be sold for prices potentially less than the internal retail price or even negative prices at the hub is reduced. A second challenge APS is facing is that their cooling water acquisition contract with the Sub Regional Operating Group (SROG) will expire soon and a renewal can only be done for a significantly higher price of the water. Therefore, APS is also seeking for alternative sources for their cooling water. An opportunity currently under investigation is to pump a limited quantity of brackish water from the regional ground water. Although much cheaper than the water from the new SROG contract, the salinity of the brackish water is so high that a blend of brackish and SROG water would need additional treatment to improve its quality for use in the cooling towers. An on-site reverse osmosis (RO) desalination plant is envisaged to reduce the salinity to an acceptable level. This report investigates the economic impact of such an RO plant. As one can see, the RO plant would help with both problems APS is facing: First, it would rise the APS base load helping to mitigate VRE induced hub price volatility. From this perspective, the RO plant can be seen as a stabilizing load on the grid as would be any other industrial process. Second, the RO’s goods, e.g. the clean water produced can be used by APS itself for a (hopefully) lower cost than 100% SROG water would be. The report will first describe the APS water procurement strategy in detail and lay out the possible alternative scenarios studied in this document. This is covered in Chapter 2. Next, in Chapter 3, the APS case model developed within the N-R HES software framework is described. This chapter includes also a description of the needed framework developments to accommodate the APS study.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Machine learning in nuclear materials research

Nuclear materials are often demanded to function for extended time in extreme environments, including high radiation fluxes with associated transmutations, high temperature and temperature gradients, mechanical stresses, and corrosive coolants. They also have a wide range of microstructural and chemical makeups, resulting in multifaceted and often out-of-equilibrium interactions. Machine learning (ML) is increasingly being used to tackle these complex time-dependent interactions and aid researchers in developing models and making predictions, sometimes with better accuracy than traditional modeling that focuses on one or two parameters at a time. Conventional practices of acquiring new experimental data in nuclear materials research are often slow and expensive, limiting the opportunity for data-centric ML, but new methods are changing that paradigm. Here we review high-throughput computational and experimental data approaches, especially robotic experimentation and active learning that is based on Gaussian process and Bayesian optimization. We show ML examples in structural materials (e.g., reactor pressure vessel (RPV) alloys and radiation detecting scintillating materials) and highlight new techniques of high-throughput sample preparation and characterizations, and automated radiation/environmental exposures and real-time online diagnostics. Herein, this review suggests that ML models of material constitutive relations in plasticity, damage, and even electronic and optical responses to radiation are likely to become powerful tools as they develop. Finally, we speculate on how the recent trends of using natural language processing (NLP) to aid the collection and analysis of literature data, interpretable artificial intelligence (AI), and the use of streamlined scripting, database, workflow management, and cloud computing platforms that will soon make the utilization of ML techniques as commonplace as the spreadsheet curve-fitting practices of today.

36 MATERIALS SCIENCE↗