Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Workflow”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

A New Workflow of X-ray CT Image Processing and Data Analysis of Structural Features in Rock Using Open-Source Software

X-ray computed tomography (CT) images of rock specimens often contain artifacts which must be corrected before scientific analyses are performed. Here, we present a new workflow of automated image processing to utilize poor-quality X-ray CT scan images. The workflow runs on the open-source image analysis software and efficiently separates desired features from low-contrast scanned images. The new workflow is a two-step technique using contrast enhancement and automated feature segmentation to generate noise-free binary images. The results of binary images using the proposed workflow and using a conventional thresholding technique are analyzed to show the quality of the proposed method. The paper also presents a workflow of estimating the structural geometries of features in two and three dimensions. The results of the structural feature analyses and computational time were compared between the open-source (ImageJ) and commercial image analysis software (Bruker Computed Tomography Analyzer). The commercial software was more computationally efficient, but the task-specific macros in open-source software enabled the user-desired automation in image processing and data extraction of desired structural features of comparable quality.

47 OTHER INSTRUMENTATION↗
DOE OSTI2021

A deep learning-accelerated data assimilation and forecasting workflow for commercial-scale geologic carbon storage

Fast assimilation of monitoring data to update forecasts of pressure buildup and carbon dioxide (CO 2 ) plume migration under geologic uncertainties is a challenging problem in geologic carbon storage. The high computational cost of data assimilation with a high-dimensional parameter space impedes fast decision-making for commercial-scale reservoir management. We propose to leverage physical understandings of porous medium flow behavior with deep learning techniques to develop a fast data assimilation-reservoir response forecasting workflow. Applying an Ensemble Smoother Multiple Data Assimilation (ES-MDA) framework, the workflow updates geologic properties and predicts reservoir performance with quantified uncertainty from pressure history and CO 2 plumes interpreted through seismic inversion. As the most computationally expensive component in such a workflow is reservoir simulation, we developed surrogate models to predict dynamic pressure and CO 2 plume extents under multi-well injection. The surrogate models employ deep convolutional neural networks, specifically, a wide residual network and a residual U-Net. The workflow is validated against a flat threedimensional reservoir model representative of a clastic shelf depositional environment. Intelligent treatments are applied to bridge between quantities in a true-3D reservoir model and those in a single-layer reservoir model. The workflow can complete history matching and reservoir forecasting with uncertainty quantification in less than one hour on a mainstream personal workstation.

25 ENERGY STORAGE↗
DOE OSTI2024

A deep learning-based workflow for fast prediction of 3D state variables in geological carbon storage: A dimension reduction approach

Deep learning (DL) models are extensively used as surrogate models for high-fidelity simulations of multiphase fluid flow in porous media at large scales, enabling fast forecasts of the spatial–temporal evolution of three-dimensional (3D) state variables in geological carbon storage (GCS). However, training these models in high-dimensional space remains computationally demanding and prone to overfitting because of limited training data. This paper presents a novel workflow to address these challenges by integrating dimension reduction (DR) methods. Here, the proposed workflow employed pre-trained DR models to extract the latent variables of geological models and state variables and utilized the multi-layer perceptron (MLP) for constructing mapping functions between the input and output variables in latent spaces. Subsequently, the pre-trained reconstruction models converted the MLP-predicted latent state variables to their original high-dimensional form. Furthermore, we proposed a novel strategy for the DR and reconstruction of 3D saturation fields to account for the unique data characteristics of sparsity, nonuniformity, and discontinuity. The proposed strategy applied PCA and inverse PCA for 2D average saturation fields and developed a DL-based 3D reconstruction model, leveraging three 2D average saturation fields as input to produce a 3D saturation field as output. The pre-training of DR and reconstruction models and training of MLP models were conducted on 84 Gulf of Mexico (GoM) simulations and evaluated on 12 testing simulations. Each simulation contained 720 monthly time steps, with the first 360 months as the injection period and the rest as the post-injection period. The proposed workflow, incorporating DR and DL models, accurately predicts the normalized 3D pressure fields, achieving mean square error (MSE) of 2.92 × 10 -7 compared to the ground truth obtained from a full-physics simulator. Furthermore, the proposed strategy outperformed PCA and convolutional autoencoder (CAE) models on 3D saturation fields, resulting in minor workflow prediction errors with an MSE of 2.93 × 10 -5 . The results suggest the proposed workflow provides sufficient predictive fidelity across temporal and spatial scales, and enables a speedup of 160 times compared to the full-physics simulator, facilitating improved decision-making and risk assessment for large-scale GCS management in real-time scenarios.

3D reconstruction model↗
DOE OSTI2023

tinyIFD: A High-Throughput Binding Pose Refinement Workflow Through Induced-Fit Ligand Docking

A critical step in structure-based drug discovery is predicting whether and how a candidate molecule binds to a model of a therapeutic target. However, substantial protein side chain movements prevent current screening methods, such as docking, from accurately predicting the ligand conformations and require expensive refinements to produce viable candidates. Here, we present the development of a high-throughput and flexible ligand pose refinement workflow, called “tinyIFD”. The main features of the workflow include the use of specialized high-throughput, small-system MD simulation code mdgx.cuda and an actively learning model zoo approach. We show the application of this workflow on a large test set of diverse protein targets, achieving 66% and 76% success rates for finding a crystal-like pose within the top-2 and top-5 poses, respectively. We also applied this workflow to the SARS-CoV-2 main protease (M pro ) inhibitors, where we demonstrate the benefit of the active learning aspect in this workflow.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2021

Common workflows for computing material properties using different quantum engines

The prediction of material properties based on density-functional theory has become routinely common, thanks, in part, to the steady increase in the number and robustness of available simulation packages. This plurality of codes and methods is both a boon and a burden. While providing great opportunities for cross-verification, these packages adopt different methods, algorithms, and paradigms, making it challenging to choose, master, and efficiently use them. We demonstrate how developing common interfaces for workflows that automatically compute material properties greatly simplifies interoperability and cross-verification. We introduce design rules for reusable, code-agnostic, workflow interfaces to compute well-defined material properties, which we implement for eleven quantum engines and use to compute various material properties. Each implementation encodes carefully selected simulation parameters and workflow logic, making the implementer’s expertise of the quantum engine directly available to non-experts. All workflows are made available as open-source and full reproducibility of the workflows is guaranteed through the use of the AiiDA infrastructure.

36 MATERIALS SCIENCE↗
DOE OSTI2024

Enabling Low-Overhead HT-HPC Workflows at Extreme Scale using GNU Parallel

GNU Parallel is a versatile and powerful tool for process parallelization widely used in scientific computing. This paper demonstrates its effective application in high-performance computing (HPC) environments, particularly focusing on its scalability and efficiency in executing large-scale high-throughput high-performance computing (HT-HPC) workflows. Through real-world examples, we highlight GNU Parallel’s performance across various HPC workloads, including GPU computing, container-based workloads, and node-local NVMe storage. Our results on two leading supercomputers, OLCF’s Frontier and NERSC’s Perlmutter, showcase GNU Parallel’s rapid process dispatching ability and its capacity to maintain low overhead even at extreme scales. We explore GNU Parallel’s application in massive parallel file transfers using a scheduled Data Transfer Node (DTN) cluster, emphasizing its broad utility in diverse scientific workflows. Beyond its direct application as a viable workflow manager, GNU Parallel can be employed in conjunction with other workflow systems as a "last-mile" parallelizing driver and as a quick prototyping tool to design and extract parallel profiles from application executions. We then argue that the potential for GNU Parallel to transform workflow management at extreme scales is substantial, paving the way for more efficient and effective scientific discoveries.

Maheshwari, Ketan↗
DOE OSTI2022

Co-scheduling Ensembles of In Situ Workflows

Molecular dynamics (MD) simulations are widely used to study large-scale molecular systems. HPC systems are ideal platforms to run these studies, however, reaching the necessary simulation timescale to detect rare processes is challenging, even with modern supercomputers. To overcome the timescale limitation, the simulation of a long MD trajectory is replaced by multiple short-range simulations that are executed simultaneously in an ensemble of simulations. Analyses are usually co-scheduled with these simulations to efficiently process large volumes of data generated by the simulations at runtime, thanks to in situ techniques. Executing a workflow ensemble of simulations and their in situ analyses requires efficient co- scheduling strategies and sophisticated management of computational resources so that they are not slowing down each other. In this paper, we propose an efficient method to co-schedule simulations and in situ analyses such that the makespan of the workflow ensemble is minimized. We present a novel approach to allocate resources for a workflow ensemble under resource constraints by using a theoretical framework modeling the workflow ensemble’s execution. We evaluate the proposed approach using an accurate simulator based on the WRENCH simulation framework on various workflow ensemble configurations. Results demonstrate the significance of co-scheduling simulations and in situ analyses that couple data together to benefit from data locality, in which inefficient scheduling decisions can lead to slowdown in makespan up to a factor of 30.

Do, Tu Mai Anh↗
DOE OSTI2025

In-Transit Data Transport Strategies for Coupled AI-Simulation Workflow Patterns

Coupled AI-Simulation workflows are becoming the major workloads for HPC facilities, and their increasing complexity necessitates new tools for performance analysis and prototyping of new in-situ workflows. We present SimAI-Bench, a tool designed to both prototype and evaluate these coupled workflows. In this paper, we use SimAI-Bench to benchmark the data transport performance of two common patterns on the Aurora supercomputer: a one-to-one workflow with co-located simulation and AI training instances, and a many-to-one workflow where a single AI model is trained from an ensemble of simulations. For the one-to-one pattern, our analysis shows that node-local and DragonHPC data staging strategies provide excellent performance compared Redis and Lustre file system. For the many-to-one pattern, we find that data transport becomes a dominant bottleneck as the ensemble size grows. Our evaluation reveals that file system is the optimal solution among the tested strategies for the many-to-one pattern.

Tummalapalli, Harikrishna [Argonne National Labora↗
DOE OSTI2025

Exascale workflow applications and middleware: An ExaWorks retrospective

Exascale computers offer transformative capabilities to combine data-driven and learning-based approaches with traditional simulation applications to accelerate scientific discovery and insight. However, these software combinations and integrations are difficult to achieve due to the challenges of coordinating and deploying heterogeneous software components on diverse and massive platforms. Here, we present the ExaWorks project, which addresses many of these challenges. We developed a workflow Software Development Toolkit (SDK), a curated collection of workflow technologies that can be composed and interoperated through a common interface, engineered following current best practices, and specifically designed to work on HPC platforms. ExaWorks also developed PSI/J, a job management abstraction API, to simplify the construction of portable software components and applications that can be used over various HPC schedulers. The PSI/J API is a minimal interface for submitting and monitoring jobs and their execution state across multiple and commonly used HPC schedulers. We also describe several leading and innovative workflow examples of ExaWorks tools used on DOE leadership platforms. Furthermore, we discuss how our project is working with the workflow community, large computing facilities, and HPC platform vendors to address the requirements of workflows sustainably at the exascale.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

STNS01-21 BEE - FY21 P6-2: Archive, clone, and re-run workflows [Slide]

BEE will give ECP a tool that great simplifies the deployment of containerized workflows on the next generation of pre-exascale and exascale systems, as well as public and private clouds. BEE allows scientists to describe their workflow using the Common Workflow Language and then deploy that workflow across the entire spectrum of systems without having to learn the specifics of each container runtime, HPC resource manager, or cloud API. BEE also streamlines the curation and sharing of common workflows among the scientific community.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Integrated End-to-end Performance Prediction and Diagnosis for Extreme Scientific Workflows (IPPD) (Final Report)

This report details the accomplishments from the ASCR funded project “Integrated End-to-end Performance Prediction and Diagnosis for Extreme Scientific Workflows” under the award numbers FWP-66406 and DE-SC0012630, with a focus on the UC San Diego (Award No. DE-SC0012630) part of the accomplishments. We refer to the project as IPPD. The main activities of IPPD were centered on the development and integration of provenance information to capture empirically workflow information and identify the sources of bottlenecks as well as variability, and modeling and simulation to provide insights and predictively explore multiple scenarios for development of advanced techniques for optimization of resources and workflow execution. After a Phase I of the project, a Phase II effort focused on three major aspects: a) observe how data is generated, distributed, and used; b) analyze how data is (repeatedly) consumed with a focus both on repeated patterns and anomalies; and c) explore how to optimize data motion. The project leveraged and extended our existing tools with new research and demonstrated our work on the Belle II workflow suite as well as on workflows from NSLS-II.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Predictive Capability Maturity Model Demonstration for Cylindrical Cavity Coupling Using Gemma in the Next Generation Workflow

The predictive capability maturity model (PCMM) uses the expert elicitation process to generate credibility evidence for a particular analysis. To ensure Gemma has the capability to efficiently produce this credibility evidence, next generation workflows (NGW) are created for the solution verification, calibration/validation, and input uncertainty quantification portions of the PCMM assessment. These workflows are then used on the Higgins cylinder problem, which is representative of applications involving external-to-internal electromagnetic field coupling through a slot. The uncertainties calculated using these workflows are then used to calculate the validation comparison error and the validation uncertainty for the model following the American Society of Mechanical Engineers (ASME) verification and validation (V&V) 20 standard. These workflows will enable analysts to iterate each element of PCMM more efficiently than if completed without using a NGW workflow. An example of this iterative process is shown in Section 7.2.

42 ENGINEERING↗
DOE OSTI2024

Lambda-PFLOTRAN 1.0: a workflow for incorporating organic matter chemistry informed by ultra high resolution mass spectrometry into biogeochemical modeling

Abstract. Organic matter (OM) composition plays a central role in microbial respiration of dissolved organic matter and subsequent biogeochemical reactions. Here, a direct connection of organic matter chemistry and thermodynamics to reactive transport simulators has been achieved through the newly developed Lambda-PFLOTRAN workflow tool that succinctly incorporates carbon chemistry data generated from Fourier transform ion cyclotron resonance mass spectrometry (FTICR-MS) into reaction networks to simulate organic matter degradation and the resulting biogeochemistry. Lambda-PFLOTRAN is a Python-based workflow, executed through a Jupyter notebook interface, that digests raw FTICR-MS data, develops a representative reaction network based on substrate-explicit thermodynamic modeling (also termed lambda modeling due to its key thermodynamic parameter λ used therein), and completes a biogeochemical simulation with the open source, reactive flow and transport code PFLOTRAN. The workflow consists of the following five steps: configuration, thermodynamic (lambda) analysis, sensitivity analysis, parameter estimation, and simulation output and visualization. Two test cases are provided to demonstrate the functionality of the Lambda-PFLOTRAN workflow. The first test case uses laboratory incubation data of temporal oxygen depletion to fit lambda parameters (i.e., maximum utilization rate and microbial carrying capacity). A slightly more complex second test case fits multiple lambda formulation and soil organic matter release parameters to temporal greenhouse gas generation measured during a soil incubation. Overall, the Lambda-PFLOTRAN workflow facilitates upscaling by using molecular-scale characterization to inform biogeochemical processes occurring at larger scales.

58 GEOSCIENCES↗
DOE OSTI2022

Watershed Workflow: A toolset for parameterizing data-intensive, integrated hydrologic models

Integrated, distributed hydrologic models leverage advances in computational power and data accessibility to improve predictive understanding of the water cycle. While impressive advances in this area of environmental modeling have been accomplished, such models are still rarely used, partially because of difficulty integrating model and data. This research describes the release of Watershed Workflow version 1.2, a new library aiming to automate and enable complex workflows defining inputs to high resolution, integrated, distributed hydrologic models. Watershed Workflow provides tools enabling the discovery, acquisition, mapping, and coordination of watershed geometry, land cover, soil properties, and meteorological data. It enables the construction of unstructured meshes that incorporate this data, and provides tools for automating a “first” simulation on any watershed in the United States. We present the design of the workflow tool, and describe best practices for its usage, culminating in a final example from watershed specification to simulation at the Coweeta Hydrologic Laboratory.

Integrated hydrologic modeling↗
DOE OSTI2021

A workflow to assess the efficacy of brine extraction for managing injection-induced seismicity potential using data from a CO 2 injection site near Decatur, Illinois

Injection of CO 2 for storage in a geologic formation increases pore pressure and alters in situ stresses. Depending on the orientation of any existing fault and fracture planes, such as critically stressed planes, this stress alteration will modify normal stresses acting on planes and could result in frictional sliding and release stored energy in the form of seismicity. Brine extraction (BE) is a technique that can be applied prior to, or during, CO 2 injection to reduce pore pressure for increasing storage capacity and, potentially, for reducing the likelihood of frictional sliding. Here a workflow is described to assess the efficacy of BE for mitigating frictional sliding (i.e., seismicity) during injection and entails: site characterization, stress calculations and failure assessment, static and dynamic modeling, and BE operational planning. Site characterization describes the stress field used to calculate the Coulomb Failure Function (CFF) that constrains allowable pore pressure changes and injection rates in the numerical simulation of CO 2 injection scenarios. The inclusion of BE in the workflow allows for determination of the potential need for pressure reduction, and evaluation of the effectiveness of this operation. Example application of the workflow using an injection field dataset near Decatur, IL, provides insight on fracture planes and stresses at the site, formation properties and the impact of variable CO 2 injection-rate targets on whether BE plans are required. The study workflow indicates that BE could enhance CO 2 injection rate by 39% and correspondingly reduce the potential for injection-induced seismicity as indicated by a reduction in CFF.

58 GEOSCIENCES↗
DOE OSTI2020

CoREC: Scalable and Resilient In-memory Data Staging for In-situ Workflows

The dramatic increase in the scale of current and planned high-end HPC systems is leading new challenges, such as the growing costs of data movement and IO, and the reduced mean time between failures (MTBF) of system components. In-situ workflows, i.e., executing the entire application workflows on the HPC system, have emerged as an attractive approach to address data-related challenges by moving computations closer to the data, and staging-based frameworks have been effectively used to support in-situ workflows at scale. However, the resilience of these staging-based solutions has not been addressed, and they remain susceptible to expensive data failures. Furthermore, naive use of data resilience techniques such as n-way replication and erasure codes can impact latency and/or result in significant storage overheads. In this article, we present CoREC, a scalable and resilient in-memory data staging runtime for large-scale in-situ workflows. CoREC uses a novel hybrid approach that combines dynamic replication with erasure coding based on data access patterns. It also leverages multiple levels of replications and erasure coding to support diverse data resiliency requirements. Furthermore, the article presents optimizations for load balancing and conflict-avoiding encoding, and a low overhead, lazy data recovery scheme. We have implemented the CoREC runtime and have deployed with the DataSpaces staging service on leadership class computing machines and present an experimental evaluation in the article. Here, the experiments demonstrate that CoREC can tolerate in-memory data failures while maintaining low latency and sustaining high overall storage efficiency at large scales.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Managing Dynamic Workflows in BEE

BEE is a powerful tool for: Managing and visualizing scientific workflows; Simplifying workflow execution on HPC and cloud platforms. BEE supports much of the CWL specification. Did not support execution of complex ”scattering” workflows. By introducing the PseudoTask: Can generate tasks to run on variable number of inputs; BEE is another step closer to supporting the entire CWL specification; BEE can now support parallelized workflows with scattering tasks.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Deploying Machine Learning Workflows into HPC environment

Outline: Workflows Overview; Common Workflow Language (CWL), Example of a CWL; BEE Overview; Machine Learning Components; Machine Learning Scientific Workflow using CWL → A new test case for BEE; Discussion: Benefits and Caveats of current ML workflow; Conclusion.

97 MATHEMATICS AND COMPUTING↗