Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Workflow”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

ProvLight: Efficient Workflow Provenance Capture on the Edge-to-Cloud Continuum

Modern scientific workflows require hybrid infrastructures combining numerous decentralized resources on the IoT/Edge interconnected to Cloud/HPC systems (aka the Computing Continuum) to enable their optimized execution. Understanding and optimizing the performance of such complex Edge-to-Cloud workflows is challenging. Capturing the provenance of key performance indicators, with their related data and processes, may assist in understanding and optimizing workflow executions. However, the capture overhead can be prohibitive, particularly in resource-constrained devices, such as the ones on the IoT/Edge.To address this challenge, based on a performance analysis of existing systems, we propose ProvLight, a tool to enable efficient provenance capture on the IoT/Edge. We leverage simplified data models, data compression and grouping, and lightweight transmission protocols to reduce overheads. We further integrate ProvLight into the E2Clab framework to enable workflow provenance capture across the Edge-to-Cloud Continuum. This integration makes E2Clab a promising platform for the performance optimization of applications through reproducible experiments.We validate ProvLight at a large scale with synthetic workloads on 64 real-life IoT/Edge devices in the FIT IoT LAB testbed. Evaluations show that ProvLight outperforms state-of-the-art systems like ProvLake and DfAnalyzer in resource-constrained devices. ProvLight is 26—37x faster to capture and transmit provenance data; uses 5—7x less CPU; 2x less memory; transmits 2x less data; and consumes 2—2.5x less energy. ProvLight [1] and E2Clab [2] are available as open-source tools.

Rosendo, Daniel↗
DOE OSTI2024

Workflow Provenance in the Computing Continuum for Responsible, Trustworthy, and Energy-Efficient AI

As Artificial Intelligence (AI) becomes more pervasive in our society, it is crucial to develop, deploy, and assess Responsible and Trustworthy AI (RTAI) models, i.e., those that consider not only accuracy but also other aspects, such as explainability, fairness, and energy efficiency. Workflow provenance data have historically enabled critical capabilities towards RTAI. Provenance data derivation paths contribute to responsible workflows through transparency in tracking artifacts and resource consumption. Provenance data are well-known for their trustworthiness helping explainability, reproducibility, and accountability. However, there are complex challenges to achieve RTAI, which are further complicated by the heterogeneous infrastructure in the computing continuum (Edge-Cloud-HPC) used to develop and deploy models. As a result, a significant research and development gap remains between workflow provenance data management and RTAI. In this paper, we present a vision of the pivotal role of workflow provenance in supporting RTAI and discuss related challenges. We present a schematic view between RTAI and provenance, and highlight open research directions.

Santos Souza, Renan↗
DOE OSTI2024

Towards Resilient Near Real-Time Analysis Workflows in Fusion Energy Science

Nuclear fusion holds the promise of an endless source of energy. Several research experiments across the world and joint modeling and simulation efforts between the nuclear physics and high performance computing communities are actively preparing the operation of the International Thermonuclear Experimental Reactor (ITER). Both experimental reactors and their simulated counterparts generate data that must be analyzed quickly and in a resilient way to support decision making for the configuration of subsequent runs or prevent a catastrophic failure. However, the cost if the traditional techniques used to improve the resilience of analysis workflows, i.e., replicating datasets and computational tasks, becomes prohibitive with explosion of the volume of data produced by modern instruments and simulations. Therefore, we advocate in this paper for an alternate approach based on data reduction and data streaming. The rationale is that by allowing for a reasonable, controlled, and guaranteed loss of accuracy it becomes possible to transfer smaller amounts of data, shorten the execution time of analysis workflows, and lower the cost of replication to increase resilience. We develop our research and development roadmap towards resilient near real-time analysis workflows in fusion energy science and present early results showing that data streaming and data reduction is a promising way to speed up the execution and improve the resilience of analysis workflows.

Suter, Fred↗
DOE OSTI2024

Dual Channel Dual Staging: Hierarchical and Portable Staging for GPU-Based In-Situ Workflow

In-situ workflows have emerged as an attractive approach for addressing data movement challenges at very large scales. Since GPU-based architectures dominate the HPC landscapes, porting these in-situ workflows, and, specifically, the inter-application data exchange, to GPU-based systems can be challenging. Technologies such as GPUDirect RDMA (GDR), which is typically used for I/O in GPU applications as an optimization that circumvents the CPU overhead, can be leveraged to support bulk data exchanges between GPU applications. However, current GDR design often lacks performance portability across HPC clusters built with different hardware configurations. Furthermore, the local CPU may also be effectively used as an auxiliary communication mechanism to offload data exchanges. In this paper, we present a dual channel dual staging approach for efficient, scalable, and performance-portable inter-application data exchange for in-situ workflows. This approach exploits the data access pattern within in-situ workflows along with the inherent execution asynchrony to accelerate data exchanges and, at the same time, improve performance portability. Specifically, the dual channel dual staging method leverages both the local CPU and the remote data staging server to build a hierarchical joint staging area and uses this staging area to transform blocking inter-application bulk data exchanges into best-effort local data movements between GPU and CPU. The dual channel dual staging is implemented as a portability extension of the Dataspaces-GPU staging framework. We present an experimental evaluation of its performance, portability, and scalability using this implementation on three leadership GPU clusters. The evaluation results demonstrate that the dual channel dual staging method saves up to 75% in data-exchange time compared to host-based, GDR, and alternate portable designs, while maintaining scalability (up to 512 GPUs) and performance portability across the three platforms.

Zhang, Bo [University of Utah]↗
DOE OSTI2024

Preparing an on-Demand Cloud Processing Workflow for NISAR Ecosystems Science Products

In preparation for the NISAR launch and data collection in 2024, the NISAR Project Science Team is building workflows for each Science Team discipline (Ecosystems, Cryosphere, and Solid Earth). This abstract focuses on the Ecosystem disciplines and the development of on-demand cloud-processing workflows for wetlands inundation, forest biomass, agricultural active crop area, and forest disturbance. The workflow simulates NISAR data using UAVSAR or ALOS-2 Single Look Complex data, which are processed to Level 2 geocoded polarimetric covariance matrix products using InSAR Scientific Computing Environment 3.0 software and to Level 3 science products using the Algorithm Theoretical Basis Documents. In this presentation, we describe these workflows and efforts to improve efficiency and data accessibility by using a cloud processing system. We present preliminary sample products from each Ecosystem discipline: inundation, forest biomass, crop area, and forest disturbance.

Christensen, Alexandra↗
DOE OSTI2024

A High-Quality Workflow for Multi-Resolution Scientific Data Reduction and Visualization

Multi-resolution methods such as Adaptive Mesh Refinement (AMR) can enhance storage efficiency for HPC applications generating vast volumes of data. However, their applicability is limited and cannot be universally deployed across all applications. Furthermore, integrating lossy compression with multi-resolution techniques to further boost storage efficiency encounters significant barriers. To this end, we introduce an innovative workflow that facilitates high-quality multi-resolution data compression for both uniform and AMR simulations. Initially, to extend the usability of multi-resolution techniques, our workflow employs a compression-oriented Region of Interest (ROI) extraction method, transforming uniform data into a multi-resolution format. Subsequently, to bridge the gap between multi-resolution techniques and lossy compressors, we optimize three distinct compressors, ensuring their optimal performance on multi-resolution data. These optimizations can improve the compression ratio of SOTA approaches by up to 3.3× under the same data quality loss. Lastly, we incorporate an advanced uncertainty visualization method into our workflow to understand the potential impacts of lossy compression. Experimental evaluation demonstrates that our workflow achieves significant compression quality improvements.

Wang, Daoce↗
DOE OSTI2025

Scalable Multi-Facility Workflows for Artificial Intelligence Applications in Climate Research

Earth observation satellites and earth system models are sources of vast, multi-modal datasets that are invaluable for advancing climate and environmental research. However, their scale and complexity pose significant challenges for processing and analysis. In this paper we discuss our experiences in developing and using a scientific research application using an automated multi-facility workflow that orchestrates data collection, preprocessing, artificial intelligence (AI) inferencing, and data movement across diverse computational resources, leveraging the Advanced Computing Ecosystem Testbed at the Oak Ridge Leadership Computing Facility (OLCF). We demonstrate that our workflow can be seamlessly integrated and orchestrated across research facilities managed by different federal agencies, thus allowing users to extract new scientific insights from climate datasets. The experimental results indicate that the multi-facility workflow significantly reduces processing time, enhances scalability, and maintains high efficiency across varying workloads. Notably, our workflow processes 12,000 high-resolution satellite images in just 44 seconds using 80 workers distributed across 10 nodes on the OLCF systems. Such high throughput is essential for dynamic tokenization and sharding of petascale satellite data for distributed AI model training and inferencing at scale across thousands of GPUs.

Kurihana, Takuya [ORNL] (ORCID:0000000156698565)↗
DOE OSTI2021

Emerging Frameworks for Advancing Scientific Workflows Research, Development, and Education

Lightning talks of the Workflows in Support of Large-Scale Science (WORKS) workshop are a venue where the workflow community (researchers, developers, and users) can discuss work in progress, emerging technologies and frameworks, and training and education materials. This paper summarizes the WORKS 2021 lightning talks, which cover four broad topics: (i) libEnsemble, a Python library to coordinate the concurrent evaluation of dynamic ensembles of calculations; (ii) Edu WRENCH, a set of online pedagogic modules that provides simulation-driven hands-on activity in the browser; (iii) VisDict, an envisioned visual dictionary framework that will translate terms, jargon, and concepts between research domains and workflow providers; and (iv) Pegasus Kickstart, a lightweight tool for capturing workflow tasks' performance, including performance metrics from Nvidia GPUs.

Casanova, Henri↗
DOE OSTI2022

Novel Proposals for FAIR, Automated, Recommendable, and Robust Workflows

Lightning talks of the Workflows in Support of Large-Scale Science (WORKS) workshop are a venue where the workflow community (researchers, developers, and users) can discuss work in progress, emerging technologies and frameworks, and training and education materials. This paper summarizes the WORKS 2022 lightning talks, which cover five broad topics: data integrity of scientific workflows; a machine learning-based recommendation system; a Python toolkit for running dynamic ensembles of simulations; a cross-platform, high-performance computing utility for processing shell commands; and a meta(data) framework for reproducing hybrid workflows.

Abhinit, Ishan↗
DOE OSTI2022

An automated workflow that generates atom mappings for large‐scale metabolic models and its application to Arabidopsis thaliana

SUMMARY Quantification of reaction fluxes of metabolic networks can help us understand how the integration of different metabolic pathways determines cellular functions. Yet, intracellular fluxes cannot be measured directly but are estimated with metabolic flux analysis (MFA), which relies on the patterns of isotope labeling of metabolites in the network. The application of MFA also requires a stoichiometric model with atom mappings that are currently not available for the majority of large‐scale metabolic network models, particularly of plants. While automated approaches such as the Reaction Decoder Toolkit (RDT) can produce atom mappings for individual reactions, tracing the flow of individual atoms of the entire reactions across a metabolic model remains challenging. Here we establish an automated workflow to obtain reliable atom mappings for large‐scale metabolic models by refining the outcome of RDT, and apply the workflow to metabolic models of Arabidopsis thaliana . We demonstrate the accuracy of RDT through a comparative analysis with atom mappings from a large database of biochemical reactions, MetaCyc. We further show the utility of our automated workflow by simulating 15 N isotope enrichment and identifying nitrogen (N)‐containing metabolites which show enrichment patterns that are informative for flux estimation in future 15 N‐MFA studies of A. thaliana . The automated workflow established in this study can be readily expanded to other species for which metabolic models have been established and the resulting atom mappings will facilitate MFA and graph‐theoretic structural analyses with large‐scale metabolic networks.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

Streaming Data in HPC Workflows Using ADIOS

The “IO Wall” problem, in which the gap between computation rate and data access rate grows continuously, poses significant problems to scientific workflows which have traditionally relied upon using the filesystem for intermediate storage between workflow stages. One way to avoid this problem in scientific workflows is to stream data directly from producers to consumers and avoiding storage entirely. However, the manner in which this is accomplished is key to both performance and usability. This paper presents the Sustainable Staging Transport, an approach which allows direct streaming between traditional file writers and readers with few application changes. SST is an ADIOS “engine”, accessible via standard ADIOS APIs, and because ADIOS allows engines to be chosen at run-time, many existing file-oriented ADIOS workflows can utilize SST for direct application-to-application communication without any source code changes. This paper describes the design of SST and presents performance results from various applications that use SST, for feeding model training with simulation data with substantially higher bandwidth than the theoretical limits of Frontier’s file system, for strong coupling of separately developed applications for multiphysics multiscale simulation, or for in situ analysis and visualization of data to complete all data processing shortly after the simulation finishes.

Podhorszki, Norbert [ORNL] (ORCID:000000019647542X↗
DOE OSTI2025

The (R)evolution of Scientific Workflows in the Agentic AI Era: Towards Autonomous Science

Modern scientific discovery increasingly requires coordinating distributed facilities and heterogeneous resources, forcing researchers to act as manual workflow coordinators rather than scientists. Advances in AI leading to AI agents show exciting new opportunities that can accelerate scientific discovery by providing intelligence as a component in the ecosystem. However, it is unclear how this new capability would materialize and integrate in the real world. To address this, we propose a conceptual framework where workflows evolve along two dimensions which are intelligence (from static to intelligent) and composition (from single to swarm) to chart an evolutionary path from current workflow management systems to fully autonomous scientific laboratories. With these trajectories in mind, we present an architectural blueprint that can help the community take the next steps towards harnessing the opportunities in autonomous science with the potential for 100x discovery acceleration and transformational scientific workflows.

Shin, Woong [ORNL] (ORCID:0000000172077814)↗
DOE OSTI2023

Improv Dynamic Workflows

A workflow is a series of dependent computations which are executed to yield an experimental result, much like sheet music describes a musical performance. Dynamic workflows are like improvisational jazz, in which the musicians create unique, situation-driven, collaborative variations on themes. Dynamic workflows can potentially yield better results, save computation, and/or save human decision-making effort versus pre-specified or manual workflows, as they need not run the exhaustive set of studies which might be required in a pre-specified set, and they make their own experimental design decisions at run time. If you have an existing Maestro study where you would like to save computation and/or human time, Improv is for you. Improv provides core capabilities for creating a simple, hierarchical structure over Maestro studies, allowing you to connect ``experimental'' Maestro strudies with decision-making code to select parameters and run corresponding studies.

Goforth, John↗
DOE OSTI2022

Sim2Ls: FAIR simulation workflows and data

Just like the scientific data they generate, simulation workflows for research should be findable, accessible, interoperable, and reusable (FAIR). However, while significant progress has been made towards FAIR data, the majority of science and engineering workflows used in research remain poorly documented and often unavailable, involving ad hoc scripts and manual steps, hindering reproducibility and stifling progress. We introduce Sim2Ls (pronounced simtools) and the Sim2L Python library that allow developers to create and share end-to-end computational workflows with well-defined and verified inputs and outputs. The Sim2L library makes Sim2Ls , their requirements, and their services discoverable, verifies inputs and outputs, and automatically stores results in a globally-accessible simulation cache and results database. This simulation ecosystem is available in nanoHUB, an open platform that also provides publication services for Sim2Ls , a computational environment for developers and users, and the hardware to execute runs and store results at no cost. We exemplify the use of Sim2Ls using two applications and discuss best practices towards FAIR simulation workflows and associated data.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2021

Development of Time Lapse VSP Integration Workflow: A Case Study at Farnsworth CO2-EOR Project

Abstract This study aims to develop a 4D Vertical Seismic Profile (VSP) integration workflow to improve the prediction of subsurface stress changes. The selected study site is a 5-spot pattern within the ongoing CO2-EOR operations at the Farnsworth Field Unit FWU in Ochiltree County, Texas. The specific pattern has undergone extensive geological and geomechanical characterization through the acquisition of 3D seismic data, geophysical well logs, and core. This workflow constrains a numerical hydromechanical model by applying a penalty function formed between "modeled" versus "observed" time-lapse compressional and shear seismic velocity changes. Analyses of geophysical logs and ultra-sonic measurements on core exhibit measurable sensitivities to changes in both fluid saturation and mean effective stress. These data are used to develop a site-specific rock physics model and stress-velocity relationship, which inform the numerical models used to generate the "modeled" portion of the penalty function. The "observed" portion of the penalty function is provided by a novel elastic full-waveform inversion of the available 3D baseline and three monitor surveys to produce high-quality estimates of time-lapse compressional and shear seismic velocity changes. The modeling workflow accounts sequentially for fluid substitution and stress impacts. Hydrodynamic and geomechanical properties of the 3D coupled numerical model are estimated through geostatistical integration of well log and core data with 3D seismic inversion products. Changes in seismic velocities due to fluid substitution are computed using the Biot-Gassmann workflow and site-specific rock physics. Stress impacts on time-lapse seismic velocity changes are modeled from the effective stress output of the hydromechanical model and are initially based on the velocity versus effective stress relationship extracted from core mechanical testing. Based on the principle of superposition of seismic wavefields, seismic velocity changes attributed to fluid substitution and that due to changes in mean effective stress are treated as linearly additive. The modeled results are upscaled using Backus averaging to reconcile scale discrepancies between the modeled and measured datasets to formulate the penalty function. This manuscript presents the forward modeling process and concludes that for the base case, the seismic velocity changes due to mean effective stress dominates over the seismic velocity changes attributed to fluid substitution because of the extensive range of the pressure perturbations. Successful minimization of this penalty function calibrates the coupled hydrodynamic geomechanical numerical model and affirms the suitability of acoustic time-lapse measurements such as 4D-VSP for geomechanical calibration.

02 PETROLEUM↗
DOE OSTI2021

Execute BEE workflows on private cloud infrastructure (STNS01-22 BEE - FY21 P6-2)

Scope and objectives: BEE provides a portable, modular, HPC-focused workflow engine capable of managing containerized applications at scale. In FY21 BEE will expand its capabilities to provide more sophisticated handling of workflows. The ability to archive, clone, and re-run workflows will be added to BEE. The kinds of resources that BEE can use to execute workflow tasks will be expanded to include public and private clouds, such as Google Cloud Platform and OpenStack.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Applying 3D Geologic Modeling Workflows to the Argillite Reference Case (Rev. 1)

The objective of this short report is to document the application of our 3D geologic modeling workflow to an argillite (shale) host rock. Over the past four years, our team at Los Alamos National Laboratory has developed a geologic modeling workflow that can be applied to generic alluvial basins such as those found in the western United States. In “frontier” or “exploratory” basins where data are sparse, the first steps are to collect, evaluate and integrate available subsurface data into conceptual geologic models. Those models form the basis for constructing the geologic framework model, a 3D geocellular model ideally constrained by seismic and borehole data. To date we have constructed our models using “synthetic” well data derived from conceptual models, without the prospect of validating our workflow using “real” subsurface data. We were tasked to investigate whether our workflow designed for alluvial basin sediments could be applied to other potential repository host rocks. This task also provided the opportunity to work with high-quality subsurface data collected specifically for siting and evaluating a nuclear waste repository. Nagra, the Swiss governmental agency responsible for the disposal of the nation’s radioactive waste, generously provided us with data from two deep boreholes drilled through their argillaceous target formation. The aim of our proof-of-concept demonstration is to evaluate whether geostatistical methods offer a viable approach to property modeling in argillaceous rocks. Nagra provided us with the well data on the condition that we maintain confidentiality with all transferred information and results. Fortunately, Nagra posts numerous technical reports on its public website that describe the subsurface geology in great detail. All of the information and illustrations in this report related to the Swiss repository enterprise are taken from the Nagra public website.

58 GEOSCIENCES↗
DOE OSTI2024

A Complete Machine-Learning-Based Workflow to Illuminate Earthquake Processes

Under this grant we developed, tested, and made available, machine learning models to improve the tasks in the earthquake monitoring workflow (Figure 1). We implemented these models as a part of an end-to-end workflow for seismic network processing and demonstrated, in a variety of settings, that these methods generalize and that they result in dramatically more comprehensive earthquake catalogs. These catalogs illuminate earthquake processes in detail and to an extent that had previously not been possible, and they do so for both tectonic seismicity and seismicity induced by fluid injection related to unconventional hydrocarbon development. This report summarizes the results from the 15 publications that resulted from this grant. Those contributes are divided into: (1) the development of specific tasks related to monitoring (7 publications), (2) the organization of those tasks into workflows for seismic monitoring (2 publications), and (3) applications of those workflows to data (6 publications).

58 GEOSCIENCES↗