SEARCH · Engineering Papers

Results for “Workflow management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 289 records · Page 16

All sources NASA NTRS DOE OSTI

NASA NTRS2004

ScyFlow: An Environment for the Visual Specification and Execution of Scientific Workflows

With the advent of grid technologies, scientists and engineers are building more and more complex applications to utilize distributed grid resources. The core grid services provide a path for accessing and utilizing these resources in a secure and seamless fashion. However what the scientists need is an environment that will allow them to specify their application runs at a high organizational level, and then support efficient execution across any given set or sets of resources. We have been designing and implementing ScyFlow, a dual-interface architecture (both GUT and APT) that addresses this problem. The scientist/user specifies the application tasks along with the necessary control and data flow, and monitors and manages the execution of the resulting workflow across the distributed resources. In this paper, we utilize two scenarios to provide the details of the two modules of the project, the visual editor and the runtime workflow engine.

McCann, Karen M.↗

DOE OSTI2020

Development of tools for coupled InSAR and Seismicity monitoring of EGS reservoir development and management (Final Report)

The report details a series of software tools and workflows to analyze surface deformation data obtained from InSAR, deformation evidence by earthquake, geological constraints, and reservoir management to define and monitor the geothermal reservoir.

15 GEOTHERMAL ENERGY↗

DOE OSTI2020

Automated and Distributed Monte Carlo Generation for GlueX

MCwrapper is a set of systems that manages the entire Monte Carlo production workflow for GlueX and provides standards for how that Monte Carlo is produced. MCwrapper was designed to be able to utilize a variety of batch systems in a way that is relatively transparent to the user, thus enabling users to quickly and easily produce valid simulated data at home institutions worldwide. Additionally, MCwrapper supports an autonomous system that takes user’s project submissions via a custom web application. The system then atomizes the project into individual jobs, matches these jobs to resources, and monitors the jobs status. The entire system is managed by a database which tracks almost all facets of the systems from user submissions to the individual jobs themselves. Users can interact with their submitted projects online via a dashboard or, in the case of testing failure, can modify their project requests from a link contained in an automated email. Beginning in 2018 the GlueX Collaboration began to utilize the Open Science Grid (OSG) to handle a bulk of simulation tasks; these tasks are currently being performed on the OSG automatically via MCwrapper. This talk will outline the entire system of MCwrapper, its use cases, and the unique challenges facing the system.

Britton, Thomas↗

DOE OSTI2024

Using stable isotopes to inform water resource management in forested and agricultural ecosystems

Present and future climatic trends are expected to markedly alter water fluxes and stores in the hydrologic cycle. In addition, water demand continues to grow due to increased human use and a growing population. Sustainably managing water resources requires a thorough understanding of water storage and flow in natural, agricultural, and urban ecosystems. Measurements of stable isotopes of water (hydrogen and oxygen) in the water cycle (atmosphere, soils, plants, surface water, and groundwater) can provide information on the transport pathways, sourcing, dynamics, ages, and storage pools of water that is difficult to obtain with other techniques. However, the potential of these techniques for practical questions has not been fully exploited yet. Here, we outline the benefits and limitations of potential applications of stable isotope methods useful to water managers, farmers, and other stakeholders. We also describe several case studies demonstrating how stable isotopes of water can support water management decision-making. Finally, we propose a workflow that guides users through a sequence of decisions required to apply stable isotope methods to examples of water management issues. We call for ongoing dialogue and a stronger connection between water management stakeholders and water stable isotope practitioners to identify the most pressing issues and develop best-practice guidelines to apply these techniques.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2022

Position Papers for the ASCR Workshop on the Management and Storage of Scientific Data

The purpose of this workshop is to identify priority research directions in the area of data management for high-performance and scientific computing above and beyond HPC’s traditional "the parallel file system is the data-management system" model. Supporting the breadth of the DOE mission, including the explosion of AI uses and the growing needs of experimental and observational science, motivates revisiting our assumptions about data management. There are many facets of this topic to explore including: (1) Interfaces for accessing data that resides on traditional persistent storage as well as memory devices; (2) Storage-system architecture design that supports scientific workflows on varied hierarchical storage and networking devices; (3) Devising metadata management infrastructure to support FAIR principles (Findability, Accessibility, Interoperability, and Reusability); (4) Capturing provenance information about scientific data; (5) Utilizing AI to learn I/O patterns of emerging workloads for efficient data management; (6) Providing data management support for AI and complex workflows; and (7) Understanding the overlap between traditional storage systems and I/O (SSIO) efforts and data management. While the program committee has identified these topics as important areas for discussion, we welcome position papers from the community that propose additional topics of interest for discussion at the workshop. The workshop agenda will include breakout sessions for discussing these and selected topic areas to inform priority research directions for data management for high-performance and scientific computing.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

Report for the ASCR Workshop on the Management and Storage of Scientific Data

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2019

Testing the efficacy of active pressure management as a tool to mitigate induced seismicity

Manipulation of subsurface stresses through fluid injection/extraction may induce seismicity through modulation of overpressure along faults. Globally, most industrial operations do not induce significant earthquakes. However, the possibility of inducing moderate to large earthquakes is recognized as a significant risk for basin-scale carbon dioxide (CO 2 ) storage operations (Niass et al., 2017). One possible method to reduce the occurrence of induced seismicity is to maintain only small (or zero) fluid overpressure changes in the subsurface through extraction of formation brine. For active pressure management (APM), operators may employ different procedures including extraction of various brine volumes and different configurations of injection/extraction well locations. However, the impact each these operational procedures on induced seismicity must be understood in order to develop management strategies. Here, we describe a numerical modeling workflow to evaluate the efficacy of APM techniques, that when applied to a specific site may be used to inform operational management strategies. We couple the multiphase reservoir model, NUFT, with the earthquake simulator, RSQSim and simulate seismicity induced by CO 2 injection into a hypothetical brine saturated reservoir. Reservoir overpressure is modified by co-extraction of brine at various fractions of the injected CO 2 volume from five potential wells. Our goal is to understand how the well-pair configuration and the extracted brine volume impact the total number and the maximum event magnitude. Additionally, the performance of each scenario is also evaluated in terms of the volume of CO 2 that can be stored before an earthquake above a tolerance threshold is induced.

58 GEOSCIENCES↗

DOE OSTI2022

Echo: Data and Analysis Management [Slides]

Echo software capabilities are described. Echo provides a comprehensive suite of tools for data wrangling, management, and analysis. It is an object-oriented approach to data analysis designed for robust and scalable execution of complex analysis workflows written in MATLAB.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2014

Nationwide Buildings Energy Research enabled through an integrated Data Intensive Scientific Workflow and Advanced Analysis Environment

Modern workflow systems enable scientists to run ensemble simulations at unprecedented scales and levels of complexity, allowing them to study system sizes previously impossible to achieve, due to the inherent resource requirements needed for the modeling work. However as a result of these new capabilities the science teams suddenly also face unprecedented data volumes that they are unable to analyze with their existing tools and methodologies in a timely fashion. In this paper we will describe the ongoing development work to create an integrated data intensive scientific workflow and analysis environment that offers researchers the ability to easily create and execute complex simulation studies and provides them with different scalable methods to analyze the resulting data volumes. The integration of simulation and analysis environments is hereby not only a question of ease of use, but supports fundamental functions in the correlated analysis of simulation input, execution details and derived results for multi-variant, complex studies. To this end the team extended and integrated the existing capabilities of the Velo data management and analysis infrastructure, the MeDICi data intensive workflow system and RHIPE the R for Hadoop version of the well-known statistics package, as well as developing a new visual analytics interface for the result exploitation by multi-domain users. The capabilities of the new environment are demonstrated on a use case that focusses on the Pacific Northwest National Laboratory (PNNL) building energy team, showing how they were able to take their previously local scale simulations to a nationwide level by utilizing data intensive computing techniques not only for their modeling work, but also for the subsequent analysis of their modeling results. As part of the PNNL research initiative PRIMA (Platform for Regional Integrated Modeling and Analysis) the team performed an initial 3 year study of building energy demands for the US Eastern Interconnect domain, which they are now planning to extend to predict the demand for the complete century. The initial study raised their data demands from a few GBs to 400GB for the 3year study and expected tens of TBs for the full century.

99 GENERAL AND MISCELLANEOUS↗

DOE OSTI2023

Integrated End-to-end Performance Prediction and Diagnosis for Extreme Scientific Workflows

This report details recent progress for the ASCR funded project “Integrated End-to-end Performance Prediction and Diagnosis for Extreme Scientific Workflows”. We refer to the project as IPPD/2, reflecting the 2017 renewal under expanded scope and partners In IPPD/2, we increased our research scope to include data motion. We are focusing on three major aspects: a) observe how data is generated, distributed, and used; b) analyze how data is (repeatedly) consumed with a focus both on repeated patterns and anomalies; and c) explore how to optimize data motion. This new work on data motion will augment and complement IPPD/2’s research that focused on the computational aspects of tasks. We leverage and extend our existing tools and demonstrate our work on the Belle II workflow suite as well as on workflows from NSLS-II. The highlights of our work are as follows: Provenance for Workflows: Provenance is used to provide information enabling quality control, re-run computational workflows, and reproduce results. IPPD/2 has been building a scalable provenance management system that enables the capture of provenance from the high-level workflow through all relevant system levels in one integrated environment. Leveraging this work, our recent efforts have included using provenance as an enabling technique. Workload characterization: Leveraging provenance and analysis, we characterize data movement within network, storage, and memory over a variety of workloads. This characterization enables an understanding by performance analysts and application developers of the range of behaviors that could be expected. Performance Prediction for Workflows: The goal of modeling distributed workflows is to understand performance bottlenecks and enable more intelligent task scheduling to optimize selected metrics of interest (e.g., task throughput or output data rate). IPPD/2 has utilized both analytical and AI/ML modeling methodologies for performance modeling. Advanced Scheduling and Fault Modeling for Workflows: Scheduling of large-scale scientific workflows on geographically distributed resources is a challenging problem. To improve workflow throughput, we combined novel scheduling algorithms with task predictions from performance modeling and fault modeling. Dynamically Alleviating Bottlenecks in Workflows: Exploiting our provenance, analysis, and modeling efforts, we have explored and developed several techniques for dynamically detecting and alleviating bottlenecks in data movement. In particular, we have spent considerable effort demonstrating our techniques on production-like workflow configurations.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

Active learning path-dependent properties using a cloud-based materials acceleration platform

Solid state materials are central to many modern technologies in which a given material may be exposed to a variety of environments. The material properties often vary with the sequence of environments in an irreversible manner, resulting in a quintessential path-dependency in experimental observables. While sequential learning techniques have been effectively deployed for accelerating learning of state properties of materials, they often use a consistent environment path in all experiments. To elevate such techniques for making optimal decisions in experimental investigations of path-dependent properties, we introduce an iterated expected information gain acquisition function that optimizes over entire experimental trajectories. This approach is implemented within a cloud-based Materials Acceleration Platform architecture utilizing an event-driven stateful broker coupled with remote HELAO (Hierarchical Experimental Laboratory Automation and Orchestration) instances and an AI science manager. The platform's efficacy was demonstrated through a case study optimizing multi-step spectro-electrochemical experiments to identify optically stable potential windows in (Co–Ni–Sb)O z metal oxides. The system successfully integrated AI-driven experiment design, remote laboratory automation, and cloud-based data infrastructure, validating the platform's capability for managing complex, adaptive, path-dependent workflows in materials discovery.

Guevarra, Dan [California Institute of Technology ↗

DOE OSTI2014

SU-E-J-129: A Strategy to Consolidate the Image Database of a VERO Unit Into a Radiotherapy Management System

Purpose: To present a strategy to integrate the imaging database of a VERO unit with a treatment management system (TMS) to improve clinical workflow and consolidate image data to facilitate clinical quality control and documentation. Methods: A VERO unit is equipped with both kV and MV imaging capabilities for IGRT treatments. It has its own imaging database behind a firewall. It has been a challenge to transfer images on this unit to a TMS in a radiation therapy clinic so that registered images can be reviewed remotely with an approval or rejection record. In this study, a software system, iPump-VERO, was developed to connect VERO and a TMS in our clinic. The patient database folder on the VERO unit was mapped to a read-only folder on a file server outside VERO firewall. The application runs on a regular computer with the read access to the patient database folder. It finds the latest registered images and fuses them in one of six predefined patterns before sends them via DICOM connection to the TMS. The residual image registration errors will be overlaid on the fused image to facilitate image review. Results: The fused images of either registered kV planar images or CBCT images are fully DICOM compatible. A sentinel module is built to sense new registered images with negligible computing resources from the VERO ExacTrac imaging computer. It takes a few seconds to fuse registered images and send them to the TMS. The whole process is automated without any human intervention. Conclusion: Transferring images in DICOM connection is the easiest way to consolidate images of various sources in your TMS. Technically the attending does not have to go to the VERO treatment console to review image registration prior delivery. It is a useful tool for a busy clinic with a VERO unit.

60 APPLIED LIFE SCIENCES↗

NASA NTRS2005

Software to Manage the Unmanageable

In 1995, NASA s Jet Propulsion Laboratory (JPL) contracted Redmond, Washington-based Lucidoc Corporation, to design a technology infrastructure to automate the intersection between policy management and operations management with advanced software that automates document workflow, document status, and uniformity of document layout. JPL had very specific parameters for the software. It expected to store and catalog over 8,000 technical and procedural documents integrated with hundreds of processes. The project ended in 2000, but NASA still uses the resulting highly secure document management system, and Lucidoc has managed to help other organizations, large and small, with integrating document flow and operations management to ensure a compliance-ready culture.

Source record↗

DOE OSTI2021

FY2021 Derivative Datasets LDRD Report

In this pilot project, the feasibility around the creation, management, and deployment of derivative datasets as a potential new class of intellectual property was explored. The objective of these derivative datasets was to address environmental, industrial, and commercial challenges in a novel way that could provide additional impact based on the research performed at PNNL. As a result of efforts in FY2021, two new derivative datasets were created using newly created code that would enable the facile creation of additional derivative datasets of varying complexity and integration from pre-existing and published originating datasets. In addition, the process workflows were documented, detailing how the creation, management, and deployment of the derivative datasets could integrate with existing processes within PNNL. Finally, exploratory interviews with industry allowed for the scoping of future collaboration and commercialization opportunities around derivative datasets.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2018

CANDLE/Supervisor: a workflow framework for machine learning applied to cancer research

BackgroundCurrent multi-petaflop supercomputers are powerful systems, but present challenges when faced with problems requiring large machine learning workflows. Complex algorithms running at system scale, often with different patterns that require disparate software packages and complex data flows cause difficulties in assembling and managing large experiments on these machines.ResultsThis paper presents a workflow system that makes progress on scaling machine learning ensembles, specifically in this first release, ensembles of deep neural networks that address problems in cancer research across the atomistic, molecular and population scales. The initial release of the application framework that we call CANDLE/Supervisor addresses the problem of hyper-parameter exploration of deep neural networks.ConclusionsInitial results demonstrating CANDLE on DOE systems at ORNL, ANL and NERSC (Titan, Theta and Cori, respectively) demonstrate both scaling and multi-platform execution.

60 APPLIED LIFE SCIENCES↗

DOE OSTI2024

GDSA framework, a computational framework for complex modeling problems in radioactive waste management

This paper details a computational framework to produce automated, graphical workflows, and how this framework can be deployed to support complex modeling problems like those in nuclear engineering. Key benefits of the framework include: automating previously manual workflows; intuitive construction and communication of workflows through a graphical interface; and automated file transfer and handling for workflows deployed across heterogeneous computing resources. This paper demonstrates the framework's application to probabilistic post-closure performance assessment of systems for deep geologic disposal of nuclear waste. However, the framework is a general capability that can help users running a variety of computational studies.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

DOE OSTI2026

iDDS: intelligent distributed dispatch and scheduling for workflow orchestration

The intelligent distributed dispatch and scheduling (iDDS) service is a versatile workflow orchestration system designed for large-scale, distributed scientific computing. iDDS extends traditional workload and data management by integrating data-aware execution, conditional logic, and programmable workflows, enabling automation of complex and dynamic processing pipelines. Originally developed for the ATLAS experiment at the large hadron collider, iDDS has evolved into an experiment-agnostic platform that supports both template-driven workflows and a Function-as-a-Task model for Python-based orchestration. This paper presents the architecture and core components of iDDS, highlighting its scalability, modular message-driven design, and integration with systems such as PanDA and Rucio. We demonstrate its versatility through real-world use cases: fine-grained tape resource optimization for ATLAS, orchestration of large Directed Acyclic Graph (DAG) workflows for the Rubin Observatory, distributed hyperparameter optimization for machine learning applications, active learning for physics analyses, and AI-assisted detector design at the electron–ion collider. By unifying workload scheduling, data movement, and adaptive decision-making, iDDS reduces operational overhead and enables reproducible, high-throughput workflows across heterogeneous infrastructures. We conclude with current challenges and future directions, including interactive, cloud-native, and serverless workflow support.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

Phase II Field Demonstration at Lansing Smith Generating Plant, Southport, Florida (Final Report)

The Final Technical Report: Field Test Design and Pressure Management Strategies for Phase II Field Demonstration of Optimal Pressure Control, Plume Management, and Produced Water Strategies presents the culmination of multi-year efforts under the U.S. Department of Energy’s Brine Extraction and Storage Test (BEST) program, led by EPRI in partnership with Gulf Power at the Plant Smith site near Panama City, Florida. The project was designed to evaluate and demonstrate the technical feasibility of managing subsurface pressures and fluid movement associated with large-scale CO₂ injection, using low-salinity water as a proxy fluid. Through a combination of field injection testing, reservoir modeling, and optimization studies, the research team developed and refined active and passive brine extraction strategies aimed at controlling injection-induced pressure buildup, mitigating risks of fault activation, and managing plume migration. The field demonstration incorporated a new injection well (TIW-2), a new monitoring/extraction well (TEMW-A), and an existing well (TIW-1) repurposed for passive pressure relief. Complementary geophysical monitoring was designed to track plume development and evaluate the effectiveness of pressure management strategies. The report details the integrated workflow encompassing wellfield development, regulatory permitting, model calibration, and survey design. It includes a comprehensive electromagnetic (EM) modeling and inversion study used to develop a cost-effective, time-lapse geophysical monitoring plan capable of imaging the evolving low-salinity plume within the highly saline Lower Tuscaloosa formation. Reservoir simulation results guided the design of the 17-month injection program and the timing of active extraction to maintain formation pressures below a hypothetical fault reactivation threshold. Supporting analyses evaluated potential injectivity risks related to clay fines migration and geochemical incompatibility, identifying practical mitigation measures such as salinity and pH control. The final design recommends a series of crosswell EM surveys—conducted before, during, and after injection—as the most effective and economical approach for plume imaging, supplemented by continuous downhole pressure and flow monitoring. Collectively, the study provides a field-ready framework for cost-effective pressure management and monitoring in support of future CO₂ storage projects.

01 COAL, LIGNITE, AND PEAT↗