Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Workflow management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 433 records · Page 24
All sourcesNASA NTRSDOE OSTI
DOE OSTI2013

SciDAC-Center for Plasma Edge Simulation Report

The Common Component Architecture (CCA) effort is the embodiment of a long-range program of research and development into the formulation, roles, and use of component technologies in high-performance scientific computing. CCA components can interoperate with other components in a variety of frameworks, including SCIRun2 from the University of Utah. The SCIRun2 framework is also developing the ability to connect components from a variety of different models through a mechanism called meta-components. The meta component model operates by providing a plugin architecture for component models. Abstract components are manipulated and managed by the SCIRun2 framework, while concrete component models perform the actual work and communicate with each other directly. We will leverage the SCIRun2 framework and the Kepler system to orchestrate components in the Fusion Simulation Project (FSP) and to provide a CCA-based interface with Kepler. The groundwork for this functionality is being performed with the Scientific Data Management center. The SDM center is developing CCA-compliant interfaces for expressing and executing workflows and create workflow components based on SCIRun and Ptolemy (Kepler) execution engines, including development of uniform interfaces for selecting, starting, and monitoring scientific workflows. Accomplishments include Introduction to CCA and Simulation Software Systems, Introduction into SCIRun2 and Bridging within SCIRun2, CCALoop: A scalable design for a distributed component framework, and Combining Workflow methodologies with Component Architectures.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2016

Automated metadata--final project report

This report summarizes the work of the Automated Metadata, Provenance Cataloging, and Navigable Interfaces: Ensuring the Usefulness of Extreme-Scale Data Project (MPO Project) funded by the United States Department of Energy (DOE), Offices of Advanced Scientific Computing Research and Fusion Energy Sciences. Initially funded for three years starting in 2012, it was extended for 6 months with additional funding. The project was a collaboration between scientists at General Atomics, Lawrence Berkley National Laboratory (LBNL), and Massachusetts Institute of Technology (MIT). The group leveraged existing computer science technology where possible, and extended or created new capabilities where required. The MPO project was able to successfully create a suite of software tools that can be used by a scientific community to automatically document their scientific workflows. These tools were integrated into workflows for fusion energy and climate research illustrating the general applicability of the project’s toolkit. Feedback was very positive on the project’s toolkit and the value of such automatic workflow documentation to the scientific endeavor.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2026

PV Operations Software Transparency: A PVMAC Industry Snapshot

The rapid growth of photovoltaic (PV) deployment has increased reliance on software platforms for monitoring, workflow automation, diagnostics, and performance analytics. As these tools play a central role in asset management and operations and maintenance (O&M), greater transparency in methodologies, data handling, and validation practices benefits the broader PV ecosystem. To better understand current practices and identify opportunities for improved clarity and interoperability, 24 software providers contributed detailed responses through the PV O&M Analytics Collaborative (PVMAC) initiative, the first structured questionnaire of its kind in the industry, covering onboarding, interoperability, data quality, diagnostics, AI/ML, and other operational categories. These providers represent over 1.1 TW of solar assets under management. The analysis shows broad adoption of digital twins, AI/ML, and API integrations, but also highlights challenges in onboarding processes, inconsistent definitions and methodologies, variability in key performance indicator (KPI) calculations, and limited independent validation. Greater standardization, clearer documentation, and stronger validation frameworks could improve transparency, comparability, and trust across PV operations software platforms.

14 SOLAR ENERGY↗
DOE OSTI2017

Lightweight Provenance Service for High-Performance Computing

Provenance describes detailed information about the history of a piece of data, containing the relationships among elements such as users, processes, jobs, and workflows that contribute to the existence of data. Provenance is key to supporting many data management functionalities that are increasingly important in operations such as identifying data sources, parameters, or assumptions behind a given result; auditing data usage; or understanding details about how inputs are transformed into outputs. Despite its importance, however, provenance support is largely underdeveloped in highly parallel architectures and systems. One major challenge is the demanding requirements of providing provenance service in situ. The need to remain lightweight and to be always on often conflicts with the need to be transparent and offer an accurate catalog of details regarding the applications and systems. To tackle this challenge, we introduce a lightweight provenance service, called LPS, for high-performance computing (HPC) systems. LPS leverages a kernel instrument mechanism to achieve transparency and introduces representative execution and flexible granularity to capture comprehensive provenance with controllable overhead. Extensive evaluations and use cases have confirmed its efficiency and usability. We believe that LPS can be integrated into current and future HPC systems to support a variety of data management needs.

Dai, Dong↗
DOE OSTI2024

AI-assisted detector design for the EIC (AID(2)E)

Artificial Intelligence is poised to transform the design of complex, large-scale detectors like ePIC at the future Electron Ion Collider. Featuring a central detector with additional detecting systems in the far forward and far backward regions, the ePIC experiment incorporates numerous design parameters and objectives, including performance, physics reach, and cost, constrained by mechanical and geometric limits. This project aims to develop a scalable, distributed AI-assisted detector design for the EIC (AID(2)E), employing state-of-the-art multiobjective optimization to tackle complex designs. Supported by the ePIC software stack and using G EANT 4 simulations, our approach benefits from transparent parameterization and advanced AI features. The workflow leverages the PanDA and iDDS systems, used in major experiments such as ATLAS at CERN LHC, the Rubin Observatory, and sPHENIX at RHIC, to manage the compute intensive demands of ePIC detector simulations. Tailored enhancements to the PanDA system focus on usability, scalability, automation, and monitoring. Ultimately, this project aims to establish a robust design capability, apply a distributed AI-assisted workflow to the ePIC detector, and extend its applications to the design of the second detector (Detector-2) in the EIC, as well as to calibration and alignment tasks. Additionally, we are developing advanced data science tools to efficiently navigate the complex, multidimensional trade-offs identified through this optimization process.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2013

Implementations of a Flexible Framework for Managing Geologic Sequestration Modeling Projects

Numerical simulation is a standard practice used to support designing, operating, and monitoring CO 2 injection projects. Although a variety of computational tools have been developed that support the numerical simulation process, many are single-purpose or platform specific and have a prescribed workflow that may or may not be suitable for a particular project. We are developing an open-source, flexible framework named Velo that provides a knowledge management infrastructure and tools to support modeling and simulation for various types of projects in a number of scientific domains. The Geologic Sequestration Software Suite (GS3) is a version of this framework with features and tools specifically tailored for geologic sequestration studies. Because of its general nature, GS3 is being employed in a variety of ways on projects with differing goals. GS3 is being used to support the Sim-SEQ international model comparison study, by providing a collaborative framework for the modeling teams and providing tools for model comparison. Another customized deployment of GS3 has been made to support the permit application process. In this case, GS3 is being used to manage data in support of conceptual model development and provide documentation and provenance for numerical simulations. An additional customized deployment of GS3 is being created for use by the United States Environmental Protection Agency (US-EPA) to aid in the CO2 injection permit application review process in one of its regions. These use cases demonstrate GS3’s flexibility, utility, and broad applicability

58 GEOSCIENCES↗
DOE OSTI2014

A Scientific Data Processing Framework for Time Series NetCDF Data

ARM Data Integrator (ADI) is a framework to streamline the development of scientific algorithms that analyze time-series NetCDF data, and to improve the content and consistency of the output data products produced by these algorithms. ADI achieves these goals by automating the process of retrieving and preparing data for analysis, supporting the definition of output data products through a graphical interface, and providing a modular, flexible software development architecture. The input data, preprocessing, and output data specifications are defined through a graphical interface and stored in a database. ADI also includes a workflow for data integration, a library of software modules to support the workflow, and a source code generator that produces C, IDL and Python templates. Data preparation support includes automated retrieval of data from input files, merging the retrieved data into appropriately sized chunks, and transformation of the data onto a common coordinate system grid. Through the graphical interface, users can view the details of both their data products and those in the ARM catalog. The variable and attribute definitions of the existing data products can be used to build new output data products. In addition, the rules that make up the ARM archive’s data standards are laid on top of the view of the new data product providing the user with a visual cue indicating where their output violates an archive standard. The necessary configurations are stored in a database that is accessed by the ADI libraries. This paper discusses the ADI framework, its supporting components, and how ADI can significantly decrease the time and cost of implementing scientific algorithms while improving the ability of scientists to disseminate their results.

Atmospheric Science↗
NASA NTRS2006

Knowledge management in the engineering design environment

The Aerospace and Defense industry is experiencing an increasing loss of knowledge through workforce reductions associated with business consolidation and retirement of senior personnel. Significant effort is being placed on process definition as part of ISO certification and, more recently, CMMI certification. The process knowledge in these efforts represents the simplest of engineering knowledge and many organizations are trying to get senior engineers to write more significant guidelines, best practices and design manuals. A new generation of design software, known as Product Lifecycle Management systems, has many mechanisms for capturing and deploying a wider variety of engineering knowledge than simple process definitions. These hold the promise of significant improvements through reuse of prior designs, codification of practices in workflows, and placement of detailed how-tos at the point of application.

designs↗
NASA NTRS2019

EARTHDATA PUB: A Data Publication Workflow Solution for NASA’s EOSDIS

Each NASA Distributed Active Archive Center (DAAC) faces the challenge of dealing with an increasingly diverse number of publishable data products from diverse data producers. Data producers, on the other hand, may experience pain points when interacting with the EOSDIS for the first time or when publishing different data at different DAACs. As a result, there has been a growing need to develop a common software framework that serves as a common interface for data producers, rigorously defines the data publication procedure for DAAC staff, facilitates the management of various data publication processes, and tracks the progress of data publication. This software should also account for the different configurations at different DAACs. Currently, two primary data publication workflow and tracking tools exist in operation at the EOSDIS: Semi-Automated ingest System (SAuS) and Data Publication workflow Portal (DAPPeR). However, neither tool is cloud-ready. Automated data processing could be managed by Cumulus, an EOSDIS cloud-based data ingest, archive and management system. However, Cumulus does not support manual tasks or on-premise implementations. We propose to develop the Earthdata Publication Minimum Viable Product (Earthdata Pub MVP) -- a cloud-hosted solution that works with both cloud and on-premise systems and implements the communications and exchange requirements generated by the Earthdata Pub information architecture team.

Rice, Justin L.↗
DOE OSTI2026

An end-to-end workflow for executing a classically bootstrapped variational quantum algorithm on an academic quantum computer

Academic quantum computing platforms often face unique challenges in executing quantum workloads due to fragmented software environments and limited engineering support. Unlike commercial ecosystems, academic devices typically evolve without full-stack integration in mind, making it difficult to run complex applications—such as variational quantum algorithms (VQA)—reliably and efficiently. Issues such as incompatible software layers and lack of automated job management significantly increase the overhead of theory-experiment collaboration. To address these challenges, we develop a modular, end-to-end workflow that decouples application-layer code from low-level hardware control, automates circuit submission and result collection, and supports fine-grained circuit-level job scheduling and recovery. The architecture employs a dual-end application programming interface (API) design, enabling robust operation across unstable or resource-constrained hardware backends. For practical use, the framework is lightweight and user-friendly, allowing rapid prototyping of full-stack workflows using basic Python tools. We validate this workflow on a high-fidelity trapped-ion quantum computer by demonstrating a variational quantum eigensolver (VQE) experiment with a classically bootstrapped ansatz initialization technique. The system successfully executed over 60,000 circuits across multiple molecular test cases with minimal human intervention, highlighting the framework’s effectiveness in enabling reproducible, resilient quantum experimentation in academic settings.

Clifford↗
DOE OSTI2023

Not just for programmers: How GitHub can accelerate collaborative and reproducible research in ecology and evolution

Abstract Researchers in ecology and evolutionary biology are increasingly dependent on computational code to conduct research. Hence, the use of efficient methods to share, reproduce, and collaborate on code as well as document research is fundamental. GitHub is an online, cloud‐based service that can help researchers track, organize, discuss, share, and collaborate on software and other materials related to research production, including data, code for analyses, and protocols. Despite these benefits, the use of GitHub in ecology and evolution is not widespread. To help researchers in ecology and evolution adopt useful features from GitHub to improve their research workflows, we review 12 practical ways to use the platform. We outline features ranging from low to high technical difficulty, including storing code, managing projects, coding collaboratively, conducting peer review, writing a manuscript, and using automated and continuous integration to streamline analyses. Given that members of a research team may have different technical skills and responsibilities, we describe how the optimal use of GitHub features may vary among members of a research collaboration. As more ecologists and evolutionary biologists establish their workflows using GitHub, the field can continue to push the boundaries of collaborative, transparent, and open research.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2025

FY 2025 Multidimensional Data Correlation Platform: Unified Software Architecture for Advanced Materials and Manufacturing Technologies Data Management and Processing

The Advanced Materials and Manufacturing Technologies (AMMT) program continues to advance a data-driven approach to demonstrate the utility of additive manufacturing for fabricating components for nuclear applications. A key scientific goal is to leverage data to better understand manufacturing outcomes and thereby improve the performance, reliability, and lifespan of nuclear components. Ultimately, this effort supports the development of standards for certification and qualification of additively manufactured components, enabling broader industry adoption. In support of this objective, the AMMT program is building and deploying a data management platform to record, index, analyze, and make available the manufacturing data generated across the AMMT program. In FY 2023, the team conceptualized the architecture of the platform and, in FY 2024, deployed the first functional version at the Oak Ridge National Laboratory (ORNL) Manufacturing Demonstration Facility (MDF). In FY 2025, the platform was officially opened to all AMMT members. To enable this expansion, core modifications and enhancements were developed, including improvements to the user interface and workflows for data entry and retrieval. Most notably, robust security and access control mechanisms were implemented to protect data and manage information sharing. This effort featured a logging system, protected views, and controlled access mechanisms. This report documents these enhancements and the transition of the platform into program-wide use.

36 MATERIALS SCIENCE↗
DOE OSTI2023

Data-driven modeling to enhance municipal water demand estimates in response to dynamic climate conditions

Altered precipitation and temperature patterns from a changing climate will affect supply, demand, and overall municipal water system operations throughout the arid western U.S. While supply forecasts leverage hydrological models to connect climate influences with surface water availability, demand forecasts typically estimate water use independent of climate and other externalities. Stemming from an increased focus on seasonal water demand management, we use the Salt Lake City, Utah municipal water system as a test bed to assess model accuracy versus complexity trade-offs between simple climate-independent econometric-based models and complex climate-sensitive data-driven models to average to extreme wet and dry climate conditions—representative of a new climate normal. Here, the climate-independent model displayed low performance during extreme dry conditions with predictions exceeding 90% and 40% of the observed monthly and seasonal volumetric demands, respectively, which we attribute to insufficient model complexity. The climate-sensitive models displayed greater accuracy in all conditions, with an ordinary least squares model demonstrating a measurable reduction in prediction bias (3.4% vs. -27.3%) and RMSE (74.0 lpcd vs. 294 lpcd) compared to the climate-independent model. The climate-sensitive workflow increased model accuracy and characterized climate-demand interactions, demonstrating a novel tool to enhance water system management.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Scientific Data Management Beyond Traditional Computing Boundaries

Scientific data management is undergoing a fundamental transformation driven by the convergence of artificial intelligence (AI)/machine learning workflows, distributed computing and storage environments, and exponential data growth. Here, we analyze how these developments address current limitations while enabling new capabilities for cross-facility collaboration and AI-driven research.

Widener, Patrick [Oak Ridge National Laboratory (O↗
DOE OSTI2019

SDN-NGenIA, a Software Defined Next Generation Integrated Architecture for HEP and Data Intensive Science

This project aims to enable the LHC and other leading programs in high energy physics and other global science domains funded by the DOE to operate with a new level of efficiency and control, through the development of a Next Generation Integrated Architecture (NGenIA) based on intelligent software defined network (SDN)-driven network systems coupled to high throughput applications. While the initial focus is on the challenging LHC use case, the systems and products being developed are general, and apply to many fields of data intensive science ranging from astrophysical sky surveys to bioinformatics and earth observation, as well as other organizations facing the challenges of extracting knowledge from distributed multi-petabyte data stores. A central concept in this development program is a new paradigm ‘consistent network operations’ among widely distributed computing and storage facilities, where stable high throughput flows, at set rates, cross load-balanced network paths, up to flexible high water marks that are adjusted in real time to accommodate other network traffic. The large smooth flows are launched and managed by SDN services that act in concert with the experiments’ site-resident data distribution and management systems, to meet the expanding needs of the science programs. The technical goals include the construction of autonomous, intelligent site-resident services that interact dynamically with network-resident services, and with the science programs’ principal data distribution and management tools, to request or command network resources in support of high throughput petascale to exascale workflow Specific work items include: 1. Developing compact Data Transfer Nodes (DTNs) with auto-tuning functions that support data transfer rates in the 100 Gbps to the 1 Tbps range when used with high throughput data transfer applications 2. Deep site orchestration among virtualized clusters, storage subsystems and subnets to successfully coschedule CPU, storage and network resources 3. Science-program designed site architectures, operational modes, and policy and resource usage priorities, adjudicated across multiple network domains and virtual organizations, using the orchestration functions and methods 4. Seamlessly extending end-to-end operation across both extra-site and intra-site boundaries through the use of Open vSwitch (OVS), FireQoS + next generation Science DMZs 5. Novel methods of system integration that enable granular control of extreme scale long distance transfers through flow matching of scattered source-destination address pairs to multi-domain dynamic circuits 6. Funneling massive sets of streams to DTNs at the site edge hosting petascale buffer pools configured for flows of 100 Gbps and up, exploiting state of the art data transfers 7. Adaptive scheduling based on pervasive end-to-end monitoring, including DTN or compute-node resident agents providing comprehensive end-system profiling 8. Developing unsupervised and supervised machine learning and modeling methods to optimize the workflow involving terabyte to multi-petabyte datasets.

47 OTHER INSTRUMENTATION↗
DOE OSTI2025

Accelerating Resilience of the Community through Holistic Engagement and use of Renewables (ARCHER) Planning Framework

The primary objective of the Accelerating Resilience of the Community through Holistic Engagement and Use of Renewables (ARCHER) initiative was to identify and incorporate the unique variations in energy burden, social vulnerability, living conditions, and access to essential services that differ across communities. By accounting for these localized factors—down to the neighborhood level—the project supports more targeted and effective investments in community resilience. The framework seeks to establish practical planning guidance, methods, and performance measures for community energy resilience, integrate community-level and electric utility system resilience planning, and assess its effectiveness through comparison with conventional and operational planning approaches. A key component of the project was its data exchange platform, which is used to evaluate and demonstrate the tools, methodologies, and planning approaches developed through ARCHER. This open-source platform enables developers and vendors of distribution and outage management systems to build upon the research by incorporating its concepts into their own tools and workflows. This capability is enabled by the transparent availability of data, functional requirements, and the underlying information model. The project yielded several important insights. First, meaningful engagement with communities is essential to achieving comprehensive resilience outcomes. Second, resilience planning is most effective when electric grid considerations and broader community needs are addressed in a coordinated manner. Third, the use of platforms that allow for real-time input from communities can enhance utility responsiveness during restoration activities. Fourth, a structured and systematic planning approach can successfully translate ARCHER concepts into practice. Fifth, the development of an integrated metric that reflects both grid performance and community impacts provides a more holistic basis for evaluating resilience. Finally, incorporating community engagement and equity considerations into grid operations is critical, particularly during severe weather events that result in extended outages.

24 POWER TRANSMISSION AND DISTRIBUTION↗
NASA NTRS1996

Software Project Management and Measurement on the World-Wide-Web (WWW)

We briefly describe a system for forms-based, work-flow management that helps members of a software development team overcome geographical barriers to collaboration. Our system, called the Web Integrated Software Environment (WISE), is implemented as a World-Wide-Web service that allows for management and measurement of software development projects based on dynamic analysis of change activity in the workflow. WISE tracks issues in a software development process, provides informal communication between the users with different roles, supports to-do lists, and helps in software process improvement. WISE minimizes the time devoted to metrics collection and analysis by providing implicit delivery of messages between users based on the content of project documents. The use of a database in WISE is hidden from the users who view WISE as maintaining a personal 'to-do list' of tasks related to the many projects on which they may play different roles.

Callahan, John↗
DOE OSTI2016

Poster - 03: How to manage a nuclear medicine PET-CT for radiation oncology patients

Purpose: Development of an adapted multidisciplinary procedure designed to optimize the clinical workflow between radiation therapy (RT) and nuclear medicine (NM) for a PET-CT located in the NM department. Methods : The radiation oncologist (RO) prescribes the PET-CT exam and the clinical RT therapist gives all the necessary information to the patient prior to the exam. The immobilization accessories are prepared in the RT department. The RT and NM therapists work together for radiotracer injection, patient positioning and scan acquisition. The nuclear medicine physician (NMP) will study the images, draw Biological Target Volumes (BTVs) and produce a full exam report. Results : All tasks related to a planning PET-CT are done within 48 hours from the request by the RO to the reception of the images with the NMP contours and report. Conclusions : By developing a complete procedure collectively between the RT and NM departments, the patient benefits of a quick access to a RT planning PET-CT exam including the active involvement of every medical practitioners in these fields.

60 APPLIED LIFE SCIENCES↗