SEARCH · Engineering Papers

Results for “Workflow management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 325 records · Page 18

All sources NASA NTRS DOE OSTI

DOE OSTI2023

Empowering Scientific Discovery Through Computing at the Advanced Photon Source

This paper explores the challenges and solutions for managing and processing the vast amount of data generated by the Advanced Photon Source (APS), a synchrotron light source facility producing ultra-bright x-rays for diverse scientific domains. With 68 experimental beamlines covering materials research, biology, and more, the APS serves a wide user base across academia, government, and industry. The ongoing upgrade of the APS storage ring and installation of new instruments will amplify data generation and processing demands. This paper discusses the approach to address these demands through automated data processing using standardized workflows that produce faster scientific insights. The APS Data Management System coordinates various data related tasks to manage storage, data transfer, metadata cataloging, data processing, and interfaces with tools provided by Globus. Through integration with the Argonne Leadership Computing Facility (ALCF), APS users can efficiently access high-performance computing resources. Standardized workflows have led to reduced computational burdens on scientists and greater accessibility of high performance computing resources. We demonstrate how standardization and collaboration enable scientists to rapidly convert raw data into meaningful scientific results, establishing a streamlined path from data collection to analysis and ultimately to publication.

Parraga, Hannah↗

DOE OSTI2026

Interpretable Models for Workflow Differentiation in High-Performance Scientific Networks

Scientific workflows in high-performance networks spawn hundreds of interdependent flows that must be managed collectively—yet existing network classifiers treat each flow in isolation, leading to fragmented QoS decisions and missed interflow patterns. We present a novel traffic classification solution that operates at the workflow level, distinguishing entire filetransfer operations from streaming analytics by capturing how concurrent flows interact and burst together. We introduce a workflow identification window (WIW) that ingests raw packet headers from parallel flows into unified tensors, preserving the spatial-temporal patterns that differentiate scientific workflows. This approach achieves 98.7% accuracy using CNN, LSTM, and hybrid architectures, while maintaining 84% accuracy on production traffic collected a week later—demonstrating robustness to temporal drift. By integrating SHAP and GradCAM explainability, we reveal that early-packet timing patterns and cross-flow correlations drive classification decisions, providing operators with interpretable insights. Our system enables coherent workflow-level QoS enforcement and dynamic bandwidth allocation in scientific networks, eliminating manual per-flow configuration while maintaining classification latency at millisecond level.

Giannakou, Anna [LBL, Berkeley]↗

DOE OSTI2021

STNS01-21 BEE - FY21 P6-2: Archive, clone, and re-run workflows [Slide]

BEE will give ECP a tool that great simplifies the deployment of containerized workflows on the next generation of pre-exascale and exascale systems, as well as public and private clouds. BEE allows scientists to describe their workflow using the Common Workflow Language and then deploy that workflow across the entire spectrum of systems without having to learn the specifics of each container runtime, HPC resource manager, or cloud API. BEE also streamlines the curation and sharing of common workflows among the scientific community.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

UNITY: Unified Memory and Storage Space

UNITY is a 36-month project focused on providing design and evaluate a new distributed storage paradigm that unifies the traditionally distinct application views of memory- and file-based data storage into a single scalable and resilient environment. The project is a collaboration among Oak Ridge National Laboratory (ORNL, Lead institution), Los Alamos National Labs (LANL), and Georgia Tech (GT). The main contributions of the GT team have been around development of low level systems software for best leveraging the capabilities of new types of persistent memory technologies, and for development of methods for intelligent data management across memory/storage substrates with heterogeneous components. GT contributed the Phoenix library for optimized checkpoint/restart for HPC I/O for systems with non-volatile memory (NVM), the NVStream library for NVM-specialized streaming I/O for HPC workflows, the CoMerge, Mnemo and Kleio solutions for intelligent data management on NVM-based systems. These contributions result in significant improvements in both application performance and system efficiency.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS2013

Workflow-Based Software Development Environment

The Software Developer's Assistant (SDA) helps software teams more efficiently and accurately conduct or execute software processes associated with NASA mission-critical software. SDA is a process enactment platform that guides software teams through project-specific standards, processes, and procedures. Software projects are decomposed into all of their required process steps or tasks, and each task is assigned to project personnel. SDA orchestrates the performance of work required to complete all process tasks in the correct sequence. The software then notifies team members when they may begin work on their assigned tasks and provides the tools, instructions, reference materials, and supportive artifacts that allow users to compliantly perform the work. A combination of technology components captures and enacts any software process use to support the software lifecycle. It creates an adaptive workflow environment that can be modified as needed. SDA achieves software process automation through a Business Process Management (BPM) approach to managing the software lifecycle for mission-critical projects. It contains five main parts: TieFlow (workflow engine), Business Rules (rules to alter process flow), Common Repository (storage for project artifacts, versions, history, schedules, etc.), SOA (interface to allow internal, GFE, or COTS tools integration), and the Web Portal Interface (collaborative web environment

Izygon, Michel E.↗

DOE OSTI2024

Streaming Large-Scale Microscopy Data to a Supercomputing Facility

Data management is a critical component of modern experimental workflows. As data generation rates increase, transferring data from acquisition servers to processing servers via conventional file-based methods is becoming increasingly impractical. The 4D Camera at the National Center for Electron Microscopy generates data at a nominal rate of 480 Gbit s -1 (87,000 frames s -1 ⁠), producing a 700 GB dataset in 15 s. To address the challenges associated with storing and processing such quantities of data, we developed a streaming workflow that utilizes a high-speed network to connect the 4D Camera’s data acquisition system to supercomputing nodes at the National Energy Research Scientific Computing Center, bypassing intermediate file storage entirely. In this work, we demonstrate the effectiveness of our streaming pipeline in a production setting through an hour-long experiment that generated over 10 TB of raw data, yielding high-quality datasets suitable for advanced analyses. Additionally, we compare the efficacy of this streaming workflow against the conventional file-transfer workflow by conducting a postmortem analysis on historical data from experiments performed by real users. Our findings show that the streaming workflow significantly improves data turnaround time, enables real-time decision-making, and minimizes the potential for human error by eliminating manual user interactions.

4D-STEM↗

DOE OSTI2023

Building the I (Interoperability) of FAIR for performance reproducibility of large-scale composable workflows in RECUP

Abstract-Scientific computing communities increasingly run their experiments using complex data- and compute-intensive workflows that utilize distributed and heterogeneous architectures targeting numerical simulations and machine learning, often executed on the Department of Energy Leadership Computing Facilities (LCFs). We argue that a principled, systematic approach to implementing FAIR principles at scale, including fine-grained metadata extraction and organization, can help with the numerous challenges to performance reproducibility posed by such workflows. We extract workflow patterns, propose a set of tools to manage the entire life cycle of performance metadata, and aggregate them in an HPC-ready framework for reproducibility (RECUP). We describe the challenges in making these tools interoperable, preliminary work, and lessons learned from this experiment.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2026

ML-AMD/exa-pd

Exa-pd is a highly parallelizable workflow for constructing multi-element phase diagrams (PDs). It uses standard sampling techniques—molecular dynamics (MD) and Monte Carlo (MC)—as implemented in the LAMMPS package, to simultaneously sample multiple phases on a fine temperature–composition mesh for free-energy calculations. The workflow uses Parsl as a global controller to manage the MD/MC jobs to achieve massive parallelization with almost ideal scalability. The resulting free energies of both liquid and solid phases (including solid solutions) are then fed to CALPHAD modeling using the PYCALPHAD package for the construction of a multi-element PD.

Zhang, Feng [Ames Laboratory (AMES), Ames, IA (Uni↗

NASA NTRS2013

NASA Tech Briefs, November 2013

Topics include: Cryogenic Liquid Sample Acquisition System for Remote Space Applications; 5 Spatial Statistical Data Fusion (SSDF); GPS Estimates of Integrated Precipitable Water Aid Weather Forecasters; Integrating a Microwave Radiometer into Radar Hardware for Simultaneous Data Collection Between the Instruments; Rapid Detection of Herpes Viruses for Clinical Applications; High-Speed Data Recorder for Space, Geodesy, and Other High-Speed Recording Applications; Datacasting V3.0; An All-Solid-State, Room-Temperature, Heterodyne Receiver for Atmospheric Spectroscopy at 1.2 THz; Stacked Transformer for Driver Gain and Receive Signal Splitting; Wireless Integrated Microelectronic Vacuum Sensor System; Fabrication Method for LOBSTER-Eye Optics in <110> Silicon; Compact Focal Plane Assembly for Planetary Science; Fabrication Methods for Adaptive Deformable Mirrors; Visiting Vehicle Ground Trajectory Tool; Workflow-Based Software Development Environment; Mobile Thread Task Manager; A Kinematic Calibration Process for Flight Robotic Arms; Magnetostrictive Alternator; Bulk Metallic Glasses and Composites for Optical and Compliant Mechanisms; Detection of Only Viable Bacterial Spores Using a Live/Dead Indicator in Mixed Populations; and Intravenous Fluid Generation System.

Source record↗

NASA NTRSDate not supplied

Practical Battery Thermal Modeling Techniques

Lithium-ion batteries are thermo-electrochemical devices, whereby nearly every facet of their functionality and performance are thermally driven. As a result, it is important to have thermal modeling techniques that effectively capture the intricacies of both the electrochemical nature of the battery and also the complex thermal network that typically results from the design of the battery thermal management system. Here we present a thermal modeling workflow and a set of general assumptions for how to construct a thermal model of a Li-ion battery pack. We use a 14-cell bank of 18650-format Li-ion cells, loosely based on a proposed alternative battery design for Orion, as the example. Although the workflow is performed with Thermal Desktop and related utilities, the focus of this presentation is less about software specific techniques, but rather is focused on the assumptions and conditions that should be used in a model (regardless of the tool used to build the model). Example cases and results will be presented for charge, discharge, and thermal runaway.

lithium-ion battery↗

DOE OSTI2025

mzPeak: Designing a Scalable, Interoperable, and Future-Ready Mass Spectrometry Data Format

Advances in mass spectrometry (MS) instrumentation, such as higher resolution, faster scan speeds, and improved sensitivity, have significantly increased the volume and complexity of data. The growing adoption of imaging and ion mobility further amplifies these challenges across MS-based omics fields, including proteomics, metabolomics, and lipidomics. While these technologies unlock new possibilities, they also present significant challenges in data management, storage, and accessibility. Existing open formats, such as the XML-based community standards mzML and imzML, struggle to meet the demands of modern MS workflows due to their large file sizes, slow data access, and limited metadata support. Vendor-specific formats, while optimized for proprietary instruments, lack interoperability, comprehensive metadata support and long-term archival reliability. This white paper lays the groundwork for mzPeak, a next-generation community data format designed to address these challenges and support high-throughput, multi-dimensional MS workflows. By adopting a hybrid model that combines efficient binary storage for numerical data and both human and machine-readable metadata storage, mzPeak will reduce file sizes, accelerate data access, and offer a scalable, adaptable solution for evolving MS technologies. For researchers, mzPeak will enable enhanced interoperability across platforms, seamless support for complex workflows including ion mobility and MS imaging, and faster data access compared to existing community formats such as mzML. Its design will ensure data is managed in compliance with regulatory standards, essential for applications such as precision medicine and chemical safety, where long-term data integrity and accessibility are critical. For vendors, mzPeak provides a streamlined, open alternative to proprietary formats, reducing the burden of regulatory compliance while aligning with the industry's push for transparency and standardization. By offering a high-performance, interoperable solution, mzPeak positions vendors to meet customer demands for sustainable data management tools which will be able to handle emerging and future data types and workflows. mzPeak aspires to become the cornerstone of MS data management, empowering researchers, vendors, and developers to innovate and collaborate more effectively.

data formats↗

DOE OSTI2017

Semiconductor Reliability Testing Data Management

iT2 Technologies is a startup developing analytical software and creating automated workflow environment in the area of big data in IC reliability. The main beneficiaries of the iT2 software will be the semiconductor companies as they will see significant savings in terms of time and cost. They will be able to efficiently perform the analysis of large amount of reliability data. A better standard of reliability and quality will also lead to the improvement in public safety in consumer, automotive, medical, and military applications. Smarter and thorough testing will reduce the risk of faulty hardware getting into the hands of the end users. The software is being developed and designed within the United States and will help keep the prominence of US in the cutting edge of high tech space. iT2 is establishing strong relations with its potential customers thanks to its key players being involved in this industry for the past several years and, thus, having a deep understanding of the market. The target clients will be all major semiconductor suppliers such as Intel, Qualcomm, Texas Instruments, OnSemi, and Broadcom and their customers including Apple, Cisco, IBM, HP, and Dell. The iT2 software will be used in a business-to-business environment for sharing engineering data and communicating results between these two classes of customers - some of which are participating in the beta testing of the iT2 software.

99 GENERAL AND MISCELLANEOUS↗

DOE OSTI2021

A machine-learning inverse model framework for rapid forecasting and history matching in unconventional reservoirs

Model-based optimization for real-time forecasting in unconventional reser-voirs requires novel methods and work?ows since the strategies and work?ows used in conventional reservoirs are either inapplicable, or prohibitively expen-sive and time-consuming. Insu?cient site data and computational expense of high-?delity simulations mean that work?ows with high-?delity simulations are not ideal for usage in comprehensive uncertainty quanti?cation stud-ies that require 1000s of forward model runs. We present an alternative, novel work?ow for unconventional reservoirs, based on the interplay between reduced-order models and machine-learning. Our physics-informed machine-learning (PIML) work?ow addresses the challenges to real-time reservoir management in uncoventionals, namely lack of data (the time-frame for which the wells have been producing), and computational expense of high-?delity modeling. We use the machine-learning paradigm of transfer-learning to bind together fast but less accurate reduced-order models with slow, but accurate high-?delity models and circumvent the di?culties inherent in the current state-of-the-art for unconventionals. Such a PIML work?ow, grounded in physics, is a viable candidate for real-time history matching and production forecasting in a fractured shale gas reservoir. The signi?cance of our approach is that while it is developed for a particu-lar well and site in the Marcelus Shale gas reservoir of the Appalachian basin (MSEEL), it is not wedded to it. We expect the same work?ow can be ap-plied to other shale formations (e.g., Woodford, Barnett, Utica, EagleFord) should site-data become available, using the same set of machine-learning techniques from transfer learning. Some ?ne-tuning (or minimal retraining of the neural networks) will be required to transfer knowledge across shale gas sites/formations but it is a clearly superior alternative to developing a new machine-learning model altogether when considering a di?erent site.

Srinivasan, Shriram↗

DOE OSTI2026

kessel

Kessel is a tool to create and drive continuous integration (CI) and developer workflows through a unified interface across multiple code projects and environments. It serves as a driver and integration layer for build systems and package managers, providing a flexible library of reusable components to build and execute complex workflows consistently.

Berger, Richard [@lanl]↗

NASA NTRS2018

Managing Variability: A Cognitive Ethnography of the Work of Airline Dispatchers

Airline dispatchers' workflow is often described in broad terms like 'flight planning' and 'flight following'. Such high-level descriptions fail to recognize the number and complexity of tasks involved in these activities. An ethnographic study was conducted at three US airlines to understand the cognitive workload involved in flight planning. Fuel planning was identified as one of five key flight planning tasks. Fuel planning was conducted concurrently with other planning and monitoring tasks which often led to interruptions. Planning fuel was dynamic, with recalculations required whenever other factors varied (e.g., payload, route, alternates). This rework increased workload and opportunities for error while reducing efficiency. Four main factors contributed variability to fuel planning: contingency planning, load planning, pilots, and station operations. Strategies for managing variability included pattern identification, use of buffers, rounding up, and leveraging software tools. Software design often added to workload by forcing dispatchers to attend to low level tasks.

dispatcher↗

DOE OSTI2025

Bridging paradigms: Designing for HPC-Quantum convergence

Here, this paper presents a comprehensive software stack architecture for integrating quantum computing (QC) capabilities with High-Performance Computing (HPC) environments. While quantum computers show promise as specialized accelerators for scientific computing, their effective integration with classical HPC systems presents significant technical challenges. We propose a hardware-agnostic software framework that supports both current noisy intermediate-scale quantum devices and future fault-tolerant quantum computers, while maintaining compatibility with existing HPC workflows. The architecture includes a quantum gateway interface, standardized APIs for resource management, and robust scheduling mechanisms to handle both simultaneous and interleaved quantum–classical workloads. Key innovations include: (1) a unified resource management system that efficiently coordinates quantum and classical resources, (2) a flexible quantum programming interface that abstracts hardware-specific details, (3) A Quantum Platform Manager API that simplifies the integration of various quantum hardware systems, and (4) a comprehensive tool chain for quantum circuit optimization and execution. We demonstrate our architecture through implementation of quantum–classical algorithms, including the variational quantum linear solver, showcasing the framework’s ability to handle complex hybrid workflows while maximizing resource utilization. This work provides a foundational blueprint for integrating QC capabilities into existing HPC infrastructures, addressing critical challenges in resource management, job scheduling, and efficient data movement between classical and quantum resources.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

VIBES: a workflow for annotating and visualizing viral sequences integrated into bacterial genomes

Abstract Bacteriophages are viruses that infect bacteria. Many bacteriophages integrate their genomes into the bacterial chromosome and become prophages. Prophages may substantially burden or benefit host bacteria fitness, acting in some cases as parasites and in others as mutualists. Some prophages have been demonstrated to increase host virulence. The increasing ease of bacterial genome sequencing provides an opportunity to deeply explore prophage prevalence and insertion sites. Here we present VIBES (Viral Integrations in Bacterial genomES), a workflow intended to automate prophage annotation in complete bacterial genome sequences. VIBES provides additional context to prophage annotations by annotating bacterial genes and viral proteins in user-provided bacterial and viral genomes. The VIBES pipeline is implemented as a Nextflow-driven workflow, providing a simple, unified interface for execution on local, cluster and cloud computing environments. For each step of the pipeline, a container including all necessary software dependencies is provided. VIBES produces results in simple tab-separated format and generates intuitive and interactive visualizations for data exploration. Despite VIBES’s primary emphasis on prophage annotation, its generic alignment-based design allows it to be deployed as a general-purpose sequence similarity search manager. We demonstrate the utility of the VIBES prophage annotation workflow by searching for 178 Pf phage genomes across 1072 Pseudomonas spp. genomes.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2015

SU-E-T-235: Data Mining for Evaluating Treatment Performances Over a Large Quantity of Data to Monitor and Improve SBRT Workflow

Purpose: To quality assure a large quantity of retrospective treatment cases for treatment performances by randomly sampling is inefficient. Here we provide a method to efficiently monitor and investigate the QA of SBRT workflow over Mosaiq. Methods: The code developed with Microsoft SQL Server Management Studio 2008R2 and VBA was used for retrieving and sorting data from Mosaiq (version 2.3–2.6 during 2012–2015). SBRT patients were filtered by fractional dose over 350cGy and total fraction number less than 6, which SBRT prescriptions were defined. The quality assurance on the SBRT workflow was focused on the treatment deliveries such as patient positioning setup, CBCT indicated offsets and couch shifted corrections. The treatment delivery were done by Varian Truebeam systems and the record/verify by Mosaiq. Results: Total 82 SBRT patients corresponding to 103 courses and 854 CBCT images were found by the retrieval query. Most centers record daily pre-treatment (Pre-Tx: before treatment shift) image-guided shifts along treatment course for inter-fraction motion record, and it is useful to also verify it with post-treatment imaging (Post-Tx: after treatment CBCT verification) to verify intra-fraction motion. Analyzing the details of daily recorded shifts can reveals the information of patient-setup and staff’s record/verify behaviors. 3 examples were provided as solid evidences and on-going rectification for preventing future mistakes. Conclusions: The report gave feasible examples for inspector to verify a large amount of data during site investigation. This program can also be extended to a scheduled data mining with software to periodical analyze the timely records in Mosaiq, for example, a various control charts for different QA purposes. As the current trend of automation in radiation therapy field, the data mining would be a necessary tool in the future, just as the automatic plan quality evaluation has been under development in Eclipse.

60 APPLIED LIFE SCIENCES↗