Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “workflow systems”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Autonomy Loops for Monitoring, Operational Data Analytics, Feedback, and Response in HPC Operations

Many High Performance Computing (HPC) facilities have developed and deployed frameworks in support of continuous monitoring and operational data analytics (MODA) to help improve efficiency and throughput. Because of the complexity and scale of systems and workflows and the need for low-latency response to address dynamic circumstances, automated feedback and response have the potential to be more effective than current human-in-the-loop approaches which are laborious and error prone. Progress has been limited, however, by factors such as the lack of infrastructure and feedback hooks, and successful deployment is often site- and case-specific. In this position paper we report on the outcomes and plans from a recent Dagstuhl Seminar, seeking to carve a path for community progress in the development of autonomous feedback loops for MODA, based on the established formalism of similar (MAPE-K) loops in autonomous computing and self-adaptive systems. By defining and developing such loops for significant cases experienced across HPC sites, we seek to extract commonalities and develop conventions that will facilitate interoperability and interchangeability with system hardware, software, and applications across different sites, and will motivate vendors and others to provide telemetry interfaces and feedback hooks to enable community development and pervasive deployment of MODA autonomy loops.

autonomy loops↗
DOE OSTI2026

A Simple Panel System to Overcome Interface Challenges for Retrofits: Preprint

Retrofitting buildings is usually an expensive and labor-intensive process. Weatherization measures can improve comfort and energy affordability to some extent, but deep energy retrofits are needed to optimize performance and comfort, and to achieve significant energy cost savings. Barriers to deep energy retrofits include a limited supply of skilled labor, different building types, planning complexity, split incentives, and a long or non-existent ROI horizon. The "Simple Panel System" (SPS) workflow developed and demonstrated in this effort streamlines deep energy retrofits by applying advanced site capture, machine learning, and mixed reality to panelized construction. The result is a one-stop, product-independent solution for rapidly scalable retrofits with the potential to reduce construction time and project costs by 50%. Soft costs are reduced by more than 66%, total costs by more than 50%, and field construction time by more than 50% - not to mention the reduction in construction waste, improvement in working conditions, and the ability to scale without an influx of skilled labor. This paper presents the SPS and the preliminary results and findings from the pilot project.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2024

Autonomous Electrochemistry Platform with Real-Time Normality Testing of Voltammetry Measurements Using ML

Electrochemistry workflows utilize various instruments and computing systems to execute workflows consisting of electrocatalyst synthesis, testing and evaluation tasks. The heterogeneity of the software and hardware of these ecosystems makes it challenging to orchestrate a complete workflow from production to characterization by automating its tasks. We propose an autonomous electrochemistry computing platform for a multi-site ecosystem that provides the services for remote experiment steering, real-time measurement transfer, and AI/ML-driven analytics. We describe the integration of a mobile robot and synthesis workstation into the ecosystem by developing custom hub-networks and software modules to support remote operations over the ecosystem’s wireless and wired networks. We describe a workflow task for generating I-V voltammetry measurements using a potentiostat, and a machine learning framework to ensure their normality by detecting abnormal conditions such as disconnected electrodes. We study a number of machine learning methods for the underlying detection problem, including smooth, non-smooth, structural and statistical methods, and their fusers. We present experimental results to illustrate the effectiveness of this platform, and also validate the proposed ML method by deriving its rigorous generalization equations.

Alnajjar, Anees↗
DOE OSTI2021

libEnsemble: A Library to Coordinate the Concurrent Evaluation of Dynamic Ensembles of Calculations

Almost all applications stop scaling at some point; those that don't are seldom performant when considering time to solution on anything but aspirational/unicorn resources. Recognizing these tradeoffs as well as greater user functionality in a near-term exascale computing era, we present libEnsemble, a library aimed at particular scalability- and capability-stretching uses. libEnsemble enables running concurrent instances of an application in dynamically allocated ensembles through an extensible Python library. Here, we highlight the structure, execution, and capabilities of the library on leading pre-exascale environments as well as advanced capabilities for exascale environments and beyond.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

JADE

JADE provides a simple, flexible interface to submit jobs to an HPC. JADE is an HPC workflow management system that enables researchers to run simulations efficiently. It provides a simple, flexible interface to define jobs, as well as dependencies between jobs, and then dispatch them to an HPC with maximized parallelization. It also provides detailed error reporting and resource utilization statistics to aid in debugging problems.

Thom, Daniel↗
DOE OSTI2020

Constructing a Testing Application for GWMS

Many of Fermilab’s High Energy Physics experiments require High Throughput Computing to carry out simulations, data reconstruction, and data analysis. Glidein Workflow Management System (GWMS) is a tool that distributes High Throughput Computing. The purpose of GWMS is to provide convenient access to Grid resources and sites. GWMS is coupled to HTCondor. HTCondor is the workload management system that is used for scheduling and job control. Users submit jobs to a local HTCondor queue and GWMS will make sure the job will run on one of the many remote resources.

Neely-Brown, LeRayah↗
DOE OSTI2025

Just-In-Time Workflow Management for DUNE

The poster describes the justIN workflow management system funded by the UK for DUNE and now used for all DUNE centrally managed data processing and simulation

McNab, Andrew [CERN]↗
DOE OSTI2026

geoPFA: A Python-Based Open-Source Software for 3D Geothermal PFA

This work presents a novel Python-based framework, geoPFA, for conducting 3D play fairway analysis (PFA) tailored to superhot geothermal systems. The workflow has been applied to the Nesjavellir field in Iceland, a candidate site for the third Iceland Deep Drilling Project's superhot production scenarios. This application demonstrates the value of modular, transparent, and extensible workflows for integrating geological, geophysical, and simulation-derived datasets in high-enthalpy environments. Preliminary results indicate favorable zones consistent with known hydrothermal activity. The geoPFA library will soon be publicly available, offering a scalable and reproducible approach to geothermal exploration across varied geological contexts.

15 GEOTHERMAL ENERGY↗
DOE OSTI2021

Visual HPC Workflows for the Analysis of System Dynamics Models

Visual analytics supported by high performance computing (HPC) accelerates and enhances the discovery, exploration, and analysis of causal patterns in complex system dynamics (SD) models. We present a suite of visualization-assisted ensemble-based techniques for hypothesis generation and testing, and for sensitivity analysis. By employing HPC to provide parallel, on-demand simulation of SD models, one can “steer” an ensemble of simulated scenarios in real time as one first formulates and then informally tests those hypotheses: this provides rapid feedback for analysts to refine their understanding of the causal relationships emergent from a model. Such understandings can be followed and augmented by rigorous application of statistical methods, namely global variance-based sensitivity analysis, Monte-Carlo filtering, adaptive regional sensitivity analysis, and self-organized maps: here timely computation relies on HPC, while effective presentation emphasizes high-dimensional multivariate data visualization. Immersive visualization in virtual 3D environments provides an excellent adjunct to the traditional 2D graphics typically used for SD models, as it generates an embodied understanding of model behavior and facilitates an active, collaborative critique of model structure and output. Finally, we summarize prospects for HPC-enabled visual analytics applied to SD modeling.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2022

Workshop Report: 2021 International Symposium of Quantitative Codesign of Supercomputers

This report provides general information about the First International Symposium on the Quantitative Design of Supercomputers (SQCS). This new workshop was held in conjunction with Supercomputing ’21 on November 19, 2021, in St Louis Missouri. The symposium aims at combining two methodologies—collaborative codesign and data-driven analysis—to realize the potential of supercomputing more fully. We refer to the design solutions that rely on intelligence from data-driven insights across applications, systems, system software, workflows, and facilities as quantitative codesign of supercomputers.

Jones, Terry↗
DOE OSTI2022

Workshop Report: 2022 International Symposium of Quantitative Codesign of Supercomputers

This report provides general information about the Second International Symposium on the Quantitative Design of Supercomputers (SQCS). This new workshop was held in conjunction with Supercomputing ’22 on November 13, 2022, in Dallas Texas. The symposium aims at combining two methodologies—collaborative codesign and data-driven analysis—to realize the potential of supercomputing more fully. We refer to the design solutions that rely on intelligence from data-driven insights across applications, systems, system software, workflows, and facilities as quantitative codesign of supercomputers.

Jones, Terry↗
DOE OSTI2026

Noise-aware optimization in nominally identical manufacturing and measuring systems for high-throughput parallel workflows

Device-to-device variability in experimental noise critically impacts reproducibility, especially in automated, high-throughput systems like additive manufacturing farms. While manageable in small labs, such variability can escalate into serious risks at larger scales, such as architectural 3D printing, where noise may cause structural or economic failures. This contribution presents a noise-aware decision-making algorithm that quantifies and models device-specific noise profiles to manage variability adaptively. It uses distributional analysis and pairwise divergence metrics with clustering to choose between single-device and robust multi-device Bayesian optimization strategies. Unlike conventional methods that assume homogeneous devices or enforce generic robustness, the proposed framework explicitly determines whether shared optimization across devices is appropriate based on the degree of inter-device noise heterogeneity. This enables improved performance, reproducibility, and efficiency. An experimental case study involving three nominally identical 3D printers (same brand, model, and close serial numbers) demonstrates reduced redundancy, lower resource usage, and improved reliability, along with improved convergence stability and solution quality through the selection of the appropriate optimization strategy based on the degree of inter-device noise heterogeneity. Overall, this framework establishes a general approach for precision- and resource-aware optimization in scalable, automated experimental platforms, demonstrated here on a representative multi-device 3D printing case study.

Schenk, Christina↗
DOE OSTI2021

Workflows Community Summit: Tightening the Integration between Computing Facilities and Scientific Workflows

Scientific workflows are used almost universally across science domains for solving complex and largescale computing and data analysis problems. The importance of workflows is highlighted by the fact that they have underpinned some of the most significant discoveries of the past decades. Many of these workflows have significant computational, storage, and communication demands, and thus must execute on a range of large-scale computer systems, from local clusters to public clouds and upcoming exascale HPC platforms. Managing these executions is often a significant undertaking, requiring a sophisticated and versatile software infrastructure. Historically, infrastructures for workflow execution consisted of complex, integrated systems, developed in-house by workflow practitioners with strong dependencies on a range of legacy technologies—even including sets of ad hoc scripts. Due to the increasing need to support workflows, dedicated workflow systems were developed to provide abstractions for creating, executing, and adapting workflows conveniently and efficiently while ensuring portability. While these efforts are all worthwhile individually, there are now hundreds of independent workflow systems. These workflow systems are created and used by thousands of researchers and developers, leading to a rapidly growing corpus of workflows research publications. The resulting workflow system technology landscape is fragmented, which may present significant barriers for future workflow users due to many seemingly comparable, yet usually mutually incompatible, systems that exist. In order to tackle some of the challenges described above, the DOE-funded ExaWorks and NSF-funded WorkflowsRI projects have organized in 2021 a series of events entitled the “Workflows Community Summit”. The third edition of the “Workflows Community Summit” explored workflows challenges and opportunities from the perspective of computing centers and facilities. This third summit builds on two prior summits (https://workflowsri.org/summits) that (i) established a high level vision for workflows research; and (ii) explored technical approaches for realizing that vision. The third summit brought together a small group of facilities representatives with the aim to understand how workflows are currently being used at each facility, how facilities would like to interact with workflow developers and users, how workflows fit with facility roadmaps, and what opportunities there are for tighter integration between facilities and workflows. This report documents and organizes the wealth of information provided by the participants before, during, and after the summit.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Towards an Introspective Dynamic Model of Globally Distributed Computing Infrastructures

Large-scale scientific collaborations like ATLAS, Belle II, CMS, DUNE, and others involve hundreds of research institutes and thousands of researchers spread across the globe. These experiments generate petabytes of data, with volumes soon expected to reach exabytes. Consequently, there is a growing need for computation, including structured data processing from raw data to consumer-ready derived data, extensive Monte Carlo simulation campaigns, and a wide range of end-user analysis. To manage these computational and storage demands, centralized workflow and data management systems are implemented. However, decisions regarding data placement and payload allocation are often made disjointly and via heuristic means. A significant obstacle in adopting more effective heuristic or AI-driven solutions is the absence of a quick and reliable introspective dynamic model to evaluate and refine alternative approaches. In this study, we aim to develop such an interactive system using real-world data. By examining job execution records from the PanDA workflow management system, we have pinpointed key performance indicators such as queuing time, error rate, and the extent of remote data access. The dataset includes five months of activity. Additionally, we are creating a generative AI model to simulate time series of payloads, which incorporate visible features like category, event count, and submitting group, as well as hidden features like the total computational load—derived from existing PanDA records and computing site capabilities. These hidden features, which are not visible to job allocators, whether heuristic or AI-driven, influence factors such as queuing times and data movement.

kilic, Ozgur Ozan [Brookhaven National Laboratory ↗
DOE OSTI2025

A2SD: Accelerating Scientific Innovation Through Autonomous Discovery Systems

The 2025 Advancing Autonomous Scientific Discovery (A2SD) workshop convened researchers from academia, national laboratories, and industry to explore the transformative role of autonomy in scientific discovery. The workshop highlighted a convergence of artificial intelligence, robotics, and computational workflows into autonomous systems capable of accelerating the scientific process. Presentations and discussions spanned autonomous experimentation, intelligent workflow orchestration, digital twins, and agent-based systems for managing complex research ecosystems. Key challenges discussed included interoperability across heterogeneous infrastructures, near real-time data management under FAIR principles, reproducibility, and the integration of human oversight. The workshop also emphasized the need for modular software interfaces, federated learning models, and education initiatives to support a next-generation scientific workforce.

Taufer, Michela [University of Tennessee, Knoxvill↗
DOE OSTI2022

Quantifying the Impact of Advanced Web Platforms on High Performance Computing Usage

The deployment of Science Gateways for High Performance Computing (HPC) systems can alter long-accepted usage patterns on supercomputing systems in positive ways as an ever-increasing number of users migrate their workflows to HPC systems. Idaho National Laboratory (INL) has deployed two separate advanced web platforms, Open OnDemand and NICE DCV, for integration with HPC resources to improve web accessibility for HPC users. Researchers conducted a multi-year study on how HPC usage pat- terns changed in the presence of these platforms. This work reports the results of that study and quantifies the observed impacts, including adoption by visualization and Jupyter Notebook/Lab users, decreased job submission friction, rapid uptake of HPC by Windows users, and increased overall system utilization. The most significant impacts were observed from the deployment of Open OnDemand, and this work also identifies some best practices for Open OnDemand deployment for HPC datacenters.

97 MATHEMATICS AND COMPUTING↗