A Classroom-Based Distributed Workflow Initiative for the Early Involvement of Undergraduate Students in Scientific Research
Abstract not provided
SEARCH · Engineering Papers
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Abstract not provided
The need for real-time processing to enable automated decision making and experimental steering has driven a shift from high-performance computing workflows on a centralized system to a distributed approach that integrates remote data sources, edge devices, and diverse compute facilities. Under this paradigm, data can be processed close to the source where it is generated, thus reducing latency and bandwidth usage. System resilience is thus a key challenge, requiring distributed workflows to survive component failures and to meet stringent quality-of-service requirements, which results in the need to mitigate anomalies such as congestion and low availability of resources. To address these challenges, we propose Diaspora, a unified resilience framework that is inspired by event-driven communication patterns used in public clouds. Specifically, we propose an event fabric that extends across sites, facilities, and computations to provide timely, reliable, and accurate information about data, application, and resource status. On top of the event fabric, we build resilience-enabling services that combine QoS-aware data streaming, resilient data views, resilient compute and data resources, and anomaly detection and prediction, all of which collectively enhance workflow resilience for these scientific cases.
This report introduces publications that report the results of a project that aimed to design a computational framework that enables computational experimentation at scale while supporting the model of “submit locally, compute globally”. The project focuses on estimating application resource needs, finding the appropriate computing resources, acquiring those resources,deploying the applications and data on the resources, managing applications and resources during run.
This manuscript develops a workflow, driven by data analytics algorithms, to support the optimization of the economic performance of an Integrated Energy System. The goal is to determine the optimum mix of capacities from a set of different energy producers (e.g., nuclear, gas, wind and solar). A stochastic-based optimizer is employed, based on Gaussian Process Modeling, which requires numerous samples for its training. Each sample represents a time series describing the demand, load, or other operational and economic profiles for various types of energy producers. These samples are synthetically generated using a reduced order modeling algorithm that reads a limited set of historical data, such as demand and load data from past years. Numerous data analysis methods are employed to construct the reduced order models, including, for example, the Auto Regressive Moving Average, Fourier series decomposition, and the peak detection algorithm. All these algorithms are designed to detrend the data and extract features that can be employed to generate synthetic time histories that preserve the statistical properties of the original limited historical data. The optimization cost function is based on an economic model that assesses the effective cost of energy based on two figures of merit: the specific cash flow stream for each energy producer and the total Net Present Value. An initial guess for the optimal capacities is obtained using the screening curve method. The results of the Gaussian Process model-based optimization are assessed using an exhaustive Monte Carlo search, with the results indicating reasonable optimization results. The workflow has been implemented inside the Idaho National Laboratory’s Risk Analysis and Virtual Environment (RAVEN) framework. The main contribution of this study addresses several challenges in the current optimization methods of the energy portfolios in IES: First, the feasibility of generating the synthetic time series of the periodic peak data; Second, the computational burden of the conventional stochastic optimization of the energy portfolio, associated with the need for repeated executions of system models; Third, the inadequacies of previous studies in terms of the comparisons of the impact of the economic parameters. The proposed workflow can provide a scientifically defendable strategy to support decision-making in the electricity market and to help energy distributors develop a better understanding of the performance of integrated energy systems.
Here we report that Least Cost Analysis (LCA) is a geospatially focused quantitative approach that can help archaeologists better understand how people may have moved across a landscape. At its core is the assumption that humans make decisions about movement as fully rational actors with complete knowledge of the landscape, attempting to minimize the cost of that movement as they travel from one location to another. LCA is most often used to construct a small number of hypothetical routes between locations of interest and, when used effectively, can even lead to the creation of entire transportation networks. Desktop GIS software packages, both commercial and open source, make running LCA relatively straightforward, but that does not necessarily mean that the output is informative or accurate. The following tutorial is designed to expose the reader to the foundational steps required to complete an LCA workflow, along with the scientific rationale behind each step and how to avoid potential pitfalls that one might run into along the way. Finally, relevant literature is cited throughout the tutorial in order to provide context for the technical methods that underpin available tools, to suggest various ways that LCA can be used to solve archaeological problems, and to show the art of the possible.
In situ approaches can accelerate the pace of scientific discoveries by allowing scientists to perform data analysis at simulation time. Current in situ workflow systems, however, face challenges in handling the growing complexity and diverse computational requirements of scientific tasks. In this work, we present Wilkins, an in situ workflow system that is designed for ease-of-use while providing scalable and efficient execution of workflow tasks. Wilkins provides a flexible workflow description interface, employs a high-performance data transport layer based on HDF5, and supports tasks with disparate data rates by providing a flow control mechanism. Wilkins seamlessly couples scientific tasks that already use HDF5, without requiring task code modifications. We demonstrate the above features using both synthetic benchmarks and two science use cases in materials science and cosmology.
The world-class research infrastructure at the US Department of Energy (DOE) Office of Science (SC) provides the research community with premier observational, experimental, computational, and network capabilities. Each user facility is designed to provide unique capabilities to advance the core DOE mission in science and technology for its SC program to stimulate rich scientific discoveries and enhance its innovation ecosystem. Research communities gather and flourish around each user facility, bringing together new and enhanced perspectives. The continual reinvention of the practice of science — as users and staff forge novel approaches expressed in research workflows — unlocks new discoveries and propels scientific progress. Within this research ecosystem, the high-performance computing (HPC) and networking user facilities stewarded by the SC’s Advanced Scientific Computing Research (ASCR) program play a dynamic cross-cutting role, enabling complex workflows demanding high-performance data, networking, and computing solutions. The ASCR facilities enterprise seeks to understand and meet the needs and requirements across SC and DOE domain science programs and priority efforts, highlighted by the formal requirements review methodology. Between July and December 2024, the Energy Sciences Network (ESnet) and the Office of High Energy Physics (HEP) of the DOE-SC organized an ESnet requirements review of HEP-supported program activities. Preparation for these events included identification of key stakeholders: program and facility management, research groups, and technology providers. Each stakeholder group was asked to prepare formal case study documents about its relationship to the HEP program to build a complete understanding of the current, near-term, and long-term status, expectations, and processes that will support the science going forward.
The 2025 Advancing Autonomous Scientific Discovery (A2SD) workshop convened researchers from academia, national laboratories, and industry to explore the transformative role of autonomy in scientific discovery. The workshop highlighted a convergence of artificial intelligence, robotics, and computational workflows into autonomous systems capable of accelerating the scientific process. Presentations and discussions spanned autonomous experimentation, intelligent workflow orchestration, digital twins, and agent-based systems for managing complex research ecosystems. Key challenges discussed included interoperability across heterogeneous infrastructures, near real-time data management under FAIR principles, reproducibility, and the integration of human oversight. The workshop also emphasized the need for modular software interfaces, federated learning models, and education initiatives to support a next-generation scientific workforce.
Artificial intelligence (AI) and machine learning (ML) have been shown to be increasingly helpful tools in a growing number of use-cases relevant to scientific research, despite significant software-related obstacles. There exist large technical costs to setting up, using, and maintaining AI/ML models in production. This often prevents researchers from utilizing these models in their work. The growing field of machine learning operations (MLOps) aims to automate much of the AI/ML life cycle while increasing access to these models. This paper presents our initial work in creating a nuclear energy MLOps platform for use by researchers at Idaho National Laboratory (INL) and aims to reduce the barriers of using AI/ML in scientific research. We hope to promote the integration of the latest AI/ML technologies into researchers' workflows and create more opportunity for scientific innovation. In this paper we discuss how our MLOps efforts aim to increase usage and the impact of AI/ML models created by engineers. We also present several use-cases that are currently integrated. Finally, we evaluate the maturity of our project as well as our plans for future functionality.
Cinema is a workflow and toolkit for capturing, storing and exploring extreme scale scientific data. It is a highly interactive image-based approach to data analysis and exploration that promotes investigation of large scientific datasets, and is easily integrated into existing workflows through extensions to widely used open source tools. In short, many different images of the dataset are captured and stored in a database, along with metadata describing the images. This provides a method of exploration that is unlike other current methods. This novel approach supports interactive exploration of a wide range of results, while still significantly reducing data movement and storage. The Cinema toolkit includes database specifications and software such as database writers, database viewers, algorithms and other support software that enable a wide variety of applications to write, operate on, and view Cinema databases.
Understanding the complex interactions of protein posttranslational modifications (PTMs) represents a major challenge in metabolic engineering, synthetic biology, and the biomedical sciences. In this work, we present a workflow that integrates multiplex automated genome editing (MAGE), genome-scale metabolic modeling, and atomistic molecular dynamics to study the effects of PTMs on metabolic enzymes and microbial fitness. This workflow incorporates complementary approaches across scientific disciplines; provides molecular insight into how PTMs influence cellular fitness during nutrient shifts; and demonstrates how mechanistic details of PTMs can be explored at different biological scales. As a proof of concept, we present a global analysis of PTMs on enzymes in the metabolic network of Escherichia coli. Based on our workflow results, we conduct a more detailed, mechanistic analysis of the PTMs in three proteins: enolase, serine hydroxymethyltransferase, and transaldolase. Finally, application of this workflow identified the roles of specific PTMs in observed experimental phenomena and demonstrated how individual PTMs regulate enzymes, pathways, and, ultimately, cell phenotypes.
UWLi is a user interface for modifying Universal Workflow Language (UWL) files. UWL is a data format used to represent high fidelity scientific procedures in a generalized, field agnostic workflow format. See related publication: https://arxiv.org/pdf/2409.05899
Agentic systems now address tasks across theoretical, phenomenological, and experimental high energy physics (HEP), but their scientific capabilities remain difficult to reuse across different large language models, providers, and harnesses. We argue that stable parts of these workflows should be promoted into versioned scientific operations and exposed through common protocols. Existing general-purpose harnesses can then be specialized for HEP through task-specific sets of tools and skills, while community-maintained registries would make these capabilities discoverable and citable. We identify mismatches in conventions, assumptions, and domains of validity among independently developed operations as a potential obstacle to their composition, and discuss machine-readable scientific contracts as one possible solution. These design principles and evaluation guidelines provide a near-term path toward a portable and community-maintained agentic harness for HEP.
Novel uses of graphical processing units for accelerated computation revolutionized the field of high-performance scientific computing by providing specialized workflows tailored to algorithmic requirements. As the era of Moore’s law draws to a close, many new non–von Neumann processors are emerging as potential computational accelerators, including those based on the principles of neuromorphic computing, tensor algebra, and quantum information. While development of these new processors is continuing to mature, the potential impact on accelerated computing is anticipated to be profound. We discuss how different processing models can advance computing in key scientific paradigms: machine learning and constraint satisfaction. Significantly, each of these new processor types utilizes a fundamentally different model of computation, and this raises questions about how to best use such processors in the design and implementation of applications. While many processors are being developed with a specific domain target, the ubiquity of spin-glass models and neural networks provides an avenue for multi-functional applications. Furthermore, this also hints at the infrastructure needed to integrate next-generation processing units into future high-performance computing systems.
In recent years, advanced and complex analysis workflows have gained increasing importance in the ATLAS experiment at CERN, one of the large scientific experiments at LHC. Support for such workflows has allowed users to exploit remote computing resources and service providers distributed worldwide, overcoming limitations on local resources and services. The spectrum of computing options keeps increasing across the Worldwide LHC Computing Grid (WLCG), volunteer computing, high-performance computing, commercial clouds, and emerging service levels like Platform-as-a-Service (PaaS), Container-as-a-Service (CaaS) and Function-as-a-Service (FaaS), each one providing new advantages and constraints. Users can significantly benefit from these providers, but at the same time, it is cumbersome to deal with multiple providers, even in a single analysis workflow with fine-grained requirements coming from their applications’ nature and characteristics. In this paper, we will first highlight issues in geographically-distributed heterogeneous computing, such as the insulation of users from the complexities of dealing with remote providers, smart workload routing, complex resource provisioning, seamless execution of advanced workflows, workflow description, pseudointeractive analysis, and integration of PaaS, CaaS, and FaaS providers. We will also outline solutions developed in ATLAS with the Production and Distributed Analysis (PanDA) system and future challenges for LHC Run4.
The Joint Laboratory for Extreme-Scale Computing (JLESC) focuses on software challenges in high-performance computing systems to meet the needs of today’s science campaigns, which often require large resources, consist of multiple tasks, and generate vast amounts of data. In this context, extreme-scale workflows have been the key factor in enabling scientific discoveries by helping scientists automate the dependencies and data exchanges between workflow tasks, instead of managing those manually. Here, in this paper, we present representative extreme-scale workflows and feature workflow systems developed by JLESC participating institutions. We present lessons learned while developing these tools, alongside with the open challenges and future research directions in the field of extreme-scale workflows.
Over a three year period, from 2016-2019, this project demonstrated the scientific benefits of integrating the Titan supercomputer at Oak Ridge Leadership Computing Facility into traditional high throughput grid based distributed computing systems managed by PanDA, the workflow management system used for the execution of all distributed computing applications by the ATLAS experiment at the Large Hadron Collider. PanDA manages millions of batch jobs daily at hundreds of clusters worldwide on request by thousands of physicist users, and processes more than an exabyte of data annually using grid middleware. High levels of operational use of Titan was sustained by PanDA in order to meet the physics goals of ATLAS. The success of this project led to the use of other supercomputers worldwide by ATLAS, and to the adoption of PanDA by other experiments and other scientists. Multiple innovative operational and computer science research goals were achieved supporting the use of supercomputers for scientific domains with large scale distributed data and distributed processing needs.
Rapid improvements in instrumentation at neutron and light source combined with new capabilities in computing, data analytics, and machine learning offer unprecedented scientific capabilities to address fundamental and applied science questions in a wide area of subjects from quantum materials to biological systems. In order to make these capabilities accessible to the broader scientific community, significant developments are needed related to workflow spanning from experimental control to advanced materials simulations as well as the underpinning scientific software. In this chapter, we explore these needs and recent developments at the Spallation Neutron Source (SNS) at Oak Ridge National Laboratory. A central part of the workflow at the SNS is the data reduction framework Mantid which is developed as a partnership between European and US neutron-scattering facilities.