SEARCH · Engineering Papers

Results for “Scientific Workflow”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 487 records · Page 27

All sources NASA NTRS DOE OSTI

DOE OSTI2013

Ultra-scale Visualization Climate Data Analysis Tools (UV-CDAT)

For the past three years, a large analysis and visualization effort—funded by the Department of Energy’s Office of Biological and Environmental Research (BER), the National Aeronautics and Space Administration (NASA), and the National Oceanic and Atmospheric Administration (NOAA)—has brought together a wide variety of industry-standard scientific computing libraries and applications to create Ultra-scale Visualization Climate Data Analysis Tools (UV-CDAT) to serve the global climate simulation and observational research communities. To support interactive analysis and visualization, all components connect through a provenance application–programming interface to capture meaningful history and workflow. Components can be loosely coupled into the framework for fast integration or tightly coupled for greater system functionality and communication with other components. The overarching goal of UV-CDAT is to provide a new paradigm for access to and analysis of massive, distributed scientific data collections by leveraging distributed data architectures located throughout the world. The UV-CDAT framework addresses challenges in analysis and visualization and incorporates new opportunities, including parallelism for better efficiency, higher speed, and more accurate scientific inferences. Today, it provides more than 600 users access to more analysis and visualization products than any other single source.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2017

Towards Common Components and a Standard Model for Workflow Systems

The role of scalable high-performance workflows and flexible workflow management systems that can support multiple simulations will continue to increase in importance. For example, with the end of Dennard scaling, there is a need to substitute a single long running simulation with multiple repeats of shorter simulations, or concurrent replicas. Further, many scientific problems involve ensembles of simulations in order to solve a higher-level problem or produce statistically meaningful results. However most supercomputing software development and performance enhancements have focused on optimizing single- simulation performance. On the other hand, there is a strong inconsistency in the definition and practice of workflows and workflow management systems. This inconsistency often centers around the difference between several different types of workfiows, including modeling and simulation, grid, uncertainty quantification, and purely conceptual workflows. This work explores this phenomenon by examining the different types of workflows and workflow management systems, reviewing the perspective of a large supercomputing facility, examining the common features and problems of workflow management systems, and finally presenting a proposed solution based on the concept of common building blocks. The implications of the continuing proliferation of workflow management systems and the lack of interoperability between these systems are discussed from a practical perspective. In doing so, we have begun an investigation of the design and implementation of open workflow systems for supercomputers based upon common components.

Billings, Jay Jay↗

DOE OSTI2015

HPCCloud: A Cloud/Web-Based Simulation Environment

Advanced modeling and simulation has enabled the design of a variety of innovative products and the analysis of numerous complex phenomenon. There are significant barriers to widespread adoption of these tools. In particular, advanced modeling and simulation: (1) is considered complex to use, (2) needs in-house expertise, and (3) requires high capital costs. In this paper, we describe the development of an end-to-end, advanced modeling and simulation cloud platform that encapsulates best practices for scientific computing in the cloud, and demonstrate using Hydra-TH as a prototypical application. As an alternative to traditional advanced modeling and simulation workflows, our Web-based approach simplifies the processes, decreases the need for in-house computational science and engineering experts, and lowers the capital investments. In addition to providing significantly improved, intuitive software, the environment offers reproducible workflows where the full lifecycle of data from input to final analyzed results can be saved, shared, and even published.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2017

Control and Information Systems for the National Ignition Facility

Orchestration of every National Ignition Facility (NIF) shot cycle is managed by the Integrated Computer Control System (ICCS), which uses a scalable software architecture running code on more than 1950 front-end processors, embedded controllers, and supervisory servers. The ICCS operates laser and industrial control hardware containing 66 000 control and monitor points to ensure that all of NIF’s laser beams arrive at the target within 30 ps of each other and are aligned to a pointing accuracy of less than 50 μm root-mean-square, while ensuring that a host of diagnostic instruments record data in a few billionths of a second. NIF’s automated control subsystems are built from a common object-oriented software framework that distributes the software across the computer network and achieves interoperation between different software languages and target architectures. A large suite of business and scientific software tools supports experimental planning, experimental setup, facility configuration, and post-shot analysis. Standard business services using open-source software, commercial workflow tools, and database and messaging technologies have been developed. An information technology infrastructure consisting of servers, network devices, and storage provides the foundation for these systems. Thus, this work is an overview of the control and information systems used to support a wide variety of experiments during the National Ignition Campaign.

42 ENGINEERING↗

DOE OSTI2026

FY26 Progress Report on the Operation of the Activated Materials Laboratory at the Advanced Photon Source as a Nuclear Science User Facilities Partner Facility

The Activated Materials Laboratory (AML), located in the Long Beamline Building of the Advanced Photon Source (APS) at Argonne National Laboratory (ANL), provides a centralized radiological capability for preparing, handling, and supporting synchrotron experiments on activated materials. As a Nuclear Science User Facilities (NSUF) partner facility, the AML enables the receipt of radioactive shipments, open-form sample handling, encapsulation, transport of specimens to and from APS beamlines, and experimentation with dedicated equipment. The partnership includes the APS 1-ID and 20-ID beamlines, which provide high-energy x-ray scattering, tomography, and diffraction microscopy techniques for ex-situ, in-situ, and grainresolved three-dimensional (3D) characterization. All samples supported through the AML and partner beamlines must meet radiological limits of less than 100 mrem/h at 30 cm. This FY26 progress report summarizes the first full year of AML operations as an NSUF partner facility and highlights progress in both user support and capability development. By the end of FY26, the AML had received a total of 10 NSUF-awarded projects, including 2 Consolidated Innovative Nuclear Research (CINR), 7 Rapid Turnaround Experiment (RTE), and 1 Super RTE projects. Beamtime was fully delivered for 3 RTE projects and partially delivered for 1 CINR project, demonstrating successful workflows for receipt, encapsulation, beamline transfer, and radiological experiment execution. These efforts demonstrated safe radiological experiments at APS beamlines for samples with dose rates above the historical 5 mrem/h threshold and now up to 100 mrem/h at 30 cm, marking an important milestone for neutron-irradiated materials research. During FY26, the AML also expanded its experimental capabilities. A Psylotech xTS load frame with in-grip rotation was deployed for room-temperature mechanical testing with threedimensional x-ray characterization during interrupted loading. A Linkam TS1500V vacuum heater was commissioned for thermally driven studies, and temperature calibration experiments were performed to establish specimen-relevant thermal profiles. In parallel, a customized split-tube furnace for high-temperature mechanical testing entered commissioning, and initial work began on robotic sample handling to reduce worker dose and improve operational efficiency. Data workflow and beamline operations continued to mature. Together, these developments demonstrate that the AML is becoming a unique national resource for safe, efficient, and scientifically advanced characterization of activated materials at the APS.

Zhang, Xuan↗

NASA NTRS2019

Surface Biology & Geology Pathfinder Data Analysis Pipeline

NASA's future global orbital mission, currently in development as the Surface Biology and Geology (SBG) Designated Observable study, will acquire relatively high resolution solar-reflected spectroscopy and thermal infrared observations. Innovative processes must be utilized for handling the high volume of data anticipated to be collected, which is anticipated to exceed 100 terabytes/day, greater than NASA's total extant airborne hyperspectral data collection. Collecting, processing/re-processing, disseminating, and exploiting this volume of data presents new challenges. To begin addressing them, NASA is drawing upon the expertise developed from its astrophysics programs to address Earth science and applications. Specifically, NASA is adapting the science processing operations technology developed for the Kepler and TESS planet-hunting missions for imaging spectroscopy data processing. This technology development has been the foundation for the remarkable scientific successes of Kepler and TESS. The Kepler/TESS data processing technology provides a scalable architecture for robust, repeatable, and replicable science and application products while enabling the Earth science community to develop, test, and implement new algorithms. Our effort to leverage this existing capability has begun by ingesting data and applying workflows from the EO-1/Hyperion 17-year mission archive that provides globally sampled visible through shortwave infrared spectra that are representative of SBG data types and volumes. This pathfinding data processing system will help define the solutions to processing SBG data volumes and will enable the scientific community to interact with the data and processing pipeline to create new science products.

Jenkins, Jon↗

DOE OSTI2016

Kernels for scalable data analysis in science: Towards an architecture-portable future

In this paper, we pose and address some of the unique challenges in the analysis of scientific Big Data on supercomputing platforms. Our approach identifies, implements and scales numerical kernels that are critical to the instantiation of theory-inspired analytic workflows on modern computing architectures. We present the benefits of scalable kernels towards constructing algorithms such as principal component analysis and non-negative matrix factorization on an image-analysis use case at the Oak Ridge Leadership Computing Facility (OLCF). Based on experience with the use-case, we conclude that piecing scalable analytic kernels into user-defined analytic workflows are a flexible, modular and agile way to enable architecture-portable productivity for the data-intensive sciences.

Sukumar, Sreenivas R.↗

DOE OSTI2018

Coupling Exascale Multiphysics Applications: Methods and Lessons Learned

With the growing computational complexity of science and the complexity of new and emerging hardware, it is time to re-evaluate the traditional monolithic design of computational codes. One new paradigm is constructing larger scientific computational experiments from the coupling of multiple individual scientific applications, each targeting their own physics, characteristic lengths, and/or scales. We present a framework constructed by leveraging capabilities such as in-memory communications, workflow scheduling on HPC resources, and continuous performance monitoring. This code coupling capability is demonstrated by a fusion science scenario, where differences between the plasma at the edges and at the core of a device have different physical descriptions. This infrastructure not only enables the coupling of the physics components, but it also connects in situ or online analysis, compression, and visualization that accelerate the time between a run and the analysis of the science content. Results from runs on Titan and Cori are presented as a demonstration.

Choi, Jong Youl↗

DOE OSTI2019

The Evolution of the Pegasus Workflow Management Software

Since 2001 the Pegasus Workflow Management System has evolved into a robust and scalable system that automates the execution of a number of complex applications running on a variety of heterogeneous, distributed high-throughput, and high-performance computing environments. Pegasus was built on the principle of separation between the workflow description and workflow execution, providing the ability to port and adapt the workflow based on the target execution environment. Through its user-driven research and development it has adapted to the needs of a number of scientific communities, utilizing and developing novel algorithms and software engineering solutions. This paper describes the evolution of Pegasus over time and provides motivations behind the design decisions. The paper concludes with selected lessons learned.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

STZ: A High Quality and High Speed Streaming Lossy Compression Framework for Scientific Data

Error-bounded lossy compression is one of the most efficient solutions to reduce the volume of scientific data. For lossy compression, progressive decompression and random-access decompression are critical features that enable on-demand data access and flexible analysis workflows. However, these features can severely degrade compression quality and speed. To address these limitations, we propose a novel streaming compression framework that supports both progressive decompression and random-access decompression while maintaining high compression quality and speed. Our contributions are three-fold: (1) we design the first compression framework that simultaneously enables both progressive decompression and random-access decompression; (2) we introduce a hierarchical partitioning strategy to enable both streaming features, along with a hierarchical prediction mechanism that mitigates the impact of partitioning and achieves high compression quality—even comparable to state-of-the-art (SOTA) non-streaming compressor SZ3; and (3) our framework delivers high compression and decompression speed, up to 6.7 × faster than SZ3.

Wang, Daoce [University of Nebraska, Omaha]↗

DOE OSTI2018

Machine Detection of Enhanced Electromechanical Energy Conversion in PbZr 0.2 Ti 0.8 O 3 Thin Films

Many energy conversion, sensing, and microelectronic applications based on ferroic materials are determined by the domain structure evolution under applied stimuli. New hyperspectral, multidimensional spectroscopic techniques now probe dynamic responses at relevant length and time scales to provide an understanding of how these nanoscale domain structures impact macroscopic properties. Such approaches, however, remain limited in use because of the difficulties that exist in extracting and visualizing scientific insights from these complex datasets. Using multidimensional band-excitation scanning probe spectroscopy and adapting tools from both computer vision and machine learning, an automated workflow is developed to featurize, detect, and classify signatures of ferroelectric/ferroelastic switching processes in complex ferroelectric domain structures. This approach enables the identification and nanoscale visualization of varied modes of response and a pathway to statistically meaningful quantification of the differences between those modes. Lastly, among other things, the importance of domain geometry is spatially visualized for enhancing nanoscale electromechanical energy conversion.

36 MATERIALS SCIENCE↗

DOE OSTI2016

The ADAQ framework: An integrated toolkit for data acquisition and analysis with real and simulated radiation detectors

Here, the ADAQ framework is a collection of software tools that is designed to streamline the acquisition and analysis of radiation detector data produced in modern digital data acquisition (DAQ) systems and in Monte Carlo detector simulations. The purpose of the framework is to maximize user scientific pro-ductivity by minimizing the effort and expertise required to fully utilize radiation detectors in a variety of scientific and engineering disciplines. By using a single set of tools to span the real and simulation domains, the framework eliminates redundancy and provides an integrated workflow for high-fidelity comparison between experimental and simulated detector performance. Built on the ROOT data analysis framework, the core of the ADAQ framework is a set of Cþþ and Python libraries that enable high-level control of digital DAQ systems and detector simulations with data stored into standardized binary ROOT files for further analysis. Two graphical user interface programs utilize the libraries to create powerful tools: ADAQAcquisition handles control and readout of real-world DAQ systems and ADAQAnalysis provides data analysis and visualization methods for experimental and simulated data. At present, the ADAQ framework supports digital DAQ hardware from CAEN S.p.A. and detector simulations performed in Geant4; however, the modular design will facilitate future extension to other manufacturers and simulation platforms.

detector simulation↗

DOE OSTI2016

MetaboTools: A comprehensive toolbox for analysis of genome-scale metabolic models

Metabolomic data sets provide a direct read-out of cellular phenotypes and are increasingly generated to study biological questions. Previous work, by us and others, revealed the potential of analyzing extracellular metabolomic data in the context of the metabolic model using constraint-based modeling. With the MetaboTools, we make our methods available to the broader scientific community. The MetaboTools consist of a protocol, a toolbox, and tutorials of two use cases. The protocol describes, in a step-wise manner, the workflow of data integration, and computational analysis. The MetaboTools comprise the Matlab code required to complete the workflow described in the protocol. Tutorials explain the computational steps for integration of two different data sets and demonstrate a comprehensive set of methods for the computational analysis of metabolic models and stratification thereof into different phenotypes. The presented workflow supports integrative analysis of multiple omics data sets. Importantly, all analysis tools can be applied to metabolic models without performing the entire workflow. Taken together, the MetaboTools constitute a comprehensive guide to the intra-model analysis of extracellular metabolomic data from microbial, plant, or human cells. In conclusion, this computational modeling resource offers a broad set of computational analysis tools for a wide biomedical and non-biomedical research community.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2026

Opportunities in AI/ML for the Rubin LSST Dark Energy Science Collaboration

The Vera C. Rubin Observatory's Legacy Survey of Space and Time (LSST) will produce unprecedented volumes of heterogeneous astronomical data (images, catalogs, and alerts) that challenge traditional analysis pipelines. The LSST Dark Energy Science Collaboration (DESC) aims to derive robust constraints on dark energy and dark matter from these data, requiring methods that are statistically powerful, scalable, and operationally reliable. Artificial intelligence and machine learning (AI/ML) are already embedded across DESC science workflows, from photometric redshifts and transient classification to weak lensing inference and cosmological simulations. Yet their utility for precision cosmology hinges on trustworthy uncertainty quantification, robustness to covariate shift and model misspecification, and reproducible integration within scientific pipelines. This white paper surveys the current landscape of AI/ML across DESC's primary cosmological probes and cross-cutting analyses, revealing that the same core methodologies and fundamental challenges recur across disparate science cases. Since progress on these cross-cutting challenges would benefit multiple probes simultaneously, we identify key methodological research priorities, including Bayesian inference at scale, physics-informed methods, validation frameworks, and active learning for discovery. With an eye on emerging techniques, we also explore the potential of the latest foundation model methodologies and LLM-driven agentic AI systems to reshape DESC workflows, provided their deployment is coupled with rigorous evaluation and governance. Finally, we discuss critical software, computing, data infrastructure, and human capital requirements for the successful deployment of these new methodologies, and consider associated risks and opportunities for broader coordination with external actors.

Aubourg, Eric [APC, Paris] (ORCID:000000025592023X↗

DOE OSTI2016

Data Provenance Hybridization Supporting Extreme-Scale Scientific WorkflowApplications

As high performance computing (HPC) infrastructures continue to grow in capability and complexity, so do the applications that they serve. HPC and distributed-area computing (DAC) (e.g. grid and cloud) users are looking increasingly toward workflow solutions to orchestrate their complex application coupling, pre- and post-processing needs To gain insight and a more quantitative understanding of a workflow’s performance our method includes not only the capture of traditional provenance information, but also the capture and integration of system environment metrics helping to give context and explanation for a workflow’s execution. In this paper, we describe IPPD’s provenance management solution (ProvEn) and its hybrid data store combining both of these data provenance perspectives.

provenance↗

DOE OSTI2026

Best practices in software development for robust and reproducible geoscientific models based on insights from the Global Carbon Budget's dynamic vegetation models

Computational models play an increasingly vital role in scientific research by enabling the numerical simulation of complex processes. Such models are also fundamental in geosciences. For instance, they offer critical insights into the impacts of global change on the Earth system today and in the future. Beyond their value as research tools, models are also software products and should therefore adhere to certain established software engineering standards. However, scientists are rarely trained as software developers, which can lead to potential deficiencies in software quality like unreadable, inefficient, or erroneous code. The complexity of models, coupled with their integration into broader workflows, also often makes it challenging to reproduce results, evaluate processes, and build upon them. In this paper, we review the state and current practices of the development processes of the state-of-the-art land surface models used by the Global Carbon Budget. We combine the experience of modelers from the respective research groups with the expertise of software engineers from tech companies to outline key principles and tools for improving software quality in research. We explore four main areas: (1) model testing and validation, (2) scientific, technical, and user documentation, (3) version control, continuous integration, and code review, and (4) the portability and reproducibility of workflows. Our review reveals that while modeling communities are incorporating many best practices, significant room for improvement remains in areas such as automated testing, automated documentation, and reproducibility. Therefore, we here identify and promote essential software engineering practices, including numerous examples of practices from within the community that can serve as guidelines for other models and could help streamline processes across the entire community. We conclude with an open-source example implementation of these principles, demonstrating portable and reproducible data flows, a continuous integration setup, and web-based visualizations. This example may serve as a practical resource for model developers, users, and all scientists engaged in scientific programming.

Gregor, Konstantin [Technical Univ. of Munich (Ger↗

DOE OSTI2018

Computing environments for reproducibility: Capturing the “Whole Tale”

The act of sharing scientific knowledge is rapidly evolving away from traditional articles and presentations to the delivery of executable objects that integrate the data and computational details (e.g., scripts and workflows) upon which the findings rely. This envisioned coupling of data and process is essential to advancing science but faces technical and institutional barriers. The Whole Tale project aims to address these barriers by connecting computational, data-intensive research efforts with the larger research process transforming the knowledge discovery and dissemination process into one where data products are united with research articles to create "living publications" or tales. The Whole Tale focuses on the full spectrum of science, empowering users in the long tail of science, and power users with demands for access to big data and compute resources. We report here on the design, architecture, and implementation of the Whole Tale environment.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

HDF5 in the exascale era: Delivering efficient and scalable parallel I/O for exascale applications

Accurately modeling real-world systems requires scientific applications at exascale to generate massive amounts of data and manage data storage efficiently. However, parallel input and output (I/O) faces challenges due to new application workflows and the state-of-the-art memory, interconnect, and storage architectures considered in exascale designs. The storage hierarchy has expanded with node-local persistent memory, solid-state storage, and traditional disk and tape-based storage, thus requiring efficiency at each layer and much more efficient data movement among these layers. This paper discusses how the ExaHDF5 project improved the I/O performance and data management for exascale architectures by enhancing HDF5, a widely used parallel I/O library. The team developed an Asynchronous I/O Virtual Object Layer (VOL) connector that allowed overlapping I/O with computation. They also created a Cache VOL to complement asynchronous I/O by incorporating fast storage layers, such as burst buffer and node-local storage, into the parallel I/O workflow through caching and staging data. Additionally, the team enabled data aggregation and I/O at the node level by using a Subfiling Virtual File Driver (VFD). To demonstrate superior I/O performance with HDF5 at exascale, the ExaHDF5 team collaborated with several exascale applications. In this paper, we show I/O performance improvements for three applications: Cabana (a particle-based simulation library), EQSIM (a regional earthquake simulation software), and E3SM (a climate system modeling library).

Asynchronous I/Ol↗