Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “compute workflow”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 397 records · Page 22
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

Consist v0.1.0

A Python library for provenance tracking, intelligent caching, and data virtualization in scientific simulation workflows. It automatically records code, configuration, and input data to skip redundant computations and enables querying results across many runs without manual bookkeeping. Designed to support multi-model simulation workflows like the BEAM CORE toolset at LBL, but designed to be extensible to a wide range of research workflows. Combines lineage tracking features as provided by OpenLineage with deterministic hashing like SnakeMake, and adds powerful analysis tools on model outputs.

Needell, Zachary [Lawrence Berkeley National Labor↗
DOE OSTI2023

Navier: Dataflow Architecture for Computation Chemistry

Navier’s objectives were two evaluate the use of emerging technologies, especially dataflow accelerators, for high-performance computing (HPC) applications, specifically in the domain of chemistry, and to develop a prototype software stack to support such applications. Navier builds on capabilities previously developed by synergistic projects, such as PNNL Data Model Convergence (DMC) LDRD Hardware Advanced Workflows (HAW) and DuOMO, as well as DOE ARIAA. Throughout its 18 months, the Navier team developed new capabilities and artifacts at all levels of the HW/SW stack, provided a seamless way to integrate novel computing architectures (Sambanova SN10 and Xilinx Versal AI) into an existing software stack, developed chemistry workflows, data analytics tools, and HPC molecular dynamics workflows that leverage the developed stack and PNNL institutional investments in emerging architectures. Navier also explored the use of active learning to accelerate a computational chemistry workflow for organic molecules on PNNL Junction cluster (in collaboration with AMD/Xilinx). Navier developed tools, methodologies, and studies for hardware software co-design and (sparse) dataflow accelerators that are composable and can be used together or separately. These methodologies are now used in other projects, such as DOE AMAIS and HPDA. This report describes Navier’s achievement, the developed tools and methodologies, and the research findings and conclusions.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2026

Depletion Benchmark Analysis on a Lead Fast Reactor Using PyARC/OpenMC

PyARC is a user-friendly fast reactor analysis tool that automates multiphysics workflows using the “extended suite” of Argonne Reactor Computation (ARC) codes by providing a single common input for model definition, code execution, and output post-processing. A lead fast reactor (LFR) benchmark model is used to perform depletion calculations using the newly integrated OpenMC depletion capability in PyARC, building on previous analysis using the ARC codes through PyARC and Serpent. Results for core lifetime k-effective, shutdown decay heat, and end-of-life heavy-metal inventory are compared to verify the PyARC/OpenMC integration against the PyARC/ARC workflow and Serpent for depletion analysis of LFR designs. The results show satisfactory agreement among all three methods, with remaining discrepancies largely attributable to differences in nuclear data libraries and decay-chain modeling detail rather than to fundamental modeling limitations.

Kiesling, Kalin R.↗
DOE OSTI2024

Ramen

Many computationally unintensive models for material process-structure-property connections exist in the literature (e.g. journal articles and textbooks). However, to our knowledge there is no existing compilation of these models in one package. Many industrially and research relevant workflows involve combinations of many of these models and if the models have inconsistent interfaces, building these workflows can be challenging. Ramen provides a collection of computationally unintensive models for calculating material process-structure-property relationships from the literature, as well as related tools (e.g. for plotting results). These models use a consistent interface structure and all leverage material data files and data structures from the Mist library (also developed at ORNL). Ramen provides a structured package for combining many standard models in one place, with a standardized interface. This approach makes combining models in long/complex chains easy and straightforward.

DeWitt, Stephen [Oak Ridge National Laboratory (OR↗
DOE OSTI2025

Using the ATLAS experiment software on heterogeneous resources

With the large dataset expected from 2030 onwards by the HL-LHC at CERN, the ATLAS experiment is reaching the limits of the current data processing model in terms of traditional CPU resources based on x86_64 architectures and an extensive program for software upgrades towards the HL-LHC has been set up. The ARM CPU architecture is becoming a competitive and energy efficient alternative. Accelerators like GPUs are available in any recent HPC. In the past years ATLAS has successfully ported its full data processing and simulation software framework Athena to ARM and has invested significant effort in porting parts of the reconstruction and simulation algorithms to GPUs. We report on the successful usage of the ATLAS experiment offline and online software framework Athena on ARM and GPUs through the PanDA workflow management system at various WLCG sites. Furthermore we report on performance optimizations of the builds for ARM CPUs and the GPU integration efforts. We will discuss performance comparisons of different ARM and x86_64 architectures on WLCG resources and Cloud compute providers like GCP and AWS using ATLAS productions workflows as used in the Hep-Score23 benchmark suite.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2021

DOE JGI Metagenome Workflow

The DOE Joint Genome Institute (JGI) Metagenome Workflow performs metagenome data processing, including assembly; structural, functional, and taxonomic annotation; and binning of metagenomic data sets that are subsequently included into the Integrated Microbial Genomes and Microbiomes (IMG/M) (I.-M. A. Chen, K. Chu, K. Palaniappan, A. Ratner, et al., Nucleic Acids Res, 49:D751–D763, 2021, https://doi.org/10.1093/nar/gkaa939) comparative analysis system and provided for download via the JGI data portal (https://genome.jgi.doe.gov/portal/). This workflow scales to run on thousands of metagenome samples per year, which can vary by the complexity of microbial communities and sequencing depth. Here, we describe the different tools, databases, and parameters used at different steps of the workflow to help with the interpretation of metagenome data available in IMG and to enable researchers to apply this workflow to their own data. We use 20 publicly available sediment metagenomes to illustrate the computing requirements for the different steps and highlight the typical results of data processing. The workflow modules for read filtering and metagenome assembly are available as a workflow description language (WDL) file (https://code.jgi.doe.gov/BFoster/jgi_meta_wdl). The workflow modules for annotation and binning are provided as a service to the user community at https://img.jgi.doe.gov/submit and require filling out the project and associated metadata descriptions in the Genomes OnLine Database (GOLD) (S. Mukherjee, D. Stamatis, J. Bertsch, G. Ovchinnikova, et al., Nucleic Acids Res, 49:D723–D733, 2021, https://doi.org/10.1093/nar/gkaa983).

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2021

The Exascale Framework for High Fidelity coupled Simulations (EFFIS): Enabling whole device modeling in fusion science

We present the Exascale Framework for High Fidelity coupled Simulations (EFFIS), a workflow and code coupling framework developed as part of the Whole Device Modeling Application (WDMApp) in the Exascale Computing Project. EFFIS consists of a library, command line utilities, and a collection of run-time daemons. Together, these software products enable users to easily compose and execute workflows that include: strong or weak coupling, in situ (or offline) analysis/visualization/monitoring, command-and-control actions, remote dashboard integration, and more. We describe WDMApp physics coupling cases and computer science requirements that motivate the design of the EFFIS framework. Furthermore, we explain the essential enabling technology that EFFIS leverages: ADIOS for performant data movement, PerfStubs/TAU for performance monitoring, and an advanced COUPLER for transforming coupling data from its native format to the representation needed by another application. Finally, we demonstrate EFFIS using coupled multi-simulation WDMApp workflows and exemplify how the framework supports the project’s needs. We show that EFFIS and its associated services for data movement, visualization, and performance collection does not introduce appreciable overhead to the WDMApp workflow and that the resource-dominant application’s idle time while waiting for data is minimal.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

US Department of Energy, Office of Science, High-Performance Computing Facility: 2023 Operational Assessment Oak Ridge Leadership Computing Facility

The Oak Ridge Leadership Computing Facility (OLCF) was established to accelerate scientific discovery by providing world-leading computational performance and advanced data infrastructure. As a US Department of Energy (DOE) Office of Science user facility, the OLCF has managed the successful deployment and operation of a succession of leadership-class resources dedicated to open science. In addition to these resources, the OLCF staff continually strive to develop innovative processes and technologies, improve security, and empower users through effective allocation management and comprehensive user support and training. These efforts support the advancement of science by the OLCF users and benefit high-performance computing (HPC) facilities around the world. In calendar year (CY) 2023, the OLCF supported 1,676 users and 598 projects and exceeded all targets for user satisfaction. The facility received an average satisfaction score of 4.52 out of 5 on the annual user survey, and 94% of respondents reported a high satisfaction rate with the OLCF overall. Of the 3,619 user tickets submitted in CY 2023, OLCF staff resolved 97% within 3 business days. The facility opened Frontier to full scientific operations this year. Two projects conducted on Frontier received the Association for Computing Machinery (ACM) Gordon Bell Prize and the Gordon Bell Special Prize for Climate Modeling, and a third earned a nomination as a Gordon Bell Prize finalist. The facility’s previous flagship machine, Summit, gained new life and was extended through 2024 in part to help provide resources to the Integrated Research Infrastructure (IRI) projects and the National Artificial Intelligence Research Resource (NAIRR) pilot program. The facility instantiated an Advanced Computing Ecosystem testbed in part to support IRI workflows. OLCF made interactivity easier and more accessible to users than ever through tools like Jupyter notebooks and workflows.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Heterogeneous data-processing optimization with CLARA’s adaptive workflow orchestrator

The hardware landscape used in HEP and NP is changing from homogeneous multi-core systems towards heterogeneous systems with many different computing units, each with their own characteristics. To achieve maximum performance with data processing, the main challenge is to place the right computing on the right hardware. In this paper, we discuss CLAS12 charge particle tracking workflow orchestration that allows us to utilize both CPU and GPU to improve the performance. The tracking application algorithm was decomposed into micro-services that are deployed on CPU and GPU processing units, where the best features of both are intelligently combined to achieve maximum performance. In this heterogeneous environment, CLARA aims to match the requirements of each micro-service to the strength of a CPU or a GPU architecture. A predefined execution of a micro-service on a CPU or a GPU may not be the most optimal solution due to the streaming data-quantum size and the data-quantum transfer latency between CPU and GPU. So, the CLARA workflow orchestrator is designed to dynamically assign micro-service execution to a CPU or a GPU, based on the online benchmark results analyzed for a period of real-time data-processing.

Gyurjyan, Vardan↗
DOE OSTI2024

Thermo-Fluid Modeling Framework for Supercomputing Digital Twins: Part 2, Automated Cooling Models

The development of digital twins for the purpose of improving the energy efficiency of supercomputing facilities is a non-trivial endeavor that is complicated by the difficulty of creating physics-based thermo-fluid cooling system models (CSMs). Within ExaDigit---an open-source framework for liquid-cooled supercomputing digital twins---a thermo-fluid modeling framework is being developed. This effort has been segmented into two with two companion papers describing each portion of the overall effort. Part 1 focuses on the development of a cooling system library in Dymola for the Frontier supercomputer at Oak Ridge National Laboratory {\cite{Kumar2024}. Part 2, this paper, describes an effort to create a template-based auto-generation methodology for CSMs, called \textit{AutoCSM}. In this paper, an overview of the initial AutoCSM architecture and workflow is provided, along with a practical example using the Oak Ridge Leadership Computing Facility's (OLCF) Frontier supercomputer CSM. AutoCSM will (1) improve ExaDigiT's user accessibility by providing a flexible workflow for modularizing the creation of the CSM system and control logic, (2) decrease the development time of CSMs, and (3) standardize the method for incorporating CSMs into the ExaDigiT framework.

Greenwood, Scott↗
DOE OSTI2023

Data Science Infrastructure SOFTWARE

LANL science workflows generate complex data sets and ensembles of data requiring significant compute and storage resources. The Data Science Infrastructure (DSI) project focuses on data-driven approaches to make data more readily available to LANL projects. DSI workflows leverage metadata stored in data-agnostic databases, supported by an abstraction layer API to simplify searching and accessing data across simulation runs, experimental runs, filesystems and environments. The abstraction layer API allows the user to query a range of data types: raw output, processed data, configuration data, machine learning models, performance data, etc. In addition to the abstraction backend API, the DSI project is developing client-driven query APIs and UIs to support specific user workflows.

Turton, Terece↗
NASA NTRSDate not supplied

TPSAS-NF1676L-33992-DND

The CERES Science Team integrates and fuses observations from 6 CERES instruments aboard the Terra, Aqua, S-NPP, and NOAA-20 missions with data from more than 20 other unique data sources. Following the November 2017 launch of CERES Flight Model 6 (FM6) onboard NOAA20, CERES has now amassed over 80 instrument-years of valuable Earth radiation budget data. The rapidly growing volume of CERES data coupled with the introduction of new data products alongside improvements to existing science algorithms fosters the requirement for faster, more flexible, and scalable data production and orchestration. New virtualized, cloud-centric compute hardware hosted by the NASA Langley Research Center’s (LaRC) Atmospheric Sciences Data Center (ASDC) provides an ideal environment for these ever-increasing data production demands for CERES. This poster discusses updates to the implementation of the CERES Data Management Team’s (DMT) CERES AuTomAted job Loading sYSTem (CATALYST), a custom data processing workflow engine for CERES, to use on-demand computing resources to perform automated CERES data production processing in a Linux-based container environment. Linux containers provide CERES the flexibility to build multiple production environments in containers tailored for specific workloads and allow effortless provisioning of resources based on the CERES Science Team’s data production requirements.

Thomas N. Hillyer↗
DOE OSTI2021

Building Toward the Future in Chemical and Materials Simulation with Accessible and Intelligently Designed Web Applications

Over the last few decades, significant progress has been made in the development and use of electronic structure and other molecular simulation methods. As these methods become more mature and are able to simulate larger and more complex chemical simulations, the need for improvement in scientific visualization, molecular builders, simplified input to simulation methods, and the development of new approaches and languages to describe simulations, along with workflows to carry them out, becomes more apparent. In this chapter, we describe our recent efforts in developing a prototype open-source computational tool called Arrows that combines NWChem, SQL and NoSQL databases, email, web APIs, and web applications in a way that make molecular and materials modeling accessible to all scientists and engineers. At the same time, because of its simplified input, it provides a framework for expert users to carry out large numbers of calculations and run complex workflows.

2↗
DOE OSTI2021

Evaluating Symmetry in 3-dimensional Simulation Data

The increasing use of three-dimensional simulations has led to an increased need for effective verification and validation, or V&V. Unfortunately, the high computational cost associated with three-dimensional simulations makes many standard V&V techniques cost-prohibitive. Given these circumstances, we must pay close attention to V&V metrics that can be evaluated using only a single simulation. One of the simplest of these metrics is the evaluation of physical symmetries. This report details a method of computing the deviation from symmetry in a three-dimensional dataset with an arbitrarily distributed mesh. Examples are presented for spherical and axisymmetric systems using the Paraview post-processing analysis workflow.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

ESnet-JLab FPGA Accelerated Transport (control plane) [EJFAT (udplbd2)] v2.0

The ESnet-JLab FPGA Accelerated Transport system is a solution for streaming high-speed scientific measurement data from Data Acquisition Systems (DAQs) to high-performance computing facilties. It is generally compatible with many science workflows, and makes no assumptions about the specifics of any particular experiment. This program (udplbd version 2) implements the control plane for the system. It is responsible for programming network forwarding rules into the data plane (implemented by the hardware designed named udplb, described separately). It also implements the control loop necessary to match up offered workload with available capacity on high-performance compute nodes.

Howard, Derek [Lawrence Berkeley National Laborato↗
DOE OSTI2021

Analysis-Specific Fast Simulation at the LHC with Deep Learning

Abstract We present a fast-simulation application based on a deep neural network, designed to create large analysis-specific datasets. Taking as an example the generation of W + jet events produced in $$\sqrt{s}=$$ s = 13 TeV proton–proton collisions, we train a neural network to model detector resolution effects as a transfer function acting on an analysis-specific set of relevant features, computed at generation level, i.e., in absence of detector effects. Based on this model, we propose a novel fast-simulation workflow that starts from a large amount of generator-level events to deliver large analysis-specific samples. The adoption of this approach would result in about an order-of-magnitude reduction in computing and storage requirements for the collision simulation workflow. This strategy could help the high energy physics community to face the computing challenges of the future High-Luminosity LHC.

Chen, C.↗
DOE OSTI2024

Electronic structure simulations in the cloud computing environment

The transformative impact of modern computational paradigms and technologies, such as high-performance computing, quantum computing, and cloud computing, has opened up profound new opportunities for scientific simulations. Scalable computational chemistry is one beneficiary of this technological progress. The main focus of this paper is on the performance of various quantum chemical formulations, ranging from low-order methods to high-accuracy approaches, implemented in different computational chemistry packages, such as NWChem, NWChemEx, SPEC, ExaChem, and FLOSIC codes on the Azure Quantum Element (AQE) Microsoft cloud services. We pay particular attention to the intricate workflows for performing composite chemistry simulations, associated data curation, and mechanisms for accuracy assessment, as defined by the enabling cloud Computational Chemistry as a Service (CCaaS). Our focus also extends to Arrows' automated workflow for high throughput simulations. Finally, we provide a perspective on the role of cloud computing in supporting the mission of leadership computational facilities (LCFs).

computational chemistry, electronic structure, Clo↗