Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “workflows”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 613 records · Page 34

Data Services for Visualization and Analysis - ASC Level II Milestone (7186)

A new in transit Data Service is presented and compared to the traditional file-based workflow and the newly refactored in situ Catalyst workflow. Each workflow is enabled by the IOSS mesh interface equipped with data management layers for Exodus and CGNS (file-based), Catalyst (in situ), and FAODEL (in transit). FAODEL is a distributed object store that can transmit data across MPI allocations. Catalyst is a Para View-based visualization capability developed as part of the CSSE Data Services effort. The workflows considered here take SPARC data into Catalyst for visualization post-processing. Although still in unoptimized form, we show that the in transit approach is a viable alternative to file-based and in situ workflows and offers several advantages to both simulation and post-processing developers. Since IOSS is a mature interface with wide adoption across Sandia and externally, each workflow can be reconfigured to use different simulations that generate mesh data and post-processing tools that consume it.

97 MATHEMATICS AND COMPUTING↗

Implementation of a feature selection algorithm in FARM to identify important state variables and time-invariant matrices

The FARM (Feasible Actuator Range Modifier) software module is a component of the RAVEN-based FORCE framework for analysis of Integrated Energy Systems (IES). FARM aids the HERON software module in the evaluation of the optimal dispatch for the different IES components. Set-point trajectories are required to meet limits on both production variables (i.e., the variables to be optimized such as the electrical power, the hydrogen production rate, etc.) and process variables tied to the service life of equipment (e.g., steam flowrate, vessel pressure, turbine firing temperature, etc.). To evaluate the feasibility of HERON generated set-points and to do so in an acceptable time, FARM employs reduced order models to represent the dynamic behavior of the systems to be dispatched. These surrogate models take the form of a linear dynamic system with sets of Linear Parameter Varying (LPV) matrices that are mapped to the system operating space. These matrices are derived from the trajectories of system state variables and system output variables during transients. The accuracy of LPV matrices depends on the selection of state variables. In previous reports, state variables were selected by adopting a complicated workflow requiring multiple software licenses and an advanced level of user expertise. In this report, a new workflow that automates the state variable selection process is presented. It significantly reduces the frequency of user interventions and does not require multiple software licenses. Each module in the new workflow is described in detail, and the input / output examples in each step of the workflow are provided. It was demonstrated that this workflow can greatly reduce the complexity of the state variable selection process, and that the updated FARM-Gamma and FARM-Delta validators can benefit from this workflow when solving the power dispatch problem of a representative IES test case. Finally, some code improvements that can further enhance the efficiency are suggested.

42 ENGINEERING↗

FY25 Theory and Simulation Performance Target: Development of an integrated modeling framework for fusion reactor design and assessment (Final Report)

This report documents the FY25 Theory and Simulation Performance Target (TSPT) of developing an integrated modeling framework for fusion reactor design and assessment (FREDA). Over Q1-Q4, new capabilities were developed across both plasma and engineering domains and demonstrated on an example representation of a Compact Advanced Tokamak with a Dual Cooled Lead Lithium blanket. This represents a first-of-a-kind demonstration of coupled core-to-wall-to-engineering for a reactor. Self-consistent CESOL workflows were applied to provide core, pedestal, and SOL prediction; new modules were developed for energetic particle stability (FAR3D) and transport (TGLF-EP) analysis; and boundary plasma modeling (SOLPS-ITER, BOUT++/Hermes-3) was expanded to evaluate wall and divertor heat fluxes and interface with engineering thermal analysis. A parameterized CAD tool, TRACER, was expanded to generate medium-fidelity divertor, blanket, and coil geometries; OpenFOAM and Diablo workflows were applied for first-wall and divertor thermal analyses with helium cooling; and reduced-order models were created for high-mass-flux divertor cooling. Magnet multiphysics capabilities were verified between Elmer, Diablo, and a new MFEM-based solver, and workflows enable stress, thermal, and neutron-fluence analysis of TF coils with neutronics-driven heating. Nuclear and blanket analysis workflows were demonstrated, including tritium breeding, transport, and CFD-informed thermo-mechanical assessment. Preliminary multi-fidelity uncertainty quantification workflows were applied to boundary modeling codes and shown to achieve variance reductions with fewer high-fidelity boundary simulations. Key findings highlight the challenges of resolving the ITEP gap to find suitable balance between wall and divertor loads, neutron heating, and practical limits of PFC cooling. Next step priorities are to develop automated workflows to check boundary code convergence and detachment, implement tighter physics-engineering CAD provenance tracking, and inclusion of plasma-material interface models for SLAG and tungsten cracking behavior. Collectively, these developments establish sophisticated capabilities for predictive, multi-fidelity, whole-device modeling that integrates plasma physics, materials, magnets, and nuclear engineering to guide pathways to viable Fusion Pilot Plant design points.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Panorama 360 (Final Report)

This is the final technical report for the DOE-funded Panorama 360 project. Panorama 360 provided a resource for the collection, analysis, and sharing of performance data about end-to-end scientific workflows executing on DOE facilities. The work focused on workflows that include experimental data generation at DOE facilities. The main activities of Panorama 360 include the development of: 1. A distributed repository that stores different types of workflow execution data (e.g., point and time series performance traces at fine- and coarse-grained levels); 2. A set of open-source data capture, curation, and publishing tools fully integrated with a state-of-the-art workflow management system that automates data ingestion to the repository and enables users to discover, query, and process data from the repository; 3. A set of analysis algorithms and machine learning based tools to perform analysis and characterization of the gathered data, which can be used to detect anomalous performance or system faults; and 4. Best practices and recommendations for workflow evaluation, analysis, execution, and architectures.

97 MATHEMATICS AND COMPUTING↗

Panorama 360 (Final Report)

This final technical report from the lead institution, USC grant #DE-SC0012636, serves as the final technical report for collaborative institution UNC-CH grant #DE-SC0012390. The goal was to develop a repository and associated capabilities for data collection, ingestion, and analysis for a broad class of DOE applications that span experimental and simulation science workflows. In particular, this work focuses on workflows that include experimental data generation at DOE facilities. The main activities of Panorama 360 include the development of: (1) A distributed repository that stores different types of workflow execution data (e.g., point and time series performance traces at fine- and coarse-grained levels); (2) A set of open-source data capture, curation, and publishing tools fully integrated with a state-of-the-art workflow management system that automates data ingestion to the repository and enables users to discover, query, and process data from the repository; (3) A set of analysis algorithms and machine learning based tools to perform analysis and characterization of the gathered data, which can be used to detect anomalous performance or system faults; and (4) Best practices and recommendations for workflow evaluation, analysis, execution, and architectures.

97 MATHEMATICS AND COMPUTING↗

Data Federation Challenges in Remote Near-Real-Time Fusion Experiment Data Processing

Fusion energy experiments and simulations provide critical information needed to plan future fusion reactors. As next-generation devices like ITER move toward long-pulse experiments, analyses, including AI and ML, should be performed in a wide range of time and computing constraints, from near-real-time constraints, between-shot analysis, and to campaign-wide long-term analysis. However, the data volume, velocity, and variety make it extremely challenging for analyses using only local computational resources. Researchers need the ability to compose and execute workflows spanning edge resources to large-scale high-performance computing facilities.We present Delta, a system to address data analysis challenges, including AI/ML, in fusion science, by leveraging the ADIOS I/O library and middleware, to support executing science workflows over the wide area network for near-real-time streaming. We discuss the data federation challenges in performing remote workflows, focusing on on-going research work in (1) managing, reducing, and streaming data to minimize I/O and data movement overheads, (2) decompressing and reorganizing data for analysis, and (3) executing workflows for automated data analysis. We introduce examples for deep-learning based data analysis for the fusion domain and demonstrate how we use Delta to construct end-to-end workflows for a fusion device in Korea, connecting a remote DOE facility in the USA. The capability demonstrated by this project is the basis for improving the state of the art for near-real-time data federation amongst remote facilities.

Choi, Jong Youl↗

Deep learning-accelerated 3D carbon storage reservoir pressure forecasting based on data assimilation using surface displacement from InSAR

Fast forecasting of the reservoir pressure distribution during geologic carbon storage (GCS) by assimilating monitoring data is a challenging problem. Due to high drilling cost, GCS projects usually have spatially sparse measurements from few wells, leading to high uncertainties in reservoir pressure prediction. To address this challenge, we use low-cost Interferometric Synthetic-Aperture Radar (InSAR) data as monitoring data to infer reservoir pressure build up. We develop a deep learning-accelerated workflow to assimilate surface displacement maps interpreted from InSAR and to forecast dynamic reservoir pressure. Employing an Ensemble Smoother Multiple Data Assimilation (ES-MDA) framework, the workflow updates three-dimensional (3D) geologic properties and predicts reservoir pressure with quantified uncertainties. We use a synthetic commercial-scale GCS model with bimodally distributed permeability and porosity to demonstrate the efficacy of the workflow. A two-step CNN-PCA approach is employed to parameterize the bimodal fields. The computational efficiency of the workflow is boosted by two residual U-Net based surrogate models for surface displacement and reservoir pressure predictions, respectively. The workflow can complete data assimilation and reservoir pressure forecasting in half an hour on a personal computer.

25 ENERGY STORAGE↗

Utilizing Distributed Heterogeneous Computing with PanDA in ATLAS

In recent years, advanced and complex analysis workflows have gained increasing importance in the ATLAS experiment at CERN, one of the large scientific experiments at LHC. Support for such workflows has allowed users to exploit remote computing resources and service providers distributed worldwide, overcoming limitations on local resources and services. The spectrum of computing options keeps increasing across the Worldwide LHC Computing Grid (WLCG), volunteer computing, high-performance computing, commercial clouds, and emerging service levels like Platform-as-a-Service (PaaS), Container-as-a-Service (CaaS) and Function-as-a-Service (FaaS), each one providing new advantages and constraints. Users can significantly benefit from these providers, but at the same time, it is cumbersome to deal with multiple providers, even in a single analysis workflow with fine-grained requirements coming from their applications’ nature and characteristics. In this paper, we will first highlight issues in geographically-distributed heterogeneous computing, such as the insulation of users from the complexities of dealing with remote providers, smart workload routing, complex resource provisioning, seamless execution of advanced workflows, workflow description, pseudointeractive analysis, and integration of PaaS, CaaS, and FaaS providers. We will also outline solutions developed in ATLAS with the Production and Distributed Analysis (PanDA) system and future challenges for LHC Run4.

97 MATHEMATICS AND COMPUTING↗

A New Integrated Analysis Suite for Fast-Ion Study in KSTAR

Here, an integrated workflow for fast-ion analysis was developed by adapting the One Modeling Framework for Integrated Task (OMFIT) workflow manager to support a standard and unified analysis platform for KSTAR users. The newly established analysis suite offers a graphical user interface–based workflow to enable users to readily access and handle experimental data archived in various data formats and servers. Further, users can analyze the data by importing modules designed for conducting certain tasks, such as profile fitting, equilibrium reconstruction, and postprocessing of tokamak data. The procedures for preparing the inputs for fast-ion simulations are streamlined by a common workflow manager, which enables the parallel processing of various tasks to efficiently analyze large fast-ion datasets. The OMFIT platform comprises a flexible Python-based application that enables users to freely manipulate the Python scripts for applications that are unavailable in the standard workflow. The framework also offers mapping tools to translate the output data into the Integrated Modeling and Analysis Suite format to maintain application compatibility for future ITER burning plasma experiments.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Streaming Large-Scale Microscopy Data to a Supercomputing Facility

Data management is a critical component of modern experimental workflows. As data generation rates increase, transferring data from acquisition servers to processing servers via conventional file-based methods is becoming increasingly impractical. The 4D Camera at the National Center for Electron Microscopy generates data at a nominal rate of 480 Gbit s -1 (87,000 frames s -1 ⁠), producing a 700 GB dataset in 15 s. To address the challenges associated with storing and processing such quantities of data, we developed a streaming workflow that utilizes a high-speed network to connect the 4D Camera’s data acquisition system to supercomputing nodes at the National Energy Research Scientific Computing Center, bypassing intermediate file storage entirely. In this work, we demonstrate the effectiveness of our streaming pipeline in a production setting through an hour-long experiment that generated over 10 TB of raw data, yielding high-quality datasets suitable for advanced analyses. Additionally, we compare the efficacy of this streaming workflow against the conventional file-transfer workflow by conducting a postmortem analysis on historical data from experiments performed by real users. Our findings show that the streaming workflow significantly improves data turnaround time, enables real-time decision-making, and minimizes the potential for human error by eliminating manual user interactions.

4D-STEM↗

A Task Based Approach for Co-Scheduling Ensemble Workloads on Heterogeneous Nodes

Scientific workflows consist of multiple, connected applications, with data and results flowing from one to another in a pipeline. Traditionally, such workflows are executed in sequential order, storing intermediate data in storage disks. Co-scheduling application workflows concurrently on the same compute nodes would greatly reduce the cost of moving data to/from storage and allow real-time analysis of intermediate results. Nevertheless, most parallel programming runtimes do not allow seamless integration of various applications in a scientific workflow, in part due to the complexity of managing data and resources. The situation is even more complicated for heterogeneous systems. In this work we extend the Minos Computing Library (MCL) runtime to accelerate pipe-lined and parallel workloads where multiple applications are running in the same system. MCL’s asynchronous task library and runtime dynamically manages resources to allow co-scheduling of multiple processes sharing heterogeneous resources. In addition, we design a custom ex- tension of the Open Compute Language (OpenCL) to enable multiple processes to share device memory. We enable MCL to coordinate these shared buffers to allow for easy, fast data sharing between applications. Using malleable micro-benchmarks and two application workflows that combine scientific simulation and AI-based analysis, we show that our method outperforms traditional approaches.

Index Terms—Parallel systems, Scheduling and Task ↗

Ensemble Simulations on Leadership Computing Systems

Scientific productivity can be enhanced through workflow management tools, relieving large High Performance Computing (HPC) system users from the tedious tasks of scheduling and designing the complex computational execution of scientific applications. This paper presents a study on the usage of ensemble workflow tools to accelerate science using the Summit and Frontier supercomputing systems. The research aims to connect science domain simulations using Oak Ridge Leadership Computing Facility (OLCF) supercomputing platforms with ensemble workflow methods in order to accelerate HPC-enabled discovery and boost scientific impact. We present the coupling, porting and optimization of Radical-Cybertools on three applications: Chroma, NAMD and LAMMPS. The tools augment traditional HPC monolithic runs with a pilot scheduler. Lessons-learned are discussed for physics, biology and materials science applications. We discuss intrinsic limitations of coupling and porting ensemble workflow tools to applications that run on large HPC systems. The origins of technical challenges and their solutions developed during the implementation process are discussed. Data management strategies, OLCF’s policies for ensembles, and natively supported workflow tools are also summarized.

Georgiadou, Antigoni [ORNL] (ORCID:000000020977631↗

Advancing subsurface analysis: Integrating computer vision and deep learning for the near real-time interpretation of borehole image logs in the Illinois Basin-Decatur Project

The accurate quantification and mapping of subsurface natural fracture systems using borehole imaging logs are critical for the success of CO 2 sequestration in geologic formations, optimization of engineered geothermal systems, and hydrocarbon production enhancement. However, traditional interpretation processes suffer from time-consuming procedures and human bias. To address these challenges and expedite fracture analysis, we investigated the application of integrated computer vision and DL workflows to automate image log analysis. Specifically, the design of our workflow was crafted to swiftly detect fractures and baffles by using actual electrical resistivity of borehole wall from microresistivity imaging device alongside their binary representation. This novel approach significantly reduces computational time while providing invaluable insights. By incorporating conventional logging and microseismic data, we present a regional subsurface natural fracture mapping technique. Through the minimization of human bias in image log analysis, our automated workflow achieves reduced fracture interpretation time and costs while ensuring robust and reproducible results. We demonstrated the efficacy of our approach by applying the workflow to the Illinois Basin-Decatur Project site. The automated workflow successfully identified major fractured zones, multiple baffles, and an interbedded layer with a high resolution of 0.01 ft or 0.12 in. (0.3 cm) and can be upscaled to any desired resolution. Validation through microseismic and image log interpretations allows for accurate and near-real-time mapping of fractures and baffles, significantly enhancing CO 2 pressure forecasting and postinjection site care. Our approach stands out due to its robustness, consistency, and reduced computational cost compared with alternative feature extraction technologies. It presents exciting possibilities for advancing CO 2 sequestration and engineered geothermal efforts by offering comprehensive and efficient fracture mapping solutions. This technology can contribute significantly to the optimization of CO 2 sequestration projects, facilitating sustainable environmental practices, and combating climate change.

Geochemistry & Geophysics↗

Hydrologic Model Data for the East Fork Poplar Creek Watershed Simulated with the Advanced Terrestrial Simulator (ATS): Streamflow and Network Expansion–Contraction Dynamics

This dataset supports hydrologic modeling and stream network expansion–contraction analysis for the East Fork Poplar Creek (EFPC) Watershed in Tennessee. It includes a Jupyter notebook for model setup, model configuration files, simulation outputs, and derived products used to evaluate model performance and investigate stream dynamics under varying hydrologic conditions. The dataset was generated using the Watershed Workflow Python package and the Advanced Terrestrial Simulator (ATS), enabling integrated surface–subsurface hydrologic simulations using a stream-aligned mesh. Outputs include high-resolution time series of streamflow, active network length, water table depth, and related hydrologic variables. Also included are spatially explicit stream persistency indices and classifications of reaches as perennial or non-perennial. These data facilitate reproducibility and support further research on stream intermittency and variability in network extent.The model data archive is organized in following directories:1) model_setup_inputsContains the Watershed Workflow Jupyter notebooks (accessed through any open source code editor), selected input datasets, and resulting ATS input files, including XML files (access through any open source code editor), computational mesh (.exo files can be viewed using Paraview), and meteorological forcing files (.h5 files can be accessed through h5py python package and HDFView open source software). 2) model_outputsIncludes ATS simulation outputs relevant to this study. Time series of spatially integrated or averaged variables (e.g., streamflow, water table depth) are provided as CSV files. Select spatial fields (e.g., ponded depth and water table depth) are saved as pickled Python objects to reduce file size, and can be accessed through pickle package in Python. Key geometry objects from Watershed Workflow—such as the surface mesh and river tree—are also included to support analysis of streamflow persistency and expansion–contraction dynamics. These files can also be accessed through Watershed Workflow Python package.3) model_evaluationProvides observed streamflow time series and field survey-based flow regime classifications used to evaluate model performance. Jupyter notebooks for processing ATS outputs and comparing model predictions with observations to build confidence in the model prior to scientific analysis are also included.4) Q_L_relationshipsContains workflows for generating time series of discharge, active network length, and related hydrologic variables used in the stream network expansion–contraction analysis. Includes routines for delineating baseflow-dominated periods. For each catchment, notebooks and processed data (as pickled DataFrames accessed through Pandas Python package) are provided. 5) figure_scriptsProvides the Jupyter notebooks used to generate the figures presented in the paper.

54 ENVIRONMENTAL SCIENCES↗

Proxy-Based Bayesian Inversion Of Poroelastic Simulations To Interpret Strain Tensor Data Measured During Well Testing

The long runtimes of 3D poroelastic numerical simulators makes it impractical to interpret deformation datasets using many inversion schemes. Recent advances in instrumentation have made it possible to measure the strain tensor during well testing, but the lack of robust inversion methods is limiting the ability to interpret these data. We have developed an inversion workflow that reduces the number of computations required to complete a Bayesian inversion using DREAMzs. The workflow trains a KNN model using output from the poroelastic simulator, and then uses the KNN model as a proxy for the simulator during inversion. The workflow also includes a strategy for ensuring the results from the proxy model converge to the results from the simulator, ensuring the accuracy of the final results. An idealized example configured to represent a well test in a deep aquifer is used to verify that the workflow correctly identifies parameters and characterizes noise. Field data measured using strainmeters during an injection test at an oil reservoir in Oklahoma are used to evaluate performance with a real dataset. The workflow identified 265 history matching solutions out of 1240 total simulation runs (21% acceptance ratio), and the results are used to characterize posterior parameter distribution and evaluate the prediction uncertainty. This approach makes it feasible to invert strain data measured during well testing and this has the potential to improve the characterization of aquifers and reservoirs.

Roudini, Soheil↗

An intelligent Data Delivery Service for and beyond the ATLAS experiment

The intelligent Data Delivery Service (iDDS) has been developed to cope with the huge increase of computing and storage resource usage in the coming LHC data taking. It has been designed to intelligently orchestrate workflows and data management systems, decoupling data pre-processing, delivery, and primary processing in large scale workflows. It is an experiment-agnostic service that has been deployed to serve data carousel (orchestrating efficient processing of tape-resident data), machine learning hyperparameter optimization, active learning, and other complex multi-stage workflows defined via DAG (Directed Acyclic Graph), CWL (Common Workflow Language) and other descriptions, including a growing number of analysis workflows. We will at first introduce some deployed use cases in a summary. Then we will focus on new improvements and use cases under developments in ATLAS, Rubin Observatory and sPHENIX, together with future efforts.

97 MATHEMATICS AND COMPUTING↗

Mapping Vegetation at Species Level with High-Resolution Multispectral and Lidar Data Over a Large Spatial Area: A Case Study with Kudzu

Mapping vegetation species is critical to facilitate related quantitative assessment, and mapping invasive plants is important to enhance monitoring and management activities. Integrating high-resolution multispectral remote-sensing (RS) images and lidar (light detection and ranging) point clouds can provide robust features for vegetation mapping. However, using multiple sources of high-resolution RS data for vegetation mapping on a large spatial scale can be both computationally and sampling intensive. Here, we designed a two-step classification workflow to potentially decrease computational cost and sampling effort and to increase classification accuracy by integrating multispectral and lidar data in order to derive spectral, textural, and structural features for mapping target vegetation species. We used this workflow to classify kudzu, an aggressive invasive vine, in the entire Knox County (1362 km2) of Tennessee (U.S.). Object-based image analysis was conducted in the workflow. The first-step classification used 320 kudzu samples and extensive, coarsely labeled samples (based on national land cover) to generate an overprediction map of kudzu using random forest (RF). For the second step, 350 samples were randomly extracted from the overpredicted kudzu and labeled manually for the final prediction using RF and support vector machine (SVM). Computationally intensive features were only used for the second-step classification. SVM had constantly better accuracy than RF, and the producer’s accuracy, user’s accuracy, and Kappa for the SVM model on kudzu were 0.94, 0.96, and 0.90, respectively. SVM predicted 1010 kudzu patches covering 1.29 km2 in Knox County. We found the sample size of kudzu used for algorithm training impacted the accuracy and number of kudzu predicted. The proposed workflow could also improve sampling efficiency and specificity. Our workflow had much higher accuracy than the traditional method conducted in this research, and could be easily implemented to map kudzu in other regions as well as map other vegetation species.

59 BASIC BIOLOGICAL SCIENCES↗

NASA Earth eXchange (NEX) App Store

NASA Earth Exchange (NEX), and her public cloud version OpenNEX, have become platforms supporting scientific collaboration, knowledge sharing and research for the entire Earth science community. To date, a number of custom tools and capabilities have been integrated into the platforms. However, such integration has to undergo a case-by-case manual process thus lacks scalability. This timely project builds an App Store onto OpenNEX as a building block. Climate data analytics tools/programs can be easily uploaded, shared, organized, searched, and recommended like photos and videos on the YouTube. The foundation of our App Store is a provenance server, which not only records metadata but also execution history of climate data analytics apps including the input data and parameters, output data and products, who runs the app for which purpose, and how apps may be chained into workflows. Researchers can thus understand, reproduce, and repurpose existing apps and workflows. Machine learning approaches are applied to mine provenance to provide recommend-as-you-go services for Earth scientists, such as to recommend suitable apps and workflow snippets. A browser-based workflow tool is also provided for researchers to explore the provenance server and design value-added workflows. Scalability, sustainability, extensibility, usability, adaptability, security and privacy are considered in the App Store.

eXchange↗