Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “distributed parallelization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 649 records · Page 36
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Phoenix: A Scalable Streaming Hypergraph Analysis Framework

We present Phoenix, a scalable hypergraph analytics framework for data analytics and knowledge discovery that was implemented on the leadership class computing platforms at Oak Ridge National Laboratory (ORNL). Our software framework comprises a distributed implementation of a streaming server architecture which acts as a gateway for various hypergraph generators/external sources to connect. Phoenix has the capability to utilize diverse hypergraph generators, including HyGen, a very large-scale hypergraph generator developed by ORNL. Phoenix incorporates specific algorithms for efficient data representation by exploiting hidden structures of the hypergraphs. Our experimental results demonstrate Phoenix’s scalable and stable performance on massively parallel computing platforms. Phoenix’s superior performance is due to the merging of high-performance computing with data analytic.

Kurte, Kuldeep↗
DOE OSTI2026

Commuting embeddings for parallel strategies in non-local games

Non-local games provide a versatile framework for probing quantum correlations and for benchmarking the power of entanglement. In finite dimensions, the standard method for playing several games in parallel requires a tensor product of the local Hilbert spaces, which scales additively in the number of qubits. In this work, we show that this additive cost can be reduced by exploiting algebraic embeddings. We introduce two forms of compressions. First, when a referee selects one game from a finite collection of games at random, the game quantum strategy can be implemented using a maximally entangled state of dimension equal to the largest individual game, thereby eliminating the need for repeated state preparations. Second, we establish conditions under which several games can be played simultaneously in parallel on fewer qubits than the tensor product baseline. These conditions are expressed in terms of commuting embeddings of the game algebras. Moreover, we provide a constructive framework for building such embeddings. Using tools from Lie theory, we show that aligning the various game algebras into a common Cartan decomposition enables such a qubit reduction. Beyond the theoretical contribution, our framework casts NLGs as algebraic primitives for distributed and resource-constrained quantum computations and suggested NLGs as a comparable device-independent dimension witness.

Commuting embeddings↗
DOE OSTI2023

Tonian Carbonates Record Phosphate-Rich Shallow Seas

The early-middle Neoproterozoic is thought to have witnessed significant perturbations to marine P cycling, in turn facilitating the rise of eukaryote-dominated primary production. However, with few robust constraints on aqueous P concentrations, current understanding of Neoproterozoic P cycling is generally model-dependent. To provide new geochemical constraints, we combined microanalytical data sets with solid-state Nuclear Magnetic Resonance, synchrotron-based X-ray Absorption Near Edge Structure spectroscopy, and micro-X-ray Fluorescence imaging to characterize the speciation and distribution of P in Tonian shallow-water carbonate rocks. These data reflect shallow water phosphate concentrations 10–100× higher than modern systems, supporting the hypothesis that tectonically-driven influxes in P periodically initiated kinetically-controlled CaCO 3 deposition, in turn destabilizing marine carbonate chemistry, climate, and nutrient inventories. Alongside these observations, a new compilation and statistical analysis of mudstone geochemistry data indicates that, in parallel, C org and P burial increased across later Tonian continental margins until becoming decoupled at the close of the Tonian, implicating widespread N-limitation triggered by increasing atmospheric O 2 .

58 GEOSCIENCES↗
DOE OSTI2023

Conceptual design of DIII-D experiments to diagnose the lifetime of spin polarized fuel

Abstract In magnetic fusion experiments, the cross sections for the D-T and D- 3 He fusion reactions are increased by as much as 50% if the fuel remains spin polarized parallel to the magnetic field. The goal of this study is to assess the feasibility of lifetime measurements of spin polarization, in magnetic fusion relevant conditions, on the DIII-D tokamak using relative changes in charged fusion product (CFP) loss measurements that depend upon the differential fusion cross section d σ /dΩ. Relative measurements that capture changes in the escaping CFP pitch, poloidal, and energy distributions are studied in two realistic TRANSP calculated plasma scenarios: (a) vector-polarized 3 He and D pellets are injected into a hot hydrogen plasma to produce thermonuclear reactions, and (b) a tensor-polarized deuterium pellet is injected into an L-mode hydrogen background plasma that includes unpolarized 3 He neutral beam injection. Ideal CFP signals in both scenarios show substantial pitch sensitivity to polarization for 14.7 MeV proton detection at a poloidal angle of − 56 ∘ (on the outer wall in the ion ∇ B direction), and strong sensitivity to polarization for 3.6 MeV alpha flux detection by an array of poloidal detectors. Energy-resolved measurements of 14.7 MeV protons are also sensitive to the degree of polarization for the − 56 ∘ port in the beam-plasma scenario. A realistic assessment of CFP signals in the thermonuclear scenario show count rates in the range of 2 × 10 4 cps for pitch-resolved proton detection and 2 × 10 5 cps for alpha flux measurements. Reduced chi-squared χ r 2 calculations show polarization lifetime measurements are feasible for either proton or alpha detection of both enhanced or suppressed polarization for the thermonuclear scenario. Measurements of gamma rays produced in the weak D + 3 He → γ + 5 Li branch complement the CFP measurements.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2021

Ensuring statistical reproducibility of ocean model simulations in the age of hybrid computing

Novel high performance computing systems that feature hybrid architectures require large scale code refactoring to unravel underlying exploitable parallelism. Such redesign can often be accompanied with machine-precision changes as the order of computation cannot always be maintained. For chaotic systems like climate models, these round-off level differences can grow rapidly. Systematic errors may also manifest initially as machine-precision differences. Isolating genuine round off level differences from such errors remains a challenge. Here, we apply two-sample equality of distribution tests to evaluate statistical reproducibility of the ocean model component of US Department of Energy's Energy Exascale Earth System Model (E3SM). A 2-year control simulation ensemble is compared to a modified ensemble as a test case - after a known non-bit-for-bit change in a model component is introduced - to evaluate the null hypothesis that the two ensembles are statistically indistinguishable. To quantify the false negative rates of these tests, we conduct a formal power analysis using a targeted suite of short simulation ensembles. The ensemble suite contains several perturbed ensembles, each with a progressively different climate than the baseline ensemble - obtained by perturbing the magnitude of a single model tuning parameter, the Gent and McWilliams κ, in a controlled manner. The null hypothesis is evaluated for each of perturbed ensembles using these tests. The power analysis informs on the detection limits of the tests for given ensemble size allowing model developers to evaluate the impact of an introduced non-bit-for-bit change to the model.

Mahajan, Salil↗
DOE OSTI2022

Accelerating shared file checkpoint with local burst buffers

A data management system and method for accelerating shared file checkpointing. Written application data is aggregated in an application data file created in a local burst buffer memory at a compute node, and an associated data mapping built index to maintain information related to the offsets into a shared file at which segments of the application data is to be stored in a parallel file system, and where in the buffer those segments are located. The node asynchronously transfers a data file containing the application data and the associated data mapping index to a file server for shared file storage. The data management system and method further accelerates shared file checkpointing in which a shared file, together with a map file that specifies how the shared file is to be distributed, is asynchronously transferred to local burst buffer memories at the nodes to accelerate reading of the shared file.

Gooding, Thomas↗
DOE OSTI2025

Practical Implementation of GPU-based Computing at the Grid Edge for Resilience Scenarios

This paper presents a practical implementation of GPU-accelerated computing at the grid edge to enhance power system resilience through next-generation smart meters. Advanced Metering Infrastructure (AMI) systems rely predominantly on centralized processing architectures, which limit real-time response capabilities during grid disturbances. This work proposes the integration of GPU-enabled computational platforms directly within smart meter to enable local execution support for power system analytics, fault detection algorithms, and optimization routines. The proposed framework uses the Julia programming language to leverage highperformance parallel computing capabilities while maintaining code portability and development efficiency. We use two experimental scenarios to benchmark the computational feasibility of this approach: sparse linear system solutions representative of power flow analyses, and multi-stage production cost simulations incorporating unit commitment and economic dispatch operations. Results demonstrate that computationally intensive power system algorithms, such as those supporting resilience scenario calculations, can be effectively executed at the distribution edge using commercially available embedded GPU hardware. Keywords—GPU acceleration, edge computing, smart meters, grid resilience, AMI, resilience.

De Souza, Reubun [School of Electrical Engineering↗
DOE OSTI2020

Research Roadmap on Grid-Forming Inverters

This research roadmap is intended to fill the knowledge gap by providing a system view of grid-forming inverter-based resource controls and their impact on grid stability, which we believe is central to meeting some of the challenges to operating the future North American electric power system. This includes the roles and requirements of grid-forming inverter-based resources—including solar photovoltaics, wind generators, and energy storage. For this roadmap, we focus on a specific family of grid-forming inverter control approaches that do not rely on an external voltage source (i.e., no phase-locked loop) and that can share load without explicit communications. Although the roadmap is focused narrowly on system challenges for grid-forming controls and power system stability, including interactions with protection, we hope it serves as a foundational element for future system-of-systems roadmapping needed in a broader grid modernization effort with increasing deployments of inverter-based resources. The roadmap first introduces formal definitions for the grid stability topics and then describes the differences between grid-forming and traditional grid-following control approaches for inverter-based resources. The core of the roadmap consists of a review of current research and an outline of research needs related to five grid-forming inverter topics: frequency control, voltage control, system protection, fault ride-through and voltage recovery, and modeling and simulation. The review both delineates contemporary advances and highlights open research questions that must be addressed to enable the widespread adoption of inverter-based resources across the grid. Feedback from industry on these research questions is incorporated, including discussions during the Workshop on Grid-forming Inverters for Low-inertia Power Systems. The workshop included industry presentations and discussion of ongoing research, technology gaps, and piloting needs. This roadmap concludes by offering a multiyear perspective on the gradual field validation of grid-forming inverters. This perspective recognizes that the scale and scope of the types of power systems that inverters will be called on to provide grid-forming services will and should begin modestly. Specifically, this roadmap recognizes that inverter controls today are predominantly grid-following and that future power systems will involve a mix of inverter-based resources with both grid-following and grid-forming control capabilities. Growth over time will depend on how well grid-forming inverters perform and what advantages they bring as penetration levels (instantaneous and average) of inverter-based resources increases. This recognition, in turn, establishes a natural sequence of priorities for the research questions that must be addressed. Following this multiyear perspective, the roadmap offers short descriptions of two specific near-term research priorities: the review of regulatory and technical standards and the development of advanced modeling techniques. These priorities are foundational. We recommend immediate pursuit of them in parallel with and in direct support of the research outlined by our multiyear perspective.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2026

Fokker-Planck simulations of fast ion ICRF and electron EC heating in a mirror plasma using CQL3D-m

The CQL3D-m continuum bounce-average Fokker-Planck code is adapted for magnetic mirror plasmas [1] and is now routinely used in no-free-parameter classical integrated modeling of mirror devices [2, 3]. In the present effort, we report on two RF methods of plasma heating in mirror machine. The fast ions (FI) are heated by Fast waves at 2nd-4th harmonic, where FIs originate from neutral beam injection at 45 degrees to the magnetic field. The scenario shows an efficient ion heating near the FI bouncing point. The electrons are heated by X-mode launched from the high magnetic field side towards the resonance. Different from the tokamak applications, CQL3D-m provides an evolving self-consistent ambipolar parallel electric field, which determines the shape of the loss cone and hence an accurate confinement time of both ions and electrons. Also, it includes a description of ion and electron sources and sinks (related to charge exchange and impact ionization) which are updated at every time step. CQL3D-m utilizes a fully nonlinear Coulomb collision operator that is important for the significantly non-Maxwellian ion distributions typically established in mirror plasmas.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2024

An experimentally validated electro-thermal EV battery pack model incorporating cycle-life aging and cell-to-cell variations

Lithium-ion batteries are used in a wide variety of applications. To meet the power and energy demands of these applications battery packs are composed of hundreds to thousands of cells. The electrical and thermal interactions between cells introduce additional complexity in the pack dynamics. To capture these effects, a battery pack model composed of 192 cells based on a first-generation (2012) Nissan Leaf battery pack is developed in MATLAB/Simulink/Simscape. Here, with this model, we simulate the electrical dynamics (using a first-order equivalent-circuit model), the thermal dynamics (using a first-order lumped-parameter thermal model), and the aging dynamics (using a semi-empirical severity factor-based model) of every cell in the pack and we also create a pack thermal model that explicitly captures the heat exchange between the modules, and the cells contained within, during operation. The models are calibrated and validated, both at the cell and pack level, with experimental data. Two different case studies of this pack model are investigated. In the first case study, an initial, normally-distributed, cell-to-cell capacity variation is introduced and its effect on the pack voltage and module temperatures is studied. In the second case study, we deliberately insert cells with lower than nominal capacity into the pack and we investigate how this type of initial cell-to-cell capacity variation affects the pack’s ability to deliver energy over time. Finally, we also study how parallel-connected cells can reduce the effects of cell-to-cell variations at the expense of increased aging of the pack overall.

25 ENERGY STORAGE↗
DOE OSTI2022

Synchronous High-frequency Distributed Readout For Edge Processing At The Fermilab Main Injector And Recycler

The Main Injector (MI) was commissioned using data acquisition systems developed for the Fermilab Main Ring in the 1980s. New VME-based instrumentation was commissioned in 2006 for beam loss monitors (BLM)[2], which provided a more systematic study of the machine and improved displays of routine operation. However, current projects are demanding more data and at a faster rate from this aging hardware. One such project, Real-time Edge AI for Distributed Systems (READS), requires the high-frequency, low-latency collection of synchronized BLM readings from around the approximately two-mile accelerator complex. Significant work has been done to develop new hardware to monitor the VME backplane and broadcast BLM measurements over Ethernet, while not disrupting the existing operations critical functions of the BLM system. This paper will detail the design, implementation, and testing of this parallel data pathway.

43 PARTICLE ACCELERATORS↗
DOE OSTI2022

Science of the Van Allen Probes Science Operations Centers

The Van Allen Probes mission operations materialized through a distributed model in which operational responsibility was divided between the Mission Operations Center (MOC) and separate instrument specific SOCs. The sole MOC handled all aspects of telemetering and receiving tasks as well as certain scientifically relevant ancillary tasks. Each instrument science team developed individual instrument specific SOCs proficient in unique capabilities in support of science data acquisition, data processing, instrument performance, and tools for the instrument team scientists. In parallel activities, project scientists took on the task of providing a significant modeling tool base usable by the instrument science teams and the larger scientific community. With a mission as complex as Van Allen Probes, scientific inquiry occurred due to constant and significant collaboration between the SOCs and in concert with the project science team. Planned cross-instrument coordinated observations resulted in critical discoveries during the seven-year mission. Instrument cross-calibration activities elucidated a more seamless set of data products. Specific topics include post-launch changes and enhancements to the SOCs, discussion of coordination activities between the SOCs, SOC specific analysis software, modeling software provided by the Van Allen Probes project, and a section on lessons learned. One of the most significant lessons learned was the importance of the original decision to implement individual team SOCs providing timely and well-documented instrument data for the NASA Van Allen Probes Mission scientists and the larger magnetospheric and radiation belt scientific community.

47 OTHER INSTRUMENTATION↗
DOE OSTI2020

Distributed fiber sensing systems for 3D combustion temperature field monitoring in coal-fired boilers using optically generated acoustic waves (Final Report)

In this project, we have developed and tested three kinds of fiber optic sensing systems for real time monitoring of temperature variations within an industrial scale boiler furnace. The fiber optic sensing systems target spatial and temporal distributions of high temperature profiles in a boiler furnace in fossil power plants. The reconstructed temperature profile will provide critical input for the control mechanisms to optimize the combustion process. This temperature profile will address the essential problem for fossil power plants in achieving higher efficiency and fewer pollutant emissions. Acoustic pyrometer systems have been used to reconstruct temperature field of power plant boilers based on measuring TOF (times-of-flight) of sound waves along some straight paths in a 2D cross-section of the boiler. In this project, optically generated acoustic signals from a fiber optic sensing system have replaced the acoustic signals generated from an electrical transducer. A 3D reconstruction algorithm replaced the previous 2D model. In this project, three kinds of fiber optic sensing systems have been developed and tested. They are fiber optic sensing system I, fiber optic sensing system II (Distributed Sensing System I) and fiber optic sensing system III (Distributed Sensing System II). For fiber optic sensing system I, the fiber optic ultrasound generator acts as a signal generator. A microphone, hydrophone or other electronic devices serve as a signal receiver. In this system, there are one generator and one receiver. Distance test, water temperature test, air temperature test, air temperature reconstruction, and GE ISBF pilot test were performed by Fiber optic sensing system I. The fiber optic sensing system I successfully detected temperature in all these tests. We got 2D temperature reconstruction results by using the fiber optic sensing system I and it matched the reference data. The fiber optic sensing system I successfully survived in GE ISBF boiler environment (480 °F). For fiber optic sensing system II (Distributed Sensing System I), it is an all optical ultrasound system. The fiber optic ultrasound generator acts as a signal generator. Fiber Bragg Grating (FBG) and Fabry-Perot (FP) sensor act as a signal receiver. In this system, there is one generator and one receiver. Aluminum plate temperature test, furnace high temperature test, and GE ISBF pilot test were performed by the fiber optic sensing system II. Fiber optic sensing system II successfully detected the temperature in all these tests. The fiber optic sensing system II successfully survived at up to 700 °C furnace environment and 320 °C GE ISBF boiler environment. For fiber optic sensing system III (Distributed Sensing System II), it is also an all optical ultrasound system. The fiber optic ultrasound generator acts as a signal generator. Multiple FP fiber sensors act as signal receivers. In this system, there is one generator and three receivers. Three GE ISBF pilot tests were performed by fiber optic sensing system III. The fiber optic sensing system III survived in the cold flow tests in GE’s ISBF pilot test facility. However, we didn’t get high temperature data by using this system since the nanosecond laser issues. During the period of the project, test trials, data simulation and algorithm optimization was performed successfully. For real time temperature field construction, the sampling rate must be fast enough to capture the field variations. The technology of Code-division multiple access (CDMA) is well studied which could allow parallel multiplexing, even if signals overlap in time or frequencies. Moreover, it has been known that extending the length of signal significantly improves SNR. For acoustic signals, these multiplexing techniques have also been widely used, mainly for sonar and acoustic communications. The CDMA modulation technique has been proposed and studied to guarantee high network throughput, low channel access delay and low energy consumption. We have studied the temperature field reconstruction using Gaussian Radial Basis Functions (GRBF)-based approximation approach. Reconstruction of 3D temperature field using Neural Networks with measured TOF and known propagation paths is feasible. 2D and 3D temperature field reconstruction simulation results are achieved. The milestone status is shown in Table 1. We finished milestone 1-8 and milestone 10. For milestone 9, we did three pilot tests by using the fiber optic sensing system III (Distributed Sensing System II) at GE Power. However, due to the failure of the ns laser, we did not get the temperature results. We conducted some additional tasks that were not originally proposed: 1) We fabricated a fiber optic sensing system I and did a pilot test based on this system. 2) In the proposal, we proposed two pilot tests at GE Power. In reality, we finished at least seven pilot tests at GE Power. GE Power has made a lot of efforts for supporting the pilot tests. 3) We got a simulation results based on CDMA. In summary, most of the tasks have been accomplished. The outcome of this project removed a few barriers that hinder the achievement of the final product of the distributed sensing systems. With the successful accomplishment of this project, a prototype of the fiber optic sensing system can be fabricated to attract more interests from companies and other funding agencies.

47 OTHER INSTRUMENTATION↗
DOE OSTI2021

Transient cycling of nitrogen, organic carbon and oxygen within the free-flowing Columbia River corridor: Linking exposure time dependent biogeochemical reactions to river stage fluctuations (Final project report)

The objective of this project was to develop next-generation techniques for representing the transformations of complex reaction networks like those impacting transient river-corridors, then apply them to develop upscaling tools. The project contributed significantly to the development of Lagrangian “mass transfer particle tracking” (MTPT) tools that are the first numerical framework that explicitly separates mixing and spreading processes; a crucial distinction for accurate representations of reaction rates across scales. The MTPT approach uses an operator splitting scheme where physical transport processes are simulated using classical random walk methods and each particle is treated as a “container” that carries with it any number of chemical species. Mixing between particles is simulated using a colocation-probability based mass transfer kernel and reactions are evaluated on each particle after mixing. The method is stable, accurate, and also allows the explicit modeling of residence time distributions of the individual containers of mass. The MTPT scheme allowed us to explore several research questions related to reactive transport in river corridors and complex reaction networks. We found that it is possible to quantify the exposure time distributions (ETDs) of reactants and that this does suggest some pathways to upscaling. However, comparable approximations can be obtained using simplified (reduced dimensionality) MTPT simulations in less time with similar uncertainty, so development of extensive ETD-based methods was not productive. The major findings are that i) the MTPT schemes are robust and highly accurate across a range of arbitrarily complex reactions, ii) the methods can be efficiently parallelized and the parallel performance characteristics of MTPT are predictable, and iii) the MTPT tools allow simultaneous tracking of residence time. The development of these research tools into efficient software packages continues. The methods are already available to the community because open-source, working examples have been included with all publications.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

How good are polarizable and flexible models for water: Insights from a many-body perspective

We present a systematic analysis of state-of-the-art polarizable and flexible water models from a many-body perspective, with a specific focus on their ability to represent the Born–Oppenheimer potential energy surface of water from the gas to the liquid phase. Using coupled cluster data in the completed basis set limit as a reference, we examine the accuracy of the polarizable models in reproducing individual many-body contributions to interaction energies and harmonic frequencies of water clusters and compare their performance with that of MB-pol, an explicit many-body model that has been shown to correctly predict the properties of water across the entire phase diagram. Based on these comparisons, we use MB-pol as a reference to analyze the ability of the polarizable models to reproduce the energy landscape of liquid water under ambient conditions. We find that, while correctly reproducing the energetics of minimum-energy structures, the polarizable models examined in this study suffer from inadequate representations of many-body effects for distorted configurations. To investigate the role played by geometry-dependent representations of 1-body charge distributions in reproducing coupled cluster data for both interaction and many-body energies, we introduce a simplified version of MB-pol that adopts fixed atomic charges and demonstrate that the new model retains the same accuracy as the original MB-pol model. Based on the analyses presented in this study, we believe that future developments of both polarizable and explicit many-body models should continue in parallel and would benefit from synergistic efforts aimed at integrating the best aspects of the two theoretical/computational frameworks.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2025

Exact signed distance fields using parallel Fast Sweeping Method

Signed distance fields are often used in multiphysics simulations to track material interfaces. We present a simple methodology based on the fast sweeping method to generate the exact signed distance from triangular meshes and linear paths on Cartesian grids. The methodology propagates the closest primitive to the boundary to the rest of the domain following the characteristics. A local upwind criterion is used to decide between the new and existing closest primitive at each grid point while capturing the correct sign of the global function. The methodology has optimal computational complexity and runs efficiently in distributed-memory architectures. We include 2D and 3D test cases along with a resolution study up to 0.512 trillion zones and 1,000 computer cores. The solution strategy can also be applied to other types of meshes or collections of primitives.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Characterizing the performance of node-aware strategies for irregular point-to-point communication on heterogeneous architectures

Supercomputer architectures are trending toward higher computational throughput due to the inclusion of heterogeneous compute nodes. These multi-GPU nodes increase on-node computational efficiency, while also increasing the amount of data to be communicated and the number of potential data flow paths. In this work, we characterize the performance of irregular point-to-point communication with MPI on heterogeneous compute environments through performance modeling, demonstrating the limitations of standard communication strategies for both device-aware and staging-through-host communication techniques. Presented models suggest staging communicated data through host processes then using node-aware communication strategies for high inter-node message counts. Notably, the models also predict that node-aware communication utilizing all available CPU cores to communicate inter-node data leads to the most performant strategy when communicating with a high number of nodes. Furthermore, model validation is provided via a case study of irregular point-to-point communication patterns in distributed sparse matrix–vector products. Importantly, we include a discussion on the implications model predictions have on communication strategy design for emerging supercomputer architectures.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

I/O in Machine Learning Applications on HPC Systems: A 360-degree Survey

Growing interest in Artificial Intelligence (AI) has resulted in a surge in demand for faster methods of Machine Learning (ML) model training and inference. This demand for speed has prompted the use of high performance computing (HPC) systems that excel in managing distributed workloads. Because data is the main fuel for AI applications, the performance of the storage and I/O subsystem of HPC systems is critical. In the past, HPC applications accessed large portions of data written by simulations or experiments or ingested data for visualizations or analysis tasks. ML workloads perform small reads spread across a large number of random files. This shift of I/O access patterns poses several challenges to modern parallel storage systems. In this paper, we survey I/O in ML applications on HPC systems, and target literature within a 6-year time window from 2019 to 2024. We define the scope of the survey, provide an overview of the common phases of ML, review available profilers and benchmarks, examine the I/O patterns encountered during offline data preparation, training, and inference, and explore I/O optimizations utilized in modern ML frameworks and proposed in recent literature. Lastly, we seek to expose research gaps that could spawn further R&D.

97 MATHEMATICS AND COMPUTING↗