SEARCH · Engineering Papers

Results for “time to failure”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 649 records · Page 36

All sources NASA NTRS DOE OSTI

DOE OSTI2026

A Physics-Aligned Multi-Domain Machine Learning Framework for Time-Localised Diagnosis of Power Electronics Faults

This paper presents a physics-aligned framework for fault diagnosis in multi-phase power-electronic systems using cycle-synchronous windowing and multi-domain features derived from Fourier, wavelet, and Hilbert–Huang representations. While both logistic regression and multilayer perceptron (MLP) models achieve perfect performance under standard evaluation, blind unseen testing reveals a critical failure in a baseline MLP. This is shown to arise from model selection based on validation accuracy. Using validation-loss-based selection restores correct unseen performance and improves confidence. Feature ablation shows that Fourier and wavelet features dominate, while computational analysis indicates that feature extraction, particularly HHT, governs runtime.

Kumar, Praveen [ORNL] (ORCID:0000000291877857)↗

DOE OSTI2022

Long term performance of pilot methanogenic digester filled with seashell wastes treating slaughterhouse wastes: Biogas production and environmental impact

High ammonia and organic acids contents of decomposed slaughterhouse waste (SHW) often result in the severe inhibition or even failure of the anaerobic digestion (AD). The two-stage AD system can alleviate the ammonia inhibition; however, addition of alkali is often needed to maintain the neutral condition. In this study, seashell wastes containing high percentage of calcium carbonate (CaCO 3 ) were for the first time added in a pilot scale methanogenic digester treating chicken blood waste to retain the microorganisms and provide pH buffers for the substrate. A long-duration performance of this digester was conducted for 400 days under different organic loading rates (OLRs). The results show that methane yields of the digester were 244.8 and 224.9 mL g –1 COD added at OLR of 0.4 and 0.6 g COD L –1 d –1 , respectively, higher than the results of previous studies treating the similar feedstock. An AD system treating blood wastes of a typical poultry slaughterhouse was also proposed for which energy balance and carbon footprint analyses were conducted. The energy production of 0.35 MJ kg –1 blood and carbon emissions reduction of 0.05 kg CO 2 -eq kg –1 blood can be achieved from treating blood waste of the broiler chicken. Finally, this study reveals that using seashell wastes in an AD treating SHW can facilitate the reuse of aquaculture wastes and improve the sustainability of meat industry.

09 BIOMASS FUELS↗

DOE OSTI2026

Citizen science coupled with machine learning to quantify green-blue infrastructure cooling potential in Maricopa County, Arizona

Here, this study investigates the spatiotemporal cooling performance of green and blue infrastructure (GBI) in the Dobson Ranch urban neighborhood in Phoenix, Arizona. We leveraged citizen science near-surface (2 m) air temperature (Tair) measurements to train a highly accurate Tair predicting LightGBM machine learning model (R 2 : 0.986, MAE: 0.251 °C, RMSE: 0.585 °C). On June 16, 2024, the park area exhibited approximately 1 °C cooling effect (relative to the neighborhood mean) during both day and night. In contrast, the nearby artificial lake exhibited a stronger cooling effect of 2.4 °C during the day but a slight warming of 0.3 °C at night. At 00:00, locations 50 m downwind of the park were 0.3 °C warmer than the park, while locations 50 m upwind were 0.8 °C warmer. At 11:00, we observed that the downwind area is 0.8 °C cooler and the upwind area is 0.6 °C warmer—at the same 50 m distances relative to the park. We also observed 1 °C cooler and warmer effects respectively at the same 50 m downwind and upwind locations at 19:00 on June 17, 2024. Our data-driven analysis highlights potential limitations of car-traverse measurements, showing that failure to account for temporal variations during the traverse can lead to overestimation of Tair at night and underestimation during the day. Our analysis also showed only a weak correlation (coefficient: 0.48) between Landsat-derived land surface temperature (LST) and model predicted Tair at the time of the local Landsat overpass (∼11.00). This highlights the potential error of relying solely on LST for human thermal exposure analysis—particularly within the heterogenous built-environment.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2020

Initial development of a high temperature life prediction method directly accounting for variability in material properties

This report presents the initial development of a life prediction method that accounts for the variability of the material properties in Grade 91 steel. We account for material variability by both fitting a variable 3-parameter Weibull distribution to the rupture data of the material and by quantifying the effect that the creep deformation properties have on stresses via a Monte Carlo approach. To ensure a reasonable time frame when applying Monte Carlo to the creep stress, we propose a methodology that models the material as an extremely viscous Stokes flow with a non-Newtonian viscosity, therefore solving the problem in a single step. We use the final probabilistic model for structural failure developed in this document to evaluate a Flat Head Vessel designed with the Section III, Division 5 ASME Boiler Pressure and Vessel Code rules with a 100,000h design life and assess its probability of premature failure.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

DOE OSTI2021

Biomechanical Comparison of Anatomic Restoration of the Ulnar Footprint vs Traditional Ulnar Tunnels in Ulnar Collateral Ligament Reconstruction

Background: Current techniques for ulnar collateral ligament (UCL) reconstruction do not reproduce the anatomic ulnar footprint of the UCL. The purpose of this study was to describe a novel UCL reconstruction technique that utilizes proximal-to-distal ulnar bone tunnels to better re-create the anatomy of the UCL and to compare the biomechanical profile at time zero among this technique, the native UCL, and the traditional docking technique. Hypothesis: The biomechanical profile of the anatomic technique is similar to the native UCL and traditional docking technique. Study Design: Controlled laboratory study. Methods: Ten matched cadaveric elbows were potted with the forearm in neutral rotation. The palmaris longus tendon graft was harvested, and bones were sectioned 14 cm proximal and distal to the elbow joint. Specimen testing included (1) native UCL testing performed at 90° of flexion with 0.5 N·m of valgus moment preload, (2) cyclic loading from 0.5 to 5 N·m of valgus moment for 1000 cycles at 1 Hz, and (3) load to failure at 0.2 mm/s. Elbows then underwent UCL reconstruction with 1 elbow of each pair receiving the classic docking technique using either anatomic (proximal to distal) or traditional (anterior to posterior) tunnel locations. Specimen testing was then repeated as described. Results: There were no differences in maximum load at failure between the anatomic and traditional tunnel location techniques (mean ± SD, 34.90 ± 10.65 vs 37.28 ± 14.26 N·m; P = .644) or when including the native UCL (45.83 ± 17.03 N·m; P = .099). Additionally, there were no differences in valgus angle after 1000 cycles across the anatomic technique (4.58°± 1.47°), traditional technique (4.08°± 1.28°), and native UCL (4.07°± 1.99°). The anatomic group and the native UCL had similar valgus angles at failure (24.13°± 5.86° vs 20.13°± 5.70°; P = .083), while the traditional group had a higher valgus angle at failure when compared with the native UCL (24.88°± 6.18° vs 19.44°± 5.86°; P = .015). Conclusion: In this cadaveric model, UCL reconstruction with the docking technique utilizing proximal-to-distal ulnar tunnels better restored the ulnar footprint while providing valgus stability comparable with reconstruction with the docking technique using traditional anterior-to-posterior ulnar tunnel locations. These results suggest that utilization of the anatomic tunnel location in UCL reconstruction has similar biomechanical properties to the traditional method at the time of initial fixation (ie, not accounting for healing after reconstruction in vivo) while keeping the ulnar tunnels farther from the ulnar nerve. Further studies are warranted to determine if an anatomically based UCL reconstruction results in differing outcomes than traditional reconstruction techniques. Clinical Relevance: Current UCL reconstruction techniques do not accurately re-create the ulnar UCL footprint. The UCL is a dynamic constraint to valgus loads at the elbow, and a more anatomic reconstruction may afford more natural joint kinematics. This more anatomic technique performs similarly to the traditional docking technique at time zero, and the results of this study may offer a starting point for future in vivo studies.

Chang, Edward S.↗

DOE OSTI2021

DEM analysis of the effect of interface transition zone on dynamic splitting tensile behavior of high-strength concrete based on multi-phase model

Highlights: • Crushable aggregates with the real geometry shape are generated by using clump-cluster method. • Flat-Joint Model is adopted as the constitutive law. • The tensile strength is almost not affected by ITZ strength when its strength exceeds 0.8 times that of mortar. • A modified formula that can reflect the functional relationship between the ITZ strength and the TDIF is proposed. The tensile strength is one of the main parameters in the design of concrete structures and has an important influence on the cracking performance and durability of concrete materials. As the weak region in concrete, the strength of the interface transition zone (ITZ) directly affects the dynamic mechanical response of concrete. Meanwhile, considering that the tensile performance of concrete is far weaker than its compressive performance, the study of the strength of ITZ is of great significance to the dynamic tensile fracture characteristics and mechanical properties of concrete. In this paper, a method combining 3D scanning technology and Clump-Cluster was proposed to generate a real geometry shape model of crushable aggregate. On this basis, the 3D discrete element model (DEM) of high-strength concrete was constructed using the Flat-Joint Model (FJM). The effects of the ITZ strength on the failure mode, number of microcracks, tensile strength and strain rate effect of high-strength concrete under strain rates from 0.01/s to 100/s were studied by splitting tensile test. The simulation results indicate that the number of microcracks generated decreases with the increase of ITZ strength, and the proportion of ITZ cracks decreases obviously. When the ITZ strength exceeds the critical value of 0.8 times that of mortar, the tensile strength is almost not affected by ITZ strength. In addition, based on the numerical simulation results, a modified formula that can reflect the functional relationship between the ITZ strength and the dynamic increasing factor of the concrete tensile strength (TDIF) is proposed, and the rationality of the TDIF modified formula is verified.

36 MATERIALS SCIENCE↗

DOE OSTI2021

JWST Noise Floor. II. Systematic Error Sources in JWST NIRCam Time Series

The James Webb Space Telescope (JWST) holds great promise for characterizing atmospheres of transiting exoplanets, potentially providing insights into Earth-sized planets within the habitable zones of M-dwarf host stars if photon-limited performance can be achieved. Here, we discuss the systematic error sources that are expected to be present in grism time-series observations with the NIRCam instrument. We find that pointing jitter and high-gain antenna moves in addition to the detectors’ subpixel crosshatch patterns will produce relatively small variations (less than 6 parts per million, ppm). The time-dependent aperture losses due to thermal instabilities in the optics can also be kept to below 2 ppm. To achieve these low noise values, it is important to employ a sufficiently large (more than 11) extraction aperture. Persistence due to charge-trapping will have a minor (less than 3 ppm) effect on the time series 20 minutes into an exposure and is expected to play a much smaller role than it does for the Hubble Space Telescope WFC3 detectors. We expect detector temperature fluctuations to contribute less than 3 ppm. In total, our estimated noise floor from known systematic error sources is only 9 ppm per visit. We urge caution, however, because unknown systematic error sources could be present in flight and will only be measurable on astrophysical sources such as quiescent stars. We find that reciprocity failure may introduce a perennial instrument offset at the 40 ppm level, so corrections may be needed when a multi-instrument multi-observatory spectrum is stitched together over wide wavelength ranges.

47 OTHER INSTRUMENTATION↗

DOE OSTI2025

Machine learning for photovoltaic single axis tracker fault detection and classification

More than 81% of the annual capacity of utility-scale photovoltaic (PV) power plants in the U.S. use single-axis trackers (SATs) due to SATs delivering 4% in capacity factor on average over fixed-array systems. However, SATs are subject to faults, such as software misconfigurations and mechanical failures, resulting in suboptimal tracking. If left undetected, the overall power yield of the PV power plant is reduced significantly. Minimizing downtime and ensuring efficient operation of SATs requires robust detection and diagnosis mechanisms for SAT faults. We present a machine learning framework for implementing real-time SAT fault detection and classification. Our implementation of the proposed framework reliably identifies measurements taken from a test PV system undergoing emulated SAT faults relative to state-of-the-art algorithms and produces nearly zero false positives on our testing days. Code and data are available at https://pvpmc.sandia.gov/tools.

Fault classification↗

DOE OSTI2024

Resilient design in nuclear energy: Critical lessons from a cross-disciplinary analysis of the Fukushima Dai-ichi nuclear accident

This paper presents a multidisciplinary analysis of the Fukushima Dai-ichi Nuclear Power Plant accident. Along with the latest observations and simulation studies, we synthesize the time-series and event progressions during the accident across multiple disciplines, including in-plant physics and engineering systems, operators’ actions, emergency responses, meteorology, radionuclide release and transport, land contamination, and health impacts. We identify three key factors that exacerbated the consequences of the accident: (1) the failure of Unit 2 containment venting, (2) the insufficient integration of radiation measurements and meteorology data in the evacuation strategy, and (3) the limited risk assessment and emergency preparedness. We conclude with new research and development directions to improve the resilience of nuclear energy systems and communities, including (1) meteorology-informed proactive venting, (2) machine learning-enabled adaptive evacuation zones, and (3) comprehensive risk-informed emergency planning while leveraging the experience from responses to other disasters.

63 RADIATION, THERMAL, AND OTHER ENVIRON. POLLUTAN↗

DOE OSTI2026

CHESS 2025: Waveform LiDAR data from NEON AOP surveys

This dataset provides Level 1 (L1) full-waveform light detection and ranging (LiDAR) data collected for the 2025 Colorado Headwaters Ecological Spectroscopy Study (CHESS). These data were acquired to enable characterization of vegetation structure and other three-dimensional features of the land surface, and to evaluate structural changes that may have occurred between a prior LiDAR acquisition in 2018 and the 2025 overflight. Waveform LiDAR data can provide more detailed information about objects on the ground than discrete point clouds typically do, and they are often used for granular target segmentation and characterization of subcanopy vegetation. The data were acquired over three study domains in the Upper Gunnison river basin: the upper East River watershed (CRBU); Almont Triangle and Taylor Canyon (ALMO); and Upper Taylor River watershed (UPTA) between 2025-06-13 and 2025-07-15. LiDAR data were acquired using the Optech Galaxy Prime Airborne LiDAR Terrain Mapper onboard the National Ecological Observatory Network (NEON) Airborne Observation Platform (AOP). These are the primary waveform LiDAR data delivered by NEON and are provided per flightline in compressed Pulsewaves format, an open-source binary file standard. A Pulsewaves object comprises a two files: a pulse (.pls) file, which stores the geographic origin, outgoing vector, and metadata for every laser pulse emitted by the scanner, and a wave file (.wvs), which stores the sequential amplitude samples of the outgoing pulse and the returning signals. The files are published here in their compressed forms (.plz, .wvz). All waveform data were processed following the theoretical workflow described in the NEON L0-to-L1 Waveform LiDAR Algorithm Theoretical Basis Document (Krause and Goulden 2022a); however, the Pulsewaves output format differs from a legacy format described in that document. Waveform amplitude samples are recorded at 1 nanosecond intervals. All coordinates are provided in meters. Horizontal coordinates are referenced in Universal Transverse Mercator (UTM) zone 13N and the World Geodetic System (WGS) 1984 ensemble datum. Elevations are referenced to Geoid12A. Waveform data for the UPTA survey area were collected without incident and the published records are complete. However, both the ALMO and CRBU collections experienced issues that resulted in incomplete data for those areas. On collection day 2018-06-16 a hardware failure caused the waveform digitizer to lose data from the eastern edge of the ALMO site (Figure 22). The waveform data for flightlines 2–20 could not be extracted from the digitizer, and the data proved unrecoverable. As a result, a portion of the site does not have coverage with waveform data. Although no hardware failure was observed during collection over the CRBU area, final waveform files generated by vendor software contained only ~25% of the expected number of return pulses. After discovery, NEON initiated troubleshooting with the vendor. The root cause of the data ablation had not been identified at the time of publication. Additional data will be published in an update to this package if further recovery proves successful. CHESS Project Description: The Colorado Headwaters Ecological Spectroscopy Study (CHESS) comprised a multi-week airborne remote sensing and field observation campaign in the Upper Gunnison Basin, Colorado, conducted in June and July of 2025. Airborne remote sensing was conducted by the National Ecological Observatory Network Airborne Observation Platform (NEON AOP), concurrent with a field campaign run by the Rocky Mountain Biological Laboratory (RMBL), the Lawrence Berkeley National Laboratory (LBNL) and SLAC National Accelerator Laboratory Watershed Function Science Focus Area (SFA), and NASA-JPL (Jet Propulsion Laboratory) Earth Surface Mineral Dust Source Investigation (EMIT) program. Between June 10 and July 18, 2025, the NEON AOP flight team collected high-resolution aerial imaging spectroscopy and Light Detection and Ranging (LiDAR) data over three domains: the Upper East River (CRBU), Almont Triangle (ALMO), and the Upper Taylor Basin (UPTA). In coordination with the flights, a field campaign acquired ground-truth observations, including observations of vegetation composition, foliar traits, forest demography, and subsurface properties in 18 core sampling areas within the domains. Additional surface water observations were taken at over 380 point locations. All CHESS campaign datasets can be found within the CHESS ESS-DIVE data portal: https://data.ess-dive.lbl.gov/portals/chess. Funding Acknowledgement: Field and remote-sensing data acquisition was performed under a grant from the National Aeronautics and Space Administration (80NSSC24K1005). This work was also supported by the Watershed Function Science Focus Area at Lawrence Berkeley National Laboratory funded by the US Department of Energy, Office of Science, Biological and Environmental Research under Contract No. DE-AC02-05CH11231.

2018 NEON and 2025 CHESS Campaigns↗

DOE OSTI2021

Chamber Interior Viewing System (CIVS 2.0)

The NIF facility is used to conduct fusion and high-density scientific research. The experiments are conducted inside of a 10-meter diameter vacuum chamber called the “Target Chamber”. The target chamber is covered in ports to permit adding equipment to the exterior. Some of this equipment extends into the target chamber such as the target positioners and diagnostics. Other equipment sits on the surface and looks inward, such as the CIVS system. CIVS stands for Chamber Interior Viewing System. The CIVS system provides simple black and white digital images of the interior of the target chamber in real time. These images are used for positioner alignment, trouble shooting and observing for positioner collision. The views through these cameras need to be clear enough for the operators to safely conduct these tasks. One of the products of the experiments conducted in the target chamber are neutrons. These neutrons expand outward from the center of the target chamber with great energy. If the neutrons hit the sensor chip on a digital camera they will “kill” the sensor pixel where they contact, and the result will be a permanently bright white pixel on the image. Over time these “dead” pixels will build up and make the camera less and less usable providing a “staticky” image. The current solution of replacing the cameras when they become damaged is unsustainable due to both the cameras being out of production and the increased rate of camera failure based upon the increased number and energy of neutrons being generated. Current supplies of the correct camera type are dwindling with no source of replacement and the projected rate of replacement is unsustainable. Without a redesign, the function provided by the current CIVS system will be lost.

42 ENGINEERING↗

DOE OSTI2023

Arctic Amplification in the Community Earth System Models (CESM1 and CESM2)

We compare the Arctic amplification (AA) produced by the two Community Earth System Models CESM1 and CESM2, members of the CEMIP5 (Coupled Models Intercomparison Project phase 5) and CEMIP6 collections, respectively. We find that the CESM1 model reproduces the recent high values of the AA deduced from the observed temperature much better than the CESM2. The correlation coefficient within the 1970–2012 time period between CESM1-simulated AA and the observed one is 0.47, while the CESM2 simulation leads to an anticorrelation of r = –0.53. Even the more successful model (CESM1) is not able to reproduce recent high AA values of 4–5. The main cause of this failure is the model’s overestimate of the rate of increase in the mean global temperature in years post 1990. When the CESM1 model’s simulated trend of the mean global temperature is replaced in the expression for the AA by the observed temperature trend, the correlation coefficient increases from 0.47 to 0.75. The CESM1 model is among the best north American models in AA simulation while the CESM2 model is among the least successful.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2025

Machine Learning for Predicting Multipactor Susceptibility in Planar RF Structures

Multipactor discharge is a persistent challenge in high-power microwave (HPM) and accelerator systems, where secondary electron avalanches can cause heating, vacuum degradation, and failure. This work presents the first supervised machine learning (ML) framework for multipactor prediction, trained on high-fidelity 3D Particle-in-Cell (PIC) simulation data in planar geometries. The model maps operational, geometric, and material-dependent secondary electron yield (SEY) parameters to the time-averaged electron growth rate, enabling rapid reconstruction of susceptibility charts. Among the models evaluated, tree-based ensemble methods such as Random Forest and Extra Trees demonstrate superior generalization to unseen materials compared to neural networks such as multilayer perceptron (MLP). Performance metrics, including Intersection over Union (IoU), Structural Similarity Index Measure (SSIM), and Pearson correlation, show close agreement with simulation benchmarks. Principal Component Analysis attributes generalization limits to material feature-space disjointedness.

43 PARTICLE ACCELERATORS↗

DOE OSTI2021

Preventing stent failure using adaptive shear responsive endovascular implant

A stent apparatus, system, and method that senses wall shear stress by measuring fluid flow at localized areas within the stent, that processes measured information through an integrated circuit, and selectively sends power to mechanically controllable stent surfaces which results in localized geometric changes. In various embodiments the stent apparatus, system, and method sends data to outside the body in real time.

Mukerjee, Erik V.↗

DOE OSTI2024

Preventing stent failure using adaptive shear responsive endovascular implant

Mukerjee, Erik V.↗

DOE OSTI2020

Global flow structure and exact formal transseries of the Gubser flow in kinetic theory

In this work we introduce the generic conditions for the existence of a non-equilibrium attractor that is an invariant manifold determined by the long-wavelength modes of the physical system. We investigate the topological properties of the global flow structure of the Gubser flow for the Israel-Stewart theory and a kinetic model for the Boltzmann equation by employing Morse-Smale theory. We present a complete classification of the invariant submanifolds of the flow and determine all the possible flow lines connecting any pair of UV/IR fixed points. The formal transseries solutions to the Gubser dynamical system around the early-time (UV) and late-time (IR) fixed points are constructed and analyzed. It is proven that these solutions are purely perturbative (or power-law asymptotic) series with a finite radius of convergence. Based on these analyses, we find that Gubser-like expanding kinetic systems do not hydrodynamize owing to the failure of the hydrodynamization process which heavily relies on the classification of (non)hydrodynamic modes in the IR regime. This is in contrast to longitudinal boost-invariant plasmas where the asymptotic dynamics is described by a few terms of the hydrodynamic gradient expansion. We finally compare our results for both Bjorken and Gubser conformal kinetic models.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

DOE OSTI2025

Potentials of mean force fail to describe chemical bond-breaking in solution

Many liquid phase studies assume that the potential energy surfaces of reacting molecules are the same as in the gas phase, neglecting complex solvent dynamics that can completely alter the nature of chemical reactivity. Even studies that include solvent effects typically only consider them in an average, equilibrium way as part of a potential of mean force (PMF). In this work, we use mixed quantum/classical simulations to compare how equilibrium and non-equilibrium solvent motions affect the photodissociation of a simple diatomic molecule, NaK + , in liquid tetrahydrofuran. A PMF analysis shows that as the excited-state molecule dissociates with the solvent at equilibrium, the bonding electron remains associated with K + at short bond distances but eventually localizes on Na + at the end of dissociation. When we examine non-equilibrium dynamical photodissociation trajectories, however, we find that they fall into three distinct categories: about a quarter of them have the bonding electron mainly associated with Na + , another quarter stay mainly associated with K + , and about half have the bonding electron shared roughly equally between the two ions. The results show that equilibrium PMFs cannot accurately describe the dynamics of bond-breaking chemical reactions in solution because there is insufficient time for the solvent to reach equilibrium on the time scale over which bond dissociation occurs. Furthermore, our analysis shows that the solvent coupling between the electronic energy surfaces is similar at and away from equilibrium, suggesting that other factors, such as solute velocity-driven solvent memory effects, play a more important role in explaining the failure of the equilibrium PMF to predict the non-equilibrium dynamics.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2022

Hvac: Removing I/O Bottleneck for Large-Scale Deep Learning Applications

Scientific communities are increasingly adopting deep learning (DL) models in their applications to accelerate scientific discovery processes. However, with rapid growth in the computing capabilities of HPC supercomputers, large-scale DL applications have to spend a significant portion of training time performing I/O to a parallel storage system. Previous research works have investigated optimization techniques such as prefetching and caching. Unfortunately, there exist non-trivial challenges to adopting the existing solutions on HPC supercomputers for large-scale DL training applications, which include non-performance and/or failures at extreme scale, lack of portability and generality in design, complex deployment methodology, and being limited to a specific application or dataset. To address these challenges, we propose High-Velocity AI Cache (HVAC), a distributed read-cache layer that targets and fully exploits the node-local storage or near node-local storage technology. HVAC seamlessly accelerates read I/O by aggregating node-local or near node-local storage, avoiding metadata lookups and file locking while preserving portability in the application code. We deploy and evaluate HVAC on 1,024 nodes (with over 6000 NVIDIA V100 GPUS) of the Summit supercomputer. In particular, we evaluate the scalability, efficiency, accuracy, and load distribution of HVAC compared to GPFS and XFS-on-NVMe. With four different DL applications, we observe an average 25 % performance improvement atop GPFS and 9% drop against XFS-on-NVMe, which scale linearly and are considered the performance upper bound. We envision HVAC as an important caching library for upcoming HPC supercomputers such as Frontier.

Khan, Awais↗