Chasing the Rabbit: A Systematic Exploration of Software-Defined HPC storage
Explore the source record for details and available documents.
Computer science↗
SEARCH · Engineering Papers
Results for “HPC storage”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
An Efficient Storage-Driven Machine Learning Model for Performance in the Era of Multimodal Scientific Data
Scientific workflows are increasingly relying on machine learning (ML), simulation, and hybrid techniques to predict, understand, and optimize the behavior of complex experiments. High-performance computing has greatly improved researchers’ ability to acquire diverse data modalities in these workflows. Recent studies suggest that the performance of machine learning models can be improved by integrating data from various sources. Unfortunately, these workloads pose unprecedent pressure on the network storage to meet the demands associated with accessing these multimodal data. To mitigate the impact of intensive IO, we propose a solution that utilizes a multi-tier High-Performance Computing (HPC) distributed storage and data processing framework, placing computation where the data resides for better performance. By adopting this project, the scientific community will gain new opportunities to explore multimodal storage-driven possibilities, integrating multiple scientific data sources with advanced streaming frameworks. Additionally, our framework effectively utilizes computing resources and bridges the gaps identified by HPC experts. Our proposed approach tackles scalability and persistence challenges by leveraging native persistency, which has posed difficulties in traditional approaches. Furthermore, we seek to enhance fault-tolerance and load-balance of computations by leveraging real-time streaming in diverse scientific computing environments, thereby propelling advanced scientific computing research into the next generation.
97 MATHEMATICS AND COMPUTING↗
From Failure to Insight: Analyzing Disk Breakdowns in Large-Scale HPC Environments
Disk failure data provides valuable insights for preventing failures, enhancing storage robustness, guiding system design and deployment, and ensuring reliable operations at data centers. This paper introduces two disk failure datasets collected from large-scale HPC production environments over the past five years, comprising over 5,000 failure records from more than 40,000 disks. We analyzed these datasets across multiple dimensions, including temporal, spatial, and relational trends, and performed a comprehensive reliability assessment. Our analysis yielded numerous observations and insights that influence various operational aspects of HPC storage systems. We believe this study offers a holistic understanding of disk failure trends likely to interest the HPC storage community.
George, Anjus↗
Evolving Storage and Cyber Infrastructure at the NASA Center for Climate Simulation
This talk will describe recent developments at the NASA Center for Climate Simulation, which is funded by NASAs Science Mission Directorate, and supports the specialized data storage and computational needs of weather, ocean, and climate researchers, as well as astrophysicists, heliophysicists, and planetary scientists. To meet requirements for higher-resolution, higher-fidelity simulations, the NCCS augments its High Performance Computing (HPC) and storage retrieval environment. As the petabytes of model and observational data grow, the NCCS is broadening data services offerings and deploying and expanding virtualization resources for high performance analytics.
Cyberinfrastructure↗
Disk Failure Dataset from the Campaign Storage System
This dataset consists of 1,389 disk (HDD) failure events collected from the Campaign storage system at LANL. The Campaign system supported various compute platforms throughout its lifespan, including Cielo, Fire, Ice, and notably, the Trinity supercomputer. Each recorded event includes its detection timestamp (in ISO 8601 format) and details such as its location within the storage system—rack, enclosure, and drive slot number. The data, spanning from May 4, 2021, to July 25, 2023 (2 years, 2 months, and 22 days), represents failure events from the terminal years of Campaign's operational period, accounting for 26% of its total operational time.
97 MATHEMATICS AND COMPUTING↗
Improvement in HPC performance through HIPPI RAID storage
In 1986, RAID (redundant array of inexpensive (or independent) disks) technology was introduced as a viable solution to the I/O bottleneck. A number of different RAID levels were defined in 1987 by the Computer Science Division (EECS) University of California, Berkeley, each with specific advantages and disadvantages. With multiple RAID options available, taking advantage of RAID technology required matching particular RAID levels with specific applications. It was not possible to use one RAID device to address all applications. Maximum Strategy's Gen 4 Storage Server addresses this issue with a new capability called programmable RAID level partitioning. This capability enables users to have multiple RAID levels coexist on the same disks, thereby providing the versatility necessary for multiple concurrent applications.
Homan, Blake↗
A High-Quality Workflow for Multi-Resolution Scientific Data Reduction and Visualization
Multi-resolution methods such as Adaptive Mesh Refinement (AMR) can enhance storage efficiency for HPC applications generating vast volumes of data. However, their applicability is limited and cannot be universally deployed across all applications. Furthermore, integrating lossy compression with multi-resolution techniques to further boost storage efficiency encounters significant barriers. To this end, we introduce an innovative workflow that facilitates high-quality multi-resolution data compression for both uniform and AMR simulations. Initially, to extend the usability of multi-resolution techniques, our workflow employs a compression-oriented Region of Interest (ROI) extraction method, transforming uniform data into a multi-resolution format. Subsequently, to bridge the gap between multi-resolution techniques and lossy compressors, we optimize three distinct compressors, ensuring their optimal performance on multi-resolution data. These optimizations can improve the compression ratio of SOTA approaches by up to 3.3× under the same data quality loss. Lastly, we incorporate an advanced uncertainty visualization method into our workflow to understand the potential impacts of lossy compression. Experimental evaluation demonstrates that our workflow achieves significant compression quality improvements.
Wang, Daoce↗
Accelerating Application Bulk Synchronous Writes in HPC Environments
High-bandwidth storage tiers are becoming more common for their capability to absorb high-rate, bursty I/Os. Notably, the designs of these fast storage tiers differ from system to system. The variation of these layers and non-uniform methods of access can pose chal- lenges for applications seeking to run at multiple HPC facilities. Therefore, in this work, we present Spectral, a rapid-output ab- straction library to accelerate application, bulk-synchronous writes on HPC systems. We design Spectral to enable applications to use high-bandwidth storage, such as node-local storage and dis- tributed, write-caches (e.g., burst buffers) transparently without requiring modifications to the application or file system source code. The key idea is to allow applications to spend most of the time performing productive work and to not require any source code changes for maximum portability on different HPC archi- tectures. Spectral internally re-routes write-only files through available, high-performance I/O resources before ultimately mi- grating them to the shared global parallel file system. For instance, on Summit, Spectral transparently places application outputs on node-local storage and then utilizes asynchronous migration to the center-wide GPFS file system. We evaluate Spectral on the Summit HPC system (1024 nodes) using the IOR benchmark and real scientific applications. Spectral shows linear performance scaling, improving application write performance by over an order of magnitude when compared to GPFS.
Khan, Awais↗
I/O in Machine Learning Applications on HPC Systems: A 360-degree Survey
Growing interest in Artificial Intelligence (AI) has resulted in a surge in demand for faster methods of Machine Learning (ML) model training and inference. This demand for speed has prompted the use of high performance computing (HPC) systems that excel in managing distributed workloads. Because data is the main fuel for AI applications, the performance of the storage and I/O subsystem of HPC systems is critical. In the past, HPC applications accessed large portions of data written by simulations or experiments or ingested data for visualizations or analysis tasks. ML workloads perform small reads spread across a large number of random files. This shift of I/O access patterns poses several challenges to modern parallel storage systems. In this paper, we survey I/O in ML applications on HPC systems, and target literature within a 6-year time window from 2019 to 2024. We define the scope of the survey, provide an overview of the common phases of ML, review available profilers and benchmarks, examine the I/O patterns encountered during offline data preparation, training, and inference, and explore I/O optimizations utilized in modern ML frameworks and proposed in recent literature. Lastly, we seek to expose research gaps that could spawn further R&D.
97 MATHEMATICS AND COMPUTING↗
Workload-Adaptive Scheduling for Efficient Use of Parallel File Systems in High-Performance Computing Clusters
Results from work on feedback control of a queuing system for HPC jobs with shared storage resources.
Allan, Benjamin Andrew↗
Lustre Unveiled: Evolution, Design, Advancements, and Current Trends
The Lustre filesystem serves as a vital element in high-performance parallel storage, meeting the rising demands of scientific, research, and enterprise environments. Widely deployed across HPC environments, ranging from small-scale applications in AI/ML, to domains like oil and gas, drug discovery, and meteorology, and manufacturing, Lustre addresses the universal challenge of efficiently accessing vast and ever-increasing volumes of data. Lustre is the filesystem of choice on six out of the top 10 fastest supercomputers in the world today, over 65% of the top 100, and also for over 60% of the top 500. Despite its widespread popularity, there is a lack of a complete and up-to-date reference, covering Lustre’s evolution, design, and various advancements made over the years. In this journal, we aim to fill this gap by providing a comprehensive journey of Lustre, including its history with significant contributions to HPC, detailed architecture and design elements, exploration of advancements added through its evolution, and future directions. Additionally, we present a comparison of Lustre with other prominent storage technologies of the era. To illustrate the current state of Lustre, we analyze several filesystem trends, including utilization, performance, and usage patterns on Orion, the Lustre filesystem on the first exascale supercomputer Frontier. We hope that this journal serves as a comprehensive educational reference for the current and future generations interested in HPC filesystem storage aspects.
97 MATHEMATICS AND COMPUTING↗
Data Management in the Continuum: Cross-facility Object-based Data Transfers
Scientific workflows are evolving from relying on a monolithic storage subsystem at a single High-Performance Computing (HPC) facility to using geographically distributed file systems, repositories, and cloud storage. As a result, storing, accessing, transferring, and managing scientific data have become highly complex and prone to performance inefficiencies. This paper delves into these challenges by exploring an optimized end-to-end interface designed to seamlessly connect various local and remote storage systems, enabling efficient data movement of objects across HPC–Cloud and HPC–HPC environments. We showcase this capability through an object-focused data management runtime system, discuss the effects of relaxed consistency semantics in distributed object scenarios, and illustrate its application in an earthquake simulation workflow. Besides reducing the amount of data by selectively transferring regions of interest, our facility-local results achieved a speedup of 45 × over an optimized HDF5 usage and 15 × over the HDF5 with caching by using the new interface in PDC-XF.
Bez, Jean Luca↗
dCache project status and update
The dCache project delivers an open-source, massively scalable, distributed storage system deployed internationally to satisfy today’s scientists’ ever-demanding storage requirements. Its multifaceted approach supports different use cases with the same storage, from high throughput data ingest, data sharing over wide area networks, efficient access from HPC clusters, and longterm data persistence on tertiary storage. Even though dCache was initially developed for HEP experiments, today, it is used by various scientific communities, including astrophysics, biomed, and life science, each with their specific requirements. To match the needs of these new communities and keep up with the scaling demands of existing experiments, dCache is permanently evolving. With this contribution, we would like to highlight the recent developments in dCache regarding integration with CERN Tape Archive (CTA), advanced metadata handling, token-based authorization support, bulk API for QoS transitions, REST API to control interaction with the tape system, and future development directions.
Mkrtchyan, Tigran [DESY]↗
dCache: The Storage System of Choice for Data-Intensive Applications
The ever-increasing volumes of data produced by modern scientific facilities like EuXFEL and LHC put significant stress on data management infrastructure operated by laboratories and research centers. The challenges to be addressed span the entire data life cycle, from ingest and efficient data analysis to long-term preservation, typically involving large tape libraries. dCache, a storage system developed in collaboration between the Deutsches Elektronen-Synchrotron (DESY), Fermi National Accelerator Laboratory, and Nordic e-Infrastructure Collaboration (NeIC), is designed to manage a large number of disk servers and to facilitate transparent data migration to and from archival storage. Its multifaceted approach offers a unified method to support a variety of scientific use cases with the same storage infrastructure, including high-throughput data ingest, data sharing over wide area networks, efficient access from HPC clusters, and long-term data preservation on tertiary storage. Initially developed for high energy physics (HEP) experiments, dCache is now used by various scientific communities, including astrophysics, biomedical research, and life sciences, each having specific requirements. This paper presents architecture, deployment strategies, performance and scalability enhancements, and recent advancements in dCache addressing the needs of scientific communities. Finally, we touch on the development and release process, ensuring the software’s high quality.
DCache↗
High-Fidelity and High-Performance Computational Simulations for Rapid Design Optimization of Sulfur Thermal Energy Storage (CRADA Report)
NREL and Element 16 collaborated on sulfur thermal energy storage modeling using NREL’s high performance computing (HPC) resources to assist its application in industrial processes. Industrial process heat (IPH) accounts for ~70% of US manufacturing energy use and is primarily produced by fossil fuel combustion. Approximately, 1500 TWht (~60% Terawatt hour thermal) of IPH demand is in the temperature range of 100-300°C. Industrial applications in this temperature range include drying, hydrothermal processing, thermal enhanced oil recovery, food and beverage, bioethanol production, etc. Cost-effective thermal energy storage (TES) that increases the utilization of waste and renewable heat (solar, geothermal, etc.) could provide significant energy savings and reliable heat sources, decrease emissions, and increase US manufacturing competitiveness through reductions in fuel consumption. This HPC4EI project facilitated Element 16’s development of low-cost and high-impact molten sulfur TES for dispatchable IPH. The development of a high-fidelity model validated by experimental data and HPC simulations enabled the successful resolution of the complex interplay between fluid dynamics and heat transfer processes during transient operation of sulfur TES, overcoming the numerical challenges posed by the non-linear temperature-dependent physical properties of sulfur. The project helped accelerate Element 16’s molten sulfur TES product design and support its broad applications.
25 ENERGY STORAGE↗
Dynamic Analysis of the STARC-ABL Propulsion System
In the pursuit of Electrified Aircraft Propulsion (EAP), much of the attention is on the development of hybrid electric concept vehicles and their propulsion systems from a steady state performance perspective. While it is steady-state performance that largely determines the efficiency of civil air transports, engine operability and transient performance define constraints for the steady state design that impact efficiency and system viability. Neglecting dynamics and control technologies can result in an over-designed, sub optimal propulsion system or a concept that is not feasible. Thus, dynamic system studies were conducted on the propulsion system of the conceptual aircraft design known as the Single-aisle Turboelectric AiRCraft with Aft Boundary Layer propulsor (STARC-ABL). This paper describes the development of a controller to verify the baseline concept's feasibility from an operability perspective. Further, studies were conducted to identify excessive stability margin in the baseline design that could be traded for potential benefits in efficiency through an engine re design. This study revealed the potential to reduce the high pressure compressor (HPC) stall margin by 3%. Finally, a study was conducted to investigate the potential benefit of adding energy storage to the STARC-ABL concept that further improves operability and enables more gains in engine efficiency and performance. The energy storage provided an additional 0.5% stall margin can be removed from the HPC.
STARC-ABL↗
Dynamic Analysis of the STARC-ABL Propulsion System
In the pursuit of Electrified Aircraft Propulsion (EAP), much of the attention is on the development of hybrid electric concept vehicles and their propulsion systems from a steady state performance perspective. While it is steady-state performance that largely determines the efficiency of civil air transports, engine operability and transient performance define constraints for the steady state design that impact efficiency and system viability. Neglecting dynamics and control technologies can result in an over-designed, sub optimal propulsion system or a concept that is not feasible. Thus, dynamic system studies were conducted on the propulsion system of the conceptual aircraft design known as the Single-aisle Turboelectric AiRCraft with Aft Boundary Layer propulsor (STARC-ABL). This paper describes the development of a controller to verify the baseline concept's feasibility from an operability perspective. Further, studies were conducted to identify excessive stability margin in the baseline design that could be traded for potential benefits in efficiency through an engine re design. This study revealed the potential to reduce the high pressure compressor (HPC) stall margin by 3%. Finally, a study was conducted to investigate the potential benefit of adding energy storage to the STARC-ABL concept that further improves operability and enables more gains in engine efficiency and performance. The energy storage provided an additional 0.5% stall margin can be removed from the HPC.
STARC-ABL↗
Dynamic Analysis of the STARC-ABL Propulsion System
In the pursuit of Electrified Aircraft Propulsion (EAP), much of the attention is on the development of hybrid electric concept vehicles and their propulsion systems from a steady state performance perspective. While it is steady-state performance that largely determines the efficiency of civil air transports, engine operability and transient performance define constraints for the steady state design that impact efficiency and system viability. Neglecting dynamics and control technologies can result in an over-designed, sub optimal propulsion system or a concept that is not feasible. Thus, dynamic system studies were conducted on the propulsion system of the conceptual aircraft design known as the Single-aisle Turboelectric AiRCraft with Aft Boundary Layer propulsor (STARC-ABL). This paper describes the development of a controller to verify the baseline concept's feasibility from an operability perspective. Further, studies were conducted to identify excessive stability margin in the baseline design that could be traded for potential benefits in efficiency through an engine re design. This study revealed the potential to reduce the high pressure compressor (HPC) stall margin by 3%. Finally, a study was conducted to investigate the potential benefit of adding energy storage to the STARC-ABL concept that further improves operability and enables more gains in engine efficiency and performance. The energy storage provided an additional 0.5% stall margin can be removed from the HPC.
dynamic systems analysis↗