Engineering Papers⌕ Search

Engineering topics

Bent, John

Publications and source records attributed to Bent, John.

From Failure to Insight: Analyzing Disk Breakdowns in Large-Scale HPC Environments

Disk failure data provides valuable insights for preventing failures, enhancing storage robustness, guiding system design and deployment, and ensuring reliable operations at data centers. This paper introduces two disk failure datasets collected from large-scale HPC production environments over the past five years, comprising over 5,000 failure records from more than 40,000 disks. We analyzed these datasets across multiple dimensions, including temporal, spatial, and relational trends, and performed a comprehensive reliability assessment. Our analysis yielded numerous observations and insights that influence various operational aspects of HPC storage systems. We believe this study offers a holistic understanding of disk failure trends likely to interest the HPC storage community.

George, Anjus↗

Design Considerations and Analysis of Multi-Level Erasure Coding in Large-Scale Data Centers

Multi-level erasure coding (MLEC) has seen large deployments in the field, but there is no in-depth study of design considerations for MLEC at scale. In this paper, we provide comprehensive design considerations and analysis of MLEC at scale. We introduce the design space of MLEC in multiple dimensions, including various code parameter selections, chunk placement schemes, and various repair methods. We quantify their performance and durability, and show which MLEC schemes and repair methods can provide the best tolerance against independent/correlated failures and reduce repair network traffic by orders of magnitude. To achieve this, we use various evaluation strategies including simulation, splitting, dynamic programming, and mathematical modeling. We also compare the performance and durability of MLEC with other EC schemes such as SLEC and LRC and show that MLEC can provide high durability with higher encoding throughput and less repair network traffic over both SLEC and LRC.

Wang, Meng↗

Architecture and method for a burst buffer using flash technology

A parallel supercomputing cluster includes compute nodes interconnected in a mesh of data links for executing an MPI job, and solid-state storage nodes each linked to a respective group of the compute nodes for receiving checkpoint data from the respective compute nodes, and magnetic disk storage linked to each of the solid-state storage nodes for asynchronous migration of the checkpoint data from the solid-state storage nodes to the magnetic disk storage. Each solid-state storage node presents a file system interface to the MPI job, and multiple MPI processes of the MPI job write the checkpoint data to a shared file in the solid-state storage in a strided fashion, and the solid-state storage node asynchronously migrates the checkpoint data from the shared file in the solid-state storage to the magnetic disk storage and writes the checkpoint data to the magnetic disk storage in a sequential fashion.

97 MATHEMATICS AND COMPUTING↗

Method and apparatus for offloading compute resources to a flash co-processing appliance

Solid-State Drive (SSD) burst buffer nodes are interposed into a parallel supercomputing cluster to enable fast burst checkpoint of cluster memory to or from nearby interconnected solid-state storage with asynchronous migration between the burst buffer nodes and slower more distant disk storage. The SSD nodes also perform tasks offloaded from the compute nodes or associated with the checkpoint data. For example, the data for the next job is preloaded in the SSD node and very fast uploaded to the respective compute node just before the next job starts. During a job, the SSD nodes perform fast visualization and statistical analysis upon the checkpoint data. The SSD nodes can also perform data reduction and encryption of the checkpoint data.

97 MATHEMATICS AND COMPUTING↗

Mahanaxar: quality of service guarantees in high-bandwidth, real-time streaming data storage

Large radio telescopes, cyber-security systems monitoring real-time network traffic, and others have specialized data storage needs: guaranteed capture of an ultra-high-bandwidth data stream, retention of the data long enough to determine what is 'interesting,' retention of interesting data indefinitely, and concurrent read/write access to determine what data is interesting, without interrupting the ongoing capture of incoming data. Mahanaxar addresses this problem. Mahanaxar guarantees streaming real-time data capture at (nearly) the full rate of the raw device, allows concurrent read and write access to the device on a best-effort basis without interrupting the data capture, and retains data as long as possible given the available storage. It has built in mechanisms for reliability and indexing, can scale to meet arbitrary bandwidth requirements, and handles both small and large data elements equally well. Results from our prototype implementation shows that Mahanaxar provides both better guarantees and better performance than traditional file systems.

97 MATHEMATICS AND COMPUTING↗

...And Eat it Too: High Read Performance in Write-Optimized HPC I/O Middleware File Formats

As HPC applications run on increasingly high process counts on larger and larger machines, both the frequency of checkpoints needed for fault tolerance and the resolution and size of Data Analysis Dumps are expected to increase proportionally. In order to maintain an acceptable ratio of time spent performing useful computation work to time spent performing I/O, write bandwidth to the underlying storage system must increase proportionally to this increase in the checkpoint and computation size. Unfortunately, popular scientific self-describing file formats such as netCDF and HDF5 are designed with a focus on portability and flexibility. Extra care and careful crafting of the output structure and API calls is required to optimize for write performance using these APIs. To provide sufficient write bandwidth to continue to support the demands of scientific applications, the HPC community has developed a number of I/O middleware layers, that structure output into write-optimized file formats. However, the obvious concern with any write optimized file format would be a corresponding penalty on reads. In the log-structured filesystem, for example, a file generated by random writes could be written efficiently, but reading the file back sequentially later would result in very poor performance. Simulation results require efficient read-back for visualization and analytics, and though most checkpoint files are never used, the efficiency of a restart is very important in the face of inevitable failures. The utility of write speed improving middleware would be greatly diminished if it sacrificed acceptable read performance. In this paper we examine the read performance of two write-optimized middleware layers on large parallel machines and compare it to reading data natively in popular file formats. The two I/O middleware layer examined in this paper are the Adaptable IO System (ADIOS), a library-based approach developed at Oak Ridge National Laboratory to provide a high-level IO API that can be used in place of netCDF or HDF5 to do much more aggressive write-behind and efficient reordering of data locations within the file; and the Parallel Log-structured Filesystem (PLFS), a stackable FUSE filesystem approach developed at Los Alamos National Laboratory that decouples concurrent writes to improve the speed of checkpoints. Since ADIOS is an I/O componentization that affords selection of different I/O methods at or during runtime, through a single API, users can have access to MPI-IO, Posix-IO, HDF5, netCDF, and staging methods. The ADIOS BP file format is a new log-file format that has a superset of the features of both HDF5 and netCDF, but is designed to be portable and flexible while being optimized for writing. PLFS, on the other hand, is mounted as a stackable filesystem on top of an existing parallel filesystem. Reads or writes to the PLFS filesystem are transparently translated into operations on per-process log files stored in the underlying parallel filesystem. Since PLFS performs this translation without application modification, users can write in HDF5, netCDF, or app-specific file formats and PLFS will store the writes in a set of efficiently written log-formatted files, while presenting the user with a logical 'flat' file on reads. Despite their different approaches, the commonality behind both of these middleware systems is that they both write to a log file format. As shown in previous publications, writes are fully optimized in both systems, sometimes resulting in 100x improvements over writing data in popular file formats. But as mentioned above, writing is only one part of the story. In this paper we examine the read performance of our middleware layers on large parallel machines and compare these to reading data either natively or from other popular file formats. We compare the reading performance in two different scenarios: (1) Reading back restarts from the same number of processors as wrote the data and (2) Reading back restart data from a different number of processors. We observe that not only can write-optimized I/O middleware be built to not penalize read speeds, but for important workloads, techniques that improve write performance can, perhaps counterintuitively, improve read speeds over reading from a contiguously organized file format. In the remainder of this paper, we investigate this further through case studies of PLFS and ADIOS on simulation checkpoint restarts.

99 GENERAL AND MISCELLANEOUS↗

Data-driven batch schuduling

In this paper, we develop data-driven strategies for batch computing schedulers. Current CPU-centric batch schedulers ignore the data needs within workloads and execute them by linking them transparently and directly to their needed data. When scheduled on remote computational resources, this elegant solution of direct data access can incur an order of magnitude performance penalty for data-intensive workloads. Adding data-awareness to batch schedulers allows a careful coordination of data and CPU allocation thereby reducing the cost of remote execution. We offer here new techniques by which batch schedulers can become data-driven. Such systems can use our analytical predictive models to select one of the four data-driven scheduling policies that we have created. Through simulation, we demonstrate the accuracy of our predictive models and show how they can reduce time to completion for some workloads by as much as 80%.

97 MATHEMATICS AND COMPUTING↗