Engineering Papers⌕ Search

Engineering topics

Kordenbrock, Todd

Publications and source records attributed to Kordenbrock, Todd.

Viability of S3 Object Storage for the ASC Program at Sandia

Recent efforts at Sandia such as DataSEA are creating search engines that enable analysts to query the institution’s massive archive of simulation and experiment data. The benefit of this work is that analysts will be able to retrieve all historical information about a system component that the institution has amassed over the years and make better-informed decisions in current work. As DataSEA gains momentum, it faces multiple technical challenges relating to capacity storage. From a raw capacity perspective, data producers will rapidly overwhelm the system with massive amounts of data. From an accessibility perspective, analysts will expect to be able to retrieve any portion of the bulk data, from any system on the enterprise network. Sandia’s Institutional Computing is mitigating storage problems at the enterprise level by procuring new capacity storage systems that can be accessed from anywhere on the enterprise network. These systems use the simple storage service, or S3, API for data transfers. While S3 uses objects instead of files, users can access it from their desktops or Sandia’s high-performance computing (HPC) platforms. S3 is particularly well suited for bulk storage in DataSEA, as datasets can be decomposed into object that can be referenced and retrieved individually, as needed by an analyst. In this report we describe our experiences working with S3 storage and provide information about how developers can leverage Sandia’s current systems. We present performance results from two sets of experiments. First, we measure S3 throughput when exchanging data between four different HPC platforms and two different enterprise S3 storage systems on the Sandia Restricted Network (SRN). Second, we measure the performance of S3 when communicating with a custom-built Ceph storage system that was constructed from HPC components. Overall, while S3 storage is significantly slower than traditional HPC storage, it provides significant accessibility benefits that will be valuable for archiving and exploiting historical data. There are multiple opportunities that arise from this work, including enhancing DataSEA to leverage S3 for bulk storage and adding native S3 support to Sandia’s IOSS library.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Lightweight File System (LWFS) v. 1.0

The Lightweight File System (LWFS) is a storage system that provides a minimal set of I/O-system functionality required by file system and/or I/O library implementations for massively parallel machines. In particular, the LWFS-core consists of a scalable security model, an efficient data-movement protocol, and a direct interface to object-based storage devices. Higher-level services such as namespace management, consistency semantics, reliability, and so forth are layered on top of the core services to provide application-specific functionality as needed. The LWFS code contains implementations of the core services and reference implementations of a number of supplemental services for namespace management and transaction support.

Oldfield, Ron↗

Trilinos I/O Support (Trios)

Trilinos I/O Support (Trios) is a new capability area in Trilinos that serves two important roles: (1) it provides and supports I/O libraries used by in-production scientific codes; (2) it provides a research vehicle for the evaluation and distribution of new techniques to improve I/O on advanced platforms. This paper provides a brief overview of the production-grade I/O libraries in Trios as well as some of the ongoing research efforts that contribute to the experimental libraries in Trios.

Oldfield, Ron A.↗

Trilinos I/O Support (Trios)

Trilinos I/O Support (Trios) is a new capability area in Trilinos that serves two important roles: (1) it provides and supports I/O libraries used by in-production scientific codes; (2) it provides a research vehicle for the evaluation and distribution of new techniques to improve I/O on advanced platforms. This paper provides a brief overview of the production-grade I/O libraries in Trios as well as some of the ongoing research efforts that contribute to the experimental libraries in Trios.

97 MATHEMATICS AND COMPUTING↗

Managing variability in the IO performance of petascale storage systems.

Significant challenges exist for achieving peak or even consistent levels of performance when using IO systems at scale. They stem from sharing IO system resources across the processes of single large-scale applications and/or multiple simultaneous programs causing internal and external interference, which in turn, causes substantial reductions in IO performance. This paper presents interference effects measurements for two different file systems at multiple supercomputing sites. These measurements motivate developing a 'managed' IO approach using adaptive algorithms varying the IO system workload based on current levels and use areas. An implementation of these methods deployed for the shared, general scratch storage system on Oak Ridge National Laboratory machines achieves higher overall performance and less variability in both a typical usage environment and with artificially introduced levels of 'noise'. The latter serving to clearly delineate and illustrate potential problems arising from shared system usage and the advantages derived from actively managing it.

99 GENERAL AND MISCELLANEOUS↗

Trios

The Trilinos I/O support (Trios) software provides general I/O support for applications using Trilinos libraries. In particular, this software will include parallel I/O support to read and write selected Trilinos data structures to commonly-used file formats such as netCDF, hdf5, and Exodus.

Oldfield, RonA.↗