Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “storage and I/O”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3

Spaceborne optical disk controller development

The current status and potential applications of an optical-disk buffer (ODB) memory system being developed by an interagency consortium including NASA and the USAF are reviewed. The design goals for the ODB include usable capacity 1 Tb, maximum data rate 1.6 Gb/s, read error rate less than 10 to the -12th, time to initial access less than 100 ms, and unlimited read/write cycles. Present efforts focus on a brassboard ODB which employs 12 14-inch magnetooptic disks and 24 nine-diode read/write heads. A typical space application of an optical disk mass memory system (ODMMS) is discussed: as communications buffer, temporary storage, and/or multiuser I/O buffer for data management on the Space Station Earth Observing System. Environmental, operational, system-architecture, and functional-separation factors; critical design issues; and standardization questions for spaceborne ODMMSs are examined in detail.

Shull, Thomas A.↗

Functional requirements document for the Earth Observing System Data and Information System (EOSDIS) Scientific Computing Facilities (SCF) of the NASA/MSFC Earth Science and Applications Division, 1992

Five scientists at MSFC/ESAD have EOS SCF investigator status. Each SCF has unique tasks which require the establishment of a computing facility dedicated to accomplishing those tasks. A SCF Working Group was established at ESAD with the charter of defining the computing requirements of the individual SCFs and recommending options for meeting these requirements. The primary goal of the working group was to determine which computing needs can be satisfied using either shared resources or separate but compatible resources, and which needs require unique individual resources. The requirements investigated included CPU-intensive vector and scalar processing, visualization, data storage, connectivity, and I/O peripherals. A review of computer industry directions and a market survey of computing hardware provided information regarding important industry standards and candidate computing platforms. It was determined that the total SCF computing requirements might be most effectively met using a hierarchy consisting of shared and individual resources. This hierarchy is composed of five major system types: (1) a supercomputer class vector processor; (2) a high-end scalar multiprocessor workstation; (3) a file server; (4) a few medium- to high-end visualization workstations; and (5) several low- to medium-range personal graphics workstations. Specific recommendations for meeting the needs of each of these types are presented.

Botts, Michael E.↗

Defining and satisfying the computing requirements of the EOS scientific computing facilities at NASA/MSFC

The computing requirements of the individual scientific computing facilities (SCFs) have been evaluated. The requirements under consideration include CPU-intensive vector and scalar processing, visualization, data storage, connectivity, and I/O peripherals. A hierarchy consisting of shared and individual resources is proposed to most effectively meet the total SCF computing requirements. This hierarchy includes a supercomputer class vector processor; a high-end scalar multiprocessor workstation; a high-speed, large capacity file server; a few medium to high-end visualization workstations; and low to medium range personal graphics workstations.

Botts, Michael E.↗

Incorporating Oracle on-line space management with long-term archival technology

The storage requirements of today's organizations are exploding. As computers continue to escalate in processing power, applications grow in complexity and data files grow in size and in number. As a result, organizations are forced to procure more and more megabytes of storage space. This paper focuses on how to expand the storage capacity of a Very Large Database (VLDB) cost-effectively within a Oracle7 data warehouse system by integrating long term archival storage sub-systems with traditional magnetic media. The Oracle architecture described in this paper was based on an actual proof of concept for a customer looking to store archived data on optical disks yet still have access to this data without user intervention. The customer had a requirement to maintain 10 years worth of data on-line. Data less than a year old still had the potential to be updated thus will reside on conventional magnetic disks. Data older than a year will be considered archived and will be placed on optical disks. The ability to archive data to optical disk and still have access to that data provides the system a means to retain large amounts of data that is readily accessible yet significantly reduces the cost of total system storage. Therefore, the cost benefits of archival storage devices can be incorporated into the Oracle storage medium and I/O subsystem without loosing any of the functionality of transaction processing, yet at the same time providing an organization access to all their data.

Moran, Steven M.↗

RH1020 Single Event Clock Upset Summary Report

This report summarizes the testing and analysis of "single event clock upset' in the RH1020. Also included are SEU-rate predictions and design recommendations for risk analysis and reduction. The subject of "upsets" in the RH1020 is best understood by using a model consisting of a global clock buffer and a D-type flip-flop as the basic memory unit. The RH1020 is built on the ACT 1 family architecture. As such, it has one low-skew global clock buffer with a TTL-level input threshold that is accessed via a single dedicated pin. The clock signal is driven to full CMOS levels, buffered, and sent to individual row buffers with one buffer per channel. For low-skew performance, the outputs of all of the RH1020 row buffers are shorted together via metal lines, as is done in the A1020B. All storage in the RH1020 consists of routed flip-flops, constructed with multiplexors and feedback through the routing segments. A simple latch can be constructed from a single (combinatorial or C) module; an edge-triggered flip-flop is constructed using two concatenated latches. There is no storage in the I/O modules. The front end of the clock buffering circuitry, at a common point relative to the row buffer, is a sub-circuit that was determined to be the most susceptible to heavy ions. This is due, in part, to its smaller transistors compared to the rest of the circuitry. This conclusion is also supported by SPICE simulations and an analysis of the heavy ion data, described in this report. The edge triggered D flip-flop has two single-event-upset modes. Mode one, called C-module upset, is caused by a heavy ion striking the C-module's sensitive area on the silicon and produces a soft single bit error at the output of the flip-flop. Mode two, called clock upset, is caused by a heavy ion strike on the clock buffer, generating a runt pulse interpreted as a false clock signal and consequently producing errors at the flip-flop outputs. C-module upset sensitivity in the RH1020 is essentially the same as that of its ACT 1 siblings (A1020, A1020A and A1020B), which were well tested, analyzed, and documented in the literature.

Katz, Richard B.↗

Privateer

Privateer is a general-purpose data store that optimizes the tradeoff between storage space utilization and I/O performance. Privateer uses memory-mapped I/O with private mapping and an optimized writeback mechanism to maximize write parallelism and eliminate redundant writes; it also uses contentaddressable storage to optimize storage space via de-duplication.

Iwabuchi, Keita↗

HDF5 in the exascale era: Delivering efficient and scalable parallel I/O for exascale applications

Accurately modeling real-world systems requires scientific applications at exascale to generate massive amounts of data and manage data storage efficiently. However, parallel input and output (I/O) faces challenges due to new application workflows and the state-of-the-art memory, interconnect, and storage architectures considered in exascale designs. The storage hierarchy has expanded with node-local persistent memory, solid-state storage, and traditional disk and tape-based storage, thus requiring efficiency at each layer and much more efficient data movement among these layers. This paper discusses how the ExaHDF5 project improved the I/O performance and data management for exascale architectures by enhancing HDF5, a widely used parallel I/O library. The team developed an Asynchronous I/O Virtual Object Layer (VOL) connector that allowed overlapping I/O with computation. They also created a Cache VOL to complement asynchronous I/O by incorporating fast storage layers, such as burst buffer and node-local storage, into the parallel I/O workflow through caching and staging data. Additionally, the team enabled data aggregation and I/O at the node level by using a Subfiling Virtual File Driver (VFD). To demonstrate superior I/O performance with HDF5 at exascale, the ExaHDF5 team collaborated with several exascale applications. In this paper, we show I/O performance improvements for three applications: Cabana (a particle-based simulation library), EQSIM (a regional earthquake simulation software), and E3SM (a climate system modeling library).

Asynchronous I/Ol↗

Integration of RNTuple in ATLAS Athena

After using ROOT’s TTree I/O subsystem for over two decades and storing more than an exabyte of compressed High Energy Physics (HEP) data, advances in technology have motivated a complete redesign, RNTuple, which breaks backward-compatibility to take better advantage of these storage options. The RNTuple I/O subsystem has been designed to address performance bottlenecks and other shortcomings of TTree. Specifically, RNTuple comes with an updated, more compact binary data format that can be stored both in ROOT files and natively in object stores. It is designed for modern storage hardware (e.g. high-throughput low-latency NVMe SSDs), and provides robust and easy to use interfaces. The binary format of RNTuple is scheduled to become production grade in 2024, and recently has become mature enough to start exploring the integration into software used by HEP experiments. In this contribution, we discuss the developments to support the features as required by the ATLAS analysis Event Data Model (EDM) in RNTuple, which will enable its integration into the Athena software framework. With these developments in place, we evaluate the performance of the current most recent versions of RNTuple-based ATLAS data sets and compare this to that of TTree.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

ROOT I/O compression improvements for HEP analysis

We overview recent changes in the ROOT I/O system, increasing performance and enhancing it and improving its interaction with other data analysis ecosystems. Both the newly introduced compression algorithms, the much faster bulk I/O data path, and a few additional techniques have the potential to significantly to improve experiment's software performance. The need for efficient lossless data compression has grown significantly as the amount of HEP data collected, transmitted, and stored has dramatically increased during the LHC era. While compression reduces storage space and, potentially, I/O bandwidth usage, it should not be applied blindly: there are significant trade-offs between the increased CPU cost for reading and writing files and the reduce storage space.

Shadura, Oksana↗

Improving I/O Performance for Exascale Applications through Online Data Layout Reorganization

The applications being developed within the U.S. Exascale Computing Project (ECP) to run on imminent Exascale computers will generate scientific results with unprecedented fidelity and record turn-around time. Many of these codes are based on particle-mesh methods and use advanced algorithms, especially dynamic load-balancing and mesh-refinement, to achieve high performance on Exascale machines. Yet, as such algorithms improve parallel application efficiency, they raise new challenges for I/O logic due to their irregular and dynamic data distributions. Thus, while the enormous data rates of Exascale simulations already challenge existing file system write strategies, the need for efficient read and processing of generated data introduces additional constraints on the data layout strategies that can be used when writing data to secondary storage. We review these I/O challenges and introduce two online data layout reorganization approaches for achieving good tradeoffs between read and write performance. We demonstrate the benefits of using these two approaches for the ECP particle-in-cell simulation WarpX, which serves as a motif for a large class of important Exascale applications. Here, we show that by understanding application I/O patterns and carefully designing data layouts we can increase read performance by more than 80 percent.

97 MATHEMATICS AND COMPUTING↗

Timely Reporting of Heavy Hitters Using External Memory

Given an input stream S of size N, a Φ-heavy hitter is an item that occurs at least ΦN times in S. The problem of finding heavy-hitters is extensively studied in the database literature. In this work, we study a real-time heavy-hitters variant in which an element must be reported shortly after we see its T = Φ N-th occurrence (and hence it becomes a heavy hitter). We call this the Timely Event Detection (TED) Problem. The TED problem models the needs of many real-world monitoring systems, which demand accurate (i.e., no false negatives) and timely reporting of all events from large, high-speed streams with a low reporting threshold (high sensitivity). Like the classic heavy-hitters problem, solving the TED problem without false-positives requires large space (Ω (N) words). Thus in-RAM heavy-hitters algorithms typically sacrifice accuracy (i.e., allow false positives), sensitivity, or timeliness (i.e., use multiple passes). We show how to adapt heavy-hitters algorithms to external memory to solve the TED problem on large high-speed streams while guaranteeing accuracy, sensitivity, and timeliness. Our data structures are limited only by I/O-bandwidth (not latency) and support a tunable tradeoff between reporting delay and I/O overhead. With a small bounded reporting delay, our algorithms incur only a logarithmic I/O overhead. We implement and validate our data structures empirically using the Firehose streaming benchmark. Multi-threaded versions of our structures can scale to process 11M observations per second before becoming CPU bound. In comparison, a naive adaptation of the standard heavy-hitters algorithm to external memory would be limited by the storage device’s random I/O throughput, i.e., ≈100K observations per second.

97 MATHEMATICS AND COMPUTING↗

Optimizing Metadata Exchange: Leveraging DAOS for ADIOS Metadata I/O

In HPC I/O middleware like the Adaptable I/O System (ADIOS) often mediates data transfers between applications. The metadata I/O generated by such systems often presents significant scaling and performance limitations. This work seeks improvement opportunities for metadata I/O by leveraging the DAOS storage systems, a recent storage system solution deployed on high-end systems such as the Aurora supercomputer. We investigate the tradeoffs and the design space for integrating I/O engines for the ADIOS middleware based on the different storage mechanisms supported by DAOS. We present a new DAOS-Array-ChunkSize-aligned engine which provides up to 2.3× improved performance than when using the existing DAOS-POSIX interface, without requiring any application modifications.

Venkatesh, Ranjan Sarpangala↗

On-the-fly data set combinations with RNTuple

With the expected data volume increase for HL-LHC and the even more complex computing challenges set by future colliders, the need for efficient data storage and processing becomes more pressing. ROOT’s next-generation data format and I/O subsystem, RNTTuple, is designed to address these challenges. RNTTuple already demonstrates a clear improvement in storage and I/O efficiency, as well as overall stability and robustness with respect to its predecessor, TTTree. These improvements provide a solid baseline to introduce novel extensions to common high-energy and nuclear physics (HENP) workflows. Notably, many workflows could benefit from the ability to arbitrarily join and chain data set samples at runtime, which could reduce overall storage requirements and improve application runtime and ergonomics. In this paper, we present the RNTupleProcessor, which enables HENP data set combinations with RNTuple. We will discuss the main design considerations, present the interfaces to support data set combinations and show how they integrate in typical workflows.

de Geus, Florine Willemijn [CERN; Twente U., Ensch↗

Mochi: Composing Data Services for High-Performance Computing Environments

Technology enhancements and the growing breadth of application workflows running on high-performance computing (HPC) platforms drive the development of new data services that provide high performance on these new platforms, provide capable and productive interfaces and abstractions for a variety of applications, and are readily adapted when new technologies are deployed. The Mochi framework enables composition of specialized distributed data services from a collection of connectable modules and subservices. Rather than forcing all applications to use a one-size-fits-all data staging and I/O software configuration, Mochi allows each application to use a data service specialized to its needs and access patterns. This paper introduces the Mochi framework and methodology. The Mochi core components and microservices are described. Examples of the application of the Mochi methodology to the development of four specialized services are detailed. Finally, a performance evaluation of a Mochi core component, a Mochi microservice, and a composed service providing an object model is performed. The paper concludes by positioning Mochi relative to related work in the HPC space and indicating directions for future work.

97 MATHEMATICS AND COMPUTING↗

Machine Learning Assisted HPC Workload Trace Generation for Leadership Scale Storage Systems

Monitoring and analyzing a wide range of I/O activities in an HPC cluster is important in maintaining mission-critical performance in a large-scale, multi-user, parallel storage system. Center-wide I/O traces can provide high-level information and fine-grained activities per application or per user running in the system. Studying such large-scale traces can provide helpful insights into the system. It can be used to develop predictive methods for making predictive decisions, adjusting scheduling policies, or providing decisions for the design of next-generation systems. However, sharing real-world I/O traces to expedite such research efforts leaves a few concerns; i) the cost of sharing the large traces is expensive due to this large size, and ii) privacy concern is an issue.We address such issues by building an end-to-end machine learn- ing (ML) workflow that can generate I/O traces for large-scale HPC applications. We leverage ML based feature selection and gener- ative models for I/O trace generation. The generative models are trained on I/O traces collected by the darshan I/O characterization tool over a period of one year. We present a two-step generation process consisting of two deep-learning models, called the feature generator and the trace generator. The combination of two-step generative models provides robustness by reducing the bias of the model and accounting for the stochastic nature of the I/O traces across different runs of an application. We evaluate the performance of the generative models and show that the two-step model can generate time-series I/O traces with less than 20% root mean square error.

Paul, Arnab↗

In Situ Machine Learning for Intelligent Data Capture on Exascale Platforms. Final Report

In many dynamic systems, interesting events occur locally in time and space. Examples of such systems include ignition events in combustion simulations, material fractures in mechanics simulations, and extreme weather events in climate simulations. Due to memory constraints and data I/O costs, current simulation workflows save data at regularly spaced time-steps, at a fixed rate determined before the start of the simulation. Often this mode of operation results in missed events of interest, necessitating a simulation restart from before an event occurred with more frequent data saves. This data saving workflow is grossly inefficient and is already a bottleneck in the computing process. We propose to develop machine learning algorithms that can detect when interesting dynamical events are occurring, triggering data saves. These machine learning algorithms will perform in situ anomaly detection to flag regions with different dynamical properties than those previously recorded. The adaptive data saves would be local in time and space to match the event of interest, thereby enabling a much more efficient workflow that will reduce data I/O costs and data storage memory requirements. The algorithms will be tested on two applications: auto-ignition simulations and climate simulations. A critical component of this project will be developing machine learning algorithms that can be deployed efficiently in situ on HPC platforms with out-of-the-box functionality. The development of in situ machine learning methods to detect anomalous events would enable a more efficient and effective workflow, in which all the relevant data are saved in a single simulation run, without re-starts or scientist intervention.

42 ENGINEERING↗

Large space structures control algorithm characterization

Feedback control algorithms are developed for sensor/actuator pairs on large space systems. These algorithms have been sized in terms of (1) floating point operation (FLOP) demands; (2) storage for variables; and (3) input/output data flow. FLOP sizing (per control cycle) was done as a function of the number of control states and the number of sensor/actuator pairs. Storage for variables and I/O sizing was done for specific structure examples.

Fogel, E.↗

I/O Error Analyzer (UNIVAC 1108 Version)

IOALZ4 is an Assembly-language utility program for UNIVAC 1108, operational under level 33 of EXEC 8 operating system. It scans user-selected portions of system log file, whether located on tape or mass storage, searching for and processing I/O error entries.

Vaughan, E. T.↗