Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “I/O performance”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 361 records · Page 20
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

SZ3: A Modular Framework for Composing Prediction-Based Error-Bounded Lossy Compressors

Today's scientific simulations require a significant reduction of data volume because of extremely large amounts of data they produce and the limited I/O bandwidth and storage space. Here, error-bounded lossy compression has been considered one of the most effective solutions to the above problem. In practice, however, the best-fit compression method often needs to be customized or optimized in particular because of diverse characteristics in different datasets and various user requirements on the compression quality and performance. In this paper, we address this issue with a novel modular, composable compression framework named SZ3. Our contributions are four-folds. (1) We develop SZ3 which features an innovative modular abstraction for the prediction-based compression framework, such that compression modules can be plugged in easily to create new compressors based on characteristics of data and user requirements. (2) We create a new compression pipeline by SZ3 for GAMESS data, which significantly improves the compression ratios over state-of-the-art compressors. (3) We develop an adaptive compression pipeline by SZ3 for APS data with minimal efforts, which leads to the best rate-distortion among all existing error-bounded lossy compressors for any bit-rate. (4) We compare the sustainability of SZ3 with leading error-bounded prediction-based compressors, and then demonstrate the necessity of diverse pipelines by integrating and evaluating several compression pipelines on diverse scientific datasets from multiple disciplines. Experiments show that SZ3 incurs very limited overhead in compressor integration and our customized compression pipelines lead to up to 20% improvement in compression ratios under the same data distortion, when compared with the best existing approach.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2011

MicroShell Minimalist Shell for Xilinx Microprocessors

MicroShell is a lightweight shell environment for engineers and software developers working with embedded microprocessors in Xilinx FPGAs. (MicroShell has also been successfully ported to run on ARM Cortex-M1 microprocessors in Actel ProASIC3 FPGAs, but without project-integration support.) Micro Shell decreases the time spent performing initial tests of field-programmable gate array (FPGA) designs, simplifies running customizable one-time-only experiments, and provides a familiar-feeling command-line interface. The program comes with a collection of useful functions and enables the designer to add an unlimited number of custom commands, which are callable from the command-line. The commands are parameterizable (using the C-based command-line parameter idiom), so the designer can use one function to exercise hardware with different values. Also, since many hardware peripherals instantiated in FPGAs have reasonably simple register-mapped I/O interfaces, the engineer can edit and view hardware parameter settings at any time without stopping the processor. MicroShell comes with a set of support scripts that interface seamlessly with Xilinx's EDK tool. Adding an instance of MicroShell to a project is as simple as marking a check box in a library configuration dialog box and specifying a software project directory. The support scripts then examine the hardware design, build design-specific functions, conditionally include processor-specific functions, and complete the compilation process. For code-size constrained designs, most of the stock functionality can be excluded from the compiled library. When all of the configurable options are removed from the binary, MicroShell has an unoptimized memory footprint of about 4.8 kB and a size-optimized footprint of about 2.3 kB. Since MicroShell allows unfettered access to all processor-accessible memory locations, it is possible to perform live patching on a running system. This can be useful, for instance, if a bug is discovered in a routine but the system cannot be rebooted: Shell allows a skilled operator to directly edit the binary executable in memory. With some forethought, MicroShell code can be located in a different memory location from custom code, permitting the custom functionality to be overwritten at any time without stopping the controlling shell.

Werne, Thomas A.↗
NASA NTRS1988

Field study of communication and workload in police helicopters - Implications for AI cockpit design

This paper reports on the work performed by civilian helicopter crews, using audio and video recordings and a variety of workload measures (heart rate and subjective ratings) obtained in a field study of public service helicopter missions. The number and frequency of communications provided a significant source of workload. This is relevant to the design of automated cockpit systems, since many designs presuppose the use of voice I/O systems. Fluency of communications (including pauses, hesitation markers, repetitions, and false starts) furnished an early indication of the effects of fatigue. Three workload measures were correlated to identify high workload segments of flight, and to suggest alternate task allocations between crew members.

Linde, Charlotte↗
NASA NTRS1977

A simulation model for wind energy storage systems. Volume 1: Technical report

A comprehensive computer program for the modeling of wind energy and storage systems utilizing any combination of five types of storage (pumped hydro, battery, thermal, flywheel and pneumatic) was developed. The level of detail of Simulation Model for Wind Energy Storage (SIMWEST) is consistent with a role of evaluating the economic feasibility as well as the general performance of wind energy systems. The software package consists of two basic programs and a library of system, environmental, and load components. The first program is a precompiler which generates computer models (in FORTRAN) of complex wind source storage application systems, from user specifications using the respective library components. The second program provides the techno-economic system analysis with the respective I/O, the integration of systems dynamics, and the iteration for conveyance of variables. SIMWEST program, as described, runs on the UNIVAC 1100 series computers.

Warren, A. W.↗
NASA NTRS1996

The Galley Parallel File System

Most current multiprocessor file systems are designed to use multiple disks in parallel, using the high aggregate bandwidth to meet the growing I/0 requirements of parallel scientific applications. Many multiprocessor file systems provide applications with a conventional Unix-like interface, allowing the application to access multiple disks transparently. This interface conceals the parallelism within the file system, increasing the ease of programmability, but making it difficult or impossible for sophisticated programmers and libraries to use knowledge about their I/O needs to exploit that parallelism. In addition to providing an insufficient interface, most current multiprocessor file systems are optimized for a different workload than they are being asked to support. We introduce Galley, a new parallel file system that is intended to efficiently support realistic scientific multiprocessor workloads. We discuss Galley's file structure and application interface, as well as the performance advantages offered by that interface.

Nieuwejaar, Nils↗
NASA NTRS1993

Enhancing real-time flight simulation execution by intercepting Run-Time Library calls

Standard operating system input-output (I/O) procedures impose a large time penalty on real-time program execution. These procedures are generally invoked by way of Run-Time Library (RTL) calls. To reduce the time penalty, as well as add flexibility, a technique has been developed to dynamically intercept these calls. The design and implementation of this technique, as applied to FORTRAN WRITE statements, are described. Measured performance gains using this RTL intercept technique are on the order of 1000 percent.

Reinbachs, Namejs↗
DOE OSTI2020

Characterization and identification of HPC applications at leadership computing facility

High Performance Computing (HPC) is an important method for scientific discovery via large-scale simulation, data analysis, or artificial intelligence. Leadership-class supercomputers are expensive, but essential to run large HPC applications. The Petascale era of supercomputers began in 2008, with the first machines achieving performance in excess of one petaflops, and with the advent of new supercomputers in 2021 (e.g., Aurora, Frontier), the Exascale era will soon begin. However, the high theoretical computing capability (i.e., peak FLOPS) of a machine is not the only meaningful target when designing a supercomputer, as the resources demand of applications varies. A deep understanding of the characterization of applications that run on a leadership supercomputer is one of the most important ways for planning its design, development and operation. In order to improve our understanding of HPC applications, user demands and resource usage characteristics, we perform correlative analysis of various logs for different subsystems of a leadership supercomputer. This analysis reveals surprising, sometimes counter-intuitive patterns, which, in some cases, conflicts with existing assumptions, and have important implications for future system designs as well as supercomputer operations. For example, our analysis shows that while the applications spend significant time on MPI, most applications spend very little time on file I/O. Combined analysis of hardware event logs and task failure logs show that the probability of a hardware FATAL event causing task failure is low. Combined analysis of control system logs and file I/O logs reveals that pure POSIX I/O is used more widely than higher level parallel I/O. Based on holistic insights of the application gained through combined and co-analysis of multiple logs from different perspectives and general intuition, we engineer features to "fingerprint" HPC applications. We use t-SNE (a machine learning technique for dimensionality reduction) to validate the explainability of our features and finally train machine learning models to identify HPC applications or group those with similar characteristic. To the best of our knowledge, this is the first work that combines logs on file I/O, computing, and inter-node communication for insightful analysis of HPC applications in production.

Liu, Zhengchun↗
DOE OSTI2022

Management and Storage of Scientific Data

Scientific discoveries rely heavily on efficient access, search, and management of massive data sets. Data management technologies have, for decades, provided foundational capabilities for scientific computing. Just as storage, input/output (I/O), and data management have been fundamental to simulation-based science for many years, so too are capable data-management technologies key to the success of today’s scientific workflows utilizing data intensive and machine learning (ML) techniques. The Department of Energy, Office of Science, Advanced Scientific Computing Research (ASCR) program has invested broadly in data-management research focused on high-performance computing (HPC) systems, from parallel file systems that store data to application software that makes these systems more productive. Still, advances in technology combined with growing diversity of supported science strongly motivate continued investment in this area. In January 2022, ASCR convened a workshop to identify priority research directions in the area of data management for high-performance and scientific computing. Attendees were challenged to identify promising approaches that would support the breadth of the DOE mission, including the explosion of artificial intelligence (AI) uses and the growing needs of experimental and observational science. Technological and science drivers were identified and considered as they relate to key aspects of data management such as interfaces, architectural design, and FAIR principles (Findable, Accessible, Interoperable, and Reusable). The thoughts of the workshop participants were distilled into a set of four priority research directions with the potential for high impact on DOE science. These research directions are summarized in the following pages.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2023

Spatiotemporally Adaptive Compression for Scientific Dataset with Feature Preservation – A Case Study on Simulation Data with Extreme Climate Events Analysis

Scientific discoveries are increasingly constrained by limited storage space and I/O capacities. For time-series simulations and experiments, their data often need to be decimated over timesteps to accommodate storage and I/O limitations. In this paper, we propose a technique that addresses storage costs while improving post-analysis accuracy through spatiotemporal adaptive, error-controlled lossy compression. We investigate the trade-off between data precision and temporal output rates, revealing that reducing data precision and increasing timestep frequency lead to more accurate analysis outcomes. Additionally, we integrate spatiotemporal feature detection with data compression and demonstrate that performing adaptive error-bounded compression in higher dimensional space enables greater compression ratios, leveraging the error propagation theory of a transformation-based compressor. To evaluate our approach, we conduct experiments using the well-known E3SM climate simulation code and apply our method to compress variables used for cyclone tracking. Our results show a significant reduction in storage size while enhancing the quality of cyclone tracking analysis, both quantitatively and qualitatively, in comparison to the prevalent timestep decimation approach. Compared to three state-of-the-art lossy compressors lacking feature preservation capabilities, our adaptive compression framework improves perfectly matched cases in TC tracking by 26.4-51.3% at medium compression ratios and by 77.3-571.1% at large compression ratios, with a merely 5–11% computational overhead.

Gong, Qian↗
DOE OSTI2022

SMALE: Enhancing Scalability of Machine Learning Algorithms on Extreme-Scale Computing Platforms

Deployment and execution of machine learning tasks on extreme-scale computing platforms face several significant technical challenges: 1) High computing cost incurred by dense networks – The computing workload of deep networks with densely-connected topology increases rapidly with the network size, imposing a non-scalable computing model of extreme-scale computing platforms; 2) Non-optimized workload distribution – Many advanced deep learning algorithms, e.g., sparsification and irregular net-work topology, produce very unbalanced workload distribution on extreme-scale computing platforms. The computation efficiency is greatly hindered by the incurred data and computation redundancies as well as long tails of the node with extensive workload; 3) Constraints in data movement and I/O bottle-neck – Inter-node data movement in extreme-scale computing platforms are associated with high energy and latency costs, and subject to the constraints of I/O bandwidth; and 4) Generalization of algorithm realization and acceleration on computing platforms – The large varieties of machine learning algorithms and structures of extreme-scale computing platforms make the derivation of a generalized algorithm realization and acceleration method very challenging, which, however, is the requirement by domain scientists and interested users. We call the above challenges Smale’s Problems in Machine Learning and Understanding for High-Performance Computing Scientific Discovery. The objective of our three-year research project is to develop a holistic innovation set at structure, assembly, and acceleration layers of machine learning algorithms to address the above challenges in algorithm deployment and execution. Three tasks are particularly performed, including: At the algorithm structure level, we investigate the techniques that can structurally sparsify on the topology of deep networks for computing workload reduction. We also study clustering and pruning techniques that can optimize the workload distributions over the extreme-scale computing platforms; At the algorithm assembly level, we derive a unified learning framework for unsupervised transfer learning and dynamic growing capabilities. Novel training methods are also exploited to enhance the training efficiency of the proposed framework; At the algorithm acceleration level, we will develop a series of techniques that can accelerate the computation of sparse matrix operations, which are one of the core executions in deep learning and optimize memory access of the concerned platforms. Our proposed techniques attack the fundamental problems in machine learning algorithms running on extreme-scale computing platforms by vertically integrating the solutions at three closely entangled layers, paving the long-term scaling path of machine learning applications under DOE context. Three tasks corresponding to the above respective research orientations are performed during the three-year project period with our collaborators at ORNL. The outcome of the proposed project is anticipated to form a holistic solution set of novel algorithms and network topologies, efficient training techniques, and fast acceleration methods to promote the computing scalability of the machine learning applications of particular interest to DOE.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1972

A model for the I/O-channel traffic in computer systems.

A new model is proposed which is based on the assumption that the actual traffic to be processed by an I/O channel system is generated as the overflow from a fictitious server system having Poisson traffic as the input. The number of servers and the mean arrival rate can be readily adjusted to verify the main value and the variance of the actual traffic to be handled. A closed solution for the stationary probabilities of state is obtained, and the parameters conventionally used to describe the grade of performance of queuing systems (such as mean queue length, mean waiting time, and probability of waiting) are defined.

Kuemmerle, K.↗
DOE OSTI2022

Fast Gaussian Process Estimation for Large-Scale In Situ Inference using Convolutional Neural Networks

Exascale computing will bring with it significant I/O limitations. One foreseeable consequence of such restrictions is that the user can save only a small fraction of complex simulation data to disk for subsequent analysis. An alternative is to fit statistical models to data in situ, that is, inside the simulation as it runs. This option requires extremely fast statistical estimation to avoid slowing down the simulation. Gaussian processes (GPs) have state-of-the-art predictive performance for modeling spatial data. However, standard estimation methods for GPs scale quite poorly to large data sets as parameter estimation requires inverting a covariance matrix to the size of the data set. In the presented work, we use a convolutional neural network (CNN) to predict the GP parameters for a spatial data set, from a simulation or otherwise, rather than optimize the parameters directly. Here, our presented case study models spatial data from E3SM, the Department of Energy’s Exascale climate model. The CNN is trained on synthetic data simulated from GP models with known parameters and then applied to data from the climate simulation. In the presented examples, the neural network scheme produces parameter estimates that compare well with standard methods such as maximum likelihood estimation in predictive performance but is obtained four orders of magnitude faster.

big data↗
DOE OSTI2024

Control System Upgrade for Canal ARC SAW

Upgrading the specific components of the Arc Saw, such as the HMI, the components of the chassis involving the I/O modules, servo motor modules, and PLC modules offers numerous advantages. By enhancing the HMI, the operators will benefit having a user-friendly interface, enabling smoother control and monitoring of the ARC SAW’s functionality. Upgrading the modules will contribute to an improved precision and responsiveness in changing modes of the ARC SAW as well as the adjusting the saw blades position, further enhancing the system's overall performance. These upgrades, while maintaining the same functionality, enable the latest technological advancements without needing significant changes or disruptions to existing operations.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
NASA NTRS2011

SpaceCube 2.0: An Advanced Hybrid Onboard Data Processor

The SpaceCube 2.0 is a compact, high performance, low-power onboard processing system that takes advantage of cutting-edge hybrid (CPU/FPGA/DSP) processing elements. The SpaceCube 2.0 design concept includes two commercial Virtex-5 field-programmable gate array (FPGA) parts protected by gradiation hardened by software" technology, and possesses exceptional size, weight, and power characteristics [5x5x7 in., 3.5 lb (approximately equal to 12.7 x 12.7 x 17.8 cm, 1.6 kg) 5-25 W, depending on the application fs required clock rate]. The two Virtex-5 FPGA parts are implemented in a unique back-toback configuration to maximize data transfer and computing performance. Draft computing power specifications for the SpaceCube 2.0 unit include four PowerPC 440s (1100 DMIPS each), 500+ DSP48Es (2x580 GMACS), 100+ LVDS high-speed serial I/Os (1.25 Gbps each), and 2x190 GFLOPS single-precision (65 GFLOPS double-precision) floating point performance. The SpaceCube 2.0 includes PROM memory for CPU boot, health and safety, and basic command and telemetry functionality; RAM memory for program execution; and FLASH/EEPROM memory to store algorithms and application code for the CPU, FPGA, and DSP processing elements. Program execution can be reconfigured in real time and algorithms can be updated, modified, and/or replaced at any point during the mission. Gigabit Ethernet, Spacewire, SATA and highspeed LVDS serial/parallel I/O channels are available for instrument/sensor data ingest, and mission-unique instrument interfaces can be accommodated using a compact PCI (cPCI) expansion card interface. The SpaceCube 2.0 can be utilized in NASA Earth Science, Helio/Astrophysics and Exploration missions, and Department of Defense satellites for onboard data processing. It can also be used in commercial communication and mapping satellites.

Lin, Michael↗
NASA NTRS2001

Flight Computer Processing Avionics for Space Station Microgravity Experiments: A Risk Assessment of Commercial Off-the-Shelf Utilization

NASA/Marshall Space Flight Center (MSFC) is continually looking for ways to reduce the costs and schedule and minimize the technical risks during the development of microgravity programs. One of the more prominent ways to minimize the cost and schedule is to use off-the-shelf hardware (OTS). However, the use of OTS often increases the risk. This paper addresses relevant factors considered during the selection and utilization of commercial off-the-shelf (COTS) flight computer processing equipment for the control of space station microgravity experiments. The paper will also discuss how to minimize the technical risks when using COTS processing hardware. Two microgravity experiments for which the COTS processing equipment is being evaluated for are the Equiaxed Dendritic Solidification Experiment (EDSE) and the Self-diffusion in Liquid Elements (SDLE) experiment. Since MSFC is the lead center for Microgravity research, EDSE and SDLE processor selection will be closely watched by other experiments that are being designed to meet payload carrier requirements. This includes the payload carriers planned for the International Space Station (ISS). The purpose of EDSE is to continue to investigate microstructural evolution of, and thermal interactions between multiple dendrites growing under diffusion controlled conditions. The purpose of SDLE is to determine accurate self-diffusivity data as a function of temperature for liquid elements selected as representative of class-like structures. In 1999 MSFC initiated a Center Director's Discretionary Fund (CDDF) effort to investigate and determine the optimal commercial data bus architecture that could lead to faster, better, and lower cost data acquisition systems for the control of microgravity experiments. As part of this effort various commercial data acquisition systems were acquired and evaluated. This included equipment with various form factors, (3U, 6U, others) and equipment that utilized various bus structures, (VME, PC104, STD bus). This evaluation of hardware was performed in conjunction with a trade study that considered over twenty (20) different factors relevant to the selection of an optimum design approach. These factors included; safety, sizing and timing, radiation hardness and single event upset, power consumption, heat dissipation, size and volume, expected service life, maintainability, heritage, operating systems, requirements for software reuse, availability of compatible interface boards, relative cost, schedule, reliability, EMI/EMC factors, "hot swap" capability, standards for conduction cooling, I/O capabilities, unique carrier requirements and operating system considerations. The approach to evaluate Safety as part of this study included a review of the Preliminary Hazard Analysis (PHA) for each of the experiment designs and a determination of how each hazard could be addressed and eliminated when different processors were selected. This included evaluating various design approaches and trade-offs between fault tolerant designs and fail-safe designs in accordance with NSTS 1700.7B. This will include the results of radiation testing where available. Various operating systems, such as VxWorks, Linux, QNX, and Embedded NT are evaluated and the advantages and disadvantages of their utilization are also addressed. Design implementation strategies for the various operating systems are considered and discussed. This paper presents the results and recommendations from this trade study. Preliminary conclusions from this study are that safety concerns from lack or radiation testing on COTS equipment can be addressed by additional testing and design considerations, the PC104 bus provided adequate I/O for the SDLE and EDSE microgravity experiments, and PC104 bus components offered significant advantages over VME and cPCI for weight and space reductions.

Estes, Howard↗
DOE OSTI2026

rustpix

rustpix is a high-performance, open-source Rust library with first-class Python bindings (via PyO3) for processing pixel-detector data in neutron imaging. It targets time-stamping detectors such as Timepix3 (TPX3) at ORNL's Spallation Neutron Source (VENUS beamline), where each detected neutron deposits charge across a cluster of pixels within a very high-rate event stream (96M+ hits/sec). rustpix parses TPX3 event data in parallel using memory-mapped I/O, offers four interchangeable clustering algorithms (ABS adjacency-based search, DBSCAN, graph/union-find connected components, and a parallel grid method), and extracts weighted, super-resolved centroids to produce neutron-event lists. A streaming architecture lets it process files larger than available memory. rustpix is distributed as a pip-installable Python package (with NumPy integration), Rust crates, a command-line tool, and an interactive GUI; it writes HDF5, Apache Arrow, and CSV; and it is designed to extend to TPX4 and other detector types. Released as open-source under the MIT License.

Zhang, Chen [Oak Ridge National Laboratory (ORNL),↗
NASA NTRS1990

High-performance optical disk mass storage for aerospace imaging systems

The specifications and techniques related to the development of the NASA Spaceflight Optical Disk Recorder (SODR) are presented. The system is being developed for the temporary storage of high-resolution sensor data that will be collected at high data rates by future space programs. Typical spaceflight recording requirements are given for the Space Station Freedom, the Earth Observing System, and some planetary missions. SODR module characteristics are listed, emphasizing the two independent optical disk recorders and the random access to 5 gigabytes of data on each side of the disks. The modular design includes two I/O ports providing 150 Mbits/sec user data rate each, and multimodule reconfigurable architecture permitting the synchronization of up to 16 modules with a maximum data rate of 4.8 gigabits/sec.

Ravner, Stephen M.↗