Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “parallel I/O”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8
All sourcesNASA NTRSDOE OSTI
NASA NTRS1987

Plasma simulation using the massively parallel processor

Two dimensional electrostatic simulation codes using the particle-in-cell model are developed on the Massively Parallel Processor (MPP). The conventional plasma simulation procedure that computes electric fields at particle positions by means of a gridded system is found inefficient on the MPP. The MPP simulation code is thus based on the gridless system in which particles are assigned to processing elements and electric fields are computed directly via Discrete Fourier Transform. Currently, the gridless model on the MPP in two dimensions is about nine times slower that the gridded system on the CRAY X-MP without considering I/O time. However, the gridless system on the MPP can be improved by incorporating a faster I/O between the staging memory and Array Unit and a more efficient procedure for taking floating point sums over processing elements. The initial results suggest that the parallel processors have the potential for performing large scale plasma simulations.

Lin, C. S.↗
NASA NTRS1990

Prefetching in file systems for MIMD multiprocessors

The question of whether prefetching blocks on the file into the block cache can effectively reduce overall execution time of a parallel computation, even under favorable assumptions, is considered. Experiments have been conducted with an interleaved file system testbed on the Butterfly Plus multiprocessor. Results of these experiments suggest that (1) the hit ratio, the accepted measure in traditional caching studies, may not be an adequate measure of performance when the workload consists of parallel computations and parallel file access patterns, (2) caching with prefetching can significantly improve the hit ratio and the average time to perform an I/O (input/output) operation, and (3) an improvement in overall execution time has been observed in most cases. In spite of these gains, prefetching sometimes results in increased execution times (a negative result, given the optimistic nature of the study). The authors explore why it is not trivial to translate savings on individual I/O requests into consistently better overall performance and identify the key problems that need to be addressed in order to improve the potential of prefetching techniques in the environment.

Kotz, David F.↗
NASA NTRS1985

Evaluation of existing and proposed computer architectures for future ground-based systems

Parallel processing architectures and techniques used in current supercomputers are described and projections are made of future advances. Presently, the von Neumann sequential processing pattern has been accelerated by having separate I/O processors, interleaved memories, wide memories, independent functional units and pipelining. Recent supercomputers have featured single-input, multiple data stream architectures, which have different processors for performing various operations (vector or pipeline processors). Multiple input, multiple data stream machines have also been developed. Data flow techniques, wherein program instructions are activated only when data are available, are expected to play a large role in future supercomputers, along with increased parallel processor arrays. The enhanced operational speeds are essential for adequately treating data from future spacecraft remote sensing instruments such as the Thematic Mapper.

Schulbach, C.↗
DOE OSTI2026

FitCache: A Transparent Drop-In Framework for Multi-Tier Caching to Accelerate Distributed Deep Learning Workloads

Training in Deep learning (DL) remains highly compute- and data-intensive, with I/O becoming a critical bottleneck as models and datasets scale. Recent studies report that data loading can dominate training time, especially on large-scale HPC systems with shared parallel file systems (PFS). Existing caching approaches either rely on single-tier designs or require intrusive modifications to training pipelines, limiting their portability and effectiveness. In this work, we present FitCache, a transparent drop-in framework for multi-tier caching to accelerate distributed DL training by coordinating fast local memory (e.g., DRAM, Persistent Memory (PMem)) and NVMe as hierarchical caches atop PFS. Our design adapts to hardware diversity, i.e., if NVMe is missing, memory transparently acts as a caching tier, ensuring stable performance. FitCache transparently intercepts I/O requests and issues concurrent fetches across all tiers, returning data from the fastest responder without centralized metadata or static redirection paths. FitCache adapts to dynamic workloads and heterogeneous clusters while maintaining POSIX compatibility. Experiments on Frontier (2048 GPUs) and smaller research clusters show that FitCache reduces training time by up to 40% and per-batch I/O latency by up to 71.6% compared to Lustre Orion PFS, offering a drop-in solution for scalable DL training.

Hu, Guangxing [ORNL] (ORCID:0009000283203614)↗
DOE OSTI2025

ChatHPC: Building the Foundations for a Productive and Trustworthy AI-Assisted HPC Ecosystem

ChatHPC democratizes large language models for the high-performance computing (HPC) community by providing the infrastructure, ecosystem, and knowledge needed to apply modern generative AI technologies to rapidly create specific capabilities for critical HPC components while using relatively modest computational resources. Our divide-and-conquer approach focuses on creating a collection of reliable, highly specialized, and optimized AI assistants for HPC based on the cost-effective and fast Code Llama fine-tuning processes and expert supervision. We target major components of the HPC software stack, including programming models, runtimes, I/O, tooling, and math libraries. Thanks to AI, ChatHPC provides a more productive HPC ecosystem by boosting important tasks related to portability, parallelization, optimization, scalability, and instrumentation, among others. With relatively small datasets (on the order of KB), the AI assistants, which are created in a few minutes by using one node with two NVIDIA H100 GPUs and the ChatHPC library, can create new capabilities with Meta’s 7-billion parameter Code Llama base model to produce high-quality software with a level of trustworthiness of up to 90% higher than the 1.8-trillion parameter OpenAI ChatGPT-4o model for critical programming tasks in the HPC software stack.

Young, Aaron [ORNL] (ORCID:0000000254484667)↗
DOE OSTI2020

PFLOTRAN

PFLOTRAN is a GNU LGPL licensed code that simulates single phase water and multiphase (air, co2, water) flow, energy transport, geochemical reaction and biological reaction in the earth's subsurface environment. The problem may be discretized on structured and unstructured grids. The code is designed to run in parallel on supercomputers and leverages the open source PETSc library for solvers and data structures and open source HDF5 for binary file I/O. See www.pflotran.org and/or doc-dev.pflotran.org for further information. SAND2020-12161 M Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Hammond, Glenn↗
DOE OSTI2021

X-composer: enabling cross-environments in-situ workflows between HPC and cloud

As large-scale scientific simulations and big data analyses become more popular, it is increasingly more expensive to store huge amounts of raw simulation results to perform post-analysis. To minimize the expensive data I/O, "in-situ" analysis is a promising approach, where data analysis applications analyze the simulation generated data on the fly without storing it first. However, it is challenging to organize, transform, and transport data at scales between two semantically different ecosystems due to the distinct software and hardware difference. To tackle these challenges, we design and implement the X-Composer framework. X-Composer connects cross-ecosystem applications to form an "in-situ" scientific workflow, and provides a unified approach and recipe for supporting such hybrid in-situ workflows on distributed heterogeneous resources. X-Composer reorganizes simulation data as continuous data streams and feeds them seamlessly into the Cloud-based stream processing services to minimize I/O overheads. For evaluation, we use X-Composer to set up and execute a cross-ecosystem workflow, which consists of a parallel Computational Fluid Dynamics simulation running on HPC, and a distributed Dynamic Mode Decomposition analysis application running on Cloud. Our experimental results show that X-Composer can seamlessly couple HPC and Big Data jobs in their own native environments, achieve good scalability, and provide high-fidelity analytics for ongoing simulations in real-time.

Wang, Dali↗
NASA NTRS1982

On the inversion of block tridiagonals without storage constraints

A strategy was developed to permit trade-offs between the number of floating point operations required and the storage requirements for the solution of certain difference problems, such as block tridiagonal systems of equations. This is done by recomputing some intermediate results instead of storing them. Reducing the storage to the square root of the current requirement roughly doubles the number of computations. Reducing the storage more than this tends to make the number of computations prohibitively large. In theory, if m is the order of each sub-matrix in the block tridiagonal matrix, one can solve any linear system with only 5m(2) + 1 temporary storage cells. In many cases m is a constant and quite small. For example, in solving a factored form of the three-dimensional Navier-Stokes equations, the size m of the block tridiagonals is 5. This method lends itself to efficient use on computers with parallel processing or vector processing architectures. On these computers the larger number of floating point operations is more than offset by the decrease in I/O and the increased percentage of vector operations made possible by this algorithm.

Merriam, M. L.↗
NASA NTRS1985

On the factorization of block-tridiagonals without storage constraints

In many programs solving difference equations, problem size is restricted by the number of available memory cells. A strategy has been developed to permit trade-offs between the number of floating point operations required and storage requirements for the solution of certain problems such as block tridiagonal systems of equations. This is done by recomputing some intermediate results instead of storing them. Reducing the storage to the square root of the current requirement will roughly double the number of computations. In theory, if m is the order of each sub-matrix in the block tridiagonal matrix, one can solve any linear system with only 5 sq m + 1 temporary storage cells. This method lends itself to efficient use on computers with parallel processing or vector processing architectures. On these computers the larger number of floating point operations is more than offset by the decrease in I/O and the increased percentage of vector operations made possible by this algorithm.

Merriam, M. L.↗
NASA NTRS1992

Spaceflight optical disk recorder development

Mass memory systems based on rewriteable optical disk media are expected to play an important role in meeting the data system requirements for future NASA spaceflight missions. NASA has established a program to develop a high performance (high rate, large capacity) optical disk recorder focused on use aboard unmanned Earth orbiting platforms. An expandable, adaptable system concept is proposed based on disk drive modules and a modular controller. Drive performance goals are 10 gigabyte capacity, 300 megabit/s transfer rate, 10 exp -12 corrected bit error rate, and 150 millisec access time. This performance is achieved by writing eight data tracks in parallel on both sides of a 14 in. optical disk using two independent heads. System goals are 160 gigabyte capacity, 1.2 gigabits/s data rate with concurrent I/O, 250 millisec access time, and two to five year operating life on orbit. The system can be configured to meet various applications. This versatility is provided by the controller. The controller provides command processing, multiple drive synchronization, data buffering, basic file management, error processing, and status reporting. Technology developments, design concepts, current status including a computer model of the system and a Controller breadboard, and future plans for the Drive and Controller are presented.

Jurczyk, Stephen G.↗
NASA NTRS1996

Cooperative high-performance storage in the accelerated strategic computing initiative

The use and acceptance of new high-performance, parallel computing platforms will be impeded by the absence of an infrastructure capable of supporting orders-of-magnitude improvement in hierarchical storage and high-speed I/O (Input/Output). The distribution of these high-performance platforms and supporting infrastructures across a wide-area network further compounds this problem. We describe an architectural design and phased implementation plan for a distributed, Cooperative Storage Environment (CSE) to achieve the necessary performance, user transparency, site autonomy, communication, and security features needed to support the Accelerated Strategic Computing Initiative (ASCI). ASCI is a Department of Energy (DOE) program attempting to apply terascale platforms and Problem-Solving Environments (PSEs) toward real-world computational modeling and simulation problems. The ASCI mission must be carried out through a unified, multilaboratory effort, and will require highly secure, efficient access to vast amounts of data. The CSE provides a logically simple, geographically distributed, storage infrastructure of semi-autonomous cooperating sites to meet the strategic ASCI PSE goal of highperformance data storage and access at the user desktop.

Gary, Mark↗
DOE OSTI2021

Online data analysis and reduction: An important co-design motif for extreme-scale computers

A growing disparity between supercomputer computation speeds and I/O rates means that it is rapidly becoming infeasible to analyze supercomputer application output only after that output has been written to a file system. Instead, data-generating applications must run concurrently with data reduction and/or analysis operations, with which they exchange information via high-speed methods such as interprocess communications. The resulting parallel computing motif, online data analysis and reduction (ODAR), has important implications for both application and HPC systems design. Here we introduce the ODAR motif and its co-design concerns, describe a co-design process for identifying and addressing those concerns, present tools that assist in the co-design process, and present case studies to illustrate the use of the process and tools in practical settings.

Data Analysis↗
NASA NTRS1972

An On Board Processor (OBP) for OAO C

A stored program computer and its application on OAO is considered. The parallel computer has a memory capacity of 16,384 words of 18 bits each, one central processor unit, two 4096 word memory units, and one input/output unit. The I/O has no direct data connection with the CPU so that all data flow between these two units must pass through memory by way of the memory data bus. The primary functions of the onboard computer are auxiliary command storage, spacecraft monitoring and malfunction reporting, data compression and status summary, and possible performance of emergency corrective action.

Hartenstein, R. G.↗
NASA NTRS1992

Wilbur: A low-cost CCD camera system for MDM Observatory

The recent availability of several 'off-the-shelf' components, particularly CCD control electronics from SDSU, has made it possible to put together a flexible CCD camera system at relatively low cost and effort. The authors describe Wilbur, a complete CCD camera system constructed for the Michigan-Dartmouth-MIT Observatory. The hardware consists of a Loral 2048(exp 2) CCD controlled by the SDSU electronics, an existing dewar design modified for use at MDM, a Sun Sparcstation 2 with a commercial high-speed parallel controller, and a simple custom interface between the controller and the SDSU electronics. The camera is controlled from the Sparcstation by software that provides low-level I/O in real time, collection of additional information from the telescope, and a simple command interface for use by an observer. Readout of the 2048(exp 2) array is complete in under two minutes at 5 e(sup -) read noise, and readout time can be decreased at the cost of increased noise. The system can be easily expanded to handle multiple CCD's/multiple readouts, and can control other dewars/CCD's using the same host software.

Metzger, M. R.↗
NASA NTRS1987

Systems simulations supporting NASA telerobotics

Two simulation and analysis environments have been developed to support telerobotics research at the Langley Research Center. One is a high-fidelity, nonreal-time, interactive model called ROBSIM, which combines user-generated models of workspace environment, robots, and loads into a working system and simulates the interaction among the system components. Models include user-specified actuator, sensor, and control parameters, as well as kinematic and dynamic characteristics. Kinematic, dynamic, and response analyses can be selected, with system configuration, task trajectories, and arm states displayed using computer graphics. The second environment is a real-time, manned Telerobotic Systems Simulation (TRSS) which uses the facilities of the Intelligent Systems Research Laboratory (ISRL). It utilizes a hierarchical structure of functionally distributed computers communicating over both parallel and high-speed serial data paths to enable studies of advanced telerobotic systems. Multiple processes perform motion planning, operator communications, forward and inverse kinematics, control/sensor fusion, and I/O processing while communicating via common memory. This paper describes both ROBSIM and TRSS, including their capability, status, and future plans. Also described is the architecture of ISRL and recent telerobotic system studies in ISRL.

Harrison, F. Wallace, Jr.↗
NASA NTRS1987

Systems simulations supporting NASA telerobotics

Two simulation and analysis environments have been developed to support telerobotics research at the Langley Research Center. One is a high-fidelity, nonreal-time, interactive model called ROBSIM, which combines user-generated models of workspace environment, robots, and loads into a working system and simulates the interaction among the system components. Models include user-specified actuator, sensor, and control parameters, as well as kinematic and dynamic characteristics. Kinematic, dynamic, and response analyses can be selected, with system configuration, task trajectories, and arm states displayed using computer graphics. The second environment is a real-time, manned Telerobotic Systems Simulation (TRSS) which uses the facilities of the Intelligent Systems Research Laboratory (ISRL). It utilizes a hierarchical structure of functionally distributed computers communicating over both parallel and high-speed serial data paths to enable studies of advanced telerobotic systems. Multiple processes perform motion planning, operator communications, forward and inverse kinematics, control/sensor fusion, and I/O processing while communicating via common memory. Both ROBSIM and TRSS, including their capability, status, and future plans are discussed. Also described is the architecture of ISRL and recent telerobotic system studies in ISRL.

Harrison, F. W., Jr.↗
NASA NTRSDate not supplied

Rapid Lightweight Firmware Architecture of the Mobile Metamaterial Internal Co-Integrator Robot

The Mobile Metamaterial Internal Co-Integrator (MMIC-I) is a structure assembly and servicing robot for in-space servicing, assembly, and manufacturing of primary structures and infrastructure. MMIC-I is a battery-powered crawling robot that can travel through periodic structures such as trusses and open framework mechanical metamaterials. It does this through sequences of component extension, contraction, and gripping. This paper provides a detailed discussion of MMIC-I’s lightweight and rapidly developed firmware architecture, to enable demonstration of robot locomotion, secondary operations, and communications with a central command source. The rationale for the lightweight rapid development approach is to allow for assessment of long term system requirements in parallel with the mechatronics development, including optimization of system and subsystem power densities, to inform a future choice of flight ready software frameworks. MMIC-I system computing and I/O requirements are much lower than what is provided by proven baseline computing hardware for existing flight ready software frameworks such as the core Flight System, F prime, and the Robot Operating System. Development of earth gravity ground demonstration of the robotic systems is greatly benefited by limited power and mass factors for computing hardware. Here, we implement inter-process communication, commanding, and telemetry with the Espressif ESP32 module running the Arduino OS.

Damiana Catanoso↗
NASA NTRS2016

Electrical Performance of a High Temperature 32-I/O HTCC Alumina Package

A high temperature co-fired ceramic (HTCC) alumina material was previously electrically tested at temperatures up to 550 C, and demonstrated improved dielectric performance at high temperatures compared with the 96% alumina substrate that we used before, suggesting its potential use for high temperature packaging applications. This paper introduces a prototype 32-I/O (input/output) HTCC alumina package with platinum conductor for 500 C low-power silicon carbide (SiC) integrated circuits. The design and electrical performance of this package including parasitic capacitance and parallel conductance of neighboring I/Os from 100 Hz to 1 MHz in a temperature range from room temperature to 550 C are discussed in detail. The parasitic capacitance and parallel conductance of this package in the entire frequency and temperature ranges measured does not exceed 1.5 pF and 0.05 microsiemens, respectively. SiC integrated circuits using this package and compatible printed circuit board have been successfully tested at 500 C for over 3736 hours continuously, and at 700 C for over 140 hours. Some test examples of SiC integrated circuits with this packaging system are presented. This package is the key to prolonged T greater than or equal to 500 C operational testing of the new generation of SiC high temperature integrated circuits and other devices currently under development at NASA Glenn Research Center.

dielectric performance↗