Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Scalable I/O”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
NASA NTRS1993

Diskless supercomputers: Scalable, reliable I/O for the Tera-Op technology base

Computing is seeing an unprecedented improvement in performance; over the last five years there has been an order-of-magnitude improvement in the speeds of workstation CPU's. At least another order of magnitude seems likely in the next five years, to machines with 500 MIPS or more. The goal of the ARPA Teraop program is to realize even larger, more powerful machines, executing as many as a trillion operations per second. Unfortunately, we have seen no comparable breakthroughs in I/O performance; the speeds of I/O devices and the hardware and software architectures for managing them have not changed substantially in many years. We have completed a program of research to demonstrate hardware and software I/O architectures capable of supporting the kinds of internetworked 'visualization' workstations and supercomputers that will appear in the mid 1990s. The project had three overall goals: high performance, high reliability, and scalable, multipurpose system.

Katz, Randy H.↗
NASA NTRS2003

Scalability of a Low-Cost Multi-Teraflop Linux Cluster for High-End Classical Atomistic and Quantum Mechanical Simulations

Scalability of a low-cost, Intel Xeon-based, multi-Teraflop Linux cluster is tested for two high-end scientific applications: Classical atomistic simulation based on the molecular dynamics method and quantum mechanical calculation based on the density functional theory. These scalable parallel applications use space-time multiresolution algorithms and feature computational-space decomposition, wavelet-based adaptive load balancing, and spacefilling-curve-based data compression for scalable I/O. Comparative performance tests are performed on a 1,024-processor Linux cluster and a conventional higher-end parallel supercomputer, 1,184-processor IBM SP4. The results show that the performance of the Linux cluster is comparable to that of the SP4. We also study various effects, such as the sharing of memory and L2 cache among processors, on the performance.

Kikuchi, Hideaki↗
NASA NTRSDate not supplied

MAPL3: A Hierarchical Framework Based on ESMF

MAPL is a hierarchical, ESMF-based, coupling framework developed to support NASA's GEOS data assimilation system, but also now supports some external applications from partner institutions. The framework continues to evolve in response novel coupling requirements and is currently undergoing a substantial re-engineering effort to improve extensibility and maintainability. Some of the major new features that will be provided in the upcoming release of MAPL3 include the following: (1) "weak" NUOPC compliance that expresses a MAPL hierarchy as single NUOPC model, (2) component extensions that represent an improvement over conventional component couplers in many contexts, (3) an expansion of the concept of import and export to include component "services" such as advection, turbulence, etc., (4) automated OpenMP threading of column-based component subtrees, (5) and (5) improved I/O scalability.

GEOS↗
NASA NTRS2013

The NASA-Goddard Multi-Scale Modeling Framework - Land Information System: Global Land/atmosphere Interaction with Resolved Convection

The present generation of general circulation models (GCM) use parameterized cumulus schemes and run at hydrostatic grid resolutions. To improve the representation of cloud-scale moist processes and landeatmosphere interactions, a global, Multi-scale Modeling Framework (MMF) coupled to the Land Information System (LIS) has been developed at NASA-Goddard Space Flight Center. The MMFeLIS has three components, a finite-volume (fv) GCM (Goddard Earth Observing System Ver. 4, GEOS-4), a 2D cloud-resolving model (Goddard Cumulus Ensemble, GCE), and the LIS, representing the large-scale atmospheric circulation, cloud processes, and land surface processes, respectively. The non-hydrostatic GCE model replaces the single-column cumulus parameterization of fvGCM. The model grid is composed of an array of fvGCM gridcells each with a series of embedded GCE models. A horizontal coupling strategy, GCE4fvGCM4Coupler4LIS, offered significant computational efficiency, with the scalability and I/O capabilities of LIS permitting landeatmosphere interactions at cloud-scale. Global simulations of 2007e2008 and comparisons to observations and reanalysis products were conducted. Using two different versions of the same land surface model but the same initial conditions, divergence in regional, synoptic-scale surface pressure patterns emerged within two weeks. The sensitivity of largescale circulations to land surface model physics revealed significant functional value to using a scalable, multi-model land surface modeling system in global weather and climate prediction.

cumulus schemes↗
NASA NTRS2013

Profiling and Improving I/O Performance of a Large-Scale Climate Scientific Application

Exascale computing systems are soon to emerge, which will pose great challenges on the huge gap between computing and I/O performance. Many large-scale scientific applications play an important role in our daily life. The huge amounts of data generated by such applications require highly parallel and efficient I/O management policies. In this paper, we adopt a mission-critical scientific application, GEOS-5, as a case to profile and analyze the communication and I/O issues that are preventing applications from fully utilizing the underlying parallel storage systems. Through in-detail architectural and experimental characterization, we observe that current legacy I/O schemes incur significant network communication overheads and are unable to fully parallelize the data access, thus degrading applications' I/O performance and scalability. To address these inefficiencies, we redesign its I/O framework along with a set of parallel I/O techniques to achieve high scalability and performance. Evaluation results on the NASA discover cluster show that our optimization of GEOS-5 with ADIOS has led to significant performance improvements compared to the original GEOS-5 implementation.

Exascale↗
NASA NTRS1994

High Speed Computing, LANs, and WAMs

Optical fiber networks may one day offer potential capacities exceeding 10 terabits/sec. This paper describes present gigabit network techniques for distributed computing as illustrated by the CASA gigabit testbed, and then explores future all-optic network architectures that offer increased capacity, more optimized level of service for a given application, high fault tolerance, and dynamic reconfigurability.

all-optics networks terabit fiber optic networks s↗
NASA NTRS2019

Leveraging STARE for Co-aligned Data Locality with netCDF and Python MPI

We have leveraged STARE indexing to package partitioned data chunks from diverse datasets into netCDF files, distributed them on a cluster of 16 lightweight nodes with their placements spatiotemporally co-aligned, and demonstrated a few integrative analyses using netCDF parallel I/O and Python MPI, with single-user performance and scalability comparable to, or even better than, that of a parallel array database management system (ADBMS) such as SciDB. However, records of the node location and STARE index ranges for each data chunk, similar to the chunk maps of SciDB, must be maintained and consulted by the I/O and analysis code for coordinating the analytic operations in parallel, in order to achieve the good performance and scalability.

Kwo-Sen Kuo↗
NASA NTRS1992

NHT-1 I/O Benchmarks

The NHT-1 benchmarks am a set of three scalable I/0 benchmarks suitable for evaluating the I/0 subsystems of high performance distributed memory computer systems. The benchmarks test application I/0, maximum sustained disk I/0, and maximum sustained network I/0. Sample codes are available which implement the benchmarks.

Carter, Russell↗
NASA NTRSDate not supplied

NASA POWER: Providing Analysis-Ready, Cloud-Optimized Data for AI /ML Training and Applications in Earth Science

As global demand for sustainable development grows, the integration of Earth Observation (EO) data into decision making frameworks has become a primary objective for the scientific community. The NASA Prediction of Worldwide Energy Resources (POWER) project serves as a bridge between NASA EO data and the specialized needs of the renewable energy, sustainable infrastructure and agroclimatology communities. In this poster presentation we will present an overview of POWER data products and services along with its use in diverse research to decision-making workflows. By providing over 40 years of high-resolution historical, hourly and daily solar and meteorological data, POWER transforms satellite observations and global model reanalysis into actionable, Analysis-Ready Dataset (ARD). Currently, the project delivers over 250 industry-friendly parameters to the users from different NASA datasets like CERES SYN1Deg, MERRA-2, and IMERG alongside downscaled CMIP6 climate model data, fulfilling over 16 million requests from 50,000 unique users monthly. To ensure data quality and traceability, these parameters are rigorously validated against the ground-based observations from the Baseline Surface Radiation Network (BSRN) and the Global Surface Summary of the Day (GSOD) – these results will be discussed in the presentation. A newly introduced web-based PaRameter Uncertainty ViEwer (PRUVE) tool will be presented that provides an online validation platform to the users that benchmarks satellite-based and assimilation data products against these surface measurements. To reduce technical barriers to data adoption, POWER data is accessible through RESTful APIs, ESRI ArcGIS Image Services, a web-based Data Access Viewer tool, allowing users to visualize, validate and apply the dataset. For efficient data delivery POWER data is cloud-optimized into Zarr datastore accessible through NASA managed Amazon S3 ensures high-performance allowing users to integrate EO directly into operational pipelines. These customized services will be presented. Use cases from application will be presented from the energy sector - such as for design of generation systems, performance monitoring of solar power plants, in infrastructure sector- optimizing building energy efficiency and thermal comfort, in agriculture – such as driving crop simulation and yield forecasting models to enable climate resilient farming. Furthermore, the shift toward machine learning (ML) in EO research that has positioned POWER as a key provider for training datasets which will be discussed. Use-cases will be presented to showcase how NASA data is enabling the development of predictive tools for climate variability and resource management. The poster will present POWER’s future plans including technology development to enhance data traceability and reproducibility and improving I/O performance to support the rapid integration of new EO products, ensuring that POWER remains a robust scalable backend for the evolving landscape of AI-driven Earth Science. Additionally, POWER is developing an AI Agent and an MCP-Server to enable industry AI-Agentic workflows.

Neha Khadka↗
NASA NTRS2026

Hydrology Copilot: A Cloud-Native Ai System for Hydrological Data Analysis

The emergence of AI-driven Earth observation systems promises to broaden access to petabyte-scale geospatial data beyond domain specialists. However, translating this vision into operational scientific infrastructure requires addressing fundamental challenges in data virtualization, code transparency, and domain-specific reasoning. We present Hydrology Copilot, a cloud-native AI framework for natural-language-driven analysis of Earth observation data. To demonstrate operational capabilities at scale, we implement the system using NASA's North American Land Data Assimilation System version 3 (NLDAS-3), which provides surface meteorological forcing and land-surface model output across North and Central America at 1-km resolution, from which drought diagnostics are derived. The system integrates five core contributions: (1) scalable data virtualization using Kerchunk-based cloud optimized access, achieving a 1.5 to 4.6 times improvement in I/O latency across benchmark queries spanning regional single-day extractions (4.6 times speedup) to continental monthly aggregations (1.5 times speedup); (2) transparent code generation through Microsoft Azure AI Foundry agents that expose executable Python workflows for scientific verification; (3) persistent conversational memory enabling multi-turn analytical discourse across sessions; (4) intelligent query validation that enforces dataset boundaries and resolves ambiguous requests before execution; and (5) a multi-agent architecture coordinating query parsing, code generation, and visualization. We evaluate the system through drought-monitoring workflows, demonstrating reliable code generation, accurate results validated against reference computations and the operational U.S. Drought Monitor, and efficient operation across increasingly complex tasks. By bridging natural-language interfaces with rigorous hydrological analysis, Hydrology Copilot advances beyond proof-of-concept demonstrations to provide a deployable framework for operational Earth science applications.

Data virtualization↗
NASA NTRS1995

High Performance Parallel Computational Nanotechnology

At a recent press conference, NASA Administrator Dan Goldin encouraged NASA Ames Research Center to take a lead role in promoting research and development of advanced, high-performance computer technology, including nanotechnology. Manufacturers of leading-edge microprocessors currently perform large-scale simulations in the design and verification of semiconductor devices and microprocessors. Recently, the need for this intensive simulation and modeling analysis has greatly increased, due in part to the ever-increasing complexity of these devices, as well as the lessons of experiences such as the Pentium fiasco. Simulation, modeling, testing, and validation will be even more important for designing molecular computers because of the complex specification of millions of atoms, thousands of assembly steps, as well as the simulation and modeling needed to ensure reliable, robust and efficient fabrication of the molecular devices. The software for this capacity does not exist today, but it can be extrapolated from the software currently used in molecular modeling for other applications: semi-empirical methods, ab initio methods, self-consistent field methods, Hartree-Fock methods, molecular mechanics; and simulation methods for diamondoid structures. In as much as it seems clear that the application of such methods in nanotechnology will require powerful, highly powerful systems, this talk will discuss techniques and issues for performing these types of computations on parallel systems. We will describe system design issues (memory, I/O, mass storage, operating system requirements, special user interface issues, interconnects, bandwidths, and programming languages) involved in parallel methods for scalable classical, semiclassical, quantum, molecular mechanics, and continuum models; molecular nanotechnology computer-aided designs (NanoCAD) techniques; visualization using virtual reality techniques of structural models and assembly sequences; software required to control mini robotic manipulators for positional control; scalable numerical algorithms for reliability, verifications and testability. There appears no fundamental obstacle to simulating molecular compilers and molecular computers on high performance parallel computers, just as the Boeing 777 was simulated on a computer before manufacturing it.

Saini, Subhash↗
NASA NTRS1996

RANS-MP: A Portable Parallel Navier-Stokes Solver

RANS-MP, a new implementation of a single-grid Navier-Stokes solver using the diagonalized Beam-Warming approximate-factorization scheme, is presented. This first release of the completely rewritten solver employs the following optimizations: (1) Bi-directional multi-partition method for the ADI solver part; this improves granularity and load balance; (2) Improved cache usage through elimination of non-unit-stride array access (possible in part due to multi-partitioning); (3) Preprocessing of communicating boundary conditions to streamline logic during time stepping; (4) Truly parallel, high-performance I/O using the newly-developed MPI-IO library; (5) Elimination of large amounts of redundant operations through efficient use of workspace. Results of some realistic wing computations on the IBM SP2 computer will be presented. We will demonstrate that excellent absolute performance and scalability are obtained with RANS-MP, even for relatively small grid sizes. Besides high performance, an outstanding feature of RANS-MP is its true portability, due to the use of the portable message passing and I/O libraries MPI and MPI-IO.

VanderWijngaart, Rob F.↗
NASA NTRS1994

The architecture of the High Performance Storage System (HPSS)

The rapid growth in the size of datasets has caused a serious imbalance in I/O and storage system performance and functionality relative to application requirements and the capabilities of other system components. The High Performance Storage System (HPSS) is a scalable, next-generation storage system that will meet the functionality and performance requirements or large-scale scientific and commercial computing environments. Our goal is to improve the performance and capacity of storage by two orders of magnitude or more over what is available in the general or mass marketplace today. We are also providing corresponding improvements in architecture and functionality. This paper describes the architecture and functionality of HPSS.

Teaff, Danny↗
NASA NTRS1999

I/O Parallelization for the Goddard Earth Observing System Data Assimilation System (GEOS DAS)

The National Aeronautics and Space Administration (NASA) Data Assimilation Office (DAO) at the Goddard Space Flight Center (GSFC) has developed the GEOS DAS, a data assimilation system that provides production support for NASA missions and will support NASA's Earth Observing System (EOS) in the coming years. The DAO's support of the EOS project along with the requirement of producing long-term reanalysis datasets with an unvarying system levy a large I/O burden on the future system. The DAO has been involved in prototyping parallel implementations of the GEOS DAS for a number of years and is now converting the production version from shared-memory parallelism to distributed-memory parallelism using the portable Message-Passing Interface (MPI). If the MPI-based GEOS DAS is to meet these production requirements, we must make I/O from the parallel system efficient. We have designed a scheme that allows efficient I/O processing while retaining portability, reducing the need for post-processing, and producing data formats that are required by our users, both internal and external. The first phase of the GEOS DAS Parallel I/O System (GPIOS) will expand upon the common method of gathering global data to a Single PE for output. Instead of using a PE also tasked with primary computation, a number of PEs will be dedicated to I/O and its related tasks. This allows the data transformations and formatting required prior to output to take place asynchronously with respect to the GEOS DAS assimilation cycle, improving performance and generating output data sets in a format convenient for our users. I/O PEs can be added as needed to handle larger data volumes or to meet user file specifications. We will show I/O performance results from a prototype MPI GCM integrated with GPIOS. Phase two of GPIOS development will examine ways of integrating new software technologies to further improve performance and build scalability into the system. The maturing of MPI-IO implementations and other supporting libraries such as parallel HDF should provide performance gains while retaining portability.

Lucchesi, R.↗
NASA NTRS2012

SpaceCube Mini

This version of the SpaceCube will be a full-fledged, onboard space processing system capable of 2500+ MIPS, and featuring a number of plug-andplay gigabit and standard interfaces, all in a condensed 3x3x3 form factor [less than 10 watts and less than 3 lb (approximately equal to 1.4 kg)]. The main processing engine is the Xilinx SIRF radiation- hardened-by-design Virtex-5 FX-130T field-programmable gate array (FPGA). Even as the SpaceCube 2.0 version (currently under test) is being targeted as the platform of choice for a number of the upcoming Earth Science Decadal Survey missions, GSFC has been contacted by customers who wish to see a system that incorporates key features of the version 2.0 architecture in an even smaller form factor. In order to fulfill that need, the SpaceCube Mini is being designed, and will be a very compact and low-power system. A similar flight system with this combination of small size, low power, low cost, adaptability, and extremely high processing power does not otherwise exist, and the SpaceCube Mini will be of tremendous benefit to GSFC and its partners. The SpaceCube Mini will utilize space-grade components. The primary processing engine of the Mini is the Xilinx Virtex-5 SIRF FX-130T radiation-hardened-by-design FPGA for critical flight applications in high-radiation environments. The Mini can also be equipped with a commercial Xilinx Virtex-5 FPGA with integrated PowerPCs for a low-cost, high-power computing platform for use in the relatively radiation- benign LEOs (low-Earth orbits). In either case, this version of the Space-Cube will weigh less than 3 pounds (.1.4 kg), conform to the CubeSat form-factor (10x10x10 cm), and will be low power (less than 10 watts for typical applications). The SpaceCube Mini will have a radiation-hardened Aeroflex FPGA for configuring and scrubbing the Xilinx FPGA by utilizing the onboard FLASH memory to store the configuration files. The FLASH memory will also be used for storing algorithm and application code for the PowerPCs and the Xilinx FPGA. In addition, it will feature highspeed DDR SDRAM (double data rate synchronous dynamic random-access memory) to store the instructions and data of active applications. This version will also feature SATA-II and Gigabit Ethernet interfaces. Furthermore, there will also be general-purpose, multi-gigabit interfaces. In addition, the system will have dozens of transceivers that can support LVDS (low-voltage differential signaling), RS-422, or SpaceWire. The SpaceCube Mini includes an I/O card that can be customized to meet the needs of each mission. This version of the SpaceCube will be designed so that multiple Minis can be networked together using SpaceWire, Ethernet, or even a custom protocol. Scalability can be provided by networking multiple SpaceCube Minis together. Rigid-Flex technology is being targeted for the construction of the SpaceCube Mini, which will make the extremely compact and low-weight design feasible. The SpaceCube Mini is designed to fit in the compact CubeSat form factor, thus allowing deployment in a new class of missions that the previous SpaceCube versions were not suited for. At the time of this reporting, engineering units should be available in the summer 2012.

Michael Lin↗
NASA NTRS2012

An Application-Based Performance Evaluation of NASAs Nebula Cloud Computing Platform

The high performance computing (HPC) community has shown tremendous interest in exploring cloud computing as it promises high potential. In this paper, we examine the feasibility, performance, and scalability of production quality scientific and engineering applications of interest to NASA on NASA's cloud computing platform, called Nebula, hosted at Ames Research Center. This work represents the comprehensive evaluation of Nebula using NUTTCP, HPCC, NPB, I/O, and MPI function benchmarks as well as four applications representative of the NASA HPC workload. Specifically, we compare Nebula performance on some of these benchmarks and applications to that of NASA s Pleiades supercomputer, a traditional HPC system. We also investigate the impact of virtIO and jumbo frames on interconnect performance. Overall results indicate that on Nebula (i) virtIO and jumbo frames improve network bandwidth by a factor of 5x, (ii) there is a significant virtualization layer overhead of about 10% to 25%, (iii) write performance is lower by a factor of 25x, (iv) latency for short MPI messages is very high, and (v) overall performance is 15% to 48% lower than that on Pleiades for NASA HPC applications. We also comment on the usability of the cloud platform.

Saini, Subhash↗
NASA NTRS1988

A case for Redundant Arrays of Inexpensive Disks (RAID)

Increasing performance of CPUs and memories will be squandered if not matched by a similar performance increase in I/O. While the capacity of Single Large Expensive Disks (SLED) has grown rapidly, the performance improvement of SLED has been modest. Redundant Arrays of Inexpensive Disks (RAID), based on the magnetic disk technology developed for personal computers, offers an attractive alternative to SLED, promising improvements of an order of magnitude in performance, reliability, power consumption, and scalability. This paper introduces five levels of RAIDs, giving their relative cost/performance, and compares RAID to an IBM 3380 and a Fujitsu Super Eagle.

Patterson, David A.↗
NASA NTRS1999

The SGI/CRAY T3E: Experiences and Insights

The focus of the HPCC Earth and Space Sciences (ESS) Project is capability computing - pushing highly scalable computing testbeds to their performance limits. The drivers of this focus are the Grand Challenge problems in Earth and space science: those that could not be addressed in a capacity computing environment where large jobs must continually compete for resources. These Grand Challenge codes require a high degree of communication, large memory, and very large I/O (throughout the duration of the processing, not just in loading initial conditions and saving final results). This set of parameters led to the selection of an SGI/Cray T3E as the current ESS Computing Testbed. The T3E at the Goddard Space Flight Center is a unique computational resource within NASA. As such, it must be managed to effectively support the diverse research efforts across the NASA research community yet still enable the ESS Grand Challenge Investigator teams to achieve their performance milestones, for which the system was intended. To date, all Grand Challenge Investigator teams have achieved the 10 GFLOPS milestone, eight of nine have achieved the 50 GFLOPS milestone, and three have achieved the 100 GFLOPS milestone. In addition, many technical papers have been published highlighting results achieved on the NASA T3E, including some at this Workshop. The successes enabled by the NASA T3E computing environment are best illustrated by the 512 PE upgrade funded by the NASA Earth Science Enterprise earlier this year. Never before has an HPCC computing testbed been so well received by the general NASA science community that it was deemed critical to the success of a core NASA science effort. NASA looks forward to many more success stories before the conclusion of the NASA-SGI/Cray cooperative agreement in June 1999.

Bernard, Lisa Hamet↗