Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data storage”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 487 records · Page 27
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

HDF5 in the exascale era: Delivering efficient and scalable parallel I/O for exascale applications

Accurately modeling real-world systems requires scientific applications at exascale to generate massive amounts of data and manage data storage efficiently. However, parallel input and output (I/O) faces challenges due to new application workflows and the state-of-the-art memory, interconnect, and storage architectures considered in exascale designs. The storage hierarchy has expanded with node-local persistent memory, solid-state storage, and traditional disk and tape-based storage, thus requiring efficiency at each layer and much more efficient data movement among these layers. This paper discusses how the ExaHDF5 project improved the I/O performance and data management for exascale architectures by enhancing HDF5, a widely used parallel I/O library. The team developed an Asynchronous I/O Virtual Object Layer (VOL) connector that allowed overlapping I/O with computation. They also created a Cache VOL to complement asynchronous I/O by incorporating fast storage layers, such as burst buffer and node-local storage, into the parallel I/O workflow through caching and staging data. Additionally, the team enabled data aggregation and I/O at the node level by using a Subfiling Virtual File Driver (VFD). To demonstrate superior I/O performance with HDF5 at exascale, the ExaHDF5 team collaborated with several exascale applications. In this paper, we show I/O performance improvements for three applications: Cabana (a particle-based simulation library), EQSIM (a regional earthquake simulation software), and E3SM (a climate system modeling library).

Asynchronous I/Ol↗
NASA NTRS1994

A distributed parallel storage architecture and its potential application within EOSDIS

We describe the architecture, implementation, use of a scalable, high performance, distributed-parallel data storage system developed in the ARPA funded MAGIC gigabit testbed. A collection of wide area distributed disk servers operate in parallel to provide logical block level access to large data sets. Operated primarily as a network-based cache, the architecture supports cooperation among independently owned resources to provide fast, large-scale, on-demand storage to support data handling, simulation, and computation.

Johnston, William E.↗
DOE OSTI2011

Tolerating Correlated Failures for Generalized Cartesian Distributions via Bipartite Matching

Faults are expected to play an increasingly important role in how algorithms and applications are designed to run on future extreme-scale systems. A key ingredient of any approach to fault tolerance is effective support for fault tolerant data storage. A typical application execution consists of phases in which certain data structures are modified while others are read-only. Often, read-only data structures constitute a large fraction of total memory consumed. Fault tolerance for read-only data can be ensured through the use of checksums or parities, without resorting to expensive in-memory duplication or checkpointing to secondary storage. In this paper, we present a graph-matching approach to compute and store parity data for read-only matrices that are compatible with fault tolerant linear algebra (FTLA). Typical approaches only support blocked data distributions with each process holding one block with the parity located on additional processes. The matrices are assumed to be blocked by a cartesian grid with each block assigned to a process. We consider a generalized distribution in which each process can be assigned arbitrary blocks. We also account for the fact that multiple processes might be part of the same failure unit, say an SMP node. The flexibility enabled by our novel application of graph matching extends fault tolerance support to data distributions beyond those supported by prior work. We evaluate the matching implementations and cost to compute the parity and recover lost data, demonstrating the low overhead incurred by our approach.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
NASA NTRS1993

Flexible storage medium for write-once optical tape

A write-once data storage media was developed which is suitable for optical tape applications. The media is manufactured using a continuous film process to deposit a ternary alloy of tin, bismuth, and copper. This laser sensitive layer is sputter deposited onto commercial plastic web as a single-layer thin film. A second layer is sequentially deposited on top of the alloy to enhance the media performance and act as an abrasion resistant hard overcoat. The media was observed to have laser write sensitivities of less than 2.0 njoules/bit, carrier-to-noise levels of greater than 50dB's, modulation depths of approximately 100 percent, read-margins of greater than 35, uniform grain sizes of less than 200 Angstroms, and a media lifetime that exceeds 10 years. Prototype tape media was produced for use in the CREO drive system. The active and overcoat materials are first sputter deposited onto three mil PET film in a single pass through the vacuum coating system, and then converted down into multiple reels of 35mm x 880m tape. One mil PET film was also coated in this manner and then slit and packaged into 3480 tape cartridges.

Strandjord, Andrew J. G.↗
DOE OSTI2024

International Offshore Geologic Carbon Storage Inventory and Data Collection

We present an interactive data collection to aggregate, understand, and disseminate the data that are publicly available to support offshore GCS which can be leveraged by stakeholders to understand where GCS may be viable offshore, create GCS project analogs, and address challenges to GCS in offshore environments. The Offshore Geologic Carbon Storage Data Collection is an Experience Builder web application of multiple web mapping applications, aggregated into a single tool for each data type for access, visualization, and exploration. We also present a spatial inventory of global offshore GCS efforts to visualize the scale and locations of actualized and potential offshore GCS. It includes project location, project type and stage, CO2 storage resource potential, injection rate, reservoir and seal geology, and key literature references. Quantitative and qualitative comparisons of the distribution and magnitude of projects by their attributes lends spatial insight into the status of global GCS operations and storage resource potential, thereby enabling comparative assessments and cross-cutting knowledge transfer for projects in development. These datasets illuminate trends in ongoing offshore projects and can be leveraged by stakeholders to estimate storage resources, identify subsurface analogs, review regulations, and address challenges to offshore GCS. Additionally, opportunities for concurrent decarbonization strategies can be identified.

Mulhern, Julia↗
DOE OSTI2021

Error-controlled, progressive, and adaptable retrieval of scientific data with multilevel decomposition

Extreme-scale simulations and high-resolution instruments have been generating an increasing amount of data, which poses significant challenges to not only data storage during the run, but also post-processing where data will be repeatedly retrieved and analyzed for a long period of time. The challenges in satisfying a wide range of post-hoc analysis needs while minimizing the I/O overhead caused by inappropriate and/or excessive data retrieval should never be left unmanaged. In this paper, we propose a data refactoring, compressing, and retrieval framework capable of 1) fine-grained data refactoring with regard to precision; 2) incrementally retrieving and recomposing the data in terms of various error bounds; and 3) adaptively retrieving data in multi-precision and multi-resolution with respect to different analysis. With the progressive data re-composition and the adaptable retrieval algorithms, our framework significantly reduces the amount of data retrieved when multiple incremental precision are requested and/or the downstream analysis time when coarse resolution is used. Experiments show that the amount of data retrieved under the same progressively requested error bound using our framework is 64% less than that using state-of-the-art single-error-bounded approaches. Parallel experiments with up to 1, 024 cores and ~ 600 GB data in total show that our approach yields 1.36× and 2.52× performance over existing approaches in writing to and reading from persistent storage systems, respectively.

Liang, Xin↗
DOE OSTI2022

Data reduction in deterministic neutron transport calculations using machine learning

Neutron cross section matrices for fission and scattering data are required for each material, temperature, and enrichment level to calculate the neutron transport equation accurately. Here, this information can be a limiting factor when using the multigroup discrete ordinates (S N ) method when the number of energy groups is large. Machine Learning (ML) can be used to replace the need for the cross section matrices by reproducing the function that maps the scalar flux to the scattering and fission sources. Through the use of autoencoders and Deep Jointly-Informed Neural Networks (DJINN), the data storage requirements are reduced by 94% of the original data for a 618 group problem. This is accomplished while preserving the scalar flux, maintaining generality, and decreasing wall clock times.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2014

The Practical Obstacles of Data Transfer: Why researchers still love scp

The importance of computing facilities is heralded every six months with the announcement of the new Top500 list, showcasing the world s fastest supercomputers. Unfortu- nately, with great computing capability does not come great long-term data storage capacity, which often means users must move their data to their local site archive, to remote sites where they may be doing future computation or anal- ysis, or back to their home institution, else face the dreaded data purge that most HPC centers employ to keep utiliza- tion of large parallel filesystems low to manage performance and capacity. At HPC centers, data transfer is crucial to the scientific workflow and will increase in importance as computing systems grow in size. The Energy Sciences Net- work (ESnet) recently launched its fifth generation network, a 100 Gbps high-performance, unclassified national network connecting more than 40 DOE research sites to support scientific research and collaboration. Despite the tenfold increase in bandwidth to DOE research sites amenable to multiple data transfer streams and high throughput, in prac- tice, researchers often under-utilize the network and resort to painfully-slow single stream transfer methods such as scp to avoid the complexity of using multiple stream tools such as GridFTP and bbcp, and contend with frustration from the lack of consistency of available tools between sites. In this study we survey and assess the data transfer methods pro- vided at several DOE supported computing facilities, includ- ing both leadership-computing facilities, connected through ESnet. We present observed transfer rates, suggested opti- mizations, and discuss the obstacles the tools must overcome to receive wide-spread adoption over scp.

Nam, Hai Ah↗
DOE OSTI2024

Accurate assessment of land–atmosphere coupling in climate models requires high-frequency data output

Land–atmosphere (L–A) interactions are important for understanding convective processes, climate feedbacks, the development and perpetuation of droughts, heatwaves, pluvials, and other land-centered climate anomalies. Local L–A coupling (LoCo) metrics capture relevant L–A processes, highlighting the impact of soil and vegetation states on surface flux partitioning and the impact of surface fluxes on boundary layer (BL) growth and development and the entrainment of air above the BL. A primary goal of the Climate Process Team in the Coupling Land and Atmospheric Subgrid Parameterizations (CLASP) project is parameterizing and characterizing the impact of subgrid heterogeneity in global and regional Earth system models (ESMs) to improve the connection between land and atmospheric states and processes. A critical step in achieving that aim is the incorporation of L–A metrics, especially LoCo metrics, into climate model diagnostic process streams. However, because land–atmosphere interactions span timescales of minutes (e.g., turbulent fluxes), hours (e.g., BL growth and decay), days (e.g., soil moisture memory), and seasons (e.g., variability in behavioral regimes between soil moisture and latent heat flux), with multiple processes of interest happening in different geographic regions at different times of year, there is not a single metric that captures all the modes, means, and methods of interaction between the land and the atmosphere. And while monthly means of most of the LoCo-relevant variables are routinely saved from ESM simulations, data storage constraints typically preclude routine archival of the hourly data that would enable the calculation of all LoCo metrics. Here, we outline a reasonable data request that would allow for adequate characterization of sub-daily coupling processes between the land and the atmosphere, preserving enough sub-daily output to describe, analyze, and better understand L–A coupling in modern climate models. A secondary request involves embedding calculations within the models to determine mean properties in and above the BL to further improve characterization of model behavior. Higher-frequency model output will (i) allow for more direct comparison with observational field campaigns on process-relevant timescales, (ii) enable demonstration of inter-model spread in L–A coupling processes, and (iii) aid in targeted identification of sources of deficiencies and opportunities for improvement of the models.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS1980

Earth observation data systems in the 1980's

The requirements for future data systems for earth resource observation data are examined. Estimates are made for both expected data volumes and data delivery requirements. Research and development activities presently underway are described, including (1) high-speed processors such as the massively parallel processor; (2) data storage systems, including optical disk configurations and (3) distributed data systems such as high-speed local networks and the applications data service system.

Bracken, P. A.↗
NASA NTRS2005

Ground System for Solar Dynamics Observatory (SDO) Mission

NASA s Goddard Space Flight Center (GSFC) has recently completed its Critical Design Review (CDR) of a new dual Ka and S-band ground system for the Solar Dynamics Observatory (SDO) Mission. SDO, the flagship mission under the new Living with a Star Program Office, is one of GSFC s most recent large-scale in-house missions. The observatory is scheduled for launch in August 2008 from the Kennedy Space Center aboard an Atlas-5 expendable launch vehicle. Unique to this mission is an extremely challenging science data capture requirement. The mission is required to capture 99.99% of available science over 95% of all observation opportunities. Due to the continuous, high volume (150 Mbps) science data rate, no on-board storage of science data will be implemented on this mission. With the observatory placed in a geo-synchronous orbit at 36,000 kilometers within view of dedicated ground stations, the ground system will in effect implement a "real-time" science data pipeline with appropriate data accounting, data storage, data distribution, data recovery, and automated system failure detection and correction to keep the science data flowing continuously to three separate Science Operations Centers (SOCs). Data storage rates of approx. 45 Tera-bytes per month are expected. The Mission Operations Center (MOC) will be based at GSFC and is designed to be highly automated. Three SOCs will share in the observatory operations, each operating their own instrument. Remote operations of a multi-antenna ground station in White Sands, New Mexico from the MOC is part of the design baseline.

Tann, Hun K.↗
NASA NTRS2024

Accurate Assessment of Land-Atmosphere Coupling in Climate Models Requires High Frequency Data Output

Land-atmosphere (L-A) interactions are important for understanding convective processes, climate feedbacks, the development and perpetuation of droughts, heatwaves, pluvials, and other land-centred climate anomalies. Local L-A coupling (LoCo) metrics capture relevant L-A processes, highlighting the impact of soil and vegetation states on surface flux partitioning, and the impact of surface fluxes on boundary layer (BL) growth, development, and entrainment of air above the BL. A primary goal of the Climate Process Team on Coupling Land and Atmospheric Subgrid Parameterizations (CLASP) is parameterizing and characterizing the impact of subgrid heterogeneity in global and regional earth system models (ESMs) to improve the connection between land and atmospheric states and processes. A critical step in achieving that aim is the incorporation of L-A metrics, especially LoCo metrics, into climate model diagnostic process streams. However, because land-atmosphere interactions span time scales of minutes (e.g., turbulent fluxes), hours (e.g., BL growth and decay), days (e.g., soil moisture memory), and seasons (e.g., variability of behavioural regimes between soil moisture and latent heat flux), with multiple processes of interest happening in different geographic regions at different times of year, there is not a single metric that captures all the modes, means, and methods of interaction between the land and the atmosphere. And while monthly means of most of the LoCo-relevant variables are routinely saved from ESM simulations, data storage constraints typically preclude routine archival of the hourly data that would enable the calculation of all LoCo metrics. Here we outline a reasonable data request that would allow for adequate characterization of sub-daily coupling processes between the land and the atmosphere, preserving enough sub-daily output to describe, analyse, and better understand L-A coupling in modern climate models. A secondary request involves embedding calculations within the models to determine mean properties in and above the BL to further improve characterization of model behaviour. Higher-frequency model output will (i) allow for more direct comparison with observational field campaigns on process-relevant time scales, (ii) enable demonstration of inter-model spread in L-A coupling processes, and (iii) aid in targeted identification of sources of deficiencies and opportunities for improvement of the models.

Kirsten L. Findell↗
DOE OSTI2015

Climate Science Performance, Data and Productivity on Titan

Climate Science models are flagship codes for the largest of high performance computing (HPC) resources, both in visibility, with the newly launched Department of Energy (DOE) Accelerated Climate Model for Energy (ACME) effort, and in terms of significant fractions of system usage. The performance of the DOE ACME model is captured with application level timers and examined through a sizeable run archive. Performance and variability of compute, queue time and ancillary services are examined. As Climate Science advances in the use of HPC resources there has been an increase in the required human and data systems to achieve programs goals. A description of current workflow processes (hardware, software, human) and planned automation of the workflow, along with historical and projected data in motion and at rest data usage, are detailed. The combination of these two topics motivates a description of future systems requirements for DOE Climate Modeling efforts, focusing on the growth of data storage and network and disk bandwidth required to handle data at an acceptable rate.

Mayer, Benjamin W↗
NASA NTRS2018

Accessing Data Stored in Amazon S3 Using the Hyrax OPeNDAP Server

For three years we have been investigating data storage and retrieval in the Amazon Cloud, striving to optimize both storage structures and (cache-enhanced retrieval processes. These optimizations are hidden from data users, who simply employ the Data Access Protocol (DAP), a popular, Web-based Application Programmer Interface (API) developed by OPeNDAP. We present our collected findings and their realization in the Hyrax data server. Specific techniques include caching data on spinning disk; accessing sharded data in place; optimizing the organizational structures for data within the flat key-value space of Simple Storage Service (S3); and employing an API extension that permits simultaneous operations on many datasets in a single request.

Data access↗
DOE OSTI2021

Improving the Accessibility and Usability of Geothermal Information with Data Lakes and Data Pipelines on the Geothermal Data Repository: Preprint

The Geothermal Data Repository (GDR) provides universal access to data and information resulting from research and development activities funded by the Department of Energy (DOE). The GDR has extended this universal access to big data through integration with data lakes developed by the Open Energy Data Initiative (OEDI). Previously, large datasets such as seismic waveform or distributed acoustic sensing (DAS) data could only be accessed by institutions with high performance data storage and compute capabilities, effectively limiting the accessibility of big data to national labs, larger universities, and major corporations. Moreover, the time and resources needed to transport big data and configure them can produce additional barriers to use. Many of the standard formats used for structured data models (also known as content models) are incapable of handling big data and can introduce additional usability problems, often requiring data to be reformatted prior to use. This paper will explore how recent integrations between the GDR and the OEDI data lake have improved the accessibility and usability of geothermal data in a big way, making the data available to a broader audience, and enabling collaborative analysis and innovation across the greater geothermal industry.

access↗
DOE OSTI2021

Improving the Accessibility and Usability of Geothermal Information with Data Lakes and Data Pipelines on the Geothermal Data Repository

The Geothermal Data Repository (GDR) provides universal access to data and information resulting from research and development activities funded by the Department of Energy (DOE). The GDR has extended this universal access to big data through integration with data lakes developed by the Open Energy Data Initiative (OEDI). Previously, large datasets such as seismic waveform or distributed acoustic sensing (DAS) data could only be accessed by institutions with high performance data storage and compute capabilities, effectively limiting the accessibility of big data to national labs, larger universities, and major corporations. Moreover, the time and resources needed to transport big data and configure them can produce additional barriers to use. Many of the standard formats used for structured data models (also known as content models) are incapable of handling big data and can introduce additional usability problems, often requiring data to be reformatted prior to use. This paper will explore how recent integrations between the GDR and the OEDI data lake have improved the accessibility and usability of geothermal data in a big way, making the data available to a broader audience, and enabling collaborative analysis and innovation across the greater geothermal industry.

access↗
NASA NTRS1992

National Media Laboratory

A review of the National Media Laboratory (NML) is presented. The mission of the NML is to support current government user data storage needs and assist them in getting the most efficient 'commercial' solutions in the future. The motivation for a National Media Laboratory is as follows: recording systems are the major government image and data exploitation bottleneck; government data recording performance and storage requirements lead commercial practice by 3-5 years; the supporting commercial recorder industry is large but principally focused on video not data formats; lack of standards; and lack of transfer of commercial knowledge base to program offices and operational sites.

Clark, Robert↗