Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “DATA STORAGE”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 469 records · Page 26
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

International Offshore Geologic Carbon Storage Inventory and Data Collection

We present an interactive data collection to aggregate, understand, and disseminate the data that are publicly available to support offshore GCS which can be leveraged by stakeholders to understand where GCS may be viable offshore, create GCS project analogs, and address challenges to GCS in offshore environments. The Offshore Geologic Carbon Storage Data Collection is an Experience Builder web application of multiple web mapping applications, aggregated into a single tool for each data type for access, visualization, and exploration. We also present a spatial inventory of global offshore GCS efforts to visualize the scale and locations of actualized and potential offshore GCS. It includes project location, project type and stage, CO2 storage resource potential, injection rate, reservoir and seal geology, and key literature references. Quantitative and qualitative comparisons of the distribution and magnitude of projects by their attributes lends spatial insight into the status of global GCS operations and storage resource potential, thereby enabling comparative assessments and cross-cutting knowledge transfer for projects in development. These datasets illuminate trends in ongoing offshore projects and can be leveraged by stakeholders to estimate storage resources, identify subsurface analogs, review regulations, and address challenges to offshore GCS. Additionally, opportunities for concurrent decarbonization strategies can be identified.

Mulhern, Julia↗
DOE OSTI2021

Error-controlled, progressive, and adaptable retrieval of scientific data with multilevel decomposition

Extreme-scale simulations and high-resolution instruments have been generating an increasing amount of data, which poses significant challenges to not only data storage during the run, but also post-processing where data will be repeatedly retrieved and analyzed for a long period of time. The challenges in satisfying a wide range of post-hoc analysis needs while minimizing the I/O overhead caused by inappropriate and/or excessive data retrieval should never be left unmanaged. In this paper, we propose a data refactoring, compressing, and retrieval framework capable of 1) fine-grained data refactoring with regard to precision; 2) incrementally retrieving and recomposing the data in terms of various error bounds; and 3) adaptively retrieving data in multi-precision and multi-resolution with respect to different analysis. With the progressive data re-composition and the adaptable retrieval algorithms, our framework significantly reduces the amount of data retrieved when multiple incremental precision are requested and/or the downstream analysis time when coarse resolution is used. Experiments show that the amount of data retrieved under the same progressively requested error bound using our framework is 64% less than that using state-of-the-art single-error-bounded approaches. Parallel experiments with up to 1, 024 cores and ~ 600 GB data in total show that our approach yields 1.36× and 2.52× performance over existing approaches in writing to and reading from persistent storage systems, respectively.

Liang, Xin↗
DOE OSTI2022

Data reduction in deterministic neutron transport calculations using machine learning

Neutron cross section matrices for fission and scattering data are required for each material, temperature, and enrichment level to calculate the neutron transport equation accurately. Here, this information can be a limiting factor when using the multigroup discrete ordinates (S N ) method when the number of energy groups is large. Machine Learning (ML) can be used to replace the need for the cross section matrices by reproducing the function that maps the scalar flux to the scattering and fission sources. Through the use of autoencoders and Deep Jointly-Informed Neural Networks (DJINN), the data storage requirements are reduced by 94% of the original data for a 618 group problem. This is accomplished while preserving the scalar flux, maintaining generality, and decreasing wall clock times.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2014

The Practical Obstacles of Data Transfer: Why researchers still love scp

The importance of computing facilities is heralded every six months with the announcement of the new Top500 list, showcasing the world s fastest supercomputers. Unfortu- nately, with great computing capability does not come great long-term data storage capacity, which often means users must move their data to their local site archive, to remote sites where they may be doing future computation or anal- ysis, or back to their home institution, else face the dreaded data purge that most HPC centers employ to keep utiliza- tion of large parallel filesystems low to manage performance and capacity. At HPC centers, data transfer is crucial to the scientific workflow and will increase in importance as computing systems grow in size. The Energy Sciences Net- work (ESnet) recently launched its fifth generation network, a 100 Gbps high-performance, unclassified national network connecting more than 40 DOE research sites to support scientific research and collaboration. Despite the tenfold increase in bandwidth to DOE research sites amenable to multiple data transfer streams and high throughput, in prac- tice, researchers often under-utilize the network and resort to painfully-slow single stream transfer methods such as scp to avoid the complexity of using multiple stream tools such as GridFTP and bbcp, and contend with frustration from the lack of consistency of available tools between sites. In this study we survey and assess the data transfer methods pro- vided at several DOE supported computing facilities, includ- ing both leadership-computing facilities, connected through ESnet. We present observed transfer rates, suggested opti- mizations, and discuss the obstacles the tools must overcome to receive wide-spread adoption over scp.

Nam, Hai Ah↗
DOE OSTI2024

Accurate assessment of land–atmosphere coupling in climate models requires high-frequency data output

Land–atmosphere (L–A) interactions are important for understanding convective processes, climate feedbacks, the development and perpetuation of droughts, heatwaves, pluvials, and other land-centered climate anomalies. Local L–A coupling (LoCo) metrics capture relevant L–A processes, highlighting the impact of soil and vegetation states on surface flux partitioning and the impact of surface fluxes on boundary layer (BL) growth and development and the entrainment of air above the BL. A primary goal of the Climate Process Team in the Coupling Land and Atmospheric Subgrid Parameterizations (CLASP) project is parameterizing and characterizing the impact of subgrid heterogeneity in global and regional Earth system models (ESMs) to improve the connection between land and atmospheric states and processes. A critical step in achieving that aim is the incorporation of L–A metrics, especially LoCo metrics, into climate model diagnostic process streams. However, because land–atmosphere interactions span timescales of minutes (e.g., turbulent fluxes), hours (e.g., BL growth and decay), days (e.g., soil moisture memory), and seasons (e.g., variability in behavioral regimes between soil moisture and latent heat flux), with multiple processes of interest happening in different geographic regions at different times of year, there is not a single metric that captures all the modes, means, and methods of interaction between the land and the atmosphere. And while monthly means of most of the LoCo-relevant variables are routinely saved from ESM simulations, data storage constraints typically preclude routine archival of the hourly data that would enable the calculation of all LoCo metrics. Here, we outline a reasonable data request that would allow for adequate characterization of sub-daily coupling processes between the land and the atmosphere, preserving enough sub-daily output to describe, analyze, and better understand L–A coupling in modern climate models. A secondary request involves embedding calculations within the models to determine mean properties in and above the BL to further improve characterization of model behavior. Higher-frequency model output will (i) allow for more direct comparison with observational field campaigns on process-relevant timescales, (ii) enable demonstration of inter-model spread in L–A coupling processes, and (iii) aid in targeted identification of sources of deficiencies and opportunities for improvement of the models.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS1980

Earth observation data systems in the 1980's

The requirements for future data systems for earth resource observation data are examined. Estimates are made for both expected data volumes and data delivery requirements. Research and development activities presently underway are described, including (1) high-speed processors such as the massively parallel processor; (2) data storage systems, including optical disk configurations and (3) distributed data systems such as high-speed local networks and the applications data service system.

Bracken, P. A.↗
NASA NTRS2005

Ground System for Solar Dynamics Observatory (SDO) Mission

NASA s Goddard Space Flight Center (GSFC) has recently completed its Critical Design Review (CDR) of a new dual Ka and S-band ground system for the Solar Dynamics Observatory (SDO) Mission. SDO, the flagship mission under the new Living with a Star Program Office, is one of GSFC s most recent large-scale in-house missions. The observatory is scheduled for launch in August 2008 from the Kennedy Space Center aboard an Atlas-5 expendable launch vehicle. Unique to this mission is an extremely challenging science data capture requirement. The mission is required to capture 99.99% of available science over 95% of all observation opportunities. Due to the continuous, high volume (150 Mbps) science data rate, no on-board storage of science data will be implemented on this mission. With the observatory placed in a geo-synchronous orbit at 36,000 kilometers within view of dedicated ground stations, the ground system will in effect implement a "real-time" science data pipeline with appropriate data accounting, data storage, data distribution, data recovery, and automated system failure detection and correction to keep the science data flowing continuously to three separate Science Operations Centers (SOCs). Data storage rates of approx. 45 Tera-bytes per month are expected. The Mission Operations Center (MOC) will be based at GSFC and is designed to be highly automated. Three SOCs will share in the observatory operations, each operating their own instrument. Remote operations of a multi-antenna ground station in White Sands, New Mexico from the MOC is part of the design baseline.

Tann, Hun K.↗
NASA NTRS2024

Accurate Assessment of Land-Atmosphere Coupling in Climate Models Requires High Frequency Data Output

Land-atmosphere (L-A) interactions are important for understanding convective processes, climate feedbacks, the development and perpetuation of droughts, heatwaves, pluvials, and other land-centred climate anomalies. Local L-A coupling (LoCo) metrics capture relevant L-A processes, highlighting the impact of soil and vegetation states on surface flux partitioning, and the impact of surface fluxes on boundary layer (BL) growth, development, and entrainment of air above the BL. A primary goal of the Climate Process Team on Coupling Land and Atmospheric Subgrid Parameterizations (CLASP) is parameterizing and characterizing the impact of subgrid heterogeneity in global and regional earth system models (ESMs) to improve the connection between land and atmospheric states and processes. A critical step in achieving that aim is the incorporation of L-A metrics, especially LoCo metrics, into climate model diagnostic process streams. However, because land-atmosphere interactions span time scales of minutes (e.g., turbulent fluxes), hours (e.g., BL growth and decay), days (e.g., soil moisture memory), and seasons (e.g., variability of behavioural regimes between soil moisture and latent heat flux), with multiple processes of interest happening in different geographic regions at different times of year, there is not a single metric that captures all the modes, means, and methods of interaction between the land and the atmosphere. And while monthly means of most of the LoCo-relevant variables are routinely saved from ESM simulations, data storage constraints typically preclude routine archival of the hourly data that would enable the calculation of all LoCo metrics. Here we outline a reasonable data request that would allow for adequate characterization of sub-daily coupling processes between the land and the atmosphere, preserving enough sub-daily output to describe, analyse, and better understand L-A coupling in modern climate models. A secondary request involves embedding calculations within the models to determine mean properties in and above the BL to further improve characterization of model behaviour. Higher-frequency model output will (i) allow for more direct comparison with observational field campaigns on process-relevant time scales, (ii) enable demonstration of inter-model spread in L-A coupling processes, and (iii) aid in targeted identification of sources of deficiencies and opportunities for improvement of the models.

Kirsten L. Findell↗
DOE OSTI2015

Climate Science Performance, Data and Productivity on Titan

Climate Science models are flagship codes for the largest of high performance computing (HPC) resources, both in visibility, with the newly launched Department of Energy (DOE) Accelerated Climate Model for Energy (ACME) effort, and in terms of significant fractions of system usage. The performance of the DOE ACME model is captured with application level timers and examined through a sizeable run archive. Performance and variability of compute, queue time and ancillary services are examined. As Climate Science advances in the use of HPC resources there has been an increase in the required human and data systems to achieve programs goals. A description of current workflow processes (hardware, software, human) and planned automation of the workflow, along with historical and projected data in motion and at rest data usage, are detailed. The combination of these two topics motivates a description of future systems requirements for DOE Climate Modeling efforts, focusing on the growth of data storage and network and disk bandwidth required to handle data at an acceptable rate.

Mayer, Benjamin W↗
NASA NTRS2018

Accessing Data Stored in Amazon S3 Using the Hyrax OPeNDAP Server

For three years we have been investigating data storage and retrieval in the Amazon Cloud, striving to optimize both storage structures and (cache-enhanced retrieval processes. These optimizations are hidden from data users, who simply employ the Data Access Protocol (DAP), a popular, Web-based Application Programmer Interface (API) developed by OPeNDAP. We present our collected findings and their realization in the Hyrax data server. Specific techniques include caching data on spinning disk; accessing sharded data in place; optimizing the organizational structures for data within the flat key-value space of Simple Storage Service (S3); and employing an API extension that permits simultaneous operations on many datasets in a single request.

Data access↗
DOE OSTI2021

Improving the Accessibility and Usability of Geothermal Information with Data Lakes and Data Pipelines on the Geothermal Data Repository: Preprint

The Geothermal Data Repository (GDR) provides universal access to data and information resulting from research and development activities funded by the Department of Energy (DOE). The GDR has extended this universal access to big data through integration with data lakes developed by the Open Energy Data Initiative (OEDI). Previously, large datasets such as seismic waveform or distributed acoustic sensing (DAS) data could only be accessed by institutions with high performance data storage and compute capabilities, effectively limiting the accessibility of big data to national labs, larger universities, and major corporations. Moreover, the time and resources needed to transport big data and configure them can produce additional barriers to use. Many of the standard formats used for structured data models (also known as content models) are incapable of handling big data and can introduce additional usability problems, often requiring data to be reformatted prior to use. This paper will explore how recent integrations between the GDR and the OEDI data lake have improved the accessibility and usability of geothermal data in a big way, making the data available to a broader audience, and enabling collaborative analysis and innovation across the greater geothermal industry.

access↗
DOE OSTI2021

Improving the Accessibility and Usability of Geothermal Information with Data Lakes and Data Pipelines on the Geothermal Data Repository

The Geothermal Data Repository (GDR) provides universal access to data and information resulting from research and development activities funded by the Department of Energy (DOE). The GDR has extended this universal access to big data through integration with data lakes developed by the Open Energy Data Initiative (OEDI). Previously, large datasets such as seismic waveform or distributed acoustic sensing (DAS) data could only be accessed by institutions with high performance data storage and compute capabilities, effectively limiting the accessibility of big data to national labs, larger universities, and major corporations. Moreover, the time and resources needed to transport big data and configure them can produce additional barriers to use. Many of the standard formats used for structured data models (also known as content models) are incapable of handling big data and can introduce additional usability problems, often requiring data to be reformatted prior to use. This paper will explore how recent integrations between the GDR and the OEDI data lake have improved the accessibility and usability of geothermal data in a big way, making the data available to a broader audience, and enabling collaborative analysis and innovation across the greater geothermal industry.

access↗
NASA NTRS1992

National Media Laboratory

A review of the National Media Laboratory (NML) is presented. The mission of the NML is to support current government user data storage needs and assist them in getting the most efficient 'commercial' solutions in the future. The motivation for a National Media Laboratory is as follows: recording systems are the major government image and data exploitation bottleneck; government data recording performance and storage requirements lead commercial practice by 3-5 years; the supporting commercial recorder industry is large but principally focused on video not data formats; lack of standards; and lack of transfer of commercial knowledge base to program offices and operational sites.

Clark, Robert↗
NASA NTRS2004

Impact of CGNS on CFD Workflow

CFD tools are an integral part of industrial and research processes, for which the amount of data is increasing at a high rate. These data are used in a multi-disciplinary fluid dynamics environment, including structural, thermal, chemical or even electrical topics. We show that the data specification is an important challenge that must be tackled to achieve an efficient workflow for use in this environment. We compare the process with other software techniques, such as network or database type, where past experiences showed how difficult it was to bridge the gap between completely general specifications and dedicated specific applications. We show two aspects of the use of CFD General Notation System (CGNS) that impact CFD workflow: as a data specification framework and as a data storage means. Then, we give examples of projects involving CFD workflows where the use of the CGNS standard leads to a useful method either for data specification, exchange, or storage.

Poinot, M.↗
NASA NTRS1991

Laser beam modeling in optical storage systems

A computer model has been developed that simulates light propagating through an optical data storage system. A model of a laser beam that originates at a laser diode, propagates through an optical system, interacts with a optical disk, reflects back from the optical disk into the system, and propagates to data and servo detectors is discussed.

Treptau, J. P.↗
NASA NTRS1980

Optical mass memory system (AMM-13). AMM-13 system segment specification

The performance, design, development, and test requirements for an optical mass data storage and retrieval system prototype (AMM-13) are established. This system interfaces to other system segments of the NASA End-to-End Data System via the Data Base Management System segment and is designed to have a storage capacity of 10 to the 13th power bits (10 to the 12th power bits on line). The major functions of the system include control, input and output, recording of ingested data, fiche processing/replication and storage and retrieval.

Bailey, G. A.↗
DOE OSTI2021

Storage Futures Study: Storage Technology Modeling Input Data Report

The Storage Futures Study (SFS) is a multiyear research project to explore the role and impact of energy storage in the evolving electricity sector of the United States. The SFS is designed to examine the potential impact of energy storage technology advancement on the deployment of utility-scale storage and the adoption of distributed storage, and the implications for future power system infrastructure investment and operations. This specific report synthesizes current and projected cost performance assumptions along with location availability for storage technologies through 2050 that will be used in scenario analysis for the SFS at both the bulk power and distribution system scales. For comparison and context, this report also presents a synthesis of current cost and performance characteristics of energy storage technologies for storage durations ranging from minutes to months and including mechanical, thermal, and electrochemical storage technologies for the electricity sector. This information is intended to cover a broad range of storage technologies that are currently receiving significant attention from the investment community as well as in the media. In the report, we emphasize that energy storage technologies must be described in terms of both their power (kilowatts [kW]) capacity and energy (kilowatt-hours [kWh]) capacity to assess their costs and potential use cases.

24 POWER TRANSMISSION AND DISTRIBUTION↗