Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “DATA STORAGE”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,279 records · Page 71
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

mzPeak: Designing a Scalable, Interoperable, and Future-Ready Mass Spectrometry Data Format

Advances in mass spectrometry (MS) instrumentation, such as higher resolution, faster scan speeds, and improved sensitivity, have significantly increased the volume and complexity of data. The growing adoption of imaging and ion mobility further amplifies these challenges across MS-based omics fields, including proteomics, metabolomics, and lipidomics. While these technologies unlock new possibilities, they also present significant challenges in data management, storage, and accessibility. Existing open formats, such as the XML-based community standards mzML and imzML, struggle to meet the demands of modern MS workflows due to their large file sizes, slow data access, and limited metadata support. Vendor-specific formats, while optimized for proprietary instruments, lack interoperability, comprehensive metadata support and long-term archival reliability. This white paper lays the groundwork for mzPeak, a next-generation community data format designed to address these challenges and support high-throughput, multi-dimensional MS workflows. By adopting a hybrid model that combines efficient binary storage for numerical data and both human and machine-readable metadata storage, mzPeak will reduce file sizes, accelerate data access, and offer a scalable, adaptable solution for evolving MS technologies. For researchers, mzPeak will enable enhanced interoperability across platforms, seamless support for complex workflows including ion mobility and MS imaging, and faster data access compared to existing community formats such as mzML. Its design will ensure data is managed in compliance with regulatory standards, essential for applications such as precision medicine and chemical safety, where long-term data integrity and accessibility are critical. For vendors, mzPeak provides a streamlined, open alternative to proprietary formats, reducing the burden of regulatory compliance while aligning with the industry's push for transparency and standardization. By offering a high-performance, interoperable solution, mzPeak positions vendors to meet customer demands for sustainable data management tools which will be able to handle emerging and future data types and workflows. mzPeak aspires to become the cornerstone of MS data management, empowering researchers, vendors, and developers to innovate and collaborate more effectively.

data formats↗
NASA NTRS1991

A system approach to archival storage

The introduction and viewgraphs of a discussion on a system approach to archival storage presented at the National Space Science Data Center (NSSDC) Mass Storage Workshop is included. The use of D-2 iron particles for archival storage is discussed along with how acceleration factors relating short-term tests to archival life times can be justified. Ampex Recording Systems is transferring D-2 video technology to data storage applications, and encountering concerns about corrosion. To protect the D-2 standard, Battelle tests were done on all four tapes in the Class 2 environment. Error rates were measured before and after the test on both exposed and control groups.

Corcoran, John W.↗
NASA NTRS2013

MERRA/AS: The MERRA Analytic Services Project Interim Report

MERRA AS is a cyberinfrastructure resource that will combine iRODS-based Climate Data Server (CDS) capabilities with Coudera MapReduce to serve MERRA analytic products, store the MERRA reanalysis data collection in an HDFS to enable parallel, high-performance, storage-side data reductions, manage storage-side driver, mapper, reducer code sets and realized objects for users, and provide a library of commonly used spatiotemporal operations that can be composed to enable higher-order analyses.

MapReduce↗
DOE OSTI2016

A method for simultaneous linear optics and coupling correction for storage rings with turn-by-turn beam position monitor data

We propose a method to simultaneously correct linear optics errors and linear coupling for storage rings using turn-by-turn (TbT) beam position monitor (BPM) data. The independent component analysis (ICA) method is used to isolate the betatron normal modes from the measured TbT BPM data. The betatron amplitudes and phase advances of the projections of the normal modes on the horizontal and vertical planes are then extracted, which, combined with dispersion measurement, are used to fit the lattice model. The fitting results are used for lattice correction. The method has been successfully demonstrated on the NSLS-II storage ring.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2017

Multivariate Empirical Mode Decomposition Based Signal Analysis and Efficient-Storage in Smart Grid

Wide-area-measurement systems (WAMSs) are used in smart grid systems to enable the efficient monitoring of grid dynamics. However, the overwhelming amount of data and the severe contamination from noise often impede the effective and efficient data analysis and storage of WAMS generated measurements. To solve this problem, we propose a novel framework that takes advantage of Multivariate Empirical Mode Decomposition (MEMD), a fully data-driven approach to analyzing non-stationary signals, dubbed MEMD based Signal Analysis (MSA). The frequency measurements are considered as a linear superposition of different oscillatory components and noise. The low-frequency components, corresponding to the long-term trend and inter-area oscillations, are grouped and compressed by MSA using the mean shift clustering algorithm. Whereas, higher-frequency components, mostly noise and potentially part of high-frequency inter-area oscillations, are analyzed using Hilbert spectral analysis and they are delineated by statistical behavior. By conducting experiments on both synthetic and real-world data, we show that the proposed framework can capture the characteristics, such as trends and inter-area oscillation, while reducing the data storage requirements

Big Data↗
DOE OSTI2015

Adding Data Management Services to Parallel File Systems

The objective of this project, called DAMASC for “Data Management in Scientific Computing”, is to coalesce data management with parallel file system management to present a declarative interface to scientists for managing, querying, and analyzing extremely large data sets efficiently and predictably. Managing extremely large data sets is a key challenge of exascale computing. The overhead, energy, and cost of moving massive volumes of data demand designs where computation is close to storage. In current architectures, compute/analysis clusters access data in a physically separate parallel file system and largely leave it scientist to reduce data movement. Over the past decades the high-end computing community has adopted middleware with multiple layers of abstractions and specialized file formats such as NetCDF-4 and HDF5. These abstractions provide a limited set of high-level data processing functions, but have inherent functionality and performance limitations: middleware that provides access to the highly structured contents of scientific data files stored in the (unstructured) file systems can only optimize to the extent that file system interfaces permit; the highly structured formats of these files often impedes native file system performance optimizations. We are developing Damasc, an enhanced high-performance file system with native rich data management services. Damasc will enable efficient queries and updates over files stored in their native byte-stream format while retaining the inherent performance of file system data storage via declarative queries and updates over views of underlying files. Damasc has four key benefits for the development of data-intensive scientific code: (1) applications can use important data-management services, such as declarative queries, views, and provenance tracking, that are currently available only within database systems; (2) the use of these services becomes easier, as they are provided within a familiar file-based ecosystem; (3) common optimizations, e.g., indexing and caching, are readily supported across several file formats, avoiding effort duplication; and (4) performance improves significantly, as data processing is integrated more tightly with data storage. Our key contributions are: SciHadoop which explores changes to MapReduce assumption by taking advantage of semantics of structured data while preserving MapReduce’s failure and resource management; DataMods which extends common abstractions of parallel file systems so they become programmable such that they can be extended to natively support a variety of data models and can be hooked into emerging distributed runtimes such as Stanford’s Legion; and Miso which combines Hadoop and relational data warehousing to minimize time to insight, taking into account the overhead of ingesting data into data warehousing.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2022

GridDS: Data Science Toolkit for Energy Grid Data

According to the U.S. Energy Information Administration (EIA), the demand for energy is expected to increase 50% by the year 20501. While energy standards, such as the Institute of Electrical and Electronics Engineers (IEEE) Standard 1547, (Basso 2015) and monitoring with wide area management systems (WAMS) (Liu 2017, Zhou 2016) have enabled large scale data collection and storage, the application of this data in mitigating costs associated with increased consumer demand is an ongoing focus for energy research. This ubiquitous data collection presents a promising opportunity for machine learning and data science to improve efficiency of distributed energy resources (DERs). The GridDS software toolkit is designed to leverage advanced metering infrastructure (AMI), outage management systems data (OMS), Supervisory control Data Acquisition (SCADA), and geographic information systems (GIS) to forecast future energy demands and detect incipient grid failures. GridDS is a python software library designed to be modular and generalizable to data recorded by DERs. In adapting to disparate datasets recorded by various WAMS, GridDS provides a range of unique functionality not presently implemented in current WAMS which have highly specific software infrastructure by design. GridDS functionality ranges from data specification and preparation, to training and validation for state of the art machine learning, to interactive data visualization. For data intake, GridDS combines: Pandera: a library for creating data specifications. TimeScaleDB: a postgresSQL database infrastructure for efficient storage of timeseries data. Dataset class: A custom dataset class / interface that ensures modularity between a range of synthetic and live recorded datasets. Is

Ladd, Alexander↗
DOE OSTI2025

Existing Hydropower Assets (EHA) Annual Net Generation Plant Database, 2003-2024

Existing Hydropower Asset (EHA) Annual Net Generation is a geospatial point-level dataset containing annual net generation over time (2003-2024) and key characteristics of operational U.S. hydropower plants with 1 megawatt or greater of nameplate capacity. EIA 923 and EHA are the primary sources of the derived data. Pumped storage and hybrid plants are excluded.

Johnson, Megan [ORNL] (ORCID:0000000290141741)↗
DOE OSTI2021

The feasibility of MT tipper data to monitor CO2 storage sites

Monitoring carbon storage sites using geophysical techniques is a critical component to the success and safety of storage programs. Currently, the primary methods of monitoring such sites are seismic and, to a much lesser extent, electromagnetics using active sources. The cost of such methods, especially seismic, can be prohibitively expensive. Natural source lectromagnetics, or magnetotellurics (MT), represents a low-cost, and underutilized method with the potential to aid CO2 monitoring efforts. Specifically, the tipper of MT data gives insight to the dimensionality of the subsurface and is able to detect the expansion front of a CO2 plume in a saline reservoir. We analyze the feasibility of using the tipper to monitor two shallow CO2 plumes and conclude that the tipper may be a suitable method for long-term monitoring.<br>

Kohnke, Colton↗
DOE OSTI2022

Rock Physics-Based Data Assimilation of Integrated Continuous Active-Source Seismic and Pressure Monitoring Data during Geological Carbon Storage

Summary There has been substantial controversy concerning the role of geological carbon storage (GCS) in sequestering anthropogenic carbon emissions to mitigate climate change and global warming. Arguments center on the inability to monitor a geological storage site precisely and continuously, especially highlighting the associated costs and spatiotemporal trade-offs when using conventional subsurface monitoring techniques (well logs, core samples, chemical tracers, and 4D seismics). Active surveillance of GCS sites is essential for managing and mitigating potential leaks but is also required by regulation. With the goal of enhancing the monitoring capability at GCS sites, we present a rock physics-based joint data assimilation model to study a popular GCS site at Cranfield, Mississippi, USA. Synthetic continuous active-source seismic monitoring (CASSM) data (in the form of Vp and Qp measurements) and wellbore pressure monitoring data are assimilated with an ensemble of reservoir realizations to monitor gas saturation and reservoir pressure changes over a period of 100 years. Synthetic seismic attributes are generated using rock physics models (RPMs) and wellbore pressure monitoring data are extracted from the ground truth. Two assimilation methods, ensemble Kalman filter (EnKF) and ensemble Kalman smoother (EnKS), are tested in an observation system simulation experiment (OSSE) environment to assess the prediction accuracy of the individual and composite observation systems. The joint monitoring system achieves more accurate estimates of gas saturation and pressure, across the time span from start of injection to end of forecast, as compared to a single type of monitoring tool and irrespective of data assimilation algorithm choice. These results indicate that jointly assimilated data from two types of sensors (in this case, crosswell seismic and downhole pressure) may lead to a more risk-reducing monitoring design. One would expect that more data, vis-à-vis inclusion of a new sensor type, will improve the accuracy of any GCS monitoring system. However, from a practical standpoint, one important question is whether such a gain in accuracy is worth the additional cost associated with the new sensor. This paper focuses on quantifying the gain in accuracy, such that a practitioner can answer this question.

Engineering↗
DOE OSTI2015

Parallel log structured file system collective buffering to achieve a compact representation of scientific and/or dimensional data

Collective buffering and data pattern solutions are provided for storage, retrieval, and/or analysis of data in a collective parallel processing environment. For example, a method can be provided for data storage in a collective parallel processing environment. The method comprises receiving data to be written for a plurality of collective processes within a collective parallel processing environment, extracting a data pattern for the data to be written for the plurality of collective processes, generating a representation describing the data pattern, and saving the data and the representation.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
NASA NTRS1976

Study report on laser storage and retrieval of image data

The theoretical foundation is presented for a system of real-time nonphotographic and nonmagnetic digital laser storage and retrieval of image data. The system utilizes diffraction-limited laser focusing upon thin metal films, melting elementary holes in the metal films in laser focus. The metal films are encapsulated in rotating flexible mylar discs which act as the permanent storage carries. Equal sized holes encompass two dimensional digital ensembles of information bits which are time-sequentially (bit by bit) stored and retrieved. The bits possess the smallest possible size, defined by the Rayleigh criterion of coherent physical optics. Space and time invariant reflective read-out of laser discs with a small laser, provides access to the stored digital information. By eliminating photographic and magnetic data processing, which characterize the previous state of the art, photographic grain, diffusion, and gamma-distortion do not exist. Similarly, magnetic domain structures, magnetic gaps, and magnetic read-out are absent with a digital laser disc system.

Becker, C. H.↗
NASA NTRS2021

Data Assimilation of Terrestrial Water Storage Observations to Estimate Precipitation Fluxes: A Synthetic Experiment

The Gravity Recovery and Climate Experiment (GRACE) mission and its Follow-On (GRACE-FO) mission provide unprecedented observations of terrestrial water storage (TWS) dynamics at basin to continental scales. Established GRACE data assimilation techniques directly adjust the simulated water storage components to improve the estimation of groundwater, streamflow, and snow water equivalent. Such techniques artificially add/subtract water to/from prognostic variables, thus upsetting the simulated water balance. To overcome this limitation, we propose and test an alternative assimilation scheme in which precipitation fluxes are adjusted to achieve the desired changes in simulated TWS. Using a synthetic data assimilation experiment, we show that the scheme improves performance skill in precipitation estimates in general, but that it is more robust for snowfall than for rainfall, and it fails in certain regions with strong horizontal gradients in precipitation. The results demonstrate that assimilation of TWS observations can help correct (adjust) the model’s precipitation forcing and, in turn, enhance model estimates of TWS, snow mass, soil moisture, runoff, and evaporation. A key limitation of the approach is the assumption that all errors in TWS originate from errors in precipitation. Nevertheless, the proposed approach produces more consistent improvements in simulated runoff than the established GRACE data assimilation techniques.

Data Assimilation↗
DOE OSTI2017

Measuring and interpreting borehole strainmeter data to improve CO2 storage

Injecting fluids into a well deforms the enveloping rocks in a complex pattern that increases in magnitude and expands outward with time, and measurements of this strain field may be useful for monitoring the injection process. A pair of instruments capable of measuring four components of strain and two components of tilt has been developed. One instrument is designed to be removeable, whereas the other one is grouted in place. The removeable strainmeter provides versatility and lower cost because it can be relocated, whereas the grout-in instrument provides better coupling to the formation, which should improve the strain signal. The new strainmeters have been deployed along with a Gladwin strainmeter at the Avant Field, an oil field north of Tulsa, Oklahoma. Oil and water are being pumped out, and water is being injected in multiple boreholes that intersect the Bartlesville formation at a depth of approximately 500 m at the Avant Field. The strainmeters have been deployed at a depth of 30 m near well 9A, an injection well near the edge of the field. Results from a brief, 4-hr-long injection test into well 9A show that the radial and circumferential strain increase (become tensile) with time during injection, reaching a maximum of several 10s of nanostrain. In another example, data from the Gladwin strainmeter showed the strain field changing with time when an injection well 1 km away was shut-in. This caused the strain along a radial direction to become tensile, while the circumferential strain became compressive. This trend reversed a week later when injection into the well resumed. The major axis of the principle strain aligned with the shut-in well. The observed strain signals are similar to results from poroelastic analyses, and preliminary inverse analyses sugest that strain signals can be used to estimate reservoir characteristics at the Avant Field.

Murdoch, Larry↗
DOE OSTI2022

Data-driven offshore CO 2 saline storage assessment methodology

The world produces approximately 50 billion tonnes of greenhouse gases annually. This is measured in CO 2 -equivalents, and geologic CO 2 storage has the potential to advance decarbonization and mitigate greenhouse gas emissions. New technologies to assess offshore carbon storage are needed to address resource, regulatory, and commercial needs. Although most efforts to assess storage resources focus on onshore criteria, offshore reservoirs offer significant storage potential and distinct development challenges. Potential advantages of offshore carbon storage include being further from human population centers and less potential to interact with groundwater. The U.S. Department of Energy's method for evaluating storage capacity in non-oil-bearing saline reservoirs has been enhanced to support assessments for offshore environments in the Offshore CO 2 Saline Storage methodology (OCSS). This methodology applies data-driven capabilities to estimate saline storage capacity while accounting for features specific to offshore reservoirs. Features include changes in CO 2 density and sedimentary differences that impact estimates of permanence and capacity. The Offshore CO 2 Saline Storage Calculator mechanizes OCSS to estimate storage capacity. This paper presents the methodology and estimates for 18 geologic domains in the Gulf of Mexico. Potential storage distributions, sensitivity analyses, and the incorporation of spatial data and tools to support safe site selection are also discussed.

58 GEOSCIENCES↗