Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “DATA STORAGE”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 487 records · Page 27
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

Distributed Tomographic Reconstruction with Quantization

Conventional tomographic reconstruction typically depends on centralized servers for both data storage and computation, leading to concerns about memory limitations and data privacy. Distributed reconstruction algorithms mitigate these issues by partitioning data across multiple nodes, reducing server load and enhancing privacy. However, these algorithms often encounter challenges related to memory constraints and communication overhead between nodes. In this paper, we introduce a decentralized Alternating Directions Method of Multipliers (ADMM) with configurable quantization. By distributing local objectives across nodes, our approach is highly scalable and can efficiently reconstruct images while adapting to available resources. To overcome communication bottlenecks, we propose two quantization techniques based on K-means clustering and JPEG compression. Numerical experiments with benchmark images illustrate the tradeoffs between communication efficiency, memory use, and reconstruction accuracy.

Miao, Runxuan↗
DOE OSTI2019

Multi-Formalism Modeling for Disaster Resilience, Forecasting, and Response

Disaster response preparedness, risk mitigation, and effective post-event response are critical for national security, public health, and economic integrity. For areas impacted by extreme events, timely and accurate impact assessments of lifeline infrastructures are critical. Traditional disaster management and situational awareness platforms are not capable of handling the massive influx of available information from traditional and non-traditional sources. Next-generation analytic assessment approaches, where use of dynamically sourced, heterogeneous data with near real-time availability are integrated into a continually-updated analytics framework. The objective of this research was to develop a foundational multi-formalism modeling platform to integrate novel data sources for use in improving situational awareness after a disaster event. Specifically, to 1) provide an architecture for machine-learning based data fusion various multi-sensor datasets; 2) classify, geolocate, and temporally bin novel data sources for use in spatially-enabled machine-learning based data fusion; and 3) derive confidence-based observation data with machine-learning based prediction to fill spatial and temporal observation gaps. This project developed a reusable cloud-based pipeline architecture for ingesting, storing, and analyzing relevant data, executing a range of deep-learning models, and visualize the information geographically. The pipeline development was informed by previous PNNL investments (StreamKit) and built upon leading-edge technologies consisting of: 1) a raw datastore; 2) a data enrichment pipeline with custom and third-party analytics; 3) enriched data storage; and 4) a custom UI to display the enriched data. Machine learning/deep learning models were selected and tested to classify social media images for relevancy and presence of flood waters. Our custom trained relevance models were run on a validation dataset and the overall accuracy of the relevance classifiers ranged from 65% to 90%. We found that convolutional neural network (CNN) models were both faster and more accurate than more traditional machine learning models. We applied the relevance classifier to remove non-relevant images and then undertook a limited training of CNN models to detect flood on ~1,800 labeled images. The testing accuracy of the flood detection models were ~88%. To improve the geographic location of both social media and oblique aerial images we tested a multi-step workflow built on a pre-trained deep-learning model. To identify similarity features from test and reference scenes, we applied the Xception and Faiss models to a set of test images with inaccurate spatial locations, as well and precisely-located reference scenes from the general area. Matching reference scenes to the test scenes with the closest similarity index had limited success in improving the precision of test scene location and additional methods have been identified for testing. Data fusion between authoritative geographic data (satellite imaging) and non-authoritative crowdsourced data is an underlying theme throughout this research and methods are tested bring these data to a common geographic form through feature matching and transformations. The automated extraction of supporting information for disaster response from oblique aerial images or ground-level social media images is accomplished through semantic image segmentation and transformation. The non-authoritative data are used to validate the authoritative data collections and processing and optionally, t+0 flood models. In addition, the common form geographic data are combined with other static landscape features to develop spatially-explicit training vectors used in deep generative statistical inference models to develop complex non-linear relationships in static and dynamic data and ultimately derive a spatially continuous and probabilistic assessment of flood damage with the event domain.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2021

National Renewable Energy Laboratory, South Table Mountain Campus, Golden, Colorado

The National Renewable Energy Laboratory (NREL) South Table Mountain campus, located in Golden, Colorado, has multiyear datasets from buildings with approximately 1,100,000 sq ft of floor area. These 16 buildings include a large office, nine laboratories, and other education or public assembly facilities such as an education center, a warehouse, quick service restaurants, two site entrance security buildings, and a parking garage—all constructed and improved from 1982 to 2014. These datasets are also available from the NREL data storage and management platform SkySparks. The datasets contain electricity meter data (poser, voltage, and current) with 1-minute resolution and building automation system (BAS) data with resolutions of 5 or 15 minutes. Some subsystems (e.g., lighting) or equipment electricity data have the same intervals as above. Some mechanical equipment state or condition data, such as flow and temperature, are captured with 5- to 15-minute intervals. The types of measurement points (i.e., electrical meter power and zone temperature) and associated data quality across the campus are typical, which helps reduce the unreliability of data quality.

STM, Systems and equipment operational, Energy use↗
NASA NTRS1984

NASA metrology information system: A NEMS subsystem

the NASA Metrology Information Systems (NMIS) is being developed as a standardized tool in managing the NASA field Center's instrument calibration programs. This system, as defined by the NASA Metrology and Calibration Workshop, will function as a subsystem of the newly developed NASA Equipment Management System (NEMS). The Metrology Information System is designed to utilize and update applicable NEMS data fields for controlled property and to function as a stand alone system for noncontrolled property. The NMIS provides automatic instrument calibration recall control, instrument historical performance data storage and analysis, calibration and repair labor and parts cost data, and instrument user and location data. Nineteen standardized reports were developed to analyze calibration system operations.

German, E. S., Jr.↗
NASA NTRS1988

NASA follow-on on the Fiji South Pacific Severe Storm Warning System Project (SPSSD/WS)

The follow-on agreement will implement needed systems enhancements of the satellite ground station installed under the previous SPSSD/WS project. These enhancements include the purchase and installation of an Uninterruptible Power Supply (UPS) and lightning protection unit, hardware modifications to provide system redundancy and increased data storage capacity, software modifications for the new Japanese GMS digital data, upgrades of the image processing software, and both hardware maintenance and tropical cyclone analysis training, and a non-renewable grant to provide emergency field repairs and replacement/spare parts. In March 1988, the UPS and lightning protection unit was installed at the Fiji Meteorological Service by NASA personnel. A tape recorder and demodulator was shipped to Fiji to record the new digital GMS data. Data tapes are not yet available from the Japanese Meteorological Service of the new digital format. This data is required to test the GMS digital software being developed for the Fiji SPSSD/WS facility.

Vermillion, C.↗
NASA NTRS1992

Vector quantization for volume rendering

Volume rendering techniques typically process volumetric data in raw, uncompressed form. As algorithmic and architectural advances improve rendering speeds, however, larger data sets will be evaluated requiring consideration of data storage and transmission issues. In this paper, we analyze the data compression requirements for volume rendering applications and present a solution based on vector quantization. The proposed system compresses volumetric data and then renders images directly from the new data format. Tests on a fluid flow data set demonstrate that good image quality may be achieved at a compression ratio of 17:1 with only a 5 percent cost in additional rendering time.

Ning, Paul↗
NASA NTRS1993

Magnetic field sources and their threat to magnetic media

Magnetic storage media (tapes, disks, cards, etc.) may be damaged by external magnetic fields. The potential for such damage has been researched, but no objective standard exists for the protection of such media. This paper summarizes a magnetic storage facility standard, Publication 933, that ensures magnetic protection of data storage media.

Jewell, Steve↗
DOE OSTI2014

Fleet DNA (Presentation)

The Fleet DNA project objectives include capturing and quantifying drive cycle and technology variation for the multitude of medium- and heavy-duty vocations; providing a common data storage warehouse for medium- and heavy-duty vehicle fleet data across DOE activities and laboratories; and integrating existing DOE tools, models, and analyses to provide data-driven decision making capabilities. Fleet DNA advantages include: for Government - providing in-use data for standard drive cycle development, R&D, tech targets, and rule making; for OEMs - real-world usage datasets provide concrete examples of customer use profiles; for fleets - vocational datasets help illustrate how to maximize return on technology investments; for Funding Agencies - ways are revealed to optimize the impact of financial incentive offers; and for researchers -a data source is provided for modeling and simulation.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
NASA NTRS2000

Ultra-High Density Holographic Memory Module with Solid-State Architecture

NASA's terrestrial. space, and deep-space missions require technology that allows storing. retrieving, and processing a large volume of information. Holographic memory offers high-density data storage with parallel access and high throughput. Several methods exist for data multiplexing based on the fundamental principles of volume hologram selectivity. We recently demonstrated that a spatial (amplitude-phase) encoding of the reference wave (SERW) looks promising as a way to increase the storage density. The SERW hologram offers a method other than traditional methods of selectivity, such as spatial de-correlation between recorded and reconstruction fields, In this report we present the experimental results of the SERW-hologram memory module with solid-state architecture, which is of particular interest for space operations.

Markov, Vladimir B.↗
NASA NTRS2018

How to Cloud for Earth Scientists: An Introduction

This presentation is a tutorial on getting started with cloud computing for the purposes of Earth Observation datasets. We first discuss some of the main advantages that cloud computing can provide for the Earth scientist: copious processing power, immense and affordable data storage, and rapid startup time. We also talk about some of the challenges of getting the most out of cloud computing: re-organizing the way data are analyzed, handling node failures and attending.

cloud storage↗
DOE OSTI2025

Homomorphic data compression for real time photon correlation analysis

The construction of highly coherent X-ray sources, combined with next-generation detectors that are larger and faster, has enabled new research opportunities across the scientific landscape. Among the techniques that benefit most from these advancements is X-ray photon correlation spectroscopy (XPCS), where faster acquisition unlocks the ability to study faster dynamics within samples. However, faster acquisition on larger detectors also introduces unprecedented challenges for online data processing and offline data storage. Such challenges are particularly prominent for XPCS, where real time analyses require simultaneous calculation of all the previously acquired data in the time series. We present a homomorphic compression scheme to effectively reduce the computational time and memory space required for XPCS analysis. Leveraging similarities in the mathematical expression between a matrix-based compression algorithm and the correlation calculation, our approach allows direct operation on the compressed data without their decompression. The offline compression scheme extends storage capacity by a factor of 40 while preserving key features in the lossy compressed data. Meanwhile, the online compression scheme reduces the computational time to below 1 ms, enabling real time calculation of the correlation functions at kHz framerate. Our demonstration of a homomorphic compression of scientific data provides an effective solution to the big data challenge at coherent light sources. Beyond the example shown in this work, the framework can be extended to facilitate real-time operations directly on a compressed data stream for other techniques.

36 MATERIALS SCIENCE↗
NASA NTRS2002

Enabling Earth Science: The Facilities and People of the NCCS

The NCCS's mass data storage system allows scientists to store and manage the vast amounts of data generated by these computations, and its high-speed network connections allow the data to be accessed quickly from the NCCS archives. Some NCCS users perform studies that are directly related to their ability to run computationally expensive and data-intensive simulations. Because the number and type of questions scientists research often are limited by computing power, the NCCS continually pursues the latest technologies in computing, mass storage, and networking technologies. Just as important as the processors, tapes, and routers of the NCCS are the personnel who administer this hardware, create and manage accounts, maintain security, and assist the scientists, often working one on one with them.

Source record↗
DOE OSTI2020

Energy Storage Grand Challenge: Energy Storage Market Report

As part of the Department of Energy's (DOE) Energy Storage Grand Challenge (ESGC), DOE intends to synthesize and disseminate best available energy storage data, information, and analysis to inform decision-making and accelerate technology adoption. The ESGC Roadmap provides options for addressing technology development, commercialization, manufacturing, valuation, and workforce challenges to position the United States for global leadership in the energy storage technologies of the future. This report provides a baseline understanding of the numerous, dynamic energy storage markets that fall within the scope of the ESGC via an integrated presentation of deployment, investment, and manufacturing data from the best, publicly-available sources. This report covers the following energy storage technologies: lithium ion batteries, lead acid batteries, pumped storage hydropower, comrpessed air energy storage, redox flow batteries, hydrogen, building thermal energy storage, and select long duration energy storage technologies. Not all energy storage technologies and markets could be addressed in this report. Due to the wide array of energy technologies, market niches, and data availability issues, this market report only included a select group of technologies. For example, thermal energy storage technologies are very broadly defined and cover a wide range of potential markets, technology readiness levels, and primary energy sources. In other areas, data scarcity necessitates a greater understanding of future applications and emerging science. Future efforts will update data presented in this report and be expanded to include other energy storage technologies. This data-driven assessment of the current status of energy storage markets is essential to track progress toward the goals described in the Energy Storage Grand Challenge and inform the decision-making of a broad range of stakeholders. At the same time, gaps identified through the development of this report can point to areas where further data collection and analysis could provide an even greater level of understanding of the full range of markets and technologies. Finally, numerous complementary analyses are planned, underway, or completed that will provide a deeper understanding of the specific technologies and markets covered at a high-level in this report.

25 ENERGY STORAGE↗
DOE OSTI2021

Tension-free Dirac strings and steered magnetic charges in 3D artificial spin ice

3D nano-architectures presents a new paradigm in modern condensed matter physics with numerous applications in photonics, biomedicine, and spintronics. They are promising for the realization of 3D magnetic nano-networks for ultra-fast and low-energy data storage. Frustration in these systems can lead to magnetic charges or magnetic monopoles, which can function as mobile, binary information carriers. However, Dirac strings in 2D artificial spin ices bind magnetic charges, while 3D dipolar counterparts require cryogenic temperatures for their stability. Here, we present a micromagnetic study of a highly frustrated 3D artificial spin ice harboring tension-free Dirac strings with unbound magnetic charges at room temperature. We use micromagnetic simulations to demonstrate that the mobility threshold for magnetic charges is by 2 eV lower than their unbinding energy. By applying global magnetic fields, we steer magnetic charges in a given direction omitting unintended switchings. The introduced system paves the way toward 3D magnetic networks for data transport and storage.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Ocelot: An Interactive, Efficient Distributed Compression-As-a-Service Platform With Optimized Data Compression Techniques

Large volumes of data generated by scientific simulations, genome sequencing, and other applications need to be moved among clusters for data collection/analysis. Data compression techniques have effectively reduced data storage and transfer costs. However, users' requirements on interactively controlling both data quality and compression ratios are non-trivial to fulfill. Here, we propose a novel Compression-as-a-Service (CaaS) platform called Ocelot with four important contributions: (1) It offers real-time visualization, interactive compression, and transfer of scientific datasets. (2) It incorporates new strategies for compressing diverse types of datasets more effectively than traditional methods. (3) It provides an effective method for estimating the compression ratio and execution time of compression tasks. (4) Experiments on multiple real-world datasets on geographically distributed computers show that Ocelot can significantly improve data transfer efficiency with a performance gain of more than 10x in computing clusters with relatively slow networks.

compression as a service (CaaS)↗
DOE OSTI2022

Large Scale Caching and Streaming of Training Data for Online Deep Learning

The training of deep neural network models on large data remains a difficult problem, despite progress towards scalable techniques. In particular, there is a mismatch between the random but predetermined order in which AI flows select training samples and the streaming I/O patterns for which traditional HPC data storage (e.g., parallel file systems) are designed. In addition, as more data are obtained, it is feasible neither simply to train learning models incrementally, due to catastrophic forgetting (i.e., bias towards new samples), nor to train frequently from scratch, due to prohibitive time and/or resource constraints. In this paper, we study data management techniques that combine caching and streaming with rehearsal support in order to enable efficient access to training samples in both offline training and continual learning. We revisit state-of-art streaming approaches based on data pipelines that transparently handle prefetching, caching, shuffling, and data augmentation, and discuss the challenges and opportunities that arise when combining these methods with data-parallel training techniques. We also report on preliminary experiments that evaluate the I/O overheads involved in accessing the training samples from a parallel file system (PFS) under several concurrency scenarios, highlighting the impact of the PFS on the design of the data pipelines.

data pipelines↗
NASA NTRS1976

A microprocessor controlled pressure scanning system

A microprocessor-based controller and data logger for pressure scanning systems is described. The microcomputer positions and manages data from as many as four 48-port electro-mechanical pressure scanners. The maximum scanning rate is 80 pressure measurements per second (20 ports per second on each of four scanners). The system features on-line calibration, position-directed data storage, and once-per-scan display in engineering units of data from a selected port. The system is designed to be interfaced to a facility computer through a shared memory. System hardware and software are described. Factors affecting measurement error in this type of system are also discussed.

Anderson, R. C.↗