SEARCH · Engineering Papers

Results for “DATA STORAGE”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 631 records · Page 35

All sources NASA NTRS DOE OSTI

NASA NTRS1996

The medium is NOT the message or Indefinitely long-term file storage at Leeds University

Approximately 3 years ago we implemented an archive file storage system which embodies experiences gained over more than 25 years of using and writing file storage systems. It is the third in-house system that we have written, and all three systems have been adopted by other institutions. This paper discusses the requirements for long-term data storage in a university environment, and describes how our present system is designed to meet these requirements indefinitely. Particular emphasis is laid on experiences from past systems, and their influence on current system design. We also look at the influence of the IEEE-MSS standard. We currently have the system operating in five UK universities. The system operates in a multi-server environment, and is currently operational with UNIX (SunOS4, Solaris2, SGI-IRIX, HP-UX), NetWare3 and NetWare4. PCs logged on to NetWare can also archive and recover files that live on their hard disks.

Holdsworth, David↗

DOE OSTI2024

A High-level Design for Bidirectional Data Streaming to High-Performance Computing Systems from External Science Facilities

Cutting-edge science is increasingly data-driven due to the emergence of scientific machine learning models that can guide scientists toward fruitful areas of exploration. Experimental science facilities such as light and neutron sources, particle colliders, and radio astronomy telescopes are also producing raw measurement data at rates that exceed available data storage and computing capacity at those facilities. As a result, scientific workflows are being developed that concurrently couple experiments at science facilities with high-performance computing (HPC) facilities to enable analysis of experimental data while the experiment is ongoing, and where analysis results are potentially fed back to the experiment for use in experimental control and/or steering in a time-sensitive manner. Our goal is to design, prototype, and deploy a new capability for the Oak Ridge Leadership Computing Facility (OLCF) that enables such workflows through support for bidirectional, memory-based streaming of data from external experiments into and out of OLCF HPC systems. This high-level design document describes the related work and motivating use cases that inform our understanding of the technical requirements for this capability, and describes a proposed architectural solution that meets these requirements and our plans for demonstrating the capability.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2016

AiiDA: automated interactive infrastructure and database for computational science

Computational science has seen in the last decades a spectacular rise in the scope, breadth, and depth of its efforts. Notwithstanding this prevalence and impact, it is often still performed using the renaissance model of individual artisans gathered in a workshop, under the guidance of an established practitioner. Great benefits could follow instead from adopting concepts and tools coming from computer science to manage, preserve, and share these computational efforts. We illustrate here our paradigm sustaining such vision, based around the four pillars of Automation, Data, Environment, and Sharing. We then discuss its implementation in the open-source AiiDA platform (http://www.aiida.net), that has been tuned first to the demands of computational materials science. AiiDA’s design is based on directed acyclic graphs to track the provenance of data and calculations, and ensure preservation and searchability. Remote computational resources are managed transparently, and automation is coupled with data storage to ensure reproducibility. Last, complex sequences of calculations can be encoded into scientific workflows. We believe that AiiDA’s design and its sharing capabilities will encourage the creation of social ecosystems to disseminate codes, data, and scientific workflows.

Materials Science↗

NASA NTRS1993

The importance of robust error control in data compression applications

Data compression has become an increasingly popular option as advances in information technology have placed further demands on data storage capabilities. With compression ratios as high as 100:1 the benefits are clear; however, the inherent intolerance of many compression formats to error events should be given careful consideration. If we consider that efficiently compressed data will ideally contain no redundancy, then the introduction of a channel error must result in a change of understanding from that of the original source. While the prefix property of codes such as Huffman enables resynchronisation, this is not sufficient to arrest propagating errors in an adaptive environment. Arithmetic, Lempel-Ziv, discrete cosine transform (DCT) and fractal methods are similarly prone to error propagating behaviors. It is, therefore, essential that compression implementations provide sufficient combatant error control in order to maintain data integrity. Ideally, this control should be derived from a full understanding of the prevailing error mechanisms and their interaction with both the system configuration and the compression schemes in use.

Woolley, S. I.↗

NASA NTRS2017

Data Production on Past and Future NASA Missions

Data return is a metric that is commonly publicized for all space science missions. In the early days of the Space Program, this figure was small, and could be described in bits or maybe even megabits. But now, missions are capable of returning data volumes two or three orders of magnitude larger. For example, Voyager 1 and 2 combined produced a little over 5 Terabits of data in 39 years of operation. In contrast, the Cassini mission, launched two decades after Voyager, produced about one and a half times those data volumes in half the time. NISAR, an Earth Science Mission currently in implementation, plans to produce over 28 Petabits of raw data in just 3 years. This means that NISAR will produce about as many data in 30 days as the combined data production of nearly all planetary missions to date. These increases in capability are a result of technology enhancements in two main areas: telecommunications architecture (both space and ground segments) and data storage technology. This paper describes the progression of these two technologies over the course of more than three decades of space missions and provides additional insight into the design of the end-to-end NISAR Data System Architecture. Trends in the data are briefly explored and compared to Moore’s Law which provides only a qualitative model for memory growth but not for data production. In summary, early missions are found to be driven by unrefined processes while later missions, having utilized earlier lessons learned, focus more on improvements to flight and ground capabilities. Data return seems to fall into three categories. First, deep space missions are driven by the large distances that limit data return to the Earth. Next, the orbiter infrastructure around Mars helps these missions generate more data than other deep space spacecraft. Finally, near-Earth missions have the greatest capabilities for the studied metrics due to their close proximity to Earth and the ground network availability.

Xaypraseuth, Peter↗

NASA NTRS1970

Optically activated magnetic recording tape

Optically activated data storage medium visually and electromagnetically reproduces a recorded signal. In an electric field, particles in heat-fluidized thermoplastic layer form a visible image of the recorded signal. Refluidizing the thermoplastic layer erases the signals. Very high data packing densities are achieved.

Marks, A. M.↗

DOE OSTI2015

Online Data Handling and Storage at the CMS Experiment

During the LHC Long Shutdown 1, the CMS Data Acquisition (DAQ) system underwent a partial redesign to replace obsolete network equipment, use more homogeneous switching technologies, and support new detector back-end electronics. The software and hardware infrastructure to provide input, execute the High Level Trigger (HLT) algorithms and deal with output data transport and storage has also been redesigned to be completely file- based. All the metadata needed for bookkeeping are stored in files as well, in the form of small documents using the JSON encoding. The Storage and Transfer System (STS) is responsible for aggregating these files produced by the HLT, storing them temporarily and transferring them to the T0 facility at CERN for subsequent offline processing. The STS merger service aggregates the output files from the HLT from ~62 sources produced with an aggregate rate of ~2GB/s. An estimated bandwidth of 7GB/s in concurrent read/write mode is needed. Furthermore, the STS has to be able to store several days of continuous running, so an estimated of 250TB of total usable disk space is required. In this article we present the various technological and implementation choices of the three components of the STS: the distributed file system, the merger service and the transfer system.

Andre, J. M.↗

NASA NTRS1975

Preliminary data processing system for telemetry data from Interkosmos-8 type satellites

The preliminary data processing system (PDPS), a set of algorithms, programs, equipment, data storage and memory devices, and all devices controlling the processing, is described using the Intercosmos-8 satellite as an example. Each stage of the system is considered along with its content and sequence. The essential elements of the data processing structure are described and the concepts underlying the selection of these elements are discussed.

Vilchinskaya, A. S.↗

NASA NTRS1999

Development of a COTS Mass Storage Unit for the Space Readiness Coherent Lidar Experiment

The technology to develop a Mass Storage Unit (MSU) using commercial-off-the-shelf (COTS) hard drives is an on-going challenge to meet the Space Readiness Coherent Lidar Experiment (SPARCLE) program requirements. A conceptual view of SPARCLE's laser collecting atmospheric data from the shuttle is shown in Figure 1. The determination to develop this technology required several in depth studies before an actual COTS hard drive was selected to continue this effort. Continuing the development of the MSU can, and will, serve future NASA programs that require larger data storage and more on-board processing.

Liggin, Karl↗

DOE OSTI2023

Compression of tokamak boundary plasma simulation data using a maximum volume algorithm for matrix skeleton decomposition

This report demonstrates satisfactory data compression of SOLPS-ITER simulation output ranging from 2D fields, 1D profiles, and 0D scalar variables with a novel matrix decomposition approach. The singular value decomposition (SVD) scales poorly for large matrix sizes and is unsuited to the application on high dimensional data common to fusion plasma physics simulation. In this work, we employ the columns-submatrix-rows (CUR) matrix factorization technique in order to compute a low-rank approximation up to two orders of magnitude faster than the SVD, but within a nominal L2-norm relative error of ε = 10 –2 . In addition, the CUR approach maintains the original format of the data, in its extracted columns and rows, allowing for interpretable data storage at the original resolution of the simulation. We utilize an iterative algorithm to compute the CUR decomposition of simulation output by maximizing the volume, or linearly independent information content, of a low-rank submatrix contained within the data. Experiments over $\textit{n} × \textit{n}$ randomized test matrices with embedded rank-deficient features show that this maximum volume implementation of CUR matrix approximation has reduced asymptotic computational complexity on the order of n compared to the SVD, which scales approximately as $n^3$. These results show that the CUR technique can be used to effectively select time step snapshots (columns) of over 140 SOLPS-ITER output variables and the associated discretized coordinate timeseries (rows) allowing for reconstruction of the complete simulation dynamics.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

DOE OSTI2022

Identifying Hydrometeorological Factors Influencing Reservoir Releases Using Machine Learning Methods

Simulation of reservoir releases plays a critical role in social-economic functioning and our nation's security. How-ever, it is challenging to predict the reservoir release accurately because of many influential factors from natural environments and engineering controls such as the reservoir inflow and storage. Moreover, climate change and hydrological intensification causing the extreme precipitation and temperature make the accurate prediction of reservoir releases even more challenging. Machine learning (ML) methods have shown some successful applications in simulating reservoir releases. However, previous studies mainly used inflow and storage data as inputs and only considered their short-term influences (e.g, previous one or two days). In this work, we use long short-term memory (LSTM) networks for reservoir release prediction based on four input variables including inflow, storage, precipitation, and temperature and consider their long-term influences. We apply the LSTM model to 30 reservoirs in Upper Colorado River Basin, United States. We analyze the prediction performance using six statistical metrics. More importantly, we investigate the influence of the input hydrometeorological factors, as well as their temporal effects on reservoir release decisions. Results indicate that inflow and storage are the most influential factors but the inclusion of precipitation and temperature can further improve the prediction of release especially in low flows. Additionally, the inflow and storage have a relatively long-term effect on the release. These findings can help optimize the water resources management in the reservoirs.

Fan, Ming↗

DOE OSTI2018

Video Compression Routines Final Report CRADA No. TSB-1183-95

The purpose of this CRADA was to establish a framework for LLNL and NOVA Management, Inc. to Cooperatively develop video compression routines to be used to condense maintenance documentation provided on computers issued to Army mechanics to help them diagnose and repair equipment faults. In the past, paper technical manuals were the only means of communicating fault isolation, diagnostics and repair action information to maintenance personnel. The need for this information was at the weapon system itself as well as at the maintenance depot. This project addressed the need for means of efficient storage and retrieval of video (bandwidth issue) by maintenance personnel as they moved through the diagnostic procedures. Several research and development projects have employed interactive stored video as a means of providing maintenance information to the soldier. The problem with this approach was the massive data storage requirement. The overall objective of the project was to develop fast decompression and playback routines for full motion, coIor video. A survey was done of existing standards and potential new routines and techniques for video compression and playback to determine which were suitable for use on the CTS. Playback speeds were preferred to beat 30 frames per second on the target architecture. Compression ratios were expected to store several minutes of synchronized video in 1 MB. Playback routines were Windows 3.1 compatible. Identification of any hardware upgrades associated with playback were made when necessary. A demonstration of existing LLNL algorithms for still frame decompression which represented a current benchmark of data compression performance was completed. NOVA Management and LLNL used some of the compression technologies developed or currently being developed at LLNL to address the requirements of this SBIR. LLNL started by evaluating current compression technologies or combinations thereof to estimate the best compression ratio LLNL could achieve at this stage. LLNL ported some of our software from the SUN workstation environment to the PC or equivalent environment (e.g., MAC) so as to closely match the computer environment suggested by the Army. Finally, LLNL designed and outlined new algorithms to meet the desired video compression of CTS.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS2016

An Efficient, FPGA-Based, Cluster Detection Algorithm Implementation for a Strip Detector Readout System in a Time Projection Chamber Polarimeter

A fundamental challenge in a spaceborne application of a gas-based Time Projection Chamber (TPC) for observation of X-ray polarization is handling the large amount of data collected. The TPC polarimeter described uses the APV-25 Application Specific Integrated Circuit (ASIC) to readout a strip detector. Two dimensional photoelectron track images are created with a time projection technique and used to determine the polarization of the incident X-rays. The detector produces a 128x30 pixel image per photon interaction with each pixel registering 12 bits of collected charge. This creates challenging requirements for data storage and downlink bandwidth with only a modest incidence of photons and can have a significant impact on the overall mission cost. An approach is described for locating and isolating the photoelectron track within the detector image, yielding a much smaller data product, typically between 8x8 pixels and 20x20 pixels. This approach is implemented using a Microsemi RT-ProASIC3-3000 Field-Programmable Gate Array (FPGA), clocked at 20 MHz and utilizing 10.7k logic gates (14% of FPGA), 20 Block RAMs (17% of FPGA), and no external RAM. Results will be presented, demonstrating successful photoelectron track cluster detection with minimal impact to detector dead-time.

Polarimeter↗

DOE OSTI2024

Zero-Power Analog Optical Processing

The motivation behind this research is the growing challenge of handling the massive amounts of data generated by modern imaging systems. Conventional digital image processing techniques are struggling to keep pace with the demands of high-resolution and high-speed imaging systems for remote sensing due to their high-power consumption and data storage requirements. We present a novel approach based on analog photonics to address this challenge. The proposed system utilizes a silicon-photonics-based image encoder positioned after image formation and initial optical-to-electrical conversion. The photonic encoder compresses image data using a passive disordered photonic structure to perform kernel-type random projections of the raw data. The compressed data is then processed by a back-end neural network, which reconstructs the original image with high fidelity (structural similarity exceeding 90%). Our proposed approach has the potential to compress images with ~ 1000X lower power consumption compared to digital approaches with data rates exceeding 1 terapixel/second.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS1994

Lessons learned supporting onboard solid-state recorders

With the advance of semiconductor technology, Solid-State Recorders (SSR) have matured and been accepted as primary onboard data storage devices. Their high reliability, simpler interface and control, and high flexibility have made the SSR's a superb choice in today's spacecraft design. While there are many benefits, the use of SSR's may also add significant complexity to ground data systems. For instance, real-time and playback data may be interleaved into the same data stream, making data sequencing and time ordering difficult. Stored data may be played back out of time order, increasing processing load significantly. Data may also be played back after being sorted by Virtual Channels in the SSR, potentially creating bursts in packet rates that exceed the real-time processing capabilities of the ground systems. This paper presents a summary of lessons learned through the efforts in supporting a number of NASA's missions that employ SSR's. It describes various problems encountered through the design process, and their potential impact on ground system performance, resources, and cost. Recommended approaches to minimizing the impact are demonstrated by examples. The discussion leads to the conclusion that the use of SSR's demands an even higher level of cooperation between spacecraft and ground system designers in order to build the most cost effective end-to-end system.

Shi, Jeff↗

NASA NTRS2021

The Analysis of Using Satellite Soil Moisture Observations for Flood Detection, Evaluating Over the Thailand’s Great Flood of 2011

A flood monitoring and warning system provides critical information that can protect property and save lives. A basin-scale flood monitoring system requires an effective observation platform that offers extensive ground coverage of flood conditions, low latency, and high spatiotemporal resolution. While satellite imagery offers substantial spatial flood extent in detail due to its high spatial resolution, the coarse temporal resolution and cloud obstruction limit its near real-time application. Daily soil moisture data derived from satellite sensors at a scale of a few km can be used to monitor extreme wet surface conditions arising in flood occurrences. This study analyses the flood detection capabilities of several sources of soil moisture information, including the Soil Moisture and Ocean Salinity mission (SMOS), the Advanced Microwave Scanning Radiometer on EOS (AMSR-E), the Advanced SCATterometer (ASCAT) on MetOp, the Global Land Data Assimilation System (GLDAS), and the WaterGAP Global Hydrology Model (WGHM). In addition to soil moisture, the analysis includes measurements of surface reflectance from the Moderate Resolution Imaging Spectroradiometer (MODIS), precipitation measurements from the Tropical Rainfall Measuring Mission (TRMM), and terrestrial water storage estimates from the Gravity Recovery And Climate Experiment (GRACE) as proxies for flood inundations. The analysis was conducted over the Chao Phraya River Basin (CPB) in Thailand, where the Great Flood of 2011 led to one of the most significant economic losses in the country's history. Satellite-derived soil moisture exhibits a stronger correlation with the flood inundations than the precipitation, model-derived soil moisture, and terrestrial water storage data. SMOS soil moisture observation agrees best with the MODIS-derived flood extent/occurrence, both in terms of spatial distribution and timing, and providing approximated flood lead-time of one week or longer. A neural network constructed from SMOS and MODIS data is used to predict flood intensity/occurrence (given soil moisture input) with a predicted time window from eight days to thirty-two days. The short-term prediction (e.g., eight days) achieves the highest accuracy with an averaged recovery rate of approximately 60% (correlation coefficient). This study's results suggest a potential application of satellite soil moisture data in assisting flood monitoring and warning systems.

Satellite soil moisture↗

NASA NTRSDate not supplied

(ODIN): An Open Source, Low-Latency Data Integration & Visualization Framework for the NASA System Wide Safety Project's Disaster Response Safety Demonstration Series

The Open Data Integration Framework (ODIN) is an open source, low latency data integration and visualization framework (https://github.com/NASARace/race-odin) developed under NASA’s System WideSafety Program to demonstrate new safety capabilities designed to improve US airspace operations. Safety demonstrations are a set of increasingly complex (from public safety perspective) disaster response scenarios under which air systems must operate with increased capacity and include: 1) Wildland fire response, 2) Hurricane relief and recovery, 4) Emergency medical delivery via UAS and 4) Urban disaster relief. To accommodate disaster response, ODIN is field deployable and can scale on one or more multi-core, commodity laptops operating with full to limited or intermittent internet connectivity, conditions likely encountered during operations. ODIN runs as webserver with local, persistent data storage to serve either public or a secured, ad hoc network (e.g., an incident command post). The current released ODIN, ODIN-Fire is tailored for wildland fire management incorporating information on satellite overpasses with links to the near real-time data and imagery from the respective agencies. Included are winds data, an important variable for emergency responders and airspace operations, and high-resolution wind forecasts generated by super-computing resources and ingested into ODIN. As an open-source project, ODIN has attracted interest from multiple entities. We will show how 1) a commercial field instrument and data provider uses ODIN to help users visualize, publish and integrate their in-situ sensor network data and 2) ODIN’s capabilities to ingest, integrate and display near-real time satellite data with air traffic and a USFS winds forecast model used in fire response and post-fire assessment. Within NASA ODIN demonstrated novel, near terminal airspace safety capabilities for a project close-out event and previously it monitored the national airspace in real-time to meet an agency milestone. ODIN is presently under development for the anticipated hurricane relief and response demonstration notionally scheduled for the 2025-27 time frame and is available from NASA's github at the above link.

Aeronautics↗

DOE OSTI2015

Atomic memory access hardware implementations

Atomic memory access requests are handled using a variety of systems and methods. According to one example method, a data-processing circuit having an address-request generator that issues requests to a common memory implements a method of processing the requests using a memory-access intervention circuit coupled between the generator and the common memory. The method identifies a current atomic-memory access request from a plurality of memory access requests. A data set is stored that corresponds to the current atomic-memory access request in a data storage circuit within the intervention circuit. It is determined whether the current atomic-memory access request corresponds to at least one previously-stored atomic-memory access request. In response to determining correspondence, the current request is implemented by retrieving data from the common memory. The data is modified in response to the current request and at least one other access request in the memory-access intervention circuit.

97 MATHEMATICS AND COMPUTING↗