Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data storage”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 649 records · Page 36
All sourcesNASA NTRSDOE OSTI
NASA NTRS1993

Data processing and calibration for tunable diode laser harmonic absorption spectrometers

Data processing and calibration methods are described for tunable diode laser absorption spectrometers which produce harmonic absorption spectra as raw data for measuring gas mixing ratios down to parts-per-trillion levels at a variety of pressures. The methods, which take advantage of modern computer speed, memory, and data storage capabilities, are applicable to the detection of weakly absorbing gases in quantitative industrial monitoring, in addition to aircraft and balloon atmospheric measurements for which they were designed. Algorithms for calibration and data analysis, including rejection of erroneous spectra, variation of effective integration time, spectral alignment prior to averaging, and plotting and archiving of results, have been tested on actual stratospheric laser spectra recorded by the Aircraft Laser Infrared Absorption Spectrometer (ALIAS) spectrometer in numerous flights of NASA's ER-2 aircraft.

May, Randy D.↗
NASA NTRS1996

The medium is NOT the message or Indefinitely long-term file storage at Leeds University

Approximately 3 years ago we implemented an archive file storage system which embodies experiences gained over more than 25 years of using and writing file storage systems. It is the third in-house system that we have written, and all three systems have been adopted by other institutions. This paper discusses the requirements for long-term data storage in a university environment, and describes how our present system is designed to meet these requirements indefinitely. Particular emphasis is laid on experiences from past systems, and their influence on current system design. We also look at the influence of the IEEE-MSS standard. We currently have the system operating in five UK universities. The system operates in a multi-server environment, and is currently operational with UNIX (SunOS4, Solaris2, SGI-IRIX, HP-UX), NetWare3 and NetWare4. PCs logged on to NetWare can also archive and recover files that live on their hard disks.

Holdsworth, David↗
DOE OSTI2024

A High-level Design for Bidirectional Data Streaming to High-Performance Computing Systems from External Science Facilities

Cutting-edge science is increasingly data-driven due to the emergence of scientific machine learning models that can guide scientists toward fruitful areas of exploration. Experimental science facilities such as light and neutron sources, particle colliders, and radio astronomy telescopes are also producing raw measurement data at rates that exceed available data storage and computing capacity at those facilities. As a result, scientific workflows are being developed that concurrently couple experiments at science facilities with high-performance computing (HPC) facilities to enable analysis of experimental data while the experiment is ongoing, and where analysis results are potentially fed back to the experiment for use in experimental control and/or steering in a time-sensitive manner. Our goal is to design, prototype, and deploy a new capability for the Oak Ridge Leadership Computing Facility (OLCF) that enables such workflows through support for bidirectional, memory-based streaming of data from external experiments into and out of OLCF HPC systems. This high-level design document describes the related work and motivating use cases that inform our understanding of the technical requirements for this capability, and describes a proposed architectural solution that meets these requirements and our plans for demonstrating the capability.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2016

AiiDA: automated interactive infrastructure and database for computational science

Computational science has seen in the last decades a spectacular rise in the scope, breadth, and depth of its efforts. Notwithstanding this prevalence and impact, it is often still performed using the renaissance model of individual artisans gathered in a workshop, under the guidance of an established practitioner. Great benefits could follow instead from adopting concepts and tools coming from computer science to manage, preserve, and share these computational efforts. We illustrate here our paradigm sustaining such vision, based around the four pillars of Automation, Data, Environment, and Sharing. We then discuss its implementation in the open-source AiiDA platform (http://www.aiida.net), that has been tuned first to the demands of computational materials science. AiiDA’s design is based on directed acyclic graphs to track the provenance of data and calculations, and ensure preservation and searchability. Remote computational resources are managed transparently, and automation is coupled with data storage to ensure reproducibility. Last, complex sequences of calculations can be encoded into scientific workflows. We believe that AiiDA’s design and its sharing capabilities will encourage the creation of social ecosystems to disseminate codes, data, and scientific workflows.

Materials Science↗
NASA NTRS1993

The importance of robust error control in data compression applications

Data compression has become an increasingly popular option as advances in information technology have placed further demands on data storage capabilities. With compression ratios as high as 100:1 the benefits are clear; however, the inherent intolerance of many compression formats to error events should be given careful consideration. If we consider that efficiently compressed data will ideally contain no redundancy, then the introduction of a channel error must result in a change of understanding from that of the original source. While the prefix property of codes such as Huffman enables resynchronisation, this is not sufficient to arrest propagating errors in an adaptive environment. Arithmetic, Lempel-Ziv, discrete cosine transform (DCT) and fractal methods are similarly prone to error propagating behaviors. It is, therefore, essential that compression implementations provide sufficient combatant error control in order to maintain data integrity. Ideally, this control should be derived from a full understanding of the prevailing error mechanisms and their interaction with both the system configuration and the compression schemes in use.

Woolley, S. I.↗
NASA NTRS2017

Data Production on Past and Future NASA Missions

Data return is a metric that is commonly publicized for all space science missions. In the early days of the Space Program, this figure was small, and could be described in bits or maybe even megabits. But now, missions are capable of returning data volumes two or three orders of magnitude larger. For example, Voyager 1 and 2 combined produced a little over 5 Terabits of data in 39 years of operation. In contrast, the Cassini mission, launched two decades after Voyager, produced about one and a half times those data volumes in half the time. NISAR, an Earth Science Mission currently in implementation, plans to produce over 28 Petabits of raw data in just 3 years. This means that NISAR will produce about as many data in 30 days as the combined data production of nearly all planetary missions to date. These increases in capability are a result of technology enhancements in two main areas: telecommunications architecture (both space and ground segments) and data storage technology. This paper describes the progression of these two technologies over the course of more than three decades of space missions and provides additional insight into the design of the end-to-end NISAR Data System Architecture. Trends in the data are briefly explored and compared to Moore’s Law which provides only a qualitative model for memory growth but not for data production. In summary, early missions are found to be driven by unrefined processes while later missions, having utilized earlier lessons learned, focus more on improvements to flight and ground capabilities. Data return seems to fall into three categories. First, deep space missions are driven by the large distances that limit data return to the Earth. Next, the orbiter infrastructure around Mars helps these missions generate more data than other deep space spacecraft. Finally, near-Earth missions have the greatest capabilities for the studied metrics due to their close proximity to Earth and the ground network availability.

Xaypraseuth, Peter↗
DOE OSTI2012

Adapting federated cyberinfrastructure for shared data collection facilities in structural biology

Early stage experimental data in structural biology is generally unmaintained and inaccessible to the public. It is increasingly believed that this data, which forms the basis for each macromolecular structure discovered by this field, must be archived and, in due course, published. Furthermore, the widespread use of shared scientific facilities such as synchrotron beamlines complicates the issue of data storage, access and movement, as does the increase of remote users. This work describes a prototype system that adapts existing federated cyberinfrastructure technology and techniques to significantly improve the operational environment for users and administrators of synchrotron data collection facilities used in structural biology. This is achieved through software from the Virtual Data Toolkit and Globus, bringing together federated users and facilities from the Stanford Synchrotron Radiation Lightsource, the Advanced Photon Source, the Open Science Grid, the SBGrid Consortium and Harvard Medical School. The performance and experience with the prototype provide a model for data management at shared scientific facilities.

59 BASIC BIOLOGICAL SCIENCES↗
NASA NTRS1970

Optically activated magnetic recording tape

Optically activated data storage medium visually and electromagnetically reproduces a recorded signal. In an electric field, particles in heat-fluidized thermoplastic layer form a visible image of the recorded signal. Refluidizing the thermoplastic layer erases the signals. Very high data packing densities are achieved.

Marks, A. M.↗
DOE OSTI2015

Online Data Handling and Storage at the CMS Experiment

During the LHC Long Shutdown 1, the CMS Data Acquisition (DAQ) system underwent a partial redesign to replace obsolete network equipment, use more homogeneous switching technologies, and support new detector back-end electronics. The software and hardware infrastructure to provide input, execute the High Level Trigger (HLT) algorithms and deal with output data transport and storage has also been redesigned to be completely file- based. All the metadata needed for bookkeeping are stored in files as well, in the form of small documents using the JSON encoding. The Storage and Transfer System (STS) is responsible for aggregating these files produced by the HLT, storing them temporarily and transferring them to the T0 facility at CERN for subsequent offline processing. The STS merger service aggregates the output files from the HLT from ~62 sources produced with an aggregate rate of ~2GB/s. An estimated bandwidth of 7GB/s in concurrent read/write mode is needed. Furthermore, the STS has to be able to store several days of continuous running, so an estimated of 250TB of total usable disk space is required. In this article we present the various technological and implementation choices of the three components of the STS: the distributed file system, the merger service and the transfer system.

Andre, J. M.↗
NASA NTRS1975

Preliminary data processing system for telemetry data from Interkosmos-8 type satellites

The preliminary data processing system (PDPS), a set of algorithms, programs, equipment, data storage and memory devices, and all devices controlling the processing, is described using the Intercosmos-8 satellite as an example. Each stage of the system is considered along with its content and sequence. The essential elements of the data processing structure are described and the concepts underlying the selection of these elements are discussed.

Vilchinskaya, A. S.↗
NASA NTRS1999

Development of a COTS Mass Storage Unit for the Space Readiness Coherent Lidar Experiment

The technology to develop a Mass Storage Unit (MSU) using commercial-off-the-shelf (COTS) hard drives is an on-going challenge to meet the Space Readiness Coherent Lidar Experiment (SPARCLE) program requirements. A conceptual view of SPARCLE's laser collecting atmospheric data from the shuttle is shown in Figure 1. The determination to develop this technology required several in depth studies before an actual COTS hard drive was selected to continue this effort. Continuing the development of the MSU can, and will, serve future NASA programs that require larger data storage and more on-board processing.

Liggin, Karl↗
DOE OSTI2023

Compression of tokamak boundary plasma simulation data using a maximum volume algorithm for matrix skeleton decomposition

This report demonstrates satisfactory data compression of SOLPS-ITER simulation output ranging from 2D fields, 1D profiles, and 0D scalar variables with a novel matrix decomposition approach. The singular value decomposition (SVD) scales poorly for large matrix sizes and is unsuited to the application on high dimensional data common to fusion plasma physics simulation. In this work, we employ the columns-submatrix-rows (CUR) matrix factorization technique in order to compute a low-rank approximation up to two orders of magnitude faster than the SVD, but within a nominal L2-norm relative error of ε = 10 –2 . In addition, the CUR approach maintains the original format of the data, in its extracted columns and rows, allowing for interpretable data storage at the original resolution of the simulation. We utilize an iterative algorithm to compute the CUR decomposition of simulation output by maximizing the volume, or linearly independent information content, of a low-rank submatrix contained within the data. Experiments over $\textit{n} × \textit{n}$ randomized test matrices with embedded rank-deficient features show that this maximum volume implementation of CUR matrix approximation has reduced asymptotic computational complexity on the order of n compared to the SVD, which scales approximately as $n^3$. These results show that the CUR technique can be used to effectively select time step snapshots (columns) of over 140 SOLPS-ITER output variables and the associated discretized coordinate timeseries (rows) allowing for reconstruction of the complete simulation dynamics.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2022

Identifying Hydrometeorological Factors Influencing Reservoir Releases Using Machine Learning Methods

Simulation of reservoir releases plays a critical role in social-economic functioning and our nation's security. How-ever, it is challenging to predict the reservoir release accurately because of many influential factors from natural environments and engineering controls such as the reservoir inflow and storage. Moreover, climate change and hydrological intensification causing the extreme precipitation and temperature make the accurate prediction of reservoir releases even more challenging. Machine learning (ML) methods have shown some successful applications in simulating reservoir releases. However, previous studies mainly used inflow and storage data as inputs and only considered their short-term influences (e.g, previous one or two days). In this work, we use long short-term memory (LSTM) networks for reservoir release prediction based on four input variables including inflow, storage, precipitation, and temperature and consider their long-term influences. We apply the LSTM model to 30 reservoirs in Upper Colorado River Basin, United States. We analyze the prediction performance using six statistical metrics. More importantly, we investigate the influence of the input hydrometeorological factors, as well as their temporal effects on reservoir release decisions. Results indicate that inflow and storage are the most influential factors but the inclusion of precipitation and temperature can further improve the prediction of release especially in low flows. Additionally, the inflow and storage have a relatively long-term effect on the release. These findings can help optimize the water resources management in the reservoirs.

Fan, Ming↗
DOE OSTI2018

Video Compression Routines Final Report CRADA No. TSB-1183-95

The purpose of this CRADA was to establish a framework for LLNL and NOVA Management, Inc. to Cooperatively develop video compression routines to be used to condense maintenance documentation provided on computers issued to Army mechanics to help them diagnose and repair equipment faults. In the past, paper technical manuals were the only means of communicating fault isolation, diagnostics and repair action information to maintenance personnel. The need for this information was at the weapon system itself as well as at the maintenance depot. This project addressed the need for means of efficient storage and retrieval of video (bandwidth issue) by maintenance personnel as they moved through the diagnostic procedures. Several research and development projects have employed interactive stored video as a means of providing maintenance information to the soldier. The problem with this approach was the massive data storage requirement. The overall objective of the project was to develop fast decompression and playback routines for full motion, coIor video. A survey was done of existing standards and potential new routines and techniques for video compression and playback to determine which were suitable for use on the CTS. Playback speeds were preferred to beat 30 frames per second on the target architecture. Compression ratios were expected to store several minutes of synchronized video in 1 MB. Playback routines were Windows 3.1 compatible. Identification of any hardware upgrades associated with playback were made when necessary. A demonstration of existing LLNL algorithms for still frame decompression which represented a current benchmark of data compression performance was completed. NOVA Management and LLNL used some of the compression technologies developed or currently being developed at LLNL to address the requirements of this SBIR. LLNL started by evaluating current compression technologies or combinations thereof to estimate the best compression ratio LLNL could achieve at this stage. LLNL ported some of our software from the SUN workstation environment to the PC or equivalent environment (e.g., MAC) so as to closely match the computer environment suggested by the Army. Finally, LLNL designed and outlined new algorithms to meet the desired video compression of CTS.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2011

Real-time online monitoring of radwaste storage: A proof-of-principle test prototype

A prototype array of modular sensors for online monitoring of radioactive waste was developed at INFN-LNS. We implemented a new kind of gamma mini-sensor, based on Silicon Photomultipliers and scintillating fibers, that behaves like a cheap scintillating Geiger-Muller counter. It can be placed in shape of a fine grid around each single waste drum in a repository. After simulating the counting response of thin scintillating fibers to gamma radiation, we verified our simulation results with measurements using radioactive sources. Front-end electronics and an FPGA-based counting system were developed to handle the field data, also implementing data transmission, a graphical user interface and a data storage system. A test of four sensors in a real radwaste storage site was performed with promising results. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
NASA NTRS2016

An Efficient, FPGA-Based, Cluster Detection Algorithm Implementation for a Strip Detector Readout System in a Time Projection Chamber Polarimeter

A fundamental challenge in a spaceborne application of a gas-based Time Projection Chamber (TPC) for observation of X-ray polarization is handling the large amount of data collected. The TPC polarimeter described uses the APV-25 Application Specific Integrated Circuit (ASIC) to readout a strip detector. Two dimensional photoelectron track images are created with a time projection technique and used to determine the polarization of the incident X-rays. The detector produces a 128x30 pixel image per photon interaction with each pixel registering 12 bits of collected charge. This creates challenging requirements for data storage and downlink bandwidth with only a modest incidence of photons and can have a significant impact on the overall mission cost. An approach is described for locating and isolating the photoelectron track within the detector image, yielding a much smaller data product, typically between 8x8 pixels and 20x20 pixels. This approach is implemented using a Microsemi RT-ProASIC3-3000 Field-Programmable Gate Array (FPGA), clocked at 20 MHz and utilizing 10.7k logic gates (14% of FPGA), 20 Block RAMs (17% of FPGA), and no external RAM. Results will be presented, demonstrating successful photoelectron track cluster detection with minimal impact to detector dead-time.

Polarimeter↗
DOE OSTI2011

Using Dashboards to Improve Energy and Comfort in Federal Buildings

Federal agencies are taking many steps to improve the sustainability of their operations, including improving the energy efficiency of their buildings, promoting recycling and reuse of materials, encouraging carpooling and alternative transit schemes, and installing low flow water fixture units are just a few of the common examples. However, an often overlooked means of energy savings is to provide feedback to building users about their energy use through information dashboards connected to a building?s energy information system. An Energy Information System (EIS), broadly defined, is a package of performance monitoring software, data acquisition hardware, and communication systems that is used to collect, store, analyze, and display energy information. At a minimum, the EIS provides the whole-building energy-use information (Granderson 2009a). We define a ?dashboard? as a display and visualization tool that utilizes the EIS data and technology to provide critical information to users. This information can lead to actions resulting in energy savings, comfort improvements, efficient operations, and more. The tools to report analyzed information have existed in the information technology as business intelligence (Few 2006). The dashboard is distinguished from the EIS as a whole, which includes additional hardware and software components to collect and storage data, and analysis for resources and energy management (Granderson 2009b). EIS can be used for a variety of uses, including benchmarking, base-lining, anomaly detection, off-hours energy use evaluation, load shape optimization, energy rate analysis, retrofit and retro-commissioning savings (Granderson 2009a). The use of these EIS features depends on the specific users. For example, federal and other building managers may use anomaly detection to identify energy waste in a specific building, or to benchmark energy use in similar buildings to identify energy saving potential and reduce operational cost. There are several vendors of EIS technology that provide information on energy and other environmental variables in buildings.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗