Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “DATA STORAGE”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,333 records · Page 74
All sourcesNASA NTRSDOE OSTI
DOE OSTI2013

POTENTIAL DROUGHT IMPACTS ON ELECTRICITY GENERATION IN TEXAS

Many power plants in the Electric Reliability Council of Texas (ERCOT) region require a large amount of water for system cooling. To improve the understanding of potential risks of electricity generation curtailment due to drought, an assessment of water availability and its potential impacts on generation during drought was performed. For this impact analysis, we identified three drought scenarios based on historical drought records and projected climate data from the Geophysical Fluid Dynamics Laboratory global climate model, for greenhouse gas emission scenario A2 defined by the Intergovernmental Panel on Climate Change . The three drought scenarios are (1) 2011 drought conditions (the worst drought in history), with the current level of water use; (2) a single-year drought (2022) projected for the period of 2020-2030, with the assumed projected water use level for 2030; and (3) a multiple-year drought constructed with climate data for 1950-1957 and water demand projected for 2030. The projected drought scenario in 2022 and the historical droughts in 2011 and 1950-1957 represent two different precipitation patterns in the Texas-Gulf river basin. The hydrologic model constructed for the Texas-Gulf river basin covers most of the ERCOT region. The model incorporates climate and water use data that correspond to three drought scenarios, respectively, to estimate evapotranspiration, water yield from watersheds, stream flow and water storage in reservoirs. Using criteria based on observed (< 50% storage) and predicted (< 55% storage) reservoir data, we identified 15 low-storage reservoirs in 2011, 10 in 2022, and 20 in 1956 (the last year of the multiple-year drought). The power plants that are supported by these reservoirs would be potentially at risk of being derated for thermoelectric cooling because of a lack of water supply. These power plants are located mainly in watersheds near and between Houston and Austin, as well as surrounding Dallas.

Yan, Y. Eugene↗
DOE OSTI2025

dCache: The Storage System of Choice for Data-Intensive Applications

The ever-increasing volumes of data produced by modern scientific facilities like EuXFEL and LHC put significant stress on data management infrastructure operated by laboratories and research centers. The challenges to be addressed span the entire data life cycle, from ingest and efficient data analysis to long-term preservation, typically involving large tape libraries. dCache, a storage system developed in collaboration between the Deutsches Elektronen-Synchrotron (DESY), Fermi National Accelerator Laboratory, and Nordic e-Infrastructure Collaboration (NeIC), is designed to manage a large number of disk servers and to facilitate transparent data migration to and from archival storage. Its multifaceted approach offers a unified method to support a variety of scientific use cases with the same storage infrastructure, including high-throughput data ingest, data sharing over wide area networks, efficient access from HPC clusters, and long-term data preservation on tertiary storage. Initially developed for high energy physics (HEP) experiments, dCache is now used by various scientific communities, including astrophysics, biomedical research, and life sciences, each having specific requirements. This paper presents architecture, deployment strategies, performance and scalability enhancements, and recent advancements in dCache addressing the needs of scientific communities. Finally, we touch on the development and release process, ensuring the software’s high quality.

DCache↗
DOE OSTI2019

A global R&D storage platform implementing distributed Quality of Service [PowerPoint]

The objectives of the program include: Provide facility R&D to match software R&D; Providing a federated system for the optimization of the various storage types (QoS) and reduction of the required manpower to operate storage; and, ''Data lake'': Hiding complexity behind common interfaces and leave it to the distributed lake infrastructure to solve those issues, being: Transpareent change of storage technologies (time, price); and, Data available for low latency access when needed (staging) and safely archived if configured (Long term archive).

43 PARTICLE ACCELERATORS↗
NASA NTRS2016

Using Cloud-Based Storage Technologies for Earth Science Data

Cloud based infrastructure may offer several key benefits of scalability, built in redundancy and reduced total cost of ownership as compared with a traditional data center approach. However, most of the tools and software systems developed for NASA data repositories were not developed with a cloud based infrastructure in mind and do not fully take advantage of commonly available cloud-based technologies. Object storage services are provided through all the leading public (Amazon Web Service, Microsoft Azure, Google Cloud, etc.) and private (Open Stack) clouds, and may provide a more cost-effective means of storing large data collections online. We describe a system that utilizes object storage rather than traditional file system based storage to vend earth science data. The system described is not only cost effective, but shows superior performance for running many different analytics tasks in the cloud. To enable compatibility with existing tools and applications, we outline client libraries that are API compatible with existing libraries for HDF5 and NetCDF4. Performance of the system is demonstrated using clouds services running on Amazon Web Services.

Data↗
NASA NTRS1969

IBM-1620 monitor 2-D disk-storage subroutines

Set of subroutines provides the FORTRAN user with protected, permanent, disk storage of data on an IBM 1620 Monitor 11-D system. The program consists of a set of four subroutines and a utility program. It allows block data to be transferred directly between assigned core locations and disk storage.

Krejci, H. F.↗
NASA NTRS1982

The Oklahoma Geographic Information Retrieval System

The Oklahoma Geographic Information Retrieval System (OGIRS) is a highly interactive data entry, storage, manipulation, and display software system for use with geographically referenced data. Although originally developed for a project concerned with coal strip mine reclamation, OGIRS is capable of handling any geographically referenced data for a variety of natural resource management applications. A special effort has been made to integrate remotely sensed data into the information system. The timeliness and synoptic coverage of satellite data are particularly useful attributes for inclusion into the geographic information system.

Blanchard, W. A.↗
NASA NTRS2001

The Role of Advanced Information System Technology in Remote Sensing for NASA's Earth Science Enterprise in the 21st Century

Future NASA Earth observing satellites will carry high-precision instruments capable of producing large amounts of scientific data. The strategy will be to network these instrument-laden satellites into a web-like array of sensors to facilitate the collection, processing, transmission, storage, and distribution of data and data products - the essential elements of what we refer to as "Information Technology." Many of these Information Technologies will enable the satellite and ground information systems to function effectively in real-time, providing scientists with the capability of customizing data collection activities on a satellite or group of satellites directly from the ground. In future systems, extremely large quantities of data collected by scientific instruments will require the fastest processors, the highest communication channel transfer rates, and the largest data storage capacity to insure that data flows smoothly from the satellite-based instrument to the ground-based archive. Autonomous systems will control all essential processes and play a key role in coordinating the data flow through space-based communication networks. In this paper, we will discuss those critical information technologies for Earth observing satellites that will support the next generation of space-based scientific measurements of planet Earth, and insure that data and data products provided by these systems will be accessible to scientists and the user community in general.

Prescott, Glenn↗
NASA NTRS1996

The Global File System

The global file system (GFS) is a prototype design for a distributed file system in which cluster nodes physically share storage devices connected via a network-like fiber channel. Networks and network-attached storage devices have advanced to a level of performance and extensibility so that the previous disadvantages of shared disk architectures are no longer valid. This shared storage architecture attempts to exploit the sophistication of storage device technologies whereas a server architecture diminishes a device's role to that of a simple component. GFS distributes the file system responsibilities across processing nodes, storage across the devices, and file system resources across the entire storage pool. GFS caches data on the storage devices instead of the main memories of the machines. Consistency is established by using a locking mechanism maintained by the storage devices to facilitate atomic read-modify-write operations. The locking mechanism is being prototyped in the Silicon Graphics IRIX operating system and is accessed using standard Unix commands and modules.

Soltis, Steven R.↗
DOE OSTI2015

BRAINformat: A Data Standardization Framework for Neuroscience Data

Neuroscience is entering the era of `extreme data` with little experience and few plans for the associated volume, velocity, variety, and veracity challenges. This is a serious impediment for both the sharing of data across labs, as well as the utilization of modern and high-performance computing capabilities to enable data driven discovery. Here, we introduce BRAINformat, a novel file format and model for management and storage of neuroscience data. The BRAINformat library defines application-independent design concepts and modules that together create a general framework for standardization of scientific data. We describe the formal specification of scientific data standards, which facilitates sharing and verification of data and formats. We introduce the concept of Managed Objects, enabling semantic components of data formats to be specified as self-contained units, supporting modular and reusable design of data format components and file storage. The BRAINformat is built off of HDF5, enabling portable, scalable, and self-describing data storage. We introduce the novel concept of Relationship Attributes for modeling and use of semantic relationships between data objects, and discuss the annotation of data using dedicated data annotation modules provided by the BRAINformat library. Based on these concepts we implement dedicated, application-oriented modules and design a data standard for neuroscience data.The BRAINformat software library is open source, easy-to-use, and provides detailed user and developer documentation and is freely available at: https://bitbucket.org/oruebel/brainformat.

59 BASIC BIOLOGICAL SCIENCES↗
NASA NTRS1989

Range Filtering For Navigation By Satellite

Less processing and storage of data needed. Mathematical basis developed for Kalman filtering of sequential measurements of range from single-channel receiver in Global Positioning System (GPS) to each of several navigation satellites. Range filtering involves separate applications of kinematic Kalman tracking filter to measurements of signal from each satellite in use. State vector consists of pseudorange and time derivatives. Main advantages: simplification of tuning and decreases in required amounts of storage and processing of data for navigation in ships, airplanes, and ground vehicles.

Paielli, Russell↗
NASA NTRS1998

Second International Workshop on Software Engineering and Code Design in Parallel Meteorological and Oceanographic Applications

This report contains the abstracts and technical papers from the Second International Workshop on Software Engineering and Code Design in Parallel Meteorological and Oceanographic Applications, held June 15-18, 1998, in Scottsdale, Arizona. The purpose of the workshop is to bring together software developers in meteorology and oceanography to discuss software engineering and code design issues for parallel architectures, including Massively Parallel Processors (MPP's), Parallel Vector Processors (PVP's), Symmetric Multi-Processors (SMP's), Distributed Shared Memory (DSM) multi-processors, and clusters. Issues to be discussed include: (1) code architectures for current parallel models, including basic data structures, storage allocation, variable naming conventions, coding rules and styles, i/o and pre/post-processing of data; (2) designing modular code; (3) load balancing and domain decomposition; (4) techniques that exploit parallelism efficiently yet hide the machine-related details from the programmer; (5) tools for making the programmer more productive; and (6) the proliferation of programming models (F--, OpenMP, MPI, and HPF).

OKeefe, Matthew↗
NASA NTRS1997

Digital storage and analysis of color Doppler echocardiograms

Color Doppler flow mapping has played an important role in clinical echocardiography. Most of the clinical work, however, has been primarily qualitative. Although qualitative information is very valuable, there is considerable quantitative information stored within the velocity map that has not been extensively exploited so far. Recently, many researchers have shown interest in using the encoded velocities to address the clinical problems such as quantification of valvular regurgitation, calculation of cardiac output, and characterization of ventricular filling. In this article, we review some basic physics and engineering aspects of color Doppler echocardiography, as well as drawbacks of trying to retrieve velocities from video tape data. Digital storage, which plays a critical role in performing quantitative analysis, is discussed in some detail with special attention to velocity encoding in DICOM 3.0 (medical image storage standard) and the use of digital compression. Lossy compression can considerably reduce file size with minimal loss of information (mostly redundant); this is critical for digital storage because of the enormous amount of data generated (a 10 minute study could require 18 Gigabytes of storage capacity). Lossy JPEG compression and its impact on quantitative analysis has been studied, showing that images compressed at 27:1 using the JPEG algorithm compares favorably with directly digitized video images, the current goldstandard. Some potential applications of these velocities in analyzing the proximal convergence zones, mitral inflow, and some areas of future development are also discussed in the article.

NASA Discipline Cardiopulmonary↗
DOE OSTI2014

Big Data over a 100G network at Fermilab

As the need for Big Data in science becomes ever more relevant, networks around the world are upgrading their infrastructure to support high-speed interconnections. To support its mission, the high-energy physics community as a pioneer in Big Data has always been relying on the Fermi National Accelerator Laboratory to be at the forefront of storage and data movement. This need was reiterated in recent years with the data-taking rate of the major LHC experiments reaching tens of petabytes per year. At Fermilab, this resulted regularly in peaks of data movement on the Wide area network (WAN) in and out of the laboratory of about 30 Gbit/s and on the Local are network (LAN) between storage and computational farms of 160 Gbit/s. To address these ever increasing needs, as of this year Fermilab is connected to the Energy Sciences Network (ESnet) through a 100 Gb/s link. To understand the optimal system-and application-level configuration to interface computational systems with the new highspeed interconnect, Fermilab has deployed a Network Research & Development facility connected to the ESnet 100G Testbed. For the past two years, the High Throughput Data Program (HTDP) has been using the Testbed to identify gaps in data movement middleware [5] when transferring data at these high-speeds. The program has published evaluations of technologies typically used in High Energy Physics, such as GridFTP [4], XrootD [9], and Squid [8]. Furthermore, this work presents the new R&D facility and the continuation of the evaluation program.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2000

A Spacebased Ocean Surface Exchange Data Analysis System

Emerging technologies have provided unprecedented opportunities to transform information into knowledge and disseminate them in a much faster, cheaper, and userfriendly mode. We have set up a system to produce and disseminate high level (gridded) ocean surface wind data from the NASA Scatterometer and European Remote Sensing missions. The data system is being expanded to produce real-time gridded ocean surface winds from an improved sensor SeaWinds on the Quikscat Mission. The wind field will be combined with hydrologic parameters from the Tropical Rain Measuring Mission to monitor evolving weather systems and natural hazard in real time. It will form the basis for spacebased Ocean Surface Exchange Data Analysis System (SOSEDAS) which will include the production of ocean surface momentum, heat, and water fluxes needed for interdisciplinary studies of ocean-atmosphere interaction. Various commercial or non-commercial software tools have been compared and selected in terms of their ability in database management, remote data accessing, graphical interface, data quality, storage needs and transfer speed, etc. Issues regarding system security and user authentication, distributed data archiving and accessing, strategy to compress large-volume geophysical and satellite data/image. and increasing transferring speed are being addressed. A simple and easy way to access information and derive knowledge from spacebased data of multiple missions is being provided. The evolving 'knowledge system' will provide relevant infrastructure to address Earth System Science, make inroads in educating an informed populace, and illuminate decision and policy making.

Tang, Wenqing↗
DOE OSTI2021

Real-Time Lossless Compression for Ultra-High-Density Synchrophasor and Point on Wave Data

Modern advanced Phasor Measurement Units (PMUs) are developed with ultra-high reporting rates to meet the demand for monitoring the power systems dynamics in detail. Due to the large volume of data, the communication and storage systems are seriously challenged with the presence of Ultra-High-Density (UHD) synchrophasor and Point on Wave (POW) data. Therefore, it is an urgent task to compress the UHD data for more efficient communication and data storage. This paper proposes several methods to compress the synchrophasor and POW data in a lossless manner. First, an Improved-Time-Series-Special Compression (ITSSC) method is proposed to compress the UHD frequency data. Second, a Delta-difference Huffman method is combined with the TSSC algorithm to compress the UHD phase angle data. Finally, a cyclical high-order delta modulation method is proposed to compress the UHD POW data. The proposed models are extensively tested and compared with different existing lossless compression algorithms using the field-collected synchrophasor and POW data at different reporting rates. The results indicate that the proposed algorithms are efficient in performing lossless compression for the UHD synchrophasor and POW data in real time.

42 ENGINEERING↗
DOE OSTI2013

OpenMSI: A High-Performance Web-Based Platform for Mass Spectrometry Imaging

Mass spectrometry imaging (MSI) enables researchers to directly probe endogenous molecules directly within the architecture of the biological matrix. Unfortunately, efficient access, management, and analysis of the data generated by MSI approaches remain major challenges to this rapidly developing field. Despite the availability of numerous dedicated file formats and software packages, it is a widely held viewpoint that the biggest challenge is simply opening, sharing, and analyzing a file without loss of information. Here we present OpenMSI, a software framework and platform that addresses these challenges via an advanced, high-performance, extensible file format and Web API for remote data access (http://openmsi.nersc.gov). The OpenMSI file format supports storage of raw MSI data, metadata, and derived analyses in a single, self-describing format based on HDF5 and is supported by a large range of analysis software (e.g., Matlab and R) and programming languages (e.g., C++, Fortran, and Python). Careful optimization of the storage layout of MSI data sets using chunking, compression, and data replication accelerates common, selective data access operations while minimizing data storage requirements and are critical enablers of rapid data I/O. The OpenMSI file format has shown to provide >2000-fold improvement for image access operations, enabling spectrum and image retrieval in less than 0.3 s across the Internet even for 50 GB MSI data sets. To make remote high-performance compute resources accessible for analysis and to facilitate data sharing and collaboration, we describe an easy-to-use yet powerful Web API, enabling fast and convenient access to MSI data, metadata, and derived analysis results stored remotely to facilitate high-performance data analysis and enable implementation of Web based data sharing, visualization, and analysis.

59 BASIC BIOLOGICAL SCIENCES↗