Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data storage data management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 163 records · Page 9

Conceptual Study of Intelligent Data Archives of the Future

A conceptual architecture study is under way to address the problem of getting the most scientific value from the large volumes of Earth and space science data that NASA expects to accumulate in the future. This involves efficient storage and access, but goes beyond that to facilitate intelligent data understanding and utilization through modeling realistic virtual entities with predictive capabilities. The objective of the study is to formulate ideas and concepts and to provide recommendations that lead to prototyping and implementation in the period from 2010 to 2020. The approach consists of the definition of future scenarios and needs for data usage in applications (in consultation with scientific and applications users), projection of advances in technologies, and an abstraction of an intelligent archive architecture. Strategic evolution is considered in various areas such as storage, data, information and knowledge management, data ingest and mining, user interfaces, and advances in intelligent data understanding algorithms.

Ramapriyan, H. K.↗

Space Telecommunications Radio System (STRS) Application Repository Design and Analysis

The Space Telecommunications Radio System (STRS) Application Repository Design and Analysis document describes the STRS application repository for software-defined radio (SDR) applications intended to be compliant to the STRS Architecture Standard. The document provides information about the submission of artifacts to the STRS application repository, to provide information to the potential users of that information, and for the systems engineer to understand the requirements, concepts, and approach to the STRS application repository. The STRS application repository is intended to capture knowledge, documents, and other artifacts for each waveform application or other application outside of its project so that when the project ends, the knowledge is retained. The document describes the transmission of technology from mission to mission capturing lessons learned that are used for continuous improvement across projects and supporting NASA Procedural Requirements (NPRs) for performing software engineering projects and NASAs release process.

information retrieval↗

Big Data Challenges for Large Radio Arrays

Future large radio astronomy arrays, particularly the Square Kilometre Array (SKA), will be able to generate data at rates far higher than can be analyzed or stored affordably with current practices. This is, by definition, a "big data" problem, and requires an end-to-end solution if future radio arrays are to reach their full scientific potential. Similar data processing, transport, storage, and management challenges face next-generation facilities in many other fields.

Combining↗

Structuring and storing signals with their metadata: practical considerations

This chapter aims to provide a comprehensive overview on structuring signal data and their metadata, highlighting key considerations for optimal management and storage. Specifically, it focuses on: (a) the relevance of data organization; (b) what data to store and what to keep; and (c) data management methods. Thus, this chapter answers where and how to store metadata efficiently. Chapter 3 explains what is considered metadata. Chapters 5 and 6 present how to collect certain metadata through dedicated sensor validation tests (Chapter 5) or algorithmic analysis (Chapter 6).

Nicolaï, Niels↗

The challenge of a data storage hierarchy

A discussion of Mesa Archival Systems' data archiving system is presented. This data archiving system is strictly a software system that is implemented on a mainframe and manages the data into permanent file storage. Emphasis is placed on the fact that any kind of client system on the network can be connected through the Unix interface of the data archiving system.

Ruderman, Michael↗

Fifth NASA Goddard Conference on Mass Storage Systems and Technologies

This document contains copies of those technical papers received in time for publication prior to the Fifth Goddard Conference on Mass Storage Systems and Technologies held September 17 - 19, 1996, at the University of Maryland, University Conference Center in College Park, Maryland. As one of an ongoing series, this conference continues to serve as a unique medium for the exchange of information on topics relating to the ingestion and management of substantial amounts of data and the attendant problems involved. This year's discussion topics include storage architecture, database management, data distribution, file system performance and modeling, and optical recording technology. There will also be a paper on Application Programming Interfaces (API) for a Physical Volume Repository (PVR) defined in Version 5 of the Institute of Electrical and Electronics Engineers (IEEE) Reference Model (RM). In addition, there are papers on specific archives and storage products.

Kobler, Benjamin↗

Recent evolution of the offline computing model of the NOvA experiment

The NOvA experiment at Fermilab is a long-baseline neutrino experiment designed to study ν e appearance in a ν μ beam. Over the last few years there has been intense work to streamline the computing infrastructure in preparation for data, which started to flow in from the far detector in Fall 2013. Major accomplishments for this effort include migration to the use of off-site resources through the use of the Open Science Grid and upgrading the file-handling framework from simple disk storage to a tiered system using a comprehensive data management and delivery system to find and access files on either disk or tape storage. NOvA has already produced more than 6.5 million files and more than 1 PB of raw data and Monte Carlo simulation files which are managed under this model. In addition, the current system has demonstrated sustained rates of up to 1 TB/hour of file transfer by the data handling system. NOvA pioneered the use of new tools and this paved the way for their use by other Intensity Frontier experiments at Fermilab. Most importantly, the new framework places the experiment's infrastructure on a firm foundation, and is ready to produce the files needed for first physics.

97 MATHEMATICS AND COMPUTING↗

Proactive Data Containers for Scientific Storage (Final Report)

Emerging HPC systems are expected to be deployed with an unprecedented level of complexity, due to a deep system memory/storage hierarchy and heterogeneity of the storage hardware. This hierarchy is expected to range from CPU cache through several levels of volatile memory to nonvolatile memory, traditional hard disks, and tape. Simple and efficient methods of data management and movement through this hierarchy is critical for scientific applications using exascale systems. Existing storage system and I/O (SSIO) technologies face severe challenges in dealing with these requirements. POSIX and MPI I/O standards that are the basis for existing I/O libraries and parallel file systems present fundamental challenges in the areas of scalable metadata operations, semantics-based data movement performance tuning, asynchronous operation, and support for scalable consistency of distributed operations. Moving toward new paradigms for SSIO in the extreme-scale era, we have proposed to investigate novel object-based data abstractions and storage mechanisms that take advantage of the deep storage hierarchy and enable proactive automated performance tuning. In order to achieve these overarching goals, we initiated an effort to develop a fundamental new data abstraction, called Proactive Data Containers (PDC). A PDC is a container within a locus of storage (memory, NVRAM, disk, etc.) that stores science data in an object-centric manner. Managing data as objects enables powerful optimization opportunities for data movement and transformations. The R&D focus of this project are: 1) formulation of object-oriented PDCs and their mapping in different levels of the exascale storage hierarchy; 2) efficient strategies for moving data in deep storage hierarchies using PDCs; 3) techniques for transforming and reorganizing data based on application requirements; and 4) novel analysis paradigms for enabling data transformations and user-defined analysis on data in PDCs. Toward achieving these overarching goals, we designed an object-centric application programing interface (API) for HPC, scalable metadata management for object-centric storage systems, and data movement optimizations such as Data Elevator for moving data between two levels of storage devices and TAPIOCA for efficient aggregation of data on compute nodes. We then implemented several components of the PDC system. They include metadata management, data placement services, remote procedure calls, data aggregation, etc. We have put them together into the overall PDC framework.

97 MATHEMATICS AND COMPUTING↗

Proxy-based Bayesian inversion of strain tensor data measured during well tests

Recent instrument developments have made it possible to measure the strain tensor caused by injecting or pumping fluid from aquifers or reservoirs, but the full value of these data is limited because the long runtimes of poroelastic forward models makes it impractical to use many inversion schemes. This limits the interpretation of strain data for managing the recovery of resources or storage of wastes in the subsurface. This paper describes a method of inverting deformation data using a poroelastic numerical simulator so the results can be used to manage reservoirs or aquifers. We developed a workflow designed to reduce the number of simulations sufficiently to make it feasible to use DREAMzs, an advanced Bayesian inversion method that translates the uncertainties from different sources into unbiased posterior parameter distributions and uncertainty envelopes around the field data. Using a KNN proxy model for the poroelastic simulator is key to reducing the overall computations, and the workflow includes a strategy for ensuring the proxy model results converge on the results from the simulator. The workflow is tested using an idealized example that verifies the ability to correctly identify parameters and characterize noise used to perturb the data. Field data from an injection test at an oil reservoir near Tulsa, Oklahoma, are also used to evaluate the efficacy of the workflow with a real dataset. The workflow identified 265 history matching solutions out of 1240 total simulation runs (21% acceptance ratio), where the results were used to characterize posterior parameter distribution and evaluate the prediction uncertainty. Furthermore, this workflow is significant because it enables strain tensor, or other geomechanical measurements to be interpreted to guide decision-making during energy and environmental processes in the subsurface.

42 ENGINEERING↗

An Intelligent Distributed Ledger Construction Algorithm for IoT

Blockchain is the next generation of secure data management that creates near-immutable decentralized storage. Secure cryptography created a niche for blockchain to provide alternatives to well-known security compromises. However, design bottlenecks with traditional blockchain data structures scale poorly with increased network usage and are extremely computation-intensive. This made the technology difficult to combine with limited devices, like those in Internet of Things networks. In protocols like IOTA, replacement of blockchain's linked-list queue processing with a lightweight dynamic ledger showed remarkable throughput performance increase. However, current stochastic algorithms for ledger construction suffer distinct trade-offs between efficiency and security. This work proposed a machine-learning approach with a multi-arm bandit that resolved these issues and was designed for auditing on limited devices. This algorithm was tested in a reinforcement-learning environment simulating the IOTA ledger's construction with a decision tree. This study showed through regret analysis and experimentation that this approach was secure against impulse manipulation attacks while remaining energy-efficient. Although the IOTA protocol was a pioneer for lightweight distributed ledgers, it is expected that future blockchain protocols will adopt techniques similar to those presented in this work.

multi-arm bandit↗

Hybrid data storage system in an HPC exascale environment

A computer-executable method, system, and computer program product for managing I/O requests from a compute node in communication with a data storage system, including a first burst buffer node and a second burst buffer node, the computer-executable method, system, and computer program product comprising striping data on the first burst buffer node and the second burst buffer node, wherein a first portion of the data is communicated to the first burst buffer node and a second portion of the data is communicated to the second burst buffer node, processing the first portion of the data at the first burst buffer node, and processing the second portion of the data at the second burst buffer node.

97 MATHEMATICS AND COMPUTING↗

QoS-Enabled, High-performance Storage Systems for Data-Intensive Scientific Computing (Final Report)

This project's work resulted in the following research projects: (1) BORG - Block-reORGanization for Self-optimizing Storage Systems; (2) ABLE - Active Block Layer Extensions; (3) EXCES - EXternal Caching in Energy-Saving Storage Systems; and (4) GRIO - Guaranteed-Rate I/O Scheduler. These projects together help in substantially advancing the over-arching project goal of developing "QoS-Enabled, High-Performance Storage Systems".

99 GENERAL AND MISCELLANEOUS↗

Analysis of the access patterns at GSFC distributed active archive center

The Goddard Space Flight Center (GSFC) Distributed Active Archive Center (DAAC) has been operational for more than two years. Its mission is to support existing and pre Earth Observing System (EOS) Earth science datasets, facilitate the scientific research, and test Earth Observing System Data and Information System (EOSDIS) concepts. Over 550,000 files and documents have been archived, and more than six Terabytes have been distributed to the scientific community. Information about user request and file access patterns, and their impact on system loading, is needed to optimize current operations and to plan for future archives. To facilitate the management of daily activities, the GSFC DAAC has developed a data base system to track correspondence, requests, ingestion and distribution. In addition, several log files which record transactions on Unitree are maintained and periodically examined. This study identifies some of the users' requests and file access patterns at the GSFC DAAC during 1995. The analysis is limited to the subset of orders for which the data files are under the control of the Hierarchical Storage Management (HSM) Unitree. The results show that most of the data volume ordered was for two data products. The volume was also mostly made up of level 3 and 4 data and most of the volume was distributed on 8 mm and 4 mm tapes. In addition, most of the volume ordered was for deliveries in North America although there was a significant world-wide use. There was a wide range of request sizes in terms of volume and number of files ordered. On an average 78.6 files were ordered per request. Using the data managed by Unitree, several caching algorithms have been evaluated for both hit rate and the overhead ('cost') associated with the movement of data from near-line devices to disks. The algorithm called LRU/2 bin was found to be the best for this workload, but the STbin algorithm also worked well.

Johnson, Theodore↗

Using cross-layer adaptations for dynamic data management in large scale coupled scientific workflows, In: SC '13 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis

As system scales and application complexity grow, managing and processing simulation data has become a significant challenge. While recent approaches based on data staging and in-situ/in-transit data processing are promising, dynamic data volumes and distributions, such as those occurring in AMR-based simulations, make the efficient use of these techniques challenging. In this paper we propose cross-layer adaptations that address these challenges and respond at runtime to dynamic data management requirements. Specifically we explore (1) adaptations of the spatial resolution at which the data is processed, (2) dynamic placement and scheduling of data processing kernels, and (3) dynamic allocation of in-transit resources. We also exploit coordinated approaches that dynamically combine these adaptations at the different layers. We evaluate the performance of our adaptive cross-layer management approach on the Intrepid IBM-BlueGene/P and Titan Cray-XK7 systems using Chombo-based AMR applications, and demonstrate its effectiveness in improving overall time-to-solution and increasing resource efficiency.

Computer Science↗

CO 2 Storage Site Screening Platform Development and CO 2 Storage Resource Analysis in SECARB Offshore Reservoirs Using SAS Viya

A major goal of the SECARB Offshore Partnership (DE-FE0031557) is to screen deep saline aquifers and hydrocarbon reservoirs in the central Gulf of Mexico for CO 2 sequestration and CO 2 -enhanced oil and gas recovery (EOR/EGR) and estimate the corresponding CO 2 storage resources for select reservoirs. CO 2 storage potential associated with offshore CO 2 -EOR is considerable and likely represents “low hanging fruit” for near-term CO 2 storage given the in-place infrastructure in the region. It is for these reasons that this assessment focuses on oil and gas fields. To this end, three major objectives have been completed and include (1) managing geological data derived from different sources, (2) building a reservoir screening platform for CO 2 storage, and (3) ranking the reservoirs based on the estimated CO 2 storage resources. The SAS ® Viya platform was used for data management and analytics. The Viya platform is a cloud service platform that provides data integration, data management, quick analytics, data visualization, machine learning functions, and application programming interfaces (APIs) for multi-programming languages. Different sources of data containing geologic information, reservoir properties, and EOR/EGR information were collected, cleaned, formatted, and loaded into the SAS ® Viya platform for evaluation. The major geological characteristics of both shelf and deep-water areas of the central Gulf were examined and compared to define the appropriate reservoir screening criteria. Next, a CO 2 storage site screening system was built in the SAS ® Viya platform with the pre-defined criteria. Finally, the CO 2 storage resources of the screened reservoirs were calculated and reported at the BOEM field level to identify fields with the highest estimated CO 2 storage resource. The fields with the largest total estimated CO 2 storage resource are located in the Mississippi Canyon protraction area. Due to proximity to the Mississippi Delta (indicative of less infrastructure) and large estimated CO 2 storage resources, future development activities may wish to focus efforts in the Mississippi Canyon protraction area.

02 PETROLEUM↗

Geographic Information Systems for Assessing Existing and Potential Bio-energy Resources: Their Use in Determining Land Use and Management Options which Minimize Ecological and Landscape Impacts in Rural Areas

A management construct is described which forms part of an overall landscape ecological planning model which has as a principal objective the extension of the traditional descriptive land use mapping capabilities of geographic information systems into land management realms. It is noted that geographic information systems appear to be moving to more comprehensive methods of data handling and storage, such as relational and hierarchical data management systems, and a clear need has simultaneously arisen therefore for planning assessment techniques and methodologies which can actually use such complex levels of data in a systematic, yet flexible and scenario dependent way. The descriptive of mapping method proposed broaches such issues and utilizes a current New England bioenergy scenario, stimulated by the use of hardwoods for household heating purposes established in the post oil crisis era and the increased awareness of the possible landscape and ecological ramifications of the continued increasing use of the resource.

Jackman, A. E.↗

The Kepler Science Operations Center Pipeline Framework Extensions

The Kepler Science Operations Center (SOC) is responsible for several aspects of the Kepler Mission, including managing targets, generating on-board data compression tables, monitoring photometer health and status, processing the science data, and exporting the pipeline products to the mission archive. We describe how the generic pipeline framework software developed for Kepler is extended to achieve these goals, including pipeline configurations for processing science data and other support roles, and custom unit of work generators that control how the Kepler data are partitioned and distributed across the computing cluster. We describe the interface between the Java software that manages the retrieval and storage of the data for a given unit of work and the MATLAB algorithms that process these data. The data for each unit of work are packaged into a single file that contains everything needed by the science algorithms, allowing these files to be used to debug and evolve the algorithms offline.

Klaus, Todd C.↗