Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data storage data management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3

Towards Efficient Scientific Data Management Using Cloud Storage

A software prototype allows users to backup and restore data to/from both public and private cloud storage such as Amazon's S3 and NASA's Nebula. Unlike other off-the-shelf tools, this software ensures user data security in the cloud (through encryption), and minimizes users operating costs by using space- and bandwidth-efficient compression and incremental backup. Parallel data processing utilities have also been developed by using massively scalable cloud computing in conjunction with cloud storage. One of the innovations in this software is using modified open source components to work with a private cloud like NASA Nebula. Another innovation is porting the complex backup to- cloud software to embedded Linux, running on the home networking devices, in order to benefit more users.

He, Qiming↗

An emerging network storage management standard: Media error monitoring and reporting information (MEMRI) - to determine optical tape data integrity

Sophisticated network storage management applications are rapidly evolving to satisfy a market demand for highly reliable data storage systems with large data storage capacities and performance requirements. To preserve a high degree of data integrity, these applications must rely on intelligent data storage devices that can provide reliable indicators of data degradation. Error correction activity generally occurs within storage devices without notification to the host. Early indicators of degradation and media error monitoring 333 and reporting (MEMR) techniques implemented in data storage devices allow network storage management applications to notify system administrators of these events and to take appropriate corrective actions before catastrophic errors occur. Although MEMR techniques have been implemented in data storage devices for many years, until 1996 no MEMR standards existed. In 1996 the American National Standards Institute (ANSI) approved the only known (world-wide) industry standard specifying MEMR techniques to verify stored data on optical disks. This industry standard was developed under the auspices of the Association for Information and Image Management (AIIM). A recently formed AIIM Optical Tape Subcommittee initiated the development of another data integrity standard specifying a set of media error monitoring tools and media error monitoring information (MEMRI) to verify stored data on optical tape media. This paper discusses the need for intelligent storage devices that can provide data integrity metadata, the content of the existing data integrity standard for optical disks, and the content of the MEMRI standard being developed by the AIIM Optical Tape Subcommittee.

Podio, Fernando↗

Management and Storage of Scientific Data

Scientific discoveries rely heavily on efficient access, search, and management of massive data sets. Data management technologies have, for decades, provided foundational capabilities for scientific computing. Just as storage, input/output (I/O), and data management have been fundamental to simulation-based science for many years, so too are capable data-management technologies key to the success of today’s scientific workflows utilizing data intensive and machine learning (ML) techniques. The Department of Energy, Office of Science, Advanced Scientific Computing Research (ASCR) program has invested broadly in data-management research focused on high-performance computing (HPC) systems, from parallel file systems that store data to application software that makes these systems more productive. Still, advances in technology combined with growing diversity of supported science strongly motivate continued investment in this area. In January 2022, ASCR convened a workshop to identify priority research directions in the area of data management for high-performance and scientific computing. Attendees were challenged to identify promising approaches that would support the breadth of the DOE mission, including the explosion of artificial intelligence (AI) uses and the growing needs of experimental and observational science. Technological and science drivers were identified and considered as they relate to key aspects of data management such as interfaces, architectural design, and FAIR principles (Findable, Accessible, Interoperable, and Reusable). The thoughts of the workshop participants were distilled into a set of four priority research directions with the potential for high impact on DOE science. These research directions are summarized in the following pages.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Using expert systems to implement a semantic data model of a large mass storage system

The successful development of large volume data storage systems will depend not only on the ability of the designers to store data, but on the ability to manage such data once it is in the system. The hypothesis is that mass storage data management can only be implemented successfully based on highly intelligent meta data management services. There now exists a proposed mass store system standard proposed by the IEEE that addresses many of the issues related to the storage of large volumes of data, however, the model does not consider a major technical issue, namely the high level management of stored data. However, if the model were expanded to include the semantics and pragmatics of the data domain using a Semantic Data Model (SDM) concept, the result would be data that is expressive of the Intelligent Information Fusion (IIF) concept and also organized and classified in context to its use and purpose. The results are presented of a demonstration prototype SDM implemented using the expert system development tool NEXPERT OBJECT. In the prototype, a simple instance of a SDM was created to support a hypothetical application for the Earth Observing System, Data Information System (EOSDIS). The massive amounts of data that EOSDIS will manage requires the definition and design of a powerful information management system in order to support even the most basic needs of the project. The application domain is characterized by a semantic like network that represents the data content and the relationships between the data based on user views and the more generalized domain architectural view of the information world. The data in the domain are represented by objects that define classes, types and instances of the data. In addition, data properties are selectively inherited between parent and daughter relationships in the domain. Based on the SDM a simple information system design is developed from the low level data storage media, through record management and meta data management to the user interface.

Roelofs, Larry H.↗

Programming Abstractions for Managing Workflows on Tiered Storage Systems

Scientific workflows in High Performance Computing (HPC) environments are processing large amounts of data. The storage hierarchy on HPC systems is getting deeper, driven by new technologies (NVRAMs, SSDs, etc.) There is a need for new programming abstractions that allow users to seamlessly manage data at the workflow level on multi-tiered storage systems, and provide optimal workflow performance and use of storage resources. In previous work, we introduced a software architecture Managing Data on Tiered Storage for Scientific Workflows (MaDaTS) that used a Virtual Data Space (VDS) abstraction to hide the complexities of the underlying storage system while allowing users to control data management strategies. In this article, we detail the data-centric programming abstractions that allow users to manage a workflow around its data on the storage layer. The programming abstractions simplify data management for scientific workflows on multi-tiered storage systems, without affecting workflow performance or storage capacity. We measure the overheads and effectiveness introduced by the programming abstractions of MaDaTS. Our results show that these abstractions can optimally use the storage capacity in lesser capacity storage tiers, and simplify data management without adding any performance overheads.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

EMASS (trademark): An expandable solution for NASA space data storage needs

The data acquisition, distribution, processing, and archiving requirements of NASA and other U.S. Government data centers present significant data management challenges that must be met in the 1990's. The Earth Observing System (EOS) project alone is expected to generate daily data volumes greater than 2 Terabytes (2 x 10(exp 12) Bytes). As the scientific community makes use of this data, their work will result in larger, increasingly complex data sets to be further exploited and managed. The challenge for data storage systems is to satisfy the initial data management requirements with cost effective solutions that provide for planned growth. The expendable architecture of the E-Systems Modular Automated Storage System (EMASS(TM)), a mass storage system which is designed to support NASA's data capture, storage, distribution, and management requirements into the 21st century is described.

Peterson, Anthony L.↗

EMASS (tm): An expandable solution for NASA space data storage needs

The data acquisition, distribution, processing, and archiving requirements of NASA and other U.S. Government data centers present significant data management challenges that must be met in the 1990's. The Earth Observing System (EOS) project alone is expected to generate daily data volumes greater than 2 Terabytes (2(10)(exp 12) Bytes). As the scientific community makes use of this data their work product will result in larger, increasingly complex data sets to be further exploited and managed. The challenge for data storage systems is to satisfy the initial data management requirements with cost effective solutions that provide for planned growth. This paper describes the expandable architecture of the E-Systems Modular Automated Storage System (EMASS (TM)), a mass storage system which is designed to support NASA's data capture, storage, distribution, and management requirements into the 21st century.

Peterson, Anthony L.↗

ATLAS Data Carousel

The ATLAS experiment at CERN's LHC stores detector and simulation data in raw and derived data formats across more than 150 Grid sites world-wide, currently in total about 200PB on disk and 250PB on tape. Data have different access characteristics due to various computational workflows, and can be accessed from different media, such as remote I/O, disk cache on hard disk drives or SSDs. Also, larger data centers provide the majority of offline storage capability via tape systems. For the High-Luminosity LHC (HL-LHC), the estimated data storage requirements are several factors bigger than the present forecast of available resources, based on a flat budget assumption. On the computing side, ATLAS Distributed Computing was very successful in the last years with high performance and high throughput computing integration and in using opportunistic computing resources for the Monte Carlo simulation. On the other hand, equivalent opportunistic storage does not exist. ATLAS started the "Data Carousel" project to increase the usage of less expensive storage, i.e. tapes or even commercial storage, so it is not limited to tape technologies exclusively. Data Carousel orchestrates data processing between workload management, data management, and storage services with the bulk data resident on offline storage. The processing is executed by staging and promptly processing a sliding window of inputs onto faster buffer storage, such that only a small percentage of input data are available at any one time. With this project, we aim to demonstrate that this is the natural way to dramatically reduce our storage cost. The first phase of the project was started in the fall of 2018 and was related to I/O tests of the sites archiving systems. Phase II now requires a tight integration of the workload and data management systems. Additionally, the Data Carousel studies the feasibility to run multiple computing workflows from tape. The project is progressing very well and the results presented in this document will be used before the LHC Run 3.

97 MATHEMATICS AND COMPUTING↗

Livewire Data Platform-A Solution for Energy Efficient Mobility Systems (EEMS) Data Sharing

This presentation is an overview of progress made on the Livewire Data Platform since June 2021. Livewire is a publicly accessible platform for sharing energy efficiency and mobility research data funded by DOE's Vehicle Technologies Office. Core services and platform capabilities of Livewire include: Free, secure data storage; Access management that allows data owners to control who sees their data; Data collection and preservation; Quality characterization; Detailed access and download metrics; and Increased visibility of projects and data.

energy efficiency mobility↗

Livewire Data Platform-A Solution for Energy Efficient Mobility Systems (EEMS) Data Sharing

This presentation is an overview of progress made on the Livewire Data Platform since June 2022. Livewire is a publicly accessible platform for sharing energy efficiency and mobility research data funded by DOE's Vehicle Technologies Office. Core services and platform capabilities of Livewire include: Free, secure data storage; Access management that allows data owners to control who sees their data; Data collection and preservation; Quality characterization; Detailed access and download metrics; and Increased visibility of projects and data. Anyone can create an account and access data at https://livewire.energy.gov/.

energy efficiency↗

Notes on a storage manager for the Clouds kernel

The Clouds project is research directed towards producing a reliable distributed computing system. The initial goal is to produce a kernel which provides a reliable environment with which a distributed operating system can be built. The Clouds kernal consists of a set of replicated subkernels, each of which runs on a machine in the Clouds system. Each subkernel is responsible for the management of resources on its machine; the subkernal components communicate to provide the cooperation necessary to meld the various machines into one kernel. The implementation of a kernel-level storage manager that supports reliability is documented. The storage manager is a part of each subkernel and maintains the secondary storage residing at each machine in the distributed system. In addition to providing the usual data transfer services, the storage manager ensures that data being stored survives machine and system crashes, and that the secondary storage of a failed machine is recovered (made consistent) automatically when the machine is restarted. Since the storage manager is part of the Clouds kernel, efficiency of operation is also a concern.

Pitts, David V.↗

ARCHIE: Data Analysis Acceleration with Array Caching in Hierarchical Storage

© 2018 IEEE. Scientific data analysis typically involves reading massive amounts of data that was generated by simulations, experiments, and observations. Performance of reading such large volumes of data from disk-based file systems is often poor because of the slow and mechanical components in the disks. Recent supercomputing systems are adding non-volatile storage layers in a hierarchy to handle the performance gap between fast main memory and slow disk-based storage. Software libraries for managing this hierarchy not only need efficient reading of data but also reduce user-involvement for cross-layer data movement. Furthermore, these libraries need to support array data access patterns into hierarchical storage management as scientific data is often organized in array-based data structures. Existing software typically manage individual storage layers requiring significant manual process in moving data among them. In this paper, we introduce a new array caching in hierarchical storage (ARCHIE) to accelerate array data analysis in a seamless fashion. ARCHIE evaluates array access patterns and prefetches data with array semantics between storage layers. Our evaluation shows that ARCHIE outperforms state-of-the-art file systems, i.e., Lustre and DataWarp, on a production supercomputing system by up to 5.8× in accessing data by scientific analysis applications.

Dong, Bin↗

MERRA/AS: The MERRA Analytic Services Project Interim Report

MERRA AS is a cyberinfrastructure resource that will combine iRODS-based Climate Data Server (CDS) capabilities with Coudera MapReduce to serve MERRA analytic products, store the MERRA reanalysis data collection in an HDFS to enable parallel, high-performance, storage-side data reductions, manage storage-side driver, mapper, reducer code sets and realized objects for users, and provide a library of commonly used spatiotemporal operations that can be composed to enable higher-order analyses.

MapReduce↗

Adapting Digital Libraries to Continual Evolution

In this paper, we describe five investment streams (data storage infrastructure, knowledge management, data production control, data transport and security, and personnel skill mix) that need to be balanced against short-term operating demands in order to maximize the probability of long-term viability of a digital library. Because of the rapid pace of information technology change, a digital library cannot be a static institution. Rather, it has to become a flexible organization adapted to continuous evolution of its infrastructure.

Barkstrom, Bruce R.↗

Development of a Technology Roadmap for Online Monitoring of Nuclear Power Plants

The nuclear power industry has recently recognized the potential cost saving that can be achieved from migrating the current manual and labor dependent surveillance and preventive maintenance activities to data-driven online monitoring methods. Since then, several efforts have been launched with various degrees of momentum to tackle specific surveillance or preventive maintenance activities. The light water reactor sustainability program (LWRS) program has anticipated this need and launched an effort that aimed to develop a technology roadmap for the nuclear power industry. Because the roadmap has to be focused towards the user or application’s value, and because nuclear power plants have different process efficiencies and deficiencies, the technology roadmap has to be plant independent. Achieving this objective required optimizing the extent of details described by a technology roadmap. The approach followed is for the roadmap to describe the sequence of activities needed to automate a process or equipment activity, but not describe the sequence of processes or equipment to target. For example, the steps required to migrate the inspection of a pump are plant independent. However, defining that the migration of feed water pumps inspections to online monitoring should be performed before inspections of feed water valves is dependent on multiple plant factors such as the plant requirements, labor work capacity, and process or equipment conditions. The technology roadmap was broken down into six elements, four of which have a sequential path of advancement, while the other two are supporting elements to the sequential elements. The sequential part of the roadmap analyzes data collection, analytics, visualization, and management. For these elements of the technology roadmap, the migration of an activity is defined in three states. The base state is defined as the most primitive, manual, and labor dependent process used by some nuclear power plants. The modern state is defined as the state that can be achieved if the activities are augmented or replaced by current technologies that are commercially available or are soon to be available. The state of the art is defined as using technologies of the future, i.e. concept and technologies that are being researched. The data collection part of the technology roadmap defines three states of every process performed in a nuclear power plant. However, the data analytics part is described based on equipment, because automating an activity performed on one equipment required considering multiple data sources. The visualization and data management elements are process or equipment independent. Visualization describes the means to use human factor science and technologies advancement in presenting the data collected and the data analytics results. The data management targets storage, communication, and computational power need, in addition to data integration and sharing. Since storage, communication, and computational power technologies and path for deployment are known to any information technology organization, this effort targeted the cost of data management. This has a direct impact on the feasibility of the automation effort. The supporting elements analyzes the value of automating the activity and the change challenges to tackle. For the support elements, the roadmap targeted describing activities that need to be performed as part of the roadmap development. The value element of the roadmap describes how to analyze the cost saving and the associated risk. The change management identifies multiple factors to consider such as regulation, cybersecurity, resource availability, supporting infrastructure availability, feasibility (power, fitness, environment tolerance, facilities), and cultural change.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Opportunities and data requirements for data-driven prognostics and health management in liquid hydrogen storage systems

During the past decade, Prognostics and Health Management (PHM) has become an important set of tools in various areas of industry and academic reliability engineering. PHM consists of a variety of mathematical and computational methods used to support data-driven decision-making to increase the safety, availability, and reliability of complex engineering systems. In particular, PHM can provide crucial insight into reliability and safety design improvements for developing technologies where historical performance and failure data are limited. This is the case of hydrogen fueling and storage technologies. This work presents a high-level approach for designing data-driven PHM applications for bulk liquid hydrogen (LH 2 ) storage systems for hydrogen fueling stations. This paper addresses core aspects of the design, development, and implementation of data-driven PHM applications that can improve the reliability assessment of hydrogen components. The analysis focuses on the relationship between data availability and diagnostic/prognostic capabilities; potential challenges; and integration schemes for current risk mitigation measures. We identify potential condition-monitoring data sources for key components in an LH 2 storage system, including storage tanks, piping, and pumps. Further, we determine that the short-term goals for the implementation of data-driven models in PHM frameworks in hydrogen systems should focus on developing adequate data collection and analysis strategies, as well as exploring the effect on reliability, safety, and regulations for hydrogen systems.

08 HYDROGEN↗

System and Method for Providing a Climate Data Persistence Service

A system, method and computer-readable storage devices for providing a climate data persistence service. A system configured to provide the service can include a climate data server that performs data and metadata storage and management functions for climate data objects, a compute-storage platform that provides the resources needed to support a climate data server, provisioning software that allows climate data server instances to be deployed as virtual climate data servers in a cloud computing environment, and a service interface, wherein persistence service capabilities are invoked by software applications running on a client device. The climate data objects can be in various formats, such as International Organization for Standards (ISO) Open Archival Information System (OAIS) Reference Model Submission Information Packages, Archive Information Packages, and Dissemination Information Packages. The climate data server can enable scalable, federated storage, management, discovery, and access, and can be tailored for particular use cases.

Schnase, John L.↗

Southeast Regional CO 2 Utilization and Storage Acceleration Partnership (SECARB-USA): Data Management Plan

The “Southeast Regional CO 2 Utilization and Storage Acceleration Partnership” (SECARB-USA) is designed to assess the region’s subsurface resources and burgeoning technology applications. This work supports the U.S. Department of Energy’s (DOE) Office of Fossil Energy’s (FE) mission to help the United States meet its need for secure, affordable, and environmentally sound fossil energy supplies by utilizing the advancements made by the current RCSP initiative to continue to identify and address knowledge gaps.

42 ENGINEERING↗