Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data storage data management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 505 records · Page 28

Acquisition and Development of Selected Cost Data for Saline Storage and Enhanced Oil Recovery (EOR)

This report presents cost data and recommends cost estimation methods for costs associated with CO 2 injection and surface facility management, including well equipment, onsite booster compression. Where water is produced for pressure management in saline aquifers, costs for water production, water injection, and associated surface treatment and wellhead equipment (along with associated piping and facilities) is provided. Comparable cost estimation recommendations are provided for CO 2 -EOR operations, including the costs associated with oil and water management and production, and CO 2 recycling.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

ROOT RNTuple and EOS: The Next Generation of Event Data I/O

For several years, the ROOT team is developing the new RNTuple I/O subsystem in preparation of the next generation of collider experiments. Both HL-LHC and DUNE are expected to start data taking by the end of this decade. They pose unprecedented challenges to event data I/O in terms of data rates, event sizes, and event complexity. At the same time, the I/O landscape is becoming more diverse. HPC cluster file systems and object stores, NVMe disk cache layers in analysis facilities, and S3 storage on cloud resources are mixing with traditional XRootD-managed spinning disk pools.The ROOT team will finalize a first production version of the RNTuple binary format by the end of 2024. After this point, ROOT will provide backward compatibility for RNTuple data. This contribution provides an overview of the RNTuple feature set, the related R&D activities and the long-term vision for RNTuple. We report on performance, interface design, tooling, robustness, integration with experiment frameworks, and validation results, as well as recent R&D on parallel reading and writing and exploitation of modern hardware and storage systems. We will give an outlook on possible future features after a first production release.Collaboratively, the IT and EP departments at CERN have launched a formal project within the Research and Computing sector to evaluate the novel data format for physics analysis data utilized in LHC experiments and other fields. This part of the project focuses on validating the scalability of the EOS storage backend during the transition from the over 25 years old TTree production format to the newly developed RNTuple format, using both replicated and erasure-coded storage profiles.

Blomer, Jakob [CERN]↗

Distributed Resources for the Earth System Grid Federation (ESGF) Advanced Management (DREAM). Final Report

Distributed Resources for the Earth System Grid Federation (ESGF) Advanced Management (DREAM) is a proposed system that will enable data from an infinite number of diverse sources to be organized and accessed from anywhere using any handheld or other computer device. The approach offers a powerful roadmap for the creation and integration of a unified knowledge base of an entire ecosystem, including its many geophysical, geographical, social, political, agricultural, energy, transportation, and cyber aspects. The resulting aggregation of data has the potential to generate an informational universe of unprecedented size that has never before been possible due to the prohibitive costs, managerial complexity, and technical barriers associated with ever-changing exponential-growth data flows. We envision that DREAM will accelerate discovery by enabling climate researchers, among other types of researchers, to manage, analyze, and visualize data from earth-scale measurements and simulations. DREAM’s success will be built on proven components that leverage existing services and resources. A key building block for DREAM will be the ESGF, chaired by Dean N. Williams. Expanding on the existing ESGF, the project will ensure that the access, storage, movement, and analysis of the large quantities of data that are processed and produced by diverse science projects can be dynamically distributed with proper resource management. Much of the Office of Science data is currently generated by multiple stand-alone facilities. DREAM can collect data accumulated from these facilities and incorporate it into a fully integrated network accessible from anywhere in the world. The result is a completely new paradigm shift for data management, analysis, and visualization enabling researchers to: Manage their calculations, data, tools, and research results; Ensure that all data are sharable, reproducible and (re)usable—accompanied by appropriate metadata describing its provenance, syntax, and semantics at creation; Advance application performance by selectively adapting APIs and services in response to scientific requirements and architectural complexities; and Provide scalable interactive resource management—navigate data and metadata at multiple levels, provide architecture-aware data integration, analysis and visualization tools. We will engage closely with DOE, NASA, and NOAA science groups working at the leading edge of computing. These engagements—in domains such as biology, climate, and hydrology—will allow us to advance disciplinary science goals and inform our development of technologies that can accelerate discovery across DOE more broadly. We will advertise and promote our technologies via dedicated workshops, tutorials, and sessions at conferences, stand-alone events with broad inter-disciplinary invitation, and engagements with leadership facilities.

54 ENVIRONMENTAL SCIENCES↗

An intelligent Data Delivery Service for and beyond the ATLAS experiment

The intelligent Data Delivery Service (iDDS) has been developed to cope with the huge increase of computing and storage resource usage in the coming LHC data taking. iDDS has been designed to intelligently orchestrate workflow and data management systems, decoupling data pre-processing, delivery, and main processing in various workflows. It is an experiment-agnostic service around a workflow-oriented structure to work with existing and emerging use cases in ATLAS and other experiments. Here we will present the motivation for iDDS, its design schema and architecture, use cases and current status, and plans for the future.

97 MATHEMATICS AND COMPUTING↗

An intelligent Data Delivery Service for and beyond the ATLAS experiment

The intelligent Data Delivery Service (iDDS) has been developed to cope with the huge increase of computing and storage resource usage in the coming LHC data taking. It has been designed to intelligently orchestrate workflows and data management systems, decoupling data pre-processing, delivery, and primary processing in large scale workflows. It is an experiment-agnostic service that has been deployed to serve data carousel (orchestrating efficient processing of tape-resident data), machine learning hyperparameter optimization, active learning, and other complex multi-stage workflows defined via DAG (Directed Acyclic Graph), CWL (Common Workflow Language) and other descriptions, including a growing number of analysis workflows. We will at first introduce some deployed use cases in a summary. Then we will focus on new improvements and use cases under developments in ATLAS, Rubin Observatory and sPHENIX, together with future efforts.

97 MATHEMATICS AND COMPUTING↗

Data management system technology discipline

Viewgraphs on data management system technology discipline for Space Station Freedom are presented. Topics covered include: systems technology area needs; storage technology area needs; processor technology area needs; communications technology area needs; software system technology area needs; human interface technology area needs; software development and verification; and onboard communications.

Benz, Harry F.↗

Integrated Systems Health Management for Intelligent Systems

The implementation of an integrated system health management (ISHM) capability is fundamentally linked to the management of data, information, and knowledge (DIaK) with the purposeful objective of determining the health of a system. Management implies storage, distribution, sharing, maintenance, processing, reasoning, and presentation. ISHM is akin to having a team of experts who are all individually and collectively observing and analyzing a complex system, and communicating effectively with each other in order to arrive at an accurate and reliable assessment of its health. In this chapter, concepts, procedures, and approaches are presented as a foundation for implementing an ISHM capability relevant to intelligent systems. The capability stresses integration of DIaK from all elements of a system, emphasizing an advance toward an on-board, autonomous capability. Both ground-based and on-board ISHM capabilities are addressed. The information presented is the result of many years of research, development, and maturation of technologies, and of prototype implementations in operational systems.

Figueroa, Fernando↗

Prospective Seal Name Catalog for U.S. Sedimentary Basins V1.0

The Prospective Seal Name Catalog for U.S. Sedimentary Basins V1.0 is an aggregation of prospective seal units for prospective storage resources within the USA for geologic carbon storage in both onshore and offshore basins. The catalog is intended to guide users to available literature and data resources. Prospective seal names are displayed with respect to the associated prospective storage resource target, for which the specified prospective seal may act as a confining layer for CO2 storage. A prospective seal unit’s lithology, position with respect to the reservoir (Primary, Secondary, Intraformational, etc.), and age (Geologic Period) are included when stated in the original literature or data resource. The information source for each record in the catalog includes the published citation, year of publication, resource type (Journal Article, Event Proceedings, etc.) and additional record notes.

Basin↗

A Vision for Coupling Operation of US Fusion Facilities with HPC Systems and the Implications for Workflows and Data Management

The operation of large US Department of Energy (DOE) research facilities, like the DIII-D National Fusion Facility, results in the collection of complex multi-dimensional scientific datasets, both experimental and model-generated. In the future, it is envisioned that integrated data analysis coupled with large-scale high performance computing (HPC) simulations will be used to improve experimental planning and operation. Practically, massive data sets from these simulations provide the physics basis for generation of both reduced semi-analytic and machine-learning-based models. Storage of both HPC simulation datasets (generated from US DOE leadership computing facilities) and experimental datasets presents significant challenges. In this paper, we present a vision for a DOE-wide data management workflow that integrates US DOE fusion facilities with leadership computing facilities. Data persistence and long-term availability beyond the length of allocated projects is essential, particularly for verification and recalibration of artificial intelligence and machine learning (AI/ML) models. Because these data sets are often generated and shared among hundreds of users across multiple leadership computing facility centers, they would benefit from cross-platform accessibility, persistent identifiers (e.g. DOI, or digital object identifier), and provenance tracking. Here, the ability to handle different data access patterns suggests that a combination of low cost, high latency (e.g. for storing ML training sets) and high cost, low latency systems (e.g. for real-time, integrated machine control feedback) may be needed.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

The COLD-SAT Experiment for Cryogenic Fluid Management Technology

Future national space transportation missions will depend on the use of cryogenic fluid management technology development needs for these missions. In-space testing will be conducted in order to show low gravity cryogenic fluid management concepts and to acquire a technical data base. Liquid H2 is the preferred test fluid due to its propellant use. The design of COLD-SAT (Cryogenic On-orbit Liquid Depot Storage, Acquisition, and Transfer Satellite), an Expendable Launch Vehicle (ELV) launched orbital spacecraft that will perform subcritical liquid H2 storage and transfer experiments under low gravity conditions is studied. An Atlas launch vehicle will place COLD-SAT into a circular orbit, and the 3-axis controlled spacecraft bus will provide electric power, experiment control, and data management, attitude control, and propulsive accelerations for the experiments. Low levels of acceleration will provide data on the effects that low gravity might have on the heat and mass transfer processes used. The experiment module will contain 3 liquid H2 tanks; fluid transfer, pressurization and venting equipment; and instrumentation.

Schuster, J. R.↗

Autoclave Corrosion Experiment of Zircaloy-4 at Temperatures Between 180 deg. C and 270 deg. C - 19167

Corrosion of zirconium alloys is a key issue for the overall nuclear industry that has been studied under various temperatures for plant operation, severe accidents, storage, and final disposal of spent cladding. The present work aims to investigate the corrosion kinetics of zirconium alloys at intermediate temperature regions between the operation and storage/disposal conditions, to examine if the extrapolation of high temperature data to lower temperatures is valid. To achieve this, an autoclave experiment, which is a common way to study corrosion behavior, was applied for up to 485 days at 4 different temperatures: 180 deg. C, 210 deg. C, 240 deg. C and 270 deg. C. The weight gain changes due to the oxide growth of the PWR cladding tube (Zircaloy-4) showed that the corrosion kinetics in temperatures above 210 deg. C follow the cubic rate law. The activation energy evaluated by the Arrhenius relationship of the cubic rate constants was determined and found to be 132.8 ± 0.6 kJ/mol, which corresponds to that generally reported under the operation temperatures. However, the kinetics at 180 deg. C was found to be close to the parabolic rate law, which suggests the kinetics changes from cubic to parabolic around 200 deg. C. Thus, it is unlikely to predict accurately long-term corrosion behavior under storage and disposal conditions by simply extrapolating the high temperature corrosion data. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

A Task Based Approach for Co-Scheduling Ensemble Workloads on Heterogeneous Nodes

Scientific workflows consist of multiple, connected applications, with data and results flowing from one to another in a pipeline. Traditionally, such workflows are executed in sequential order, storing intermediate data in storage disks. Co-scheduling application workflows concurrently on the same compute nodes would greatly reduce the cost of moving data to/from storage and allow real-time analysis of intermediate results. Nevertheless, most parallel programming runtimes do not allow seamless integration of various applications in a scientific workflow, in part due to the complexity of managing data and resources. The situation is even more complicated for heterogeneous systems. In this work we extend the Minos Computing Library (MCL) runtime to accelerate pipe-lined and parallel workloads where multiple applications are running in the same system. MCL’s asynchronous task library and runtime dynamically manages resources to allow co-scheduling of multiple processes sharing heterogeneous resources. In addition, we design a custom ex- tension of the Open Compute Language (OpenCL) to enable multiple processes to share device memory. We enable MCL to coordinate these shared buffers to allow for easy, fast data sharing between applications. Using malleable micro-benchmarks and two application workflows that combine scientific simulation and AI-based analysis, we show that our method outperforms traditional approaches.

Index Terms—Parallel systems, Scheduling and Task ↗

The medical science DMZ: a network design pattern for data-intensive medical science

Abstract Objective We describe a detailed solution for maintaining high-capacity, data-intensive network flows (eg, 10, 40, 100 Gbps+) in a scientific, medical context while still adhering to security and privacy laws and regulations. Materials and Methods High-end networking, packet-filter firewalls, network intrusion-detection systems. Results We describe a “Medical Science DMZ” concept as an option for secure, high-volume transport of large, sensitive datasets between research institutions over national research networks, and give 3 detailed descriptions of implemented Medical Science DMZs. Discussion The exponentially increasing amounts of “omics” data, high-quality imaging, and other rapidly growing clinical datasets have resulted in the rise of biomedical research “Big Data.” The storage, analysis, and network resources required to process these data and integrate them into patient diagnoses and treatments have grown to scales that strain the capabilities of academic health centers. Some data are not generated locally and cannot be sustained locally, and shared data repositories such as those provided by the National Library of Medicine, the National Cancer Institute, and international partners such as the European Bioinformatics Institute are rapidly growing. The ability to store and compute using these data must therefore be addressed by a combination of local, national, and industry resources that exchange large datasets. Maintaining data-intensive flows that comply with the Health Insurance Portability and Accountability Act (HIPAA) and other regulations presents a new challenge for biomedical research. We describe a strategy that marries performance and security by borrowing from and redefining the concept of a Science DMZ, a framework that is used in physical sciences and engineering research to manage high-capacity data flows. Conclusion By implementing a Medical Science DMZ architecture, biomedical researchers can leverage the scale provided by high-performance computer and cloud storage facilities and national high-speed research networks while preserving privacy and meeting regulatory requirements.

Peisert, Sean↗

Enabling High Availability Service with oVirt Virtualization and CephFS

The online computing environment at STAR has generated demand for high availability of services (HAS) and a resilient uptime guarantee. Such services include databases, webservers, and storage systems that user and sub-systems tend to rely on for their critical workflows. Standard deployment of services on bare metal creates a problem if the fundamental hardware fails or loses connectivity. Additionally, the configuration of redundant failover nodes (a secondary Web service for example) requires constant syncing of configuration and content and sometimes manual interaction for switching hardware (DNS name comes to mind). For this project, we will focus on two tools: oVirt and Ceph. oVirt is an OpenSource virtualization management application that enables central management of hardware nodes, storage, and network resources used to deploy and monitor virtual machines. oVirt supports the deployment of a virtual environment for your data center leveraging automatic provisioning, live migration, and the ability to easily scale the number of hypervisors. oVirt enables the use of multiple storage technologies where you can store virtual machines, images, and templates within one or multiple storage systems. STAR's recent efforts focused on deploying a CephFS POSIX compliant distributed storage system, we would enable the ability to couple our Ceph storage with the oVirt virtualization management system. When designing an intricate system for hosting critical services, it is a requirement to circumvent single points of failure. This work will involve the testing and viability of such an approach along with test cases for high availability, live migration, and service on demand.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Supporting Global Air Quality Management Needs With A Flexible Data Fusion Tool for Estimation and Forecasting in Google Earth Engine

High spatial and temporal resolution air quality estimation and forecasting can be enhanced by combining global data sources, like chemical transport models and satellite remote sensing, with local information from regulatory and low-cost air quality monitors. Successful integration of data from these diverse sources is complicated by many factors, however, including differences in spatial and temporal resolution, data availability and latency issues, varying data quality, and large computational and data storage requirements. This presentation will provide an overview of a NASA-funded effort to develop the foundation for future operationalization of air quality forecasting for world-wide end-users and integration into their air quality management decision processes, which will be achieved in future phases of this multi-year project. We will summarize our progress in developing a data fusion system using the Google Earth Engine platform which can integrate model, satellite, and surface-level monitoring datasets to enhance estimation and forecasting of air-quality-relevant pollutants at sub-daily and sub-city scales. The tool is being developed in close cooperation with several city- and regional-level air quality managers in the USA and around the world. Our end-goal is to provide these air quality managers with the information they need to assess and anticipate the impacts of poor air quality, track changes in air quality due to ongoing mitigation efforts and land use changes, and identify ways to improve their air quality monitoring strategies. This presentation will focus on recent advances achieved through the project, including integration of multiple air quality datasets in a prototype data fusion system in Google Earth Engine, the quantification of uncertainties associated with our data fusion approach, and the development of user interfaces and visualization tools to convey air quality information in a way which best meets end-user needs.

Carl Malings↗

Supporting Global Air Quality Management Needs With A Flexible Data Fusion Tool for Estimation and Forecasting in Google Earth Engine

High spatial and temporal resolution air quality estimation and forecasting can be enhanced by combining global data sources, like chemical transport models and satellite remote sensing, with local information from regulatory and low-cost air quality monitors. Successful integration of data from these diverse sources is complicated by many factors, however, including differences in spatial and temporal resolution, data availability and latency issues, varying data quality, and large computational and data storage requirements. This presentation will provide an overview of a NASA-funded effort to develop the foundation for future operationalization of air quality forecasting for world-wide end-users and integration into their air quality management decision processes, which will be achieved in future phases of this multi-year project. We will summarize our progress in developing a data fusion system using the Google Earth Engine platform which can integrate model, satellite, and surface-level monitoring datasets to enhance estimation and forecasting of air-quality-relevant pollutants at sub-daily and sub-city scales. The tool is being developed in close cooperation with several city- and regional-level air quality managers in the USA and around the world. Our end-goal is to provide these air quality managers with the information they need to assess and anticipate the impacts of poor air quality, track changes in air quality due to ongoing mitigation efforts and land use changes, and identify ways to improve their air quality monitoring strategies. This presentation will focus on recent advances achieved through the project, including integration of multiple air quality datasets in a prototype data fusion system in Google Earth Engine, the quantification of uncertainties associated with our data fusion approach, and the development of user interfaces and visualization tools to convey air quality information in a way which best meets end-user needs.

Carl Malings↗

CARETS: A prototype regional environmental information system. Volume 3: Toward a national land use information system

It is recommended that a national land use information system be established by an agency of the Federal Government. This recommendation comes at a time of increasing demand for scientific information in support of environmentally relevant land use planning and management at all levels of government. It is also a time when new airborne and spaceborne remote sensors, tested in cooperation with the National Aeronautics and Space Administration (NASA) and the Earth Resources Observation Systems (EROS) Program of the Department of the Interior, make possible the gathering of land use information rapidly and on an unprecedented scale. Furthermore, information handling technology is developing toward a capability to receive, store, and disseminate the huge quantities of data that would be involved. The recommendation for the national land use information system is based upon careful analysis of the results of remote sensing experiments funded by NASA, EROS, and the Geography Program of the Geological Survey, with specific examples drawn from the demonstration project known as the Central Atlantic Regional Ecological Test Site (CARETS). CARETS is cast in the framework of a regional land use information system, channeling the flow bf information generated in response to users' declaration of their needs, through stages dealing with remote sensing data gathering systems, data processing and land mensuration, calibration in-terms of environmental impact, and evaluation with feedback from users. The proposed system would develop and implement a unified approach to the description and interpretation of the changing uses of the nation's land resources, building upon the base of interagency and intergovernmental cooperation already achieved in the experimental work to date. The land use data base that is being derived from high-altitude aerial color infrared photography would be the initial component of the recommended system. High-altitude photographic coverage would immediately be extended to as much of the nation as possible as technological developments and economic considerations permit. The system would later expand to include multiple-sensor, multiple-platform data sources. Six system characteristics are recommended: (1) High capacity storage of data available for quick retrieval, inexpensive processing, and update, (2) provision of accuracy appropriate to the scale of survey or to the level of detail dictated by different types of management and decision requirements; (3) permanent, publicly accessible sensor records for historical interpretation; (4) compatibility of the recording, storage, and retrieval system with all types of inputs, from ground observer to satellite; (5) products of diverse formats and scales, responsive to user feedback; (6) and standardization of formats, scales, and storage inputs to permit nationwide comparability.

Land use mapping↗

ICI optical data storage tape: An archival mass storage media

At the 1991 Conference on Mass Storage Systems and Technologies, ICI Imagedata presented a paper which introduced ICI Optical Data Storage Tape. This paper placed specific emphasis on the media characteristics and initial data was presented which illustrated the archival stability of the media. More exhaustive analysis that was carried out on the chemical stability of the media is covered. Equally important, it also addresses archive management issues associated with, for example, the benefits of reduced rewind requirements to accommodate tape relaxation effects that result from careful tribology control in ICI Optical Tape media. ICI Optical Tape media was designed to meet the most demanding requirements of archival mass storage. It is envisaged that the volumetric data capacity, long term stability and low maintenance characteristics demonstrated will have major benefits in increasing reliability and reducing the costs associated with archival storage of large data volumes.

Ruddick, Andrew J.↗