Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data storage data management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6

Volume server: A scalable high speed and high capacity magnetic tape archive architecture with concurrent multi-host access

A major challenge facing data processing centers today is data management. This includes the storage of large volumes of data and access to it. Current media storage for large data volumes is typically off line and frequently off site in warehouses. Access to data archived in this fashion can be subject to long delays, errors in media selection and retrieval, and even loss of data through misplacement or damage to the media. Similarly, designers responsible for architecting systems capable of continuous high-speed recording of large volumes of digital data are faced with the challenge of identifying technologies and configurations that meet their requirements. Past approaches have tended to evaluate the combination of the fastest tape recorders with the highest capacity tape media and then to compromise technology selection as a consequence of cost. This paper discusses an architecture that addresses both of these challenges and proposes a cost effective solution based on robots, high speed helical scan tape drives, and large-capacity media.

Rybczynski, Fred↗

Implementation and Evaluation of Active Storage in Modern Parallel File Systems

Active Storage is a technology aimed at reducing the bandwidth requirements of current supercomputing systems, and leveraging the processing power of the storage nodes used by some modern file systems. Active Storage achieves both objectives by moving certain processing tasks to the storage nodes, near the data they manage. Our proposal for Active Storage has several key features: user-space implementation which facilitates the port to different file systems, analytical model to anticipate the performance of Active Storage with respect to a traditional system, support for striped files and complex format files such as netCDF, scientific-friendly programming and runtime environment, etc.

99 GENERAL AND MISCELLANEOUS↗

Challenges and Opportunities of User-Level File Systemsfor HPC

The performance gap between magnetic disks and data processing on HPC systems has become that huge that an efficient data processing can only be achieved by introducing non-volatile memory (NVRAM) as a new storage tier. Although the benefits of hierarchical storage have been adequately demonstrated to the point that the newest leadership class HPC systems will employ burst buffers, critical questions remain for supporting hierarchical storage systems, including: How should we present hierarchical storage systems to user applications, such that they are easy to use and that application code is portable across systems? How should we manage data movement through a storage hierarchy for best performance and resilience of data? How do the particular I/O use cases mandate the way we manage data? There have been many efforts to explore this space in the form of file systems, with increasingly more implemented at the user level. This is because it is relatively easy to swap in new, specialized user-level file systems for use by applications on a case-by-case basis, as opposed to the current mainstream approach of using general-purpose, system-level file systems which may not be optimized for HPC workloads and must be installed by administrators. In contrast, file systems at the user level can be tailored for specific HPC workloads for high performance and can be used by applications without administrator intervention. Many user-level file system developers have found themselves “having to reinvent the wheel” to implement various optimizations in their file systems. Thus, a main goal of this meeting was to bring together experts in I/O performance, file systems, and storage, and collectively explore the space of current and future problems and solutions for I/O on hierarchical storage systems in order to begin a community effort in enabling user-level file system support for HPC systems. We had a lively week of learning about each other’s approaches as well as unique I/O use cases that can influence the design of a community-driven file and storage system standards. The agenda for this meeting contained talks from participants on the following high level topics: HPC storage and I/O support today; what do HPC users need for I/O; existing userlevel file system efforts; object stores and other alternative storage systems; and components for building user-level file systems. The talks were short and intended to be conversation starters for more in-depth discussions with the whole group. The participants engaged in lengthy discussions on various questions that arose from the talks including: Are we ready to program to a memory hierarchy versus block devices? Are the needs of HPC users reflected in our existing file systems and storage systems? Should we drop or keep POSIX moving forward? What do we mean when we say “user-level file system”? Do we all mean the same thing? How should the IO 500 benchmark be defined so it is fair and useful? and How are stage-in and stage-out actually going to work? The report for this seminar contains a record of the talks from the participants as well as the resulting discussions. Our hope is that the effort initiated during this seminar will result in long-term collaborations that will benefit the HPC community as a whole.

97 MATHEMATICS AND COMPUTING↗

Managing high-bandwidth real-time data storage

There exist certain systems which generate real-time data at high bandwidth, but do not necessarily require the long-term retention of that data in normal conditions. In some cases, the data may not actually be useful, and in others, there may be too much data to permanently retain in long-term storage whether it is useful or not. However, certain portions of the data may be identified as being vitally important from time to time, and must therefore be retained for further analysis or permanent storage without interrupting the ongoing collection of new data. We have developed a system, Mahanaxar, intended to address this problem. It provides quality of service guarantees for incoming real-time data streams and simultaneous access to already-recorded data on a best-effort basis utilizing any spare bandwidth. It has built in mechanisms for reliability and indexing, can scale upwards to meet increasing bandwidth requirements, and handles both small and large data elements equally well. We will show that a prototype version of this system provides better performance than a flat file (traditional filesystem) based version, particularly with regard to quality of service guarantees and hard real-time requirements.

99 GENERAL AND MISCELLANEOUS↗

Giovanni in the Cloud: Earth Science Data Exploration in Amazon Web Services

Giovanni is an exploration tool at the NASA Goddard Earth Sciences Data Information Services Center (GES DISC), providing 22 analysis and visualization services for over 1600 Earth Science data variables. Owing to its popularity, Giovanni has experienced a consistent growth in overall demand, with periodic usage spikes attributed to trainings by education organizations, extensive data analysis in response to natural disasters, preparations for science meetings, etc. Furthermore, the new generation of spaceborne sensors and high resolution models have resulted in an exponential growth in data volume with data distributed across the traditional boundaries of data centers. Seamless exploration of data (without users having to worry about data center boundaries) has been a key recommendation of the GES DISC User Working Group. These factors have required new strategies for delivering acceptable performance. The cloud-based Giovanni, built on Amazon Web Services (AWS), evaluates (1) AWS native solutions to provide a scalable, serverless architecture; (2) open standards for data storage in the Cloud; (3) a cost model for operations; and (4) end-user performance. Our preliminary findings indicate that the use of serverless architecture has a potential to significantly reduce development and operational cost of Giovanni. The combination of using AWS managed services, storage of data in open standards, and schema-on-read data access strategy simplifies data access and analytics, in addition to making data more accessible to the end users of Giovanni through popular programming languages.

Giovanni↗

Geospatial Data Platform for All

Spatiotemporal data has evolved in scale due to augmented use in cross-domain applications. Simultaneously, there is substantial growth in the availability of Geographic Information Systems (GIS) data provided by the United States Geological Survey (USGS) along with other federal, state, county, or local agencies through open-data portals and public access APIs. However, data availability does not equate with accessibility. Large-scale analyses and applications require robust, performant data management with co-location of data storage and computing. The insufficiency of data management infrastructure compels researchers to adopt ad hoc project- specific GIS data storage solutions (e.g., copying data to High-Performance computer file systems). As an ad hoc storage strategy does not scale, it hampers cross-domain analyses causing difficulty in data reuse and utilizing existing code bases. Furthermore, GIS data is complex and requires expertise to analyze and manipulate due to its intricate data structures and data-specific projection transformations. Despite the challenges, we recognize that derived GIS data products, e.g., satellite or LIDAR-based images, can be used in downstream applications such as AI by domain, but non-GIS experts. To address the data needs and overcome the challenges, we are working towards a GIS Data Platform focused on efficient data storage, data discovery and access, and an API to enable common workflows. We propose a knowledge-graph (KG) approach for data discovery, whereby datasets are semantically linked to higher- level constructs such as projects and research areas. The semantic data links enable researchers to explore datasets in a top-down approach by specifying relevant and meaningful terms (assists in finding hidden data). An advantage is that the nodes and edges in a knowledge graph create built-in semantic documentation. Deeper spatiotemporal connections between data sources can be encoded via Graph Neural Networks (GNN) (Zhang et al., 2021). The KG approach can be extended to integrate the data itself in a Virtual KG (VKG). Our work will derive inspiration from large-scale VKG efforts that have been undertaken or are currently underway as part of the OpenStreetMap project (Ding et al., 2021). For DOE Data Days, we share the proposed geospatial data platform hybrid (cloud/on-prem) architecture, our work-to-date on storing, retrieving, and transforming LiDAR and raster data relevant to two important NREL use-cases, including the Renewable Energy Potential (reV) Model, and present our proposal for a KG based data discovery engine.

data platform↗

Region templates: Data representation and management for high-throughput image analysis

Compartment fires exhibit unique features associated with smoke accumulation and restricted air ventilation. The objective of the present study is to evaluate the ability of CFD-based fire models to simulate these features and in particular the effects of flame extinction that results from under-ventilated fire conditions. The study is performed using FireFOAM; FireFOAM is an advanced fire modeling software developed by FM Global and is based on a general-purpose open-source software called OpenFOAM. A new flame extinction model based on the concept of a critical value of the flame Damköhler number is incorporated into FireFOAM. The performance of the extinction model is evaluated via comparisons with a previously developed experimental database corresponding to a reduced-scale, heptane-fueled, compartment fire configuration. The numerical simulations also include a description of fuel evaporation driven by the computed gas-to-liquid heat feedback. Comparisons between experimental data and numerical results provide a suitable test bed to evaluate the ability of CFD-based fire models to describe the transition from over- to under-ventilated fire conditions, as well as the transition from extinction-free conditions to conditions in which the flame experiences partial or total quenching.

Energy & Fuels↗

A Conditions Data Management System for HEP Experiments

Conditions data infrastructure for both ATLAS and CMS have to deal with the management of several Terabytes of data. Distributed computing access to this data requires particular care and attention to manage request-rates of up to several tens of kHz. Thanks to the large overlap in use cases and requirements, ATLAS and CMS have worked towards a common solution for conditions data management with the aim of using this design for data-taking in Run 3. In the meantime other experiments, including NA62, have expressed an interest in this cross- experiment initiative. For experiments with a smaller payload volume and complexity, there is particular interest in simplifying the payload storage. The conditions data management model is implemented in a small set of relational database tables. A prototype access toolkit consisting of an intermediate web server has been implemented, using standard technologies available in the Java community. Access is provided through a set of REST services for which the API has been described in a generic way using standard Open API specications, implemented in Swagger. Such a solution allows the automatic generation of client code and server stubs and further allows changes in the backend technology transparently. An important advantage of using a REST API for conditions access is the possibility of caching identical URLs, addressing one of the biggest challenges that large distributed computing solutions impose on conditions data access, avoiding direct DB access by means of standard web proxy solutions.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

NASA Delay Tolerant Networks: Operational, Evolving, an Ready for Expansion

The future of humanity’s presence beyond Earth depends on the successful commercialization of space. For commercialization to succeed, companies need cost-efficient architectures to support their business models and minimize risks for human capital, design, development, and operations. An ongoing challenge to any space enterprise is the reality that terrestrial network technologies are insufficient to provide reliable communications between assets in space. Whether you need to ensure your valuable data is safely transmitted to the ground or reliably delivered between platforms in orbit, ensuring data integrity over intermittent communication links is a necessity. Current solutions to space communications rely heavily on manual recording, storing, and retrieval of data from spacecraft. The current standard in space communication protocols, Consultative Committee for Space Data Systems (CCSDS) Space Packet standard, is reliant on inflexible network architectures based around mission-critical infrastructure to ensure data delivery. However, by automating the recording, storing, retrieval, and verification of data with Delay Tolerant Networks (DTN), the operator is freed from the dependence on manual data management and expensive mission critical infrastructure. NASA has been developing delay tolerant systems since the late 1990’s. Multiple DTN implementations have been established during that time, each suited to different use cases. Most notably, the DTN deployment for the International Space Station (ISS) includes demonstration of two DTN technologies: Interplanetary Overlay Network (ION) and Delay Tolerant Network Marshall Enterprise (DTNME). Beyond ISS, there are even more NASA DTN deployments being considered. Now that DTN implementations are maturing, it is appropriate to reflect upon these decades of work, review the integration and performance of the existing ISS deployment, and explore the future possibilities for DTN deployment industry-wide. The ISS DTN deployment is a complex architecture consisting of different DTN implementations for the onboard and ground network environments. The ION DTN implementation is being used in the on-board network. The Huntsville Operations Support Center (HOSC) DTN implementation, DTNME, is used by the ground network supporting ISS and will soon be a second onboard gateway too. The two implementations work cooperatively to provide high fidelity data services to flight operations users and payload developers across the globe. Though the two implementations yield a quality service, limitations are evident. Data rate, data storage, and device management are constrained by the services themselves and the complex nature of the deployment. Evolution of operations concepts will improve system capabilities and stability, but significant improvement will require additional development to the implementations themselves and to the overall deployment architecture. Taking advantage of the ongoing development and operation of the ISS DTN service will be central to the success of the future evolutions of NASA DTN deployments while demonstrating the benefits of DTN’s low-cost reliable data communication protocols for the growing commercial space industry. A broad effort on DTN integration and support is necessary to promote expansion beyond existing applications. NASA is developing several useful DTN implementations across a number of different systems: ION, DTNME, High-Rate DTN (HDTN), Bundle Protocol Library (BPLib), and others. To prevent fragmentation, DTN implementation teams need to communicate, collaborate, and integrate with one another to build a solid operational foundation for new DTN deployments. The establishment of a group that can assist new DTN users with understanding the purpose of each DTN implementation, provide best practices, and serve as a general knowledge base is paramount. Potential use of DTN on Gateway and other future NASA missions further drives the need for streamlined communication between DTN implementation teams. A well-integrated and highly engaged NASA DTN working group should help provide system architects the best DTN solutions for future commercial space efforts. This paper will first review the history of DTN implementations, explore the shortcoming of current space networking solutions given available limits in technology, and therefore establish the need for Delay Tolerant Networking in space communications. Secondly, the authors will explore NASA’s array of DTN implementations and highlight their usefulness to space applications. Thirdly, this paper will establish general DTN implementation distinguishing factors. Fourthly, the authors will discuss attempts to create a generic DTN comparison matrix, and the authors will review potential future topics in DTN innovation and collaboration, highlighting several key future efforts. Finally, this paper will describe how the institution of a NASA DTN Working Group will benefit DTN adoption across the governmental and commercial space sector. The goal of this paper is to encourage enthusiasm for DTN, share strategies for improving DTN on both current and future applications, promote the collaboration of DTN implementation groups within the international space operations community, and open the conversations about DTN, priorities, complexities, and innovation to the wider spaceflight industry.

DTN↗

NASA Delay Tolerant Networks: Operational, Evolving, an Ready for Expansion

The future of humanity’s presence beyond Earth depends on the successful commercialization of space. For commercialization to succeed, companies need cost-efficient architectures to support their business models and minimize risks for human capital, design, development, and operations. An ongoing challenge to any space enterprise is the reality that terrestrial network technologies are insufficient to provide reliable communications between assets in space. Whether you need to ensure your valuable data is safely transmitted to the ground or reliably delivered between platforms in orbit, ensuring data integrity over intermittent communication links is a necessity. Current solutions to space communications rely heavily on manual recording, storing, and retrieval of data from spacecraft. The current standard in space communication protocols, Consultative Committee for Space Data Systems (CCSDS) Space Packet standard, is reliant on inflexible network architectures based around mission-critical infrastructure to ensure data delivery. However, by automating the recording, storing, retrieval, and verification of data with Delay Tolerant Networks (DTN), the operator is freed from the dependence on manual data management and expensive mission critical infrastructure. NASA has been developing delay tolerant systems since the late 1990’s. Multiple DTN implementations have been established during that time, each suited to different use cases. Most notably, the DTN deployment for the International Space Station (ISS) includes demonstration of two DTN technologies: Interplanetary Overlay Network (ION) and Delay Tolerant Network Marshall Enterprise (DTNME). Beyond ISS, there are even more NASA DTN deployments being considered. Now that DTN implementations are maturing, it is appropriate to reflect upon these decades of work, review the integration and performance of the existing ISS deployment, and explore the future possibilities for DTN deployment industry-wide. The ISS DTN deployment is a complex architecture consisting of different DTN implementations for the onboard and ground network environments. The ION DTN implementation is being used in the on-board network. The Huntsville Operations Support Center (HOSC) DTN implementation, DTNME, is used by the ground network supporting ISS and will soon be a second onboard gateway too. The two implementations work cooperatively to provide high fidelity data services to flight operations users and payload developers across the globe. Though the two implementations yield a quality service, limitations are evident. Data rate, data storage, and device management are constrained by the services themselves and the complex nature of the deployment. Evolution of operations concepts will improve system capabilities and stability, but significant improvement will require additional development to the implementations themselves and to the overall deployment architecture. Taking advantage of the ongoing development and operation of the ISS DTN service will be central to the success of the future evolutions of NASA DTN deployments while demonstrating the benefits of DTN’s low-cost reliable data communication protocols for the growing commercial space industry. A broad effort on DTN integration and support is necessary to promote expansion beyond existing applications. NASA is developing several useful DTN implementations across a number of different systems: ION, DTNME, High-Rate DTN (HDTN), Bundle Protocol Library (BPLib), and others. To prevent fragmentation, DTN implementation teams need to communicate, collaborate, and integrate with one another to build a solid operational foundation for new DTN deployments. The establishment of a group that can assist new DTN users with understanding the purpose of each DTN implementation, provide best practices, and serve as a general knowledge base is paramount. Potential use of DTN on Gateway and other future NASA missions further drives the need for streamlined communication between DTN implementation teams. A well-integrated and highly engaged NASA DTN working group should help provide system architects the best DTN solutions for future commercial space efforts. This paper will first review the history of DTN implementations, explore the shortcoming of current space networking solutions given available limits in technology, and therefore establish the need for Delay Tolerant Networking in space communications. Secondly, the authors will explore NASA’s array of DTN implementations and highlight their usefulness to space applications. Thirdly, this paper will establish general DTN implementation distinguishing factors. Fourthly, the authors will discuss attempts to create a generic DTN comparison matrix, and the authors will review potential future topics in DTN innovation and collaboration, highlighting several key future efforts. Finally, this paper will describe how the institution of a NASA DTN Working Group will benefit DTN adoption across the governmental and commercial space sector. The goal of this paper is to encourage enthusiasm for DTN, share strategies for improving DTN on both current and future applications, promote the collaboration of DTN implementation groups within the international space operations community, and open the conversations about DTN, priorities, complexities, and innovation to the wider spaceflight industry.

DTN↗

Transportation Operations Model (TOM) Technical Manual

The Transportation Operations Model (TOM) is a component of the Transportation Storage Logistics (TSL) model, which assists the analyst in planning and evaluating scenarios for the transportation and storage of spent nuclear fuel (SNF). TOM was originally developed for the DOE’s Office of Civilian Radioactive Waste Management (OCRWM) in 2006. TOM provides detailed modeling of the transportation of spent nuclear fuel and the assets needed to accomplish that. This Transportation Operations Model (TOM) Technical Manual presents the theory and algorithm behind the model. Further information on the Transportation Storage Logistics model is available in the Transportation Storage Logistics Users’ Manual, and in the Transportation Storage Logistics Model Data Management Manual.

Busch, Ingrid Karin↗

Scalable training of graph convolutional neural networks for fast and accurate predictions of HOMO-LUMO gap in molecules

Abstract Graph Convolutional Neural Network (GCNN) is a popular class of deep learning (DL) models in material science to predict material properties from the graph representation of molecular structures. Training an accurate and comprehensive GCNN surrogate for molecular design requires large-scale graph datasets and is usually a time-consuming process. Recent advances in GPUs and distributed computing open a path to reduce the computational cost for GCNN training effectively. However, efficient utilization of high performance computing (HPC) resources for training requires simultaneously optimizing large-scale data management and scalable stochastic batched optimization techniques. In this work, we focus on building GCNN models on HPC systems to predict material properties of millions of molecules. We use HydraGNN, our in-house library for large-scale GCNN training, leveraging distributed data parallelism in PyTorch. We use ADIOS, a high-performance data management framework for efficient storage and reading of large molecular graph data. We perform parallel training on two open-source large-scale graph datasets to build a GCNN predictor for an important quantum property known as the HOMO-LUMO gap. We measure the scalability, accuracy, and convergence of our approach on two DOE supercomputers: the Summit supercomputer at the Oak Ridge Leadership Computing Facility (OLCF) and the Perlmutter system at the National Energy Research Scientific Computing Center (NERSC). We present our experimental results with HydraGNN showing (i) reduction of data loading time up to 4.2 times compared with a conventional method and (ii) linear scaling performance for training up to 1024 GPUs on both Summit and Perlmutter.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

MetaCat - metadata catalog for data management systems

Metadata management is one of three major areas and parts of functionality of scientific data management along with replica management and workflow management. Metadata is the information describing the data stored in a data item, a file or an object. It includes the data item provenance, recording conditions, format and other attributes. MetaCat is a metadata management database designed and developed for High Energy Physics experiments. As a component of a data management system, it’s main objectives are to provide efficient metadata storage and management and fast data items selection functionality. MetaCat is supposed to work on the scale of 100 million files (or objects) and beyond. The article will discuss the functionality of MetaCat and technological solutions used to implement the product.

Mandrichenko, Igor↗

Smart Optical RAM for Fast Information Management and Analysis

Statement of Problem Instruments for high speed and high capacity in-situ data identification, classification and storage capabilities are needed by NASA for the information management and analysis of extremely large volume of data sets in future space exploration, space habitation and utilization, in addition to the various missions to planet-earth programs. Parameters such as communication delays, limited resources, and inaccessibility of human manipulation require more intelligent, compact, low power, and light weight information management and data storage techniques. New and innovative algorithms and architecture using photonics will enable us to meet these challenges. The technology has applications for other government and public agencies.

Liu, Hua-Kuang↗

Adoption of a token-based authentication model for the CMS Submission Infrastructure

The CMS Submission Infrastructure (SI) is the main computing resource provisioning system for CMS workloads. A number of HTCondor pools are employed to manage this infrastructure, which aggregates geographically distributed resources from the WLCG and other providers. Historically, the model of authentication among the diverse components of this infrastructure has relied on the Grid Security Infrastructure (GSI), based on identities and X509 certificates. In contrast, commonly used modern authentication standards are based on capabilities and tokens. The WLCG has identified this trend and aims at a transparent replacement of GSI for all its workload management, data transfer and storage access operations, to be completed during the current LHC Run 3. As part of this effort, and within the context of CMS computing, the Submission Infrastructure group is in the process of phasing out the GSI part of its authentication layers, in favor of IDTokens and Scitokens. The use of tokens is already well integrated into the HTCondor Software Suite, which has allowed us to fully migrate the authentication between internal components of SI. Additionally, recent versions of the HTCondor-CE support tokens as well, enabling CMS resource requests to Grid sites employing this CE technology to be granted by means of token exchange. After a rollout campaign to sites, successfully completed by the third quarter of 2022, the totality of HTCondor CEs in use by CMS are already receiving Scitoken-based pilot jobs. On the ARC CE side, a parallel campaign was launched to foster the adoption of the REST interface at CMS sites (required to enable token-based job submission via HTCondor-G), which is nearing completion as well. In this contribution, the newly adopted authentication model will be described. We will then report on the migration status and final steps towards complete GSI phase out in the CMS SI.

Pérez-Calero Yzquierdo, Antonio↗

Near-field radiative thermal transport: From theory to experiment

Radiative thermal transport via the fluctuating electromagnetic near-field has recently attracted increasing attention due to its fundamental importance and its impact on a range of applications from data storage to thermal management and energy conversion. After a brief historical account of radiative thermal transport, we summarize the basics of fluctuational electrodynamics, a theoretical framework for the study of radiative heat transfer in terms of thermally excited propagating and evanescent electromagnetic waves. Various approaches to modeling near-field thermal transport are briefly discussed, together with key results and proposals for manipulation and utilization of radiative heat flow. Subsequently, we review the experimental advances in the characterization of both near-field heat flow and energy density. We conclude with remarks on the opportunities and challenges for future explorations of radiative heat transfer at the nanoscale.

36 MATERIALS SCIENCE↗