SEARCH · Engineering Papers

Results for “distributed storage”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3

All sources NASA NTRS DOE OSTI

DOE OSTI2021

Strong and Efficient Consistency with Consistency-aware Durability

We introduce consistency-aware durability or C ad , a new approach to durability in distributed storage that enables strong consistency while delivering high performance. We demonstrate the efficacy of this approach by designing cross-client monotonic reads , a novel and strong consistency property that provides monotonic reads across failures and sessions in leader-based systems; such a property can be particularly beneficial in geo-distributed and edge-computing scenarios. We build O rca , a modified version of ZooKeeper that implements C ad and cross-client monotonic reads. We experimentally show that O rca provides strong consistency while closely matching the performance of weakly consistent ZooKeeper. Compared to strongly consistent ZooKeeper, O rca provides significantly higher throughput (1.8--3.3×) and notably reduces latency, sometimes by an order of magnitude in geo-distributed settings. We also implement C ad in Redis and show that the performance benefits are similar to that of C ad ’s implementation in ZooKeeper.

Computer Science↗

DOE OSTI2023

Safety of Mobile Hydrogen and Fuel Cell Technology Applications: An Investigation by the Hydrogen Safety Panel

Safe practices in the production, storage, distribution, and use of hydrogen are essential for the widespread acceptance of hydrogen and fuel cell technologies. A significant safety incident could damage public perception of hydrogen and fuel cells. Recent incidents involving multi-cylinder hydrogen transport vehicles in the United States have brought attention to the potential impacts of mobile hydrogen storage and transportation. Road transportation of bulk gaseous hydrogen presents unique hazards that can be very different from those for stationary equipment, and new equipment developers may have less experience and expertise than seasoned gas providers. In response to the aforementioned incidents, and in support of hydrogen and fuel cell activities in California specifically, the Hydrogen Safety Panel (HSP) has investigated the safety of mobile hydrogen and fuel cell applications (mobile auxiliary/emergency fuel cell power units, mobile fuelers, multi-cylinder transport vehicles, unmanned aircraft power supplies, and mobile hydrogen generators). The HSP examined the applications, requirements, and performance of mobile applications that are being used extensively outside of California to understand how safety considerations are applied. This report discusses the results of the HSP’s evaluation of hydrogen and fuel cell mobile applications along with recommendations to address relevant safety issues.

08 HYDROGEN↗

DOE OSTI2022

Data-driven offshore CO 2 saline storage assessment methodology

The world produces approximately 50 billion tonnes of greenhouse gases annually. This is measured in CO 2 -equivalents, and geologic CO 2 storage has the potential to advance decarbonization and mitigate greenhouse gas emissions. New technologies to assess offshore carbon storage are needed to address resource, regulatory, and commercial needs. Although most efforts to assess storage resources focus on onshore criteria, offshore reservoirs offer significant storage potential and distinct development challenges. Potential advantages of offshore carbon storage include being further from human population centers and less potential to interact with groundwater. The U.S. Department of Energy's method for evaluating storage capacity in non-oil-bearing saline reservoirs has been enhanced to support assessments for offshore environments in the Offshore CO 2 Saline Storage methodology (OCSS). This methodology applies data-driven capabilities to estimate saline storage capacity while accounting for features specific to offshore reservoirs. Features include changes in CO 2 density and sedimentary differences that impact estimates of permanence and capacity. The Offshore CO 2 Saline Storage Calculator mechanizes OCSS to estimate storage capacity. This paper presents the methodology and estimates for 18 geologic domains in the Gulf of Mexico. Potential storage distributions, sensitivity analyses, and the incorporation of spatial data and tools to support safe site selection are also discussed.

58 GEOSCIENCES↗

DOE OSTI2024

MDLoader: A Hybrid Model-Driven Data Loader for Distributed Graph Neural Network Training

Scalable data management is essential for processing large scientific dataset on HPC platforms for distributed deep learning. In-memory distributed storage is preferred for its speed, enabling rapid, random, and frequent data access required by stochastic optimizers. Processes use one-sided or collective communication to fetch remote data, with optimal performance depending on (i) dataset characteristics, (ii) training scale, and (iii) interconnection network. Empirical analysis shows collective communication excels with larger mini-batch sizes and/or fewer processes, whereas one-sided communication outperforms at larger scales. We propose MDLoader, a hybrid in-memory data loader for distributed graph neural network training. MDLoader features a model-driven performance estimator that dynamically selects between one-sided and collective communication at the beginning of training using Tree of Parzen Estimators (TPE). Evaluations on NERSC Perlmutter and OLCF Summit show MDLoader outperforms single-backend loaders by up to 2.83 × and predicts the suitable communication method with 96.3% (Perlmutter) and 94.3% (Summit) success rate.

Bae, Jonghyun↗

DOE OSTI2022

Community-Scale Solar Deployment in the Northwest Arctic

NANA Regional Corporation (NRC, or NANA) was formed as an Alaska Native Corporation (ANC) pursuant to the Alaska Native Claims Settlement Act of 1971. Our lands cover 39,000 square miles of the Northwest Arctic region of Alaska. We partner closely with the Northwest Arctic Borough (NAB) and our 11 remote communities on numerous projects and activities, especially around clean energy initiatives that improve quality of life and help to reduce extremely high energy costs experienced by the communities in our region. Collectively, the regional partnership has developed numerous successful solar, wind, biomass, and energy storage, distribution upgrade, and efficiency projects. To further progress, we created the Northwest Arctic Energy Steering Committee to share and replicate these benefits across the region. NANA’s mission is to provide economic opportunities for our more than 13,500 Iñupiat shareholders and to protect and enhance NANA lands.

14 SOLAR ENERGY↗

DOE OSTI2024

Soil Carbon Stocks Not Linked to Aboveground Litter Input and Chemistry of Old-Growth Forest and Adjacent Prairie

The long-standing assumption that aboveground plant litter inputs have a substantial influence on soil organic carbon storage (SOC) and dynamics has been challenged by a new paradigm for SOC formation and persistence. We tested the importance of plant litter chemistry on SOC storage, distribution, composition, and age by comparing two highly contrasting ecosystems: an old-growth coast redwood (Sequoia sempervirens) forest, with highly aromatic litter, and an adjacent coastal prairie, with more easily decomposed litter. We hypothesized that if plant litter chemistry was the primary driver, redwood would store more and older SOC that was less microbially processed than prairie. Total soil carbon stocks to 110 cm depth were higher in prairie (35 kg C m —2 ) than redwood (28 kg C m —2 ). Radiocarbon values indicated shorter SOC residence times in redwood than prairie throughout the profile. Higher amounts of pyrogenic carbon and a higher degree of microbial processing of SOC appear to be instrumental for soil carbon storage and persistence in prairie, while differences in fine-root carbon inputs likely contribute to younger SOC in redwood. We conclude that at these sites fire residues, root inputs, and soil properties influence soil carbon dynamics to a greater degree than the properties of aboveground litter.

13C-NMR spectroscopy↗

DOE OSTI2022

U.S. national water and energy land dataset for integrated multisector dynamics research

Abstract Understanding resource demands and tradeoffs among energy, water, and land socioeconomic sectors requires an explicit consideration of spatial scale. However, incorporation of land dynamics within the energy-water nexus has been limited due inconsistent spatial units of observation from disparate data sources. Herein we describe the development of a National Water and Energy Land Dataset (NWELD) for the conterminous United States. NWELD is a 30-m, 86-layer rasterized dataset depicting the land use of mappable components of the United States energy sector life cycles (and related water used for energy), specifically the extraction, development, production, storage, distribution, and operation of eight renewable and non-renewable technologies. Through geospatial processing and programming, the final products were assembled using four different methodologies, each depending upon the nature and availability of raw data sources. For validation, NWELD provided a relatively accurate portrayal of the spatial extent of energy life cycles yet displayed low measures of association with mainstream land cover and land use datasets, indicating the provision of new land use information for the energy-water nexus.

58 GEOSCIENCES↗

DOE OSTI2025

dCache project status and update

The dCache project delivers an open-source, massively scalable, distributed storage system deployed internationally to satisfy today’s scientists’ ever-demanding storage requirements. Its multifaceted approach supports different use cases with the same storage, from high throughput data ingest, data sharing over wide area networks, efficient access from HPC clusters, and longterm data persistence on tertiary storage. Even though dCache was initially developed for HEP experiments, today, it is used by various scientific communities, including astrophysics, biomed, and life science, each with their specific requirements. To match the needs of these new communities and keep up with the scaling demands of existing experiments, dCache is permanently evolving. With this contribution, we would like to highlight the recent developments in dCache regarding integration with CERN Tape Archive (CTA), advanced metadata handling, token-based authorization support, bulk API for QoS transitions, REST API to control interaction with the tape system, and future development directions.

Mkrtchyan, Tigran [DESY]↗

DOE OSTI2024

To Derive or Not to Derive: I/O Libraries Take Charge of Derived Quantities Computation

The ever-increasing volume of data produced by HPC simulations necessitates scalable methods for data exploration and knowledge extraction. Scientific data analysis often involves complex queries across distributed datasets, requiring manipulation of multiple primary variables and generating derived data that needs to be handled efficiently, creating challenges for applications that need to parse many large datasets. Relying on individual applications to handle all intermediate data generally leads to redundant computations across studies and unnecessary data transfers. In this paper, we investigate the performance of different approaches where applications define derived variables as quantities of interest (QoIs) and offload the computation and transfer of these QoIs to the I/O library. This significantly reduces redundancy and optimizes data movement across the distributed storage and processing infrastructure by allowing control over when and where derived variables are computed. We present a detailed analysis of the performance-storage trade-offs associated with different solutions and showcase results for our study on two large-scale datasets created from climate and combustion simulations.

Gainaru, Ana↗

DOE OSTI2020

Streaming Data Reorganization at Scale with DeltaFS Indexed Massive Directories

We report complex storage stacks providing data compression, indexing, and analytics help leverage the massive amounts of data generated today to derive insights. It is challenging to perform this computation, however, while fully utilizing the underlying storage media. This is because, while storage servers with large core counts are widely available, single-core performance and memory bandwidth per core grow slower than the core count per die. Computational storage offers a promising solution to this problem by utilizing dedicated compute resources along the storage processing path. We present DeltaFS Indexed Massive Directories (IMDs), a new approach to computational storage. DeltaFS IMDs harvest available (i.e., not dedicated) compute, memory, and network resources on the compute nodes of an application to perform computation on data. We demonstrate the efficiency of DeltaFS IMDs by using them to dynamically reorganize the output of a real-world simulation application across 131,072 CPU cores. DeltaFS IMDs speed up reads by 1,740x while only slightly slowing down the writing of data during simulation I/O for in situ data processing.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

Recent Developments in Sensor Technologies for Enabling the Hydrogen Economy

Efforts to create a sustainable hydrogen economy are gaining momentum as governments all over the world are investing in hydrogen production, storage, distribution, and delivery technologies to develop a hydrogen infrastructure. This involves transporting hydrogen in gaseous or liquid form or using carrier gases such as methane, ammonia, or mixtures of methane and hydrogen. Hydrogen is a colorless, odorless gas and can easily leak into the atmosphere leading to economic loss and safety concerns. Therefore, deployment of robust low-cost sensors for various scenarios involving hydrogen is of paramount importance. Here, we review some recent developments in hydrogen sensors for applications such as leak detection, safety, process monitoring in production, transport and use scenarios. The status of methane and ammonia sensors is covered due to their important role in hydrogen production and transportation using existing natural gas and ammonia infrastructure. This review further provides an overview of existing commercial hydrogen sensors and also addresses the potential for hydrogen as an interferent gas for currently used sensors. This review can help developers and users make informed decisions about how to drive hydrogen sensor technology forward and to incorporate hydrogen sensors into the various hydrogen deployment projects in the coming decade.

08 HYDROGEN↗

DOE OSTI2023

Orchestration of materials science workflows for heterogeneous resources at large scale

In the era of big data, materials science workflows need to handle large-scale data distribution, storage, and computation. Any of these areas can become a performance bottleneck. We present a framework for analyzing internal material structures (e.g., cracks) to mitigate these bottlenecks. We demonstrate the effectiveness of our framework for a workflow performing synchrotron X-ray computed tomography reconstruction and segmentation of a silica-based structure. Our framework provides a cloud-based, cutting-edge solution to challenges such as growing intermediate and output data and heavy resource demands during image reconstruction and segmentation. Specifically, our framework efficiently manages data storage, scaling up compute resources on the cloud. The multi-layer software structure of our framework includes three layers. A top layer uses Jupyter notebooks and serves as the user interface. A middle layer uses Ansible for resource deployment and managing the execution environment. A low layer is dedicated to resource management and provides resource management and job scheduling on heterogeneous nodes (i.e., GPU and CPU). At the core of this layer, Kubernetes supports resource management, and Dask enables large-scale job scheduling for heterogeneous resources. The broader impact of our work is four-fold: through our framework, we hide the complexity of the cloud’s software stack to the user who otherwise is required to have expertise in cloud technologies; we manage job scheduling efficiently and in a scalable manner; we enable resource elasticity and workflow orchestration at a large scale; and we facilitate moving the study of nonporous structures, which has wide applications in engineering and scientific fields, to the cloud. While we demonstrate the capability of our framework for a specific materials science application, it can be adapted for other applications and domains because of its modular, multi-layer architecture.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

A New Capability of E4D For 3D Parallel Joint Inversion of DC Resistivity And Traveltime Data on Unstructured Mesh

A major challenge in interpreting geophysical data is how to derive consistent three-dimensional (3D) earth models of different physical properties from spatially and temporally limited measurements. Joint inversion with cross-gradient constraints is an approach to find such models by imposing structural similarities between different physical parameters. We have developed a parallel distributed-memory joint inversion code for direct-current (DC) resistivity and traveltime data using the cross-gradient constraint on unstructured mesh. The code utilizes existing E4D framework for parallel forward simulation, distributed storage and computation of the Jacobian matrix of forward operator, and parallel execution of matrix-vector multiplication during inversion. Besides, the joint inversion is solved by nonlinear conjugate gradient algorithm parallelized for DC resistivity and traveltime data. The joint inversion capability of E4D was tested using synthetic data from cross-borehole DC resistivity and traveltime data. The results indicate that the shape and size of the anomalies from the joint inversion are more reliable than those from separate inversions.

58 GEOSCIENCES↗

DOE OSTI2023

Recent Developments in Sensor Technologies for Enabling the Hydrogen Economy

08 HYDROGEN↗

DOE OSTI2025

An Nginx-based Content Distribution Network for HEP

With the move to HTTP/WebDAV and JSON Web Tokens as a standard protocol for transfers within the WLCG distributed storage network, a large amount of off-the-shelf technologies become viable for meeting the requirements of a Storage Element (SE). In this work, we explore the capabilities and performance of the OpenResty framework, which extends the nginx server with the LuaJIT scripting language, to recreate the feature set of a SE. We demonstrate token-authenticated HTTP read, write, and WebDAV third-party copy features, as well as a storage federation with HTTP redirect, proxy, and caching capabilities. We further explore the performance scaling in terms of throughput and requests per second.

Aarora, Aashay [UC, San Diego]↗

DOE OSTI2026

Measuring labor productivity dynamics in U.S. industrial and electric power sectors: a case study (2014–2023)

This study proposes a subsystem methodology for measuring labor productivity in the U.S. industrial and electric power sectors by leveraging public data available between 2014 and 2023. Building on Pasinetti’s framework and subsequent developments, the approach employs Vertically Integrated Sectors (VIS) to account for both direct and indirect productivity effects. The novelty of this work is twofold. First, it enables the estimation of productivity trends over time, providing a robust foundation for empirical analysis. Second, it applies the methodology to a case study of the electric generation sector, highlighting its practical relevance. Using data from the Bureau of Economic Analysis, the Bureau of Labor Statistics, and the Impact Analysis for Planning (IMPLAN) tool, the study reveals significant discrepancies between conventional productivity measures and those derived from the VIS approach. Furthermore, the proposed method aligns with the principles of Integrated Energy Systems by capturing the interrelations among generation, distribution, storage, and consumption. This alignment underscores its utility and applications for energy-related policy and planning. Overall, the findings contribute to more precise labor productivity assessments, supporting informed decision-making and future research. Additionally, the method highlights the importance of considering the whole supply chain, providing interrelated metrics for labor productivity which includes both, direct and indirect effects on the final labor productivity metric. By incorporating intersectoral dependencies, this method offers a more comprehensive and accurate measure of labor productivity compared to traditional metrics.

11 - NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

DOE OSTI2021

Analyzing the Economic and Technical Potential of H2@Scale

H2@Scale is one of EERE's flagship initiatives to enable clean and affordable hydrogen production, storage, distribution, and utilization across multiple sectors in the economy. The January H2IQ hour will discuss results from a recent multi-laboratory analysis report quantifying the technical and economic potential of H2@Scale, given the use of hydrogen as an energy intermediate to integrate sectors in the U.S. energy system. Mark Ruth of the National Renewable Energy Laboratory, the lead author of the report, will discuss specific examples of the role hydrogen can play in industry and transportation sectors, such as steelmaking, synthetic fuels, and blending into natural gas pipelines. He will also explain how hydrogen production can be integrated with the electric grid to monetize low-cost energy and incentivize growth in renewable energy supply. The H2IQ Hour will dive into estimates for future hydrogen consumption in current and emerging sectors, given R&D advances and varying prices of natural gas and electricity.

39 EE - Hydrogen and Fuel Cell Technologies (EE-3F↗

DOE OSTI2022

Designing the PR100 Study: Puerto Rico Grid Resilience and Transition to 100% Renewable Energy

Puerto Rico has committed to meeting its electricity needs with 100% renewable energy by 2050, along with realizing interim goals of 40% by 2025, 60% by 2040, the phaseout of coal-fired generation by 2028, and a 30% improvement in energy efficiency by 2040 as established in Act 17. [1] To meet these goals and support widespread end-use electrification, the territory must explore how renewable energy and other generation technologies can be developed with energy storage, distributed generation, distribution control, electric vehicles, and energy efficient and responsive loads in each of Puerto Rico's cities and communities. To support Puerto Rico in reaching its renewable energy goals; help ensure energy system resilience against future extreme weather events; improve energy justice; and provide inputs to LUMA Energy's walk, jog, and run approach in its coordinated planning roadmap, the U.S. Department of Energy, National Renewable Energy Laboratory, Argonne National Laboratory, Lawrence Berkeley National Laboratory, Oak Ridge National Laboratory, Pacific Northwest National Laboratory and Sandia National Laboratories propose to evaluate 100% renewable energy pathways through an integrated analysis process. This talk will present steps in designing a two-year study entitled Puerto Rico Grid Resilience and Transitions to 100% Renewable Energy (PR100). [1] U.S. Energy Information Administration (EIA). 2020. Puerto Rico Territory Energy Profile.

100% renewable energy↗