Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “DATA BASE MANAGEMENT”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 613 records · Page 34
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

The future low-temperature geochemical data-scape as envisioned by the U.S. geochemical community

Data sharing benefits the researcher, the scientific community, and the public by allowing the impact of data to be generalized beyond one project and by making science more transparent. However, many scientific communities have not developed protocols or standards for publishing, citing, and versioning datasets. One community that lags in data management is that of low-temperature geochemistry (LTG). This paper resulted from an initiative from 2018 through 2020 to convene LTG and data scientists in the U.S. to strategize future management of LTG data. Through webinars, a workshop, a preprint, a townhall, and a community survey, the group of U.S. scientists discussed the landscape of data management for LTG – the data-scape. Currently this data-scape includes a “street bazaar” of data repositories. This was deemed appropriate in the same way that LTG scientists publish articles in many journals. The variety of data repositories and journals reflect that LTG scientists target many different scientific questions, produce data with extremely different structures and volumes, and utilize copious and complex metadata. Nonetheless, the group agreed that publication of LTG science must be accompanied by sharing of data in publicly accessible repositories, and, for sample-based data, registration of samples with globally unique persistent identifiers. LTG scientists should use certified data repositories that are either highly structured databases designed for specialized types of data, or unstructured generalized data systems. Recognizing the need for tools to enable search and cross-referencing across the proliferating data repositories, the group proposed that the overall data informatics paradigm in LTG should shift from “build data repository, data will come” to “publish data online, cybertools will find”. Funding agencies could also provide portals for LTG scientists to register funded projects and datasets, and forge approaches that cross national boundaries. Finally, the needed transformation of the LTG data culture requires emphasis in student education on science and management of data.

58 GEOSCIENCES↗
DOE OSTI2024

Ultrahigh-resolution mass spectrometry data associated with the manuscript “A functional microbiome catalog crowdsourced from North American rivers"

This data package is associated with the publication “A functional microbiome catalog crowdsourced from North American rivers” submitted to Nature (Borton et al., 2024); (https://www.biorxiv.org/content/10.1101/2023.07.22.550117v1). Predicting elemental cycles and maintaining water quality under increasing anthropogenic influence requires understanding the spatial drivers of river microbiomes. However, the unifying microbial determinants governing river biogeochemistry are hindered by a lack of genome-resolved functional insights and sampling across multiple rivers. Here we employed a community science effort to accelerate the sampling of river microbiomes to create the Genome Resolved Open Watersheds database (GROWdb). GROWdb is a publicly available resource that paves the way for watershed predictive modeling and microbiome-based management practices. This resource profiled the identity, distribution, function, and expression of thousands of microbial genomes across rivers covering 90% of United States watersheds. We identified the most cosmopolitan microbiome members, while also revealing local drivers of strain endemism across ecological dimensions. We provide the first evidence that microbial functional trait expression followed the tenets of the River Continuum Concept, suggesting the structure and function of river microbiomes is predictable. The Fourier-transform ion cyclotron resonance mass spectrometry (FTICR-MS) data were one of many different data types used in establishing the ecological dimensions along which different microbes were detected .This data package only contains the processed FTICR-MS data associated with this manuscript; all other data is accessible via Zenodo (https://zenodo.org/records/8173287), GitHub (https://github.com/jmikayla1991/Genome-Resolved-Open-Watersheds-database-GROWdb), KBase (https://doi.org/10.25982/109073.30/1895615), and NCBI via Bioproject PRJNA946291.This dataset consists of (1) a file-level metadata (flmd) file; (2) a data dictionary (dd) file; (3) a readme; (4) three Fourier-transform ion cyclotron resonance mass spectrometry (FTICR-MS) processed data files (a ‘data’ file containing peak-by-sample observations, a ‘mol’ file containing peak metadata, and a transformation profile containing transformation-by-sample observations). All files are .csv or .pdf.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Infrastructure-Based Cooperative Perception at a Traffic Intersection: Overview and Challenges

Traffic intersections are crucial and challenging nodes in transportation networks where multiple lanes of vehicles and pedestrians converge. About one-quarter of traffic fatalities and about one-half of all traffic injuries in the United States happen at traffic intersections . Effective management of these intersections is important to ensure safety and efficiency of all users - vehicles, pedestrians, cyclists, and vulnerable road users (VRUs). With advancements in sensor perception technologies such as radar, light detection and ranging (lidar), and cameras, traffic intersections are developing into dynamic and data-rich environments. By using these data to create a real-time digital twin, we can enable real-time data-driven decision making and a range of applications such as sharing perception information to connected vehicles (CVs) and connected autonomous vehicles (CAVs), safety affirmative signaling, and curb optimizing to improve efficiency and enhance safety.This paper presents an overview of the concept and examines the challenges involved in implementing an infrastructure-based cooperative perception engine at a traffic intersection. In addition to outlining the physical components, this study also addresses important challenges involved in a multi-sensor system. We present results from deploying the National Renewable Energy Laboratory's (NREL's) Infrastructure Perception and Control (IPC) mobile trailer at a traffic intersection in the city of Colorado Springs, Colorado, USA that employed multiple radars and lidars to capture the data. This study provides necessary practical learning for the Cooperative Driving Automation (CDA) and traffic engineering communities for next-generation infrastructure-based cooperative perception that promises improvements in signal control for optimized traffic flow, among other applications, and documents findings for ongoing research and development efforts in other areas.

ADVANCED PROPULSION SYSTEMS,MATHEMATICS AND COMPUT↗
DOE OSTI2021

Broad Distribution of Local Polar States Generates Large Electrothermal Properties in Pb-Free Relaxor Ferroelectrics

Electrothermal energy conversion provides attractive solutions for global energy management, such as energy harvesting from waste heat using pyroelectric energy conversion (PEC) and efficient cooling of portable electronics or data servers using the electrocaloric effect. Relaxor ferroelectrics are attractive for electrothermal energy conversion because of their large pyroelectric coefficients over a wide temperature range. Although Pb-based relaxors are well-known, toxicity concerns have mandated the intense search for Pb-free alternatives. Here, we engineered (Ba,Ca)TiO 3 -based relaxors based on a multisite doping strategy, which show promising electrothermal performance, viz. a maximum PEC efficiency of 14% and electrocaloric refrigeration capacity of 115 J/kg. Using local-scale structural analysis, we provide an atomistic model for large electrothermal properties in the newly designed Pb-free ferroelectrics, whereby a temperature-independent continuous distribution of cation displacement directions creates easy pathways for microscopic polarization reorientation. This research provides key structural insight for future atomic-scale engineering of environmentally sustainable ferroelectrics in energy applications.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2023

High-throughput screening of hypothetical metal-organic frameworks for thermal conductivity

Thermal energy management in metal-organic frameworks (MOFs) is an important, yet often neglected, challenge for many adsorption-based applications such as gas storage and separations. Despite its importance, there is insufficient understanding of the structure-property relationships governing thermal transport in MOFs. To provide a data-driven perspective into these relationships, here we perform large-scale computational screening of thermal conductivity k in MOFs, leveraging classical molecular dynamics simulations and 10,194 hypothetical MOFs created using the ToBaCCo 3.0 code. We found that high thermal conductivity in MOFs is favored by high densities (> 1.0 g cm -3 ), small pores (< 10 Å), and four-connected metal nodes. We also found that 36 MOFs exhibit ultra-low thermal conductivity (< 0.02 W m -1 K -1 ), which is primarily due to having extremely large pores (~65 Å). Furthermore, we discovered six hypothetical MOFs with very high thermal conductivity (>10 Wm -1 K -1 ), the structures of which we describe in additional detail.

36 MATERIALS SCIENCE↗
DOE OSTI2022

DGTile

SAND2022-12898 O DGTile is a lightweight C++17 adaptive mesh library meant to support explicit discontinuous Galerkin applications on high performance computing machines. DGTile uses a block-based adaptive mesh refinement approach, where the underlying mesh data structure is an octree in three dimensions, where each leaf node of the tree represents a Cartesian grid. Over each grid, DGTile provides modal discontinuous Galerkin basis functions to facilitate simulations. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Granzow, Brian↗
DOE OSTI2024

Impact of Vegetation Assimilation on Flash Drought Characteristics across the Continental United States

Predicting and managing the impacts of flash droughts is difficult owing to their rapid onset and intensification. Flash drought monitoring often relies on assessing changes in root-zone soil moisture. However, the lack of widespread soil moisture measurements means that flash drought assessments often use process-based model data like that from the North American Land Data Assimilation System (NLDAS). Such reliance opens flash drought assessment to model biases, particularly from vegetation processes. Here, we examine the influence of vegetation on NLDAS-simulated flash drought characteristics by comparing two experiments covering 1981–2017: open loop (OL), which uses NLDAS surface meteorological forcing to drive a land surface model using prognostic vegetation, and data assimilation (DA), which instead assimilates near-real-time satellite-derived leaf area index (LAI) into the land surface model. The OL simulation consistently underestimates LAI across the United States, causing relatively high soil moisture values. Both experiments produce similar geographic patterns of flash droughts, but OL produces shorter duration events and regional trends in flash drought occurrence that are sometimes opposite to those in DA. Across the Midwest and Southern United States, flash droughts are 4 weeks (about 70%) longer on average in DA than OL. Moreover, across much of the Great Plains, flash drought occurrence has trended upward according to the DA experiment, opposite to the trend in OL. This sensitivity of flash drought to the representation of vegetation suggests that representing plants with greater fidelity could aid in monitoring flash droughts and improve the prediction of flash drought transitions to more persistent and damaging long-term droughts.

Drought↗
DOE OSTI2022

Deployment of Energy Efficiency and Clean Energy on Indians Lands – A Community Solar Project

The Eastern Band of Cherokee Indians (EBCI or Tribe) is a federally recognized Indian Tribe located on ancestral homelands in western North Carolina. The Tribe is comprised of just over 16,000 enrolled members, with an estimated 8,000 – 10,000 residing on tribal lands. The Qualla Boundary is comprised of over 56,000 acres of contiguous trust land adjacent to the Great Smoky Mountain National Park. The EBCI also possess an assortment of scattered parcels of varying sizes across western North Carolina, some in trust status and others in deeded or fee simple. This project was implemented on a trust parcel in Cherokee County, NC. The Tribe was awarded the maximum amount of $\$ $1,000,000 for the installation of a 705.6 kilowatt (kW) solar PV array to supplement the power consumed and reduce annual utility costs at the Harrah’s Cherokee Valley River (HCVR) casino and hotel in Murphy, NC. The casino is a tribal entity of the Tribe. The total cost of the project was $\$ $2.364 million. The system was officially commissioned on April 12, 2019. For the purpose of calculating the amount of power produced and reduction in utility costs, the time frame assessed is May 2019 through April 2020. All power generated was consumed onsite.Based on the first full year of data, the project achieved the estimated 8.3% reduction in utility costs. A total of six casino and tribal staff were trained and received a North American Board of Certified Energy Practitioners (NABCEP) certification, giving our staff the resources and skills to manage and operate the solar PV array. Based on the Tribal match of $\$ $1,377,797 and estimated net annual benefit of $89,000, the estimated return on investment (ROI) is 15.48 years.

14 SOLAR ENERGY↗
DOE OSTI2022

Validation and Parametric Investigations of an Internal Permanent Magnet Motor Using a Lumped Parameter Thermal Model

One of the key challenges for the electric vehicle industry is to develop high-power-density electric motors. Achieving higher power density requires efficient heat removal from inside the motor. In order to improve thermal management, a multiphysics modeling framework that is able to accurately predict the behavior of the motor, while being computationally efficient, is essential. This paper first presents a detailed validation of a lumped parameter thermal network (LPTN) model of an Internal Permanent Magnet synchronous motor within the commercially available MOTOR-CAD modeling environment. The validation is based on temperature comparison with experimental data and with more detailed finite element analysis (FEA). All critical input parameters of the LPTN are considered in detail for each layer of the stator, especially the contact resistances between the impregnation, liner, laminations, and housing. Finally, a sensitivity analysis for each of the critical input parameters is provided. A maximum difference of 4% - for the highest temperature in the slot-winding and the end-winding - was found between the LPTN and the experimental data. Comparing the results from the LPTN and the FEA model, the maximum difference was 2% for the highest temperature in the slot-winding and end-winding. As for the LPTN sensitivity analysis, the thermal parameter with the highest sensitivity was found to be the liner-to-lamination contact resistance.

42 ENGINEERING↗
DOE OSTI2021

Developing an Automated Uncertainty Quantification Tool to Improve Watershed-Scale Predictions of Water and Nutrient Cycling

Managing the flow of water, nutrients, and contaminants in watersheds is vital to addressing pressing issues related to water scarcity, access to clean drinking water, energy production, resilience to natural and anthropogenic perturbations, and ecological restoration. Decisions about the management of watersheds critically depend on the accuracy with which the flow of water and chemicals through the watershed can be predicted by computer models. Prediction uncertainty can be reduced by matching the model to data, which are collected in the field at great expense. The contribution of watershed characterization data to reducing uncertainty of relevant model predictions can be evaluated in a so-called data-worth analysis, which provides transparent, quantitative metrics about a data set’s value for the support of relevant watershed management objectives. To achieve this goal, we developed a software package that implements the data-worth analysis approach for use with state-of-the-art watershed models. The purpose of the proposed data-worth analysis is to help decision-makers allocate resources for watershed characterization such that the uncertainty in model predictions can be significantly reduced, which leads to better, more effective management decisions. At the same time, watershed characterization costs can be reduced. The specific technical objectives of this SBIR/STTR Phase II project were to develop a framework and associated software toolsets that implement the uncertainty quantification and data-worth analysis approach for use with state-of-the-art watershed models. This goal was achieved by (A) developing a user-friendly, robust software package that is accessible to a wide audience, including watershed managers, policy-makers, and public stakeholders; (B) by demonstrating application of the prototype on several use cases that are representative of complex watershed management challenges spanning a range of scales and that consider different open-source, DOE-based codes and other modeling platforms; and (C) by gathering information about the needs and requirements from potential users to help guide future developments, ensuring that the final product will be commercially viable. The developed software consists of a graphical user interface that guides the user through a sequence of analysis steps, supported by toolsets that leverage state-of-the-art computational simulation-optimization capabilities. A prototype of the software runs on multiple platforms (PC, Mac, multi-processor Linux environment), is linked to diverse watershed simulators (e.g., ECOSYS, TOUGH2, TOUGHREACT, Amanzi-ATS), performs multiple analysis tasks (predictive simulations, sensitivity analysis, uncertainty analysis, automatic parameter estimation, and data-worth analysis, multicomponent geothermometry), and is readily extensible to include external simulators and analysis tools. The software is being commercialized and will be continually updated to address user needs.

58 GEOSCIENCES↗
DOE OSTI2026

Data for Immediate Impacts of Soybean Cover Crop on Bacterial Community Composition and Diversity in Soil Under Long-Term Saccharum Monoculture

Saccharum yield decline results from long-term monoculture practices. Changes in cropping management can improve soil health and productivity. Below-ground bacterial community diversity and composition across soybean (Glycine max (L.) Merr) cover crop, Saccharum monoculture (30+ year) and fallowed soil were determined. Near full length (~1,400 base pairs) of 16S rRNA gene sequences were extracted from the rhizospheres of sugarcane and soybean and fallowed soil were compared. Higher soil bacterial diversity was observed in the soybean cover crop than sugarcane monoculture across all measured indices (observed operationational taxonomic units, Chao1, Shannon, reciprocal Simpson and Jackknife). Acidocateria, Proteobacteria, Bacteroidetes and Planctomycetes were the most abundant bacterial phyla across the treatments. Indicator species analysis identified nine indicator phyla. Planctomycetes, Armatimonadetes and candidate phylum FBP were associated with soybean; Proteobacteria and Firmicutes were linked with sugarcane and Gemmatimonadetes, Nitrospirae, Rokubacteria and unclassified bacteria were associated with fallowed soil. Non-metric multidimensional scaling analysis showed distinct groupings of bacterial operational taxonomic units (97% identity) according to management system (soybean, sugarcane or fallow) indicating compositional differences among treatments. This is confirmed by the results of the multi-response permutation procedures (A = 0.541, p = 0.00045716). No correlation between soil parameters and bacterial community structure was observed according to Mantel test (r = 211865, p = 0.14). Use of soybean cover-crop fostered bacterial diversity and altered community structure. This indicates cover crops could have a restorative effect and potentially promote sustainability in long-term Saccharum production systems.

Field Data↗
DOE OSTI2020

Analysis of Air-Purifying Respirator (APR) and Powered Air-Purifying Respirator (PAPR) Cartridge Performance Testing on a Hanford AX Tank Farm Exhauster Slipstream Volume 2: Raw Analytical Data

As the Tank Operations Contractor for U.S. Department of Energy operations at the Hanford site in Washington State, Washington River Protection Solutions (WRPS) is responsible for managing highly radioactive wastes stored in tanks at Hanford. To protect workers at Hanford Tank Farms, WRPS tests air-purifying respirator (APR) and powered air-purifying respirator (PAPR) chemical cartridges commonly used at the tank farms. The tests were conducted to determine the period of time the cartridges would provide adequate performance for APRs and PAPRs when workers are exposed to a mixture of Chemicals of Potential Concern (COPC) from any vapors exiting headspaces in the storage tanks. Occupational Safety and Health Administration (OSHA) Standard 29 Code of the Federal Regulations (CFR) 1910.134(d)(3)(iii)(b)(2) specifies that for protection against gases and vapors, employers shall implement a schedule for cartridges to ensure that change-outs occur before the end of service life. The change schedule can be based on objective information or data that ensures cartridge change-outs occur before the end of their service life. The primary function of the WRPS Cartridge Test Program is to obtain objective data to determine service lives for the APR and PAPR cartridges used at Hanford Tank Farms. WRPS contracted with Pacific Northwest National Laboratory to analyze the test data and offer an independent analysis and any recommendations. This report summarizes data analysis of APR and PAPR cartridge testing on a vapor slipstream from the Hanford AX tank farm exhauster. Volume 1 of this report documents the testing, data analysis, results, conclusions, and recommendations resulting from the cartridge testing on AX exhauster slipstream vapors. Volume 2 provides an introduction to the raw data, including analytical laboratory analysis results that supported the analysis and conclusions documented in Volume 1.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2025

Complexity Reduction Methods for Large-Scale Spatially Explicit Biofuels Network Design

The size and complexity of energy system optimization models have increased significantly in recent years, driven by the availability of high-resolution spatial data. We present complexity reduction and solution methods that enable us to efficiently represent high-resolution spatial data in the network design of large-scale energy systems. We aim to reduce the size and enhance the computational efficiency of network design models without sacrificing solution accuracy. Specifically, we first present how to aggregate highly granular data into larger resolutions without averaging out their specific properties through a composite-curve-based approach and then develop a method to linearly represent these curves. Second, we utilize a general clustering method to determine groups of geographically proximate biomass fields and establish a single transportation arc for all of them, reducing the number of transportation-related variables while maintaining an accurate representation of the system. Finally, we introduce a two-step algorithm that decomposes large-scale network design problems into two smaller, more manageable subproblems. We demonstrate the application of our methods using a case study of switchgrass-to-biofuels network design in the eight states of the U.S. Midwest, using realistic and highly explicit spatial data.

09 BIOMASS FUELS↗
DOE OSTI2020

Analysis of Air-Purifying Respirator (APR) Cartridge Performance Testing on Hanford Tanks SX-101 and SX-104: Volume 2 - Raw Analytical Data

As the Tank Operations Contractor for U.S. Department of Energy operations at the Hanford site in Washington State, Washington River Protection Solutions (WRPS) is responsible for managing highly radioactive wastes stored in tanks at Hanford. To protect workers at Hanford Tank Farms, WRPS tests air-purifying respirator (APR) and powered air-purifying respirator (PAPR) chemical cartridges commonly used at the tank farms. The tests were conducted to determine the period of time the cartridges would provide adequate performance for APRs and PAPRs when workers are exposed to a mixture of Chemicals of Potential Concern (COPC) from any vapors exiting headspaces in the storage tanks. Occupational Safety and Health Administration (OSHA) Standard 29 Code of the Federal Regulations (CFR) 1910.134(d)(3)(iii)(b)(2) specifies that for protection against gases and vapors, employers shall implement a schedule for cartridges to ensure that change-outs occur before the end of service life. The change schedule can be based on objective information or data that ensures cartridge change-outs occur before the end of their service life. The primary function of the WRPS Cartridge Test Program is to obtain objective data to determine service lives for the APR and PAPR cartridges used at Hanford Tank Farms. WRPS contracted with Pacific Northwest National Laboratory to analyze the test data and offer an independent analysis and any recommendations. This report summarizes data analysis of APR cartridge testing on headspace vapors from Hanford SX-101 and SX-104 single-shell tanks. Volume 1 of this report documents the testing, data analysis, results, conclusions, and recommendations resulting from the APR testing on SX-101 and SX-104 headspace vapors. Volume 2 provides an introduction to the raw data, including analytical laboratory analysis results that supported the analysis and conclusions documented in Volume 1.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2023

Proxy-based Bayesian inversion of strain tensor data measured during well tests

Recent instrument developments have made it possible to measure the strain tensor caused by injecting or pumping fluid from aquifers or reservoirs, but the full value of these data is limited because the long runtimes of poroelastic forward models makes it impractical to use many inversion schemes. This limits the interpretation of strain data for managing the recovery of resources or storage of wastes in the subsurface. This paper describes a method of inverting deformation data using a poroelastic numerical simulator so the results can be used to manage reservoirs or aquifers. We developed a workflow designed to reduce the number of simulations sufficiently to make it feasible to use DREAMzs, an advanced Bayesian inversion method that translates the uncertainties from different sources into unbiased posterior parameter distributions and uncertainty envelopes around the field data. Using a KNN proxy model for the poroelastic simulator is key to reducing the overall computations, and the workflow includes a strategy for ensuring the proxy model results converge on the results from the simulator. The workflow is tested using an idealized example that verifies the ability to correctly identify parameters and characterize noise used to perturb the data. Field data from an injection test at an oil reservoir near Tulsa, Oklahoma, are also used to evaluate the efficacy of the workflow with a real dataset. The workflow identified 265 history matching solutions out of 1240 total simulation runs (21% acceptance ratio), where the results were used to characterize posterior parameter distribution and evaluate the prediction uncertainty. Furthermore, this workflow is significant because it enables strain tensor, or other geomechanical measurements to be interpreted to guide decision-making during energy and environmental processes in the subsurface.

42 ENGINEERING↗
DOE OSTI2021

Rate-Based Absorption Modeling for Postcombustion CO 2 Capture with Additively Manufactured Structured Packing

Carbon capture using amine-based solvents in an absorption process is a leading candidate for reducing greenhouse gas emissions in industrial flue gas streams. To reduce operating costs and associated parasitic energy of these processes, process intensification utilizing additively manufactured structured packing has emerged as a new technology to manage exothermic reactions during absorption while improving CO 2 capture. A rate-based model framework has been developed for these novel packings that incorporates the mass and heat transfer phenomena for amine-based absorption of CO 2 . The rate-based model framework is first benchmarked using available solubility data and pilot plant data for aqueous monoethanolamine. The model validation shows accurate prediction of both CO 2 equilibrium partial pressures and ion speciation for solubility data as well as CO 2 capture, temperature profile, and solvent composition for pilot plant data in the absorption column. The model framework is then applied toward predicting the CO 2 capture performance of additively manufactured structured packing. Simulations agree with experimental data in predicting the CO 2 capture and the capture performance increase due to cooling within the structured packing device. Finally, advantages of this rate-based model framework are the utilization of correlations that may predict mass transfer and heat transfer coefficients of the packing based on the geometric properties of the device and the implementation of the model framework in open-source programming.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2021

Enabling Scalable and Extensible Memory-mapped Datastores in Userspace

Exascale workloads are expected to incorporate data-intensive processing in close coordination with traditional physics simulations. These emerging scientific, data-analytics and machine learning applications need to access a wide variety of datastores in flat files and structured databases. Programmer productivity is greatly enhanced by mapping datastores into the application process's virtual memory space to provide a unified “in-memory” interface. Currently, memory mapping is provided by system software primarily designed for generality and reliability. However, scalability at high concurrency is a formidable challenge on exascale systems. Also, there is a need for extensibility to support new datastores potentially requiring HPC data transfer services. In this article, we present UMap , a scalable and extensible userspace service for memory-mapping datastores. Furthermore, through decoupled queue management, concurrency aware adaptation, and dynamic load balancing, UMap enables application performance to scale even at high concurrency. We evaluate UMap in data-intensive applications, including sorting, graph traversal, database operations, and metagenomic analytics. Our results show that UMap as a userspace service outperforms an optimized kernel-based service across a wide range of intra-node concurrency by 1.22-1.9 × . We performed two case studies to demonstrate UMap 's extensibility. First, a new datastore residing in remote memory is incorporated into UMap as an application-specific plugin. Second, we present a persistent memory allocator Metall built atop UMap for unified storage/memory.

97 MATHEMATICS AND COMPUTING↗