Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data Variables”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 343 records · Page 19
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

SULI Technical Report: WIRE (Draft)

Local climate undoubtedly affects facility maintenance requirements; however, knowledge of the link between weather and facility condition over time is grounded in the 1990s and needs to be updated. WIRE aims to quantify the relationship between climate and infrastructure by calculating climate-based service life estimates to be used in BUILDER, a program the Department of Energy (DOE) uses for facility management. Phase One of this project is a case study on window-or-wall-mounted AC units. The independent variables influencing service life are load, maintenance quality, and effective age, with load calculated from three weather variables: temperature, humidity, and solar radiation. Given sufficient data, service life can be calculated as a function of these variables via regression. WIRE is currently collecting data. Preliminary estimated service life projections as a function of load derived from Lawrence Livermore National Laboratory (LLNL) data support the need for updated analysis on the relationship between climate and facility condition. With the groundwork laid, WIRE can generate deliverables immediately upon arrival of data. The methods used for this case study constitute a framework for deriving the relationship between weather variables and service life for other components.

42 ENGINEERING↗
DOE OSTI2023

Dayflow: CONUS Daily Streamflow Reanalysis, Version 2 (DayflowV2)

The DayflowV2 dataset provides multiple meteorologic forcings driven hourly streamflow information for approximately 2.7 million NHDPlusV2 stream reaches in the conterminous US (CONUS). DaymetV4, Stage-IV, and Analysis of Period of Record for Calibration (AORC) forcings and their corresponding hybrids drive a nationally scalable modeling framework integrating the simulated runoff from the Variable Infiltration Capacity (VIC) model with the Routing Application for Parallel computatIon of Discharge (RAPID) routing model. Streamflow with (Assimilated) and without (Naturalized) streamflow assimilation at US Geological Survey (USGS) streamflow monitoring sites are included in DayflowV2. A comprehensive evaluation of streamflow at 7,526 USGS gauges is performed for both streamflow types. The resulting key evaluation metrics are also included in the Dayflow dataset. The reanalysis data are available for variable periods; 36 years (1980-2015) for DaymetV4 (DayflowV1), 18 years (2002-2019) for Stage-IV and its hybrids, and 40 years (1980-2019) for AORC and its hybrids.

13 HYDRO ENERGY↗
DOE OSTI2020

A double peak in the seasonality of California's photosynthesis as observed from space

Solar-induced chlorophyll fluorescence (SIF) has been shown to be a powerful proxy for photosynthesis and gross primary productivity (GPP). The recently launched TROPOspheric Monitoring Instrument (TROPOMI) features the required spectral resolution and signal-to-noise ratio to retrieve SIF from space. Here, we present a downscaling method to obtain 500 m spatial resolution SIF over California. We report daily values based on a 14 d window. TROPOMI SIF data show a strong correspondence with daily GPP estimates at AmeriFlux sites across multiple ecosystems in California. We find a linear relationship between SIF and GPP that is largely invariant across ecosystems with an intercept that is not significantly different from zero. Measurements of SIF from TROPOMI agree with MODerate Resolution Imaging Spectroradiometer (MODIS) vegetation indices – the normalized difference vegetation index (NDVI), enhanced vegetation index (EVI), and near-infrared reflectance of vegetation index (NIR$_v$) – at annual timescales but indicate different temporal dynamics at monthly and daily timescales. TROPOMI SIF data show a double peak in the seasonality of photosynthesis, a feature that is not present in the MODIS vegetation indices. The different seasonality in the vegetation indices may be due to a clear-sky bias in the vegetation indices, whereas previous work has shown SIF to have a low sensitivity to clouds and to detect the downregulation of photosynthesis even when plants appear green. We further decompose the spatiotemporal patterns in the SIF data based on land cover. The double peak in the seasonality of California's photosynthesis is due to two processes that are out of phase: grasses, chaparral, and oak savanna ecosystems show an April maximum, while evergreen forests peak in June. An empirical orthogonal function (EOF) analysis corroborates the phase offset and spatial patterns driving the double peak. The EOF analysis further indicates that two spatiotemporal patterns explain 84 % of the variability in the SIF data. Results shown here are promising for obtaining global GPP at sub-kilometer spatial scales and identifying the processes driving carbon uptake.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

A novel conditional generative model for efficient ensemble forecasts of state variables in large-scale geological carbon storage

Integrating monitoring data to efficiently update reservoir pressure and CO 2 plume distribution forecasts presents a significant challenge in geological carbon storage (GCS) applications. Inverse modeling techniques are commonly used to fuse observational data and refine reservoir model parameters, thereby improving state variable forecasts. However, these techniques often rely on linear or Gaussian assumptions, which can limit their effectiveness in accurately predicting state variables. Moreover, simulating large-scale three-dimensional (3D) GCS problems is computationally expensive, making iterative runs in inverse problems prohibitive. To address these challenges, we propose a conditional generative model utilizing the score-based diffusion method for real-time 3D pressure and saturation field distribution predictions. Our approach involves solving the score function with a mini-batch-based Monte Carlo estimator to generate labeled data. This data is subsequently employed to train a fully connected neural network, enabling it to learn the conditional sample generator within a supervised learning framework. This method enables the rapid generation of a large ensemble of predictions, facilitating comprehensive uncertainty quantification of state variables. Here we applied our method to forecast the dynamic 3D distributions of pressure and saturation fields over a 30-year injection period. The statistical assessment with low root mean square error (RMSE) values demonstrates that our method can accurately predict the spatiotemporal distributions of both pressure and saturation fields. Moreover, the developed conditional generative model shows high computational efficiency by generating 100 ensemble forecasts of 3D state variables in less than 10 min. The consistency between ensemble averages and ground truth values further illustrates the model’s capability to capture state variable dynamics during the CO 2 plume injection process. Notably, the ground truth values fall within the ensemble forecasts, indicating that our uncertainty quantification effectively captures variability and potential noise in the observations. Thus, the developed conditional generative model proves to be a more efficient, accurate, and practical tool for GCS applications, facilitating timely risk analysis and informed decision-making.

58 GEOSCIENCES↗
DOE OSTI2022

Improving the LandScan USA Non-Obligate Population Estimate (NOPE)

Where do people go when they have nowhere to be? Nonobligate activities are a significant part of our social and cultural lives, but there are no existing large scale data which characterize spatial variability in population allocation for these activities. As large scale population estimates have ever-finer resolutions, gaps in our ability to estimate this population segment have an increasingly large impact on high resolution population estimates. In this paper, we demonstrate an improved method for estimating the spatial allocation of the non-obligate population - people who are not at work, school, or in another residential institution. This method builds upon on anonymized and aggregate data on visits to public places, allocating the non-obligate population proportionally to worker population while accounting for the estimated ratio of visitors to workers in public places.

Brelsford, Christa↗
DOE OSTI2025

Reproductive and leaf litterfall fluxes in forest ecosystem sites globally (1950-2022)

Forest allocation of net primary productivity (NPP) to reproduction is poorly quantified globally, despite its critical role in forest regeneration and a well-supported trade-off with allocation to growth. Although field measurements of total NPP are rare, our work finds that a proxy for reproductive carbon allocation constructed from leaf (L) and reproductive (R) litterfall fluxes, R/(R+L), is strongly correlated with R/NPP, facilitating analysis across a wide range of sites where biometric estimates of NPP are not available (R² = 0.85; Hanbury-Brown et al., 2022, Ward et al., in prep). To investigate relationships between ecosystem-scale reproductive allocation (RA) and climate, soil fertility, and stand age gradients, we conducted a literature search and synthesized 824 observations of annual average leaf and reproductive litterfall fluxes across forest sites globally. The zip file includes 1) a folder Data/ containing the litterfall data ("GlobalForestRA_data.csv") and metadata ("GlobalForestRA_metadata.doc") files. The data file includes geographic coordinates, long-term mean annual temperature and precipitation (1970-2000, extracted from WorldClim2.1), leaf and reproductive litterfall fluxes, sampling interval and protocols, forest characteristics (dominant leaf morphology, information pertaining to forest age and successional stage, and disturbance history) and soil properties (% sand, %silt, %clay, total phosphorus (P), nitrogen (N), cation exchange capacity (CEC) and pH) extracted from SoilGrids250 and from on-site measurements, where available. The metadata file contains information about each variable reported in the data file, including data sources, processing methods, and all references. The Data folder contains two additional files used to create Figure 1; these are described in greater detail in the README.2) R scripts GloalForestRA_analysis.r and GlobalForestRA_SI.r and a folder /Functions used to produce results, figures, and tables in the manuscript Ward et al. (in press)3) a README file describing how the data and R scripts can be used to reproduce statistical results, figures, and tables found in the manuscript. Ward et al. (in press)This repository can also be found at: https://github.com/r-ward/Global_Analysis_ForestRA.Ward, R.E., Zhang-Zheng, H. Aernethy, K., Adu-Bredu, S., Arroyo, L., Bailey, A. et al. (in press). Forest age rivals climate to explain reproductive allocation patterns in forest ecosystems globally. Ecology Letters. Hanbury-Brown, A.R., Ward, R.E. & Kueppers, L.M. (2022). Forest regeneration within Earth system models: current process representations and ways forward. New Phytol., 235, 20–40.Ward et al. (2025), Forest age rivals climate to explain reproductive allocation patterns in forest ecosystems globally, in prep.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Near-Real-Time Forecast of Satellite-Based Soil Moisture Using Long Short-Term Memory with an Adaptive Data Integration Kernel

Nowcasts, or near-real-time (NRT) forecasts, of soil moisture based on the Soil Moisture Active and Passive (SMAP) mission could provide substantial value for a range of applications including hazards monitoring and agricultural planning. To provide such a NRT forecast with high fidelity, we enhanced a time series deep learning architecture, long short-term memory (LSTM), with a novel data integration (DI) kernel to assimilate the most recent SMAP observations as soon as they become available. The kernel is adaptive in that it can accommodate irregular observational schedules. Testing over the CONUS, this NRT forecast product showcases predictions with unprecedented accuracy when evaluated against subsequent SMAP retrievals. It showed smaller error than NRT forecasts reported in the literature, especially at longer forecast latency. The comparative advantage was due to LSTM’s structural improvements, as well as its ability to utilize more input variables and more training data. The DI-LSTM was compared to the original LSTM model that runs without data integration, referred to as the projection model here. We found that the DI procedure removed the autocorrelated effects of forcing errors and errors due to processes not represented in the inputs, for example, irrigation and floodplain/lake inundation, as well as mismatches due to unseen forcing conditions. The effects of this purely data-driven DI kernel are discussed for the first time in the geosciences. Furthermore, this work presents an upper-bound estimate for the random component of the SMAP retrieval error.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

The Effect of Inverter Loading Ratio on Energy Estimate Bias: Preprint

Subhourly effects, particularly variability in solar irradiance, can lead to underestimation of inverter clipping losses and overestimation of energy in hourly photovoltaic system performance models, particularly for systems with high inverter loading ratios. Direct simulation of this error can be complicated by factors such as the representation of spatial and temporal variability in hourly weather data and transient system conditions. In this work we take an alternative approach using real system power measurements to show that energy predictions from typical industry models suffer from a bias that increases with inverter loading ratio. We also show that this loading ratio-dependent bias is strongly correlated with an empirical subhourly inverter clipping bias derived from real power plant data. Finally, we show that this bias is not necessarily specific to any one model or weather dataset by recreating similar biases with alternatives of each.

clipping↗
DOE OSTI2024

Subdecadal Holocene Warm-Season Temperature Variability in Central Europe Recorded by Biochemical Varves

Paleoclimate data provide important information about the character of natural climate variability. However, records with sufficient length and resolution to resolve high-frequency (decadal-scale) variability across the Holocene are scarce. We present a 10,800-year reconstruction of spring and summer temperature at three-year resolution based on biochemical varves from Lake $\dot{Z}abi$$\acute{n}skie$, Poland. The reconstruction is based on Ca/Ti ratio, which are significantly correlated with instrumental spring and summer temperature spanning 240 years. Major climate events of the Holocene period are represented in the reconstruction, including the Holocene Thermal Maximum, 8.2 ka Event, Medieval Climate Anomaly, and Little Ice Age. A low-frequency 8,000-year decreasing trend in warm-season temperatures is driven by declining summer insolation. Temperature variability is highest during the early Holocene, likely related to warmer and drier conditions. The rate of warming during the past 90 years is extremely unusual, if not unprecedented for the Holocene, based on our reconstruction.

58 GEOSCIENCES↗
DOE OSTI2025

Overview of the SCEC/USGS Community Stress Drop Validation Study Using the 2019 Ridgecrest Earthquake Sequence

We present initial findings from the ongoing Community Stress Drop Validation Study to compare spectral stress-drop estimates for earthquakes in the 2019 Ridgecrest, California, sequence. This study uses a unified dataset to independently estimate earthquake source parameters through various methods. Stress drop, which denotes the change in average shear stress along a fault during earthquake rupture, is a critical parameter in earthquake science, impacting ground motion, rupture simulation, and source physics. Spectral stress drop is commonly derived by fitting the amplitude-spectrum shape, but estimates can vary substantially across studies for individual earthquakes. Sponsored jointly by the U.S. Geological Survey and the Statewide (previously, Southern) California Earthquake Center our community study aims to elucidate sources of variability and uncertainty in earthquake spectral stress-drop estimates through quantitative comparison of submitted results from independent analyses. The dataset includes nearly 13,000 earthquakes ranging from M 1 to 7 during a two-week period of the 2019 Ridgecrest sequence, recorded within a 1° radius. Here, in this article, we report on 56 unique submissions received from 20 different groups, detailing spectral corner frequencies (or source durations), moment magnitudes, and estimated spectral stress drops. Methods employed encompass spectral ratio analysis, spectral decomposition and inversion, finite-fault modeling, ground-motion-based approaches, and combined methods. Initial analysis reveals significant scatter across submitted spectral stress drops spanning over six orders of magnitude. However, we can identify between-method trends and offsets within the data to mitigate this variability. Averaging submissions for a prioritized subset of 56 events shows reduced variability of spectral stress drop, indicating overall consistency in recovered spectral stress-drop values.

58 GEOSCIENCES↗
DOE OSTI2023

Systems and methods for screening particle source manufacturing and development test data

A computing system obtains test data for a particle source. The test data was generated by the particle source when the particle source was caused to emit particles. The test data comprises a first set of measurements of a first type and a second set of measurements of a second type. The computing system applies a data agnostic predictive model to the test data. The data agnostic predictive model is generated without a parametric analysis of variables of the first type and variables of the second type. The data agnostic predictive model outputs, based upon the test data, a value that is indicative of whether or not the test data is abnormal. Based upon the value, the computing system outputs an indication that the particle source was operating sub-optimally when emitting the particles.

Multari, Rosalie A.↗
DOE OSTI2025

Elastic Bayesian Model Calibration

Functional data are ubiquitous in scientific modeling. For instance, quantities of interest are modeled as functions of time, space, energy, density, etc. Uncertainty quantification methods for computer models with functional response have resulted in tools for emulation, sensitivity analysis, and calibration that are widely used. However, many of these tools do not perform well when the computer model’s parameters control both the amplitude variation of the functional output and its alignment (or phase variation). This paper introduces a framework for Bayesian model calibration when the model responses are misaligned functional data. The approach generates two types of data out of the misaligned functional responses: (1) aligned functions so that the amplitude variation is isolated and (2) warping functions that isolate the phase variation. These two types of data are created for the computer simulation data (both of which may be emulated) and the experimental data. The calibration approach uses both types so that it seeks to match both the amplitude and phase of the experimental data. The framework is careful to respect constraints that arise, especially when modeling phase variation, and is framed in a way that it can be done with readily available calibration software. In conclusion, we demonstrate the techniques on two simulated data examples and on two dynamic material science problems: a strength model calibration using flyer plate experiments and an equation of state model calibration using experiments performed on the Sandia National Laboratories’ Z-machine.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

A machine learning approach for clinker quality prediction and nonlinear model predictive control design for a rotary cement kiln

Abstract Cement manufacturing is energy‐intensive (5Gj/t) and comprises a significant portion of the energy footprint of concrete systems. Incorporating modern monitoring, simulation and control systems will allow lower energy use, lower environmental impact, and lower costs of this widely used construction material. One of the goals of the CESMII roadmap project on the Smart Manufacturing of Cement included developing an analytical process model for clinker quality that includes the chemistry of the kiln feed and accounts for critical process variables. This predictive model will be used in nonlinear model predictive control system designed to significantly reduce process energy use while maintaining or improving product quality. In the cement manufacturing plant used in this study, the kiln feed (meal) is tested every 12 h and used to estimate the mineral composition of the cement kiln output (clinker) using the stoichiometry‐based Bogue's model and the expertise of the plant operators. During kiln operation, kiln output (clinker) is sampled and tested every 2 h to measure its chemical and mineral composition. The predicted and measured values of the clinker composition are used by the plant operators to adjust the kiln input stream and the production process characteristics to maintain stable operation and uniform product quality. However, the time delay between prediction and testing, along with inaccuracies inherent in the Bogue's model have made any process changes designed to minimize energy use problematic, especially in‐light of potential clinker quality issues that process changes often pose. A new analytical model that integrates quality information and process operation information has been developed from data collected from 2 years of production from an operating cement facility. To make the model fuel‐type‐independent, consumed heat energy was computed in the model instead of fuel type and amount. A Feedforward Network was trained and tailored from collected data. Many data‐based simulations were conducted to quantitatively evaluate the proposed model and the 5‐fold cross‐validation procedure was used to test the models. The resulting predictive model was shown to have a low root mean square error (MSE) with respect to the estimated clinker mineral composition compared to that using the industry standard “Bogue’ model”. The end goal of this work was to develop a single machine learning tool that allows the use of quality control data and process control variables to improve energy efficiency of the process in a continuous fashion. The proposed nonlinear model predictive control system (NMPC) can generate predicted kiln production characteristics based on manipulated variables in manner that accurately follows the target product quality values. Simulation results also show that the proposed model produced accurate predictions of kiln outputs that fell within the required constraints, while manipulating control variables within typical operational ranges.

Ali, Asem M.↗
DOE OSTI2021

Kinetics of solid–liquid interface motion in molecular dynamics and phase-field models: crystallization of chromium and silicon

The results of molecular dynamics (MD) simulations of the crystallization process in one-component materials and solid solution alloys reveal a complex temperature dependence of the velocity of the crystal–liquid interface featuring an increase up to a maximum at 10–30% undercooling below the equilibrium melting temperature followed by a gradual decrease of the velocity at deeper levels of undercooling. At the qualitative level, such non-monotonous behaviour of the crystallization front velocity is consistent with the diffusion-controlled crystallization process described by the Wilson–Frenkel model, where the almost linear increase of the interface velocity in the vicinity of melting temperature is defined by the growth of the thermodynamic driving force for the phase transformation, while the decrease in atomic mobility with further increase of the undercooling drives the velocity through the maximum and into a gradual decrease at lower temperatures. At the quantitative level, however, the diffusional model fails to describe the results of MD simulations in the whole range of temperatures with a single set of parameters for some of the model materials. The limited ability of the existing theoretical models to adequately describe the MD results is illustrated in the present work for two materials, chromium and silicon. It is also demonstrated that the MD results can be well described by the solution following from the hodograph equation, previously found from the kinetic phase-field model (kinetic PFM) in the sharp interface limit. The ability of the hodograph equation to describe the predictions of MD simulation in the whole range of temperatures is related to the introduction of slow (phase field) and fast (gradient flow) variables into the original kinetic PFM from which the hodograph equation is obtained. The slow phase-field variable is responsible for the description of data at small undercoolings and the fast gradient flow variable accounts for local non-equilibrium effects at high undercoolings. The introduction of these two types of variables makes the solution of the hodograph equation sufficiently flexible for a reliable description of all nonlinearities of the kinetic curves predicted in MD simulations of Cr and Si. This article is part of the theme issue ‘Transport phenomena in complex systems (part 1)’.

Science & Technology - Other Topics↗
DOE OSTI2023

Assessing complexity and dynamics in epidemics: geographical barriers and facilitators of foot-and-mouth disease dissemination

Introduction: Physical and non-physical processes that occur in nature may influence biological processes, such as dissemination of infectious diseases. However, such processes may be hard to detect when they are complex systems. Because complexity is a dynamic and non-linear interaction among numerous elements and structural levels in which specific effects are not necessarily linked to any one specific element, cause-effect connections are rarely or poorly observed. Methods: To test this hypothesis, the complex and dynamic properties of geo-biological data were explored with high-resolution epidemiological data collected in the 2001 Uruguayan foot-and-mouth disease (FMD) epizootic that mainly affected cattle. County-level data on cases, farm density, road density, river density, and the ratio of road (or river) length/county perimeter were analyzed with an open-ended procedure that identified geographical clustering in the first 11 epidemic weeks. Two questions were asked: (i) do geo-referenced epidemiologic data display complex properties? and (ii) can such properties facilitate or prevent disease dissemination? Results: Emergent patterns were detected when complex data structures were analyzed, which were not observed when variables were assessed individually. Complex properties–including data circularity–were demonstrated. The emergent patterns helped identify 11 counties as ‘disseminators’ or ‘facilitators’ (F) and 264 counties as ‘barriers’ (B) of epidemic spread. In the early epidemic phase, F and B counties differed in terms of road density and FMD case density. Focusing on non-biological, geographical data, a second analysis indicated that complex relationships may identify B-like counties even before epidemics occur. Discussion: Geographical barriers and/or promoters of disease dispersal may precede the introduction of emerging pathogens. If corroborated, the analysis of geo-referenced complexity may support anticipatory epidemiological policies.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2020

Iodine Waste Form Corrosion: PNNL Summary Testing Report

A test method to correlate the laboratory corrosion behavior of iodine-containing waste forms (IWFs) to a conceptual long-term corrosion models is required. Several candidate methods have been evaluated previously, and each method presents unique challenges and limitations. Dynamic leach tests have emerged as the preferred candidate test method to evaluate IWFs and parameterize an eventual long-term corrosion model. This report was prepared in support of this effort to define the test suite to evaluate the corrosion of IWFs, with the goal of defining and parameterizing a conceptual long-term corrosion model. A review was presented highlighting the high degree of variability between the IWF corrosion data that is available in the literature and the need for a standardized test suite. The targeted test suite is composed of a single interval screening test, a dynamic leach test, and post-test solid characterization of the monolithic IWF samples. The data presented serves two purposes: to expand the range of IWFs evaluated with the test suite to assess method variability and to further understand the corrosion of IWFs to best inform conceptual long-term corrosion model development. A series of IWF types were included in the testing, including hot isostatically pressed Ag-mordenite, silver functionalized silica-based aerogels, iodosodalites (hot isostatic pressing, sintered, glass bonded ceramic), iodoapatite, tellurite glass and glass composite materials.

42 ENGINEERING↗
DOE OSTI2021

Southwest Pacific tropical cyclone development classification utilizing machine learning and synoptic composites

This study evaluates the ability of machine learning algorithms to classify tropical depressions (TDs) and tropical storms (TSs) in the western region of the southwest Pacific Ocean (SWPO). Decision rules are generated to predict the environment required for a depression to fully develop into a mature storm, and the most influential predictors in the classification decision are ranked. TD and TS are discriminated based on a maximum sustained wind speed threshold (≥17 ms -1 ). Various aerosol, thermodynamic, and dynamic parameters are extracted closest to the initiation point of each non-developing and developing sample. The covariates associated with each labelled sample are used to train a decision tree and random forest model. Results using a testing dataset suggest the random forest approach more accurately distinguishes between non-developing and developing samples. The classification accuracy of the decision tree and random forest are 72% and 91%, respectively. Random forest outperformed the decision tree by providing higher accuracy in test data. The most important variables for binary classification are sea salt aerosol optical depth (AOD), 1,000 mb relative humidity, and sea surface temperature. AOD is a quantitative estimate of the aerosols presents in the air through the extinction of a ray of light as it passes through the atmosphere. Mean composite maps constructed in an unsupervised manner have been created for the most important variables identified by the random forest classifier during TD and TS events to highlight the difference in geophysical and aerosol variables' climatology during the two different classifications. This work will advance the risk management strategies for northeastern Australia and other SWPO basin islands to control their tropical cyclone related losses through prioritizing forecasting variables that are the strongest predictors of the strengthening of tropical depressions into tropical cyclones.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Distributed out-of-memory NMF on CPU/GPU architectures

We propose an efficient distributed out-of-memory implementation of the non-negative matrix factorization (NMF) algorithm for heterogeneous high-performance-computing systems. The proposed implementation is based on prior work on NMFk, which can perform automatic model selection and extract latent variables and patterns from data. In this work, we extend NMFk by adding support for dense and sparse matrix operation on multi-node, multi-GPU systems. The resulting algorithm is optimized for out-of-memory problems where the memory required to factorize a given matrix is greater than the available GPU memory. Memory complexity is reduced by batching/tiling strategies, and sparse and dense matrix operations are significantly accelerated with GPU cores (or tensor cores when available). Input/output latency associated with batch copies between host and device is hidden using CUDA streams to overlap data transfers and compute asynchronously, and latency associated with collective communications (both intra-node and inter-node) is reduced using optimized NVIDIA Collective Communication Library (NCCL) based communicators. Benchmark results show significant improvement, from 32X to 76x speedup, with the new implementation using GPUs over the CPU-based NMFk. Good weak scaling was demonstrated on up to 4096 multi-GPU cluster nodes with approximately 25,000 GPUs when decomposing a dense 340 Terabyte-size matrix and an 11 Exabyte-size sparse matrix of density 10 -6 .

97 MATHEMATICS AND COMPUTING↗