Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “distributed training”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 325 records · Page 18
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Adaptive hyperparameter updating for training restricted Boltzmann machines on quantum annealers

Restricted Boltzmann Machines (RBMs) have been proposed for developing neural networks for a variety of unsupervised machine learning applications such as image recognition, drug discovery, and materials design. The Boltzmann probability distribution is used as a model to identify network parameters by optimizing the likelihood of predicting an output given hidden states trained on available data. Training such networks often requires sampling over a large probability space that must be approximated during gradient based optimization. Quantum annealing has been proposed as a means to search this space more efficiently which has been experimentally investigated on D-Wave hardware. D-Wave implementation requires selection of an effective inverse temperature or hyperparameter (β) within the Boltzmann distribution which can strongly influence optimization. Here, we show how this parameter can be estimated as a hyperparameter applied to D-Wave hardware during neural network training by maximizing the likelihood or minimizing the Shannon entropy. We find both methods improve training RBMs based upon D-Wave hardware experimental validation on an image recognition problem. Neural network image reconstruction errors are evaluated using Bayesian uncertainty analysis which illustrate more than an order magnitude lower image reconstruction error using the maximum likelihood over manually optimizing the hyperparameter. The maximum likelihood method is also shown to out-perform minimizing the Shannon entropy for image reconstruction.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

CAFE AU LAIT: Compute-Aware Federated Augmented Low-Rank AI Training

Federated finetuning is crucial for unlocking the knowledge embedded in pretrained Large Language Models (LLMs) when data are geographically distributed across clients. Unlike finetuning with data from a single institution, federated finetuning allows collaboration across multiple institutions, enabling the utilization of diverse and decentralized datasets while preserving data privacy. Given the high computing costs of LLM training and the emphasis on energy efficiency in Federated Learning (FL), Low-Rank Adaptation (LoRA) has emerged as a widely adopted algorithm due to its significantly reduced number of trainable parameters. However, this assumes that all data silos have the necessary computing resources to compute local updates of LLMs. Nevertheless, in practice, the computing resources across clients are highly heterogeneous: while some may have access to hundreds of GPUs, others might have limited or no GPU access. Recently, federated finetuning using synthetic data has been proposed, allowing clients to participate in a collaborative training run without training LLMs locally. However, our experimental results reveal a performance gap between models trained using synthetic data and those trained using local updates. Motivated by the observed heterogeneity in computing resources and the performance gap, we propose a novel two-stage algorithm that leverages the storage and computing capabilities of a strong server. In the first stage, under the coordination of the strong server, clients with limited computing resources collaborate to generate synthetic data, which is transferred to and stored on the strong server. In the second stage, the strong server uses this synthetic data on behalf of the resource-constrained clients to perform federated LoRA finetuning alongside clients with sufficient computing resources. This approach ensures that all clients can participate in the finetuning process. Experimental results demonstrate that incorporating local updates from even a small fraction of clients improves performance compared to using synthetic data for all clients. Furthermore, we incorporate the Gaussian mechanism in both stages to guarantee client-level differential privacy.

Wang, Jiayi [ORNL]↗
DOE OSTI2023

Large-Eddy Simulation of Flow Over Boeing Gaussian Bump Using Multiagent Reinforcement Learning Wall Model: Preprint

We develop a wall model for large-eddy simulation (LES) that takes into account various pressure-gradient effects using multi-agent reinforcement learning. The model is trained using low-Reynolds-number flow over periodic hills with agents distributed on the wall at various computational grid points. It utilizes a wall eddy-viscosity formulation as the boundary condition to apply the modeled wall shear stress. Each agent receives states based on local instantaneous flow quantities at an off-wall location, computes a reward based on the estimated wall-shear stress, and provides an action to update the wall eddy viscosity at each time step. The trained wall model is validated in wall-modeled LES of flow over periodic hills at higher Reynolds numbers, and the results show the effectiveness of the model on flow with pressure gradients. The analysis of the trained model indicates that the model is capable of distinguishing between the various pressure gradient regimes present in the flow. To further assess the robustness of the developed wall model, simulations of flow over the Boeing Gaussian bump are conducted at a Reynolds number of 2 x 10^6, based on the free-stream velocity and the bump width. The results of mean skin friction and pressure on the bump surface, as well as the velocity statistics of the flow field, are compared to those obtained from equilibrium wall model (EQWM) simulations and published experimental data sets. The developed wall model is found to successfully capture the acceleration and deceleration of the turbulent boundary layer on the bump surface, providing better predictions of skin friction near the bump peak and exhibiting comparable performance to the EQWM with respect to the wall pressure and velocity field. We also conclude that the subgrid-scale model is crucial to the accurate prediction of the flow field, in particular the prediction of separation.

boundary layer↗
DOE OSTI2026

Regularization via f -Divergence: An Application to Multi-Oxide Spectroscopic Analysis

In this paper, we explore the application of convolutional neural networks (CNNs) for predicting the chemical composition of complex geologic samples in a simulated Martian atmospheric environment. Specifically, we aim to characterize oxide weight percentages (wt.%) of rock samples analyzed by remote Laser-Induced Breakdown Spectroscopy (LIBS), framing the problem as a multi-target regression task . Neural networks trained on LIBS spectra are prone to overfitting due to high spectral complexity, limited labeled data, and measurement noise. While regularization is critical for improving generalization, common methods (e.g., ℓ 2 regularization) impose constraints not directly tied to data distribution properties. We propose a novel regularization method based on a specific ƒ-divergence induced by a graph-based estimator, designed to constrain the distributional discrepancy between predictions and targets. This regularizer serves a dual purpose: (a) mitigating overfitting by enforcing a constraint on the distributional difference between predictions and noisy targets, and (b) acting as an auxiliary loss that penalizes large divergences. To enable backpropagation, we develop a differentiable approximation of this particular ƒ-divergence, making the method feasible for neural networks. Experiments on ChemCam and SuperCam LIBS calibration spectra show that mathematical equation-divergence regularization outperforms or matches standard regularization methods (ℓ 1 , ℓ 2 , dropout) and the classical baseline, partial least squares (PLS). Combining ƒ-divergence regularization with standard regularization yields further performance gains, indicating that distributional regularization is useful in this context giving a promising direction for robust model training in planetary science applications. Source code is publicly available at Klein and Li (2025), https://doi.org/10.11578/dc.20250530.7.

58 GEOSCIENCES↗
DOE OSTI2024

Quantifying Streambed Grain Size, Uncertainty, and Hydrobiogeochemical Parameters Using Machine Learning Model YOLO

Abstract Streambed grain sizes control river hydro‐biogeochemical (HBGC) processes and functions. However, measuring their quantities, distributions, and uncertainties is challenging due to the diversity and heterogeneity of natural streams. This work presents a photo‐driven, artificial intelligence (AI)‐enabled, and theory‐based workflow for extracting the quantities, distributions, and uncertainties of streambed grain sizes from photos. Specifically, we first trained You Only Look Once, an object detection AI, using 11,977 grain labels from 36 photos collected from nine different stream environments. We demonstrated its accuracy with a coefficient of determination of 0.98, a Nash–Sutcliffe efficiency of 0.98, and a mean absolute relative error of 6.65% in predicting the median grain size of 20 ground‐truth photos representing nine typical stream environments. The AI is then used to extract the grain size distributions and determine their characteristic grain sizes, including the 10th, 50th, 60th, and 84th percentiles, for 1,999 photos taken at 66 sites within a watershed in the Northwest US. The results indicate that the 10th, median, 60th, and 84th percentiles of the grain sizes follow log‐normal distributions, with most likely values of 2.49, 6.62, 7.68, and 10.78 cm, respectively. The average uncertainties associated with these values are 9.70%, 7.33%, 9.27%, and 11.11%, respectively. These data allow for the computation of the quantities, distributions, and uncertainties of streambed HBGC parameters, including Manning's coefficient, Darcy‐Weisbach friction factor, top layer interstitial velocity magnitude, and nitrate uptake velocity. Additionally, major sources of uncertainty in grain sizes and their impact on HBGC parameters are examined.

58 GEOSCIENCES↗
DOE OSTI2023

Scaling Resolution of Gigapixel Whole Slide Images Using Spatial Decomposition on Convolutional Neural Networks

Gigapixel images are prevalent in scientific domains ranging from remote sensing, and satellite imagery to microscopy, etc. However, training a deep learning model at the natural resolution of those images has been a challenge in terms of both, overcoming the resource limit (e.g. HBM memory constraints), as well as scaling up to a large number of GPUs. In this paper, we trained Residual neural Networks (ResNet) on 22,528 x 22,528-pixel size images using a distributed spatial decomposition method on 2,304 GPUs on the Summit Supercomputer. We applied our method on a Whole Slide Imaging (WSI) dataset from The Cancer Genome Atlas (TCGA) database. WSI images can be in the size of 100,000 x 100,000 pixels or even larger, and in this work we studied the effect of image resolution on a classification task, while achieving state-of-the-art AUC scores. Moreover, our approach doesn't need pixel-level labels, since we're avoiding patching from the WSI images completely, while adding the capability of training arbitrary large-size images. This is achieved through a distributed spatial decomposition method, by leveraging the non-block fat-tree interconnect network of the Summit architecture, which enabled GPU-to-GPU direct communication. Finally, detailed performance analysis results are shown, as well as a comparison with a data-parallel approach when possible.

Tsaris, Aristeidis (aris)↗
DOE OSTI2024

Synthesizing realistic sand assemblies with denoising diffusion in latent space

Abstract The shapes and morphological features of grains in sand assemblies have far‐reaching implications in many engineering applications, such as geotechnical engineering, computer animations, petroleum engineering, and concentrated solar power. Yet, our understanding of the influence of grain geometries on macroscopic response is often only qualitative, due to the limited availability of high‐quality 3D grain geometry data. In this paper, we introduce a denoising diffusion algorithm that uses a set of point clouds collected from the surface of individual sand grains to generate grains in the latent space. By employing a point cloud autoencoder, the three‐dimensional point cloud structures of sand grains are first encoded into a lower‐dimensional latent space. A generative denoising diffusion probabilistic model is trained to produce synthetic sand that maximizes the log‐likelihood of the generated samples belonging to the original data distribution measured by a Kullback‐Leibler divergence. Numerical experiments suggest that the proposed method is capable of generating realistic grains with morphology, shapes and sizes consistent with the training data inferred from an F50 sand database. We then use a rigid contact dynamic simulator to pour the synthetic sand in a confined volume to form granular assemblies in a static equilibrium state with targeted distribution properties. To ensure third‐party validation, 50,000 synthetic sand grains and the 1542 real synchrotron microcomputed tomography (SMT) scans of the F50 sand, as well as the granular assemblies composed of synthetic sand grains are made available in an open‐source repository.

Vlassis, Nikolaos N.↗
DOE OSTI2025

LLM-Based Adaptive Distribution Voltage Regulation Under Frequent Topology Changes: An In-Context MPC Framework

This paper proposes a large language model (LLM) based adaptive inverter control for distribution voltage regulation under frequent topology changes. We leverage the ability of the LLM to perform in-context learning and create a topology-adaptive surrogate model for power flow calculation. The surrogate model is then integrated with a long short-term memory-based load forecaster and a model predictive control (MPC) scheme to achieve the optimal inverter control that adapts to frequent topology changes. Unlike many existing works that assume fixed-topology grids or require the knowledge of all possible topologies when training a model, the proposed in-context MPC method tackles the distribution voltage control problem under various topologies and adapts to unknown topologies with limited data requirement for fine-tuning. The effectiveness of our method is demonstrated on a modified IEEE 123-bus test system.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2024

New Tank Mapping Method Improves Waste Removal Process

Savannah River Mission Completion is the Liquid Waste (LW) contractor at the Savannah River Site (SRS). The LW mission is tasked with treating and disposing of legacy nuclear waste. There are multiple facilities involved in this work, including the Concentration, Storage, and Transfer Facilities (CSTF), the Defense Waste Processing Facility (DWPF), the Salt Waste Processing Facility (SWPF), and the Saltstone Production Facility (SPF). The CSTF includes 43 underground waste tanks used to store and support processing of radioactive liquid waste. Waste removal activities, such as salt dissolution campaigns and sludge agitation, are conducted within the CSTF waste tanks to convert the waste into a form that allows for downstream processing at other LW facilities. While performing these waste removal campaigns, camera inspections are performed to assess the quantity and distribution of the remaining waste within the waste tank (i.e. saltcake or sludge). Understanding the quantity and distribution of the salt/sludge within the waste tanks allows for improved waste removal strategies (e.g. mixing pump operation) and refined safety controls. Typically, several camera inspections are performed during a waste removal transfer to verify the elevation of the visible salt/sludge mounds against the known elevation of the liquid surface. The camera inspection footage must then be interpreted by a trained engineer who will develop a 2-D map that depicts the waste distribution at various elevations within the waste tank. This tank mapping is then used in conjunction with conservative assumptions to evaluate the volume of saltcake or sludge that is present within the waste tank.

Mini, Melany↗
DOE OSTI2024

Controlling radioisotope proportions when randomly sampling from Dirichlet distributions in PyRIID

As machine learning models for radioisotope quantification become more powerful, likewise the need for high-quality synthetic training data grows as well. For problem spaces that involve estimating the relative isotopic proportions of various sources in gamma spectra it is necessary to generate training data that accurately represents the variance of proportions encountered. In this report, we aim to provide guidance on how to target a desired variance of proportions which are randomly when using the PyRIID Seed Mixer, which samples from a Dirichlet distribution. We provide a method for properly parameterizing the Dirichlet distribution in order to maintain a constant variance across an arbitrary number of dimensions, where each dimension represents a distinct source template being mixed. We demonstrate that our method successfully parameterizes the Dirichlet distribution to target a specific variance of proportions, provided that several conditions are met. This allows us to follow a principled technique for controlling how random mixture proportions are generated which are then used downstream in the synthesis process to produce the final, noisy gamma spectra.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2026

Deep Learning Reconstruction of Daily Soil CO 2 Efflux Reveals Biogeochemical Insights and Reduces Annual Estimate Uncertainty Despite Limited Daily Predictability

Soil CO 2 efflux is commonly measured monthly or seasonally, leaving daily dynamics poorly resolved and contributing to global estimation uncertainty. We trained a single Long Short-Term Memory (LSTM) model to predict daily soil CO 2 efflux across 82 globally distributed sites in COSORE, with 0.2%–46.9% daily data coverage from 2003 to 2020. Despite using far fewer sites than are typically used to train a single deep learning model, with observations biased toward temperate mesic sites, the LSTM model performed well at approximately one-third of sites, reconstructed nearly 2 decades of daily efflux, and outperformed commonly used approaches for estimating daily efflux when applied to the same data set. Performance was weakest at pronounced peaks and troughs and at non-temperate sites with <1.5 years of observations and irregular data patterns. Nevertheless, annual efflux from reconstructed daily data had <40% error even at underperforming sites, substantially improving estimates derived from monthly and seasonal sampling (maximum errors of 95% and 136%, respectively). Temperature sensitivity (Q 10 ) estimated from reconstructed daily predictions closely matched estimates from daily observations, whereas Q 10 values derived from monthly or seasonal observations deviated substantially, suggesting that coarse temporal sampling may contribute to uncertainty in reported Q 10 values. Consistent daily reconstructions further enabled trend analyses for well-performing, predominantly temperate sites and showed increasing soil CO 2 efflux at most sites from 2003 to 2020, with more variable summer trends. Despite limitations, these results demonstrate the potential of LSTM models to reconstruct daily soil CO 2 efflux and reduce estimation uncertainties from sparse observations.

Smykalov, Valerie [Pennsylvania State University, ↗
DOE OSTI2024

Automated Gold Nanorod Spectral Morphology Analysis Pipeline

The development of a colloidal synthesis procedure to produce nanomaterials with high shape and size purity is often a time-consuming, iterative process. This is often due to quantitative uncertainties in the required reaction conditions and the time, resources, and expertise intensive characterization methods required for quantitative determination of nanomaterial size and shape. Absorption spectroscopy is often the easiest method for colloidal nanomaterial characterization. However, due to the lack of a reliable method to extract nanoparticle shapes from absorption spectroscopy, it is generally treated as a more qualitative measure for metal nanoparticles. This work demonstrates a gold nanorod (AuNR) spectral morphology analysis tool, called AuNR-SMA, which is a fast and accurate method to extract quantitative structural information from colloidal AuNR absorption spectra. To demonstrate the practical utility of this model, we apply it to three distinct applications. First, we demonstrate this model's utility as an automated analysis tool in a high-throughput AuNR synthesis procedure by generating quantitative size information from optical spectra. Second, we use the predictions generated by this model to train a machine learning model to predict the resulting AuNR size distributions under specified reaction conditions. Third, we apply this model to spectra extracted from the literature where no size distributions are reported and impute unreported quantitative information on AuNR synthesis. This approach can potentially be extended to any other nanocrystal system where absorption spectra are size dependent, and accurate numerical simulation of absorption spectra is possible. In addition, this pipeline could be integrated into automated synthesis apparatuses to provide interpretable data from simple measurements, help explore the synthesis science of nanoparticles in a rational manner, or facilitate closed-loop workflows.

36 MATERIALS SCIENCE↗
DOE OSTI2023

A forward modeling approach to analyzing galaxy clustering with S IM BIG

We present cosmological constraints from a simulation-based inference (SBI) analysis of galaxy clustering from the SimBIG forward modeling framework. SimBIG leverages the predictive power of high-fidelity simulations and provides an inference framework that can extract cosmological information on small nonlinear scales. In this work, we apply SimBIG to the Baryon Oscillation Spectroscopic Survey (BOSS) CMASS galaxy sample and analyze the power spectrum, P ℓ (k), to k max = 0.5 h/Mpc. We construct 20,000 simulated galaxy samples using our forward model, which is based on 2,000 high-resolution Quijote N -body simulations and includes detailed survey realism for a more complete treatment of observational systematics. We then conduct SBI by training normalizing flows using the simulated samples and infer the posterior distribution of ΛCDM cosmological parameters: Ω m , Ω b , h, n s , σ 8 . We derive significant constraints on Ω m and σ 8 , which are consistent with previous works. Our constraint on σ 8 is 27% more precise than standard P ℓ analyses because we exploit additional cosmological information on nonlinear scales beyond the limit of current analytic models, k > 0.25 h/Mpc. This improvement is equivalent to the statistical gain expected from a standard P ℓ analysis of galaxy sample ~ 60% larger than CMASS. While we focus on P ℓ in this work for validation and comparison to the literature, SimBIG provides a framework for analyzing galaxy clustering using any summary statistic. We expect further improvements on cosmological constraints from subsequent SimBIG analyses of summary statistics beyond P ℓ .

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2023

Data efficiency and extrapolation trends in neural network interatomic potentials

Abstract Recently, key architectural advances have been proposed for neural network interatomic potentials (NNIPs), such as incorporating message-passing networks, equivariance, or many-body expansion terms. Although modern NNIP models exhibit small differences in test accuracy, this metric is still considered the main target when developing new NNIP architectures. In this work, we show how architectural and optimization choices influence the generalization of NNIPs, revealing trends in molecular dynamics (MD) stability, data efficiency, and loss landscapes. Using the 3BPA dataset, we uncover trends in NNIP errors and robustness to noise, showing these metrics are insufficient to predict MD stability in the high-accuracy regime. With a large-scale study on NequIP, MACE, and their optimizers, we show that our metric of loss entropy predicts out-of-distribution error and data efficiency despite being computed only on the training set. This work provides a deep learning justification for probing extrapolation and can inform the development of next-generation NNIPs.

36 MATERIALS SCIENCE↗
DOE OSTI2022

Efficient quantum state tomography with mode-assisted training

Neural networks (NNs) representing quantum states are typically trained using Markov chain Monte Carlo based methods. However, unless specifically designed, such samplers only consist of local moves, making the slow-mixing problem prominent even for extremely simple quantum states. Here, we propose to use mode-assisted training that provides global information via the modes of the NN distribution. Further, applied to quantum state tomography using restricted Boltzmann machines, this method improves the quality of reconstructed quantum states by orders of magnitude. The method is applicable to other types of NNs and may efficiently tackle problems previously unmanageable.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2020

Probabilistic neural networks for fluid flow surrogate modeling and data recovery

We consider the use of probabilistic neural networks for fluid flow surrogate modeling and data recovery. This framework is constructed by assuming that the target variables are sampled from a Gaussian distribution conditioned on the inputs. Consequently, the overall formulation sets up a procedure to predict the hyperparameters of this distribution which are then used to compute an objective function given training data. We demonstrate that this framework has the ability to provide for prediction confidence intervals based on the assumption of a probabilistic posterior, given an appropriate model architecture and adequate training data. The applicability of the present framework to cases with noisy measurements and limited observations is also assessed. To demonstrate the capabilities of this framework, we consider canonical regression problems of fluid dynamics from the viewpoint of reduced-order modeling and spatial data recovery for four canonical data sets. The examples considered in this study arise from (i) the shallow-water equations, (ii) a two-dimensional cylinder flow, (iii) the wake of a NACA0012 airfoil with a Gurney flap, and (iv) the NOAA sea surface temperature data set. Furthermore, the present results indicate that the probabilistic neural network not only produces a machine-learning-based fluid flow surrogate model but also systematically quantifies the uncertainty therein to assist with model interpretability.

42 ENGINEERING↗
DOE OSTI2022

Skopi: a simulation package for diffractive imaging of noncrystalline biomolecules

X-ray free-electron lasers (XFELs) have the ability to produce ultra-bright femtosecond X-ray pulses for coherent diffraction imaging of biomolecules. While the development of methods and algorithms for macromolecular crystallography is now mature, XFEL experiments involving aerosolized or solvated biomolecular samples offer new challenges in terms of both experimental design and data processing. Skopi is a simulation package that can generate single-hit diffraction images for reconstruction algorithms, multi-hit diffraction images of aggregated particles for training machine learning classifiers using labeled data, diffraction images of randomly distributed particles for fluctuation X-ray scattering algorithms, and diffraction images of reference and target particles for holographic reconstruction algorithms. Skopi is a resource to aid feasibility studies and advance the development of algorithms for noncrystalline experiments at XFEL facilities.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

WaveDenoiser

We developed a robust deep learning model to effectively reduce background noise in the time domain from seismic waveforms. The deep learning model processes a 57-second three-component seismogram to predict and generate a denoised seismogram. The training was conducted using the benchmark STEAD dataset, which comprises globally distributed earthquake signals recorded at local distances ranging from 0 to 350 kilometers.

Chai, Chengping [Oak Ridge National Laboratory (OR↗