SEARCH · Engineering Papers
Results for “DNN”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
DNN-based policies for stochastic AC OPF
We report a prominent challenge to the safe and optimal operation of the modern power grid arises due to growing uncertainties in loads and renewables. Stochastic optimal power flow (SOPF) formulations provide a mechanism to handle these uncertainties by computing dispatch decisions and control policies that maintain feasibility under uncertainty. Most SOPF formulations consider simple control policies such as affine policies that are mathematically simple and resemble many policies used in current practice. Motivated by the efficacy of machine learning (ML) algorithms and the potential benefits of general control policies for cost and constraint enforcement, we put forth a deep neural network (DNN)-based policy that predicts the generator dispatch decisions in real time in response to uncertainty. The weights of the DNN are learnt using stochastic primal–dual updates that solve the SOPF without the need for prior generation of training labels and can explicitly account for the feasibility constraints in the SOPF. The advantages of the DNN policy over simpler policies and their efficacy in enforcing safety limits and producing near optimal solutions are demonstrated in the context of a chance constrained formulation on a number of test cases.
TrioSim: A Lightweight Simulator for Large-Scale DNN Workloads on Multi-GPU Systems
Deep Neural Networks (DNNs) have become increasingly capable of performing tasks ranging from image recognition to content generation. The training and inference of DNNs heavily rely on GPUs, as GPUs' massively parallel architecture delivers extremely high computing capability. With the growing complexity of DNNs and the size of training datasets, training DNNs with a large number of GPUs is becoming a prevalent strategy. Researchers have been exploring how to design software and hardware systems for GPU farms to achieve the best utilization, efficiency, and DNN accuracy during training or inference. However, when designing and deploying such systems, designers usually rely on testing on physical hardware platforms equipped with many GPUs, incurring high costs that are almost prohibitive for system designers to test different configurations and designs, even for highly resourceful companies. While an alternative solution is to test on GPU simulators, they are often too slow for these l
MIND-MAC: Multi-Level In-memory Quasi Non-Destructive MAC Operation in Compact 2T-nC FeRAM for Efficient DNN Accelerator
We present MIND-MAC, a compact 2T-nC FeRAM architecture that performs multi-level, quasi-non-destructive in-memory multiply–accumulate (MAC) for deep neural networks. By exploiting voltage-controlled partial domain switching in MFM capacitors and read-transistor amplification, the cell stores multi-bit weights and gates bit-serial inputs to produce an accumulated current on shared lines. We combine TCAD-extracted parasitics with experimentally calibrated ferroelectric models in SPICE to validate device-/circuit-level behavior, and validate multi-level sensing and QNRO with measurements on a fabricated 2T-3C test vehicle. An analytical system model maps MIND-MAC to a 6-GB main-memory in-memory compute (IMC) architecture and benchmarks VGG13 inference in 61.08 ms at 964.99 mJ. Results indicate high density, reduced rewrite overhead, and energy efficiency, positioning 2T-nC FeRAM as a promising IMC candidate for next-generation AI hardware.
DNN-based Signal Processing for Liquid Argon Time Projection Chambers
We investigate a deep learning-based signal processing for liquid argon time projection chambers (LArTPCs), a leading detector technology in neutrino physics. Identifying regions of interest (ROIs) in LArTPCs is challenging due to signal cancellation from bipolar responses and various detector effects observed in real data. We approach ROI identification as an image segmentation task, and employ a U-ResNet architecture. The network is trained on samples that incorporate detector geometry information and include a range of detector variations. Our approach significantly outperforms traditional methods while maintaining robustness across diverse detector conditions. This method has been adopted for signal processing in the Short-Baseline Neutrino program and provides a valuable foundation for future experiments such as the Deep Underground Neutrino Experiment.
Bayesian reduced-order deep learning surrogate model for dynamic systems described by partial differential equations
We propose a reduced-order deep-learning surrogate model for dynamic systems described by time-dependent partial differential equations. This method employs space–time Karhunen–Loève expansions (KLEs) of the state variables and space-dependent KLEs of space-varying parameters to identify the reduced (latent) dimensions. Subsequently, a deep neural network (DNN) is used to map the parameter latent space to the state variable latent space. An approximate Bayesian method is developed for uncertainty quantification (UQ) in the proposed KL-DNN surrogate model. The KL-DNN method is tested for the linear advection–diffusion and nonlinear diffusion equations, and the Bayesian approach for UQ is compared with the deep ensembling (DE) approach, commonly used for quantifying uncertainty in DNN models. It was found that the approximate Bayesian method provides a more informative distribution of the PDE solutions in terms of the coverage of the reference PDE solutions (the percentage of nodes where the reference solution is within the confidence interval predicted by the UQ methods) and log predictive probability. The DE method is found to underestimate uncertainty and introduce bias. For the nonlinear diffusion equation, we compare the KL-DNN method with the Fourier Neural Operator (FNO) method and find that KL-DNN is 10% more accurate and needs less training time than the FNO method.
Deep-learning-derived planetary boundary layer height from conventional meteorological measurements
Abstract. The planetary boundary layer (PBL) height (PBLH) is an important parameter for various meteorological and climate studies. This study presents a multi-structure deep neural network (DNN) model, which can estimate PBLH by integrating the morning temperature profiles and surface meteorological observations. The DNN model is developed by leveraging a rich dataset of PBLH derived from long-standing radiosonde records augmented with high-resolution micro-pulse lidar and Doppler lidar observations. We access the performance of the DNN with an ensemble of 10 members, each featuring distinct hidden-layer structures, which collectively yield a robust 27-year PBLH dataset over the southern Great Plains from 1994 to 2020. The influence of various meteorological factors on PBLH is rigorously analyzed through the importance test. Moreover, the DNN model's accuracy is evaluated against radiosonde observations and juxtaposed with conventional remote sensing methodologies, including Doppler lidar, ceilometer, Raman lidar, and micro-pulse lidar. The DNN model exhibits reliable performance across diverse conditions and demonstrates lower biases relative to remote sensing methods. In addition, the DNN model, originally trained over a plain region, demonstrates remarkable adaptability when applied to the heterogeneous terrains and climates encountered during the GoAmazon (Green Ocean Amazon; tropical rainforest) and CACTI (Cloud, Aerosol, and Complex Terrain Interactions; middle-latitude mountain) campaigns. These findings demonstrate the effectiveness of deep learning models in estimating PBLH, enhancing our understanding of boundary layer processes with implications for improving the representation of PBL in weather forecasting and climate modeling.
Deep-Learning-derived Boundary Layer Height from Meteorological Data over the SGP, GOAMAZON, CACTI
The planetary boundary-layer (PBL) height (PBLH) is an important parameter for various meteorological and climate studies. This study presents a multi-structure deep neural network (DNN) model, designed to estimate PBLH by integrating morning temperature profiles with surface meteorological observations. The DNN model is developed by leveraging a rich data set of PBLH derived from long-standing radiosonde records and augmented with high-resolution micropulse lidar and Doppler lidar observations. We access the performance of the DNN with an ensemble of 10 members, each featuring distinct hidden layer structures, which collectively yield a robust 27-year PBLH data set over the Southern Great Plains from 1994 to 2020. The influence of various meteorological factors on PBLH is rigorously analyzed through the importance test. Moreover, the DNN model's accuracy is evaluated against radiosonde observations and juxtaposed with conventional remote-sensing methodologies, including Doppler lidar, ceilometer, Raman lidar, and micropulse lidar. The DNN model exhibits reliable performance across diverse conditions and demonstrates lower biases relative to remote-sensing methods. In addition, the DNN model, originally trained over a plain region, demonstrates remarkable adaptability when applied to the heterogeneous terrains and climates encountered during the GoAmazon (tropical rainforest) and CACTI (middle-latitude mountain) campaigns. These findings demonstrate the effectiveness of deep learning models in estimating PBLH, enhancing our understanding of boundary-layer dynamics with implications for enhancing the representation of PBL in weather forecasting and climate modeling.
Physics informed deep neural network embedded in a chemical transport model for the Amazon rainforest
Secondary organic aerosols (SOA) are fine particles in the atmosphere, which interact with clouds, radiation and affect the Earth’s energy budget. SOA formation involves chemistry in gas phase, aqueous aerosols, and clouds. Simulating these chemical processes involve solving a stiff set of differential equations, which are computationally expensive steps for three-dimensional chemical transport models. Deep neural networks (DNNs) are universal function approximators that could be used to represent the complex nonlinear changes in aerosol physical and chemical processes; however, key challenges such as generalizability to extended time periods, preservation of mass balance, simulating sparse model outputs, and maintaining physical constraints have limited their use in atmospheric chemistry. Here, we develop an approach of using a physics-informed DNN that overcomes previous such challenges and demonstrates its applicability for the chemical formation processes of isoprene epoxydiol SOA (IEPOX-SOA) over the Amazon rainforest. The DNN is trained with data generated by simulating IEPOX-SOA over the entire atmospheric column, using the Weather Research and Forecasting Model coupled with Chemistry (WRF-Chem). The trained DNN is then embedded within WRF-Chem to replace the computationally expensive default solver of IEPOX-SOA formation. The trained DNN predictions generalizes well with the default model simulation of the IEPOX-SOA mass concentrations and its size distribution (20 size bins) over several days of simulations in both dry and wet seasons. The embedded DNN reduces the computational expense of WRF-Chem by a factor of 2. Our approach shows promise in terms of application to other computationally expensive chemistry solvers in climate models.
Performance Comparison of Machine Learning Models for Ultrasonic Nondestructive Evaluation of Alkali-Silica Reaction in Concrete
Alkali-silica reaction (ASR) causes concrete degradation, leading to cracking, rebar corrosion, and reduced structural integrity, which raises safety concerns. Ultrasonic nondestructive evaluation (NDE) effectively assesses concrete properties and monitors ASR progression. However, its deployment and analysis require specialized expertise and subjective interpretation. As computational power increases, artificial intelligence (AI) and machine learning (ML) algorithms are increasingly being used to automate NDE data analysis across various industries for AI-assisted automation. Regulatory agencies are adapting to this technological shift, prompting a need to evaluate current ML technologies’ capabilities and limitations in assessing concrete material properties and damage. This report presents a comparative analysis of four ML regression models for predicting concrete material damage induced by ASR expansion using long-term ultrasonic data monitoring. The models investigated include linear regression (LR), support vector regression (SVR), shallow neural networks (NN), and deep neural networks (DNN). LR, SVR, and shallow NN models use features extracted from ultrasonic signals, whereas the DNN model processes time-domain ultrasonic signals and frequency spectra directly. The study systematically compared the models’ performance from various perspectives, including model input, prediction performance, and generalization ability. The findings indicate significant variability in model performance, with some ML algorithms achieving very high or very low prediction accuracy depending on the preprocessing and feature engineering (extraction and selection) applied. Key insights include the observation that shallow ML models (LR, SVR, and shallow NNs) require meticulous preprocessing and feature extraction to achieve high accuracy. In contrast, the DNN model, although it bypasses the need for feature engineering, necessitates extensive preprocessing to mitigate noise and computational demands. The SVR model emerged as the top performer among the shallow models, and the DNN model exhibited superior performance on specific datasets but struggled with generalization across specimens from different batches. Additionally, the SVR model is sensitive to temperature variations, whereas the DNN model is robust in this regard. Using recurrent neural networks is recommended for future ASR expansion prediction studies. Recurrent neural networks’ inherent ability to capture temporal dependencies and long-term patterns makes them well suited for analyzing sequential ultrasonic monitoring data. Overall, the results and conclusions of this study could provide insights into the capabilities and effectiveness of ML when applied to ultrasonic NDE data and help identify best practices for using ML for ultrasonic NDE of concrete material properties.
Fast and accurate reduced-order modeling of a MOOSE-based additive manufacturing model with operator learning
One predominant challenge in additive manufacturing (AM) is to achieve specific material properties by manipulating manufacturing process parameters during the runtime. Such manipulation tends to increase the computational load imposed on existing simulation tools employed in AM. The goal of the present work is to construct a fast and accurate reduced-order model (ROM) for an AM model developed within the Multiphysics Object-Oriented Simulation Environment (MOOSE) framework, ultimately reducing the time/cost of AM control and optimization processes. Our adoption of the operator learning (OL) approach enabled us to learn a family of differential equations produced by altering process variables in the laser’s Gaussian point heat source. More specifically, we used the Fourier neural operator (FNO) and deep operator network (DeepONet) to develop ROMs for time-dependent responses. Furthermore, we benchmarked the performance of these OL methods against a conventional deep neural network (DNN)-based ROM. Ultimately, we found that OL methods offer comparable performance and, in terms of accuracy and generalizability, even outperform DNN at predicting scalar model responses. The DNN-based ROM afforded the fastest training time. Furthermore, all the ROMs were faster than the original MOOSE model yet still provided accurate predictions. FNO had a smaller mean prediction error than DeepONet, with a larger variance for time-dependent responses. Unlike DNN, both FNO and DeepONet were able to simulate time series data without the need for dimensionality reduction techniques. Finally, the present work can help facilitate the AM optimization process by enabling faster execution of simulation tools while still preserving evaluation accuracy.
SyReNN: A tool for analyzing deep neural networks
Deep Neural Networks (DNNs) are rapidly gaining popularity in a variety of important domains. Unfortunately, modern DNNs have been shown to be vulnerable to a variety of attacks and buggy behavior. This has motivated recent work in formally analyzing the properties of such DNNs. Furthermore, this paper introduces SyReNN, a tool for understanding and analyzing a DNN by computing its symbolic representation. The key insight is to decompose the DNN into linear functions. Our tool is designed for analyses using low-dimensional subsets of the input space, a unique design point in the space of DNN analysis tools. We describe the tool and the underlying theory, then evaluate its use and performance on three case studies: computing Integrated Gradients, visualizing a DNN’s decision boundaries, and repairing buggy DNNs.
Predicting critical heat flux with uncertainty quantification and domain generalization using conditional variational autoencoders and deep neural networks
Deep generative models (DGMs) can generate synthetic data samples that closely resemble the original dataset, addressing data scarcity. In this work, we developed a conditional variational autoencoder (CVAE) to augment critical heat flux (CHF) data used for the 2006 Groeneveld lookup table. To compare with traditional methods, a fine-tuned deep neural network (DNN) regression model was evaluated on the same dataset. Both models achieved small mean absolute relative errors, with the CVAE showing more favorable results. Uncertainty quantification (UQ) was performed using repeated CVAE sampling and DNN ensembling. The DNN ensemble improved performance over the baseline, while the CVAE maintained consistent results with less variability and higher confidence. Both models achieved small errors inside and outside the training domain, with slightly larger errors outside. Altogether, the CVAE performed better than the DNN in predicting CHF and exhibited better uncertainty behavior.
Quantifying uncertainty for deep learning based forecasting and flow-reconstruction using neural architecture search ensembles
Classical problems in computational physics such as data-driven forecasting and signal reconstruction from sparse sensors have recently seen an explosion in deep neural network (DNN) based algorithmic approaches. However, most DNN models do not provide uncertainty estimates, which are crucial for establishing the trustworthiness of these techniques in downstream decision making tasks and scenarios. In recent years, ensemble-based methods have achieved significant success for the uncertainty quantification in DNNs on a number of benchmark problems. However, their performance on real-world applications remains under-explored. In this work, we present an automated approach to DNN discovery and demonstrate how this may also be utilized for ensemble-based uncertainty quantification. Specifically, we propose the use of a scalable neural and hyperparameter architecture search for discovering an ensemble of DNN models for complex dynamical systems. We highlight how the proposed method not only discovers high-performing neural network ensembles for our tasks, but also quantifies uncertainty seamlessly. This is achieved by using genetic algorithms and Bayesian optimization for sampling the search space of neural network architectures and hyperparameters. Subsequently, a model selection approach is used to identify candidate models for an ensemble set construction. Afterwards, a variance decomposition approach is used to estimate the uncertainty of the predictions from the ensemble. We demonstrate the feasibility of this framework for two tasks — forecasting from historical data and flow reconstruction from sparse sensors for the sea-surface temperature. In conclusion, we demonstrate superior performance from the ensemble in contrast with individual high-performing models and other benchmarks.
Deep learning-based predictive models for laser direct drive at the Omega Laser Facility
The rich and complex physics of inertial confinement fusion provides a unique and challenging space for high-fidelity first-principles modeling. Consequently, simulation codes that are used to design experiments are computationally expensive and lack the predictive capability required for extensive parameter exploration in search of a high-performing design for laser direct drive. In this article, we present two deep-learning-based predictive models intended to address these difficulties. The first model (TL DNN) acts as a fast emulator of simulations as well as experiments at the Omega Laser Facility. This model is trained on a simulation database and subsequently calibrated on experimental data using transfer learning. To facilitate the development of this model, an autoencoder is developed to reduce the dimensionality of the input space by compressing the laser pulse input. The model predicts key experimental scalar observables of Omega experiments with high accuracy and minimal computational cost. This deep neural net enables rapid exploration of a high-dimensional input parameter space for an optimal implosion design. The second model (DNN SM+) aims to extend the statistical modeling work of Lees et al. [Phys. Rev. Lett. 127, 105001 (2021)], by increasing the complexity of the model space and allowing for coupling between degradation terms. Since the model capacity of DNN SM+ is higher than the model of Lees et al., DNN SM+ can potentially provide an improvement in predictive capability, and we use this model to provide insight into complicated degradation dependencies.
Multiobjective Hyperparameter Optimization for Deep Learning Interatomic Potential Training Using NSGA-II
Deep neural network (DNN) potentials are an emerging tool for simulation of dynamical atomistic systems, with the promise of quantum mechanical accuracy at speedups of 10000$\times$. As with other DNN methods, hyperparameters used during training can make a substantial difference in model accuracy, and optimal settings vary with dataset. To enable rapid tuning of hyperparameters for DNN potential training, we developed a scalable multiobjective optimization evolutionary algorithm for supercomputers and tested it on the Summit system at the Oak Ridge Leadership Computing Facility (OLCF). The multiobjective approach is required due to the coupling of two learned values defining the potential: the energy and force. Using a large-scale implementation of the NSGA-II algorithm adapted for training DNN potentials, we discovered several optimal multiobjective combinations, including best choices of activation functions, learning rate scaling scheme, and pairing of the two radial cutoffs used in the three dimensional descriptor function.
A hybrid numerical and machine learning framework for evaluating the performance of a 780 cm 2 aqueous organic redox flow battery
Aqueous organic redox flow battery (AORFB) is a promising cost-competitive technology for large-scale energy storage. Among existing work, the dihydroxyphenazine (DHP)-based AORFB has demonstrated high energy density and low capacity degradation in 10 cm2 cells during lab tests. However, its commercial-scale performance in more complex environments remains unknown, posing a barrier for commercialization. To address this gap, this work presents a comprehensive performance evaluation of a 780 cm 2 DHP-based AORFB by combining physics-based numerical model, machine learning (ML)-based surrogate models, and ML-derived sensitivity quantification. Specifically, we first select 12 key battery parameters that include 10 physicochemical quantities and 2 operation quantities, then select 6 performance metrics that include energy efficiency (EE), discharging capacity, charging energy, and power losses due to concentration, activation, and ohmic over-potentials. With such selection, 12800 combinations of the 12 parameters are subsequently generated using the Latin Hypercube Sampling method. These combinations, together with 38 pre-defined State of Charge, are then integrated to a validated AORFB model developed in COMSOL to compute the performance metrics. With both input parameters and performance metrics, 60 deep neural network (DNN) surrogate models are then trained to approximate the relationship between the 10 physicochemical quantities and 6 performance metrics at each flow rate and current density. Sensitivity scores are then calculated based on the DNN models. Two additional sensitivity analysis tools, i.e., MARS, and SHAP, are also used to cross-validate the sensitivity scores from the DNN. The results demonstrate that 1) the standard potential ranks the first in controlling EE and charging energy, 2) the membrane conductivity is most critical for power loss and EE, and 3) specific area and reaction rate control activation power loss.
Karhunen–Loève deep learning method for surrogate modeling and approximate Bayesian parameter estimation
We evaluate the performance of the Karhunen-Loève Deep Neural Network (KL-DNN) framework for surrogate modeling and approximate Bayesian parameter estimation in partial differential equation models. In the surrogate model, the Karhunen-Loève (KL) expansions are used for the dimensionality reduction of the number of unknown parameters and variables, and a deep neural network is employed to relate the reduced space of parameters to that of the state variables. The KL-DNN surrogate model is used to formulate a maximum-a-posteriori-like least-squares problem, which is randomized to draw samples of the posterior distribution of the parameters. We test the proposed framework for a hypothetical unconfined aquifer via comparison with the forward MODFLOW and inverse PEST++ iterative ensemble smoother (IES) solutions as well as the state-of-the-art Fourier neural operator (FNO) and deep operator networks (DeepONets) operator learning surrogate models. Our results show that the KL-DNN surrogate model outperforms FNO and DeepONet for forward predictions. For solving inverse problems, the randomized algorithm provides the same or more accurate Bayesian predictions of the parameters than IES as evidenced by the higher log-predictive probability of both the estimated parameter field and the forecast hydraulic head. The posterior mean obtained from the randomized algorithm is closer to the reference parameter field than that obtained with FNO as the maximum a posteriori estimate.