Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “recurrent neural networks”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 163 records · Page 9

Forward variable selection enables fast and accurate dynamic system identification with Karhunen-Loève decomposed Gaussian processes

A promising approach for scalable Gaussian processes (GPs) is the Karhunen-Loève (KL) decomposition, in which the GP kernel is represented by a set of basis functions which are the eigenfunctions of the kernel operator. Such decomposed kernels have the potential to be very fast, and do not depend on the selection of a reduced set of inducing points. However KL decompositions lead to high dimensionality, and variable selection thus becomes paramount. This paper reports a new method of forward variable selection, enabled by the ordered nature of the basis functions in the KL expansion of the Bayesian Smoothing Spline ANOVA kernel (BSS-ANOVA), coupled with fast Gibbs sampling in a fully Bayesian approach. It quickly and effectively limits the number of terms, yielding a method with competitive accuracies, training and inference times for tabular datasets of low feature set dimensionality. Theoretical computational complexities are O ( N P 2 ) in training and O ( P ) per point in inference, where N is the number of instances and P the number of expansion terms. The inference speed and accuracy makes the method especially useful for dynamic systems identification, by modeling the dynamics in the tangent space as a static problem, then integrating the learned dynamics using a high-order scheme. The methods are demonstrated on two dynamic datasets: a ‘Susceptible, Infected, Recovered’ (SIR) toy problem, along with the experimental ‘Cascaded Tanks’ benchmark dataset. Comparisons on the static prediction of time derivatives are made with a random forest (RF), a residual neural network (ResNet), and the Orthogonal Additive Kernel (OAK) inducing points scalable GP, while for the timeseries prediction comparisons are made with LSTM and GRU recurrent neural networks (RNNs) along with the SINDy package.

Hayes, Kyle↗

Spatio-Temporal Anomaly Detection with Graph Networks for Data Quality Monitoring of the Hadron Calorimeter

The Compact Muon Solenoid (CMS) experiment is a general-purpose detector for high-energy collision at the Large Hadron Collider (LHC) at CERN. It employs an online data quality monitoring (DQM) system to promptly spot and diagnose particle data acquisition problems to avoid data quality loss. In this study, we present a semi-supervised spatio-temporal anomaly detection (AD) monitoring system for the physics particle reading channels of the Hadron Calorimeter (HCAL) of the CMS using three-dimensional digi-occupancy map data of the DQM. We propose the GraphSTAD system, which employs convolutional and graph neural networks to learn local spatial characteristics induced by particles traversing the detector and the global behavior owing to shared backend circuit connections and housing boxes of the channels, respectively. Recurrent neural networks capture the temporal evolution of the extracted spatial features. We validate the accuracy of the proposed AD system in capturing diverse channel fault types using the LHC collision data sets. The GraphSTAD system achieves production-level accuracy and is being integrated into the CMS core production system for real-time monitoring of the HCAL. We provide a quantitative performance comparison with alternative benchmark models to demonstrate the promising leverage of the presented system.

43 PARTICLE ACCELERATORS↗

Deep Learning Systems for Increased Safeguards Surveillance Review Productivity

Nuclear safeguards inspectors expend significant time and maintain intense focus in reviewing video surveillance for safeguards relevant events. To increase efficiency and reduce the time burden of safeguards inspectors performing surveillance data review, this paper presents a novel deep learning (DL) systems concept to integrate generalized DL models into the safeguards surveillance review workflow. The Agency is investigating several DL algorithms for object recognition, localization, tracking, and flagging relevant activities. The project team is working closely with nuclear safeguards inspectors to identify review use cases (based on specific safeguards objectives) and collect their associated requirements. We focused on CANDU and LWR Nuclear Power Plants (NPPs) and their associated dry storage areas as these present a particularly heavy burden on the inspector surveillance review process due to the number of these facilities under safeguards worldwide. Initial DL algorithm results on safeguards data are promising. Using a convolutional neural network, the team attained a mean average precision (mAP) of 92.9% identifying and localizing spent fuel (SF) casks from a 475 surveillance image dataset. Further, the team had initial success in training a recurrent neural network to identify reactor area activities in video clips, successfully indicating when SF casks enter or exit a pool. We discuss how such DL algorithms would be integrated into the Next Generation Surveillance Review (NGSR) software application. Another issue impacting review productivity is the long time inspectors may have to wait when running these algorithms in NGSR. We present a concept to pre-process remotely collected surveillance data with DL models as the data arrives to IAEA headquarters so that results are already available when starting a new review in NGSR. The proposed DL system concept shows a pathway and workflow for increasing an inspector’s surveillance review productivity by quickly and accurately identifying declared and undeclared safeguards relevant objects and activities in large quantities of surveillance imagery data.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗

Neural Networks for Rapid Design and Analysis

Artificial neural networks have been employed for rapid and efficient dynamics and control analysis of flexible systems. Specifically, feedforward neural networks are designed to approximate nonlinear dynamic components over prescribed input ranges, and are used in simulations as a means to speed up the overall time response analysis process. To capture the recursive nature of dynamic components with artificial neural networks, recurrent networks, which use state feedback with the appropriate number of time delays, as inputs to the networks, are employed. Once properly trained, neural networks can give very good approximations to nonlinear dynamic components, and by their judicious use in simulations, allow the analyst the potential to speed up the analysis process considerably. To illustrate this potential speed up, an existing simulation model of a spacecraft reaction wheel system is executed, first conventionally, and then with an artificial neural network in place.

Sparks, Dean W., Jr.↗

Technical note: Using long short-term memory models to fill data gaps in hydrological monitoring networks

Abstract. Quantifying the spatiotemporal dynamics in subsurface hydrological flows over a long time window usually employs a network of monitoring wells. However, such observations are often spatially sparse with potential temporal gaps due to poor quality or instrument failure. In this study, we explore the ability of recurrent neural networks to fill gaps in a spatially distributed time-series dataset. We use a well network that monitors the dynamic and heterogeneous hydrologic exchanges between the Columbia River and its adjacent groundwater aquifer at the U.S. Department of Energy's Hanford site. This 10-year-long dataset contains hourly temperature, specific conductance, and groundwater table elevation measurements from 42 wells with gaps of various lengths. We employ a long short-term memory (LSTM) model to capture the temporal variations in the observed system behaviors needed for gap filling. The performance of the LSTM-based gap-filling method was evaluated against a traditional autoregressive integrated moving average (ARIMA) method in terms of error statistics and accuracy in capturing the temporal patterns of river corridor wells with various dynamics signatures. Our study demonstrates that the ARIMA models yield better average error statistics, although they tend to have larger errors during time windows with abrupt changes or high-frequency (daily and subdaily) variations. The LSTM-based models excel in capturing both high-frequency and low-frequency (monthly and seasonal) dynamics. However, the inclusion of high-frequency fluctuations may also lead to overly dynamic predictions in time windows that lack such fluctuations. The LSTM can take advantage of the spatial information from neighboring wells to improve the gap-filling accuracy, especially for long gaps in system states that vary at subdaily scales. While LSTM models require substantial training data and have limited extrapolation power beyond the conditions represented in the training data, they afford great flexibility to account for the spatial correlations, temporal correlations, and nonlinearity in data without a priori assumptions. Thus, LSTMs provide effective alternatives to fill in data gaps in spatially distributed time-series observations characterized by multiple dominant frequencies of variability, which are essential for advancing our understanding of dynamic complex systems.

54 ENVIRONMENTAL SCIENCES↗

TorchBraid: High-Performance Layer-Parallel Training of Deep Neural Networks with MPI and GPU Acceleration

TorchBraid is a high-performance implementation of layer-parallel training for deep neural networks (DNNs) supporting MPI-based parallelism and GPU acceleration. Layer-parallel training has been developed to overcome the serialization inherent in forward and backward propagation of DNNs that limits utilization of computational resources in the strong scaling limit. To achieve this, TorchBraid integrates the PyTorch neural network framework with the state-of-the-art XBraid time-parallel library. Furthermore, this article presents the use and performance of TorchBraid, in addition to solutions for overcoming the algorithmic challenges inherent in combining automatic differentiation with layer-parallel. Results are presented with and without GPU acceleration for the Tiny ImageNet and MNIST image classification data sets, as well as recurrent neural networks. Overall, TorchBraid enables fast training of DNNs, both in a strong and weak scaling context. In addition to the TorchBraid software, several new advances in applying layer-parallel algorithms are detailed. Integration of layer-parallel with data-parallel algorithms is presented for the first time, showing the computational advantages of the combination. Standard deep learning techniques, like batch-normalization, are developed for layer-parallel training. Finally, a new approach combining layer-parallel with spatial coarsening in order to accelerate training for 3D image classification shows roughly a 10× speedup over serial execution.

Layer-parallel↗

Data-driven based coordinated smart inverter control for distributed energy resources

Smart inverters (SI) for distributed energy resources (DER) are becoming popular since they have the ability to stabilize as well as restore the voltage and frequency of power systems. Aiming at establishing the mathematical models combined with SI control methods, multiple optimization methods are developed. However, the computational complexity of solving such a mathematical model with various uncertainties limits the real-time application of the SI control. To conquer this challenge, a data-driven-based SI control approach is developed to achieve coordinated control in the high penetration DER system. First, an optimization problem for maximizing the active power generation and minimizing the power loss is designed using the Volt/VAR control. To reduce the time consumption, the recurrent neural network (RNN) is proposed to model the relationship between the uncertainties and control actions during the offline site. The RNN with different sub-structures such as the long short-term memory cell and gated recurrent unit cell are included to enrich the diversity of features. In the last stage, different experiment comparisons, including multiple uncertainties maps and stateof- art machine learning methods, are conducted to verify the effectiveness of the proposed method based on the IEEE 123 bus power system. The results demonstrate that the proposed method can effectively achieve a rapid and coordinated control with a lower error rate.

Qiu, Wei↗

Watch and learn—a generalized approach for transferrable learning in deep neural networks via physical principles

Transfer learning refers to the use of knowledge gained while solving a machine learning task and applying it to the solution of a closely related problem. Such an approach has enabled scientific breakthroughs in computer vision and natural language processing where the weights learned in state-of-the-art models can be used to initialize models for other tasks which dramatically improve their performance and save computational time. Here we demonstrate an unsupervised learning approach augmented with basic physical principles that achieves fully transferrable learning for problems in statistical physics across different physical regimes. By coupling a sequence model based on a recurrent neural network to an extensive deep neural network, we are able to learn the equilibrium probability distributions and inter-particle interaction models of classical statistical mechanical systems. Our approach, distribution-consistent learning, DCL, is a general strategy that works for a variety of canonical statistical mechanical models (Ising and Potts) as well as disordered interaction potentials. Using data collected from a single set of observation conditions, DCL successfully extrapolates across all temperatures, thermodynamic phases, and can be applied to different length-scales. This constitutes a fully transferrable physics-based learning in a generalizable approach.

97 MATHEMATICS AND COMPUTING↗

Upscaling Reactive Transport and Clogging in Shale Microcracks by Deep Learning

Fracture networks in shales exhibit multiscale features. A rock system may contain a few main fractures and thousands of microcracks, whose length and aperture are orders of magnitude smaller than the former. It is computationally prohibitive to resolve all the fractures explicitly for such multiscale fracture networks. One traditional approach is to model the small-scale features (e.g., microcracks in shales) as an effective medium. Although this fracture-matrix conceptualization significantly reduces the problem complexity, there are classes of physical processes that cannot be accurately upscaled by effective medium approximations, e.g., microcrack clogging during mineral reactions. Here, we employ deep learning in place of effective medium theory to upscale physical processes in small-scale features. Specifically, we consider reactive transport in a fracture-microcrack network where microcracks can be clogged by precipitation. A deep learning multiscale algorithm is developed, in which the microcracks are upscaled as a wall boundary condition of the main fractures. The wall boundary condition is constructed by recurrent neural networks, which take concentration histories as input and predict the solute transport from main fractures to microcracks. The deep learning multiscale algorithm is firstly employed in specific scenarios, then a general model is developed which can work under various conditions. The new approach is validated against fully resolved simulations and an analytical solution, providing a reliable and efficient solution for problems that cannot be upscaled by effective medium models.

58 GEOSCIENCES↗

Neural Network-Based Electric Vehicle Range Prediction for Smart Charging Optimization

Range prediction is a standard feature in most modern road vehicles, allowing drivers to make informed decisions about when to refuel. Most vehicles make range predictions through data- or model-driven means, monitoring the average fuel consumption rate or using a tuned vehicle model to predict fuel consumption. The uncertainty of future driving conditions makes the range prediction problem challenging, particularly for less pervasive battery electric vehicles (BEV). Most contemporary machine learning-based methods attempt to forecast the battery SOC discharge profile to predict vehicle range. In this work, we propose a novel approach using two recurrent neural networks (RNNs) to predict the remaining range of BEVs and the minimum charge required to safely complete a trip. Each RNN has two outputs that can be used for statistical analysis to account for uncertainties; the first loss function leads to mean and variance estimation (MVE), while the second results in bounded interval estimation (BIE). These outputs of the proposed RNNs are then used to predict the probability of a vehicle completing a given trip without charging, or if charging is needed, the remaining range and minimum charging required to finish the trip with high probability. Training data was generated using a low-order physics model to estimate vehicle energy consumption from historical drive cycle data collected from medium-duty last-mile delivery vehicles. Here, the proposed method demonstrated high accuracy in the presence of day-to-day route variability, with the root-mean-square error (RMSE) below 6% for both RNN models.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Deep Learning Predicts Stress–Strain Relations of Granular Materials Based on Triaxial Testing Data

This study presents an AI-based constitutive modelling framework wherein the prediction model directly learns from triaxial testing data by combining discrete element modelling (DEM) and deep learning. A constitutive learning strategy is proposed based on the generally accepted frame-indifference assumption in constructing material constitutive models. The low-dimensional principal stress-strain sequence pairs, measured from discrete element modelling of triaxial testing, are used to train recurrent neural networks, and then the predicted principal stress sequence is augmented to other high-dimensional or general stress tensor via coordinate transformation. Through detailed hyperparameter investigations, it is found that long short-term memory (LSTM) and gated recurrent unit (GRU) networks have similar prediction performance in constitutive modelling problems, and both satisfactorily predict the stress responses of granular materials subjected to a given unseen strain path. Furthermore, the unique merits and ongoing challenges of data-driven constitutive models for granular materials are discussed.

42 ENGINEERING↗

Neural-network learning of SPOD latent dynamics

Here, we aim to reconstruct the latent space dynamics of high dimensional, quasi-stationary systems using model order reduction via the spectral proper orthogonal decomposition (SPOD). The proposed method is based on three fundamental steps: in the first, once that the mean flow field has been subtracted from the realizations (also referred to as snapshots), we compress the data from a high-dimensional representation to a lower dimensional one by constructing the SPOD latent space; in the second, we build the time-dependent coefficients by projecting the snapshots containing the fluctuations onto the SPOD basis and we learn their evolution in time with the aid of recurrent neural networks; in the third, we reconstruct the high-dimensional data from the learnt lower -dimensional representation. The proposed method is demonstrated on two different test cases, namely, a compressible jet flow, and a geophysical problem known as the Madden-Julian Oscillation. An extensive comparison between SPOD and the equivalent POD-based counterpart is provided and differences between the two approaches are highlighted. The numerical results suggest that the proposed model is able to provide low rank predictions of complex statistically stationary data and to provide insights into the evolution of phenomena characterized by specific range of frequencies. The comparison between POD and SPOD surrogate strategies highlights the need for further work on the characterization of the interplay of error between data reduction techniques and neural network forecasts.

97 MATHEMATICS AND COMPUTING↗

Using Neural Networks for Low Energy Reconstruction and Neutron Identification in the MicroBooNE LArTPC

Identifying and reconstructing final-state neutrons from neutrino interactions in Liquid Argon Time Projection Chambers (LArTPCs) will enhance future oscillation measurements by recovering missing energy and improving neutrino interaction channel identification. However, neutrons are challenging to reconstruct as the majority leave only small, isolated charge signatures known as blips. Here we present initial efforts to identify neutrons in the MicroBooNE LArTPC with low energy protons from neutron-argon inelastic interactions that present as blips below the traditional tracking threshold in the TPC. Unlike for tracks, there is no algorithmic method to determine direction for blips since they span only a few wires. Therefore, we developed and trained a Recurrent Neural Network (RNN) to reconstruct the directionality of proton-induced blips, allowing us to separate signal from background by selecting blips that point back to the neutrino vertex. The model achieves a preliminary average angular resolution of 17 degrees when tested on a simulated sample of protons over 6 MeV in kinetic energy. This novel tool will enhance neutron detection in LArTPCs and expand a broad range of other low-energy physics searches such as for solar and supernova neutrinos.

Silva, Liani Isabel [Unlisted, US]↗

Evapotranspiration partitioning estimates from 8 methods from 47 NEON sites, 2019-2021

This dataset provides daily estimates of evapotranspiration (ET) and the transpiration-to-evapotranspiration ratio (T/ET) across 47 terrestrial National Ecological Observatory Network (NEON) sites spanning diverse environmental and biome conditions in the United States across three years of data (2019-2021). Daily ET is reported in both energy units (MJ m⁻² day⁻¹) and equivalent water depth (mm day⁻¹), assuming a constant latent heat of vaporization of 2.45 MJ/kg. The primary method uses a hybrid recurrent neural network–Penman–Monteith framework (RNN-PM), which integrates physically based surface energy balance constraints with data-driven learning to partition ET into transpiration and evaporation components. Model inputs include in situ meteorological observations (air temperature, vapor pressure deficit, wind speed, and radiation) combined with satellite-derived land surface temperature, leaf area index, and soil moisture. For benchmarking and uncertainty assessment, T/ET estimates from seven additional models are included: Priestley-Taylor Jet Propulsion Laboratory (PT-JPL), Penman-Monteith (P-M), Two-Source Energy Balance (TSEB), Support Vector Regression (SVR), and Categorical Boosting (CatBoost), among others—spanning empirical, machine-learning, and process-based approaches (see methods section or linked publication for detailed descriptions). Data Package Contents: The dataset a csv files containing daily ET and T/ET estimates for each site and model, along with associated metadata files these variables. Data can be accessed using common spreadsheet software (e.g., Microsoft Excel, LibreOffice) or programming environments such as R or Python. Together, these data support cross-site comparisons of ecosystem water use, evaluation of ET partitioning methods, and development of improved land–atmosphere exchange models.

EARTH SCIENCE > ATMOSPHERE↗

From Coexpression to Coregulation: An Approach to Inferring Transcriptional Regulation Among Gene Classes from Large-Scale Expression Data

We provide preliminary evidence that existing algorithms for inferring small-scale gene regulation networks from gene expression data can be adapted to large-scale gene expression data coming from hybridization microarrays. The essential steps are (I) clustering many genes by their expression time-course data into a minimal set of clusters of co-expressed genes, (2) theoretically modeling the various conditions under which the time-courses are measured using a continuous-time analog recurrent neural network for the cluster mean time-courses, (3) fitting such a regulatory model to the cluster mean time courses by simulated annealing with weight decay, and (4) analysing several such fits for commonalities in the circuit parameter sets including the connection matrices. This procedure can be used to assess the adequacy of existing and future gene expression time-course data sets for determining transcriptional regulatory relationships such as coregulation.

Mjolsness, Eric↗

Construction of generalized quasilinear diffusion coefficient using neural networks with physical restrictions

The quasilinear diffusion coefficient (D QL ) derived from our machine learning framework shows comparable trends with the ground truth D QL obtained from GENRAY-CQL3D simulations. Additionally, for the strong absorption cases, the radial current drive profiles generated using the D QL from our model exhibit consistent behavior with those obtained from the original simulation. These findings indicate the potential of our surrogate modeling approach with physical restrictions to replicate key wave–plasma interaction characteristics while reducing computational costs. Traditionally, calculating D QL for wave–particle interactions relies on computationally intensive wave simulations coupled with Fokker–Planck solvers. To address this challenge, we developed a machine learning-based surrogate model with physical restrictions derived from cold plasma theory and bounce-averaged damping effects. First, we establish the propagation domain of Lower Hybrid Waves in the (N∥, ρ) space by identifying the accessibility limit and determining the upper and lower bounds of N∥ using the Potential Power Deposition (PPD) method. Subsequently, leveraging a database constructed using Latin hypercube sampling alongside the underlying physical restrictions (e.g. PPD), machine learning methods including U-Net and Recurrent Neural Networks are employed to design a physics-restricted machine learning framework capable of reconstructing D QL .

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Monitoring of Temperature Measurements for Different Flow Regimes in Water and Galinstan with Long Short-Term Memory Networks and Transfer Learning of Sensors

Temperature sensing is one of the most common measurements of a nuclear reactor monitoring system. The coolant fluid flow in a reactor core depends on the reactor power state. We investigated the monitoring and estimation of the thermocouple time series using machine learning for a range of flow regimes. Measurement data were obtained, in two separate experiments, in a flow loop filled with water and with liquid metal Galinstan. We developed long short-term memory (LSTM) recurrent neural networks (RNNs) for sensor predictions by training on the sensor’s own prior history, and transfer learning LSTM (TL-LSTM) by training on a correlated sensor’s prior history. Sensor cross-correlations were identified by calculating the Pearson correlation coefficient of the time series. The accuracy of LSTM and TL-LSTM predictions of temperature was studied as a function of Reynolds number (Re). The root-mean-square error (RMSE) for the test segment of time series of each sensor was shown to linearly increase with Re for both water and Galinstan fluids. Using linear correlations, we estimated the range of values of Re for which RMSE is smaller than the thermocouple measurement uncertainty. For both water and Galinstan fluids, we showed that both LSTM and TL-LSTM provide reliable estimations of temperature for typical flow regimes in a nuclear reactor. The LSTM runtime was shown to be substantially smaller than the data acquisition rate, which allows for performing estimation and validation of sensor measurements in real time.

Pantopoulou, Stella↗

One-Step Ahead Prediction of Thermal Mixing Tee Sensors with Long Short Term Memory (LSTM) Neural Networks

High-temperature advanced reactors under development, such as sodium fast reactors (SFR) and molten salt cooled reactors (MSCR), are expected to offer lower levelized cost of energy (LCOE) compared to existing light water reactor (LWR’s). In the existing light water reactors (LWR’s), operation and maintenance (O&M) expenses constitute the largest fraction of the total operating cost. Some of the O&M costs are related maintenance of sensors which can fail due to exposure to harsh environment in a reactor. The O&M costs of Advanced Reactor (AR)’s are expected to constitute a significant fraction of the total cost as well, because of high temperature and radiation level in AR are likely to cause material fatigue and premature failure of sensors and components. The O&M costs in AR’s could be reduced through integration of advanced informatics of performance-related sensors into a digital twin designed for reactor monitoring. For example, machine learning (ML) could be employed for real-time validation and correction of performance-related sensors, and reducing the number of performance-related physical sensor units through virtual sensing. As part of the effort, we investigate real-time validation of thermal hydraulic sensors through one-step ahead forecasting of sensor values using long short-term memory (LSTM) recurrent neural networks (RNN). The sensors are installed in a flow loop containing a thermal mixing tee, which is a common experimental model to study thermal fatigue in a thermal hydraulic loop. In addition, nonlinear transients generated in a thermal mixing tee constitute a good challenge data set for training and validation of ML algorithms. Sensors in this study include thermocouples, flow meters, and optical fibers for distributed temperature sensing. In one experiment, measurement data sets were obtained for a loop was filled with water, and in another experiment, measurements were performed on a loop filled with liquid metal Galinstan. We have also conducted preliminary investigation of one-step ahead prediction of fiber optics-based distributed temperature sensing with LSTM networks. In predicting fiber-based temperature measurements, we treated each gauge pitch of the fiber as an independent sensor. Accuracy of one-step ahead forecasting was estimated by calculating root mean square error (RMSE) for the test segment of time series of each sensor. RMSE’s for temperature sensors in water loop were, for the most part, lower than for the same sensors in Galinstan loop. The RMSE’s for flow meters were similar for both loops. The RMSE’s for distributed temperature measured with the fiber optic sensor were similar to those of the point sensors. Results of this study demonstrated the capability of LSTM one-step ahead forecasting with RMSE comparable to uncertainty in sensor measurements.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗