Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Deep Recurrent Learning”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Towards data-driven constitutive modelling for granular materials via micromechanics-informed deep learning

The analytical description of path-dependent elastic-plastic responses of a granular system is highly complicated because of continuously evolving microstructures and strain localisation within the system undergoing deformation. This study offers an alternative to the current analytical paradigm by developing micromechanics-informed machine-learning based constitutive modelling approaches for granular materials. A set of critical variables associated with the constitutive behaviour of granular materials are identified through an incremental stress-strain relationship analysis. Depending on the strategy to exploit the priori micromechanical knowledge, three different training strategies are explored. The first model uses only the measurable external variables to make stress predictions; the second model utilises a directed graph to link all the external strain sequences and internal microstructural evolution variables into a single prediction model comprised of a series of sub-mappings, and the third model explicitly integrates the physically important non-temporal properties with external strain paths into training through an enhanced Gated Recurrent Unit (GRU). These three models show satisfactory agreement with unseen test specimens based on multi-directional loading cases. The basic features and potential applications of each model are explained. Lastly, the key factors for constitutive training and limitations of the current work are also discussed in detail.

36 MATERIALS SCIENCE↗
DOE OSTI2026

Recurrent convolutional neural networks for modeling nonadiabatic dynamics of quantum-classical systems

Recurrent neural networks (RNNs) have recently been extensively applied to model the time evolution in fluid dynamics, weather predictions, and even chaotic systems due to their ability to capture temporal dependencies and sequential patterns in data. Here we present an RNN model based on convolutional neural networks for modeling the nonlinear nonadiabatic dynamics of hybrid quantum-classical systems. The dynamical evolution of the hybrid systems is governed by equations of motion for classical degrees of freedom and von Neumann equation for electrons. The Physics-Aware Recurrent Convolution (PARC) neural network structure incorporates a differentiator-integrator architecture that inductively models the spatiotemporal dynamics of generic physical systems. Here, we apply our RNN approach to learn the space-time evolution of a one-dimensional semiclassical Holstein model after an interaction quench. For shallow quenches (small changes in electron-lattice coupling), the deterministic dynamics can be accurately captured using a single-CNN-based recurrent network. In contrast, deep quenches induce chaotic evolution, making long-term trajectory prediction significantly more challenging. Nonetheless, we demonstrate that the PARC-CNN architecture can effectively learn the statistical climate of the Holstein model under deep-quench conditions.

Holstein model↗
DOE OSTI2022

Tracing and Forecasting Metabolic Indices of Cancer Patients Using Patient-Specific Deep Learning Models

We develop a patient-specific dynamical system model from the time series data of the cancer patient’s metabolic panel taken during the period of cancer treatment and recovery. The model consists of a pair of stacked long short-term memory (LSTM) recurrent neural networks and a fully connected neural network in each unit. It is intended to be used by physicians to trace back and look forward at the patient’s metabolic indices, to identify potential adverse events, and to make short-term predictions. When the model is used in making short-term predictions, the relative error in every index is less than 10% in the L ∞ norm and less than 6.3% in the L 1 norm in the validation process. Once a master model is built, the patient-specific model can be calibrated through transfer learning. As an example, we obtain patient-specific models for four more cancer patients through transfer learning, which all exhibit reduced training time and a comparable level of accuracy. This study demonstrates that this modeling approach is reliable and can deliver clinically acceptable physiological models for tracking and forecasting patients’ metabolic indices.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2021

Deep Learning Predicts Stress–Strain Relations of Granular Materials Based on Triaxial Testing Data

This study presents an AI-based constitutive modelling framework wherein the prediction model directly learns from triaxial testing data by combining discrete element modelling (DEM) and deep learning. A constitutive learning strategy is proposed based on the generally accepted frame-indifference assumption in constructing material constitutive models. The low-dimensional principal stress-strain sequence pairs, measured from discrete element modelling of triaxial testing, are used to train recurrent neural networks, and then the predicted principal stress sequence is augmented to other high-dimensional or general stress tensor via coordinate transformation. Through detailed hyperparameter investigations, it is found that long short-term memory (LSTM) and gated recurrent unit (GRU) networks have similar prediction performance in constitutive modelling problems, and both satisfactorily predict the stress responses of granular materials subjected to a given unseen strain path. Furthermore, the unique merits and ongoing challenges of data-driven constitutive models for granular materials are discussed.

42 ENGINEERING↗
DOE OSTI2022

Survival analysis of localized prostate cancer with deep learning

In recent years, data-driven, deep-learning-based models have shown great promise in medical risk prediction. By utilizing the large-scale Electronic Health Record data found in the U.S. Department of Veterans Affairs, the largest integrated healthcare system in the United States, we have developed an automated, personalized risk prediction model to support the clinical decision-making process for localized prostate cancer patients. This method combines the representative power of deep learning and the analytical interpretability of parametric regression models and can implement both time-dependent and static input data. To collect a comprehensive evaluation of model performances, we calculate time-dependent C-statistics C td over 2-, 5-, and 10-year time horizons using either a composite outcome or prostate cancer mortality as the target event. The composite outcome combines the Prostate-Specific Antigen (PSA) test, metastasis, and prostate cancer mortality. Our longitudinal model Recurrent Deep Survival Machine (RDSM) achieved C td 0.85 (0.83), 0.80 (0.83), and 0.76 (0.81), while the cross-sectional model Deep Survival Machine (DSM) attained C td 0.85 (0.82), 0.80 (0.82), and 0.76 (0.79) for the 2-, 5-, and 10-year composite (mortality) outcomes, respectively. In addition to estimating the survival probability, our method can quantify the uncertainty associated with the prediction. The uncertainty scores show a consistent correlation with the prediction accuracy. We find PSA and prostate cancer stage information are the most important indicators in risk prediction. Our work demonstrates the utility of the data-driven machine learning model in prostate cancer risk prediction, which can play a critical role in the clinical decision system.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2023

Spatio-Temporal Deep Graph Network for Event Detection, Localization, and Classification in Cyber-Physical Electric Distribution System

This work proposes a deep graph learning framework to identify, locate, and classify power, cyber, and cyber power events at the distribution system level. The proposed algorithm jointly exploits spatial, temporal, and node-level cyber and physical data features. The developed graph neural network, together with a deep autoencoder, utilizes physical measurements from distribution level phasor measurement units and cyber data from communication network logs. The spatial structure of the synchrophasor measurements and network is incorporated through a weighted adjacency matrix. The temporal structure is incorporated by defining a spatial operation in the gated recurrent unit. This spatio-temporal learning element resides inside a power event detection, localization, and classification module that provides the degree of confidence for an event label. To accurately pinpoint the location of an event to the nearest bus equipped with a measurement unit, a combination of squared error and proximity score is utilized. Also included is a cyber event detection module that employs heteroskedasticity to analyze the significance of various cyber features during different types of attacks. Finally, a dual-bit cyber-power decision table determines the nature of the event. The proposed method is validated on two distribution systems modeled in OPAL-RT/Hypersim with limited phasor measurement units for different possible physical and cyber events. Further analyses include comparison with other state-of-the-art methods and validation in the presence of measurement noise. As a result, our method outperforms existing approaches and achieves an average detection accuracy of 97.97%, F1-score of 96.88%, precision of 96.53%, and recall of 98.57%.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2025

A generative artificial intelligence framework for long-time plasma turbulence simulations

Generative deep learning techniques are employed in a novel framework for the construction of surrogate models capturing the spatiotemporal dynamics of 2D plasma turbulence. The proposed Generative Artificial Intelligence Turbulence (GAIT) framework enables the acceleration of turbulence simulations for long-time transport studies. GAIT leverages a convolutional variational auto-encoder and a recurrent neural network to generate new turbulence data from existing simulations, extending the time horizon of transport studies with minimal computational cost. The application of the GAIT framework to plasma turbulence using the Hasegawa–Wakatani (HW) model is presented, evaluating its performance via various analyses. Very good agreement is found between the GAIT and the HW models in the spatiotemporal Fourier and Proper Orthogonal Decomposition spectra, the flow topology characterized by the Okubo–Weiss parameter, and the time autocorrelation function of turbulent fluctuations. Excellent agreement has also been obtained in the probability distribution function of particle displacements and the effective turbulent diffusivity. In-depth analyses of the latent space of turbulent states, choice of hyperparameters and alternative deep learning models for the time prediction are presented. Our results highlight the potential of Artificial Intelligence-based surrogate models to overcome the computational challenges in turbulence simulation, which can be extended to other situations such as geophysical fluid dynamics.

Artificial intelligence↗
DOE OSTI2024

Spatio–Temporal Machine Learning for Regional to Continental Scale Terrestrial Hydrology

Integrated hydrologic models can simulate coupled surface and subsurface processes but are computationally expensive to run at high resolutions over large domains. Here we develop a novel deep learning model to emulate subsurface flows simulated by the integrated ParFlow–CLM model across the contiguous US. We compare convolutional neural networks like ResNet and UNet run autoregressively against our novel architecture called the Forced SpatioTemporal RNN (FSTR). The FSTR model incorporates separate encoding of initial conditions, static parameters, and meteorological forcings, which are fused in a recurrent loop to produce spatiotemporal predictions of groundwater. We evaluate the model architectures on their ability to reproduce 4D pressure heads, water table depths, and surface soil moisture over the contiguous US at 1 km resolution and daily time steps over the course of a full water year. The FSTR model shows superior performance to the baseline models, producing stable simulations that capture both seasonal and event–scale dynamics across a wide array of hydroclimatic regimes. The emulators provide over 1,000× speedup compared to the original physical model, which will enable new capabilities like uncertainty quantification and data assimilation for integrated hydrologic modeling that were not previously possible. Our results demonstrate the promise of using specialized deep learning architectures like FSTR for emulating complex process–based models without sacrificing fidelity.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Machine learning and deep learning tools for the automated capture of cancer surveillance data

The National Cancer Institute and the Department of Energy strategic partnership applies advanced computing and predictive machine learning and deep learning models to automate the capture of information from unstructured clinical text for inclusion in cancer registries. Applications include extraction of key data elements from pathology reports, determination of whether a pathology or radiology report is related to cancer, extraction of relevant biomarker information, and identification of recurrence. With the growing complexity of cancer diagnosis and treatment, capturing essential information with purely manual methods is increasingly difficult. These new methods for applying advanced computational capabilities to automate data extraction represent an opportunity to close critical information gaps and create a nimble, flexible platform on which new information sources, such as genomics, can be added. This will ultimately provide a deeper understanding of the drivers of cancer and outcomes in the population and increase the timeliness of reporting. These advances will enable better understanding of how real-world patients are treated and the outcomes associated with those treatments in the context of our complex medical and social environment.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2023

Learning macroscopic internal variables and history dependence from microscopic models

This paper concerns the study of history dependent phenomena in heterogeneous materials in a two-scale setting where the material is specified at a fine microscopic scale of heterogeneities that is much smaller than the coarse macroscopic scale of application. Here, we specifically study a polycrystalline medium where each grain is governed by crystal plasticity while the solid is subjected to macroscopic dynamic loads. The theory of homogenization allows us to solve the macroscale problem directly with a constitutive relation that is defined implicitly by the solution of the microscale problem. However, the homogenization leads to a highly complex history dependence at the macroscale, one that can be quite different from that at the microscale. In this paper, we examine the use of machine-learning, and especially deep neural networks, to harness data generated by repeatedly solving the finer scale model to: (i) gain insights into the history dependence and the macroscopic internal variables that govern the overall response; and (ii) to create a computationally efficient surrogate of its solution operator, that can directly be used at the coarser scale with no further modeling. We do so by introducing a recurrent neural operator (RNO), and show that: (i) the architecture and the learned internal variables can provide insight into the physics of the macroscopic problem; and (ii) that the RNO can provide multiscale, specifically FE 2 , accuracy at a cost comparable to a conventional empirical constitutive relation.

36 MATERIALS SCIENCE↗
DOE OSTI2020

On Neural Architectures for Astronomical Time-series Classification with Application to Variable Stars

Despite the utility of neural networks (NNs) for astronomical time-series classification, the proliferation of learning architectures applied to diverse data sets has thus far hampered a direct intercomparison of different approaches. Here we perform the first comprehensive study of variants of NN-based learning and inference for astronomical time series, aiming to provide the community with an overview on relative performance and, hopefully, a set of best-in-class choices for practical implementations. In both supervised and self-supervised contexts, we study the effects of different time-series-compatible layer choices, namely the dilated temporal convolutional neural network (dTCNs), long-short term memory NNs, gated recurrent units and temporal convolutional NNs (tCNNs). Additionally, we also study the efficacy and performance of encoder-decoder (i.e., autoencoder) networks compared to direct classification networks, different pathways to include auxiliary (non-time-series) metadata, and different approaches to incorporate multi-passband data (i.e., multiple time series per source). Performance—applied to a sample of 17,604 variable stars (VSs) from the MAssive Compact Halo Objects (MACHO) survey across 10 imbalanced classes—is measured in training convergence time, classification accuracy, reconstruction error, and generated latent variables. We find that networks with recurrent NNs generally outperform dTCNs and, in many scenarios, yield to similar accuracy as tCNNs. In learning time and memory requirements, convolution-based layers perform better. We conclude by discussing the advantages and limitations of deep architectures for VS classification, with a particular eye toward next-generation surveys such as the Legacy Survey of Space and Time, the Roman Space Telescope, and Zwicky Transient Facility.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2023

Robust deep learning framework for constitutive relations modeling

Modeling the full-range deformation behaviors of materials under complex loading and materials conditions is a significant challenge for constitutive relations (CRs) modeling. Here, we propose a general encoder-decoder deep learning framework that can model high-dimensional stress-strain data and complex loading histories with robustness and universal capability. The framework employs an encoder to project high-dimensional input information (e.g., loading history, loading conditions, and materials information) to a lower-dimensional hidden space and a decoder to map the hidden representation to the stress of interest. We evaluated various encoder architectures, including gated recurrent unit (GRU), GRU with attention, temporal convolutional network (TCN), and the Transformer encoder, on two complex stress-strain datasets that were designed to include a wide range of complex loading histories and loading conditions. All architectures achieved excellent test results with an root-mean-square error (RMSE) below 1 MPa. Additionally, we analyzed the capability of the different architectures to make predictions on out-of-domain applications, with an uncertainty estimation based on deep ensembles. The proposed approach provides a robust alternative to empirical/semi-empirical models for CRs modeling, offering the potential for more accurate and efficient materials design and optimization.

36 MATERIALS SCIENCE↗
DOE OSTI2024

STFM: Accurate Spatio-Temporal Fusion Model for Weather Forecasting

Meteorological prediction is crucial for various sectors, including agriculture, navigation, daily life, disaster prevention, and scientific research. However, traditional numerical weather prediction (NWP) models are constrained by their high computational resource requirements, while the accuracy of deep learning models remains suboptimal. In response to these challenges, we propose a novel deep learning-based model, the Spatiotemporal Fusion Model (STFM), designed to enhance the accuracy of meteorological predictions. Our model leverages Fifth-Generation ECMWF Reanalysis (ERA5) data and introduces two key components: a spatiotemporal encoder module and a spatiotemporal fusion module. The spatiotemporal encoder integrates the strengths of convolutional neural networks (CNNs) and recurrent neural networks (RNNs), effectively capturing both spatial and temporal dependencies. Meanwhile, the spatiotemporal fusion module employs a dual attention mechanism, decomposing spatial attention into global static attention and channel dynamic attention. This approach ensures comprehensive extraction of spatial features from meteorological data. The combination of these modules significantly improves prediction performance. Experimental results demonstrate that STFM excels in extracting spatiotemporal features from reanalysis data, yielding predictions that closely align with observed values. In comparative studies, STFM outperformed other models, achieving a 7% improvement in ground and high-altitude temperature predictions, a 5% enhancement in the prediction of the u/v components of 10 m wind speed, and an increase in the accuracy of potential height and relative humidity predictions by 3% and 1%, respectively. This enhanced performance highlights STFM’s potential to advance the accuracy and reliability of meteorological forecasting.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

DeepShake: Shaking Intensity Prediction Using Deep Spatiotemporal RNNs for Earthquake Early Warning

Abstract We propose a deep spatiotemporal recurrent neural network, DeepShake, to project future shaking intensity directly from current ground-motion observations. DeepShake is a network-based forecasting model, able to predict future shaking intensity at all stations within a network given previously measured ground shaking. The model is not given any a priori knowledge of station locations; instead, it learns wave propagation amplitudes and delays solely from training data. We developed DeepShake with the 35,679 earthquakes from the 2019 Ridgecrest sequence. Tasked with alerting for modified Mercalli intensity (MMI) IV+ shaking on 3568 validation earthquakes at least 5 s in advance, DeepShake achieves an equal error rate of 11.4%. For the Mw 7.1 earthquake that hit Ridgecrest on 5 July 2019, DeepShake was able to provide targeted alerts to all stations inside the network 5 s prior to the arrival of MMI IV+ waveforms. DeepShake demonstrates that deep spatiotemporal neural networks can effectively provide one-step earthquake early warning with reasonable accuracy and latency.

Geochemistry & Geophysics↗
DOE OSTI2021

Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functions

Here we propose a new type of neural networks, Kronecker neural networks (KNNs), that form a general framework for neural networks with adaptive activation functions. KNNs employ the Kronecker product, which provides an efficient way of constructing a very wide network while keeping the number of parameters low. Our theoretical analysis reveals that under suitable conditions, KNNs induce a faster decay of the loss than that by the feed-forward networks. This is also empirically verified through a set of computational examples. Furthermore, under certain technical assumptions, we establish global convergence of gradient descent for KNNs. As a specific case, we propose the Rowdy activation function that is designed to get rid of any saturation region by injecting sinusoidal fluctuations, which include trainable parameters. The proposed Rowdy activation function can be employed in any neural network architecture like feed-forward neural networks, Recurrent neural networks, Convolutional neural networks etc. The effectiveness of KNNs with Rowdy activation is demonstrated through various computational experiments including function approximation using feed-forward neural networks, solution inference of partial differential equations using the physics-informed neural networks, and standard deep learning benchmark problems using convolutional and fully-connected neural networks.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Estimating CO 2 fluxes through integrating spatial and temporal input layers via deep learning algorithms

Background Accurate estimation of net ecosystem exchange of CO 2 fluxes (Fc) is essential for understanding carbon cycle processes and assessing ecosystem carbon budgets. However, conventional modeling approaches often emphasize temporal dynamics while overlooking the pronounced spatial heterogeneity within the footprint of eddy covariance (EC) towers, potentially limiting predictive accuracy and interpretability of Fc estimates. To address this challenge, we developed a spatiotemporal model that integrates high-resolution footprint-weighted spatial information with sequential environmental drivers. Results The integrated model combines a deeper graph convolutional network to characterize fine-scale spatial variability within EC footprints and a gated recurrent unit network to capture temporal dependencies in biophysical conditions. Using multi-year flux tower observations, remote sensing vegetation indices and footprint modeling, we evaluate the proposed method across three land cover types. This spatiotemporal model consistently outperforms temporal-only and spatial-only baselines, achieving the highest overall accuracy (R 2 = 0.9569) and the lowest RMSE (1.8128 μmol m −2 s −1 ) and MAE (1.1939 μmol m −2 s −1 ). Performance gains are particularly evident in ecosystems with strong vegetation heterogeneity, where spatial structure substantially modulates Fc variability. Conclusions This study demonstrates the importance of joint modeling spatial heterogeneity and temporal dynamics for improving Fc estimation and provides a robust method for advancing footprint-based Fc estimates across diverse ecosystems, supporting refined assessments of terrestrial carbon fluxes, and enhancing scientific foundations for carbon studies.

CO2 flux estimate↗
DOE OSTI2022

A Bayesian Deep Learning Approach to Near-Term Climate Prediction

Since model bias and associated initialization shock are serious shortcomings that reduce prediction skills in state-of-the-art decadal climate prediction efforts, we pursue a complementary machine-learning-based approach to climate prediction. The example problem setting we consider consists of predicting natural variability of the North Atlantic sea surface temperature on the interannual timescale in the pre-industrial control simulation of the Community Earth System Model. While previous works have considered the use of recurrent networks such as convolutional LSTMs and reservoir computing networks in this and other similar problem settings, we currently focus on the use of feedforward convolutional networks. In particular, we find that a feedforward convolutional network with a Densenet architecture is able to outperform a convolutional LSTM in terms of predictive skill. Next, we go on to consider a probabilistic formulation of the same network based on Stein variational gradient descent and find that in addition to providing useful measures of predictive uncertainty, the probabilistic (Bayesian) version improves on its deterministic counterpart in terms of predictive skill. Finally, we characterize the reliability of the ensemble of machine learning models obtained in the probabilistic setting by using analysis tools developed in the context of ensemble numerical weather prediction.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Deep convolutional neural networks for multi-scale time-series classification and application to tokamak disruption prediction using raw, high temporal resolution diagnostic data

In this paper we discuss recent advances in deep convolutional neural networks (CNN) for sequence learning, which allow identifying long-range, multi-scale phenomena in long sequences, such as those found in fusion plasmas. We point out several benefits of these deep CNN architectures, such as not requiring experts such as physicists to hand-craft input data features, the ability to capture longer range dependencies compared to the more common sequence neural networks (recurrent neural networks like long short-term memory (LSTM) networks), and the comparative computational efficiency. We apply this neural network architecture to the popular problem of disruption prediction in fusion energy tokamaks, utilizing raw data from a single diagnostic, the Electron Cyclotron Emission imaging (ECEi) diagnostic from the DIII-D tokamak. Initial results trained on a large ECEi dataset show promise, achieving an F 1 -score of ~91% on individual time-slices using only the ECEi data. This indicates the ECEi diagnostic by itself can be sensitive to a number of pre-disruption markers useful for predicting disruptions on timescales not only for mitigation but also avoidance. Future opportunities for utilizing these deep CNN architectures with fusion data are outlined, including impact of recent upgrades to the ECEi diagnostic.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗