Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Training Time”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Dimension-adaptive machine learning-based quantum state reconstruction

Here, we introduce an approach for performing quantum state reconstruction on systems of n qubits using a machine learning-based reconstruction system trained exclusively on m qubits, where m ≥ n. This approach removes the necessity of exactly matching the dimensionality of a system under consideration with the dimension of a model used for training. We demonstrate our technique by performing quantum state reconstruction on randomly sampled systems of one, two, and three qubits using machine learning-based methods trained exclusively on systems containing at least one additional qubit. The reconstruction time required for machine learning-based methods scales significantly more favorably than the training time; hence this technique can offer an overall saving of resources by leveraging a single neural network for dimension-variable state reconstruction, obviating the need to train dedicated machine learning systems for each Hilbert space.

42 ENGINEERING↗
DOE OSTI2024

IMS Rapid Response 2024 Summary Report: A Machine Learning Potential for the Periodic Table

Stockpile stewardship and nuclear waste remediation are inherently chemically complex, involving practically the full diversity of the periodic table, but existing methods are too expensive or not functional for a large diversity of atom types. Overall, the field of machine learning interatomic potentials (MLIPs) has advanced dramatically in 2024 with large high-accuracy datasets existing for bulk, surface, and organic chemical systems and new online leaderboards for diverse chemistry. To participate in, and bring LANL interests into this ecosystem, here, we have built upon existing technologies created by LANL to create a framework capable of creating machine learning interatomic potentials (MLIPs) for over 90 atom types. Our results have created a massively diverse coordination complex training dataset more than 3 times the size of existing datasets, parallelized MLIP training over multiple GPUs, enabling the training of an MLIP spanning the periodic table at 20 times the speed of prior training on 32 GPUs. These advances are substantial towards creation on foundational MLIPs for LANL-specific application areas.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2025

Adiabatic quantum support vector machines

Adiabatic quantum computers can solve difficult optimization problems (e.g., the quadratic unconstrained binary optimization problem), and they seem well suited to train machine learning models. In this paper, we describe an adiabatic quantum approach for training support vector machines. We show that the time complexity of our quantum approach is an order of magnitude better than the classical approach. Next, we compare the test accuracy of our quantum approach against a classical approach that uses the Scikit-learn library in Python across five benchmark datasets (Iris, Wisconsin Breast Cancer (WBC), Wine, Digits, and Lambeq). We show that our quantum approach obtains accuracies on par with the classical approach. Finally, we perform a scalability study in which we compute the total training times of the quantum approach and the classical approach with an increasing number of features and an increasing number of data points in the training dataset. In conclusion, our scalability results show that the quantum approach obtains a 3.5–4.5x speedup over the classical approach on datasets with many (millions of) features.

Computational Complexity↗
DOE OSTI2023

Autonomous System Inference, Trojan, and Adversarial Reprogramming Attack and Defense (Final)

In the world of ever-advancing technology, Autonomous Systems (AS) find extensive application, bolstering functionalities of critical infrastructures such as nuclear power plants. These systems, however, are increasingly becoming a target for nefarious activities, namely through inference attacks, trojan attacks, and adversarial reprogramming. This paper delves into a comprehensive exploration of machine learning (ML)-driven autonomous control systems within advanced nuclear reactor designs, revealing the vulnerabilities and proposing strategies for defense against potential cyber-attacks. Advanced cyber-attacks against critical infrastructure and the energy sector are becoming more common. With the invention of autonomous control systems (ACS) within advanced nuclear reactor designs, system designers, reactor operators, and regulators must consider cybersecurity during the design and operational phases. This article provides a cyber threat assessment of machine learning (ML)- based digital twinning (DT) technologies in the context of advanced reactor ACS. A cyber-physical testbed was created to emulate nuclear reactor digital instrumentation and controls (I&C) and act as a basis for the ACS. The ACS was designed as two plant-level DTs predicting reactor malfunctions and determining control actions and two component-level DTs responsible for classifying component states and forecasting component inputs and outputs (I/O). Two duplicate ACS designs– one using a traditional ML framework and one using an automated ML (AutoML) framework– were created and tested against cyber-attacks on training data, real-time process data, and ML model architectures to determine their respective qualitative cyber-risk in terms of likelihood and impact. Both frameworks showed similar cyber-resilience against training, real-time, and ML architecture attacks, proving that neither is inherently more secure. Recommended safeguard and security measures are posed to system designers, reactor operators, and regulators to maintain the cybersecurity of ML-based DT technologies such as ACS, prompting a holistic view of shared responsibility for maintaining cyber-secure ML-based systems. As global reliance on generation III reactors begins to be critically assessed, the evolution towards advanced reactor systems utilizing digital instrumentation and controls (I&C) becomes not merely preferable, but essential. The integration of semi and fully autonomous control systems (ACS), powered by digital I&C and machine learning (ML)-based digital twinning (DT) technologies, emerges as a potent strategy to mitigate operations and maintenance costs, thereby enhancing the economic feasibility of novel reactor designs. However, with a staggering 500% and 380% increase in cyber-attacks reported against the energy sector by the United States Department of Energy (DoE) and the European Union respectively, a surge in cyber vulnerabilities specifically targeting the nuclear industry has been 2 markedly observed. Notable incidents, such as the W32.Ramnit spyware infiltration at the Gundremmingen nuclear power plant in Germany and the Dtrack spyware intrusion at the Kudankulam nuclear power plant in India, while not directly compromising core industrial control systems (ICS), underscore a compelling necessity to fortify cybersecurity protocols in safeguarding reactor systems against increasingly adept digital adversaries. In light of this, our investigation extends beyond conventional cybersecurity parameters, diving into the intricate web of potential vulnerabilities woven into ML-based DTs and ACS in advanced reactor systems. A crafted cyber-physical testbed and preliminary ACS were devised to act as a mirror, reflecting potential configurations of advanced reactor control designs. Moreover, this study is intertwined with a scrutinization of ML models, developed either through conventional, manually tuned methodologies or via automated means through AutoML, probing into their cyber-risk profiles within operational technology (OT) environments. Expanding on this, two distinct ACS blueprints were forged – one navigating through the corridors of traditional ML and the other traversing the path of AutoML – in an effort to holistically encapsulate the considerations pivotal to ML-based DT control system design. Employing the SANS Institute Industrial Control System (ICS) Kill Chain and the MITRE ATT&CK Tactics, Techniques, and Procedures (TTP) framework, a structured analysis was conducted, launching three targeted attacks against the training dataset, real-time dataset, and ML models, therein dissecting the potential cyber-attack implications against both ML frameworks within an ACS milieu. It is essential to note that three distinct categories of attacks were conducted against both ACS configurations, each encompassing three distinct ML-based DTs, cumulating in a total of 18 varied attacks. This exploration extends into the realms of Autonomous System Inference, Trojan, and Adversarial Reprogramming Attack and Defense, unraveling vulnerabilities, and opportunities for fortified defenses against such intrusions, particularly where ML-driven technologies, and by extension, ACS, are deployed. Final recommendations, articulated through a lens of security, safeguard, and implementation considerations, are presented for both traditional and AutoML models, anchoring upon the existing knowledge landscape and ML-based DT modeling for ACS, and are offered as a beacon to guide the nuclear industry through the intricate cybersecurity challenges that lie ahead.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2023

Probing Accuracy-Speedup Tradeoff in Machine Learning Surrogates for Molecular Dynamics Simulations

The performance promise of machine learning surrogates of molecular dynamics simulations of soft materials is significant but generally comes at the cost of acquiring large training datasets to learn the complex relationships between input soft material attributes and output properties. Under the constraint of limited high-performance computing resources, optimizing the size of the training datasets becomes paramount. Using an artificial neural network based surrogate for molecular dynamics simulations of confined electrolytes, we explore the tradeoff between surrogate accuracy and computational gains. Accuracy is assessed by computing the root-mean-square errors between the surrogate predictions and the ground truth results obtained via molecular dynamics simulations. The computational performance is judged by evaluating the speedup which incorporates the training dataset creation time. Improvement in accuracy occurs with a loss of speedup, which scales as the inverse of the training dataset size. Furthermore, the link between surrogate generalizability and the accuracy-speedup tradeoff is assessed by examining the errors incurred in surrogate predictions on unseen, interpolated input variables and developing a net speedup metric to capture the associated gains.

Anions↗
DOE OSTI2025

A Digital Twin Framework Utilizing Machine Learning for Robust Predictive Maintenance: Enhancing Tire Health Monitoring

We introduce a novel digital twin (DT) framework for the predictive maintenance of long-term physical systems. Using monitoring tire health as an application, we show how the DT framework can be used to enhance automotive safety and efficiency, and how the technical challenges can be overcome using a three-step approach. First, to manage the data complexity over a long operation span, we employ data reduction techniques to concisely represent physical tires using historical performance and usage data. Relying on these data, for fast real-time prediction, we train a transformer-based model offline on our concise dataset to predict future tire health over time, represented as remaining casing potential (RCP). Based on our architecture, our model quantifies both epistemic and aleatoric uncertainties, providing reliable confidence intervals around predicted RCP. Second, to incorporate real-time data, we update the predictive model in the DT framework, ensuring its accuracy throughout its lifespan with the aid of hybrid modeling and the use of the discrepancy function. Third, to assist decision-making in predictive maintenance, we implement a tire state decision algorithm, which strategically determines the optimal timing for tire replacement based on RCP forecasted by our transformer model. This approach ensures that our DT accurately predicts system health, continually refines its digital representation, and supports predictive maintenance decisions. Furthermore, our framework effectively embodies a physical system, leveraging big data and machine learning (ML) for predictive maintenance, model updates, and decision-making.

advanced computing infrastructure↗
DOE OSTI2024

Forward variable selection enables fast and accurate dynamic system identification with Karhunen-Loève decomposed Gaussian processes

A promising approach for scalable Gaussian processes (GPs) is the Karhunen-Loève (KL) decomposition, in which the GP kernel is represented by a set of basis functions which are the eigenfunctions of the kernel operator. Such decomposed kernels have the potential to be very fast, and do not depend on the selection of a reduced set of inducing points. However KL decompositions lead to high dimensionality, and variable selection thus becomes paramount. This paper reports a new method of forward variable selection, enabled by the ordered nature of the basis functions in the KL expansion of the Bayesian Smoothing Spline ANOVA kernel (BSS-ANOVA), coupled with fast Gibbs sampling in a fully Bayesian approach. It quickly and effectively limits the number of terms, yielding a method with competitive accuracies, training and inference times for tabular datasets of low feature set dimensionality. Theoretical computational complexities are O ( N P 2 ) in training and O ( P ) per point in inference, where N is the number of instances and P the number of expansion terms. The inference speed and accuracy makes the method especially useful for dynamic systems identification, by modeling the dynamics in the tangent space as a static problem, then integrating the learned dynamics using a high-order scheme. The methods are demonstrated on two dynamic datasets: a ‘Susceptible, Infected, Recovered’ (SIR) toy problem, along with the experimental ‘Cascaded Tanks’ benchmark dataset. Comparisons on the static prediction of time derivatives are made with a random forest (RF), a residual neural network (ResNet), and the Orthogonal Additive Kernel (OAK) inducing points scalable GP, while for the timeseries prediction comparisons are made with LSTM and GRU recurrent neural networks (RNNs) along with the SINDy package.

Hayes, Kyle↗
DOE OSTI2021

Deep Learning Approaches to Surrogates for Solving the Diffusion Equation for Mechanistic Real-World Simulations

In many mechanistic medical, biological, physical, and engineered spatiotemporal dynamic models the numerical solution of partial differential equations (PDEs), especially for diffusion, fluid flow and mechanical relaxation, can make simulations impractically slow. Biological models of tissues and organs often require the simultaneous calculation of the spatial variation of concentration of dozens of diffusing chemical species. One clinical example where rapid calculation of a diffusing field is of use is the estimation of oxygen gradients in the retina, based on imaging of the retinal vasculature, to guide surgical interventions in diabetic retinopathy. Furthermore, the ability to predict blood perfusion and oxygenation may one day guide clinical interventions in diverse settings, i.e., from stent placement in treating heart disease to BOLD fMRI interpretation in evaluating cognitive function (Xie et al., 2019; Lee et al., 2020). Since the quasi-steady-state solutions required for fast-diffusing chemical species like oxygen are particularly computationally costly, we consider the use of a neural network to provide an approximate solution to the steady-state diffusion equation. Machine learning surrogates, neural networks trained to provide approximate solutions to such complicated numerical problems, can often provide speed-ups of several orders of magnitude compared to direct calculation. Surrogates of PDEs could enable use of larger and more detailed models than are possible with direct calculation and can make including such simulations in real-time or near-real time workflows practical. Creating a surrogate requires running the direct calculation tens of thousands of times to generate training data and then training the neural network, both of which are computationally expensive. Often the practical applications of such models require thousands to millions of replica simulations, for example for parameter identification and uncertainty quantification, each of which gains speed from surrogate use and rapidly recovers the up-front costs of surrogate generation. We use a Convolutional Neural Network to approximate the stationary solution to the diffusion equation in the case of two equal-diameter, circular, constant-value sources located at random positions in a two-dimensional square domain with absorbing boundary conditions. Such a configuration caricatures the chemical concentration field of a fast-diffusing species like oxygen in a tissue with two parallel blood vessels in a cross section perpendicular to the two blood vessels. To improve convergence during training, we apply a training approach that uses roll-back to reject stochastic changes to the network that increase the loss function. The trained neural network approximation is about 1000 times faster than the direct calculation for individual replicas. Because different applications will have different criteria for acceptable approximation accuracy, we discuss a variety of loss functions and accuracy estimators that can help select the best network for a particular application. We briefly discuss some of the issues we encountered with overfitting, mismapping of the field values and the geometrical conditions that lead to large absolute and relative errors in the approximate solution.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2023

Simulating the metabolic pathway dynamics of an organism

Disclosed herein are systems and methods for determining metabolic pathway dynamics using time series multiomics data. In one example, after receiving time series multiomics data comprising time-series metabolomics data associated a metabolic pathway and time-series proteomics data associated with the metabolic pathway, derivatives of the time series multiomics data can be determined. A machine learning model, representing a metabolic pathway dynamics model, can be trained using the time series multiomics data and the derivatives of the time series multiomics data, wherein the metabolic pathway dynamics model relates the time-series metabolomics data and time-series proteomics data to the derivatives of the time series multiomics data. The method can include simulating a virtual strain of the organism using the metabolic pathway dynamics model.

Costello, Zachary↗
DOE OSTI2023

Development of Multiresolution Capabilities for the Holistic Energy Resource Optimization Network (HERON) tool A progress update

INL researchers work on technoeconomic analyses for integrated energy systems (IES) using the Framework for Optimization of ResourCes and Economics (FORCE). Within FORCE, researchers use the Holistic Energy Resource Optimization Network (HERON) tool to conduct optimization of grid portfolios under uncertain market conditions. These optimizations determine optimal capacities for all IES components and strategies for resource dispatch which maximize some economic metric (e.g., net present value). Resource dispatch occurs on finer timescales (typically hours) and thus are asked to respond to a given time series (e.g. hourly load demand profiles for a grid, or pre-determined electricity prices). Volatile and complex bidding dynamics as well as poorly forecasted weather events within deregulated markets add uncertainty to the time series; FORCE can address this uncertainty by training a reduced order model on historical time series and generate unique synthetic time series which represent individual scenarios or realizations of the market. The IES configuration can be simulated under these different sampled realizations and a stochastic optimization is conducted which optimizes the expected value of the desired economic metric. The training of a synthetic time series generator is limited by the chosen time resolution; dynamics can occur on different time scales. Seasonal demand trends can dominate faster dynamical events (such as power outages from certain sectors or severe weather events) which might not get captured correctly by the trained model. In this report, we investigate different ways of addressing the training and generation of time series on multiple time scales using three main algorithms: wavelet decomposition, dynamic mode decomposition, and generative adversarial networks for time series. We demonstrate a time series analysis that yields information on not just the frequency space but also temporal space: where a fast Fourier transform can provide what frequencies dominate, the new algorithms can provide when the frequencies dominate as well. These analyses can help improve IES optimization by allowing researchers to couple simulations at different timescales when it is most needed - seasonal, day-ahead, and real time optimization - with greater computational efficiency. Future work will include implementation of a subset of the proposed algorithms into the FORCE toolset and application of these analyses into multiple timescale optimization.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2023

Cyber threat assessment of machine learning driven autonomous control systems of nuclear power plants

We report advanced cyber-attacks against critical infrastructure and the energy sector are becoming more common. With the invention of autonomous control systems (ACS) within advanced nuclear reactor designs, system designers, reactor operators, and regulators must consider cybersecurity during the design and operational phases. This article provides a cyber threat assessment of machine learning (ML)-based digital twinning (DT) technologies in the context of advanced reactor ACS. A cyber–physical testbed was created to emulate nuclear reactor digital instrumentation and controls (I&C) and act as a basis for the ACS. The ACS was designed as two plant-level DTs predicting reactor malfunctions and determining control actions and two component-level DTs responsible for classifying component states and forecasting component inputs and outputs (I/O). Two duplicate ACS designs– one using a traditional ML framework and one using an automated ML (AutoML) framework– were created and tested against cyber-attacks on training data, real-time process data, and ML model architectures to determine their respective qualitative cyber-risk in terms of likelihood and impact. Both frameworks showed similar cyber-resilience against training, real-time, and ML architecture attacks, proving that neither is inherently more secure. Recommended safeguard and security measures are posed to system designers, reactor operators, and regulators to maintain the cybersecurity of ML-based DT technologies such as ACS, prompting a holistic view of shared responsibility for maintaining cyber-secure ML-based systems.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2021

Differentiable programming for online training of a neural artificial viscosity function within a staggered grid Lagrangian hydrodynamics scheme

Lagrangian methods to solve the inviscid Euler equations produce numerical oscillations near shock waves. A common approach to reducing these oscillations is to add artificial viscosity (AV) to the discrete equations. The AV term acts as a dissipative mechanism that attenuates oscillations by smearing the shock across a finite number of computational cells. However, AV introduces several control parameters that are not determined by the underlying physical model, and hence, in practice are tuned to the characteristics of a given problem. We seek to improve the standard quadratic-linear AV form by replacing it with a learned neural function that reduces oscillations relative to exact solutions of the Euler equations, resulting in a hybrid numerical-neural hydrodynamic solver. Because AV is an artificial construct that exists solely to improve the numerical properties of a hydrodynamic code, there is no offline ‘viscosity data’ against which a neural network can be trained before inserting into a numerical simulation, thus requiring online training. We achieve this via differentiable programming, i.e. end-to-end backpropagation or adjoint solution through both the neural and differential equation code, using automatic differentiation of the hybrid code in the Julia programming language to calculate the necessary loss function gradients. A novel offline pre-training step accelerates training by initializing the neural network to the default numerical AV scheme, which can be learned rapidly by space-filling sampling over the AV input space. We find that online training over early time steps of simulation is sufficient to learn a neural AV function that reduces numerical oscillations in long-term hydrodynamic shock simulations. These results offer an early proof-of-principle that online differentiable training of hybrid numerical schemes with novel neural network components can improve certain performance aspects existing in purely numerical schemes.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Improved machine learning algorithm for predicting ground state properties

Finding the ground state of a quantum many-body system is a fundamental problem in quantum physics. In this work, we give a classical machine learning (ML) algorithm for predicting ground state properties with an inductive bias encoding geometric locality. The proposed ML model can efficiently predict ground state properties of an n-qubit gapped local Hamiltonian after learning from only $\mathcal{O}$(log(n)) data about other Hamiltonians in the same quantum phase of matter. This improves substantially upon previous results that require $\mathcal{O}$(n c ) data for a large constant c. Furthermore, the training and prediction time of the proposed ML model scale as $\mathcal{O}$(n log n) in the number of qubits n. Numerical experiments on physical systems with up to 45 qubits confirm the favorable scaling in predicting ground state properties using a small training dataset.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2023

Highly transferable atomistic machine-learning potentials from curated and compact datasets across the periodic table

Machine learning atomistic potentials trained using density functional theory (DFT) datasets allow for the modeling of complex material properties with near-DFT accuracy while imposing a fraction of its computational cost. The curation of the DFT datasets can be extensive in size and time-consuming to train and refine. In this study, we focus on addressing these barriers by developing minimalistic and flexible datasets for many elements in the periodic table regardless of their mass, electronic configuration, and ground state lattice. These DFT datasets have on average, ~4000 different structures and 27 atoms per structure, which we found sufficient to maintain the predictive accuracy of DFT properties and notably with high transferability. We envision these highly curated training sets as starting points for the community to expand, modify, or use with other machine learning atomistic potential models, whatever may suit individual needs, further accelerating the utilization of machine learning as a tool for material design and discovery.

42 ENGINEERING↗
DOE OSTI2023

An Ensemble Score Filter for Tracking High-Dimensional Nonlinear Dynamical Systems

We propose an ensemble score filter (EnSF) for solving high-dimensional nonlinear filtering problems with superior accuracy. A major drawback of existing filtering methods, e.g., particle filters or ensemble Kalman filters, is the low accuracy in handling high-dimensional and highly nonlinear problems. EnSF attacks this challenge by exploiting the score-based diffusion model, defined in a pseudo-temporal domain, to characterizing the evolution of the filtering density. EnSF stores the information of the recursively updated filtering density function in the score function, in stead of storing the information in a set of finite Monte Carlo samples (used in particle filters and ensemble Kalman filters). Unlike existing diffusion models that train neural networks to approximate the score function, we develop a training-free score estimation that uses mini-batch-based Monte Carlo estimator to directly approximate the score function at any pseudo-spatial-temporal location, which provides sufficient accuracy in solving high-dimensional nonlinear problems as well as saves tremendous amount of time spent on training neural networks. Another essential aspect of EnSF is its analytical update step, gradually incorporating data information into the score function, which is crucial in mitigating the degeneracy issue faced when dealing with very high-dimensional nonlinear filtering problems. High-dimensional Lorenz systems are used to demonstrate the performance of our method. EnSF provides surprisingly impressive performance in reliably tracking extremely high-dimensional Lorenz systems (up to 1,000,000 dimension) with highly nonlinear observation processes, which is a well-known challenging problem for existing filtering methods.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Cy-Phy ADS: Cyber Physical Anomaly Detection Framework for EV Charging Systems

Today’s large-scale Electric Vehicle (EV) infrastructures are heavily dependent on information communication technologies to maintain their operation and to support communication within sub-system components as well as the outside world. These technologies are vulnerable to various cyber and physical threats. Timely identification and mitigation of these threats are critical for improving human safety, avoiding economic losses, and preventing catastrophic system failures. By addressing this, our work presents a ResNet Autoencoder (AE) based Cyber-Physical Anomaly Detection System (Cy-Phy ADS) for detecting anomalies in EV Controller Area Network (CAN) protocol communication. It consists of four main components: Cyber-Physical Feature Extractor, ResNet AE-based Anomaly Detection Framework, Cyber-Physical Health Metric (CPHM), and Visualization Dashboard. The presented framework was trained and tested using CAN data collected from the EV charging system testbed at the Idaho National Laboratory. The presented Cy-Phy ADS compared against six widely used unsupervised anomaly detection algorithms: One Class Support Vector Machine (OCSVM), Variational Autoencoder (VAE), LSTM Autoencoder (LSTM AE), Isolation Forest (IForest), Principle Component Analysis (PCA) and Local Outlier Factor (LOF). Here the presented approach showed the highest accuracy among the compared methods. Further, the proposed approach showed comparable performance in terms of precision, F1, and False positive rate. It also showed the lowest training and inference time compared to the neural network-based baseline algorithms compared against with. Additionally, the Cy-Phy ADS has advantages such as unsupervised training, the ability to provide a holistic metric for system health characterization, and non-linear feature extraction.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2024

A comparison of model validation approaches for echo state networks using climate model replicates

As global temperatures continue to rise, climate mitigation strategies such as stratospheric aerosol injections (SAI) are increasingly discussed, but the downstream effects of these strategies are not well understood. As such, there is interest in developing statistical methods to quantify the evolution of climate variable relationships during the time period surrounding an SAI. Feature importance applied to echo state network (ESN) models has been proposed as a way to understand the effects of SAI using a data-driven model. This approach depends on the ESN fitting the data well. If not, the feature importance may place importance on features that are not representative of the underlying relationships. Typically, time series prediction models such as ESNs are assessed using out-of-sample performance metrics that divide the times series into separate training and testing sets. However, this model assessment approach is geared towards forecasting applications and not scenarios such as the motivating SAI example where the objective is using a data driven model to capture variable relationships. Here, in this paper, we demonstrate a novel use of climate model replicates to investigate the applicability of the commonly used repeated hold-out model assessment approach for the SAI application. Simulations of an SAI are generated using a simplified climate model, and different initialization conditions are used to provide independent training and testing sets containing the same SAI event. The climate model replicates enable out-of-sample measures of model performance, which are compared to the single time series hold-out validation approach. For our case study, it is found that the repeated hold-out sample performance is comparable, but conservative, to the replicate out-of-sample performance when the training set contains enough time after the aerosol injection.

54 ENVIRONMENTAL SCIENCES↗