SEARCH · Engineering Papers
Results for “Automatic Differentiation”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Learning and meta-learning of stochastic advection–diffusion–reaction systems from sparse measurements
Physics-informed neural networks (PINNs) were recently proposed in [18] as an alternative way to solve partial differential equations (PDEs). A neural network (NN) represents the solution, while a PDE-induced NN is coupled to the solution NN, and all differential operators are treated using automatic differentiation. Here, we first employ the standard PINN and a stochastic version, sPINN, to solve forward and inverse problems governed by a non-linear advection–diffusion–reaction (ADR) equation, assuming we have some sparse measurements of the concentration field at random or pre-selected locations. Subsequently, we attempt to optimise the hyper-parameters of sPINN by using the Bayesian optimisation method (meta-learning) and compare the results with the empirically selected hyper-parameters of sPINN. In particular, for the first part in solving the inverse deterministic ADR, we assume that we only have a few high-fidelity measurements, whereas the rest of the data is of lower fidelity. Hence, the PINN is trained using a composite multi-fidelity network, first introduced in [12], that learns the correlations between the multi-fidelity data and predicts the unknown values of diffusivity, transport velocity and two reaction constants as well as the concentration field. For the stochastic ADR, we employ a Karhunen–Loève (KL) expansion to represent the stochastic diffusivity, and arbitrary polynomial chaos (aPC) to represent the stochastic solution. Correspondingly, we design multiple NNs to represent the mean of the solution and learn each aPC mode separately, whereas we employ a separate NN to represent the mean of diffusivity and another NN to learn all modes of the KL expansion. For the inverse problem, in addition to stochastic diffusivity and concentration fields, we also aim to obtain the (unknown) deterministic values of transport velocity and reaction constants. The available data correspond to 7spatial points for the diffusivity and 20 space–time points for the solution, both sampled 2000 times. We obtain good accuracy for the deterministic parameters of the order of 1–2% and excellent accuracy for the mean and variance of the stochastic fields, better than three digits of accuracy. In the second part, we consider the previous stochastic inverse problem, and we use Bayesian optimisation to find five hyper-parameters of sPINN, namely the width, depth and learning rate of two NNs for learning the modes. Here, we obtain much deeper and wider optimal NNs compared to the manual tuning, leading to even better accuracy, i.e., errors less than 1% for the deterministic values, and about an order of magnitude less for the stochastic fields.
Spatiotemporal control of laser intensity using differentiable programming
Optical techniques for spatiotemporal control can produce laser pulses with custom amplitude, phase, or polarization structure. In nonlinear optics and plasma physics, the use of structured pulses typically follows a forward design approach, in which the efficacy of a known structure is analyzed for a particular application. Inverse approaches, in contrast, enable the discovery of new structures with the potential for superior performance. Here, an implementation of the unidirectional pulse propagation equation that supports automatic differentiation is combined with gradient-based optimization to design structured pulses with features that are advantageous for a range of nonlinear optical and plasma-based applications: (1) a longitudinally uniform intensity over an extended region, (2) a superluminal intensity peak that travels many Rayleigh ranges with constant duration, spot size, and amplitude, and (3) a laser pulse that ionizes a gas to form a uniform column of plasma. In the final case, optimizing the full spatiotemporal structure improves the performance by a factor of 15 compared to optimizing only spatial or only temporal structure, highlighting the advantage of spatiotemporal control.
Dynamic energy system modeling using hybrid physics-based and machine learning encoder–decoder models
Three model configurations are presented for multi-step time series predictions of the heat absorbed by the water and steam in a thermal power plant. The models predict over horizons of 2, 4, and 6 steps into the future, where each step is a 5-minute increment. The evaluated models are a pure machine learning model, a novel hybrid machine learning and physics-based model, and the hybrid model with an incomplete dataset. The hybrid model deconstructs the machine learning into individual boiler heat absorption units: economizer, water wall, superheater, and reheater. Each configuration uses a gated recurrent unit (GRU) or a GRU-based encoder–decoder as the deep learning architecture. Mean squared error is used to evaluate the models compared to target values. The encoder–decoder architecture is over 11% more accurate than the GRU only models. The hybrid model with the incomplete dataset highlights the importance of the manipulated variables to the system. The hybrid model, compared to the pure machine learning model, is over 10% more accurate on average over 20 iterations of each model. Automatic differentiation is applied to the hybrid model to perform a local sensitivity analysis to identify the most impactful of the 72 manipulated variables on the heat absorbed in the boiler. The models and sensitivity analyses are used in a discussion about optimizing the thermal power plant.
A comparative study of calibration techniques for finite strain elastoplasticity: Numerically-exact sensitivities for FEMU and VFM
Accurate identification of material parameters is crucial for predictive modeling in computational mechanics. Here, the two primary approaches in the experimental mechanics community for calibration from full-field digital image correlation data are known as finite element model updating (FEMU) and the virtual fields method (VFM). In VFM, the objective function is a squared mismatch between internal and external virtual work or power. In FEMU, the objective function quantifies the weighted mismatch between model predictions and corresponding experimentally measured quantities of interest. It is minimized by iteratively updating the parameters of an FE model. While FEMU is seen as more flexible, VFM is commonly used instead of FEMU due to its considerably greater computational expense. However, comparisons between the two methods usually involve approximations of gradients or sensitivities with finite difference schemes, thereby making direct assessments difficult. Hence, in this study, we compare VFM and FEMU in the context of numerically-exact sensitivities obtained through local sensitivity analyses and the application of automatic differentiation software. To this end, we conduct a series of test cases to assess both methods under practical challenges using a finite strain elastoplasticity model.
A direct-adjoint approach for material point model calibration with application to plasticity
Here, this paper proposes a new approach for the calibration of material parameters in local elastoplastic constitutive models. The calibration is posed as a constrained optimization problem, where the constitutive model evolution equations for a single material point serve as constraints. The objective function quantifies the mismatch between the stress predicted by the model and corresponding experimental measurements. To improve calibration efficiency, a novel direct-adjoint approach is presented to compute the Hessian of the objective function, which enables the use of second-order optimization algorithms. Automatic differentiation is used for gradient and Hessian computations. Two numerical examples are employed to validate the Hessian matrices and to demonstrate that the Newton–Raphson algorithm consistently outperforms gradient-based algorithms such as L-BFGS-B.
A novel implicit hybrid machine learning model and its application for reinforcement learning
A novel methodology to develop implicit hybrid models is presented. PyTorch is used to integrate physics-based equations with machine learning models. Automatic differentiation of the hybrid model is leveraged to solve the implicit equations. Iterative solving enables gradient based updates to the machine learning model. The novel methodology is compared to an explicit hybrid approach on a continuously stirred tank reactor (CSTR). The novel method results in a lower modelling error. Both hybrid models effectively train with noisy data. To test the implicit hybrid model, it is employed as a reinforcement learning (RL) training model. The RL algorithm trained on the hybrid model outperforms real time optimization of the CSTR and performs nearly as well as RL trained directly on the CSTR and a traditional gradient based approach. Training RL directly on the CSTR requires over 60,000 system interactions compared to 6000 historical data points for hybrid model development.
Prediction and uncertainty quantification of shale well performance using multifidelity Monte Carlo
Uncertainty quantification is an integral component of reservoir management, especially considering the inherent uncertainty in subsurface systems. While a standard practice to estimate the uncertainty, Monte Carlo (MC) simulation is computationally intense when the sampling population comprises high-fidelity simulations. Alternatively, the Multi-fidelity Monte Carlo (MFMC) simulation overcomes this computational intensity by integrating low- and high-fidelity simulations. Our goal is to minimize the number of expensive high-fidelity simulations while maintaining accuracy and using numerous fast and cheap low-fidelity simulations to efficiently sample to input parameter space of interest. We selected gas production from unconventional wells to demonstrate the potential speedups and accuracy of the MFMC approach. The model fidelity usually determines the trade-off between accuracy and efficiency. While the high-fidelity model is more accurate, the low-fidelity model is more efficient. Our high-fidelity simulation includes reservoir simulations of a hydraulically fractured well. On the other hand, our low-fidelity model comprises the parallel-plate flow model. We used differential programming to efficiently solve the 1D flow model, where automatic differentiation is used to efficiently compute the gradients. We matched the production profile of high-fidelity simulations with our low-fidelity simulations. Then, we used a support vector regression to map the high- and low-fidelity input parameters. The mapping function is essential to tune the low-dimensional parameter space of the low-fidelity model to the high-dimensional parameter space of the high-fidelity model. We found that we can use a combination of 9 high fidelity and 10,000 low fidelity simulations to efficiently and accurately simulate pressure management. This method is at least two orders of magnitude faster than only using high-fidelity simulations. Finally, from a broader perspective, MFMC could efficiently estimate the uncertainty of various systems and models, integrating low- and high-fidelity models.
Spectrally accurate, reverse-mode differentiable bounce-averaging algorithm and its applications
We present a fast, spectrally (exponentially) accurate, automatically differentiable bounce-averaging algorithm that is used to simplify kinetic models. Using this algorithm, implemented in the DESC stellarator optimisation suite, we can perform efficient optimisation of many objectives to improve stellarator performance, such as the effective ripple 𝜖 eff metric for the neoclassical transport coefficient in the low collisionality regime and proxies for energetic particle confinement. For the first time, we optimise a finite-beta stellarator to directly reduce neoclassical ripple transport using reverse-mode differentiation. This ensures the computational cost of differentiation is independent of the number of controllable parameters.
Physics constrained learning for data-driven inverse modeling from sparse observations
Deep neural networks (DNN) have been used to model nonlinear relations between physical quantities. Those DNNs are embedded in physical systems described by partial differential equations (PDE) and trained by minimizing a loss function that measures the discrepancy between predictions and observations in some chosen norm. This loss function often includes the PDE constraints as a penalty term when only sparse observations are available. As a result, the PDE is only satisfied approximately by the solution. However, the penalty term typically slows down the convergence of the optimizer for stiff problems. We present a new approach that trains the embedded DNNs while numerically satisfying the PDE constraints. We develop an algorithm that enables differentiating both explicit and implicit numerical solvers in reverse-mode automatic differentiation. This allows the gradients of the DNNs and the PDE solvers to be computed in a unified framework. We demonstrate that our approach enjoys faster convergence and better stability in relatively stiff problems compared to the penalty method. Furthermore, our approach allows for the potential to solve and accelerate a wide range of data-driven inverse modeling, where the physical constraints are described by PDEs and need to be satisfied accurately.
Reverse-mode differentiation in arbitrary tensor network format: with application to supervised learning.
This paper describes an efficient reverse-mode differentiation algorithm for contraction operations of tensor networks that may have arbitrary and unconventional network topologies. The approach leverages the tensor contraction tree of Evenbly and Pfeifer (2014), which provides an instruction set for the contraction sequence of a network. We show that this tree can be efficiently leveraged for differentiation of a full tensor network contraction using a recursive scheme that exploits (1) the bilinear property of contraction and (2) the property that trees have single path from root to leaves. While differentiation of tensor-tensor contraction is already possible in most automatic differentiation packages, we show that exploiting these two additional properties in the specific context of contraction sequences can improve efficiency. Following a description of the algorithm and computational complexity analysis, we investigate its utility for gradient-based supervised learning for low-rank function recovery and for fitting real-world unstructured datasets. We demonstrate improved performance over alternating least-squares optimization approaches and the capability to handle heterogeneous and arbitrary tensor network formats. When compared to alternating minimization algorithms, we find that the gradient-based approach requires a smaller oversampling ratio (number of samples compared to number model parameters) for recovery. This increased efficiency extends to fitting unstructured data of varying dimensionality and when employing a variety of tensor network formats. Here, we show improved learning using the hierarchical Tucker method over the tensor-train in high-dimensional settings on a number of benchmark problems.
Differentiable Quantum Programming with Unbounded Loops
The emergence of variational quantum applications has led to the development of automatic differentiation techniques in quantum computing. Existing work has formulated differentiable quantum programming with bounded loops, providing a framework for scalable gradient calculation by quantum means for training quantum variational applications. However, promising parameterized quantum applications, e.g., quantum walk and unitary implementation, cannot be trained in the existing framework due to the natural involvement of unbounded loops. To fill in the gap, we provide the first differentiable quantum programming framework with unbounded loops, including a newly designed differentiation rule, code transformation, and their correctness proof. Technically, we introduce a randomized estimator for derivatives to deal with the infinite sum in the differentiation of unbounded loops, whose applicability in classical and probabilistic programming is also discussed. We implement our framework with Python and Q# and demonstrate a reasonable sample efficiency. Through extensive case studies, we showcase an exciting application of our framework in automatically identifying close-to-optimal parameters for several parameterized quantum applications.
A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks
Physics-informed neural networks (PINNs) have shown to be effective tools for solving both forward and inverse problems of partial differential equations (PDEs). PINNs embed the PDEs into the loss of the neural network using automatic differentiation, and this PDE loss is evaluated at a set of scattered spatio-temporal points (called residual points). The location and distribution of these residual points are highly important to the performance of PINNs. However, in the existing studies on PINNs, only a few simple residual point sampling methods have mainly been used. Here, we present a comprehensive study of two categories of sampling for PINNs: non-adaptive uniform sampling and adaptive nonuniform sampling. We consider six uniform sampling methods, including (1) equispaced uniform grid, (2) uniformly random sampling, (3) Latin hypercube sampling, (4) Halton sequence, (5) Hammersley sequence, and (6) Sobol sequence. We also consider a resampling strategy for uniform sampling. To improve the sampling efficiency and the accuracy of PINNs, we propose two new residual-based adaptive sampling methods: residual-based adaptive distribution (RAD) and residual-based adaptive refinement with distribution (RAR-D), which dynamically improve the distribution of residual points based on the PDE residuals during training. Hence, we have considered a total of 10 different sampling methods, including six non-adaptive uniform sampling, uniform sampling with resampling, two proposed adaptive sampling, and an existing adaptive sampling. We extensively tested the performance of these sampling methods for four forward problems and two inverse problems in many setups. Our numerical results presented in this study are summarized from more than 6000 simulations of PINNs. Here, we show that the proposed adaptive sampling methods of RAD and RAR-D significantly improve the accuracy of PINNs with fewer residual points for both forward and inverse problems. Furthermore, the results obtained in this study can also be used as a practical guideline in choosing sampling methods.
Integrating deep neural networks with full-waveform inversion: Reparameterization, regularization, and uncertainty quantification
Full-waveform inversion (FWI) is an accurate imaging approach for modeling the velocity structure by minimizing the misfit between recorded and predicted seismic waveforms. However, the strong nonlinearity of FWI resulting from fitting oscillatory waveforms can trap the optimization in local minima. We have adopted a neural-network-based full-waveform inversion (NNFWI) method that integrates deep neural networks with FWI by representing the velocity model with a generative neural network. Neural networks can naturally introduce spatial correlations as regularization to the generated velocity model, which suppresses noise in the gradients and mitigates local minima. Furthermore, the velocity model generated by neural networks is input to the same partial differential equation (PDE) solvers used in conventional FWI. The gradients of the neural networks and PDEs are calculated using automatic differentiation, which back propagates gradients through the acoustic PDEs and neural network layers to update the weights of the generative neural network. Experiments on 1D velocity models, the Marmousi model, and the 2004 BP model determine that NNFWI can mitigate local minima, especially for imaging high-contrast features such as salt bodies, and it significantly improves the inversion in the presence of noise. Adding dropout layers to the neural network model also allows analyzing the uncertainty of the inversion results through Monte Carlo dropout. NNFWI opens a new pathway to combine deep learning and FWI for exploiting the characteristics of deep neural networks and the high accuracy of PDE solvers. Because NNFWI does not require extra training data and optimization loops, it provides an attractive and straightforward alternative to conventional FWI.
Efficient high-dimensional variational data assimilation with machine-learned reduced-order models
Abstract. Data assimilation (DA) in geophysical sciences remains the cornerstone of robust forecasts from numerical models. Indeed, DA plays a crucial role in the quality of numerical weather prediction and is a crucial building block that has allowed dramatic improvements in weather forecasting over the past few decades. DA is commonly framed in a variational setting, where one solves an optimization problem within a Bayesian formulation using raw model forecasts as a prior and observations as likelihood. This leads to a DA objective function that needs to be minimized, where the decision variables are the initial conditions specified to the model. In traditional DA, the forward model is numerically and computationally expensive. Here we replace the forward model with a low-dimensional, data-driven, and differentiable emulator. Consequently, gradients of our DA objective function with respect to the decision variables are obtained rapidly via automatic differentiation. We demonstrate our approach by performing an emulator-assisted DA forecast of geopotential height. Our results indicate that emulator-assisted DA is faster than traditional equation-based DA forecasts by 4 orders of magnitude, allowing computations to be performed on a workstation rather than a dedicated high-performance computer. In addition, we describe accuracy benefits of emulator-assisted DA when compared to simply using the emulator for forecasting (i.e., without DA). Our overall formulation is denoted AIEADA (Artificial Intelligence Emulator-Assisted Data Assimilation).
pnnl/JAX-CanVeg
Differentiable land surface model reimplementing an existing simulator, CANOAK, in JAX—a Google-developed Python package for high-performance machine learning research using automatic differentiation. The model's purpose is to perform hybrid land surface modeling that seamlessly couples process-based components with deep neural networks
CHEMNODE: CHEMICAL KINETICS SOLVER APPROACH BASED ON NEURAL ORDINARY DIFFERENTIAL EQUATIONS
This software contains an algorithm to model and predict the time evolution of chemical kinetics in reacting flow simulations using neural ordinary differential equations (NODEs). The approach works by using artificial neural networks to predict the chemical source terms. It achieves this by using forward-mode automatic differentiation and the Levenberg-Marquardt algorithm to adjust the neural network parameters, such that the discrepancies between the actual and predicted species profiles are minimized.
JAX-CanVeg: A Differentiable Land Surface Model
Land surface models consider the exchange of water, energy, and carbon along the soil-canopy-atmosphere continuum, which is challenging to model due to their complex interdependency and associated challenges in representing and parameterizing them. Differentiable modeling provides a new opportunity to capture these complex interactions by seamlessly hybridizing process-based models with deep neural networks (DNNs), benefiting both worlds, that is, the physical interpretation of process-based models and the learning power of DNNs. Here, we developed a differentiable land model, JAX-CanVeg. The new model builds on the legacy CanVeg by incorporating advanced functionalities through JAX in the graphic processing unit support, automatic differentiation, and integration with DNNs. We demonstrated JAX-CanVeg's hybrid modeling capability by applying the model at four flux tower sites with varying aridity. To this end, we developed a hybrid version of the Ball-Berry equation that emulates the water stress impact on stomatal closure to explore the capability of the hybrid model in (a) improving the simulations of latent heat fluxes (LE) and net ecosystem exchange (NEE), (b) improving the optimization trade-off when learning observations of both LE and NEE, and (c) benefiting a multi-layer canopy model setup. Our results show that the proposed hybrid model improved the simulations of LE and NEE at all sites, with an improved optimization trade-off over the process-based model. Additionally, the multi-layer canopy set benefited hybrid modeling at some sites. Anchored in differentiable modeling, our study provides a new avenue for modeling land-atmosphere interactions by leveraging the benefits of both data-driven learning and process-based modeling.