SEARCH · Engineering Papers
Results for “gradient methods”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Optimizing the optimizer for physics-informed neural networks and Kolmogorov-Arnold networks
Physics-Informed Neural Networks (PINNs) have revolutionized the computation of PDE solutions by integrating partial differential equations (PDEs) into the neural network’s training process as soft constraints, becoming an important component of the scientific machine learning (SciML) ecosystem. More recently, physics-informed Kolmogorv-Arnold networks (PIKANs) have also shown to be effective and comparable in accuracy with PINNs. In their current implementation, both PINNs and PIKANs are mainly optimized using first-order methods like Adam, as well as quasi-Newton methods such as BFGS and its low-memory variant, L-BFGS. However, these optimizers often struggle with highly nonlinear and non-convex loss landscapes, leading to challenges such as slow convergence, local minima entrapment, and (non)degenerate saddle points. In this study, we investigate the performance of Self- Scaled BFGS (SSBFGS), Self-Scaled Broyden (SSBroyden) methods and other advanced quasi-Newton schemes, including BFGS and L-BFGS with different line search strategies. These methods dynamically rescale updates based on historical gradient information, thus enhancing training efficiency and accuracy. We systematically compare these optimizers – using both PINNs and PIKANs – on key challenging PDEs, including the Burgers, Allen-Cahn, Kuramoto-Sivashinsky, Ginzburg-Landau, and Stokes equations. Additionally, we evaluate the performance of SSBFGS and SSBroyden for Deep Operator Network (DeepONet) architectures, demonstrating their effectiveness for data-driven operator learning. Our findings provide state-of-the-art results with orders-of-magnitude accuracy improvements without the use of adaptive weights or any other enhancements typically employed in PINNs. More broadly, our work reveal insights into the effectiveness of quasi-Newton optimization strategies in significantly improving the convergence and accurate generalization of PINNs and PIKANs.
Skip-Connected Self-Recurrent Spiking Neural Networks with Joint Intrinsic Parameter and Synaptic Weight Training
As an important class of spiking neural networks (SNNs), recurrent spiking neural networks (RSNNs) possess great computational power and have been widely used for processing sequential data like audio and text. However, most RSNNs suffer from two problems. 1. Due to the lack of architectural guidance, random recurrent connectivity is often adopted, which does not guarantee good performance. 2. Training of RSNNs is in general challenging, bottlenecking achievable model accuracy. To address these problems, we propose a new type of RSNNs called Skip-Connected Self-Recurrent SNNs (ScSr-SNNs). Recurrence in ScSr-SNNs is introduced in a stereotyped manner by adding self-recurrent connections to spiking neurons. The SNNs with self-recurrent connections can realize recurrent behaviors similar to those of more complex RSNNs while the error gradients can be more straightforwardly calculated due to the mostly feedforward nature of the network. The network dynamics is enriched by skip connections between nonadjacent layers. Moreover, we propose a new backpropagation (BP) method called backpropagated intrinsic plasticity (BIP) to further boost the performance of ScSr-SNNs by training intrinsic model parameters. Unlike standard intrinsic plasticity rules that adjust the neuron's intrinsic parameters according to neuronal activity, the proposed BIP method optimizes intrinsic parameters based on the backpropagated error gradient of a well-defined global loss function in addition to synaptic weight training. Here, based on challenging speech, neuromorphic speech, and neuromorphic image datasets, the proposed ScSr-SNNs can boost performance by up to 2.85% compared with other types of RSNNs trained by state-of-the-art BP methods.
Selective capture and recovery of uranium oxide colloids from aqueous soil suspensions using high gradient magnetic filtration
High Gradient Magnetic Filtration (HGMF) is a promising method for the selective capture and recovery of uranium oxide from surface soils. To date, however, magnetic filtration of uranium oxide has only been demonstrated at a proof-of-principle scale using relatively small filters (<5 cm 3 ) at low flowrates (<60 mL/min). Here, to explore the efficacy of magnetic filtration of uranium oxide at a larger scale, a newly designed HGMF apparatus that is more than an order of magnitude larger than our earlier filters (106 cm 3 ) was designed, fabricated, and tested at relatively high flowrates. Filtration experiments were performed using aqueous uranium oxide particle suspensions with Arizona Road Dust (ARD) as a soil simulant. At a flowrate of 125 mL/min, the apparatus’ uranium capture rate was exceptionally high (96 %), but selectivity was poor due to the high rate of capture for diamagnetic soil constituents (e.g., 77 % for silicon). All particles were captured at a lower rate when the flowrate was increased to 250 mL/min, but uranium selectivity was significantly increased due to the more substantial reduction in diamagnetic particle capture (i.e., capture rate of 77 % and 15 % for uranium and silicon, respectively). When backwashing the apparatus at the same flowrates used during filtration experiments, the rate of uranium recovery tended to be fairly low. Nevertheless, higher flowrates (1 L/min) and sonication were both shown to be highly effective methods of increasing uranium recovery. Magnetic field simulations were also performed to investigate potential optimizations to the design of the apparatus. These simulations showed that the intensity of the applied magnetic field could be increased by increasing the thickness of the steel magnetic housing. Additionally, stochastic trajectory simulations were performed to investigate the potential mechanisms of particle capture.
An Empirical Quantile Estimation Approach for Chance-Constrained Nonlinear Optimization Problems
We investigate an empirical quantile estimation approach to solve chance-constrained nonlinear optimization problems. Our approach is based on the reformulation of the chance constraint as an equivalent quantile constraint to provide stronger signals on the gradient. In this approach, the value of the quantile function is estimated empirically from samples drawn from the random parameters, and the gradient of the quantile function is estimated via a finite-difference approximation on top of the quantile-function-value estimation. We establish a convergence theory of this approach within the framework of an augmented Lagrangian method for solving general nonlinear constrained optimization problems. The foundation of the convergence analysis is a concentration property of the empirical quantile process, and the analysis is divided based on whether or not the quantile function is differentiable. In contrast to the sampling-and-smoothing approach used in the literature, the method developed in this paper does not involve any smoothing function and hence the quantile-function gradient approximation is easier to implement and there are less accuracy-control parameters to tune. Furthermore, we demonstrate the effectiveness of this approach and compare it with a smoothing method for the quantile-gradient estimation. Numerical investigation shows that the two approaches are competitive for certain problem instances.
Analytic gradients for compressed multistate pair-density functional theory
Photochemical reactions often involve states that are closely coupled due to near degeneracies, for example by proximity to conical intersections. Therefore, a multistate method is used to accurately describe these states; for example, ordinary perturbation theory is replaced by quasidegenerate perturbation theory. Multiconfiguration pair-density functional theory (MC-PDFT) provides an efficient way to approximate the full dynamical correlation energy of strongly correlated systems, and we recently proposed compressed multistate pair-density functional theory (CMS-PDFT) to treat closely coupled states. In the present paper, we report the implementation of analytic gradients for CMS-PDFT in both OpenMolcas and PySCF, and here we illustrate the use of these gradients by applying the method to the excited states of formaldehyde and phenol.
Stochastic properties of ultralight scalar field gradients
Ultralight axion-like particles are well-motivated dark matter candidates that are the target of numerous direct detection efforts. In the vicinity of the Solar System, such particles can be treated as oscillating scalar fields. The velocity dispersion of the Milky Way determines a coherence time of about 10 6 oscillations, beyond which the amplitude of the axion field fluctuates stochastically. Any analysis of data from an axion direct detection experiment must carefully account for this stochastic behavior to properly interpret the results. This is especially true for experiments sensitive to the gradient of the axion field that are unable to collect data for many coherence times. Indeed, the direction, in addition to the amplitude, of the axion field gradient fluctuates stochastically. We present the first complete stochastic treatment for the gradient of the axion field, including multiple computationally efficient methods for performing likelihood-based data analysis, which can be applied to any axion signal, regardless of coherence time. Additionally, we demonstrate that ignoring the stochastic behavior of the gradient of the axion field can potentially result in failure to discover a true axion signal
Handling Iterative Solvers in an Algorithmic Differentiation Framework Using Implicit Methods
Differentiable programming is a powerful concept as it enables the seemly propagation of gradients through functions, algorithms, and/or whole physics simulations. These gradients are useful for a wide variety of applications, including sensitivity studies and machine learning, but one of particular interest is optimization. Gradient-based optimization, enabled through automatic/algorithmic differentiation (AD), can be used on predictive physical models to efficiently optimize a set of design variables. AD methods are a particularly promising approach to complex physics simulations because they can be shown to scale well with an increasing number of design variables; however, care must be taken when coupling between different models or different states of a single model.
Ab initio calculation of atomic solid hydrogen phases based on Gutzwiller many-body wave functions
We apply two ab initio many-body methods based on Gutzwiller wave functions, i.e., correlation matrix renormalization theory (CMRT) and Gutzwiller conjugate gradient minimization (GCGM), to the study of crystalline phases of atomic hydrogen. Both methods avoid empirical Hubbard U parameters and are free from double-counting issues. CMRT employs a Gutzwiller-type approximation that enables efficient calculations, while GCGM goes beyond this approximation to achieve higher accuracy at higher computational cost. By benchmarking against available quantum Monte Carlo (QMC) results, we demonstrate that while both methods are more accurate than the widely used density-functional theory, GCGM systematically captures additional correlation energy missing in CMRT, leading to significantly improved total energy predictions. We also show that by including the correlation energy Ec from local density approximation in the CMRT calculation, CMRT + E c produces energy in better agreement with the QMC results in these hydrogen lattice systems.
Approximate Dynamic Programming With Enhanced Off-Policy Learning for Coordinating Distributed Energy Resources
Herein this paper proposes an innovative approximate dynamic programming (ADP) method for distributed energy resource coordination with the loss of life of battery energy storage system (BESS) explicitly modeled. The dispatch policy is designed to account for both calendrical and cyclical aging effects on BESS, explicitly modeling the impacts of ambient temperature on BESS lifespan. The proposed ADP employs an adaptive critic method and enhanced off-policy deterministic policy gradient (DPG) strategy, addressing the limitations of the on-policy gradient-based ADP approaches, including inadequate exploration, low data usage, and computational complexity. In particular, a customized policy is proposed to guide the algorithm to explore some promising decisions and thereby improve exploration capability and learning efficiency compared to conventional DPG-based learning approaches, which may struggle to find a global optimum due to random noisy action-based exploration or require expert demonstration with extra effort. The proposed method is illustrated using the IEEE 123-node system and compared with the existing ADP methods to prove solution accuracy and demonstrate the effects of incorporating degradation models into control design. Case studies showed that the proposed ADP effectively coordinates DERs with a 10 times smaller optimization gap compared to existing methods, and the incorporation of the BESS life loss model ensures the expected lifespan and results in significant cost savings.
A hybrid calibration approach to Hertz-type contact parameters for discrete element models
This study aims at providing a hybrid calibration framework to estimate Hertz-type contact parameters (particle-scale shear modulus and Poisson ratio) for both two-dimensional and three-dimensional discrete element modelling (DEM). On the basis of statistically isotropic granular packings, a set of analytical formulae between macroscopic material parameters (Young modulus and Poisson ratio) and particle-scale Hertz-type contact parameters for granular systems are derived under small-strain isotropic stress conditions. However, the derived analytical solutions are only estimated values for general models. By viewing each DEM modelling as an implicit mathematical function taking the particle-level parameters as independent variables and employing the derived analytical solutions as the initial input parameters, an automatic iterative scheme is proposed to obtain the calibrated parameters with higher accuracies. Considering highly nonlinear features and discontinuities of the macro-micro relationship in Hertz-based discrete element models, the adaptive moment estimation algorithm is adopted in this study because of its capacity of dealing with noise gradients of cost functions. Here, the proposed method is validated with several numerical cases including randomly distributed monodisperse and polydisperse packings. Noticeable improvements in terms of calibration efficiency and accuracy have been made.
A FETI approach to domain decomposition for meshfree discretizations of nonlocal problems
We propose a domain decomposition method for the efficient simulation of nonlocal problems. Our approach is based on a multi-domain formulation of a nonlocal diffusion problem where the subdomains share “nonlocal” interfaces of the size of the nonlocal horizon. This system of nonlocal equations is first rewritten in terms of minimization of a nonlocal energy, then discretized with a meshfree approximation and finally solved via a Lagrange multiplier approach in a way that resembles the finite element tearing and interconnect method. Specifically, we propose a distributed projected gradient algorithm for the solution of the Lagrange multiplier system, whose unknowns determine the nonlocal interface conditions between subdomains. Several two-dimensional numerical tests on problems as large as 191 million unknowns illustrate the strong and the weak scalability of our algorithm, which outperforms the standard approach to the distributed numerical solution of the problem. Finally, this work is the first rigorous numerical study in a two-dimensional multi-domain setting for nonlocal operators with finite horizon and, as such, it is a fundamental step towards increasing the use of nonlocal models in large scale simulations.
Latent space dynamics identification for interface tracking with application to shock-induced pore collapse
Capturing sharp, evolving interfaces remains a central challenge in reduced-order modeling, especially when data is limited and the system exhibits localized nonlinearities or discontinuities. Here, we propose LaSDI-IT (Latent Space Dynamics Identification for Interface Tracking), a data-driven framework that combines low-dimensional latent dynamics learning with explicit interface-aware encoding to enable accurate and efficient modeling of physical systems involving moving material boundaries. At the core of LaSDI-IT is a revised autoencoder architecture that jointly reconstructs the physical field and an indicator function representing material regions or phases, allowing the model to track complex interface evolution without requiring detailed physical models or mesh adaptation. The latent dynamics are learned through linear regression in the encoded space and generalized across parameter regimes using Gaussian process interpolation with greedy sampling. We demonstrate LaSDI-IT on the problem of shock-induced pore collapse in high explosives, a process characterized by sharp temperature gradients and dynamically deforming pore geometries. The method achieves relative prediction errors below 9% across the parameter space, accurately recovers key quantities of interest such as pore area and hot spot formation, and matches the performance of dense training with only half the data. This latent dynamics prediction was 10 6 times faster than the conventional high-fidelity simulation, proving its utility for multi-query applications. These results highlight LaSDI-IT as a general, data-efficient framework for modeling discontinuity-rich systems in computational physics, with potential applications in multiphase flows, fracture mechanics, and phase change problems.
High-throughput printing of combinatorial materials from aerosols
The development of new materials and their compositional and microstructural optimization are essential in regard to next-generation technologies such as clean energy and environmental sustainability. However, materials discovery and optimization have been a frustratingly slow process. The Edisonian trial-and-error process is time consuming and resource inefficient, particularly when contrasted with vast materials design spaces 1 . Whereas traditional combinatorial deposition methods can generate material libraries 2,3 , these suffer from limited material options and inability to leverage major breakthroughs in nanomaterial synthesis. Here we report a high-throughput combinatorial printing method capable of fabricating materials with compositional gradients at microscale spatial resolution. In situ mixing and printing in the aerosol phase allows instantaneous tuning of the mixing ratio of a broad range of materials on the fly, which is an important feature unobtainable in conventional multimaterials printing using feedstocks in liquid–liquid or solid–solid phases 4–6 . We demonstrate a variety of high-throughput printing strategies and applications in combinatorial doping, functional grading and chemical reaction, enabling materials exploration of doped chalcogenides and compositionally graded materials with gradient properties. The ability to combine the top-down design freedom of additive manufacturing with bottom-up control over local material compositions promises the development of compositionally complex materials inaccessible via conventional manufacturing approaches.
Prediction of the energetic particle redistribution by an improved critical gradient model and analysis of the transport threshold
Based on the theory of critical gradient model (CGM) and following the simulation method proposed by Waltz et al., a combination of TGLFEP and EPtran code is employed to predict the energetic particle (EP) transport induced by Alfvén eigenmodes (AEs). To be consistent with the experiment, recent improvements to the simulation method include consideration of threshold evolution and orbit loss due to finite orbit width. The revised CGM is applied to simulate two DIII-D experimental discharges (#142111 and #153071). It well reproduces the experimental profiles with multiple unstable AEs and large-scale EP transport. Discharge #142111 had previously been simulated using a nonlinear MHD-kinetic code MEGA with a transport mechanism based on stochasticity induced by overlapping AE. By comparing the simulated EP profiles, we find that the AE transport threshold is approximated by both the MEGA nonlinear stability threshold and the proposed CGM threshold (error <5% for single n and <17% for multiple n simulation). Both of them are larger than the linear stability threshold of the most unstable AE mode by a quantity of the order of the flux needed to sustain EP transport by the background turbulence. Additionally, we have also applied the improved CGM to simulate the α particle redistribution for a China Fusion Engineering Test Reactor steady state scenario. Because of the clear separation between the AE unstable region and the loss cone, only a moderate α particle loss of ~9.6% is predicted.
Electropolishing-induced topographic defects in niobium: insights and implications for superconducting radio frequency applications
Electropolishing (EP) is the premier surface preparation method for high-Q, high-gradient superconducting RF cavities made of Nb. This leaves behind an apparently smooth surface, yet the achievable peak magnetic fields fall well below the superheating field of Nb, in most cases. Here, in this work, the ultimate surface finish of EP was investigated by studying its effect on highly polished Nb samples. EP introduces high slope angle sloped-steps at grain boundaries. The magnetic field enhancement and superheating field suppression factors associated with such a geometry are calculated in the London theory. Despite the by-eye smoothness of electropolished Nb, such defects compromise the stability of the low-loss Meissner state, likely limiting the achievable peak accelerating fields in superconducting RF cavities. Finally, the impact of surface roughness on impurity diffusion is investigated which can link surface roughness to the effectiveness of heat treatments like low-temperature baking or nitrogen infusion in the vortex nucleation or hydride hypotheses. Surface roughness tends to decrease the effective dose of impurities as a result of the expansion of impurities into regions with greater internal angle. The effective dose of impurities can be protected by minimizing slope angles and step heights, ensuring uniformity.
Online Convex Optimization of Programmable Quantum Computers to Simulate Time-Varying Quantum Channels
Simulating quantum channels is a fundamental primitive in quantum computing, since quantum channels define general (trace-preserving) quantum operations. An arbitrary quantum channel cannot be exactly simulated using a finite-dimensional programmable quantum processor, making it important to develop optimal approximate simulation techniques. In this paper, we study the challenging setting in which the channel to be simulated varies adversarially with time. We propose the use of matrix exponentiated gradient descent (MEGD), an online convex optimization method, and analytically show that it achieves a sublinear regret in time. Through experiments, we validate the main results for time-varying dephasing channels using a programmable generalized teleportation processor.
Electropolishing-induced topographic defects in niobium: Insights and Implications for SRF
Electropolishing (EP) is the premier surface preparation method for high-Q, high-gradient superconducting RF cavities made of Nb. This leaves behind an apparently smooth surface, yet the achievable peak magnetic fields fall well below the superheating field of Nb, in most cases. In this work, the ultimate surface finish of EP was investigated by studying its effect on highly polished Nb samples. EP introduces high slope angle sloped-steps at grain boundaries. The magnetic field enhancement and superheating field suppression factors associated with such a geometry are calculated in the London theory. Despite the by-eye smoothness of electropolished Nb, such defects compromise the stability of the low-loss Meissner state, likely limiting the achievable peak accelerating fields in superconducting RF cavities. Finally, the impact of surface roughness on impurity diffusion is investigated which can link surface roughness to the effectiveness of heat treatments like low-temperature baking or nitrogen infusion in the vortex nucleation or hydride hypotheses. Surface roughness tends to decrease the effective dose of impurities as a result of the expansion of impurities into regions with greater internal angle. The effective dose of impurities can be protected by minimizing slope angles and step heights, ensuring uniformity.