SEARCH · Engineering Papers
Results for “gradient methods”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Optimizing the optimizer for physics-informed neural networks and Kolmogorov-Arnold networks
Physics-Informed Neural Networks (PINNs) have revolutionized the computation of PDE solutions by integrating partial differential equations (PDEs) into the neural network’s training process as soft constraints, becoming an important component of the scientific machine learning (SciML) ecosystem. More recently, physics-informed Kolmogorv-Arnold networks (PIKANs) have also shown to be effective and comparable in accuracy with PINNs. In their current implementation, both PINNs and PIKANs are mainly optimized using first-order methods like Adam, as well as quasi-Newton methods such as BFGS and its low-memory variant, L-BFGS. However, these optimizers often struggle with highly nonlinear and non-convex loss landscapes, leading to challenges such as slow convergence, local minima entrapment, and (non)degenerate saddle points. In this study, we investigate the performance of Self- Scaled BFGS (SSBFGS), Self-Scaled Broyden (SSBroyden) methods and other advanced quasi-Newton schemes, including BFGS and L-BFGS with different line search strategies. These methods dynamically rescale updates based on historical gradient information, thus enhancing training efficiency and accuracy. We systematically compare these optimizers – using both PINNs and PIKANs – on key challenging PDEs, including the Burgers, Allen-Cahn, Kuramoto-Sivashinsky, Ginzburg-Landau, and Stokes equations. Additionally, we evaluate the performance of SSBFGS and SSBroyden for Deep Operator Network (DeepONet) architectures, demonstrating their effectiveness for data-driven operator learning. Our findings provide state-of-the-art results with orders-of-magnitude accuracy improvements without the use of adaptive weights or any other enhancements typically employed in PINNs. More broadly, our work reveal insights into the effectiveness of quasi-Newton optimization strategies in significantly improving the convergence and accurate generalization of PINNs and PIKANs.
Skip-Connected Self-Recurrent Spiking Neural Networks with Joint Intrinsic Parameter and Synaptic Weight Training
As an important class of spiking neural networks (SNNs), recurrent spiking neural networks (RSNNs) possess great computational power and have been widely used for processing sequential data like audio and text. However, most RSNNs suffer from two problems. 1. Due to the lack of architectural guidance, random recurrent connectivity is often adopted, which does not guarantee good performance. 2. Training of RSNNs is in general challenging, bottlenecking achievable model accuracy. To address these problems, we propose a new type of RSNNs called Skip-Connected Self-Recurrent SNNs (ScSr-SNNs). Recurrence in ScSr-SNNs is introduced in a stereotyped manner by adding self-recurrent connections to spiking neurons. The SNNs with self-recurrent connections can realize recurrent behaviors similar to those of more complex RSNNs while the error gradients can be more straightforwardly calculated due to the mostly feedforward nature of the network. The network dynamics is enriched by skip connections between nonadjacent layers. Moreover, we propose a new backpropagation (BP) method called backpropagated intrinsic plasticity (BIP) to further boost the performance of ScSr-SNNs by training intrinsic model parameters. Unlike standard intrinsic plasticity rules that adjust the neuron's intrinsic parameters according to neuronal activity, the proposed BIP method optimizes intrinsic parameters based on the backpropagated error gradient of a well-defined global loss function in addition to synaptic weight training. Here, based on challenging speech, neuromorphic speech, and neuromorphic image datasets, the proposed ScSr-SNNs can boost performance by up to 2.85% compared with other types of RSNNs trained by state-of-the-art BP methods.
Selective capture and recovery of uranium oxide colloids from aqueous soil suspensions using high gradient magnetic filtration
High Gradient Magnetic Filtration (HGMF) is a promising method for the selective capture and recovery of uranium oxide from surface soils. To date, however, magnetic filtration of uranium oxide has only been demonstrated at a proof-of-principle scale using relatively small filters (<5 cm 3 ) at low flowrates (<60 mL/min). Here, to explore the efficacy of magnetic filtration of uranium oxide at a larger scale, a newly designed HGMF apparatus that is more than an order of magnitude larger than our earlier filters (106 cm 3 ) was designed, fabricated, and tested at relatively high flowrates. Filtration experiments were performed using aqueous uranium oxide particle suspensions with Arizona Road Dust (ARD) as a soil simulant. At a flowrate of 125 mL/min, the apparatus’ uranium capture rate was exceptionally high (96 %), but selectivity was poor due to the high rate of capture for diamagnetic soil constituents (e.g., 77 % for silicon). All particles were captured at a lower rate when the flowrate was increased to 250 mL/min, but uranium selectivity was significantly increased due to the more substantial reduction in diamagnetic particle capture (i.e., capture rate of 77 % and 15 % for uranium and silicon, respectively). When backwashing the apparatus at the same flowrates used during filtration experiments, the rate of uranium recovery tended to be fairly low. Nevertheless, higher flowrates (1 L/min) and sonication were both shown to be highly effective methods of increasing uranium recovery. Magnetic field simulations were also performed to investigate potential optimizations to the design of the apparatus. These simulations showed that the intensity of the applied magnetic field could be increased by increasing the thickness of the steel magnetic housing. Additionally, stochastic trajectory simulations were performed to investigate the potential mechanisms of particle capture.
An Empirical Quantile Estimation Approach for Chance-Constrained Nonlinear Optimization Problems
We investigate an empirical quantile estimation approach to solve chance-constrained nonlinear optimization problems. Our approach is based on the reformulation of the chance constraint as an equivalent quantile constraint to provide stronger signals on the gradient. In this approach, the value of the quantile function is estimated empirically from samples drawn from the random parameters, and the gradient of the quantile function is estimated via a finite-difference approximation on top of the quantile-function-value estimation. We establish a convergence theory of this approach within the framework of an augmented Lagrangian method for solving general nonlinear constrained optimization problems. The foundation of the convergence analysis is a concentration property of the empirical quantile process, and the analysis is divided based on whether or not the quantile function is differentiable. In contrast to the sampling-and-smoothing approach used in the literature, the method developed in this paper does not involve any smoothing function and hence the quantile-function gradient approximation is easier to implement and there are less accuracy-control parameters to tune. Furthermore, we demonstrate the effectiveness of this approach and compare it with a smoothing method for the quantile-gradient estimation. Numerical investigation shows that the two approaches are competitive for certain problem instances.
Test of Shi et al. Method to Infer the Magnetic Reconnection Geometry from Spacecraft Data: MHD Simulation with Guide Field and Antiparallel Kinetic Simulation
When analyzing data from an array of spacecraft (such as Cluster or MMS) crossing a site of magnetic reconnection, it is desirable to be able to accurately determine the orientation of the reconnection site. If the reconnection is quasi-two dimensional, there are three key directions, the direction of maximum inhomogeneity (the direction across the reconnection site), the direction of the reconnecting component of the magnetic field, and the direction of rough invariance (the "out of plane" direction). Using simulated spacecraft observations of magnetic reconnection in the geomagnetic tail, we extend our previous tests of the direction-finding method developed by Shi et al. (2005) and the method to determine the structure velocity relative to the spacecraft Vstr. These methods require data from four proximate spacecraft. We add artificial noise and calibration errors to the simulation fields, and then use the perturbed gradient of the magnetic field B and perturbed time derivative dB/dt, as described by Denton et al. (2010). Three new simulations are examined: a weakly three-dimensional, i.e., quasi-two-dimensional, MHD simulation without a guide field, a quasi-two-dimensional MHD simulation with a guide field, and a two-dimensional full dynamics kinetic simulation with inherent noise so that the apparent minimum gradient was not exactly zero, even without added artificial errors. We also examined variations of the spacecraft trajectory for the kinetic simulation. The accuracy of the directions found varied depending on the simulation and spacecraft trajectory, but all the directions could be found within about 10 for all cases. Various aspects of the method were examined, including how to choose averaging intervals and the best intervals for determining the directions and velocity. For the kinetic simulation, we also investigated in detail how the errors in the inferred gradient directions from the unmodified Shi et al. method (using the unperturbed gradient) depended on the amplitude of the calibration errors. For an accuracy of 3 for the maximum gradient direction, the calibration errors could be as large as 3% of reconnection magnetic field, while for the same accuracy for the minimum gradient direction, the calibration errors could only be as large as 0.03% of the reconnection magnetic field. These results suggest that the maximum gradient direction can normally be determined by the unmodified Shi et al. method, while the modified method or some other method must be used to accurately determine the minimum gradient direction. The structure velocity was found with magnitude accurate to 2% and direction accurate to within 5%.
Analytic gradients for compressed multistate pair-density functional theory
Photochemical reactions often involve states that are closely coupled due to near degeneracies, for example by proximity to conical intersections. Therefore, a multistate method is used to accurately describe these states; for example, ordinary perturbation theory is replaced by quasidegenerate perturbation theory. Multiconfiguration pair-density functional theory (MC-PDFT) provides an efficient way to approximate the full dynamical correlation energy of strongly correlated systems, and we recently proposed compressed multistate pair-density functional theory (CMS-PDFT) to treat closely coupled states. In the present paper, we report the implementation of analytic gradients for CMS-PDFT in both OpenMolcas and PySCF, and here we illustrate the use of these gradients by applying the method to the excited states of formaldehyde and phenol.
Stochastic properties of ultralight scalar field gradients
Ultralight axion-like particles are well-motivated dark matter candidates that are the target of numerous direct detection efforts. In the vicinity of the Solar System, such particles can be treated as oscillating scalar fields. The velocity dispersion of the Milky Way determines a coherence time of about 10 6 oscillations, beyond which the amplitude of the axion field fluctuates stochastically. Any analysis of data from an axion direct detection experiment must carefully account for this stochastic behavior to properly interpret the results. This is especially true for experiments sensitive to the gradient of the axion field that are unable to collect data for many coherence times. Indeed, the direction, in addition to the amplitude, of the axion field gradient fluctuates stochastically. We present the first complete stochastic treatment for the gradient of the axion field, including multiple computationally efficient methods for performing likelihood-based data analysis, which can be applied to any axion signal, regardless of coherence time. Additionally, we demonstrate that ignoring the stochastic behavior of the gradient of the axion field can potentially result in failure to discover a true axion signal
Handling Iterative Solvers in an Algorithmic Differentiation Framework Using Implicit Methods
Differentiable programming is a powerful concept as it enables the seemly propagation of gradients through functions, algorithms, and/or whole physics simulations. These gradients are useful for a wide variety of applications, including sensitivity studies and machine learning, but one of particular interest is optimization. Gradient-based optimization, enabled through automatic/algorithmic differentiation (AD), can be used on predictive physical models to efficiently optimize a set of design variables. AD methods are a particularly promising approach to complex physics simulations because they can be shown to scale well with an increasing number of design variables; however, care must be taken when coupling between different models or different states of a single model.
Ab initio calculation of atomic solid hydrogen phases based on Gutzwiller many-body wave functions
We apply two ab initio many-body methods based on Gutzwiller wave functions, i.e., correlation matrix renormalization theory (CMRT) and Gutzwiller conjugate gradient minimization (GCGM), to the study of crystalline phases of atomic hydrogen. Both methods avoid empirical Hubbard U parameters and are free from double-counting issues. CMRT employs a Gutzwiller-type approximation that enables efficient calculations, while GCGM goes beyond this approximation to achieve higher accuracy at higher computational cost. By benchmarking against available quantum Monte Carlo (QMC) results, we demonstrate that while both methods are more accurate than the widely used density-functional theory, GCGM systematically captures additional correlation energy missing in CMRT, leading to significantly improved total energy predictions. We also show that by including the correlation energy Ec from local density approximation in the CMRT calculation, CMRT + E c produces energy in better agreement with the QMC results in these hydrogen lattice systems.
Development of Fast Algorithms Using Recursion, Nesting and Iterations for Computational Electromagnetics
In the first phase of our work, we have concentrated on laying the foundation to develop fast algorithms, including the use of recursive structure like the recursive aggregate interaction matrix algorithm (RAIMA), the nested equivalence principle algorithm (NEPAL), the ray-propagation fast multipole algorithm (RPFMA), and the multi-level fast multipole algorithm (MLFMA). We have also investigated the use of curvilinear patches to build a basic method of moments code where these acceleration techniques can be used later. In the second phase, which is mainly reported on here, we have concentrated on implementing three-dimensional NEPAL on a massively parallel machine, the Connection Machine CM-5, and have been able to obtain some 3D scattering results. In order to understand the parallelization of codes on the Connection Machine, we have also studied the parallelization of 3D finite-difference time-domain (FDTD) code with PML material absorbing boundary condition (ABC). We found that simple algorithms like the FDTD with material ABC can be parallelized very well allowing us to solve within a minute a problem of over a million nodes. In addition, we have studied the use of the fast multipole method and the ray-propagation fast multipole algorithm to expedite matrix-vector multiplication in a conjugate-gradient solution to integral equations of scattering. We find that these methods are faster than LU decomposition for one incident angle, but are slower than LU decomposition when many incident angles are needed as in the monostatic RCS calculations.
Direct Numerical Simulation of Incompressible Pipe Flow Using a B-Spline Spectral Method
A numerical method based on b-spline polynomials was developed to study incompressible flows in cylindrical geometries. A b-spline method has the advantages of possessing spectral accuracy and the flexibility of standard finite element methods. Using this method it was possible to ensure regularity of the solution near the origin, i.e. smoothness and boundedness. Because b-splines have compact support, it is also possible to remove b-splines near the center to alleviate the constraint placed on the time step by an overly fine grid. Using the natural periodicity in the azimuthal direction and approximating the streamwise direction as periodic, so-called time evolving flow, greatly reduced the cost and complexity of the computations. A direct numerical simulation of pipe flow was carried out using the method described above at a Reynolds number of 5600 based on diameter and bulk velocity. General knowledge of pipe flow and the availability of experimental measurements make pipe flow the ideal test case with which to validate the numerical method. Results indicated that high flatness levels of the radial component of velocity in the near wall region are physical; regions of high radial velocity were detected and appear to be related to high speed streaks in the boundary layer. Budgets of Reynolds stress transport equations showed close similarity with those of channel flow. However contrary to channel flow, the log layer of pipe flow is not homogeneous for the present Reynolds number. A topological method based on a classification of the invariants of the velocity gradient tensor was used. Plotting iso-surfaces of the discriminant of the invariants proved to be a good method for identifying vortical eddies in the flow field.
Approximate Dynamic Programming With Enhanced Off-Policy Learning for Coordinating Distributed Energy Resources
Herein this paper proposes an innovative approximate dynamic programming (ADP) method for distributed energy resource coordination with the loss of life of battery energy storage system (BESS) explicitly modeled. The dispatch policy is designed to account for both calendrical and cyclical aging effects on BESS, explicitly modeling the impacts of ambient temperature on BESS lifespan. The proposed ADP employs an adaptive critic method and enhanced off-policy deterministic policy gradient (DPG) strategy, addressing the limitations of the on-policy gradient-based ADP approaches, including inadequate exploration, low data usage, and computational complexity. In particular, a customized policy is proposed to guide the algorithm to explore some promising decisions and thereby improve exploration capability and learning efficiency compared to conventional DPG-based learning approaches, which may struggle to find a global optimum due to random noisy action-based exploration or require expert demonstration with extra effort. The proposed method is illustrated using the IEEE 123-node system and compared with the existing ADP methods to prove solution accuracy and demonstrate the effects of incorporating degradation models into control design. Case studies showed that the proposed ADP effectively coordinates DERs with a 10 times smaller optimization gap compared to existing methods, and the incorporation of the BESS life loss model ensures the expected lifespan and results in significant cost savings.
Numerical Evaluation of P-Multigrid Method for the Solution of Discontinuous Galerkin Discretizations of Diffusive Equations
This paper describes numerical experiments with P-multigrid to corroborate analysis, validate the present implementation, and to examine issues that arise in the implementations of the various combinations of relaxation schemes, discretizations and P-multigrid methods. The two approaches to implement P-multigrid presented here are equivalent for most high-order discretization methods such as spectral element, SUPG, and discontinuous Galerkin applied to advection; however it is discovered that the approach that mimics the common geometric multigrid implementation is less robust, and frequently unstable when applied to discontinuous Galerkin discretizations of di usion. Gauss-Seidel relaxation converges 40% faster than block Jacobi, as predicted by analysis; however, the implementation of Gauss-Seidel is considerably more expensive that one would expect because gradients in most neighboring elements must be updated. A compromise quasi Gauss-Seidel relaxation method that evaluates the gradient in each element twice per iteration converges at rates similar to those predicted for true Gauss-Seidel.
A hybrid calibration approach to Hertz-type contact parameters for discrete element models
This study aims at providing a hybrid calibration framework to estimate Hertz-type contact parameters (particle-scale shear modulus and Poisson ratio) for both two-dimensional and three-dimensional discrete element modelling (DEM). On the basis of statistically isotropic granular packings, a set of analytical formulae between macroscopic material parameters (Young modulus and Poisson ratio) and particle-scale Hertz-type contact parameters for granular systems are derived under small-strain isotropic stress conditions. However, the derived analytical solutions are only estimated values for general models. By viewing each DEM modelling as an implicit mathematical function taking the particle-level parameters as independent variables and employing the derived analytical solutions as the initial input parameters, an automatic iterative scheme is proposed to obtain the calibrated parameters with higher accuracies. Considering highly nonlinear features and discontinuities of the macro-micro relationship in Hertz-based discrete element models, the adaptive moment estimation algorithm is adopted in this study because of its capacity of dealing with noise gradients of cost functions. Here, the proposed method is validated with several numerical cases including randomly distributed monodisperse and polydisperse packings. Noticeable improvements in terms of calibration efficiency and accuracy have been made.
A FETI approach to domain decomposition for meshfree discretizations of nonlocal problems
We propose a domain decomposition method for the efficient simulation of nonlocal problems. Our approach is based on a multi-domain formulation of a nonlocal diffusion problem where the subdomains share “nonlocal” interfaces of the size of the nonlocal horizon. This system of nonlocal equations is first rewritten in terms of minimization of a nonlocal energy, then discretized with a meshfree approximation and finally solved via a Lagrange multiplier approach in a way that resembles the finite element tearing and interconnect method. Specifically, we propose a distributed projected gradient algorithm for the solution of the Lagrange multiplier system, whose unknowns determine the nonlocal interface conditions between subdomains. Several two-dimensional numerical tests on problems as large as 191 million unknowns illustrate the strong and the weak scalability of our algorithm, which outperforms the standard approach to the distributed numerical solution of the problem. Finally, this work is the first rigorous numerical study in a two-dimensional multi-domain setting for nonlocal operators with finite horizon and, as such, it is a fundamental step towards increasing the use of nonlocal models in large scale simulations.
Latent space dynamics identification for interface tracking with application to shock-induced pore collapse
Capturing sharp, evolving interfaces remains a central challenge in reduced-order modeling, especially when data is limited and the system exhibits localized nonlinearities or discontinuities. Here, we propose LaSDI-IT (Latent Space Dynamics Identification for Interface Tracking), a data-driven framework that combines low-dimensional latent dynamics learning with explicit interface-aware encoding to enable accurate and efficient modeling of physical systems involving moving material boundaries. At the core of LaSDI-IT is a revised autoencoder architecture that jointly reconstructs the physical field and an indicator function representing material regions or phases, allowing the model to track complex interface evolution without requiring detailed physical models or mesh adaptation. The latent dynamics are learned through linear regression in the encoded space and generalized across parameter regimes using Gaussian process interpolation with greedy sampling. We demonstrate LaSDI-IT on the problem of shock-induced pore collapse in high explosives, a process characterized by sharp temperature gradients and dynamically deforming pore geometries. The method achieves relative prediction errors below 9% across the parameter space, accurately recovers key quantities of interest such as pore area and hot spot formation, and matches the performance of dense training with only half the data. This latent dynamics prediction was 10 6 times faster than the conventional high-fidelity simulation, proving its utility for multi-query applications. These results highlight LaSDI-IT as a general, data-efficient framework for modeling discontinuity-rich systems in computational physics, with potential applications in multiphase flows, fracture mechanics, and phase change problems.
High-throughput printing of combinatorial materials from aerosols
The development of new materials and their compositional and microstructural optimization are essential in regard to next-generation technologies such as clean energy and environmental sustainability. However, materials discovery and optimization have been a frustratingly slow process. The Edisonian trial-and-error process is time consuming and resource inefficient, particularly when contrasted with vast materials design spaces 1 . Whereas traditional combinatorial deposition methods can generate material libraries 2,3 , these suffer from limited material options and inability to leverage major breakthroughs in nanomaterial synthesis. Here we report a high-throughput combinatorial printing method capable of fabricating materials with compositional gradients at microscale spatial resolution. In situ mixing and printing in the aerosol phase allows instantaneous tuning of the mixing ratio of a broad range of materials on the fly, which is an important feature unobtainable in conventional multimaterials printing using feedstocks in liquid–liquid or solid–solid phases 4–6 . We demonstrate a variety of high-throughput printing strategies and applications in combinatorial doping, functional grading and chemical reaction, enabling materials exploration of doped chalcogenides and compositionally graded materials with gradient properties. The ability to combine the top-down design freedom of additive manufacturing with bottom-up control over local material compositions promises the development of compositionally complex materials inaccessible via conventional manufacturing approaches.