Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Hessian approximation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

An adaptive Hessian approximated stochastic gradient MCMC method

Bayesian approaches have been successfully integrated into training deep neural networks. One popular family is stochastic gradient Markov chain Monte Carlo methods (SG-MCMC), which have gained increasing interest due to their ability to handle large datasets and the potential to avoid overfitting. Although standard SG-MCMC methods have shown great performance in a variety of problems, they may be inefficient when the random variables in the target posterior densities have scale differences or are highly correlated. Here, we present an adaptive Hessian approximated stochastic gradient MCMC method to incorporate local geometric information while sampling from the posterior. The idea is to apply stochastic approximation (SA) to sequentially update a preconditioning matrix at each iteration. The preconditioner possesses second-order information and can guide the random walk of a sampler efficiently. Instead of computing and saving the full Hessian of the log posterior, we use limited memory of the samples and their stochastic gradients to approximate the inverse Hessian-vector multiplication in the updating formula. Moreover, by smoothly optimizing the preconditioning matrix via SA, our proposed algorithm can asymptotically converge to the target distribution with a controllable bias under mild conditions. To reduce the training and testing computational burden, we adopt a magnitude-based weight pruning method to enforce the sparsity of the network. Our method is user-friendly and demonstrates better learning results compared to standard SG-MCMC updating rules. The approximation of inverse Hessian alleviates storage and computational complexities for large dimensional models. Numerical experiments are performed on several problems, including sampling from 2D correlated distribution, synthetic regression problems, and learning the numerical solutions of heterogeneous elliptic PDE. The numerical results demonstrate great improvement in both the convergence rate and accuracy.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2014

Approximate Hessian for accelerating ab initio structure relaxation by force fitting

We present a method to approximate the Hessian matrix of the Born-Oppenheimer energy landscape by using a simple force field model whose parameters are fitted to on-the-flight ab initio results. The inversed Hessian matrix is used as the preconditioner of conjugate gradient algorithms to speed up the atomic structure relaxation, resulting in a speedup factor of 2 to 5 on systems of bulk, slab, sheets, and atomic clusters. Because the force field model employed is simple and general, the parameter fitting is straightforward; the method is applicable to a variety of complicated systems for minimum structure relaxation. In the metal cluster new structure search, the new method yields better structures than the one obtained before with conventional algorithms.

36 MATERIALS SCIENCE↗
DOE OSTI2023

Hierarchical off-diagonal low-rank approximation of Hessians in inverse problems, with application to ice sheet model initialization

Obtaining lightweight and accurate approximations of discretized objective functional Hessians in inverse problems governed by partial differential equations (PDEs) is essential to make both deterministic and Bayesian statistical large-scale inverse problems computationally tractable. The cubic computational complexity of dense linear algebraic tasks, such as Cholesky factorization, that provide a means to sample Gaussian distributions and determine solutions of Newton linear systems is a computational bottleneck at large-scale. These tasks can be reduced to log-linear complexity by utilizing hierarchical off-diagonal low-rank (HODLR) matrix approximations. In this work, we show that a class of Hessians that arise from inverse problems governed by PDEs are well approximated by the HODLR matrix format. In particular, we study inverse problems governed by PDEs that model the instantaneous viscous flow of ice sheets. In these problems, we seek a spatially distributed basal sliding parameter field such that the flow predicted by the ice sheet model is consistent with ice sheet surface velocity observations. Here, we demonstrate the use of HODLR Hessian approximation to efficiently sample the Laplace approximation of the posterior distribution with covariance further approximated by HODLR matrix compression. Computational studies are performed which illustrate ice sheet problem regimes for which the Gauss–Newton data-misfit Hessian is more efficiently approximated by the HODLR matrix format than the low-rank (LR) format. We then demonstrate that HODLR approximations can be favorable, when compared to global LR approximations, for large-scale problems by studying the data-misfit Hessian associated with inverse problems governed by the first-order Stokes flow model on the Humboldt glacier and Greenland ice sheet.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

A randomized sketching trust-region secant method for low-memory dynamic optimization

The numerical solution of dynamic optimization problems is often limited by the memory required to store the state trajectory, which is used to evaluate the objective function and its derivatives. Recently, [R. Muthukumar et al., SIAM Journal on Optimization 31(2), pp. 1242–1275 (2021)] introduced a trust-region method for dynamic optimization that employs randomized sketching to compress the state trajectory, resulting in inexact derivative computations. By adaptively learning the sketch rank, the trust-region algorithm achieves rigorous convergence guarantees. Here, we extend this approach to use secant Hessian approximations. Due to the randomness introduced by the sketch, the traditional secant update formulae can produce poor Hessian approximations. In particular, the difference of two gradients, computed from two different sketches, may be inconsistent. To overcome this, we employ a sketched approximation of the Hessian application, in lieu of computing the gradient difference. We numerically demonstrate the improved stability of this approach on an example from PDE-constrained optimization.

dynamic optimization↗
NASA NTRS1998

Accelerated Training for Large Feedforward Neural Networks

In this paper we introduce a new training algorithm, the scaled variable metric (SVM) method. Our approach attempts to increase the convergence rate of the modified variable metric method. It is also combined with the RBackprop algorithm, which computes the product of the matrix of second derivatives (Hessian) with an arbitrary vector. The RBackprop method allows us to avoid computationally expensive, direct line searches. In addition, it can be utilized in the new, 'predictive' updating technique of the inverse Hessian approximation. We have used directional slope testing to adjust the step size and found that this strategy works exceptionally well in conjunction with the Rbackprop algorithm. Some supplementary, but nevertheless important enhancements to the basic training scheme such as improved setting of a scaling factor for the variable metric update and computationally more efficient procedure for updating the inverse Hessian approximation are presented as well. We summarize by comparing the SVM method with four first- and second- order optimization algorithms including a very effective implementation of the Levenberg-Marquardt method. Our tests indicate promising computational speed gains of the new training technique, particularly for large feedforward networks, i.e., for problems where the training process may be the most laborious.

Stepniewski, Slawomir W.↗
NASA NTRS1987

Recursive inverse kinematics for robot arms via Kalman filtering and Bryson-Frazier smoothing

This paper applies linear filtering and smoothing theory to solve recursively the inverse kinematics problem for serial multilink manipulators. This problem is to find a set of joint angles that achieve a prescribed tip position and/or orientation. A widely applicable numerical search solution is presented. The approach finds the minimum of a generalized distance between the desired and the actual manipulator tip position and/or orientation. Both a first-order steepest-descent gradient search and a second-order Newton-Raphson search are developed. The optimal relaxation factor required for the steepest descent method is computed recursively using an outward/inward procedure similar to those used typically for recursive inverse dynamics calculations. The second-order search requires evaluation of a gradient and an approximate Hessian. A Gauss-Markov approach is used to approximate the Hessian matrix in terms of products of first-order derivatives. This matrix is inverted recursively using a two-stage process of inward Kalman filtering followed by outward smoothing. This two-stage process is analogous to that recently developed by the author to solve by means of spatial filtering and smoothing the forward dynamics problem for serial manipulators.

Rodriguez, G.↗
DOE OSTI2014

A finite difference Davidson procedure to sidestep full ab initio hessian calculation: Application to characterization of stationary points and transition state searches

The cost of calculating nuclear hessians, either analytically or by finite difference methods, during the course of quantum chemical analyses can be prohibitive for systems containing hundreds of atoms. In many applications, though, only a few eigenvalues and eigenvectors, and not the full hessian, are required. For instance, the lowest one or two eigenvalues of the full hessian are sufficient to characterize a stationary point as a minimum or a transition state (TS), respectively. We describe here a method that can eliminate the need for hessian calculations for both the characterization of stationary points as well as searches for saddle points. A finite differences implementation of the Davidson method that uses only first derivatives of the energy to calculate the lowest eigenvalues and eigenvectors of the hessian is discussed. This method can be implemented in conjunction with geometry optimization methods such as partitioned-rational function optimization (P-RFO) to characterize stationary points on the potential energy surface. With equal ease, it can be combined with interpolation methods that determine TS guess structures, such as the freezing string method, to generate approximate hessian matrices in lieu of full hessians as input to P-RFO for TS optimization. This approach is shown to achieve significant cost savings relative to exact hessian calculation when applied to both stationary point characterization as well as TS optimization. The basic reason is that the present approach scales one power of system size lower since the rate of convergence is approximately independent of the size of the system. Therefore, the finite-difference Davidson method is a viable alternative to full hessian calculation for stationary point characterization and TS search particularly when analytical hessians are not available or require substantial computational effort.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2014

Using Hessian update formulae to construct modified Shepard interpolated potential energy surfaces: Application to vibrating surface atoms

Modified Shepard interpolation based on second order Taylor series expansions has proven to be a flexible tool for constructing potential energy surfaces in a range of situations. Extending this to gas–surface dynamics where surface atoms are allowed to move represents a substantial increase in the dimensionality of the problem, reflected in a dramatic increase in the computational cost of the required Hessian (matrix of second derivatives) evaluations. This work demonstrates that using approximate Hessians derived from well known Hessian update formulae and a single accurate Hessian can provide an effective way to avoid this expensive accurate Hessian determination.

74 ATOMIC AND MOLECULAR PHYSICS↗
DOE OSTI2019

A memory-distributed quasi-Newton solver for nonlinear programming problems with a small number of general constraints

We approach the problem of parallelizing state-of-the-art nonlinear programming optimization algorithms.Specifically, we focus on parallelizing quasi-Newton interior-point methods that use limited-memory secant Hessian approximations. Such interior-point methods are known to have better convergence properties and to be more effective on large-scale problems than gradient-based and derivative-free optimization algorithms. We target nonlinear and potentially nonconvex optimization problems with an arbitrary number of bound constraints and a small number of general equality and inequality constraints on the optimization variables. These problems occur for example in the form of optimal control, optimal design, and inverse problems governed by ordinary or partial differential equations, whenever they are expressed in a “reduced-space” optimization approach. We introduce and analyze the time and space complexity of a decomposition method for solving the quasi-Newton linear systems that leverages the fact that the quasi-Newton Hessian matrix has a small number of dense blocks that border a low-rank update of a diagonal matrix. This enables an efficient parallelization on memory-distributed computers of the iterations of the optimization algorithm, a state-of-the-art filter line-search interior-point algorithm by Wächter et. al. We illustrate the efficiency of the proposed method by solving structural topology optimization problems on up to 4608 cores on a parallel machine.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Surrogate optimization of variational quantum circuits

Variational quantum eigensolvers are touted as a near-term algorithm capable of impacting many applications. However, the potential has not yet been realized, with few claims of quantum advantage and high resource estimates, especially due to the need for optimization in the presence of noise. Finding algorithms and methods to improve convergence is important to accelerate the capabilities of near-term hardware for VQE or more broad applications of hybrid methods in which optimization is required. To this goal, we look to use modern approaches developed in circuit simulations and stochastic classical optimization, which can be combined to form a surrogate optimization approach to quantum circuits. Using an approximate (classical CPU/GPU) state vector simulator as a surrogate model, we efficiently calculate an approximate Hessian, passed as an input for a quantum processing unit or exact circuit simulator. This method will lend itself well to parallelization across quantum processing units. We demonstrate the capabilities of such an approach with and without sampling noise and a proof-of-principle demonstration on a quantum processing unit utilizing 40 qubits.

Gustafson, Erik J. [RIACS, Mtn. View] (ORCID:00000↗
DOE OSTI2016

Global adjoint tomography: first-generation model

We present the first-generation global tomographic model constructed based on adjoint tomography, an iterative full-waveform inversion technique. Synthetic seismograms were calculated using GPU-accelerated spectral-element simulations of global seismic wave propagation, accommodating effects due to 3-D anelastic crust & mantle structure, topography & bathymetry, the ocean load, ellipticity, rotation, and self-gravitation. Fréchet derivatives were calculated in 3-D anelastic models based on an adjoint-state method. The simulations were performed on the Cray XK7 named ‘Titan’, a computer with 18 688 GPU accelerators housed at Oak Ridge National Laboratory. The transversely isotropic global model is the result of 15 tomographic iterations, which systematically reduced differences between observed and simulated three-component seismograms. Our starting model combined 3-D mantle model S362ANI with 3-D crustal model Crust2.0. We simultaneously inverted for structure in the crust and mantle, thereby eliminating the need for widely used ‘crustal corrections’. We used data from 253 earthquakes in the magnitude range 5.8 ≤ M w ≤ 7.0. We started inversions by combining ~30 s body-wave data with ~60 s surface-wave data. The shortest period of the surface waves was gradually decreased, and in the last three iterations we combined ~17 s body waves with ~45 s surface waves. We started using 180 min long seismograms after the 12th iteration and assimilated minor- and major-arc body and surface waves. The 15th iteration model features enhancements of well-known slabs, an enhanced image of the Samoa/Tahiti plume, as well as various other plumes and hotspots, such as Caroline, Galapagos, Yellowstone and Erebus. Furthermore, we see clear improvements in slab resolution along the Hellenic and Japan Arcs, as well as subduction along the East of Scotia Plate, which does not exist in the starting model. Point-spread function tests demonstrate that we are approaching the resolution of continental-scale studies in some areas, for example, underneath Yellowstone. Here, this is a consequence of our multiscale smoothing strategy in which we define our smoothing operator as a function of the approximate Hessian kernel, thereby smoothing gradients less wherever we have good ray coverage, such as underneath North America.

58 GEOSCIENCES↗
NASA NTRS1982

Minimization versus homotopy algorithms

The relative merits and demerits of the minimization techniques are assessed using globally convergent quasi-Newton algorithms on the one hand and the homotopy algorithms on the other hand for the solution of problems of nonlinear structural analysis. Like the homotopy algorithms, the globally convergent quasi-Newton algorithms are equally suited for the solution of the nonlinear equations of structural analysis directly without having to pose the problem as an equivalent minimization problem. In the close neighborhood of the limit and bifurcation points quasi-Newton algorithms experience difficulties. Homotopy algorithms are robust for practically all types of nonlinear problems but are computationally not as cost effective since they provide an extremely accurate prediction of the response by calculating it as a large number of points. Globally convergent algorithms can perform well with very approximate Hessians, while homotopy algorithms require extremely accurate Hessians. While quasi-Newton algorithms can be very easily structured to exploit sparsity and symmetry, homotopy algorithms are not presently so structured and would require special modifications for exploitation of such features without sacrificing robustness and global convergence.

Kamat, M. P.↗
NASA NTRS1991

A parallel trajectory optimization tool for aerospace plane guidance

A parallel trajectory optimization algorithm is being developed. One possible mission is to provide real-time, on-line guidance for the National Aerospace Plane. The algorithm solves a discrete-time problem via the augmented Lagrangian nonlinear programming algorithm. The algorithm exploits the dynamic programming structure of the problem to achieve parallelism in calculating cost functions, gradients, constraints, Jacobians, Hessian approximations, search directions, and merit functions. Special additions to the augmented Lagrangian algorithm achieve robust convergence, achieve (almost) superlinear local convergence, and deal with constraint curvature efficiency. The algorithm can handle control and state inequality constraints such as angle-of-attack and dynamic pressure constraints. Portions of the algorithm have been tested. The nonlinear programming core algorithm performs well on a variety of static test problems and on an orbit transfer problem. The parallel search direction algorithm can reduce wall clock time by a factor of 10 for this part of the computation task.

Psiaki, Mark L.↗
NASA NTRSDate not supplied

Surrogate Optimization for Quantum Circuits

Variational quantum Eigensolvers are touted as a near-term algorithm capable of impacting many applications. However, the potential has yet to be realized with few claims of quantum advantage and high resource estimates mainly due to the need for optimization in the presence of noise. Finding algorithms and methods to improve the convergence is essential to accelerate the capabilities of near-term hardware for VQE or more broad applications of hybrid methods in which optimization is required. To this goal we look to use modern approaches recently developed in circuit simulations and stochastic classical optimization that can be combined in a surrogate optimization approach to classical circuits. Using an approximate state vector simulator, we efficiently calculate an approximate Hessian, fed as an input for a detailed quantum circuit simulator. We demonstrate the capabilities of such an approach with and without sampling noise. We also show that this method outperforms Powell in the presence of quantum circuit shot noise by a factor of 2-4

quantum computing↗
NASA NTRS1985

A Residuals Approach to Filtering, Smoothing and Identification for Static Distributed Systems

An approach for state estimation and identification of spatially distributed parameters embedded in static distributed (elliptic) system models is advanced. The method of maximum likelihood is used to find parameter values that maximize a likelihood functional for the system model, or equivalently, that minimize the negative logarithm of this functional. To find the minimum, a Newton-Raphson search is conducted that from an initial estimate generates a convergent sequence of parameter estimates. For simplicity, a Gauss-Markov approach is used to approximate the Hessian in terms of products of first derivatives. The gradient and approximate Hessian are computed by first arranging the negative log likelihood functional into a form based on the square root factorization of the predicted covariance of the measurement process. The resulting data processing approach, referred to here by the new term of predicted data covariance square root filtering, makes the gradient and approximate Hessian calculations very simple. A closely related set of state estimates is also produced by the maximum likelihood method: smoothed estimates that are optimal in a conditional mean sense and filtered estimates that emerge from the predicted data covariance square root filter.

Rodriguez, G.↗
DOE OSTI2019

Ensemble Quasi-Newton HMC

We present a modification of the Hybrid Monte Carlo algorithm for tackling the critical slowing down of generating Markov chains of lattice gauge configurations towards the continuum limit. We propose a new method to exchange information within an ensemble of Markov chains, and use it to construct an approximate inverse Hessian matrix of the action inspired from quasi-Newton algorithms for optimization. The kinetic term of the molecular dynamics evolution includes the approximate Hessian for long distance couplings among the momenta. We show the result of applying the new algorithm to the U(1) gauge theory in two dimensions, and discuss our future plans.

Jin, Xiao-Yong↗
NASA NTRS1996

Analysis of the Hessian for Aerodynamic Optimization: Inviscid Flow

In this paper we analyze inviscid aerodynamic shape optimization problems governed by the full potential and the Euler equations in two and three dimensions. The analysis indicates that minimization of pressure dependent cost functions results in Hessians whose eigenvalue distributions are identical for the full potential and the Euler equations. However the optimization problems in two and three dimensions are inherently different. While the two dimensional optimization problems are well-posed the three dimensional ones are ill-posed. Oscillations in the shape up to the smallest scale allowed by the design space can develop in the direction perpendicular to the flow, implying that a regularization is required. A natural choice of such a regularization is derived. The analysis also gives an estimate of the Hessian's condition number which implies that the problems at hand are ill-conditioned. Infinite dimensional approximations for the Hessians are constructed and preconditioners for gradient based methods are derived from these approximate Hessians.

Arian, Eyal↗