Higher-Order Corrections to Optimisers based on Newton's Method

Abstract

The Newton, Gauss–Newton and Levenberg–Marquardt methods all use the first derivative of a vector function (the Jacobian) to minimise its sum of squares. When the Jacobian matrix is ill-conditioned, the function varies much faster in some directions than others and the space of possible improvement in sum of squares becomes a long narrow ellipsoid in the linear model. This means that even a small amount of nonlinearity in the problem parameters can cause a proposed point far down the long axis of the ellipsoid to fall outside of the actual curved valley of improved values, even though it is quite nearby. This paper presents a differential equation that ‘follows’ these valleys, based on the technique of geodesic acceleration, which itself provides a 2 nd order improvement to the Levenberg–Marquardt iteration step. Higher derivatives of this equation are computed that allow n th order improvements to the optimisation methods to be derived. These higher-order accelerated methods up to 4 th order are tested numerically and shown to provide substantial reduction of both number of steps and computation time.