Iterative methods for solving nonlinear least squares problems
Iterative methods for solving nonlinear least squares problems
ITERATIVE SOLUTION↗
SEARCH · Engineering Papers
Results for “nonlinear least squares problems”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Methods for nonlinear least squares problems and convergence proofs
Explore the source record for details and available documents.
Morrison, D. D.↗
Efficient implementation of a variable projection algorithm for nonlinear least squares problems
Explore the source record for details and available documents.
Krogh, F. T.↗
A quadratic-tensor model algorithm for nonlinear least-squares problems with linear constraints
A new algorithm for solving nonlinear least-squares and nonlinear equation problems is proposed which is based on approximating the nonlinear functions using the quadratic-tensor model by Schnabel and Frank. The algorithm uses a trust region defined by a box containing the current values of the unknowns. The algorithm is found to be effective for problems with linear constraints and dense Jacobian matrices.
Hanson, R. J.↗
A new algorithm for constrained nonlinear least-squares problems, part 1
A Gauss-Newton algorithm is presented for solving nonlinear least squares problems. The problem statement may include simple bounds or more general constraints on the unknowns. The algorithm uses a trust region that allows the objective function to increase with logic for retreating to best values. The computations for the linear problem are done using a least squares system solver that allows for simple bounds and linear constraints. The trust region limits are defined by a box around the current point. In its current form the algorithm is effective only for problems with small residuals, linear constraints and dense Jacobian matrices. Results on a set of test problems are encouraging.
Hanson, R. J.↗
A partial-step algorithm for enlarging the convergence region of the nonlinear estimation problem
Weighted least squares nonlinear estimation problem, presenting partial step algorithm for convergence region enlargement
Boggs, D. H.↗
TAO Users Manual (Rev. 3.15)
The Toolkit for Advanced Optimization (TAO) focuses on the development of algorithms and software for the solution of large-scale optimization problems on high-performance architectures. Areas of interest include unconstrained and bound-constrained optimization, nonlinear least squares problems, optimization problems with partial differential equation constraints, and variational inequalities and complementarity constraints. The development of TAO was motivated by the scattered support for parallel computations and the lack of reuse of external toolkits in current optimization software. Our aim is to produce high-quality optimization software for computing environments ranging from workstations and laptops to massively parallel high-performance architectures. Our design decisions are strongly motivated by the challenges inherent in the use of large-scale distributed memory architectures and the reality of working with large, often poorly structured legacy codes for specific applications.
97 MATHEMATICS AND COMPUTING↗
ReMU: regional minimal updating for model-based derivative-free optimization
Derivative-free optimization (DFO) problems are optimization problems where derivative information is unavailable or extremely difficult to obtain. Model-based DFO solvers have been applied extensively in scientific computing. Powell's NEWUOA (2004) [Powell, The NEWUOA software for unconstrained optimization without derivatives, in Large-Scale Nonlinear Optimization, Nonconvex Optimization and its Applications Vol. 83, G. Di Pillo and M. Roma, eds., Springer, 2006, pp. 255–297] and Wild's POUNDerS (2014) [Wild, Solving derivative-free nonlinear least squares problems with POUNDERS, in Advances and Trends in Optimization with Engineering Applications, T. Terlaky, M.F. Anjos, and S. Ahmed, eds., SIAM, 2017, pp. 529–540] explore the numerical power of the minimal norm Hessian (MNH) model for DFO and contributed to the open discussion on building better models with fewer data to achieve faster numerical convergence. Another decade later, we propose the regional minimal updating (ReMU) models, and extend the previous models into a broader class, including the H 2 norm models [Xie and Yuan, Least H 2 norm updating of quadratic interpolation models for derivative-free trust-region algorithms, IMA J. Numer. Anal. 46 (2025), pp. 21–50]. This paper shows motivation behind ReMU models, computational details, theoretical and numerical results on particular extreme points and the barycentre of ReMU's weight coefficient region, and the associated KKT matrix error and distance. Novel metrics, such as the truncated Newton step error, are proposed to numerically understand the new models' properties. A new algorithmic strategy, based on iteratively adjusting the ReMU model type, is also proposed, and shows numerical advantages by combining and switching between the barycentric model and the classic least Frobenius norm model in an online fashion.
derivative-free trust-region methods↗
Derivative-free stochastic optimization via adaptive sampling strategies
In this paper, we present a novel derivative-free framework for solving unconstrained stochastic optimization problems. Many problems in fields ranging from simulation optimization to reinforcement learning to quantum computing involve settings where only stochastic function values are obtained via a zeroth-order oracle, which has no available gradient information and necessitates the usage of derivative-free optimization methodologies. Our approach includes estimating gradients using stochastic function evaluations and integrating adaptive sampling techniques to control the accuracy in these stochastic approximations. Our framework encapsulates several gradient estimation techniques, including standard finite-difference, Gaussian smoothing, sphere smoothing, randomized coordinate finite-difference, and randomized subspace finite-difference methods. We provide theoretical convergence guarantees for our framework and analyze the worst-case iteration and sample complexities associated with each gradient estimation method. Finally, we demonstrate the empirical performance of the methods on logistic regression and nonlinear least squares problems.
Adaptive sampling↗
A Scalable Space-Time Domain Decomposition Approach for Solving Large Scale Nonlinear Regularized Inverse Ill Posed Problems in 4D Variational Data Assimilation
We address the development of innovative algorithms designed to solve the strong-constraint Four Dimensional Variational Data Assimilation (4DVar DA) problems in large scale applications. We present a space-time decomposition approach which employs the whole domain decomposition, i.e. both along the spacial and temporal direction in the overlapping case, and the partitioning of both the solution and the operator. Starting from the global functional defined on the entire domain, we get to a sort of regularized local functionals on the set of sub domains providing the order reduction of both the predictive and the Data Assimilation models. The algorithm convergence is developed. Performance in terms of reduction of time complexity and algorithmic scalability is discussed on the Shallow Water Equations on the sphere. The number of state variables in the model, the number of observations in an assimilation cycle, as well as numerical parameters as the discretization step in time and in space domain are defined on the basis of discretization grid used by data available at repository Ocean Synthesis/Reanalysis Directory of Hamburg University.
97 MATHEMATICS AND COMPUTING↗
Determination of Mineral Type, Abundance, and Grain Size with Remotely Obtained Reflectance Spectra
To derive the abundances of minerals in a surface composed of more then one mineral in an intimate mixture, the bidirectional reflectance spectrum must be measured at several angles of incidence and emission. From these measurements, the scattering parameters may be derived, and the effective single scattering albedo can be computed from the models. If the scattering parameters are not derived, and the particles scatter anisotropically, then the computed effective single scattering albedo has been computed, the solution to the model is a nonlinear least squares problem with three unknowns per particle type. The solution is feasible when the complex indices of refraction and the densities of each mineral component are known. The unknowns are: the grain size of each particle type, the mass fraction of each particle type, and a grain internal scattering coefficient that describes grain imperfections.
Clark, R. N.↗
A moving discontinuous Galerkin finite element method with interface condition enforcement for compressible flows
A variation of moving discontinuous Galerkin finite element method with interface condition enforcement (MDG-ICE) is developed for solving the compressible Euler equations. The MDG-ICE method, originating from the work of Corrigan et al. [1], [2], [3], [4], is based on the space-time DG formulation, where both flow field and grid geometry are considered as independent variables and the conservation laws are enforced both on discrete elements and element interfaces. The element conservation laws are solved in the standard discontinuous solution space to determine conservative quantities, while the interface conservation is enforced using a variational formulation in a continuous space to determine discrete grid geometry. The resulting over-determined system of nonlinear equations arising from the MDG-ICE formulation can then be solved in a least-squares sense, leading to an unconstrained nonlinear least-squares problem that is regularized and solved by Levenberg-Marquardt method. A number of numerical experiments for both 1D unsteady and 2D steady state compressible flow problems are conducted to assess the accuracy and robustness of the MDG-ICE method. Numerical results obtained indicate that the MDG-ICE method is able to implicitly detect and track all types of discontinuities via interface conservation enforcement and satisfy the conservation law on both elements and interfaces via grid movement and grid management, demonstrating that an exponential rate of convergence for Sod and Lax-Harden shock tube problems can be achieved and highly accurate solutions without overheating to both double-rarefaction wave and Noh problems can be obtained.
97 MATHEMATICS AND COMPUTING↗
Optimized System Identification
In system identification, one usually cares most about finding a model whose outputs are as close as possible to the true system outputs when the same input is applied to both. However, most system identification algorithms do not minimize this output error. Often they minimize model equation error instead, as in typical least-squares fits using a finite-difference model, and it is seen here that this distinction is significant. Here, we develop a set of system identification algorithms that minimize output error for multi-input/multi-output and multi-input/single-output systems. This is done with sequential quadratic programming iterations on the nonlinear least-squares problems, with an eigendecomposition to handle indefinite second partials. This optimization minimizes a nonlinear function of many variables, and hence can converge to local minima. To handle this problem, we start the iterations from the OKID (Observer/Kalman Identification) algorithm result. Not only has OKID proved very effective in practice, it minimizes an output error of an observer which has the property that as the data set gets large, it converges to minimizing the criterion of interest here. Hence, it is a particularly good starting point for the nonlinear iterations here. Examples show that the methods developed here eliminate the bias that is often observed using any system identification methods of either over-estimating or under-estimating the damping of vibration modes in lightly damped structures.
Juang, Jer-Nan↗
Train Like a (Var)Pro: Efficient Training of Neural Networks with Variable Projection
Deep neural networks (DNNs) have achieved state-of-the-art performance across a variety of traditional machine learning tasks, e.g., speech recognition, image classification, and segmentation. The ability of DNNs to efficiently approximate high-dimensional functions has also motivated their use in scientific applications, e.g., to solve partial differential equations and to generate surrogate models. In this paper, we consider the supervised training of DNNs, which arises in many of the above applications. We focus on the central problem of optimizing the weights of the given DNN such that it accurately approximates the relation between observed input and target data. Devising effective solvers for this optimization problem is notoriously challenging due to the large number of weights, nonconvexity, data sparsity, and nontrivial choice of hyperparameters. To solve the optimization problem more efficiently, we propose the use of variable projection (VarPro), a method originally designed for separable nonlinear least-squares problems. Our main contribution is the Gauss--Newton VarPro method (GNvpro) that extends the reach of the VarPro idea to nonquadratic objective functions, most notably cross-entropy loss functions arising in classification. These extensions make GNvpro applicable to all training problems that involve a DNN whose last layer is an affine mapping, which is common in many state-of-the-art architectures. In our four numerical experiments from surrogate modeling, segmentation, and classification, GNvpro solves the optimization problem more efficiently than commonly used stochastic gradient descent (SGD) schemes. Finally, GNvpro finds solutions that generalize well, and in all but one example better than well-tuned SGD methods, to unseen data points.
97 MATHEMATICS AND COMPUTING↗
A Newton algorithm for complex curve fitting
The problem of synthesizing transfer functions from frequency response measurements is considered. Given a complex vector representing the measured frequency response of a physical system, a transfer function of specified order is determined that minimizes the sum of the magnitude-squared of the frequency response errors. This nonlinear least squares minimization problem is solved by an iterative global descent algorithm of the Newton type which converges quadratically near the minimum. The unknown transfer function is expressed as a sum of second order rational polynomials, a parameterization that facilitates a numerically robust computer implementation. The algorithm is developed for single-input, single-output, causal, stable transfer functions.
Spanos, J. T.↗
Newton algorithm for fitting transfer functions to frequency response measurements
In this paper the problem of synthesizing transfer functions from frequency response measurements is considered. Given a complex vector representing the measured frequency response of a physical system, a transfer function of specified order is determined that minimizes the sum of the magnitude-squared of the frequency response errors. This nonlinear least squares minimization problem is solved by an iterative global descent algorithm of the Newton type that converges quadratically near the minimum. The unknown transfer function is expressed as a sum of second-order rational polynomials, a parameterization that facilitates a numerically robust computer implementation. The algorithm is developed for single-input, single-output, causal, stable transfer functions. Two numerical examples demonstrate the effectiveness of the algorithm.
Spanos, J. T.↗
Photometric theory for wide-angle phenomena
An examination is made of the problem posed by wide-angle photographic photometry, in order to extract a photometric-morphological history of Comet P/Halley. Photometric solutions are presently achieved over wide angles through a generalization of an assumption-free moment-sum method. Standard stars in the field allow a complete solution to be obtained for extinction, sky brightness, and the characteristic curve. After formulating Newton's method for the solution of the general nonlinear least-square problem, an implementation is undertaken for a canonical data set. Attention is given to the problem of random and systematic photometric errors.
Usher, Peter D.↗
Quasi-periodic travelling gravity–capillary waves
We present a numerical study of spatially quasi-periodic travelling waves on the surface of an ideal fluid of infinite depth. This is a generalization of the classic Wilton ripple problem to the case when the ratio of wavenumbers satisfying the dispersion relation is irrational. We propose a conformal mapping formulation of the water wave equations that employs a quasi-periodic variant of the Hilbert transform to compute the normal velocity of the fluid from its velocity potential on the free surface. We develop a Fourier pseudo-spectral discretization of the travelling water wave equations in which one-dimensional quasi-periodic functions are represented by two-dimensional periodic functions on the torus. This leads to an overdetermined nonlinear least-squares problem that we solve using a variant of the Levenberg–Marquardt method. We investigate various properties of quasi-periodic travelling waves, including Fourier resonances, time evolution in conformal space on the torus, asymmetric wave crests, capillary wave patterns that change from one gravity wave trough to the next without repeating and the dependence of wave speed and surface tension on the amplitude parameters that describe a two-parameter family of waves.
97 MATHEMATICS AND COMPUTING↗