Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Gradient information”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

Layer-Parallel Training of Deep Residual Neural Networks

Residual neural networks (ResNets) are a promising class of deep neural networks that have shown excellent performance for a number of learning tasks, e.g., image classification and recognition. Mathematically, ResNet architectures can be interpreted as forward Euler discretizations of a nonlinear initial value problem whose time-dependent control variables represent the weights of the neural network. Hence, training a ResNet can be cast as an optimal control problem of the associated dynamical system. For similar time-dependent optimal control problems arising in engineering applications, parallel-in-time methods have shown notable improvements in scalability. This paper demonstrates the use of those techniques for efficient and effective training of ResNets. The proposed algorithms replace the classical (sequential) forward and backward propagation through the network layers with a parallel nonlinear multigrid iteration applied to the layer domain. This adds a new dimension of parallelism across layers that is attractive when training very deep networks. From this basic idea, we derive multiple layer-parallel methods. The most efficient version employs a simultaneous optimization approach where updates to the network parameters are based on inexact gradient information in order to speed up the training process. Finally, using numerical examples from supervised classification, we demonstrate that the new approach achieves a training performance similar to that of traditional methods, but enables layer-parallelism and thus provides speedup over layer-serial methods through greater concurrency.

97 MATHEMATICS AND COMPUTING↗

Rapid Optimization Library

ROL provides interfaces to and implementations of algorithms for gradient-based unconstrained and constrained optimization. ROL can be used to optimize the response of any client simulation code that evaluates scalar-valued response functions. If the client code can provide gradient information for the response function, ROL will take advantage of it, resulting in faster runtimes. ROL's interfaces are matrix-free, in other words ROL only uses evaluations of scalar-valued and vector-valued functions. ROL can be used to solve optimal design problems and inverse problems based on a variety of simulation software.

Kouri, DrewKouri↗

Robust Uncertainty Quantification using Response Surface Approximations of Discontinuous Functions

This paper considers response surface approximations for discontinuous quantities of interest. Our objective is not to adaptively characterize the manifold defining the discontinuity. Instead, we utilize an epistemic description of the uncertainty in the location of a discontinuity to produce robust bounds on sample-based estimates of probabilistic quantities of interest. We demonstrate that two common machine learning strategies for classification, one based on nearest neighbors (Voronoi cells) and one based on support vector machines, provide reasonable descriptions of the region where the discontinuity may reside. In higher dimensional spaces, we demonstrate that support vector machines are more accurate for discontinuities defined by smooth manifolds. We also show how gradient information, often available via adjoint-based approaches, can be used to define indicators to effectively detect a discontinuity and to decompose the samples into clusters using an unsupervised learning technique. Numerical results demonstrate the epistemic bounds on probabilistic quantities of interest for simplistic models and for a compressible fluid model with a shock-induced discontinuity.

97 MATHEMATICS AND COMPUTING↗

Holistic Portfolio Optimization using Directed Mutations

Genetic algorithms provide attractive options for performing nonlinear multi-objective combinatorial design optimization, and they have proven very useful for optimizing individual systems. However, conventional genetic algorithms fall short when performing holistic portfolio optimizations in which the decision variables also include the integer counts of multiple system types over multiple time periods. When objective functions are formulated as analytic functions, we can formally differentiate with respect to system counts and use the resulting gradient information to generate favorable mutations in the count variables. We apply several variations on this basic idea to an idealized hanging chain example to obtain >> 1000x speedups over conventional genetic algorithms in both single - and multi-objective cases. We develop a more complex example of a notional military portfolio that includes combinatorial design variables and dependency constraints between the design choices. In this case, our initial results are mixed, but many variations are still open to further research.

97 MATHEMATICS AND COMPUTING↗

MULTI-LEADER: MULTI-source LEarning-Accelerated Design of high-Efficiency multi-stage compRessor (Final Technical Report)

The objective of MULTI-LEADER is to cut design costs by 80% while generating more energy-efficient designs of multi-stage compressors by developing and implementing novel machine learning (ML) techniques, which enable faster and fewer design iterations, improved solver performance, and concurrent multi-disciplinary design. Current industrial practices for the design of multi-stage compressors involve simulation-based design optimization with successive levels of model fidelity, iteratively evaluated between distinct disciplines, one stage at a time to tackle the high dimensional design variations. This project addresses these key design challenges: (1) concurrent optimization of multiple stages under many non-linear constraints; (2) multitude of evaluation of high-fidelity and expensive solvers and their gradients during optimization convergence in high-dimensional design; (3) multi-disciplinary design to maximize aerodynamic performance while guaranteeing structural integrity and additive manufacturability; (4) utilization of multiple fidelity of solvers with disparate parameterization and modeling assumptions. MULTI-LEADER achieved more than 5x speed up in detailed design of more energy-efficient compressors via these machine learning (ML) innovations: (i) rapid design surrogates by multi-source learning from diverse fidelities across multiple disciplines, (ii) physics-constrained data-augmented modeling for improved empiricism, (iii) generative manifold embedding for high dimensional concurrent design without gradient information; (iv) budget-constrained fidelity-adaptive sampling towards fewer design iterations.

33 ADVANCED PROPULSION SYSTEMS↗

Differentiable Multiphysics Codes: A Breakthrough Technology for Simulation and Computing

This document summarizes the findings of a strategic planning exercise commissioned by the Weapons Simulation and Computing, Computational Physics (WSC/CP) program at the Lawrence Livermore National Laboratory (LLNL) in FY24. During the year, the committee met with multiple stakeholder communities to gather input, opinions, suggestions and concerns which have been incorporated throughout this document. The key findings from this exercise are summarized: • The development of multiphysics modelling and simulation (mod/sim) codes and software technologies, their deployment on exascale compute platforms, and their broad adoption across the NNSA is a major success of the Advanced Simulation and Computing (ASC) program and the Exascale Computing Project (ECP). Sustained investment in these core technologies is essential. • Today’s state of the art involves running ensembles of O(100K) simulations to perform uncertainty quantification (UQ) and design studies using multiple statistical methods such as Bayesian optimization to understand sensitivities of our models and explore parameterized design spaces. Even with exascale computing, we are practically limited to O(10) parameters in these studies since the number of simulations required to sample the space scales exponentially with the number of design parameters. • The data from these simulation ensembles is increasingly being used to train machine learned (ML) surrogates (or reduced order models, ROMs) which can then be used for optimization or real time design exploration. However, the trained surrogates are still limited in the number of parameters they can represent due to the sampling limitations previously noted. • Augmenting our suite of integrated multiphysics simulation codes, both current and emerging, with the ability to compute gradients (solution derivatives) of arbitrary simulation outputs with respect to (some or all) simulation inputs would be a breakthrough technology, opening the door to a new era of efficient and automated inverse design based on verified and validated mod/sim capabilities. • This capability, which we refer to as differentiable multiphysics codes (DMCs), would revolutionize both UQ and optimization studies by breaking the curse of dimensionality that presently limits our “gradient-free” ensemble based computing approach. A similar breakthrough occurred in the AI/ML community once the ability to compute gradients of arbitrary loss functions using back-propagation became commonplace. Gradient information from the multiphysics codes can also be used to dramatically improve the efficiency and scale of training of ML/ROM surrogates for rapid assessments. • Achieving this in our suite of codes will be a grand challenge, similar to the amount of effort that was required to transition from CPU to GPU computing. It will require buy-in from the entire WSC/CP program and beyond, including all integrated codes, physics and engineering models, third-party library dependencies and performance portability abstractions. It will also require investment in research and development of numerical methods for computing adjoints of coupled physics across multiple adaptively refined moving meshes and of stochastic (Monte Carlo) and mesh free (SPH) methods. • New software and numerical techniques, largely pioneered by the AI/ML community, make this feasible. Chief among these is automatic differentiation (AD), the ability to employ AD at point-wise locations in a physics calculation (instead of traditional black-box approaches) and the ability to perform “back-propagation in time” (or reverse mode AD) for non-linear partial differential equations (PDEs). Fundamentally, the conclusion of this strategic planning exercise is that the time is right to undertake a large scale effort in WSC, centered on the existing integrated codes, to continue the natural evolution of mod/sim in the age of AI/ML. Instead of attempting to replace mod/sim with purely data driven AI/ML models, we believe the key to success is to integrate AI/ML by building on top of the decades of hard-won knowledge and the verified/validated multiphysics modelling capability that is the hallmark of the ASC program.

97 MATHEMATICS AND COMPUTING↗

Automated tuning for HMC mass ratios

We extended previous work on tuning HMC parameters using gradient information to include Hasenbusch mass ratios. The inclusion of mass ratios adds many more parameters that need to be tuned, and also allows for lots of variations in the choice of integrator pattern. We investigate the effectiveness of automatically tuning a large number of HMC parameters and compare the optimally tuned versions over a range of integrator variants.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Structural optimization by methods of feasible directions.

A general design algorithm based on methods of feasible directions is presented. Zoutendijk's method of feasible directions is first presented as applied to structural design. This method is modified to improve numerical stability of the design process and is then further modified to deal efficiently with infeasible designs. The algorithm requires the analytic gradient of the objective function and the constraint functions which are active at a given stage in the design process. Gradient information is not required for nonactive constraints. Complex constraint functions may be ignored in the initial design stages because violation of these constraints is efficiently overcome later in the design process. The algorithm is demonstrated with elastic design of redundant trusses.

Vanderplaats, G. N.↗

Utilizing remote sensing data for land use decisions for Indian lands in South Dakota.

Evaluation of photographic and thermal infrared imagery acquired at an altitude of 11,500 ft above ground level (in southwestern South Dakota) on Oct. 15, 1970, June 30, 1971, and Aug. 2, 1971, for use in mapping range sites and soils. A density slicing system was used to enhance film optical density differences associated with range sites and soils. The range site boundaries in native rangeland areas were found to be delineated best by a density slicing analysis of color infrared film obtained in August 1971. The range site maps produced by density slicing were superior to existing range inventory maps. The density slicing analysis of color infrared film from the August 1971 flight was best for locating soil boundaries in rangeland areas. To adequately map soils in rangeland areas, data based on density slicing used in conjunction with slope gradient information derived from stereoscopic analyses provided more detailed and accurate information than is presently available from range site and soil maps of the area.

Frazee, C. J.↗

Study of genetic direct search algorithms for function optimization

The results are presented of a study to determine the performance of genetic direct search algorithms in solving function optimization problems arising in the optimal and adaptive control areas. The findings indicate that: (1) genetic algorithms can outperform standard algorithms in multimodal and/or noisy optimization situations, but suffer from lack of gradient exploitation facilities when gradient information can be utilized to guide the search. (2) For large populations, or low dimensional function spaces, mutation is a sufficient operator. However for small populations or high dimensional functions, crossover applied in about equal frequency with mutation is an optimum combination. (3) Complexity, in terms of storage space and running time, is significantly increased when population size is increased or the inversion operator, or the second level adaptation routine is added to the basic structure.

Zeigler, B. P.↗

Efficient calculation of optimum design sensitivity

Two methods for obtaining the sensitivity of an optimized design to changes in problem parameters are presented. The first-order method uses gradient information to estimate the required sensitivity. The second-order method uses second derivatives of the design objective and constraint functions to provide a quadratic approximation to the new design which results from changing the specified parameter. It is shown that the sensitivity of the optimized design may be discontinuous with respect to the new parameter. This discontinuity is accounted for in the present methods. These methods are compared to a previous approach based on the Kuhn-Tucker necessary conditions for optimality. It is shown that the present methods provide the appropriate sensitivity information in an efficient manner. The methods are demonstrated by a structural synthesis example.

Vanderplaats, G. N.↗

Hurricane Debby - Analysis and numerical forecasts using VAS soundings

The utility of VISSR Atmospheric Sounder (VAS) temperature and moisture soundings in defining the storm and its surroundings at subsynoptic scales has been examined using a numerical analysis and prognosis system. In particular, VAS temperature and moisture soundings and cloud and water vapor motion winds have been used in numerical analysis for three time periods. It is shown that the VAS temperature and moisture data which specify temperature and moisture well in cloud free regions are complemented by cloud and water vapor wind data which provide horizontal gradient information for the cloudy areas. The loss of analysis integrity due to the reduction of VAS data density in the cloudy regions associated with synoptic activity is ameliorated by using cloud and water vapor motion winds. The improvement in numerical forecasts resulting from the addition of these data to the numerical data base is also recorded.

Le Marshall, J. F.↗

Hurricane Debby - An illustration of the complementary nature of VAS soundings and cloud and water vapor motion winds

The utility of VISSR Atmospheric Sounder (VAS) temperature and moisture soundings and cloud and water vapor motion winds in defining a storm and its surroundings at subsynoptic scales has been examined using a numerical analysis and prognosis system. It is shown that the VAS temperature and moisture data, which specify temperature and moisture well in cloud-free areas, are complemented by cloud and water vapor motion data generated in the cloudy areas. The cloud and water vapor 'winds' provide thermal gradient information for interpolating the soundings across cloudy regions. The loss of analysis integrity due to the reduction of VAS sounding density in the cloudy regions associated with synoptic activity is ameliorated by using cloud and water vapor motion winds. The improvement in numerical forecasts resulting from the addition of these data to the numerical analysis is recorded.

Le Marshall, J. F.↗

The development of an environmental disturbance model for large space structures after the onset of thermal shock

Expressions for the temperature response across a thin large space structure during and immediately following the onset of thermal shock are obtained as a function of the properties of the material and the solar incidence angle. The thermal gradients induced due to solar radiation heating result in thermal deformation. From the temperature gradient information, expressions are developed for the time history of the thermally induced deflections based on the thermoelastic relationships. The effect of the variation of the solar incidence angle and the variation of the emissivity of the surface is considered. From the deflection time history, a model of the disturbance moments is developed and evaluated for different solar incidence angles.

Hamsath, N.↗

The assimilation of satellite soundings, winds and satellite products in a mesoscale analysis/forecast system

Investigations in FY-85 were centered on three case study days in 1982. Two of these, March 6 and April 24, were Atmospheric Variability Experiment/Verical Atmospheric Sounder (AVE/VAS) days for which high spatial and temporal resolution RAOB and Vertical Atmospheric Sounder (VAS) data sets were available. The third investigation day, April 26, was a day of interesting severe weather. In the last part of FY-84 and early FY-85 we were able to demonstrate most importantly the complimentary nature of satellite soundings and winds in a forecast/analysis system. In our variational analysis scheme, cloud drift and water vapor winds enter into the height field as gradient information. The cloud drift winds especially, have the character of supplying information in cloudy areas where satellite soundings are not possible. In the April 26 experiments, analyses and forecasts using the combination satellite winds and soundings were superior to those using only soundings. Good consistency was shown between independent satellite forecasts from different initialization times run to the same verification time. A significant accomplishment in FY-85 was expanding experiments on April 26 to include quasi-continuous initialization inserting satellite soundings and winds from several different times into an analysis/forecast. Contrary to the first set of experiments on April 26, here forecast initialization fields were not independent, but contained satellite information from two data times.

Diak, G. R.↗

Stability robustness improvement of direct eigenspace assignment based feedback systems using singular value sensitivities

A methodology to improve the stability robustness of feedback control systems designed using direct eigenspace assignment techniques is presented. The method consists of considering the sensitivity of the minimum singular value of the return difference transfer matrix at the plant input to small changes in the desired closed-loop eigenvalues and the specified elements of the desired closed-loop eigenvectors. Closed-form expressions for the gradient of the minimum return difference singular value with respect to desired closed-loop eigenvalue and eigenvector parameters are derived. Closed-form expressions for the gradients of the control feedback gains with respect to the specified eigenspace parameters are obtained as an intermediate step. The use of the gradient information to improve the guaranteed gain and phase margins in eigenspace assignment based designs is demonstrated by application to an advanced fighter aircraft.

Garg, Sanjay↗

Stability robustness improvement of direct eigenspace assignment based feedback systems using singular value sensitivities

A methodology to improve the stability robustness of feedback control systems designed using direct eigenspace assignment techniques is presented. The method consists of considering the sensitivity of the minimum singular value of the return difference transfer matrix at the plant input to small changes in the desired closed-loop eigenvalues and the specified elements of the desired closed-loop eigenvectors. Closed-form expressions for the gradient of the minimum return difference singular value with respect to desired closed-loop eigenvalue and eigenvector parameters are derived. Closed-form expressions for the gradients of the control feedback gains with respect to the specified eigenspace parameters are obtained as an intermediate step. The use of the gradient information to improve the guaranteed gain and phase margins in eigenspace assignment based designs is demonstrated by application to an advanced fighter aircraft.

Garg, Sanjay↗

Airfoil optimization with efficient gradient calculations

The viscous airfoil design analysis code XFOIL was extended to allow optimization using conformal mapping coefficients as design variables. The optimization technique used was the Steepest Descent method applied to a Penalty Function. The gradients of the aerodynamic variables with respect to the design variables were cheaply calculated as by-products of XFOIL's integral boundary layer Newton solver. The speed of the optimization process further increased by updating the Newton system boundary layer variables after each optimization step using the available gradient information. Two examples are presented.

Sorensen, Thomas↗