SEARCH · Engineering Papers
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
The relationship between minimizing a mean squared error and finding the optimal Bayesian classifier is reviewed. This provides a theoretical interpretation for the process by which neural networks are used in classification. A number of confidence measures are proposed to evaluate the performance of the neural network classifier within a statistical framework.
In this work, the permutation invariant polynomial neural network (PIP-NN) approach is employed to construct a quasi-diabatic Hamiltonian for system with non-Abelian symmetries. It provides a flexible and compact NN-based diabatic ansatz from the related approach of Williams, Eisfeld, and co-workers. The example of $H^3_+$ is studied, which is an (E + A) × (e + a) Jahn–Teller and Pseudo-Jahn–Teller system. The PIP-NN diabatic ansatz is based on the symmetric polynomial expansion of Viel and Eisfeld, the coefficients of which are expressed with neural network functions that take permutation-invariant polynomials as input. This PIP-NN-based diabatic ansatz not only preserves the correct symmetry but also provides functional flexibility to accurately reproduce ab initio electronic structure data, thus resulting in excellent fits. Further, the adiabatic energies, energy gradients, and derivative couplings are well reproduced. A good description of the local topology of the conical intersection seam is also achieved. Therefore, this diabatic ansatz completes the PIP-NN based representation of DPEM with correct symmetries and will enable us to diabatize even more complicated systems with complex symmetries.
We use a stable parallel approach to train Wasserstein Conditional Generative Adversarial Neural Networks (W-CGANs). The parallel training reduces the risk of mode collapse and enhances scalability by using multiple generators that are concurrently trained, each one of them focusing on a single data label. The use of the Wasserstein metric reduces the risk of cycling by stabilizing the training of each generator. We apply the approach on the CIFAR10 and the CIFAR100 datasets, two standard benchmark datasets with images of the same resolution, but different number of classes. Performance is assessed using the inception score, the Fréchet inception distance, and image quality. An improvement in inception score and Fréchet inception distance is shown in comparison to previous results obtained by performing the parallel approach on deep convolutional conditional generative adversarial neural networks (DC-CGANs). Weak scaling is attained on both datasets using up to 100 NVIDIA V100 GPUs on the OLCF supercomputer Summit.
Accurate self-motion estimation is critical for various navigational tasks in mobile robotics. Optic flow provides a means to estimate self-motion using a camera sensor and is particularly valuable in GPS- and radio-denied environments. The present study investigates the influence of different activation functions—ReLU, leaky ReLU, GELU, and Mish—on the accuracy, robustness, and encoding properties of convolutional neural networks (CNNs) and multi-layer perceptrons (MLPs) trained to estimate self-motion from optic flow. Our results demonstrate that networks with ReLU and leaky ReLU activation functions not only achieved superior accuracy in self-motion estimation from novel optic flow patterns but also exhibited greater robustness under challenging conditions. The advantages offered by ReLU and leaky ReLU may stem from their ability to induce sparser representations than GELU and Mish do. Our work characterizes the encoding of optic flow in neural networks and highlights how the sparseness induced by ReLU may enhance robust and accurate self-motion estimation from optic flow.
In numerous applications, the integration of prior knowledge and historical information is essential, particularly for tasks requiring the solution of ordinary or partial differential equations (ODEs/PDEs) in data-sparse or noisy environments. For instance, achieving accurate solutions to time-dependent PDEs with limited initial condition measurements necessitates an effective strategy for embedding prior knowledge. Hard-parameter sharing architectures in neural networks (NNs) have demonstrated success in both traditional and scientific machine learning domains, facilitating the learning of informative representations. Here, in this study, we introduce a novel, yet efficient, method to enhance physics-informed neural networks (PINNs) by incorporating a multi-head structure that enables the learning of functional priors from both empirical data and governing physical laws. This prior information can then be used to address data sparsity and high-level noise in solving ODE/PDE problems with uncertainty quantification (UQ). The approach, termed Multi-Head PINN (MH-PINN), consists of a shared body NN and multiple head NNs, each corresponding to an individual PINN instance. Our framework for functional prior learning is carried out in two stages: (1) training the MH-PINNs to develop a shared body NN alongside multiple head NNs, and (2) employing these trained head NNs to estimate a prior distribution through a normalizing flow-based density estimator. The learned functional prior can then be applied as a regularization mechanism in deterministic contexts or as an informative prior within a Bayesian inference framework, aiding in the resolution of subsequent ODE/PDE tasks. We evaluate the efficacy of MH-PINNs across five benchmark problems, including a high-dimensional parametric PDE, all characterized by data sparsity or substantial noise levels. Our findings reveal that MH-PINNs deliver accurate solutions and robust UQ, demonstrating adaptability across a range of complex and challenging scenarios.
Accuracy-optimized convolutional neural networks (CNNs) have emerged as highly effective models at predicting neural responses in brain areas along the primate ventral stream, but it is largely unknown whether they effectively model neurons in the complementary primate dorsal stream. We explored how well CNNs model the optic flow tuning properties of neurons in dorsal area MSTd and we compared our results with the Non-Negative Matrix Factorization (NNMF) model, which successfully models many tuning properties of MSTd neurons. To better understand the role of computational properties in the NNMF model that give rise to optic flow tuning that resembles that of MSTd neurons, we created additional CNN model variants that implement key NNMF constraints – non-negative weights and sparse coding of optic flow. While the CNNs and NNMF models both accurately estimate the observer's self-motion from purely translational or rotational optic flow, NNMF and the CNNs with nonnegative weights yield substantially less accurate estimates than the other CNNs when tested on more complex optic flow that combines observer translation and rotation. Despite its poor accuracy, NNMF gives rise to tuning properties that align more closely with those observed in primate MSTd than any of the accuracy-optimized CNNs. This work offers a step toward a deeper understanding of the computational properties and constraints that describe the optic flow tuning of primate area MSTd.
The Hückel Hamiltonian is an incredibly simple tight-binding model known for its ability to capture qualitative physics phenomena arising from electron interactions in molecules and materials. Part of its simplicity arises from using only two types of empirically fit physics-motivated parameters: the first describes the orbital energies on each atom and the second describes electronic interactions and bonding between atoms. By replacing these empirical parameters with machine-learned dynamic values, we vastly increase the accuracy of the extended Hückel model. The dynamic values are generated with a deep neural network, which is trained to reproduce orbital energies and densities derived from density functional theory. The resulting model retains interpretability, while the deep neural network parameterization is smooth and accurate and reproduces insightful features of the original empirical parameterization. Altogether, this work shows the promise of utilizing machine learning to formulate simple, accurate, and dynamically parameterized physics models.
It is critical that machine learning (ML) model predictions be trustworthy for high-throughput catalyst discovery approaches. Uncertainty quantification (UQ) methods allow estimation of the trustworthiness of an ML model, but these methods have not been well explored in the field of heterogeneous catalysis. Herein, we investigate different UQ methods applied to a crystal graph convolutional neural network to predict adsorption energies of molecules on alloys from the Open Catalyst 2020 dataset, the largest existing heterogeneous catalyst dataset. We apply three UQ methods to the adsorption energy predictions, namely k-fold ensembling, Monte Carlo dropout, and evidential regression. The effectiveness of each UQ method is assessed based on accuracy, sharpness, dispersion, calibration, and tightness. Evidential regression is demonstrated to be a powerful approach for rapidly obtaining tunable, competitively trustworthy UQ estimates for heterogeneous catalysis applications when using neural networks. Recalibration of model uncertainties is shown to be essential in practical screening applications of catalysts using uncertainties.
Accurate prediction of geothermal reservoir responses to alternative energy production scenarios is critical for optimizing the development of the underlying resources. While the conventional physics-based models offer a comprehensive prediction tool, data-driven models provide an efficient alternative to build fit-for-purpose predictive models by extracting and using the statistical patterns in the collected data to make predictions. The recurrent neural network (RNN) is a data-driven model that is commonly applied to predict time series sequences. This paper presents a variant of RNN that also utilizes the efficiency of convolutional neural networks (CNN) for the prediction of energy production from geothermal reservoirs. Specifically, a CNN–RNN architecture is developed that takes historical well controls as input (features) and their corresponding production response data as output (labels) to learn an input-output mapping that can predict the future well production responses/performance for any given future well control inputs. The model is paired with a labeling scheme to handle real field disturbances that create data gaps. In addition to the model structure, we introduce a thorough workflow for applying the model, which includes data pre-processing, feature selection, as well as different training strategies for short-term and long-term prediction. Finally, the performance and accuracy of the model are evaluated by applying it to multiple datasets, including a field reservoir model.
Autonomous manipulation is a challenging problem in field robotics due to uncertainty in object properties, constraints, and coupling phenomenon with robot control systems. Humans learn motion primitives over time to effectively interact with the environment. We postulate that autonomous manipulation can be enabled by basic sets of motion primitives as well, but do not necessitate mimicking human motion primitives. Here, this work presents an approach to generalized optimal motion primitives using physics-informed neural networks. Our simulated and experimental results demonstrate that optimality is notionally maintained where the mean maximum observed final position percent error was 0.564% and the average mean error for all the trajectories was 1.53%. These results indicate that notional generalization is attained using a physics-informed neural network approach that enables near optimal real-time adaptation of primitive motion profiles.
The 52-color asteroid survey (Bell et al., 1988) together with the 8-color asteroid survey (Zellner et al., 1985) provide a data set of asteroid spectra spanning 0.3-2.5 micrometers. An artificial neural network clusters these asteroid spectra based on their similarity to each other. We have also trained the neural network with a categorization learning output layer in a supervised mode to associate the established clusters with taxonomic classes. Results of our classification agree with Tholen's classification based on the 8-color data alone. When extending the spectral range using the 52-color survey data, we find that some modification of the Tholen classes is indicated to produce a cleaner, self-consistent set of taxonomic classes. After supervised training using our modified classes, the network correctly classifies both the training examples, and additional spectra into the correct class with an average of 90% accuracy. Our classification supports the separation of the K class from the S class, as suggested by Bell et al. (1987), based on the near-infrared spectrum. We define two end-member subclasses which seem to have compositional significance within the S class: the So class, which is olivine-rich and red, and the Sp class, which is pyroxene-rich and less red. The remaining S-class asteroids have intermediate compositions of both olivine and pyroxene and moderately red continua. The network clustering suggests some additional structure within the E-, M-, and P-class asteroids, even in the absence of albedo information, which is the only discriminant between these in the Tholen classification. New relationships are seen between the C class and related G, B, and F classes. However, in both cases, the number of spectra is too small to interpret or determine the significance of these separations.
Precise, smooth aerodynamic models are required for implementing adaptive, nonlinear control strategies. Accurate representations of aerodynamic coefficients can be generated for the complete flight envelope by combining computational neural network models with an Estimation-Before-Modeling paradigm for on-line training information. A novel method of incorporating first-partial-derivative information is employed to estimate the weights in individual feedforward neural networks for each aerodynamic coefficient. The method is demonstrated by generating a model of the normal force coefficient of a twin-jet transport aircraft from simulated flight data, and promising results are obtained.
Here we present the PI-CKL-NN method for parameter estimation in differential equation (DE) models given sparse measurements of the parameters and states. In the proposed approach, the space- or time-dependent parameters are approximated by Karhunen-Loeve (KL) expansions that are conditioned on the parameters’ measurements, and the states are approximated by deep neural networks (DNNs). The unknown weights in the KL expansions and DNNs are found my minimizing the cost function that enforces the measurements of the states the DE constraint. Regularization is achieved by adding the l2 norm of the conditional KL coefficients into the loss function. Our approach assumes that the parameter fields are correlated in space or time and enforces the statistical knowledge (the mean and the covariance function) in addition to the DE constraints and measurements as opposed to the physics-informed neural network (PINN) and other similar physics-informed machine learning methods where only DE constraints and data are used for parameter estimation. We use the PI-CKL-NN method for parameter estimation in an ordinary differential equation with an unknown time-dependent parameter and the one- and two-dimensional partial differential diffusion equations with unknown space-dependent diffusion coefficients. We also demonstrate that PI-CKL-NN is more accurate than the PINN method, especially when the observations of the parameters are very sparse
A hybrid-network method for obstacle avoidance in the truck-backing system of D. Nguyen and B. Widrow (1989) is presented. A neural network technique for vehicle navigation and control in the presence of obstacles has been developed. A potential function which peaks at the surface of obstacles and has its minimum at the proper vehicle destination is computed using a network structure. The field is guaranteed not to have spurious local minima and does not have the property of flattening-out far from the goal. A feedforward neural network is used to control the steering of the vehicle using local field information. The network is trained in an obstacle-free space to follow the negative gradient of the field, after which the network is able to control and navigate the truck to its target destination in a space of obstacles which may be stationary or movable.
The intrinsic similarity between the firing of a rule and the firing of a neuron has been captured in this research to provide a neural network development system within an existing production system (CLIPS). A very important by-product of this research has been the emergence of an integrated technique of using rule based systems in conjunction with the neural networks to solve complex problems. The systems provides a tool kit for an integrated use of the two techniques and is also extendible to accommodate other AI techniques like the semantic networks, connectionist networks, and even the petri nets. This integrated technique can be very useful in solving complex AI problems.
Reactive molecular dynamics simulations enable detailed understanding of solvent effects on chemical reaction mechanisms and reaction rates. While classical molecular dynamics using reactive force fields allows significantly longer simulation time scales and larger system sizes compared with ab initio molecular dynamics, constructing reactive force fields is a difficult and complex task. In this work, we describe a general approach following the Empirical Valence Bond (EVB) framework for constructing ab initio reactive force fields for condensed phase simulations by combining physics-based methods with neural networks (PB/NN). The physics-based terms ensure correct asymptotic behavior of electrostatic, polarization, and dispersion interactions, and are compatible with existing solvent force fields. Neural networks are utilized for versatile description of short-range orbital interactions within the transition state region, and accurate rendering of vibrational motion of the reacting complex. Herein, we demonstrate our methodology for a simple deprotonation reaction of the 1-ethyl-3-methylimidazolium (EMIM+) cation with acetate to form 1-ethyl- 3-methylimidazol-2-ylidene and acetic acid. Our PB/NN force field exhibits ~ 1 kJ/mol MAE accuracy within the transition state region for the gas-phase complex. To characterize solvent modulation of the reaction profile, we compute potentials of mean force (PMFs) for the gas-phase reaction as well as the reaction within a four ion cluster, and benchmark against ab initio molecular dynamics simulations. We find that the surrounding ionic environment significantly destabilizes formation of the carbene product, and we show that this effect is accurately captured by the reactive force field. By construction, the PB/NN potential may be directly employed for simulations of other solvents/chemical environments without additional parameterization.
Neuro-Spark, which is a new neuromorphic architecture with a field-programmable gate array (FPGA) implementation for ultrafast spiking neural network (SNN) inference at the edge, facilitates smart-pixel in-sensor filtering for high-energy physics experiments at the Large Hadron Collider (LHC). Utilizing the evolutionary optimization for neuromorphic systems (EONS) training method, we generate compact SNN models with 91% signal efficiency, akin to convolutional neural networks but with half the parameters. However, deploying near the detector poses a challenge because the SNN must handle a sustained input data rate exceeding 1013 GB/s. To overcome this, we propose a novel hardware architecture that uses high-level synthesis to construct a tuned architecture for the EONS-trained SNN. In addition to the analysis and validation with an AMD Xilinx Artix-A7 FPGA, our solution consumes only ç24% of FPGA LUT and flipflops. We also introduce an innovative quantization method that reduces FPGA resource utilization by ç15% without compromising accuracy. Our FPGA implementation achieves computing latency of ç10 ns for smart-pixel application inference on an edge FPGA.
We present a potential energy surface fitting scheme based on multiplicative artificial neural networks. It has the sum of products form required for efficient computation of the dynamics of multidimensional quantum systems with the multi configuration time dependent Hartree method. Moreover, it results in analytic potential energy matrix elements when combined with quantum dynamics methods using Gaussian basis functions, eliminating the need for a local harmonic approximation. Scaling behavior with respect to the complexity of the potential as well as the requested accuracy is discussed.