Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Neural Networks (Computer)”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 631 records · Page 35

Resilience and Robustness of Spiking Neural Networks for Neuromorphic Systems

Though robustness and resilience are commonly quoted as features of neuromorphic computing systems, the expected performance of neuromorphic systems in the face of hardware failures is not clear. In this work, we study the effect of failures on the performance of four different training algo-rithms for spiking neural networks on neuromorphic systems: two back-propagation-based training approaches (Whetstone and SLAYER), a liquid state machine or reservoir computing approach, and an evolutionary optimization-based approach (EONS). We show that these four different approaches have very different resilience characteristics with respect to simulated hardware failures. We then analyze an approach for training more resilient spiking neural networks using the evolutionary optimization approach. We show how this approach produces more resilient networks and discuss how it can be extended to other spiking neural network training approaches as well.

Schuman, Catherine↗

Scaling kinetic Monte-Carlo simulations of grain growth with combined convolutional and graph neural networks

Graph neural networks (GNN) have emerged as a promising machine learning method for microstructure simulations such as grain growth. However, accurate modeling of realistic grain boundary networks requires large simulation cells, which GNN has difficulty scaling up to. To alleviate the computational costs and memory footprint of GNN, we suggest a hybrid architecture combining a convolutional neural network (CNN) based bijective autoencoder to compress the spatial dimensions, and a GNN that evolves the microstructure in the latent space of reduced spatial sizes. Our results demonstrate that the new design significantly reduces computational costs with using fewer message passing layer (from 12 down to 3) compared with GNN alone. The reduction in computational cost becomes more pronounced as the spatial size increases, indicating strong computational scalability. For the largest mesh evaluated (160 3 ), our method reduces memory usage and runtime in inference by 117× and 115×, respectively, compared with GNN-only baseline. More importantly, it shows higher accuracy and stronger spatiotemporal capability than the GNN-only baseline, especially in long-term testing. Such combination of scalability and accuracy is essential for simulating realistic material microstructures over extended time scales. The improvements can be attributed to the bijective autoencoder’s ability to compress information losslessly from spatial domain into a high dimensional feature space, thereby producing more expressive latent features for the GNN to learn from, while also contributing its own spatiotemporal modeling capability. Training data are generated from stochastic grain growth simulations, providing realistic variability for learning robust microstructure evolution. Comprehensive system validation confirms that the model is accurate, robust, and scalable.

36 MATERIALS SCIENCE↗

Testing autonomous cyber-physical systems using fuzzing features from convolutional neural networks: work-in-progress

Autonomous cyber-physical systems rely on modern machine learning methods such as deep neural networks to control their interactions with the physical world. Testing of such intelligent cyber-physical systems is a challenge due to the huge state space associated with high-resolution visual sensory inputs. We demonstrate how fuzzing the input using patterns obtained from the convolutional filters of an unrelated convolutional neural network can be used to test computer vision algorithms implemented in intelligent cyber-physical systems. Our method discovers interesting counterexamples to a pedestrian detection algorithm implemented in the popular OpenCV library. Our approach also unearths counterexamples to the correct behavior of an autonomous car similar to NVIDIA's end-to-end self-driving deep neural net running on the Udacity open-source simulator.

Raj, Sunny↗

Rethinking the ill-posedness of the spectral function reconstruction — Why is it fundamentally hard and how Artificial Neural Networks can help

Reconstructing hadron spectral functions through Euclidean correlation functions are of the important missions in lattice QCD calculations. However, in a Källen–Lehmann (KL) spectral representation, the reconstruction is observed to be ill-posed in practice. It is usually ascribed to the fewer observation points compared to the number of points in the spectral function. In this paper, by solving the eigenvalue problem of continuous KL convolution, we show analytically that the ill-posedness of the inversion is fundamental and it exists even for continuous correlation functions. We discussed how to introduce regulators to alleviate the predicament, in which include the Artificial Neural Networks (ANNs) representations recently proposed by the Authors in another study. The uniqueness of solutions using ANNs representations is manifested analytically and validated numerically. Reconstructed spectral functions using different regularization schemes are also demonstrated, together with their eigen-mode decomposition. We observe that components with large eigenvalues can be reliably reconstructed by all methods, whereas those with low eigenvalues need to be constrained by regulators.

97 MATHEMATICS AND COMPUTING↗

Stable parallel training of Wasserstein conditional generative adversarial neural networks

In this work, we propose a stable, parallel approach to train Wasserstein conditional generative adversarial neural networks (W-CGANs) under the constraint of a fixed computational budget. Differently from previous distributed GANs training techniques, our approach avoids inter-process communications, reduces the risk of mode collapse and enhances scalability by using multiple generators, each one of them concurrently trained on a single data label. The use of the Wasserstein metric also reduces the risk of cycling by stabilizing the training of each generator. We illustrate the approach on the CIFAR10, CIFAR100, and ImageNet1k datasets, three standard benchmark image datasets, maintaining the original resolution of the images for each dataset. Performance is assessed in terms of scalability and final accuracy within a limited fixed computational time and computational resources. To measure accuracy, we use the inception score, the Fréchet inception distance, and image quality. An improvement in inception score and Fréchet inception distance is shown in comparison to previous results obtained by performing the parallel approach on deep convolutional conditional generative adversarial neural networks as well as an improvement of image quality of the new images created by the GANs approach. Weak scaling is attained on both datasets using up to 2000 NVIDIA V100 GPUs on the OLCF supercomputer Summit.

97 MATHEMATICS AND COMPUTING↗

Universal and interpretable classification of atomistic structural transitions via unsupervised graph learning

Materials processing often occurs under extreme dynamic conditions leading to a multitude of unique structural environments. These structural environments generally occur at high temperatures and/or high pressures, often under non-equilibrium conditions, which results in drastic changes in the material's structure over time. Computational techniques, such as molecular dynamics simulations, can probe the atomic regime under these extreme conditions. However, characterizing the resulting diverse atomistic structures as a material undergoes extreme changes in its structure has proved challenging due to the inherently non-linear relationship between structures as large-scale changes occur. Here, we introduce SODAS++, a universal graph neural network framework, that can accurately and intuitively quantify the atomistic structural evolution corresponding to the transition between any two arbitrary phases. We showcase SODAS++ for both solid–solid and solid–liquid transitions for systems of increasing geometric and chemical complexity, such as colloidal systems, elemental Al, rutile and amorphous TiO 2 , and the non-stoichiometric ternary alloy Ag 26 Au 5 Cu 19 . Finally, we show that SODAS++ can accurately quantify all transitions in a physically interpretable manner, showcasing the power of unsupervised graph neural network encodings for capturing the complex and non-linear pathway, a material's structure takes as it evolves.

36 MATERIALS SCIENCE↗

A Comprehensive Analysis of PINNs for Power System Transient Stability

The integration of machine learning in power systems, particularly in stability and dynamics, addresses the challenges brought by the integration of renewable energies and distributed energy resources (DERs). Traditional methods for power system transient stability, involving solving differential equations with computational techniques, face limitations due to their time-consuming and computationally demanding nature. This paper introduces physics-informed Neural Networks (PINNs) as a promising solution for these challenges, especially in scenarios with limited data availability and the need for high computational speed. PINNs offer a novel approach for complex power systems by incorporating additional equations and adapting to various system scales, from a single bus to multi-bus networks. Our study presents the first comprehensive evaluation of physics-informed Neural Networks (PINNs) in the context of power system transient stability, addressing various grid complexities. Additionally, we introduce a novel approach for adjusting loss weights to improve the adaptability of PINNs to diverse systems. Our experimental findings reveal that PINNs can be efficiently scaled while maintaining high accuracy. Furthermore, these results suggest that PINNs significantly outperform the traditional ode45 method in terms of efficiency, especially as the system size increases, showcasing a progressive speed advantage over ode45.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Prediction of Stiffness and Fatigue Lives of Polymer Matrix Composite Laminates Using Artificial Neural Networks

Machine learning (ML) models are increasingly being used in many engineering fields due to the advancements in ML algorithms and availability of high-speed computing power. One of the most popular ML class of models is artificial neural networks (ANN). ML is increasingly being used in the design and analysis of composite materials and structures, specifically in the constitutive modeling of composite materials with the focus on greatly accelerating multiscale analyses of composite materials and structures through development of surrogate models. Towards that end, both Python and MATLAB-based neural nets have been developed to predict initial stiffness and fatigue life of an eight-ply symmetric polymer matrix composite laminate. Two types of neural networks, a Multilayer Perceptron (MLP) and a Recurrent Neural Network (RNN), have been developed for both platforms. Results show that the both neural net types can provide an excellent estimate of initial stiffness as well as fatigue life of eight-ply symmetric polymer matrix composite laminate. RNNs are better able to capture the shape of the fatigue curve of a laminate. This tool can be very useful for system level studies to obtain an estimate of desired properties and life of PMC composite laminates. The associated surrogate models could also be used in composite multiscale analyses to replace the actual physics-based calculations at lower scales and thereby significantly increase the computational efficiency of such analyses and thus make multiscale analyses a viable industrial tool for large scale structural problems.

Composite↗

DETECTING FIRE WITH MACHINE LEARNING-ENABLED VISUAL MONITORING FOR NUCLEAR POWER PLANT ENVIRONMENTS

Nuclear power plants are experiencing significant cost challenges to remain competitive with other energy-generation utilities. Unlike other industries, the cost of operation and maintenance activities is mostly attributed to workforce costs. To mitigate this, nuclear power plant stakeholders are increasingly interested in the development and deployment of machine learning methods to potentially automate or augment manually intensive tasks to reduce costs, especially for monitoring activities. One monitoring function that is visually demanding and that can occur frequently to meet the requirements of a fire protection program is visually monitoring an area for fire occurrence. Currently, fire watch activities consist of a worker physically stationed at a given location with the sole responsibility of observing a given area to ensure a fire is detected and mitigated promptly. This effort focused on the development and evaluation of a suitable deep convolutional neural network to classify individual video frames at a sub-second frequency for the occurrence of “fire” and “no fire” in varying industrial environments similar to nuclear power plants. It is believed that a trained neural network model could be integrated with existing facility video surveillance camera feeds to generate alerts when fire inferences occur in individual frames captured at sub-second temporal resolutions. Extensive effort was dedicated to identifying and curating suitable imagery training data representing varying environments and scene settings with and without flame features to maximize generalization in nuclear power plant environments. The data collection effort resulted in the aggregation of a large, labeled image library exceeding 12,000 images to support model training for diverse industrial environments. A deep neural network model incorporating parallel multi-scale capabilities was developed and trained to support accurate image-based detection of flame incidents of varying sizes and spectral feature properties within heterogeneous scenes. Analysis results show that the trained model can achieve high inference accuracy despite heterogeneous scene environments and components. Testing accuracy exceeded 95.0 percent with very low false positive and false negative inferences.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

A mathematical discussion concerning the performance of multilayer perceptron-type artificial neural networks through use of orthogonal bipolar vectors

This work proposes the use of orthogonal bipolar vectors (OBV) as targets for multilayer perceptron (MLP) Artificial neural networks pertaining to the pattern recognition field. Implications involving this proposal are supported by a proper mathematical discussion. This study shows that the use of OBV as targets improves the class identification performance of MLP networks for pattern recognition with more than two classes when compared to the widely used conventional target bipolar vector (CBV). In addition to the overall performance improvement, it is possible to obtain high recognition rates with fewer training epochs, while keeping a feasible generalization with other data. This behavior is related to the Euclidean distance of the points in the output space, which is much higher in the case of OBVs. Therefore, with a greater distance between the points in the output space, the MLP network becomes less prone to associate trained outputs with incorrect targets. The mathematical discussion in this study is based on the backpropagation training algorithm. In addition to the mathematical discussion, results from experiments with real data are presented. These results follow those obtained by mathematical derivations. Real data sets used in the experiments are available from: (a) the Semeion Handwritten Digit of Machine Learning Repository, international repository; (b) Iris Image Database from the Chinese Academy of Sciences—CASIA; (c) Australian Sign Language, signs of Machine Learning Repository, international repository. Experiments were conducted to measure the mean Euclidean distance between the MLP network output subjected to a given input pattern and outputs related to other pattern classifications. Experimental results show that MLP networks trained with OBVs were able to ensure a Euclidean distance between the output and incorrect targets greater than MLP networks trained with CBVs (by up to 4 times for handwritten, by up to 12 times for iris images and by up to 10 times for Auslan signs language). As such, the adoption of OBVs as target vectors for MLPs reduces the computational effort for training the network.

97 MATHEMATICS AND COMPUTING↗

Quantum reservoir computing implementation on coherently coupled quantum oscillators

Quantum reservoir computing is a promising approach for quantum neural networks, capable of solving hard learning tasks on both classical and quantum input data. However, current approaches with qubits suffer from limited connectivity. We propose an implementation for quantum reservoir that obtains a large number of densely connected neurons by using parametrically coupled quantum oscillators instead of physically coupled qubits. We analyze a specific hardware implementation based on superconducting circuits: with just two coupled quantum oscillators, we create a quantum reservoir comprising up to 81 neurons. We obtain state-of-the-art accuracy of 99% on benchmark tasks that otherwise require at least 24 classical oscillators to be solved. Our results give the coupling and dissipation requirements in the system and show how they affect the performance of the quantum reservoir. Beyond quantum reservoir computing, the use of parametrically coupled bosonic modes holds promise for realizing large quantum neural network architectures, with billions of neurons implemented with only 10 coupled quantum oscillators.

97 MATHEMATICS AND COMPUTING↗

Machine Learning Algorithms for Automated NIF Capsule Mandrel Selection

Small shells, approximately 2 mm in diameter, made from Poly(α-methylstyrene) (PAMS) are used as mandrels in the production of glow discharge polymer capsules located at the center of inertial confinement fusion experiments. The visual inspection process of microscope images of these shell mandrels, including detection of micron-sized defects on the shell surface as well as the handling and sorting, is a very labor-intensive, repetitive, and highly subjective process that stands to benefit greatly from automation. As part of an effort to decrease the number of labor hours spent in capsule handling, inspection, and metrology, the development of robotic systems was presented in a paper by Carlson et al., “Automation in Target Fabrication” [Fusion Sci. Technol., Vol. 70, p. 274 (2016)]. The current work expands the automated image acquisition systems developed previously and adds the use of convolutional neural networks to select capsules best suited for use in the downstream production process. Through the use of these machine learning algorithms, the selection process becomes robust, repeatable, and operator independent. Finally, as an added benefit the system developed as part of this work is able to provide defect statistics on entire shell batches and feed this information upstream to the production team.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Adjoint-Operator Learning For A Neural Network

Electronic neural networks made to synthesize initially unknown mathematical models of time-dependent phenomena or to learn temporally evolving patterns by use of algorithms based on adjoint operators. Algorithms less complicated, involve less computation and solve learning equations forward in time possibly simultaneously with equations of evolution of neural network, thereby both increasing computational efficiency and making real-time applications possible.

Barhen, Jacob↗

Reinforcement Learning via Gaussian Processes with Neural Network Dual Kernels

While deep neural networks (DNNs) and Gaussian Processes (GPs) are both popularly utilized to solve problems in reinforcement learning, both approaches feature undesirable drawbacks for challenging problems. DNNs learn complex non-linear embeddings, but do not naturally quantify uncertainty and are often data-inefficient to train. GPs infer posterior distributions over functions, but popular kernels exhibit limited expressivity on complex and high-dimensional data. Fortunately, recently discovered conjugate and neural tangent kernel functions encode the behavior of overparameterized neural networks in the kernel domain. We demonstrate that these kernels can be efficiently applied to regression and reinforcement learning problems by analyzing a baseline case study.We apply GPs with neural network dual kernels to solve reinforcement learning tasks for the first time. We demonstrate, using the well understood mountain-car problem, that GPs empowered with dual kernels perform at least as well as those using the conventional radial basis function kernel. Finally, we conjecture that by inheriting the probabilistic rigor of GPs and the powerful embedding properties of DNNs, GPs using NN dual kernels will empower future reinforcement learning models on difficult domains.

97 MATHEMATICS AND COMPUTING↗

Architecture-Preserving Provable Repair of Deep Neural Networks

Deep neural networks (DNNs) are becoming increasingly important components of software, and are considered the state-of-the-art solution for a number of problems, such as image recognition. However, DNNs are far from infallible, and incorrect behavior of DNNs can have disastrous real-world consequences. This paper addresses the problem of architecture-preserving V-polytope provable repair of DNNs. A V-polytope defines a convex bounded polytope using its vertex representation. V-polytope provable repair guarantees that the repaired DNN satisfies the given specification on the infinite set of points in the given V-polytope. An architecture-preserving repair only modifies the parameters of the DNN, without modifying its architecture. The repair has the flexibility to modify multiple layers of the DNN, and runs in polynomial time. It supports DNNs with activation functions that have some linear pieces, as well as fully-connected, convolutional, pooling and residual layers. To the best our knowledge, this is the first provable repair approach that has all of these features. We implement our approach in a tool called APRNN. Using MNIST, ImageNet, and ACAS Xu DNNs, we show that it has better efficiency, scalability, and generalization compared to PRDNN and REASSURE, prior provable repair methods that are not architecture preserving.

97 MATHEMATICS AND COMPUTING↗

Quantum-inspired weight-constrained neural network: Reducing variable numbers by 100× compared to standard neural networks

Although quantum machine learning has shown great promise, the practical application of quantum computers remains constrained in the noisy intermediate-scale quantum era. To take advantage of quantum machine learning, we investigate the underlying mathematical principles of these quantum models and find that the quantum neural network with amplitude encoding is equivalent to a weight-constrained neural network. Motivated by this discovery, we develop a classical weight-constrained neural network. We find that this approach can reduce the number of variables in a classical neural network by a factor of 135 while preserving its accuracy. In addition, we develop a dropout method to enhance the robustness of quantum machine learning models, which are highly susceptible to adversarial attacks. This technique can also be applied to improve the adversarial robustness of the classical weight-constrained neural network, which is essential for industry applications, such as self-driving vehicles. Our work offers an approach to reduce the complexity of large classical neural networks, addressing a critical challenge in machine learning.

quantum algorithms & computation↗

Machine Learning Models of Plastic Flow Based on Representation Theory

We use machine learning (ML) to infer stress and plastic flow rules using data from representative polycrystalline simulations. In particular, we use so-called deep (multilayer) neural networks (NN) to represent the two response functions. The ML process does not choose appropriate inputs or outputs, rather it is trained on selected inputs and output. Likewise, its discrimination of features is crucially connected to the chosen input-output map. Furthermore, we draw upon classical constitutive modeling to select inputs and enforce well-accepted symmetries and other properties. With these developments, we enable rapid model building in real-time with experiments, and guide data collection and feature discovery.

42 ENGINEERING↗

A differentiable approach to the maximum independent set problem using dataless neural networks

The success of machine learning solutions for reasoning about discrete structures has brought attention to its adoption within combinatorial optimization algorithms. Such approaches generally rely on supervised learning by leveraging datasets of the combinatorial structures of interest drawn from some distribution of problem instances. Reinforcement learning has also been employed to find such structures. Here, in this paper, we propose a different approach in that no data is required for training the neural networks that produce the solution. In this sense, what we present is not a machine learning solution, but rather one that is dependent on neural networks and where backpropagation is applied to a loss function defined by the structure of the neural network architecture as opposed to a training dataset. In particular, we reduce the popular combinatorial optimization problem of finding a maximum independent set to a neural network and employ a dataless training scheme to refine the parameters of the network such that those parameters yield the structure of interest. Additionally, we propose a universal graph reduction procedure to handle large-scale graphs. The reduction exploits community detection for graph partitioning and is applicable to any graph type and/or density. Experimental results on both real and synthetic graphs demonstrate that our proposed method performs on par or outperforms state-of-the-art learning-based methods in terms of the size of the found set without requiring any training data.

97 MATHEMATICS AND COMPUTING↗