Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “NEURAL NETWORKS”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Towards a robust out-of-the-box neural network model for genomic data

The accurate prediction of biological features from genomic data is paramount for precision medicine and sustainable agriculture. For decades, neural network models have been widely popular in fields like computer vision, astrophysics and targeted marketing given their prediction accuracy and their robust performance under big data settings. Yet neural network models have not made a successful transition into the medical and biological world due to the ubiquitous characteristics of biological data such as modest sample sizes, sparsity, and extreme heterogeneity. Here, we investigate the robustness, generalization potential and prediction accuracy of widely used convolutional neural network and natural language processing models with a variety of heterogeneous genomic datasets. Mainly, recurrent neural network models outperform convolutional neural network models in terms of prediction accuracy, overfitting and transferability across the datasets under study. While the perspective of a robust out-of-the-box neural network model is out of reach, we identify certain model characteristics that translate well across datasets and could serve as a baseline model for translational researchers.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2023

Analytic Theory for the Dynamics of Wide Quantum Neural Networks

Here, parametrized quantum circuits can be used as quantum neural networks and have the potential to outperform their classical counterparts when trained for addressing learning problems. To date, much of the results on their performance on practical problems are heuristic in nature. In particular, the convergence rate for the training of quantum neural networks is not fully understood. Here, we analyze the dynamics of gradient descent for the training error of a class of variational quantum machine learning models. We define wide quantum neural networks as parametrized quantum circuits in the limit of a large number of qubits and variational parameters. Then, we find a simple analytic formula that captures the average behavior of their loss function and discuss the consequences of our findings. For example, for random quantum circuits, we predict and characterize an exponential decay of the residual training error as a function of the parameters of the system. Finally, we validate our analytic results with numerical experiments.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2022

Towards Compact Neural Networks via End-to-End Training: A Bayesian Tensor Approach with Automatic Rank Determination

Post-training model compression can reduce the inference costs of deep neural networks, but uncompressed training still consumes enormous hardware resources and energy. To enable low-energy training on edge devices, it is highly desirable to directly train a compact neural network from scratch with a low memory cost. Low-rank tensor decomposition is an effective approach to reduce the memory and computing costs of large neural networks. However, directly training low-rank tensorized neural networks is a very challenging task because it is hard to determine a proper tensor rank a priori, and the tensor rank controls both model complexity and accuracy. Here, this paper presents a novel end-to-end framework for low-rank tensorized training. We first develop a Bayesian model that supports various low-rank tensor formats (e.g., CANDECOMP/PARAFAC, Tucker, tensor-train, and tensor-train matrix) and reduces neural network parameters with automatic rank determination during training. Then we develop a customized Bayesian solver to train large-scale tensorized neural networks. Our training methods shows orders-of-magnitude parameter reduction and little accuracy loss (or even better accuracy) in the experiments. On a very large deep learning recommendation system with over 4.2 ×10 9 model parameters, our method can reduce the parameter number to 1.6 ×10 5 automatically in the training process (i.e., by 2.6 ×10 4 times) while achieving almost the same accuracy. Code is available at https://github.com/colehawkins/bayesian-tensor-rank-determination.

compact neural networks↗
DOE OSTI2022

GAAF: Searching Activation Functions for Binary Neural Networks Through Genetic Algorithm

Binary neural networks (BNNs) show promising utilization in cost and power-restricted domains such as edge devices and mobile systems. This is due to its significantly less computation and storage demand, but at the cost of degraded performance. To close the accuracy gap, in this paper we propose to add a complementary activation function (AF) ahead of the sign based binarization, and rely on the genetic algorithm (GA) to automatically search for the ideal AFs. These AFs can help extract extra information from the input data in the forward pass, while allowing improved gradient approximation in the backward pass. Fifteen novel AFs are identified through our GA-based search, while most of them show improved performance (up to 2.54% on ImageNet) when testing on different datasets and network models. Interestingly, periodic functions are identified as a key component for most of the discovered AFs, which rarely exist in human designed AFs. Our method offers a novel approach for designing general and application-specific BNN architecture.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2020

Atomic resolution convergent beam electron diffraction analysis using convolutional neural networks

Two types of convolutional neural network (CNN) models, a discrete classification network and a continuous regression network, were trained to determine local sample thickness from convergent beam diffraction (CBED) patterns of SrTiO 3 collected in a scanning transmission electron microscope (STEM) at atomic column resolution. Acquisition of atomic resolution CBED patterns for this purpose requires careful balancing of CBED feature size in pixels, acquisition speed, and detector dynamic range. The training datasets were derived from multislice simulations, which must be convolved with incoherent source broadening. Sample thicknesses were also determined using quantitative high-angle annular dark-field (HAADF) STEM images acquired simultaneously. The regression CNN performed well on sample thinner than 35 nm, with 70% of the CNN results within 1 nm of HAADF thickness, and 1.0 nm overall root mean square error between the two measurements. The classification CNN was trained for a thicknesses up to 100 nm and yielded 66% of CNN results within one classification increment of 2 nm of HAADF thickness. Our approach depends on methods from computer vision including transfer learning and image augmentation.

36 MATERIALS SCIENCE↗
DOE OSTI2021

Train Like a (Var)Pro: Efficient Training of Neural Networks with Variable Projection

Deep neural networks (DNNs) have achieved state-of-the-art performance across a variety of traditional machine learning tasks, e.g., speech recognition, image classification, and segmentation. The ability of DNNs to efficiently approximate high-dimensional functions has also motivated their use in scientific applications, e.g., to solve partial differential equations and to generate surrogate models. In this paper, we consider the supervised training of DNNs, which arises in many of the above applications. We focus on the central problem of optimizing the weights of the given DNN such that it accurately approximates the relation between observed input and target data. Devising effective solvers for this optimization problem is notoriously challenging due to the large number of weights, nonconvexity, data sparsity, and nontrivial choice of hyperparameters. To solve the optimization problem more efficiently, we propose the use of variable projection (VarPro), a method originally designed for separable nonlinear least-squares problems. Our main contribution is the Gauss--Newton VarPro method (GNvpro) that extends the reach of the VarPro idea to nonquadratic objective functions, most notably cross-entropy loss functions arising in classification. These extensions make GNvpro applicable to all training problems that involve a DNN whose last layer is an affine mapping, which is common in many state-of-the-art architectures. In our four numerical experiments from surrogate modeling, segmentation, and classification, GNvpro solves the optimization problem more efficiently than commonly used stochastic gradient descent (SGD) schemes. Finally, GNvpro finds solutions that generalize well, and in all but one example better than well-tuned SGD methods, to unseen data points.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

The predictive skill of convolutional neural networks models for disease forecasting

In this paper we investigate the utility of one-dimensional convolutional neural network (CNN) models in epidemiological forecasting. Deep learning models, in particular variants of recurrent neural networks (RNNs) have been studied for ILI (Influenza-Like Illness) forecasting, and have achieved a higher forecasting skill compared to conventional models such as ARIMA. In this study, we adapt two neural networks that employ one-dimensional temporal convolutional layers as a primary building block—temporal convolutional networks and simple neural attentive meta-learners—for epidemiological forecasting. We then test them with influenza data from the US collected over 2010-2019. We find that epidemiological forecasting with CNNs is feasible, and their forecasting skill is comparable to, and at times, superior to, plain RNNs. Thus CNNs and RNNs bring the power of nonlinear transformations to purely data-driven epidemiological models, a capability that heretofore has been limited to more elaborate mechanistic/compartmental disease models.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

Overview of RFID Applications Utilizing Neural Networks

As Radio Frequency Identification (RFID) methods continue to evolve to higher levels of complexity, one form of machine learning is making its appearance. The use of Neural Networks (NN) in the RFID field is steadily increasing, and in the fields of localization and activity recognition, promising results are being shown from a variety of research. RFID applications fall primarily under two types of problems including regression and classification. We analyze RIFD localization techniques which fall under regression, and activity recognition which falls under classification. Many works don’t classify themselves as activity recognition methods, but because they fall under the classification category, we still consider them as activity recognition techniques. This research overviews the Neural Network models in the localization field based on whether they can perform independently of the environment in which they were tested. For activity recognition and accessory fields, the major methods involve tag-based and tag-free approaches. In conclusion, after the models are surveyed, a comparison study is given to examine what may be the cause for increased accuracy between different Neural Network models.

42 ENGINEERING↗
DOE OSTI2020

Deep convolutional neural networks for multi-scale time-series classification and application to tokamak disruption prediction using raw, high temporal resolution diagnostic data

In this paper we discuss recent advances in deep convolutional neural networks (CNN) for sequence learning, which allow identifying long-range, multi-scale phenomena in long sequences, such as those found in fusion plasmas. We point out several benefits of these deep CNN architectures, such as not requiring experts such as physicists to hand-craft input data features, the ability to capture longer range dependencies compared to the more common sequence neural networks (recurrent neural networks like long short-term memory (LSTM) networks), and the comparative computational efficiency. We apply this neural network architecture to the popular problem of disruption prediction in fusion energy tokamaks, utilizing raw data from a single diagnostic, the Electron Cyclotron Emission imaging (ECEi) diagnostic from the DIII-D tokamak. Initial results trained on a large ECEi dataset show promise, achieving an F 1 -score of ~91% on individual time-slices using only the ECEi data. This indicates the ECEi diagnostic by itself can be sensitive to a number of pre-disruption markers useful for predicting disruptions on timescales not only for mitigation but also avoidance. Future opportunities for utilizing these deep CNN architectures with fusion data are outlined, including impact of recent upgrades to the ECEi diagnostic.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2025

Adaptive Interface-PINNs (AdaI-PINNs): An Efficient Physics-Informed Neural Networks Framework for Interface Problems

Here, we present an efficient physics-informed neural networks (PINNs) framework, termed Adaptive Interface-PINNs (AdaI-PINNs), to improve the modeling of interface problems with discontinuous coefficients and/or interfacial jumps. This framework is an enhanced version of its predecessor, Interface PINNs or I-PINNs (Sarma et al.; https://doi.org/10.1016/j.cma.2024.117135), which involves domain decomposition and assignment of different predefined activation functions to the neural networks in each subdomain across a sharp interface, while keeping all other parameters of the neural networks identical. In AdaI-PINNs, the activation functions vary solely in their slopes, which are trained along with the other parameters of the neural networks. This makes the AdaI-PINNs framework fully automated without requiring preset activation functions. Comparative studies on one-dimensional, two-dimensional, and three-dimensional benchmark elliptic interface problems reveal that AdaI-PINNs outperform I-PINNs, reducing computational costs by 2-6 times while producing similar or better accuracy.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Development of systematic uncertainty-aware neural network trainings for binned-likelihood analyses at the LHC

We propose a neural network training method capable of accounting for the effects of systematic variations of the data model in the training process and describe its extension towards neural network multiclass classification. The procedure is evaluated on the realistic case of the measurement of Higgs boson production via gluon fusion and vector boson fusion in the τ τ decay channel at the CMS experiment. The neural network output functions are used to infer the signal strengths for inclusive production of Higgs bosons as well as for their production via gluon fusion and vector boson fusion. We observe improvements of 12 and 16% in the uncertainty in the signal strengths for gluon and vector-boson fusion, respectively, compared with a conventional neural network training based on cross-entropy.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2023

Optimized asynchronous training of neural networks using a distributed parameter server with eager updates

A method of training a neural network includes, at a local computing node, receiving remote parameters from a set of one or more remote computing nodes, initiating execution of a forward pass in a local neural network in the local computing node to determine a final output based on the remote parameters, initiating execution of a backward pass in the local neural network to determine updated parameters for the local neural network, and prior to completion of the backward pass, transmitting a subset of the updated parameters to the set of remote computing nodes.

Hamidouche, Khaled↗
DOE OSTI2021

Keras2c: A library for converting Keras neural networks to real-time compatible C

With the growth of machine learning models and neural networks in measurement and control systems comes the need to deploy these models in a way that is compatible with existing systems. Existing options for deploying neural networks either introduce very high latency, require expensive and time consuming work to integrate into existing code bases, or only support a very limited subset of model types. We have therefore developed a new method called Keras2c, which is a simple library for converting Keras/TensorFlow neural network models into real-time compatible C code. It supports a wide range of Keras layers and model types including multidimensional convolutions, recurrent layers, multi-input/output models, and shared layers. Keras2c re-implements the core components of Keras/TensorFlow required for predictive forward passes through neural networks in pure C, relying only on standard library functions considered safe for real-time use. The core functionality consists of ~ 1500 lines of code, making it lightweight and easy to integrate into existing codebases. Keras2c has been successfully tested in experiments and is currently in use on the plasma control system at the DIII-D National Fusion Facility at General Atomics in San Diego.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2023

Multiverse: Bayesian model selection for neural networks

Multiverse is a code repository for a set of tools for Bayesian model selection for neural networks. The goal of the tools is to provide capabilities for selecting among prior and model specifications for Bayesian neural networks. Multiverse will include tools for creating Bayesian neural networks, evaluating the Bayesian model evidence, and performing inference in Bayesian neural networks. These components are written in Python, a high-level programming language that takes advantage of the Python ecosystem of high-quality open-source packages for machine learning and signal processing.

Klein, Natalie↗
DOE OSTI2022

Modeling design and control problems involving neural network surrogates

Here, we consider nonlinear optimization problems that involve surrogate models represented by neural networks. We demonstrate first how to directly embed neural network evaluation into optimization models, highlight a difficulty with this approach that can prevent convergence, and then characterize stationarity of such models. We then present two alternative formulations of these problems in the specific case of feedforward neural networks with ReLU activation: as a mixed-integer optimization problem and as a mathematical program with complementarity constraints. For the latter formulation we prove that stationarity at a point for this problem corresponds to stationarity of the embedded formulation. Each of these formulations may be solved with state-of-the-art optimization methods, and we show how to obtain good initial feasible solutions for these methods. We compare our formulations on three practical applications arising in the design and control of combustion engines, in the generation of adversarial attacks on classifier networks, and in the determination of optimal flows in an oil well network.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

A Classification of G -invariant Shallow Neural Networks

When trying to fit a deep neural network (DNN) to a G-invariant target function with G a group, it only makes sense to constrain the DNN to be G-invariant as well. However, there can be many different ways to do this, thus raising the problem of “G-invariant neural architecture design”: What is the optimal Ginvariant architecture for a given problem? Before we can consider the optimization problem itself, we must understand the search space, the architectures in it, and how they relate to one another. In this paper, we take a first step towards this goal; we prove a theorem that gives a classification of all G-invariant single-hidden-layer or “shallow” neural network (G-SNN) architectures with ReLU activation for any finite orthogonal group G, and we prove a second theorem that characterizes the inclusion maps or “network morphisms” between the architectures that can be leveraged during neural architecture search (NAS). The proof is based on a correspondence of every G-SNN to a signed permutation representation of G acting on the hidden neurons; the classification is equivalently given in terms of the first cohomology classes of G, thus admitting a topological interpretation. The G-SNN architectures corresponding to nontrivial cohomology classes have, to our knowledge, never been explicitly identified in the literature previously. Using a code implementation, we enumerate the G-SNN architectures for some example groups G and visualize their structure. Lastly, we prove that architectures corresponding to inequivalent cohomology classes coincide in function space only when their weight matrices are zero, and we discuss the implications of this for NAS.

Agrawal, Devanshu↗
DOE OSTI2023

Graph Neural Networks and Applied Linear Algebra v.1.0

SAND2024-01365O The Graph Neural Networks and Applied Linear Algebra is companion software for the educational article with the same title. The software provides illustrative examples of graph neural networks in Matlab and Python. These stand-alone algorithms are for educational purposes. The software also includes graph neural network-based algorithms for a trainable Jacobi iteration as well as diffusion coefficient estimation. The software provides human-interpretable implementations of Graph Neural Networks in Matlab and Python. These implementations are not optimized for performance and instead emphasize readability. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

SciDAC↗
DOE OSTI2024

Polyconvex neural network models of thermoelasticity

Machine-learning function representations such as neural networks have proven to be excellent constructs for constitutive modeling due to their flexibility to represent highly nonlinear data and their ability to incorporate constitutive constraints, which also allows them to generalize well to unseen data. Here, in this work, we extend a polyconvex hyperelastic neural network framework to (isotropic) thermo-hyperelasticity by specifying the thermodynamic and material theoretic requirements for an expansion of the Helmholtz free energy expressed in terms of deformation invariants and temperature. Different formulations which a priori ensure polyconvexity with respect to deformation and concavity with respect to temperature are proposed and discussed. The physics-augmented neural networks are furthermore calibrated with a recently proposed sparsification algorithm that not only aims to fit the training data but also penalizes the number of active parameters, which prevents overfitting in the low data regime and promotes generalization. The performance of the proposed framework is demonstrated on synthetic data, which illustrate the expected thermomechanical phenomena, and existing temperature-dependent uniaxial tension and tension-torsion experimental datasets.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗