Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “binary neural networks”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
NASA NTRS1988

Orthogonal patterns in binary neural networks

A binary neural network that stores only mutually orthogonal patterns is shown to converge, when probed by any pattern, to a pattern in the memory space, i.e., the space spanned by the stored patterns. The latter are shown to be the only members of the memory space under a certain coding condition, which allows maximum storage of M=(2N) sup 0.5 patterns, where N is the number of neurons. The stored patterns are shown to have basins of attraction of radius N/(2M), within which errors are corrected with probability 1 in a single update cycle. When the probe falls outside these regions, the error correction capability can still be increased to 1 by repeatedly running the network with the same probe.

Baram, Yoram↗
DOE OSTI2022

GAAF: Searching Activation Functions for Binary Neural Networks Through Genetic Algorithm

Binary neural networks (BNNs) show promising utilization in cost and power-restricted domains such as edge devices and mobile systems. This is due to its significantly less computation and storage demand, but at the cost of degraded performance. To close the accuracy gap, in this paper we propose to add a complementary activation function (AF) ahead of the sign based binarization, and rely on the genetic algorithm (GA) to automatically search for the ideal AFs. These AFs can help extract extra information from the input data in the forward pass, while allowing improved gradient approximation in the backward pass. Fifteen novel AFs are identified through our GA-based search, while most of them show improved performance (up to 2.54% on ImageNet) when testing on different datasets and network models. Interestingly, periodic functions are identified as a key component for most of the discovered AFs, which rarely exist in human designed AFs. Our method offers a novel approach for designing general and application-specific BNN architecture.

59 BASIC BIOLOGICAL SCIENCES↗
NASA NTRS1991

Orthogonal Patterns In A Binary Neural Network

Report presents some recent developments in theory of binary neural networks. Subject matter relevant to associate (content-addressable) memories and to recognition of patterns - both of considerable importance in advancement of robotics and artificial intelligence. When probed by any pattern, network converges to one of stored patterns.

Baram, Yoram↗
DOE OSTI2022

Implementation of a Binary Neural Network on a Passive Array of Magnetic Tunnel Junctions

The increasing scale of neural networks and their growing application space have produced demand for more energy- and memory-efficient artificial-intelligence-specific hardware. Avenues to mitigate the main issue, the von Neumann bottleneck, include in-memory and near-memory architectures, as well as algorithmic approaches. In this report we leverage the low-power and the inherently binary operation of magnetic tunnel junctions (MTJs) to demonstrate neural network hardware inference based on passive arrays of MTJs. In general, transferring a trained network model to hardware for inference is confronted by degradation in performance due to device-to-device variations, write errors, parasitic resistance, and nonidealities in the substrate. To quantify the effect of these hardware realities, we benchmark 300 unique weight matrix solutions of a two-layer perceptron to classify the Wine dataset for both classification accuracy and write fidelity. Despite device imperfections, we achieve software-equivalent accuracy of up to 95.3% with proper tuning of network parameters in 15 x 15 MTJ arrays having a range of device sizes. The success of this tuning process shows that new metrics are needed to characterize the performance and quality of networks reproduced in mixed signal hardware.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
NASA NTRS1990

Ground states of partially connected binary neural networks

Neural networks defined by outer products of vectors over (-1, 0, 1) are considered. Patterns over (-1, 0, 1) define by their outer products partially connected neural networks consisting of internally strongly connected, externally weakly connected subnetworks. Subpatterns over (-1, 1) define subnetworks, and their combinations that agree in the common bits define permissible words. It is shown that the permissible words are locally stable states of the network, provided that each of the subnetworks stores mutually orthogonal subwords, or, at most, two subwords. It is also shown that when each of the subnetworks stores two mutually orthogonal binary subwords at most, the permissible words, defined as the combinations of the subwords (one corresponding to each subnetwork), that agree in their common bits are the unique ground states of the associated energy function.

Baram, Yoram↗
DOE OSTI2023

BitGNN: Unlocking the Performance Potential of Binary Graph Neural Networks on GPUs

Graph Neural Networks (GNNs) have shown compelling results in many graph-based learning tasks. They are, however, time-consuming. Recent work has shown a promising direction in improving GNN speed and shrinking the size — network binarization, which binarizes network values and operations. Prior work, however, mainly focused on algorithm designs, leaving it open on how to fully materialize the performance potential. This work fills the gap by proposing techniques to best map binary GNNs and their computations to fit the nature of bit manipulations, optimizations and algorithms to maximize BSpMM kernel efficiency, and solutions to other factors influencing the end-to-end time on GPUs. Results on real-world graphs show that the proposed techniques outperform state of-the-art binary GNN implementations by 21-67× with little accuracy loss.

Chen, Jou-An↗
NASA NTRS1993

Classification of multispectral image data by the Binary Diamond neural network and by nonparametric, pixel-by-pixel methods

The classification of multispectral image data obtained from satellites has become an important tool for generating ground cover maps. This study deals with the application of nonparametric pixel-by-pixel classification methods in the classification of pixels, based on their multispectral data. A new neural network, the Binary Diamond, is introduced, and its performance is compared with a nearest neighbor algorithm and a back-propagation network. The Binary Diamond is a multilayer, feed-forward neural network, which learns from examples in unsupervised, 'one-shot' mode. It recruits its neurons according to the actual training set, as it learns. The comparisons of the algorithms were done by using a realistic data base, consisting of approximately 90,000 Landsat 4 Thematic Mapper pixels. The Binary Diamond and the nearest neighbor performances were close, with some advantages to the Binary Diamond. The performance of the back-propagation network lagged behind. An efficient nearest neighbor algorithm, the binned nearest neighbor, is described. Ways for improving the performances, such as merging categories, and analyzing nonboundary pixels, are addressed and evaluated.

Salu, Yehuda↗
NASA NTRS1988

Programmable synaptic chip for electronic neural networks

A binary synaptic matrix chip has been developed for electronic neural networks. The matrix chip contains a programmable 32X32 array of 'long channel' NMOSFET binary connection elements implemented in a 3-micron bulk CMOS process. Since the neurons are kept off-chip, the synaptic chip serves as a 'cascadable' building block for a multi-chip synaptic network as large as 512X512 in size. As an alternative to the programmable NMOSFET (long channel) connection elements, tailored thin film resistors are deposited, in series with FET switches, on some CMOS test chips, to obtain the weak synaptic connections. Although deposition and patterning of the resistors require additional processing steps, they promise substantial savings in silicon area. The performance of synaptic chip in a 32-neuron breadboard system in an associative memory test application is discussed.

Moopenn, A.↗
NASA NTRSDate not supplied

Early Information Parameter-Set Analysis for Satellite Close Approaches using Machine Learning

Understanding orbital mechanics is essential in space flight and navigation applications, and leveraging modern force models for flight path projection remains an important aspect in space mission design and operation. However, force models do not capture all the dynamics or perturbations in the space environment and thus are subject to errors in predicting the state vectors. The further out the predicted miss distance between spacecraft is from the time of closest approach (TCA), the larger the propagated errors in the predicted miss distance at TCA is. The dependency on these force models for spacecraft flight state prediction calls for a more reliable method that can quantify, or even reduce, these propagated errors. With recent advances in the field artificial intelligence, specifically in machine and deep learning algorithms, a model that implements these approaches can improve on the modern force model approach. The goal for this work is to provide an early-information decision-making threshold, in order to prioritize risk assessment implementation, given the ongoing increase of space objects. In analyzing the relationship of several parameters from conjunction data messages(CDMs) and solar information, early information becomes viable in miss distance prediction with unsupervised learning techniques, which learn the parameters that are linked together with miss distance and probability of collision (Pc) variables. Another approach implemented for identifying relationships within CDMs is supervised learning, in which a shallow neural network binary classifier learns to distinguish events with Pc values¡108. These parameters detected in the unsupervised process are then applied to a regression neural network, which predicts the miss distance at TCA for a specific event within a given uncertainty bound. For the regression neural network, a Long Short Term Memory (LSTM) neural network is implemented, which yields memory about each time step in an event. Using an LSTM network, the model learns to predict miss distance within 0.2km of the value measured at TCA. Although there is a limited amount of "close miss" data to train a network, the network learns to associate parameters, like large energy dissipation rates with the secondary object, with an elevated Pc

Brianna I. Robertson↗
NASA NTRS1988

Optical production systems using neural networks and symbolic substitution

Two optical implementations of production systems are advanced. The production systems operate on a knowledge base where facts and rules are encoded as formulas in propositional calculus. The first implementation is a binary neural network. An analog neural network is used to include reasoning with uncertainties. The second implementation uses a new optical symbolic substitution correlator. This implementation is useful when a set of similar situations has to be handled in parallel on one processor.

Botha, Elizabeth↗
DOE OSTI2023

Compact Ferroelectric Programmable Majority Gate for Compute-in-Memory Applications

In this study, a compact and novel ferroelectric (FE) programmable majority gate is proposed and its novel application in Binary Neural Network (BNNs) is investigated. We demonstrate: i) by integrating N metal-ferroelectric-metal (MFM) capacitors on the gate of a transistor (1T-N-MFM structure), a nonvolatile and programmable majority (MAJ) gate that performs MAJ of AND between the gate input and polarization is realized; ii) validation the functionality of our 3-input MAJ of AND gate through comprehensive theoretical and experimental investigations; iii) a compact implementation of 3-input MAJ of XNOR gate that leverages only five of our 3-input MAJ of AND gates connected in parallel; iv) application of MAJ of XNOR gates to replace the XNOR gates and the first layer of the adder tree in the BNNs for up to 21x area saving on top of eliminating the energy-hungry memory accesses due to the compute-in-memory nature.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Accelerating cavity fault prediction using deep learning at Jefferson Laboratory

Abstract Accelerating cavities are an integral part of the continuous electron beam accelerator facility (CEBAF) at Jefferson Laboratory. When any of the over 400 cavities in CEBAF experiences a fault, it disrupts beam delivery to experimental user halls. In this study, we propose the use of a deep learning model to predict slowly developing cavity faults. By utilizing pre-fault signals, we train a long short-term memory-convolutional neural network binary classifier to distinguish between radio-frequency (RF) signals during normal operation and RF signals indicative of impending faults. We optimize the model by adjusting the fault confidence threshold and implementing a multiple consecutive window criterion to identify fault events, ensuring a low false positive rate. Results obtained from analysis of a real dataset collected from the accelerating cavities simulating a deployed scenario demonstrate the model’s ability to identify normal signals with 99.99% accuracy and correctly predict 80% of slowly developing faults. Notably, these achievements were achieved in the context of a highly imbalanced dataset, and fault predictions were made several hundred milliseconds before the onset of the fault. Anticipating faults enables preemptive measures to improve operational efficiency by preventing or mitigating their occurrence.

43 PARTICLE ACCELERATORS↗
DOE OSTI2026

Quantum-Inspired Bayesian Sampling for Uncertainty Quantification and Machine Learning (Final Technical Report)

With increasing simulation and measurement data, machine learning and artificial intelligence have been widely used in computational decision-making of complex engineering systems. The resulting tools, such as uncertainty quantification solvers, reinforcement learning, and physics-informed machine learning, have achieved great success in critical DOE tasks such as material discovery and design, energy system modeling and control, and numerical weather and climate prediction. A core topic in scientific machine learning and artificial intelligence is Bayesian inference: given an observed data set, people want to estimate the posterior distribution of a (possibly large) number of hidden parameters. Due to the flexibility and weak assumptions, Bayesian sampling has been the mainstream Bayesian inference solvers despite the rapid progress of approximate Bayesian inference. Classical Bayesian sampling methods such as Markov-chain Monte Carlo suffer from a low-acceptance rate due to the random walk nature, therefore state-of-the-art techniques use Hamiltonian Monte Carlo and its variants to efficiently draw posterior samples in a high dimension. The key idea of Hamiltonian Monte Carlo and its variants is to simulate the Hamiltonian dynamics of a classical particle with a fixed mass, and their performance significantly degrades when the posterior distribution is highly spiky or has multiple modes. Leveraging the idea of quantum physics, this project has investigated new theory, algorithms and applications of Bayesian inference (especially Bayesian sampling). The main results include: (1) novel quantum-inspired Bayesian sampling methods that can lead to better accuracy for challenging multi-modal or spiky distributions, (2) more scalable machine learning framework leveraging tensor-compressed Bayesian inference, and (3) Bayesian and sampling approaches for verifying the robustness of continuous and binary neural networks.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1993

Data fusion with artificial neural networks (ANN) for classification of earth surface from microwave satellite measurements

A data fusion system with artificial neural networks (ANN) is used for fast and accurate classification of five earth surface conditions and surface changes, based on seven SSMI multichannel microwave satellite measurements. The measurements include brightness temperatures at 19, 22, 37, and 85 GHz at both H and V polarizations (only V at 22 GHz). The seven channel measurements are processed through a convolution computation such that all measurements are located at same grid. Five surface classes including non-scattering surface, precipitation over land, over ocean, snow, and desert are identified from ground-truth observations. The system processes sensory data in three consecutive phases: (1) pre-processing to extract feature vectors and enhance separability among detected classes; (2) preliminary classification of Earth surface patterns using two separate and parallely acting classifiers: back-propagation neural network and binary decision tree classifiers; and (3) data fusion of results from preliminary classifiers to obtain the optimal performance in overall classification. Both the binary decision tree classifier and the fusion processing centers are implemented by neural network architectures. The fusion system configuration is a hierarchical neural network architecture, in which each functional neural net will handle different processing phases in a pipelined fashion. There is a total of around 13,500 samples for this analysis, of which 4 percent are used as the training set and 96 percent as the testing set. After training, this classification system is able to bring up the detection accuracy to 94 percent compared with 88 percent for back-propagation artificial neural networks and 80 percent for binary decision tree classifiers. The neural network data fusion classification is currently under progress to be integrated in an image processing system at NOAA and to be implemented in a prototype of a massively parallel and dynamically reconfigurable Modular Neural Ring (MNR).

Lure, Y. M. Fleming↗
DOE OSTI2022

Prediction of Self-Diffusion in Binary Fluid Mixtures Using Artificial Neural Networks

Artificial neural networks (ANNs) were developed to accurately predict the self-diffusion constants for individual components in binary fluid mixtures. The ANNs were tested on an experimental database of 4328 self-diffusion constants from 131 mixtures containing 75 unique compounds. The presence of strong hydrogen bonding molecules may lead to clustering or dimerization resulting in non-linear diffusive behavior. To address this, self- and binary association energies were calculated for each molecule and mixture to provide information on intermolecular interaction strength and were used as input features to the ANN. An accurate, generalized ANN model was developed with an overall average absolute deviation of 4.1%. Forward input feature selection reveals the importance of critical properties and self-association energies along with other fluid properties. Additional ANNs were developed with subsets of the full input feature set to further investigate the impact of various properties on model performance. The results from two specific mixtures are discussed in additional detail: one providing an example of strong hydrogen bonding and the other an example of extreme pressure changes, with the ANN models predicting self-diffusion well in both cases.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗
DOE OSTI2024

CMOS-Based Single-Cycle in-Memory XOR/XNOR

Big data applications are on the rise, and so is the number of data centers. The ever-increasing massive data pool needs to be periodically backed up in a secure environment. Moreover, a massive amount of securely backed-up data is required for training binary convolutional neural networks for image classification. XOR and XNOR operations are essential for large-scale data copy verification, encryption, and classification algorithms. The disproportionate speed of existing compute and memory units makes the von Neumann architecture inefficient to perform these Boolean operations. Compute-in-memory (CiM) has proved to be an optimum approach for such bulk computations. The existing CiM-based XOR/XNOR techniques either require multiple cycles for computing or add to the complexity of the fabrication process. Here, we propose a CMOS-based hardware topology for single-cycle in-memory XOR/XNOR operations. Our design provides at least 2× improvement in the latency compared with other existing CMOS-compatible solutions. We verify the proposed system through circuit/system-level simulations and evaluate its robustness using a 5000-point Monte Carlo variation analysis. This all-CMOS design paves the way for practical implementation of CiM XOR/XNOR at scaled technology nodes.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1991

Non-parametric analysis of LANDSAT maps using neural nets and parallel computers

Nearest neighbor approaches and a new neural network, the Binary Diamond, are used for the classification of images of ground pixels obtained by LANDSAT satellite. The performances are evaluated by comparing classifications of a scene in the vicinity of Washington DC. The problem of optimal selection of categories is addressed as a step in the classification process.

Salu, Yehuda↗