Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Neural Networks (Computer)”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 847 records · Page 47

A scalable algorithm for the optimization of neural network architectures

In this work, we propose a new scalable method to optimize the architecture of an artificial neural network. The proposed algorithm, called Greedy Search for Neural Network Architecture, aims to determine a neural network with minimal number of layers that is at least as performant as neural networks of the same structure identified by other hyperparameter search algorithms in terms of accuracy and computational cost. Numerical results performed on benchmark datasets show that, for these datasets, our method outperforms state-of-the-art hyperparameter optimization algorithms in terms of attainable predictive performance by the selected neural network architecture, and time-to-solution for the hyperparameter optimization to complete.

97 MATHEMATICS AND COMPUTING↗

Stiff neural ordinary differential equations

Neural Ordinary Differential Equations (ODEs) are a promising approach to learn dynamical models from time-series data in science and engineering applications. This work aims at learning neural ODEs for stiff systems, which are usually raised from chemical kinetic modeling in chemical and biological systems. We first show the challenges of learning neural ODEs in the classical stiff ODE systems of Robertson’s problem and propose techniques to mitigate the challenges associated with scale separations in stiff systems. We then present successful demonstrations in stiff systems of Robertson’s problem and an air pollution problem. The demonstrations show that the usage of deep networks with rectified activations, proper scaling of the network outputs as well as loss functions, and stabilized gradient calculations are the key techniques enabling the learning of stiff neural ODEs. The success of learning stiff neural ODEs opens up possibilities of using neural ODEs in applications with widely varying time-scales, such as chemical dynamics in energy conversion, environmental engineering, and life sciences.

97 MATHEMATICS AND COMPUTING↗

Neural network uncertainty assessment using Bayesian statistics: a remote sensing application

Neural network (NN) techniques have proved successful for many regression problems, in particular for remote sensing; however, uncertainty estimates are rarely provided. In this article, a Bayesian technique to evaluate uncertainties of the NN parameters (i.e., synaptic weights) is first presented. In contrast to more traditional approaches based on point estimation of the NN weights, we assess uncertainties on such estimates to monitor the robustness of the NN model. These theoretical developments are illustrated by applying them to the problem of retrieving surface skin temperature, microwave surface emissivities, and integrated water vapor content from a combined analysis of satellite microwave and infrared observations over land. The weight uncertainty estimates are then used to compute analytically the uncertainties in the network outputs (i.e., error bars and correlation structure of these errors). Such quantities are very important for evaluating any application of an NN model. The uncertainties on the NN Jacobians are then considered in the third part of this article. Used for regression fitting, NN models can be used effectively to represent highly nonlinear, multivariate functions. In this situation, most emphasis is put on estimating the output errors, but almost no attention has been given to errors associated with the internal structure of the regression model. The complex structure of dependency inside the NN is the essence of the model, and assessing its quality, coherency, and physical character makes all the difference between a blackbox model with small output errors and a reliable, robust, and physically coherent model. Such dependency structures are described to the first order by the NN Jacobians: they indicate the sensitivity of one output with respect to the inputs of the model for given input data. We use a Monte Carlo integration procedure to estimate the robustness of the NN Jacobians. A regularization strategy based on principal component analysis is proposed to suppress the multicollinearities in order to make these Jacobians robust and physically meaningful.

Neural Networks (Computer)↗

Investigating Anomalies in Compute Clusters: An Unsupervised Learning Approach

As compute clusters continue to grow in scale and complexity, the frequency of detected anomalies in their operation significantly increases. Timely detection of anomalous events is vital to maintain system efficiency and availability. This study presents an attentionbased graph neural network (GNN) for detecting anomalies in clusters at the compute node level and for providing detailed root cause analysis. We show the effectiveness of attention-based GNNs to accurately detect and localize anomalies on real-world datasets.

Lu, Yiyang↗

Understanding machine learning approaches for partial differential equations

Recent works in computational physics have been successful in the super-resolution of numerical solutions of partial differential equations (PDEs) via neural networks. In this paper we explore the possibilities and limitations of the data-driven discretization approach implemented by Bar-Sinai et. al., while contrasting it with a simpler yet weaker approach utilizing the nangs Python library. The results demonstrate what neural network parameters optimize accuracy and performance, as well as what conditions on PDEs are necessary to maintain convergence.

97 MATHEMATICS AND COMPUTING↗

A Neural Network Correction to the Scalar Approximation in Radiative Transfer

The next generation of advanced high-resolution sensors in geostationary orbit will gather detailed information for studying the Earth system. There is an increasing desire to perform observing system simulation experiments (OSSEs) for new sensors during the development phase of the mission in order to better leverage information content from the new and existing sensors. Forward radiative transfer calculations that simulate the observing characteristics of a new instrument are the first step to an OSSE, and they are computationally intensive. The scalar approximation to the radiative transfer equation, a simplification of the vector representation, can save considerable computational cost, but produces errors in top of the atmosphere (TOA) radiance as large as 10% due to neglecting polarization effects. This article presents an artificial neural network technique to correct scalar TOA radiance over both land and ocean surfaces to within 1% of vector-calculated radiance. A neural network was trained on a database of scalar-vector TOA radiance differences at a large range of solar and viewing angles for several thousand realistic atmospheric vertical profiles that were sampled from a high resolution (7 km) global atmospheric transport model. The profiles include Rayleigh scattering and aerosol scattering and absorption. Training and validation of the neural network was demonstrated for two wavelengths in the ultraviolet-visible (US-Vis) spectral range (354 nm and 670 nm). The significant computational savings accrued from using a scalar approximation plus neural network correction approach to simulating TOA radiance will make feasible hyperspectral forward simulations of high-resolution sensors on geostationary satellites, such as TEMPO, GOES-R, GEMS, and SENTINEL-4.

TOA↗

Deep learning with mixup augmentation for improved pore detection during additive manufacturing

In additive manufacturing (AM), process defects such as keyhole pores are difficult to anticipate, affecting the quality and integrity of the AM-produced materials. Hence, considerable efforts have aimed to predict these process defects by training machine learning (ML) models using passive measurements such as acoustic emissions. This work considered a dataset in which keyhole pores of a laser powder bed fusion (LPBF) experiment were identified using X-ray radiography and then registered both in space and time to acoustic measurements recorded during the LPBF experiment. Due to AM’s intrinsic process controls, where a pore-forming event is relatively rare, the acoustic datasets collected during monitoring include more non-pores than pores. In other words, the dataset for ML model development is imbalanced. Moreover, this imbalanced and sparse data phenomenon remains ubiquitous across many AM monitoring schemes since training data is nontrivial to collect. Hence, we propose a machine learning approach to improve this dataset imbalance and enhance the prediction accuracy of pore-labeled data. Specifically, we investigate how data augmentation helps predict pores and non-pores better. This imbalance is improved using recent advances in data augmentation called Mixup, a weak-supervised learning method. Convolutional neural networks (CNNs) are trained on original and augmented datasets, and an appreciable increase in performance is reported when testing on five different experimental trials. When ML models are trained on original and augmented datasets, they achieve an accuracy of 95% and 99% on test datasets, respectively. We also provide information on how dataset size affects model performance. Lastly, we investigate the optimal Mixup parameters for augmentation in the context of CNN performance.

42 ENGINEERING↗

Neural Network Prediction of Strong Lensing Systems with Domain Adaptation and Uncertainty Quantification

Modeling strong gravitational lenses is computationally expensive for the complex data from modern and next-generation cosmic surveys. Deep learning has emerged as a promising approach for finding lenses and predicting lensing parameters, such as the Einstein radius. Mean-variance Estimators (MVEs) are a common approach for obtaining aleatoric (data) uncertainties from a neural network prediction. However, neural networks have not been demonstrated to perform well on out-of-domain target data successfully - e.g., when trained on simulated data and applied to real, observational data. In this work, we perform the first study of the efficacy of MVEs in combination with unsupervised domain adaptation (UDA) on strong lensing data. The source domain data is noiseless, and the target domain data has noise mimicking modern cosmology surveys. We find that adding UDA to MVE increases the accuracy on the target data by a factor of about two over an MVE model without UDA. Including UDA also permits much more well-calibrated aleatoric uncertainty predictions. Advancements in this approach may enable future applications of MVE models to real observational data.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Attentional Ptycho-Tomography (APT) for three-dimensional nanoscale X-ray imaging with minimal data acquisition and computation time

Abstract Noninvasive X-ray imaging of nanoscale three-dimensional objects, such as integrated circuits (ICs), generally requires two types of scanning: ptychographic, which is translational and returns estimates of the complex electromagnetic field through the IC; combined with a tomographic scan, which collects these complex field projections from multiple angles. Here, we present Attentional Ptycho-Tomography (APT), an approach to drastically reduce the amount of angular scanning, and thus the total acquisition time. APT is machine learning-based, utilizing axial self-Attention for Ptycho-Tomographic reconstruction. APT is trained to obtain accurate reconstructions of the ICs, despite the incompleteness of the measurements. The training process includes regularizing priors in the form of typical patterns found in IC interiors, and the physics of X-ray propagation through the IC. We show that APT with ×12 reduced angles achieves fidelity comparable to the gold standard Simultaneous Algebraic Reconstruction Technique (SART) with the original set of angles. When using the same set of reduced angles, then APT also outperforms Filtered Back Projection (FBP), Simultaneous Iterative Reconstruction Technique (SIRT) and SART. The time needed to compute the reconstruction is also reduced, because the trained neural network is a forward operation, unlike the iterative nature of these alternatives. Our experiments show that, without loss in quality, for a 4.48 × 93.2 × 3.92 µm 3 IC (≃6 × 10 8 voxels), APT reduces the total data acquisition and computation time from 67.96 h to 38 min. We expect our physics-assisted and attention-utilizing machine learning framework to be applicable to other branches of nanoscale imaging, including materials science and biological imaging.

47 OTHER INSTRUMENTATION↗

Intelligence Applied to Air Vehicles

The exponential growth in information technology has provided the potential for air vehicle capabilities that were previously unavailable to mission and vehicle designers. The increasing capabilities of computer hardware and software, including new developments such as neural networks, provide a new balance of work between humans and machines. This paper will describe several NASA projects, and review results and conclusions from ground and flight investigations where vehicle intelligence was developed and applied to aeronautical and space systems. In the first example, flight results from a neural network flight control demonstration will be reviewed. Using, a highly-modified F-15 aircraft, a NASA/Dryden experimental flight test program has demonstrated how the neural network software can correctly identify and respond to changes in aircraft stability and control characteristics. Using its on-line learning capability, the neural net software would identify that something in the vehicle has changed, then reconfigure the flight control computer system to adapt to those changes. The results of the Remote Agent software project will be presented. This capability will reduce the cost of future spacecraft operations as computers become "thinking" partners along with humans. In addition, the paper will describe the objectives and plans for the autonomous airplane program and the autonomous rotorcraft project. Technologies will also be developed.

Rosen, Robert↗

A Deep Learning Approach to Fast Radiative Transfer

Due to the sheer volume of data, leveraging satellite instrument observations effectively in a data assimilation context for numerical weather prediction or for remote sensing requires a radiative transfer model as an observation operator that is both fast and accurate at the same time. Physics-based line-by-line radiative transfer (RT) models fulfil the requirement for accuracy, but are too slow and too costly in computational terms for operational applications. Therefore, fast methods were developed to be able to perform fast RT calculations using techniques such as spectral sampling or pre-computed look-up tables. The operational fast models currently calculate the absorption and scattering coefficients from the pre-computed regression coefficients and atmospheric state and cloud profiles. As a novel solution to this problem, this work investigates a deep learning approach to replace the regression coefficients in the fast RT models. A selection of hidden-layer neural network configurations is trained against atmospheric transmittance profile data computed by an accurate line-by-line model and their performance is evaluated and their advantages and disadvantages are discussed.

Machine learning↗

Explaining machine-learning models for gamma-ray detection and identification

As more complex predictive models are used for gamma-ray spectral analysis, methods are needed to probe and understand their predictions and behavior. Recent work has begun to bring the latest techniques from the field of Explainable Artificial Intelligence (XAI) into the applications of gamma-ray spectroscopy, including the introduction of gradient-based methods like saliency mapping and Gradient-weighted Class Activation Mapping (Grad-CAM), and black box methods like Local Interpretable Model-agnostic Explanations (LIME) and SHapley Additive exPlanations (SHAP). In addition, new sources of synthetic radiological data are becoming available, and these new data sets present opportunities to train models using more data than ever before. In this work, we use a neural network model trained on synthetic NaI(Tl) urban search data to compare some of these explanation methods and identify modifications that need to be applied to adapt the methods to gamma-ray spectral data. We find that the black box methods LIME and SHAP are especially accurate in their results, and recommend SHAP since it requires little hyperparameter tuning. We also propose and demonstrate a technique for generating counterfactual explanations using orthogonal projections of LIME and SHAP explanations.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

End-to-End Pipeline for Trigger Detection on Hit and Track Graphs

There has been a surge of interest in applying deep learning in particle and nuclear physics to replace labor-intensive offline data analysis with automated online machine learning tasks. This paper details a novel AI-enabled triggering solution for physics experiments in Relativistic Heavy Ion Collider and future Electron-Ion Collider. The triggering system consists of a comprehensive end-to-end pipeline based on Graph Neural Networks that classifies trigger events versus background events, makes online decisions to retain signal data, and enables efficient data acquisition. Here, the triggering system first starts with the coordinates of pixel hits lit up by passing particles in the detector, applies three stages of event processing (hits clustering, track reconstruction, and trigger detection), and labels all processed events with the binary tag of trigger versus background events. By switching among different objective functions, we train the Graph Neural Networks in the pipeline to solve multiple tasks: the edge-level track reconstruction problem, the edge-level track adjacency matrix prediction, and the graph-level trigger detection problem. We propose a novel method to treat the events as track-graphs instead of hit-graphs. This method focuses on intertrack relations and is driven by underlying physics processing. As a result, it attains a solid performance (around 72% accuracy) for trigger detection and outperforms the baseline method using hit-graphs by 2% higher accuracy.

97 MATHEMATICS AND COMPUTING↗

Building an ab initio solvated DNA model using Euclidean neural networks

Accurately modeling large biomolecules such as DNA from first principles is fundamentally challenging due to the steep computational scaling of ab initio quantum chemistry methods. This limitation becomes even more prominent when modeling biomolecules in solution due to the need to include large numbers of solvent molecules. We present a machine-learned electron density model based on a Euclidean neural network framework that includes a built-in understanding of equivariance to model explicitly solvated double-stranded DNA. By training the machine learning model using molecular fragments that sample the key DNA and solvent interactions, we show that the model predicts electron densities of arbitrary systems of solvated DNA accurately, resolves polarization effects that are neglected by classical force fields, and captures the physics of the DNA-solvent interaction at the ab initio level.

59 BASIC BIOLOGICAL SCIENCES↗

Neural-based time series forecasting of loss of coolant accidents in nuclear power plants

During the last few years, deep learning in neural networks has demonstrated impressive successes in the areas of computer vision, speech and image recognition, text generation, and many others. However, sensitive engineering areas such as nuclear engineering benefited less from these efficient techniques. In this work, deep learning expert systems are utilized to model and predict time series progression of a design-basis nuclear accident, featuring a loss of coolant accident. Two major findings are accomplished in this work. First, the ability to train expert systems with high accuracy, which could help nuclear power plant operators to figure out plant responses during the accident. Second, building fast, efficient, and accurate deep models to simulate nuclear phenomena, which could be valuable to nuclear computational science. In this work, large amount of time series data is obtained from simulation tools by simulating different conditions of the base-case/nominal accident scenario. Four critical outputs/responses are monitored during the accident (e.g. temperature, pressure, break flow rate, water level). Two approaches are adopted in this work. The first approach is to use feedforward deep neural networks (DNN) to fit all time steps and outputs in a single model. The second approach is to use long short-term memory (LSTM) to fit all time steps together for each reactor response separately. Both DNN and LSTM demonstrate very good performance in predicting the test and base-case scenarios, with accuracy as low as 92% and as high as 99%, where these test scenarios are unknown to the expert systems and are not included in the model training. In addition, both approaches demonstrate a significant reduction in computational costs, as the deep expert system is able to accurately predict the accident 100,000 times faster than the original simulation tool. Given sufficient data, the methodology adopted in this study demonstrates that DNN/LSTM expert systems can be used as a decision support system to model advanced time series phenomena within nuclear power plants with high accuracy and negligible computational costs.

42 ENGINEERING↗

Nonvolatile Electrochemical Random‐Access Memory under Short Circuit

Abstract Electrochemical random‐access memory (ECRAM) is a recently developed and highly promising analog resistive memory element for in‐memory computing. One longstanding challenge of ECRAM is attaining retention time beyond a few hours. This short retention has precluded ECRAM from being considered for inference classification in deep neural networks, which is likely the largest opportunity for in‐memory computing. In this work, an ECRAM cell with orders of magnitude longer retention than previously achieved is developed, and which is anticipated to exceed ten years at 85 °C. This study hypothesizes that the origin of this exceptional retention is phase separation, which enables the formation of multiple effectively equilibrium resistance states. This work highlights the promises and opportunities to use phase separation to yield ECRAM cells with exceptionally long, and potentially permanent, retention times.

Kim, Diana S.↗

Remaining Useful Strength (RUS) Prediction of SiCf-SiCm Composite Materials Using Deep Learning and Acoustic Emission

Prognosis techniques for prediction of remaining useful life (RUL) are of crucial importance to the management of complex systems for they can lead to appropriate maintenance interventions and improvements in reliability. While various data-driven methods have been introduced to predict the remaining useful life (RUL) of machinery systems or batteries, no research has been reported on the remaining useful strength (RUS) prediction of silicon carbide fiber reinforced silicon carbide matrix (SiCf-SiCm) materials with pivotal role in its potential usage as a structural material in nuclear reactors and turbine engines. Knowledge of its degradation process is of the utmost importance to the manufacturers. For this purpose, two approaches based on the machine-learning techniques of random-forest (RF) and convolutional neural network (CNN) are proposed to predict the RUS of SiCf-SiCm using only acoustic emission (AE) signals generated during the material’s stress applying process. Experimental results show that the CNN models achieved better predictive performance than the RF models but the latter with expert-engineered features achieves better prediction for AE signals in the early stage of degradation. Additionally, our results demonstrate that both models can correctly predict the SiCf-SiCm RUS as evaluated by our robust testing method from which the best average root mean square error (RMSE) and Pearson correlation coefficient of 3.55 ksi units and 0.85 were obtained.

36 MATERIALS SCIENCE↗

Multilevel-in-width training for deep neural network regression

A common challenge in regression is that for many problems, the degrees of freedom required for a high-quality solution also allows for overfitting. Regularization is a class of strategies that seek to restrict the range of possible solutions so as to discourage overfitting while still enabling good solutions, and different regularization strategies impose different types of restrictions. In this paper, we present a multilevel regularization strategy that constructs and trains a hierarchy of neural networks, each of which has layers that are wider versions of the previous network's layers. Here, we draw intuition and techniques from the field of Algebraic Multigrid (AMG), traditionally used for solving linear and nonlinear systems of equations, and specifically adapt the Full Approximation Scheme (FAS) for nonlinear systems of equations to the problem of deep learning. Training through V-cycles then encourage the neural networks to build a hierarchical understanding of the problem. We refer to this approach as multilevel-in-width to distinguish from prior multilevel works which hierarchically alter the depth of neural networks. The resulting approach is a highly flexible framework that can be applied to a variety of layer types, which we demonstrate with both fully connected and convolutional layers. We experimentally show with PDE regression problems that our multilevel training approach is an effective regularizer, improving the generalize performance of the neural networks studied.

97 MATHEMATICS AND COMPUTING↗