Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “NEURAL NETWORKS”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 343 records · Page 19
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Structured Neural Network Modeling for Developing Digital Twins Models of Hydropower Generation Units

Dynamic modeling is a key part in the development of digital twin (DT) for dynamic systems. This is true for hydropower systems, where whole system modeling including penstock, turbine and generators, etc is important in realizing actuate modeling for the real systems. On the other hand, in response to the large variations of the power demand due to increased penetration of renewables such as wind and solar, hydropower systems are now required to operate in a large power generation range. This situation triggers the nonlinear characteristics of the generation unit with respect to its models. As such, it is imperative to use data driven modeling such as neural networks to learn the nonlinear dynamics of the hydropower generation unit. To achieve this objective, this study constructs a modeling and learning algorithm integrated with multiple structured neural network models for the modeling of turbine shaft speed, penstock pressure, and generator power output based on the generator power control setpoint, field current, and field voltage. In addition, the study uses the hydropower data from Tacoma Public Utilities to train and validate the proposed neural network algorithm. The results have shown that this structured neural network modeling approach can learn the system dynamics effectively by using the real-time data collected from the hydropower system with the desired modeling results.

Wang, Hong↗
DOE OSTI2025

An adaptive and stability-promoting layerwise training approach for sparse deep neural network architecture

This work presents a two-stage adaptive framework for progressively developing deep neural network (DNN) architectures that generalize well for a given training data set. In the first stage, a layerwise training approach is adopted where a new layer is added each time and trained independently by freezing parameters in the previous layers. We impose desirable structures on the DNN by employing manifold regularization, sparsity regularization, and physics-informed terms. We introduce a ε – δ – stability-promoting concept as a desirable property for a learning algorithm and show that employing manifold regularization yields a ε – δ stability-promoting algorithm. Further, we also derive the necessary conditions for the trainability of a newly added layer and investigate the training saturation problem. In the second stage of the algorithm (post-processing), a sequence of shallow networks is employed to extract information from the residual produced in the first stage, thereby improving the prediction accuracy. Numerical investigations on prototype regression and classification problems demonstrate that the proposed approach can outperform fully connected DNNs of the same size. Moreover, by equipping the physics-informed neural network (PINN) with the proposed adaptive architecture strategy to solve partial differential equations, we numerically show that adaptive PINNs not only are superior to standard PINNs but also produce interpretable hidden layers with provable stability. As a result, we also apply our architecture design strategy to solve inverse problems governed by elliptic partial differential equations.

42 ENGINEERING↗
DOE OSTI2023

Multifidelity Neural Network Formulations for Prediction of Reactive Molecular Potential Energy Surfaces

Here, this paper focuses on the development of multifidelity modeling approaches using neural network surrogates, where training data arising from multiple model forms and resolutions are integrated to predict high-fidelity response quantities of interest at lower cost. We focus on the context of quantum chemistry and the integration of information from multiple levels of theory. Important foundations include the use of symmetry function-based atomic energy vector constructions as feature vectors for representing structures across families of molecules and single-fidelity neural network training capabilities that learn the relationships needed to map feature vectors to potential energy predictions. These foundations are embedded within several multifidelity topologies that decompose the high-fidelity mapping into model-based components, including sequential formulations that admit a general nonlinear mapping across fidelities and discrepancy-based formulations that presume an additive decomposition. Methodologies are first explored and demonstrated on a pair of simple analytical test problems and then deployed for potential energy prediction for C 5 H 5 using B2PLYP-D3/6-311++G(d,p) for high-fidelity simulation data and Hartree–Fock 6-31G for low-fidelity data. For the common case of limited access to high-fidelity data, our computational results demonstrate that multifidelity neural network potential energy surface constructions achieve roughly an order of magnitude improvement, either in terms of test error reduction for equivalent total simulation cost or reduction in total cost for equivalent error.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2020

SuperCDMS Event Reconstruction Using Convolutional Neural Networks [Thesis]

The SuperCDMS experiment uses cryogenic silicon and germanium detectors to search for dark matter candidates such as WIMPs (Weakly Interacting Massive Particles) streaming through the Earth. Collisions in the silicon and germanium crystals are expected to produce phonons whose thermal signatures can be measured. This thesis first describes the integration of a new Signal Distribution Unit (SDU) to the SuperCDMS data acquisition system, which allows for synchronization of multiple detectors and electronic/mechanical noise characterization via accelerometer, antenna, and AC phase measurements. From SuperCDMS detector data it is necessary to reconstruct the energies of the particle events. This thesis explores the use of Convolutional Neural Networks (CNNs) to perform this reconstruction and finds that, although they perform well, changing the noise model breaks the model and requires the neural network to be retrained. In order to mitigate this issue, a new CNN model is proposed which includes the noise Power Spectral Density (PSD) of the data as an additional input to the CNN. While it proves to be effective as a denoising algorithm, it still fails for data with a different noise model. However, including data from multiple PSDs in the neural network training sample allows it to handle data with different types of noise while still maintaining the quality of the reconstruction. Nevertheless, neural networks trained even on multiple PSDs do not robustly handle data taken with PSDs dissimilar to those in the training sample, suggesting that CNNs may need to be retrained whenever the noise environment changes in a significant way.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2021

Neural Network Approaches for Mobile Spectroscopic Gamma-Ray Source Detection

Artificial neural networks (ANNs) for performing spectroscopic gamma-ray source identification have been previously introduced, primarily for applications in controlled laboratory settings. To understand the utility of these methods in scenarios and environments more relevant to nuclear safety and security, this work examines the use of ANNs for mobile detection, which involves highly variable gamma-ray background, low signal-to-noise ratio measurements, and low false alarm rates. Simulated data from a 2” × 4” × 16” NaI(Tl) detector are used in this work for demonstrating these concepts, and the minimum detectable activity (MDA) is used as a performance metric in assessing model performance.In addition to examining simultaneous detection and identification, binary spectral anomaly detection using autoencoders is introduced in this work, and benchmarked using detection methods based on Non-negative Matrix Factorization (NMF) and Principal Component Analysis (PCA). On average, the autoencoder provides a 12% and 23% improvement over NMF- and PCA-based detection methods, respectively. Additionally, source identification using ANNs is extended to leverage temporal dynamics by means of recurrent neural networks, and these time-dependent models outperform their time-independent counterparts by 17% for the analysis examined here. The paper concludes with a discussion on tradeoffs between the ANN-based approaches and the benchmark methods examined here.

Bilton, Kyle J. (ORCID:0000000184553689)↗
DOE OSTI2021

Accelerating Binarized Neural Networks via Bit-Tensor-Cores in Turing GPUs

Despite foreseeing tremendous speedups over conventional deep neural networks, the performance advantage of binarized neural networks (BNNs) has merely been showcased on general-purpose processors such as CPUs and GPUs. In fact, due to being unable to leverage bit-level-parallelism with a word-based architecture, GPUs have been criticized for extremely low utilization (1%) when executing BNNs. Consequently, the latest tensorcores in NVIDIA Turing GPUs start to experimentally support bit computation. In this work, we look into this brand new bit computation capability and characterize its unique features. We show that the stride of memory access can significantly affect performance delivery and a data-format co-design is highly desired to support the tensorcores for achieving superior performance than existing software solutions without tensorcores. We realize the tensorcore-accelerated BNN design, particularly the major functions for fully-connect and convolution layers — bit matrix multiplication and bit convolution. Evaluations on two NVIDIA Turing GPUs show that, with ResNet-18, our BTC-BNN design can process ImageNet at a rate of 5.6K images per second, 77% faster than state-of-the-art. Our BNN approach is released on https://github.com/pnnl/TCBNN.

Li, Ang↗
DOE OSTI2020

Long term predictions of coal fired power plant data using evolved recurrent neural networks

Abstract This work presents an investigation into the ability of recurrent neural networks (RNNs) to provide long term predictions of time series data generated by coal fired power plants. While there are numerous studies which have used artificial neural networks (ANNs) to predict coal plant parameters, to the authors’ knowledge these have almost entirely been restricted to predicting values at the next time step, and not farther into the future. Using a novel neuro-evolution strategy called Evolutionary eXploration of Augmenting Memory Models (EXAMM), we evolved RNNs with advanced memory cells to predict per-minute plant parameters and per-hour boiler parameters up to 8 hours into the future. These data sets were challenging prediction tasks as they involve spiking behavior in the parameters being predicted. While the evolved RNNs were able to successfully predict the spikes in the hourly data they did not perform very well in accurately predicting their severity. The per-minute data proved even more challenging as medium range predictions miscalculated the beginning and ending of spikes, and longer range predictions reverted to long term trends and ignored the spikes entirely. We hope this initial study will motivate further study into this highly challenging prediction problem. The use of fuel properties data generated by a new Coal Tracker Optimization (CTO) program was also investigated and this work shows that their use improved predictive ability of the evolved RNNs.

Automation & Control Systems↗
DOE OSTI2021

Demonstration of background rejection using deep convolutional neural networks in the NEXT experiment

Convolutional neural networks (CNNs) are widely used state-of-the-art computer vision tools that are becoming increasingly popular in high-energy physics. In this paper, we attempt to understand the potential of CNNs for event classification in the NEXT experiment, which will search for neutrinoless double-beta decay in $^{136}$Xe. To do so, we demonstrate the usage of CNNs for the identification of electron-positron pair production events, which exhibit a topology similar to that of a neutrinoless double-beta decay event. These events were produced in the NEXT-White high-pressure xenon TPC using 2.6 MeV gamma rays from a $^{228}$Th calibration source. We train a network on Monte Carlo-simulated events and show that, by applying on-the-fly data augmentation, the network can be made robust against differences between simulation and data. The use of CNNs offers significant improvement in signal efficiency and background rejection when compared to previous non-CNN-based analyses.[graphic not available: see fulltext]

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2024

Tokamak edge localized mode onset prediction with deep neural network and pedestal turbulence

Abstract A neural network, BES-ELMnet, predicting a quasi-periodic disruptive eruption of the plasma energy and particles known as edge localized mode (ELM) onset is developed with observed pedestal turbulence from the beam emission spectroscopy system in DIII-D. BES-ELMnet has convolutional and fully-connected layers, taking two-dimensional plasma fluctuations with a temporal window of size 128 µ s and generating a scalar output which can be interpreted as a probability of the upcoming ELM onset. As approximately labeled inter-ELM broadband ( 15 kHz ⩽ f ⩽ 150 kHz ) fluctuations are given to the network, BES-ELMnet learns by itself ELM-related precursors arising before the onsets through supervised learning scheme. BES-ELMnet achieves the gradually increasing ELM onset probabilities between two consecutive ELMs during the inter-ELM phases and can forecast the first ELM onsets which occur after the high confinement mode transition. We further investigate the network generality in terms of the selected frequency band to ensure the use of BES-ELMnet for various operation regimes without changing the trained architecture. Therefore, our novel prediction method will enhance a proactive high confinement mode control of fusion-grade plasmas.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2020

Reconstructing 3D Charge Depositions in the MicroBooNE Liquid Argon Time Projection Chamber using Convolutional Neural Networks

We have developed a convolutional neural network (CNN) that reconstructs the three dimensional location of ionization produced by charge particles traversing a wire-readout liquid argon time projection chamber (LArTPC) at MicroBooNE. This method is the first CNN application to reconstruct 3D space points directly from two dimensional images of the different wire plane views. I.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2022

Classifying Metal-Binding Sites with Neural Networks

A pre-trained convolutional neural network was fine-tuned for three separate classification tasks, distinguishing 2D images of: 1) single amino acids, 2) protein structural ball and stick images of metalloproteins, and 3) protein structural ball and stick images of metalloenzymes with the metal cofactors removed. Images used in the training, testing and validation are shared here.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2022

Anomaly detection with spiking neural networks

Detecting anomalies with a spiking neural network is provided. An input layer receives a number of inputs and converts them into phase-coded spikes, wherein each input is contained within a number of progressively larger neighborhoods of surrounding inputs. From the phase-coded spikes, a median value of each input is computed for each size neighborhood. An absolute difference of each input from its median value is computed for each size neighborhood. A median absolute difference (MAD) of each input is computed for each size neighborhood. For each input, an adaptive median filter (AMF) determines if a MAD for any size neighborhood exceeds a respective threshold. If one or more neighborhoods exceeds its threshold, the AMF outputs the median value of the input for the smallest neighborhood. If none of the neighborhoods exceeds the threshold, the AMF outputs the original value of the input.

Verzi, Stephen Joseph↗
DOE OSTI2024

Physics-informed heterogeneous graph neural networks for DC blocker placement

The threat of geomagnetic disturbances (GMDs) to the reliable operation of the bulk energy system has spurred the development of effective strategies for mitigating their impacts. One such approach involves placing transformer neutral blocking devices, which interrupt the path of geomagnetically induced currents (GICs) to limit their impact. The high cost of these devices and the sparsity of transformers that experience high GICs during GMD events, however, calls for a sparse placement strategy that involves high computational cost. To address this challenge, we developed a physics-informed heterogeneous graph neural network (PIHGNN) for solving the graph-based dc-blocker placement problem. Our approach combines a heterogeneous graph neural network (HGNN) with a physics-informed neural network (PINN) to capture the diverse types of nodes and edges in ac/dc networks and incorporates the physical laws of the power grid. We train the PIHGNN model using a surrogate power flow model and validate it using case studies. Results demonstrate that PIHGNN can effectively and efficiently support the deployment of GIC dc-current blockers, ensuring the continued supply of electricity to meet societal demands. Furthermore, our approach has the potential to contribute to the development of more reliable and resilient power grids capable of withstanding the growing threat that GMDs pose.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2025

Bayesian Entropy Neural Networks for physics-aware prediction

This article addresses the need for deep learning models to integrate well-defined constraints into their outputs, driven by their application in surrogate models, learning with limited data and partial information, and scenarios requiring flexible model behavior to incorporate non-data sample information. We introduce Bayesian Entropy Neural Networks (BENN), a framework grounded in Maximum Entropy (MaxEnt) principles, designed to impose constraints on Bayesian Neural Network (BNN) predictions. BENN is capable of constraining not only the predicted values but also their derivatives and variances, ensuring a more robust and reliable model output. To achieve simultaneous uncertainty quantification and constraint satisfaction, we employ the method of multipliers approach. This allows for the concurrent estimation of neural network parameters and the Lagrangian multipliers associated with the constraints. Our experiments, spanning diverse applications such as beam deflection modeling and microstructure generation, demonstrate the effectiveness of BENN. The results highlight significant improvements over traditional BNNs and showcase competitive performance relative to contemporary constrained deep learning methods.

14 SOLAR ENERGY↗
DOE OSTI2025

Short-Term Forecasting of Thermostatic and Residential Loads Using Long Short-Term Memory Recurrent Neural Networks

Internet of Things (IoT) devices in smart grids enable intelligent energy management for grid managers and personalized energy services for consumers. Investigating a smart grid with IoT devices requires a simulation framework with IoT devices modeling. However, there lack comprehensive study on the modeling of IoT devices in smart grids. This paper investigates the IoT device modeling of a thermostatic load and implements the recurrent neural networks model for short-term load forecasting in this IoT-based thermostatic load. The recurrent neural network structure is leveraged to build a load forecasting model on temporal correlation. The temporal recurrent neural network layers including long short-term memory cells are employed to learn the data from both the simulation platform and New South Wales residential datasets. The simulation results are provided for demonstration.

electric load forecasting↗
DOE OSTI2022

A physics-constrained neural network for multiphase flows

The present study develops a physics-constrained neural network (PCNN) to predict sequential patterns and motions of multiphase flows (MPFs), which includes strong interactions among various fluid phases. To predict the order parameters, which locate individual phases in the future time, a neural network (NN) is applied to quickly infer the dynamics of the phases by encoding observations. The multiphase consistent and conservative boundedness mapping algorithm (MCBOM) is next implemented to correct the predicted order parameters. This enforces the predicted order parameters to strictly satisfy the mass conservation, the summation of the volume fractions of the phases to be unity, the consistency of reduction, and the boundedness of the order parameters. Then, the density of the fluid mixture is updated from the corrected order parameters. Finally, the velocity in the future time is predicted by another NN with the same network structure, but the conservation of momentum is included in the loss function to shrink the parameter space. The proposed PCNN for MPFs sequentially performs (NN)-(MCBOM)-(NN), which avoids nonphysical behaviors of the order parameters, accelerates the convergence, and requires fewer data to make predictions. Numerical experiments demonstrate that the proposed PCNN is capable of predicting MPFs effectively.

Mechanics↗
DOE OSTI2024

Transformer Neural Networks with Spatiotemporal Attention for Predictive Control and Optimization of Industrial Processes

In the context of real-time optimization and model predictive control of industrial systems, machine learning, and neural networks represent cutting-edge tools that hold promise for enhancing dynamic modeling. This work presents a novel transformer neural network architecture for real-time optimization and model predictive control. This network design includes a modified attention mechanism inspired by positional embedding attention from vision transformers and task-specific modifications to the input-output structure of the transformer’s decoder stack. Experiments were conducted using data from a 450 MW coal-fired power plant to evaluate this approach's effectiveness. The transformer neural network was compared with conventional recurrent models, including GRU and LSTM. The transformer exhibited a 6% increase in the R-squared (R2) value of predictions and an 83% reduction in mean squared error (MSE). Computation time was also reduced by 84% compared to conventional recurrent models.

Gallup, Ethan R.↗
DOE OSTI2025

A Parallel Alternative for Energy-Efficient Neural Network Training and Inferencing

Energy efficiency of training and inferencing with large neural network models is a critical challenge facing the future of sustainable large-scale machine learning workloads. This paper introduces an alternative strategy, called phantom parallelism, to minimize the net energy consumption of traditional tensor (model) parallelism, the most energy-inefficient component of large neural network training. The approach is presented in the context of feed-forward network architectures as a preliminary, but comprehensive, proof-of-principle study of the proposed methodology. We derive new forward and backward propagation operators for phantom parallelism, implement them as custom autograd operations within an end-to-end phantom parallel training pipeline and compare its parallel performance and energy-efficiency against those of conventional tensor parallel training pipelines. Formal analyses that predict lower bandwidth and FLOP counts are presented with supporting empirical results on up to 256 GPUs that corroborate these gains. Experiments are shown to deliver ∼50% reduction in the energy consumed to train FFNs using the proposed phantom parallel approach when compared with conventional tensor parallel methods. Additionally, the proposed approach is shown to train smaller phantom models to the same model loss on smaller GPU counts as larger tensor parallel models on larger GPU counts offering the possibility for even greater energy savings.

Seal, Sudip [ORNL] (ORCID:0000000332330656)↗