Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Generative Neural Networks”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

Few-view computed tomography reconstruction using deep neural network inference

A system for generating 2D slices of a 3D image of a target volume is provided. The system receives a target sinogram collected during a computed tomography scan of the target volume. The system inputs the target sinogram to a convolutional neural network (CNN) to generate predicted 2D slices of the 3D image. The CNN is trained using training 2D slices of training 3D images. The system initializes 2D slices to the predicted 2D slices. The system reconstructs 2D slices of the 3D image from the target sinogram and the initialized 2D slices.

Kim, Hyojin↗

Deep Image Prior Enabled Full Waveform Inversion (Final Technical Report)

MS Student Naveen Gupta worked on the problem of full waveform inversion (FWI) using neural networks as shown in Figure 1. Our goal was to learn a neural network to represent the subsurface velocity model, which when fed into the FWI module (implemented using a numerical forward model of wave equations) produces amplitude estimates that match with ground-truth observations of amplitude. We used neural networks to solve the inverse problem of estimating velocity distributions for a given seismic amplitude data such that, once trained, our neural network model can generate a distribution of velocity profiles for different random vectors fed as inputs to the neural network model.

97 MATHEMATICS AND COMPUTING↗

Structured Neural Network Modeling for Developing Digital Twins Models of Hydropower Generation Units

Dynamic modeling is a key part in the development of digital twin (DT) for dynamic systems. This is true for hydropower systems, where whole system modeling including penstock, turbine and generators, etc is important in realizing actuate modeling for the real systems. On the other hand, in response to the large variations of the power demand due to increased penetration of renewables such as wind and solar, hydropower systems are now required to operate in a large power generation range. This situation triggers the nonlinear characteristics of the generation unit with respect to its models. As such, it is imperative to use data driven modeling such as neural networks to learn the nonlinear dynamics of the hydropower generation unit. To achieve this objective, this study constructs a modeling and learning algorithm integrated with multiple structured neural network models for the modeling of turbine shaft speed, penstock pressure, and generator power output based on the generator power control setpoint, field current, and field voltage. In addition, the study uses the hydropower data from Tacoma Public Utilities to train and validate the proposed neural network algorithm. The results have shown that this structured neural network modeling approach can learn the system dynamics effectively by using the real-time data collected from the hydropower system with the desired modeling results.

Wang, Hong↗

Diffusion-Model-Assisted Supervised Learning of Generative Models for Density Estimation

Here, we present a supervised learning framework of training generative models for density estimation. Generative models, including generative adversarial networks (GANs), normalizing flows, and variational auto-encoders (VAEs), are usually considered as unsupervised learning models, because labeled data are usually unavailable for training. Despite the success of the generative models, there are several issues with the unsupervised training, e.g., requirement of reversible architectures, vanishing gradients, and training instability. To enable supervised learning in generative models, we utilize the score-based diffusion model to generate labeled data. Unlike existing diffusion models that train neural networks to learn the score function, we develop a training-free score estimation method. This approach uses mini-batch-based Monte Carlo estimators to directly approximate the score function at any spatial-temporal location in solving an ordinary differential equation (ODE), corresponding to the reverse-time stochastic differential equation (SDE). This approach can offer both high accuracy and substantial time savings in neural network training. Once the labeled data are generated, we can train a simple, fully connected neural network to learn the generative model in the supervised manner. Compared with existing normalizing flow models, our method does not require the use of reversible neural networks and avoids the computation of the Jacobian matrix. Compared with existing diffusion models, our method does not need to solve the reverse-time SDE to generate new samples. As a result, the sampling efficiency is significantly improved. We demonstrate the performance of our method by applying it to a set of 2D datasets as well as real data from the University of California Irvine (UCI) repository.

97 MATHEMATICS AND COMPUTING↗

Cutting the Deployment Costs of Physics-Based MPC in Buildings by Simulation-Based Imitation Learning

It has been shown that model predictive control (MPC) is a promising solution for energy-efficient building operations. However, the deployment of MPC in a large portion of the building stock has not been possible partially because of high installation costs. Every building is unique and requires a tailored MPC solution. The best performing solutions are often based on physics-based modeling, which is, however, computationally expensive and requires dedicated software. A promising direction that tackles this problem is to train a neural network-based optimal control policy to imitate the behavior of physics-based MPC from the simulation data generated offline. The neural networks give control actions that closely approximate those produced by physics-based MPC, but with a fraction of the computational and memory requirements and without the need for licensed software. The main advantage of the proposed approach stems from simple evaluation at execution time, leading to low computational footprints and easy deployment on embedded HW platforms. In the case study, we present the energy savings potential of physics-based MPC applied to an office building in Belgium. We demonstrate how neural network approximators can be used to cut the implementation and maintenance costs of MPC deployment without compromising performance. We also critically assess the presented approach by pointing out the remaining challenges and open research questions.

Drgona, Jan↗

A data-driven CO 2 leakage detection using seismic data and spatial–temporal densely connected convolutional neural networks

In carbon capture and sequestration (also known as carbon capture and storage, or CCS), developing effective monitoring methods is needed to detect and respond to CO 2 leakage. CO 2 leakage detection methods rely on geophysical observations and monitoring sensor network. However, traditional methods usually require the development of site-specific physical models and expert interpretation, and the effectiveness of these methods can be limited to different application locations, operational scenarios, and conditions. Here, we developed a novel data-driven leakage detection method based on densely connected convolutional neural networks. Our method is an end-to-end detection approach, that differs from conventional leakage monitoring methods by directly learning a mapping relationship between seismic data and the CO 2 leakage mass. To account for the spatial and temporal characteristics of seismic data, our novel networks architecture combines 1D and 2D convolutional neural networks. To overcome the computational expense of solving optimization problems, we apply a densely-connecting strategy in our network architecture that reduces the number of network parameters. Based on the features generated by our convolutional neural networks, we further incorporate a long short-term memory network to utilize time-sequential information, which further improves the detection accuracy. Lastly, we employ our detection method to synthetic seismic datasets generated based on flow simulations of a hypothetical CO 2 storage scenario with injection into a partially compartmentalized sandstone storage reservoir. To evaluate method performance, we conducted multiple experiments including a random leakage test, a sequential test, and a robustness test. Numerical results show that our CO 2 leakage detection method successfully detects the leakage and accurately predicts the leakage mass, suggesting that it has the potential for application in monitoring of real CO 2 storage sites.

58 GEOSCIENCES↗

Stimulus-dependent suppression of chaos in recurrent neural networks

Neuronal activity arises from an interaction between ongoing firing generated spontaneously by neural circuits and responses driven by external stimuli. Using mean-field analysis, we ask how a neural network that intrinsically generates chaotic patterns of activity can remain sensitive to extrinsic input. We find that inputs not only drive network responses, but they also actively suppress ongoing activity, ultimately leading to a phase transition in which chaos is completely eliminated. The critical input intensity at the phase transition is a nonmonotonic function of stimulus frequency, revealing a 'resonant' frequency at which the input is most effective at suppressing chaos even though the power spectrum of the spontaneous activity peaks at zero and falls exponentially. A prediction of our analysis is that the variance of neural responses should be most strongly suppressed at frequencies matching the range over which many sensory systems operate.

97 MATHEMATICS AND COMPUTING↗

Heuristic Dynamic Programming for Adaptive Virtual Synchronous Generators

In this paper a neural network heuristic dynamic programing (HDP) is used for optimal control of the virtual inertia-based control of grid connected three-phase inverters. It is shown that the conventional virtual inertia controllers are not suited for non-inductive grids. A neural network-based controller is proposed to adapt to any impedance angle. Applying an adaptive dynamic programming controller instead of a supervised controlled method enables the system to adjust itself to different conditions. The proposed HDP consists of two subnetworks: critic network and action network. These networks can be trained during the same training cycle to decrease the training time. The simulation results confirm that the proposed neural network HDP controller performs better than the traditional direct-fed voltage and/or reactive power controllers in virtual inertia control schemes.

24 POWER TRANSMISSION AND DISTRIBUTION↗

3D segmentation using space carving and 2D convolutional neural networks

A system for generating a 3D segmentation of a target volume is provided. The system accesses views of an X-ray scan of a target volume. The system applies a 2D CNN to each view to generate a 2D multi-channel feature vector for each view. The system applies a space carver to generate a 3D channel volume for each channel based on the 2D multi-channel feature vectors. The system then applies a linear combining technique to the 3D channel volumes to generate a 3D multi-label map that represents a 3D segmentation of the target volume.

97 MATHEMATICS AND COMPUTING↗

Augmenting C. elegans Microscopic Dataset for Accelerated Pattern Recognition

The detection of cell shape changes in 3D time-lapse images of complex tissues is an important task. However, it is a challenging and tedious task to establish a comprehensive dataset to improve the performance of deep learning models. In the paper, we present a deep learning approach to augment 3D live images of the Caenorhabditis elegans embryo, so that we can further speed up the specific structure pattern recognition. We use an unsupervised training over unlabeled images to generate supplementary datasets for further pattern recognition. Technically, we used Alex-style neural networks in a generative adversarial network framework to generate new datasets that have common features of the C. elegans membrane structure. We also made the dataset available for a broad scientific community.

Lu, Zheng↗

Complexity-calibrated benchmarks for machine learning reveal when prediction algorithms succeed and mislead

Abstract Recurrent neural networks are used to forecast time series in finance, climate, language, and from many other domains. Reservoir computers are a particularly easily trainable form of recurrent neural network. Recently, a “next-generation” reservoir computer was introduced in which the memory trace involves only a finite number of previous symbols. We explore the inherent limitations of finite-past memory traces in this intriguing proposal. A lower bound from Fano’s inequality shows that, on highly non-Markovian processes generated by large probabilistic state machines, next-generation reservoir computers with reasonably long memory traces have an error probability that is at least $$\sim 60\%$$ ∼ 60 % higher than the minimal attainable error probability in predicting the next observation. More generally, it appears that popular recurrent neural networks fall far short of optimally predicting such complex processes. These results highlight the need for a new generation of optimized recurrent neural network architectures. Alongside this finding, we present concentration-of-measure results for randomly-generated but complex processes. One conclusion is that large probabilistic state machines—specifically, large $$\epsilon$$ ϵ -machines—are key to generating challenging and structurally-unbiased stimuli for ground-truthing recurrent neural network architectures.

97 MATHEMATICS AND COMPUTING↗

Hypothesis testing via AI: Generating physically interpretable models of scientific data with machine learning (Full Technical Report)

Deep learning has demonstrated an exceptional ability to solve complex tasks (an engineering success); however, it has done so at the expense of the ability to generate new knowledge (a scientific failure). We propose an alternative framework—entitled Deep Symbolic Regression (DSR)—in which artificial neural networks (NNs) rapidly generate hypotheses about physical relationships among inputs. This framework bypasses the need to interpret an NN altogether, while still leveraging the representational power of deep learning. The resulting models are tractable mathematical expressions, which are inherently and readily human interpretable and can provide insights into underlying physical phenomena. Further, we fold this methodology into the scientific process by allowing the scientist to directly integrate a priori knowledge and beliefs to accelerate learning. We demonstrate this methodology on symbolic regression—the problem of rediscovering underlying expressions describing a dataset—and achieve state-of-the-art performance across a wide variety of symbolic regression problems. Further, we generalize our DSR framework to apply to the more general class of symbolic optimization problems, in which one seeks to optimize a sequence of symbols or “tokens” under a black-box reward function. Examples of other symbolic optimization problems include neural architecture search and computational antibody design. Our generalized tool, Deep Symbolic Optimization (DSO), has been demonstrated on the task of learning symbolic control policies for reinforcement learning environments, and has been adopted as an enabling capability for computational antibody design.

97 MATHEMATICS AND COMPUTING↗

Neural Networks and Graph Algorithms with Next-Generation Processors

The use of graphical processors for distributed computation revolutionized the field of high performance scientific computing. As the Moore's Law era of computing draws to a close, the development of non-Von Neumann systems: neuromorphic processing units, and quantum annealers; again are redefining new territory for computational methods. While these technologies are still in their nascent stages, we discuss their potential to advance computing in two domains: machine learning, and solving constraint satisfaction problems. Each of these processors utilize fundamentally different theoretical models of computation. This raises questions about how to best use them in the design and implementation of applications. While many processors are being developed with a specific domain target, the ubiquity of spin-glass models and neural networks provides an avenue for multi-functional applications. This provides hints at the future infrastructure needed to integrate many next-generation processing units into conventional high-performance computing systems.

Hamilton, Kathleen↗

Sivers extraction with Neural Network

Pseudo-data with simulated experimental errors can be generated to train an ensemble of Artificial Neural Networks (ANN) implemented on a regression to extract Transverse Momentum-dependent Distributions (TMDs). A preliminary analysis is presented on the reliability in extraction of the Sivers function imposed in the pseudo-data given the bounds on the experimental errors, data sparsity, and complexity of phase-space.

Fernando, Ishara↗

Hierarchical modeling of molecular energies using a deep neural network

In this work, we introduce the Hierarchically Interacting Particle Neural Network (HIP-NN) to model molecular properties from datasets of quantum calculations. Inspired by a many-body expansion, HIP-NN decomposes properties, such as energy, as a sum over hierarchical terms. These terms are generated from a neural network—a composition of many nonlinear transformations—acting on a representation of the molecule. HIP-NN achieves the state-of-the-art performance on a dataset of 131k ground state organic molecules and predicts energies with 0.26 kcal/mol mean absolute error. Finally, with minimal tuning, our model is also competitive on a dataset of molecular dynamics trajectories. In addition to enabling accurate energy predictions, the hierarchical structure of HIP-NN helps to identify regions of model uncertainty.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

CrossSim Inference Manual v2.0

Neural networks are largely based on matrix computations. During forward inference, the most heavily used compute kernel is the matrix-vector multiplication (MVM): $W \vec{x} $. Inference is a first frontier for the deployment of next-generation hardware for neural network applications, as it is more readily deployed in edge devices, such as mobile devices or embedded processors with size, weight, and power constraints. Inference is also easier to implement in analog systems than training, which has more stringent device requirements. The main processing kernel used during inference is the MVM.

97 MATHEMATICS AND COMPUTING↗

AI-Powered Knowledge Graphs for Neuromorphic and Energy-Efficient Computing

The surge in scientific literature obscures breakthroughs and hinders the discovery of new research paths. We propose an artificial intelligence (AI) powered framework using large language models (LLMs) and knowledge graphs (KGs) to automate parts of scientific discovery, focusing on energy-efficient AI circuits. Our hybrid approach combines LLMs, structured data, and ontology-based reasoning to construct a comprehensive knowledge graph that integrates insights across computational neuroscience, spiking neuron models, learning rules, architectural motifs, and neuromorphic device technologies. This multi-domain representation enables the generation of hypotheses that connect biological function with implementable, energy-efficient hardware architectures. Using KG embeddings and graph neural networks, the framework generates hypotheses for novel circuits, validates them through optimization on exascale HPC systems, and with tools like SuperNeuro and Fugu, the most promising designs will be prototyped in hardware. This open-source system aims to accelerate discoveries and bridging neuroscience with hardware innovation, drive collaboration, and unlock new opportunities in low-power AI computing.

Gautam, Ashish [ORNL]↗

Full-field imaging learning machine (FILM)

A method of determining dynamic properties of a structure (linear or nonlinear) includes receiving spatio-temporal inputs, generating mode shapes and modal components corresponding to the spatio-temporal inputs using a trained deep complexity coding artificial neural network, and subsequently generating the dynamic properties by analyzing each modal component using a trained learning machine. A computing system for non-contact determination of dynamic properties of a structure includes a camera, a processor, and a memory including computer-executable instructions. When the instructions are executed, the system is caused to receive spatio-temporal image data, decompose the spatio-temporal image data into constituent manifold components using an autoencoder, and analyze the constituent manifold components using a trained learning machine to determine the dynamic properties.

Yang, Yongchao↗