Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “neural operator”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 235 records · Page 13

Deep nonparametric estimation of operators between infinite dimensional spaces

Learning operators between infinitely dimensional spaces is an important learning task arising in machine learning, imaging science, mathematical modeling and simulations, etc. This paper studies the nonparametric estimation of Lipschitz operators using deep neural networks. Non-asymptotic upper bounds are derived for the generalization error of the empirical risk minimizer over a properly chosen network class. Under the assumption that the target operator exhibits a low dimensional structure, our error bounds decay as the training sample size increases, with an attractive fast rate depending on the intrinsic dimension in our estimation. Our assumptions cover most scenarios in real applications and our results give rise to fast rates by exploiting low dimensional structures of data in operator estimation. We also investigate the influence of network structures (e.g., network width, depth, and sparsity) on the generalization error of the neural network estimator and propose a general suggestion on the choice of network structures to maximize the learning efficiency quantitatively.

97 MATHEMATICS AND COMPUTING↗

Deep Koopman learning of nonlinear time-varying systems

Here this paper presents a data-driven approach to approximate the dynamics of a nonlinear time-varying system (NTVS) by a linear time-varying system (LTVS), which results from the Koopman operator and deep neural networks. Analysis of the approximation error between states of the NTVS and the resulting LTVS is presented. Simulations on a representative NTVS show that the proposed method achieves small approximation errors, even when the system changes rapidly. Furthermore, simulations in an example of quadcopters demonstrate the computational efficiency of the proposed approach.

97 MATHEMATICS AND COMPUTING↗

A dimensionality reduction algorithm for mapping tokamak operational regimes using a variational autoencoder (VAE) neural network

Abstract A variational autoencoder (VAE) is a type of unsupervised neural network which is able to learn meaningful data representations in a reduced dimensional space. We present an application of VAE in identifying the operational stability boundary of tokamak plasma discharges. This model was implemented using a dataset of over 3000 discharges from the high beta tokamak-extended pulse (HBT-EP) device. We found the VAE model to be capable of forming a continuous low-dimensional operational space map and identifying the operational boundaries using a specified warning time window. By projecting the operational parameters onto the same reduced space, this provides an intuitive way for the machine operator or an automated control system to perform disruption avoidance using a relevant control actuator as a discharge approaches a boundary. Pre-programmed GPU control experiments were conducted to demonstrate this control technique using HBT-EP’s saddle control coils as a horizontal position actuator, showing the ability to avoid the oncoming disruptive event and extend the duration of the discharge.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Deep Nonparametric Estimation of Operators between Infinite Dimensional Spaces

Learning operators between infinitely dimensional spaces is an important learning task arising in machine learning, imaging science, mathematical modeling and simulations, etc. This paper studies the nonparametric estimation of Lipschitz operators using deep neural networks. Non-asymptotic upper bounds are derived for the generalization error of the empirical risk minimizer over a properly chosen network class. Under the assumption that the target operator exhibits a low dimensional structure, our error bounds decay as the training sample size increases, with an attractive fast rate depending on the intrinsic dimension in our estimation. Our assumptions cover most scenarios in real applications and our results give rise to fast rates by exploiting low dimensional structures of data in operator estimation. We also investigate the influence of network structures (e.g., network width, depth, and sparsity) on the generalization error of the neural network estimator and propose a general suggestion on the choice of network structures to maximize the learning efficiency quantitatively.

97 MATHEMATICS AND COMPUTING↗

Self-regulating power management for a neural network system

A neural network runs a known input data set using an error free power setting and using an error prone power setting. The differences in the outputs of the neural network using the two different power settings determine a high level error rate associated with the output of the neural network using the error prone power setting. If the high level error rate is excessive, the error prone power setting is adjusted to reduce errors by changing voltage and/or clock frequency utilized by the neural network system. If the high level error rate is within bounds, the error prone power setting can remain allowing the neural network to operate with an acceptable error tolerance and improved efficiency. The error tolerance can be specified by the neural network application.

Kegel, Andrew G.↗

Self-regulating power management for a neural network system

A neural network runs a known input data set using an error free power setting and using an error prone power setting. The differences in the outputs of the neural network using the two different power settings determine a high level error rate associated with the output of the neural network using the error prone power setting. If the high level error rate is excessive, the error prone power setting is adjusted to reduce errors by changing voltage and/or clock frequency utilized by the neural network system. If the high level error rate is within bounds, the error prone power setting can remain allowing the neural network to operate with an acceptable error tolerance and improved efficiency. The error tolerance can be specified by the neural network application.

Kegel, Andrew G.↗

High-order neural models for error correcting code

The decoding and error-correction of data transmitted over a noisy channel is in principle equivalent to the operation of a neural network performing as a content-addressable memory. For a successful application, however, the neural network has to be capable of storing arbitrary words, and it has to be guaranteed that the stored words represent the only stable attractors of the memory. This paper presents a novel high-order neural network architecture that has these characteristics. The analog nature of the network can be used to perform soft-decision decoding with any block code. The performance in terms of postdecoding bit error rate versus SNR is demonstrated for two exemplary block codes. The comparison with a conventional decoding algorithm for a (15,5) cyclic redundancy code shows, for example, that the bit error rate at 7dB SNR can be decreased by two orders of magnitude.

Jeffries, Clark↗

Generalized Quantum Convolution for Multidimensional Data

The convolution operation plays a vital role in a wide range of critical algorithms across various domains, such as digital image processing, convolutional neural networks, and quantum machine learning. In existing implementations, particularly in quantum neural networks, convolution operations are usually approximated by the application of filters with data strides that are equal to the filter window sizes. One challenge with these implementations is preserving the spatial and temporal localities of the input features, specifically for data with higher dimensions. In addition, the deep circuits required to perform quantum convolution with a unity stride, especially for multidimensional data, increase the risk of violating decoherence constraints. In this work, we propose depth-optimized circuits for performing generalized multidimensional quantum convolution operations with unity stride targeting applications that process data with high dimensions, such as hyperspectral imagery and remote sensing. We experimentally evaluate and demonstrate the applicability of the proposed techniques by using real-world, high-resolution, multidimensional image data on a state-of-the-art quantum simulator from IBM Quantum.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Oil Reservoir Properties Estimation Using Neutal Networks

This paper investigates the applicability as well as the accuracy of artificial neural networks for estimating specific parameters that describe reservoir properties based on seismic data. Our approach relies on JPL's adjoint operators general purpose neural network code to determine the best suited architecture. We believe that results presented in this work demonstrate that artificial neural networks produce surprisingly accurate estimates of the reservoir parameters.

oil reservoir neural networks geoscience data oil ↗

On the Training and Generalization of Deep Operator Networks

Here, we present a novel training method for deep operator networks (DeepONets), one of the most popular neural network models for operators. DeepONets are constructed by two subnetworks, namely the branch and trunk networks. Typically, the two subnetworks are trained simultaneously, which amounts to solving a complex optimization problem in a high dimensional space. In addition, the nonconvex and nonlinear nature makes training very challenging. To tackle such a challenge, we propose a two-step training method that trains the trunk network first and then sequentially trains the branch network. The core mechanism is motivated by the divide-and-conquer paradigm and is the decomposition of the entire complex training task into two subtasks with reduced complexity. Therein the Gram–Schmidt orthonormalization process is introduced which significantly improves stability and generalization ability. On the theoretical side, we establish a generalization error estimate in terms of the number of training data, the width of DeepONets, and the number of input and output sensors. Numerical examples are presented to demonstrate the effectiveness of the two-step training method, including Darcy flow in heterogeneous porous media.

deep operator networks↗

Trustworthy Physics-Informed Deep Learning for Predictive Scientific Computing

This project has developed powerful trustworthy physics-informed deep learning (TPiDL) models and methods to fundamentally enhance the scale and power of computational modeling in the scientific and engineering domains. Deep learning (DL) has radically advanced the state-of-the-art in machine learning, computer vision, natural language processing, and also scientific computing. Nevertheless, progress has been driven almost entirely by empirical observations, hacks, and tricks. Under the support of this project, the graph operator learning tools and advanced trustworthy physical informed neural networks have been developed. In addition, stochastic gradient replica-exchange Markov Chain Monte Carlo (MCMC) sampling algorithms have been designed to quantify the uncertainties and speed up the training of large-scale neural networks.

97 MATHEMATICS AND COMPUTING↗

B-DeepONet: An enhanced Bayesian DeepONet for solving noisy parametric PDEs using accelerated replica exchange SGLD

Here, the Deep Operator Network (DeepONet) is a neural network architecture used to approximate operators, including the solution operator of parametric PDEs. DeepONets have shown remarkable approximation ability. However, the performance of DeepONets deteriorates when the training data is polluted with noise, a scenario that occurs in practice. To handle noisy data, we propose a Bayesian DeepONet based on replica exchange Langevin diffusion (reLD). Replica exchange uses two particles. The first particle trains a DeepONet to exploit the loss landscape and make predictions. The other particle trains a different DeepONet to explore the loss landscape and escape local minima via swapping. Compared to DeepONets trained with state-of-the-art gradient-based algorithms (e.g., Adam), the proposed Bayesian DeepONet greatly improves the training convergence for noisy scenarios and accurately estimates the uncertainty. To further reduce the high computational cost of the reLD training of DeepONets, we propose (1) an accelerated training framework that exploits the DeepONet's architecture to reduce its computational cost up to 25% without compromising performance and (2) a transfer learning strategy that accelerates training DeepONets for PDEs with different parameter values. Finally, we illustrate the effectiveness of the proposed Bayesian DeepONet using four parametric PDE problems.

97 MATHEMATICS AND COMPUTING↗

Transfer learning for analysis of collective and non-collective Thomson scattering spectra

Thomson scattering (TS) diagnostics provide reliable, minimally perturbative measurements of fundamental plasma parameters, such as electron density (⁠n e ) and electron temperature (⁠T e ⁠). Deep neural networks can provide accurate estimates of ⁠n e and T e when conventional fitting algorithms may fail, such as when TS spectra are dominated by noise, or when fast analysis is required for real-time operation. Although deep neural networks typically require large training sets, transfer learning can improve model performance on a target task with limited data by leveraging pre-trained models from related source tasks, where select hidden layers are further trained using target data. We present five architecturally diverse deep neural networks, pre-trained on synthetic TS data and adapted for experimentally measured TS data, to evaluate the efficacy of transfer learning in estimating n e and T e in both the collective and non-collective scattering regimes. We evaluate errors in n e and T e estimates as a function of training set size for models trained with and without transfer learning, and we observe decreases in model error from transfer learning when the training set contains ≲ 200 experimentally measured spectra.

Artificial neural networks↗

The Effects of Spaceflight and a Spaceflight Analog on Neurocognitive Perfonnance: Extent, Longevity, and Neural Bases

We are conducting ongoing experiments in which we are performing structural and functional magnetic resonance brain imaging to identify the relationships between changes in neurocognitive function and neural structural alterations following a six month International Space Station mission and following 70 days exposure to a spaceflight analog, head down tilt bedrest. Our central hypothesis is that measures of brain structure, function, and network integrity will change from pre to post intervention (spaceflight, bedrest). Moreover, we predict that these changes will correlate with indices of cognitive, sensory, and motor function in a neuroanatomically selective fashion. Our interdisciplinary approach utilizes cutting edge neuroimaging techniques and a broad ranging battery of sensory, motor, and cognitive assessments that will be conducted pre flight, during flight, and post flight to investigate potential neuroplastic and maladaptive brain changes in crewmembers following long-duration spaceflight. Success in this endeavor would 1) result in identification of the underlying neural mechanisms and operational risks of spaceflight-induced changes in behavior, and 2) identify whether a return to normative behavioral function following re-adaptation to Earth's gravitational environment is associated with a restitution of brain structure and function or instead is supported by substitution with compensatory brain processes. With the bedrest study, we will be able to determine the neural and neurocognitive effects of extended duration unloading, reduced sensory inputs, and increased cephalic fluid distribution. This will enable us to parse out the multiple mechanisms contributing to any spaceflight-induced neural structural and behavioral changes that we observe in the flight study. In this presentation I will discuss preliminary results from six participants who have undergone the bed rest protocol. These individuals show decrements in balance and functional mobility, and alterations in brain structure and function, in association with extended bed rest.

Seidler, R. D.↗

Operator inference with roll outs for learning reduced models from scarce and low-quality data

Data-driven modeling has become a key building block in computational science and engineering. However, data that are available in science and engineering are typically scarce, often polluted with noise and affected by measurement errors and other perturbations, which makes learning the dynamics of systems challenging. Here, in this work, we propose to combine data-driven modeling via operator inference with the dynamic training via roll outs of neural ordinary differential equations. Operator inference with roll outs inherits interpretability, scalability, and structure preservation of traditional operator inference while leveraging the dynamic training via roll outs over multiple time steps to increase stability and robustness for learning from low-quality and noisy data. Numerical experiments with data describing shallow water waves and surface quasi-geostrophic dynamics demonstrate that operator inference with roll outs provides predictive models from training trajectories even if data are sampled sparsely in time and polluted with noise of up to 10%.

97 MATHEMATICS AND COMPUTING↗

Particle hit clustering and identification using point set transformers in liquid argon time projection chambers

Liquid argon time projection chambers are often used in neutrino physics and dark-matter searches because of their high spatial resolution. The images generated by these detectors are extremely sparse, as the energy values detected by most of the detector are equal to 0, meaning that despite their high resolution, most of the detector is unused in a particular interaction. Instead of representing all of the empty detections, the interaction is usually stored as a sparse matrix, a list of detection locations paired with their energy values. Traditional machine learning methods that have been applied to particle reconstruction such as convolutional neural networks (CNNs), however, cannot operate over data stored in this way and therefore must have the matrix fully instantiated as a dense matrix. Operating on dense matrices requires a lot of memory and computation time, in contrast to directly operating on the sparse matrix. We propose a machine learning model using a point set neural network that operates over a sparse matrix, greatly improving both processing speed and accuracy over methods that instantiate the dense matrix, as well as over other methods that operate over sparse matrices. Compared to competing state-of-the-art methods, our method improves classification performance by 14%, segmentation performance by more than 22%, while taking 80% less time and using 66% less memory. Compared to state-of-the-art CNN methods, our method improves classification performance by more than 86%, segmentation performance by more than 71%, while reducing runtime by 91% and reducing memory usage by 61%.

calibration and fitting methods↗

Structure-Informed Graph Learning of Networked Dependencies for Online Prediction of Power System Transient Dynamics

Online transient analysis plays an increasingly important role in dynamic power grids as the renewable generation continues growing. Traditional numerical methods for transient analysis not only are computationally intensive but also require precise contingency information as input, and therefore, are not suitable for online applications. Existing online transient assessment studies focus on the determination of post-contingency system stability or stability margin. Here, this paper develops a novel graph-learning framework, Deep-learning Neural Representation or DNR, for online prediction, of the time-series trajectories of the system states using initial system responses that can be measured by phasor measurement units (PMUs). The proposed DNR framework consists of two sequential modules: a Network Constructor that captures network dependencies among generators, and a Dynamics Predictor that predicts the system trajectories. The key to improved prediction performance is the introduction of the spatio-temporal message-passing operations into graph neural networks with structural knowledge. Its effectiveness and scalability are validated through comparative studies, demonstrating the prediction performance under different contingency scenarios for systems of different sizes. This framework provides a solution to online predicting post-fault system dynamics based on real-time PMU measurements. Additionally, it can also be applied to facilitate the offline transient simulation without simulating the entire trajectories.

24 POWER TRANSMISSION AND DISTRIBUTION↗