Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “neural operator”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 325 records · Page 18

Evaluating the Use of Foundational Chemical Language Models in Multimodal Graph Fusion

Rapid and accurate prediction of the physicochemical properties of molecules given their structures remains a key challenge in cheminformatics. Machine learning approaches offer high-throughput options, but the optimality of inductive biases and data representations are up for debate. For example, BERT-based masked language models (MLMs) can be trained in a self-supervised way on hundreds of millions to billions of readily available SMILES strings. Another option is graph neural networks (GNNs), which can operate directly on molecular structures. Yet, generating accurate molecular geometry is computationally expensive, leading to a relative scarcity in data compared to SMILES strings. It is attractive to combine these two paradigms by pre-training an LM on a large corpus of SMILES strings and embedding these representation into a geometric graph neural network. Despite the promise of such an approach, and contrary to previous studies, we find mixed results with the combination of the LMs and GNNs on several molecule datasets. In particular, we found evidence for improvement on the FreeSolv and QM7 benchmarks, but degraded performance on the ESOL, LIPO and QM9 datasets compared to a GNN baseline.

Francel, Collin [University of Alabama]↗

Development of a hybrid neural network and transfer learning model for optimized ICP-MS/MS operation

Correct function and calibration of instrumentation is a crucial assumption for any scientific experiment. One such instrument, tandem inductively coupled plasma mass spectrometer (ICP-MS/MS), has in-depth calibration settings that range across 30+ different parameters, making it difficult to determine optimal conditions without expertise and some degree of trial and error. Often, these settings are hand-tuned, a time-intensive process prone to local maxima and human error. While some automation is available, the automation also may favor local optimizations over a global optimum. In addition to these difficulties, day to day instrument variability can further complicate the calibration process. We propose a solution to this problem as a machine learning (ML) algorithm that learns how each parameter helps determine the calibration sensitivity across several elements, and re-weights parameters over time as instrument variability changes (e.g., a global neural network (NN) with a time-dependent transfer learning (TL) component). This model would be able to generate a surface of predicted calibration sensitivities and their respective parameters, and a simple multivariate algorithm would be able to pull out the optimum results with the settings associated with them. Here-in, we describe our initial findings in working towards this goal, including data extraction from historical files, exploratory data analysis, and some initial model building to better describe the data and the feasibility of our goal.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Reducing Operator Complexity of Galerkin Coarse-grid Operators with Machine Learning

Here, we propose a data-driven and machine-learning-based approach to compute non-Galerkin coarse-grid operators in multigrid (MG) methods, addressing the well-known issue of increasing operator complexity. Guided by the MG theory on spectrally equivalent coarse-grid operators, we have developed novel machine learning algorithms that utilize neural networks combined with smooth test vectors from multigrid eigenvalue problems. The proposed method demonstrates promise in reducing the complexity of coarse-grid operators while maintaining overall MG convergence for solving parametric partial differential equation problems. Numerical experiments on anisotropic rotated Laplacian and linear elasticity problems are provided to showcase the performance and comparison with existing methods for computing non-Galerkin coarse-grid operators.

97 MATHEMATICS AND COMPUTING↗

Short-term load forecasting using generalized regression and probabilistic neural networks in the electricity market

For the economic and secure operation of power systems, a precise short-term load forecasting technique is essential. Modern load forecasting techniques - especially artificial neural network methods - are particularly attractive, as they have the ability to handle the non-linear relationships between load, weather temperature, and the factors affecting them directly. A test of two different ANN models on data from Australia's Victoria market is promising. (author)

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Data-driven selection of stiff chemistry ODE solver in operator-splitting schemes

Most computational fluid dynamics simulations of practical combustion applications employ operator-splitting schemes, where chemistry and transport are separated and integrated with distinct numerical methods. The changes in composition due to chemistry are evaluated by solving ordinary differential equations (ODE) in each cell of the computational domain, which typically dominates the computational cost when detailed chemistry is considered. In this work, a data-driven approach for the selection of chemistry ODE solvers in operator-splitting schemes is presented. Neural networks are used to predict the ODE solvers CPU times and errors for a given thermochemical state. This allows the selection of an optimal ODE solver on a cell-by-cell, timestep-by-timestep basis. The models are trained using a wide set of thermochemical states generated through partially-stirred reactors and flames simulations. The methodology is validated by quantifying the prediction errors, the classification accuracy, and the computational speedup. The model predicts the optimal ODE solver for 70 to 95% of the validation cases and decreases the computional cost by a factor of 3 or more. The generalizability of the methodology to different chemical mechanisms and different fuels is assessed and it is shown that the model’s performance is only slightly degraded and its applicability is significantly enhanced if the inputs to the neural networks are restricted to a small set of thermochemical state variables present in most chemical mechanisms. In conclusion, the models are used in an homogeneous reactor case and a multi-dimensional CFD simulation of a diesel spray at high pressure where a speedup of more than 3 is achieved.

42 ENGINEERING↗

WE-AB-204-06: Pseudo-CT Generation Using Undersampled, Single-Acquisition UTE-MDixon and Direct-Mapping Artificial Neural Networks for MR-Based Attenuation Correction and Radiation Therapy Planning

Purpose: Emerging technologies such as dedicated PET/MRI and MR-therapy systems require robust and clinically practical methods for determining photon attenuation. Herein, we propose using novel MR acquisition methods and processing for the generation of pseudo-CTs. Methods: A single acquisition, 190-second UTE-mDixon sequence with 25% (angular) sampling density and 3D radial readout was performed on nine volunteers. Three water-filled tubes were placed in the FOV for trajectory-delay correction. The MR data were reconstructed to generate three primitive images acquired at TEs of 0.1, 1.5 and 2.8 ms. In addition, three derived MR images were generated, i.e. two-point Dixon water/fat separation and R2* (1/T2*) map. Furthermore, two spatial features, i.e. local binary pattern (S-1) and relative spatial coordinates (S-2), were incorporated. A direct-mapping operator was generated using Artificial Neural Networks (ANNs) for transforming the MR features to a pseudo-CT. CT images served as the training data and, using a leave-one-out method, for performance evaluation using mean prediction deviation (MPD), mean absolute prediction deviation (MAPD), and correlation coefficient (R). Results: The errors between measured CT and pseudo-CT declined dramatically when the spatial features, i.e. S-1 and S-2, were included. The MPD, MAPD, and R were, respectively, 5±57 HU, 141±41 HU, and 0.815±0.066 for results generated by the ANN trained without the spatial features and were 32±26 HU, 115±18 HU, and 0.869±0.035 with the spatial features. The estimation errors of the pseudo-CT were smaller when both the S-1 and S-2 were used together than when either the S-1 or the S-2 was used. Pseudo-CT generation (256×256×256 voxels) by ANN took < 0.5 s using a computer having an Intel i7 3.4GHz CPU and 16 GB RAM. Conclusion: The proposed direct-mapping ANN approach is a technically accurate, clinically practical method for pseudo-CT generation and can potentially help improve the accuracy of MR-AC and MR-RTP applications. Please note that the project was completed with partial funding from the Ohio Department of Development grant TECH 11-063 and a sponsored research agreement with Philips Healthcare that is managed by Case Western Reserve University. As noted in the affiliations, some of the authors are Philips employees.

60 APPLIED LIFE SCIENCES↗

WellPINN: Accurate Well Representation for Transient Fluid Pressure Diffusion in Subsurface Reservoirs With Physics‐Informed Neural Networks

Accurate representation of pumping wells is essential for reliable reservoir characterization and simulation of operational scenarios in subsurface flow models. Physics-informed neural networks (PINNs) are emerging as a promising alternative to numerical models for reservoir modeling, offering seamless integration of monitoring data and governing physical equations. However, existing PINN-based studies face major challenges in capturing fluid pressure near wells when using a source/sink term, particularly during the early stages after pumping begins. We address this problem by introducing WellPINN, a workflow in which an initially trained PINN infers fluid pressure across the entire reservoir domain using a large equivalent well radius. This initial PINN solution is then locally refined around the well by a set of subdomain PINNs that are trained for smaller equivalent well radii. Continuity across these subdomain interfaces as well as at the initial condition is ensured by hard-constraining each PINN on its subdomain boundary. Our results demonstrate WellPINN as the first workflow of its kind to focus on accurate inference of fluid pressure from pumping rates throughout the entire injection period, significantly advancing the potential of PINNs for inverse modeling and operational scenario simulations. All data and code for this paper are openly available at https://doi.org/10.20350/DIGITALCSIC/17260.

58 GEOSCIENCES↗

Predicting nonequilibrium Green’s function dynamics and photoemission spectra via nonlinear integral operator learning

Understanding the dynamics of nonequilibrium quantum many-body systems is an important research topic in a wide range of fields across condensed matter physics, quantum optics, and high-energy physics. However, numerical studies of large-scale nonequilibrium phenomena in realistic materials face serious challenges due to intrinsic high-dimensionality of quantum many-body problems and the absence of time-invariance. The nonequilibrium properties of many-body systems can be described by the dynamics of the correlator, or the Green's function of the system, whose time evolution is given by a high-dimensional system of integro-differential equations, known as the Kadanoff–Baym equations (KBEs). The time-convolution term in KBEs, which needs to be recalculated at each time step, makes it difficult to perform long-time numerical simulation. In this paper, we develop an operator-learning framework based on recurrent neural networks (RNNs) to address this challenge. We utilize RNNs to learn the nonlinear mapping between Green's functions and convolution integrals in KBEs. By using the learned operators as a surrogate model in the KBE solver, we obtain a general machine-learning scheme for predicting the dynamics of nonequilibrium Green's functions. Besides significant savings per each time step, the new methodology reduces the temporal computational complexity from $O(N_t^3)$ to $O(N_t)$ where N t is the number of steps taken in a simulation, thereby making it possible to study large many-body problems which are currently infeasible with conventional KBE solvers. Through various numerical examples, we demonstrate the effectiveness of the operator-learning based approach in providing accurate predictions of physical observables such as the reduced density matrix and time-resolved photoemission spectra. Moreover, our framework exhibits clear numerical convergence and can be easily parallelized, thereby facilitating many possible further developments and applications.

97 MATHEMATICS AND COMPUTING↗

The efficacy of using human myoelectric signals to control the limbs of robots in space

This project was designed to investigate the usefulness of the myoelectric signal as a control in robotics applications. More specifically, the neural patterns associated with human arm and hand actions were studied to determine the efficacy of using these myoelectric signals to control the manipulator arm of a robot. The advantage of this approach to robotic control was the use of well-defined and well-practiced neural patterns already available to the system, as opposed to requiring the human operator to learn new tasks and establish new neural patterns in learning to control a joystick or mechanical coupling device.

Clark, Jane E.↗

Improving Trust in Deep Neural Networks with Nearest Neighbors

Deep neural networks are used increasingly for perception and decision-making in UAVs. For example, they can be used to recognize objects from images and decide what actions the vehicle should take. While deep neural networks can perform very well at complex tasks, their decisions may be unintuitive to a human operator. When a human disagrees with a neural network prediction, due to the black box nature of deep neural networks, it can be unclear whether the system knows something the human does not or whether the system is malfunctioning. This uncertainty is problematic when it comes to ensuring safety. As a result, it is important to develop technologies for explaining neural network decisions for trust and safety. This paper explores a modification to the deep neural network classification layer to produce both a predicted label and an explanation to support its prediction. Specifically, at test time, we replace the final output layer of the neural network classifier by a k-nearest neighbor classifier. The nearest neighbor classifier produces 1) a predicted label through voting and 2) the nearest neighbors involved in the prediction, which represent the most similar examples from the training dataset. Because prediction and explanation are derived from the same underlying process, this approach guarantees that the explanations are always relevant to the predictions. We demonstrate the approach on a convolutional neural network for a UAV image classification task. We perform experiments using a forest trail image dataset and show empirically that the hybrid classifier can produce intuitive explanations without loss of predictive performance compared to the original neural network. We also show how the approach can be used to help identify potential issues in the network and training process.

Lee, Ritchie↗

Convolutional neural networks for intra-hour solar forecasting based on sky image sequences

Accurate and timely solar forecasts play an increasingly critical role in power systems. Compared to longer forecasting timescales, very short-term solar forecasting has lagged behind in both research and practice. In this paper, we propose deep convolutional neural networks (CNNs) to provide operational intra-hour (10-minute-ahead to 60-minute-ahead) solar forecasts. We develop two CNN structures inspired by a widely-used CNN architecture. The CNNs are tailored to our solar forecasting regression tasks and rely solely on sky image sequences. Case studies based on six years of data (over 150,000 data points) demonstrate that the best CNN model has forecast skill scores of 20%-39% over the naive persistence of cloudiness benchmark, even at these very short timescales. The CNNs also have consistently superior performance when compared to shallow machine learning models with meteorological predictors, where the improvement averages around 7%. The sensitivity analyses show that the sky image length, resolution, and weather conditions have impacts on the deep learning model accuracy. In our intra-hour problem with specific setups, two sky images with a 10-minute 128 x 128 resolution yield the most accurate forecasts. Current limitations, future work, and deployment challenges and solutions are also discussed.

14 SOLAR ENERGY↗

Attentional Ptycho-Tomography (APT) for three-dimensional nanoscale X-ray imaging with minimal data acquisition and computation time

Abstract Noninvasive X-ray imaging of nanoscale three-dimensional objects, such as integrated circuits (ICs), generally requires two types of scanning: ptychographic, which is translational and returns estimates of the complex electromagnetic field through the IC; combined with a tomographic scan, which collects these complex field projections from multiple angles. Here, we present Attentional Ptycho-Tomography (APT), an approach to drastically reduce the amount of angular scanning, and thus the total acquisition time. APT is machine learning-based, utilizing axial self-Attention for Ptycho-Tomographic reconstruction. APT is trained to obtain accurate reconstructions of the ICs, despite the incompleteness of the measurements. The training process includes regularizing priors in the form of typical patterns found in IC interiors, and the physics of X-ray propagation through the IC. We show that APT with ×12 reduced angles achieves fidelity comparable to the gold standard Simultaneous Algebraic Reconstruction Technique (SART) with the original set of angles. When using the same set of reduced angles, then APT also outperforms Filtered Back Projection (FBP), Simultaneous Iterative Reconstruction Technique (SIRT) and SART. The time needed to compute the reconstruction is also reduced, because the trained neural network is a forward operation, unlike the iterative nature of these alternatives. Our experiments show that, without loss in quality, for a 4.48 × 93.2 × 3.92 µm 3 IC (≃6 × 10 8 voxels), APT reduces the total data acquisition and computation time from 67.96 h to 38 min. We expect our physics-assisted and attention-utilizing machine learning framework to be applicable to other branches of nanoscale imaging, including materials science and biological imaging.

47 OTHER INSTRUMENTATION↗

From disorganized data to emergent dynamic models: Questionnaires to partial differential equations

Starting with sets of disorganized observations of spatially varying and temporally evolving systems, obtained at different (also disorganized) sets of parameters, we demonstrate the data-driven derivation of parameter dependent, evolutionary partial differential equation (PDE) models capable of generating the data. This tensor type of data is reminiscent of shuffled (multidimensional) puzzle tiles. The independent variables for the evolution equations (their “space” and “time”) as well as their effective parameters are all emergent , i.e. determined in a data-driven way from our disorganized observations of behavior in them. We use a diffusion map based questionnaire approach to build a smooth parametrization of our emergent space/time/parameter space for the data. This approach iteratively processes the data by successively observing them on the “space,” the “time” and the “parameter” axes of a tensor. Once the data become organized, we use machine learning (here, neural networks) to approximate the operators governing the evolution equations in this emergent space. Our illustrative examples are based (i) on a simple advection–diffusion model; (ii) on a previously developed vertex-plus-signaling model of Drosophila embryonic development; and (iii) on two complex dynamic network models (one neuronal and one coupled oscillator model) for which no obvious smooth embedding geometry is known a priori. This allows us to discuss features of the process like symmetry breaking, translational invariance, and autonomousness of the emergent PDE model, as well as its interpretability.

generative models↗

Easy-plane spin Hall nano-oscillators as spiking neurons for neuromorphic computing

Here we show analytically using a macrospin approximation that easy-plane spin Hall nano-oscillators excited by a spin current polarized perpendicularly to the easy plane have phase dynamics analogous to that of Josephson junctions. Similarly to Josephson junctions, they can reproduce the spiking behavior of biological neurons that is appropriate for neuromorphic computing. To take advantage of typical spin-orbit torques, we use a nanoconstriction geometry, in which the magnetostatic interaction and magnetocrystalline anisotropy are tuned to create an easy plane that includes the interface normal direction. We perform micromagnetic simulations of such oscillators realized in this geometry and show that the easy-plane spiking dynamics is preserved in this experimentally feasible architecture. Finally we simulate two elementary neural network blocks that implement operations essential for neuromorphic computing. First, we show that output spikes energies from two neurons can be summed and injected into a following layer neuron and second, we demonstrate that outputs can be multiplied by synaptic weights implemented by locally modifying the anisotropy.

36 MATERIALS SCIENCE↗

Neural Networks Of VLSI Components

Concept for design of electronic neural network calls for assembly of very-large-scale integrated (VLSI) circuits of few standard types. Each VLSI chip, which contains both analog and digital circuitry, used in modular or "building-block" fashion by interconnecting it in any of variety of ways with other chips. Feedforward neural network in typical situation operates under control of host computer and receives inputs from, and sends outputs to, other equipment.

Eberhardt, Silvio P.↗

An all digital implementation of a modified Hamming net for video compression with prediction and quantization circuits

The hardware and algorithms used to vector quantize (VQ) predicted pixel intensity differences for real-time video compression are described. The hardware is designed for rapid vector quantization performance, which entails the development of application-specific associative memory circuits. A modified DPCM algorithm is originally examined to determine how neural circuitry could enhance its operation. It was determined that quantization and encoding could be improved by consolidating these two functions into one, and by increasing the amount of information (i.e. number of pixels) quantized at a time. The result is a predictive scheme that vector quantizes differential values. Some of the disadvantages of VQ algorithms are solved using associative memories. The video compression algorithm and the associative memory design are described.

Kaul, Richard↗

A Diagnostic Approach for Electro-Mechanical Actuators in Aerospace Systems

Electro-mechanical actuators (EMA) are finding increasing use in aerospace applications, especially with the trend towards all all-electric aircraft and spacecraft designs. However, electro-mechanical actuators still lack the knowledge base accumulated for other fielded actuator types, particularly with regard to fault detection and characterization. This paper presents a thorough analysis of some of the critical failure modes documented for EMAs and describes experiments conducted on detecting and isolating a subset of them. The list of failures has been prepared through an extensive Failure Modes and Criticality Analysis (FMECA) reference, literature review, and accessible industry experience. Methods for data acquisition and validation of algorithms on EMA test stands are described. A variety of condition indicators were developed that enabled detection, identification, and isolation among the various fault modes. A diagnostic algorithm based on an artificial neural network is shown to operate successfully using these condition indicators and furthermore, robustness of these diagnostic routines to sensor faults is demonstrated by showing their ability to distinguish between them and component failures. The paper concludes with a roadmap leading from this effort towards developing successful prognostic algorithms for electromechanical actuators.

Balaban, Edward↗

Signal dispersion within a hippocampal neural network

A model network is described, representing two neural populations coupled so that one population is inhibited by activity it excites in the other. Parameters and operations within the model represent EPSPs, IPSPs, neural thresholds, conduction delays, background activity and spatial and temporal dispersion of signals passing from one population to the other. Simulations of single-shock and pulse-train driving of the network are presented for various parameter values. Neuronal events from 100 to 300 msec following stimulation are given special consideration in model calculations.

Horowitz, J. M.↗