Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “neural network architecture”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 307 records · Page 17
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Robustness of Deep Learning Classification to Adversarial Input on GPUs: Asynchronous Parallel Accumulation Is a Source of Vulnerability

The ability of machine learning (ML) classification models to resist small, targeted input perturbations—known as adversarial attacks—is a key measure of their safety and reliability. We show that floating-point non associativity (FPNA) coupled with asynchronous parallel programming on GPUs is sufficient to result in misclassification, without any perturbation to the input. Additionally, we show that this misclassification is particularly significant for inputs close to the decision boundary and that standard adversarial robustness results may be overestimated up to 4.6 when not considering machine-level details. We first study a linear classifier, before focusing on standard Graph Neural Network (GNN) architectures and datasets used in robustness assessments. We develop a novel black-box attack using Bayesian optimization to discover external workloads that can change the instruction scheduling which bias the output of reductions on GPUs and reliably lead to misclassification. Motivated by these results, we present a new learnable permutation (LP) gradient-based approach to learning floating-point operation orderings that lead to misclassifications. The LP approach provides a worst-case estimate in a computationally efficient manner, avoiding the need to run identical experiments tens of thousands of times over a potentially large set of possible GPU states or architectures. Finally, using instrumentation-based testing, we investigate parallel reduction ordering across different GPU architectures under external background workloads, when utilizing multi-GPU virtualization, and when applying power capping. Our results demonstrate that parallel reduction ordering varies significantly across architectures under the first two conditions, substantially increasing the search space required to fully test the effects of this parallel scheduler-based vulnerability. These results and the methods developed here can help to include machine-level considerations into adversarial robustness assessments, which can make a difference in safety and mission critical applications.

Shanmugavelu, Sanjif [Maxeler Technologies, a Groq↗
DOE OSTI2024

Equivariant, safe and sensitive — graph networks for new physics

This study introduces a novel Graph Neural Network (GNN) architecture that leverages infrared and collinear (IRC) safety and equivariance to enhance the analysis of collider data for Beyond the Standard Model (BSM) discoveries. By integrating equivariance in the rapidity-azimuth plane with IRC-safe principles, our model significantly reduces computational overhead while ensuring theoretical consistency in identifying BSM scenarios amidst Quantum Chromodynamics backgrounds. The proposed GNN architecture demonstrates superior performance in tagging semi-visible jets, highlighting its potential as a robust tool for advancing BSM search strategies at high-energy colliders.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2024

ThermoPore: Predicting part porosity based on thermal images using deep learning

Part qualification is often a critical and labor-intensive process in additive manufacturing, particularly in the detection of defects such as porosity, which stands to benefit significantly from advancements in machine learning. We present a deep learning approach for quantifying and localizing ex-situ porosity within Laser Powder Bed Fusion fabricated samples utilizing in-situ thermal image monitoring data. Our goal is to build the real time porosity map of parts based on thermal images acquired during the build. The quantification task builds upon the established Convolutional Neural Network model architecture to predict pore count and the localization task leverages the spatial and temporal attention mechanisms of the novel Video Vision Transformer model to indicate areas of expected porosity. Our model for porosity quantification achieved a R 2 score of 0.57 and our model for porosity localization produced an average Intersection over Union (IoU) score of 0.32 and a maximum of 1.0. This work is setting the foundations of part porosity “Digital Twins” based on additive manufacturing monitoring data and can be applied downstream to reduce time-intensive post-inspection and testing activities during part qualification and certification. In addition, we seek to accelerate the acquisition of crucial insights normally only available through ex-situ part evaluation by means of machine learning analysis of in-situ process monitoring data.

Deep learning↗
DOE OSTI2024

Machine learning models of intermittent operation of RO wellhead water treatment for salinity reduction and nitrate removal

Machine learning models were developed for intermittent multi-mode operation of a wellhead reverse osmosis water purification and desalination system to predict salt passage, nitrate passage, and permeate flux. The models, based on long short-term memory (LSTM) recurrent neural network (RNN) architecture, included an attention mechanism to increase model performance in proximity of the regulatory limit for nitrate. Training and testing of the models for the Startup, Production, Shutdown and Flushing operational modes were based on operational data (consisting of 22 process variables per data sample) acquired every 2–5 s over a six-month period. The significant sets of model input attributes for the different operational modes were assessed via Spearman ranking correlation, Self-Organizing Map (SOM) analysis and feed forward feature selection (FFFS). Although the variability of nitrate passage, salt passage and permeate flux was significant over the four operational modes, prediction performance for the three outcomes were with R2 and Average Absolute Relative Error (AARE) of 0.78–0.95 and 2.96–6.16 %, respectively. Model updates post membrane elements replacement demonstrated similar levels of prediction accuracy. The study results suggest that there is merit in exploring the utility of multi-mode models for sensor fault detection, data imputation, and for potential use in model-predictive control.

Intermittent RO operation↗
DOE OSTI2022

Automated bubble analysis of high-speed subcooled flow boiling images using U-net transfer learning and global optical flow

Capturing and analyzing the bubble dynamics is crucial to improving the understanding of boiling heat transfer mechanisms and predicting boiling heat transfer coefficient and boiling crisis. High speed video (HSV) imaging has been used for decades towards this end. Still, there is no universal approach to quantitatively analyze bubble dynamics from HSV images. In this study, we propose a data-driven post-processing approach to segment, track, and identify wall-attached vapor bubbles from HSV images of the boiling process in subcooled flow conditions. Firstly, we employ a transfer learning framework with a U-Net-based convolution neural network (CNN) architecture to detect and segment bubbles in HSV images of diverse contrast and surface texture using very little data (e.g., 10 images) for training. Then, we evaluate the trained CNN model with 100 ground-truth images, and the validation results show that the model accuracy and precision in detecting the optical footprint of bubbles are higher than 90%. Finally, we suggest a criterion to identify a condensing bubble based on the divergence of the bubble displacement, which is calculated from sequential segmented bubble images using a global optical flow code. Using this combination of machine learning and optical flow, we can identify nucleation sites and track the growth of bubbles nucleating at each site to quantify nucleation site density, nucleation frequency, and other fundamental boiling parameters. The proposed system is validated using results obtained on a special heater, which enables both infrared (IR) thermometry and HSV imaging on a metallic surface. We compare the fundamental boiling parameters obtained by the two different diagnostics. The results show good agreement. In conclusion, the difference between the measurements of nucleation site density, averaged nucleation frequency, and averaged growth time performed with the two techniques is always within ± 20% and mostly ± 10% of the values measured with IR thermometry.

42 ENGINEERING↗
DOE OSTI2025

Anti-symmetric barron functions and their approximation with sums of determinants

A fundamental problem in quantum physics is to encode functions that are completely anti-symmetric under permutations of identical particles. The architecture of neural network models for the electron wave function typically comprises an equivariant component followed by a summation of determinants. The recently introduced Generic Antisymmetric (GA) block is designed to enhance the expressivity of such neural wave functions, and it was found that the 2-layer GA block achieved more accurate energies than the corresponding single-determinant FermiNet architecure, suggesting its promise as a way to improve the expressivity of neural wave functions. In this paper we show how the function expressed by the 2-layer GA block can be decomposed into a sum of determinants. We formalize this result by defining the antisymmetric Barron space as a generalized version of the 2-layer GA block and providing an appromation theorem for this function class. This result can be viewed as a negative result showing that the 2-layer GA block is not more expressive than using multiple determinants.

Abrahamsen, Nilin↗
DOE OSTI2023

Multi-module-based CVAE to predict HVCM faults in the SNS accelerator

We present a multi-module framework based on Conditional Variational Autoencoder (CVAE) to detect anomalies in the power signals coming from multiple High Voltage Converter Modulators (HVCMs). We condition the model with the specific modulator type to capture different representations of the $\mathcal{normal}$ waveforms and to improve the sensitivity of the model to identify a specific type of fault when we have limited samples for a given module type. We studied several Artificial Neural Network (ANN) architectures for our CVAE model and evaluated the model performance by looking at their loss landscape for stability and generalization. Our results for the Spallation Neutron Source (SNS) experimental data show that the trained model generalizes well to detecting multiple fault types for several HVCM module types. The results of this study can be used to improve the HVCM reliability and overall SNS uptime.

43 PARTICLE ACCELERATORS↗
DOE OSTI2021

Machine learning based rate optimization under geologic uncertainty

We propose a novel approach for rate optimization during a waterflood under geologic uncertainty in reservoir properties such as permeability and porosity. The traditional approach typically involves several runs of the forward simulator. This may not scale well when the optimization is to be performed at the full field-level and over multiple geologic realizations. A machine-learning (ML) based approach which is quick and scalable for rate optimization over multiple geologic realizations is proposed instead. The training data for the model is generated by running the forward simulator with randomly assigned well rates using multiple geologic realizations. A reduced order representation of the permeability heterogeneity in each of the realizations is derived using a grid connectivity transformation (GCT). This step involves finding basis functions corresponding to the different modal frequencies of the grid connectivity represented by the grid Laplacian. The projection of the heterogeneous property field along these basis functions gives the basis coefficients that form the reduced order representation. Subsequently, for each training datapoint, streamlines are traced and the minimum time of flight (TOF) representing the tracer breakthrough time at each producer is recorded. The basis coefficients and well rates are fed to a machine learning model as input and the minimum TOF at the producers forms the output of the model. This trained model can then be used along with an optimizer for computing the optimal injection rates to maximize the injection sweep efficiency. This corresponds to minimizing the variance in the minimum TOF within each well group. Different architectures of neural network are tested using 5-fold cross validation to decide the best ML model to compute the streamline time of flight. The trained model is used to perform well rate optimization over multiple realizations of geology by using a risk tolerance penalty. The optimal well rates thus obtained are compared with two cases: a) equal well rates assigned to all injectors and producers and b) well rates obtained by optimizing over a single realization without considering the uncertainty in geology. The optimal well rates are seen to offer better oil recovery and sweep efficiency than both cases.

02 PETROLEUM↗
DOE OSTI2021

Explore Spatio‐Temporal Learning of Large Sample Hydrology Using Graph Neural Networks

Abstract Streamflow forecasting over gauged and ungauged basins play a vital role in water resources planning, especially under the changing climate. Increased availability of large sample hydrology data sets, together with recent advances in deep learning techniques, has presented new opportunities to explore temporal and spatial patterns in hydrological signatures for improving streamflow forecasting. The purpose of this study is to adapt and benchmark several state‐of‐the‐art graph neural network (GNN) architectures, including ChebNet, Graph Convolutional Network (GCN), and GraphWaveNet, for end‐to‐end graph learning. We explicitly represent river basins as nodes in a graph, learn the spatiotemporal nodal dependencies, and then use the learned relations to predict streamflow simultaneously across all nodes in the graph. The efficacy of the developed GNN models is investigated using the Catchment Attributes and MEteorology for Large‐sample Studies (CAMELS) data set under two settings, fixed graph topology (transductive learning), and variable graph topology (inductive learning), with the latter applicable to prediction in ungauged basins (PUB). Results indicate that GNNs are generally robust and computationally efficient, achieving similar or better performance than a baseline model trained using the long short‐term memory (LSTM) network. Further analyses are conducted to interpret the graph learning process at the edge and node levels and to investigate the effect of different model configurations. We conclude that graph learning constitutes a viable machine learning‐based method for aggregating spatiotemporal information from a multitude of sources for streamflow forecasting

Sun, Alexander Y.↗
DOE OSTI2026

Image processing pipeline for AI-driven nanoparticle megalibrary characterization

Recent innovations have made it possible to produce megalibraries, millions of structurally and compositionally distinct nanoparticles on a chip. These megalibraries yield vast volumes of data that are impossible to analyze manually, necessitating the development of automated tools. In previous work, we created a binary classification machine learning model to select quality nanoparticle images for downstream analysis. In this work, we show that adding a custom image processing step before training can produce significantly higher-performing models in a fraction of the time and make them more robust to different image noise levels and microscope acquisition settings. The image processing pipeline proposed here effectively cleans raw nanoparticle images, enhances key features, and allows us to use much lower resolution images and simpler neural network model architectures. These features result in higher performance and significant cost savings. Experiments demonstrate superior performance relative to baseline, including an 18.2% improvement in recall and a 13.1% increase in accuracy. Given the high cost of downstream analysis, it is critical to minimize false positives, and our best-performing model reaches a precision of 95.9% and a weighted F-score of 95.1% on an unseen test set. Additionally, model training time is reduced from hours to less than a minute. We also show that, using this custom image processing pipeline, model performance is significantly improved at lower pixel resolutions compared to downsizing alone. We expect that adopting this pipeline for AI-driven automated nanoparticle characterization will allow researchers to rapidly and accurately analyze much greater volumes of data, thereby accelerating materials discovery.

77 NANOSCIENCE AND NANOTECHNOLOGY↗
DOE OSTI2023

Data-Driven RANS Turbulence Closures for Forced Convection Flow in Reactor Downcomer Geometry

Recent progress in data-driven turbulence modeling has shown its potential to enhance or replace traditional equation-based Reynolds-averaged Navier-Stokes (RANS) turbulence models. Here, this work utilizes invariant neural network (NN) architectures to model Reynolds stresses and turbulent heat fluxes in forced convection flows (when the models can be decoupled). As the considered flow is statistically one dimensional, the invariant NN architecture for the Reynolds stress model reduces to the linear eddy viscosity model. To develop the data-driven models, direct numerical and RANS simulations in vertical planar channel geometry mimicking a part of the reactor downcomer are performed. Different conditions and fluids relevant to advanced reactors (sodium, lead, unitary-Prandtl-number fluid, and molten salt) constitute the training database. The models enabled accurate predictions of velocity and temperature, and compared to the baseline k–τ turbulence model with the simple gradient diffusion hypothesis, do not require tuning of the turbulent Prandtl number. The data-driven framework is implemented in the open-source graphics processing unit–accelerated spectral element solver nekRS and has shown the potential for future developments and consideration of more complex mixed convection flows.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
DOE OSTI2026

Viability of perturbative expansion for quantum field theories on neurons

Neural Network (NN) architectures that break statistical independence of parameters have been proposed as a new approach for simulating local quantum field theories (QFTs) [1]. In the infinite neuron number limit, single-layer NNs can exactly reproduce QFT results. This paper examines the viability of this architecture for perturbative calculations of local QFTs for finite neuron number N using scalar ϕ 4 theory in d Euclidean dimensions as an example. We find that the renormalized O(1/N ) corrections to two-and four-point correlators yield perturbative series which are sensitive to the UV cut-off and therefore have a weak convergence. We propose a modification to the architecture to improve this convergence and discuss constraints on the parameters of the theory and the scaling of N which allow us to extract accurate field theory results.

Sen, Srimoyee [Iowa State University, Ames, IA (Un↗
DOE OSTI2024

Improving neutrino energy estimation of charged-current interaction events with recurrent neural networks in MicroBooNE

We present a deep learning-based method for estimating the neutrino energy of charged-current neutrino-argon interactions. We employ a recurrent neural network (RNN) architecture for neutrino energy estimation in the MicroBooNE experiment, utilizing liquid argon time projection chamber (LArTPC) detector technology. Traditional energy estimation approaches in LArTPCs, which largely rely on reconstructing and summing visible energies, often experience sizable biases and resolution smearing because of the complex nature of neutrino interactions and the detector response. The estimation of neutrino energy can be improved after considering the kinematics information of reconstructed final-state particles. Utilizing kinematic information of reconstructed particles, the deep learning-based approach shows improved resolution and reduced bias for the muon neutrino Monte Carlo simulation sample compared to the traditional approach. In order to address the common concern about the effectiveness of this method on experimental data, the RNN-based energy estimator is further examined and validated with dedicated data-simulation consistency tests using MicroBooNE data. We also assess its potential impact on a neutrino oscillation study after accounting for all statistical and systematic uncertainties and show that it enhances physics sensitivity. This method has good potential to improve the performance of other physics analyses. Published by the American Physical Society 2024

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2025

Dynamical Sketching for Enhanced Communication Efficiency in Federated Learning

Federated learning (FL) has revolutionized distributed machine learning by enabling collaborative model training without sharing local data. However, communication efficiency and privacy guarantees remain significant challenges. This paper introduces a dynamic sketching mechanism in FL, optimizing the trade-off between communication efficiency and model accuracy. By dynamically selecting the sketch matrix size, our approach adapts to the evolving characteristics of the data and the model, ensuring optimal performance across diverse scenarios. We leverage Bayesian optimization to systematically tune the sketch parameters, achieving an effective balance between resource efficiency and model performance. Experimental results on the MNIST dataset using a convolutional neural network (CNN) architecture validate the proposed method's efficiency and scalability. Our dynamic sketching approach significantly outperforms fixed-size sketching techniques, achieving higher compression ratios (up to 62x) and providing better privacy guarantees while maintaining high model accuracy. These findings highlight the robustness and versatility of our approach and make it a valuable solution for privacy-preserving, communication-efficient federated learning.

Afrose, Sharmin [ORNL]↗
DOE OSTI2022

Use of Event-Time Embeddings via RNN to Discern Novel Event Sequences in EHRs

In highly configurable health information technology (HIT) systems, such as VistA of the Veterans Health Administration, the variations in how the system is used among different healthcare facilities and how the data are recorded can be significant. Despite the successful standardization of care efforts, some of these variations can be indicative of HIT hazards and demand further investigation. In this work, we implemented a recurrent neural network (RNN) architecture to learn clinical provider order sequences and their temporal dynamics while predicting the orders' terminal state. We demonstrate model performance and provide a use case for the model discerning novel event sequences. This model is proposed to find novel event sequences in an operational environment.

Ozmen, Ozgur↗
DOE OSTI2025

A Weakly Supervised Machine Learning Procedure for Magnet Quench Diagnostics

Voltage taps remain the standard and reliable diagnostic tool for detecting quenches in superconducting magnets. However, they identify a quench only at the time of voltage rise and do not provide information on earlier physical precursors. In this work, we investigate whether acoustic emission data can reveal precursor activity that occurs before conventional voltage detection using machine learning techniques. We introduce an event selection method and a weakly supervised machine learning procedure to learn data-driven criteria for identifying potential acoustic precursors to quenches. Two Convolutional Neural Network (CNN) architectures are trained: one on acoustic sensor events from our selection procedure and one on the Fast Fourier Transforms (FFTs) of these events. Both networks are trained iteratively using confidence-weighted loss functions to associate certain subsets of training data with a precursor label. We evaluate the performance of these models by examining the time distribution of events classified as potential precursors relative to the quench onset. Results indicate that the proposed approach can possibly distinguish acoustic emission events occurring closer to the quench from earlier acoustic activity during ramping, suggesting the potential for flagging quench precursors in acoustic data.

Khan, Maira [Fermilab] (ORCID:0009000891602387)↗
DOE OSTI2024

Modeling the Impacts of Material Properties on Oscillatory Neuron Behavior

In this study, neuromorphic computing, which mimics the functions of biological brains, offers improvements in both latency and energy efficiency over typical von Neumann computing architectures. Spiking neural networks can be especially power-efficient because they encode information temporally and can use more sparse electrical inputs. Here, we study the design of volatile memristors (variable resistors with memory) for neuronal devices, with particular consideration toward the feasibility of all-on-chip oscillation using built-in capacitance. We use circuit simulations to model the behavior of oscillator neurons with a range of realistic material properties. We find that energy inputs increase with insulating-phase resistivity, thermal conductivity, and device aspect ratio. However, we also find that the minimum capacitance needed for oscillation decreases with increasing insulating-phase resistivity, which opposes the constraints for power efficiency. Based on published data on NbO 2 , VO 2 , and EuNiO 3 , we find that existing materials can be engineered for all-on-chip spiking using their parasitic capacitance.

36 MATERIALS SCIENCE↗