Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Backpropagation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Coded-mask-based wavefront sensing technique for APS nanofocusing beamline diagnostics

Here, we extend our recently developed coded-mask wavefront sensing technique to enable single-shot measurements of nanofocused x-ray beams. This method accurately reconstructs the focal beam profile by backpropagating the wavefront measured downstream of the beam focus. To validate its performance, we benchmarked it against the conventional fluorescence wire scan method, successfully measuring ∼120 nm focal spots at the 28-ID-B beamline of the Advanced Photon Source using a polymeric compound refractive lens. The results highlight the effectiveness of coded-mask wavefront sensing for high-precision beam profiling and its application as a real-time wavefront monitoring tool.

Shi, Xianbo [Argonne National Laboratory (ANL), Ar↗
DOE OSTI2025

Identification of tau leptons using a convolutional neural network with domain adaptation

A tau lepton identification algorithm,DeepTau, based on convolutional neural network techniques, has been developed in the CMS experiment to discriminate reconstructed hadronic decays of tau leptons (τ h ) from quark or gluon jets and electrons and muons that are misreconstructed as τ h candidates. The latest version of this algorithm, v2.5, includes domain adaptation by backpropagation, a technique that reduces discrepancies between collision data and simulation in the region with the highest purity of genuine τh candidates. Additionally, a refined training workflow improves classification performance with respect to the previous version of the algorithm, with a reduction of 30–50% in the probability for quark and gluon jets to be misidentified as τ h candidates for given reconstruction and identification efficiencies. This paper presents the novel improvements introduced in theDeepTau algorithm and evaluates its performance in LHC proton-proton collision data at √(s) = 13 and 13.6 TeV collected in 2018 and 2022 with integrated luminosities of 60 and 35 fb -1 , respectively. Techniques to calibrate the performance of the τ h identification algorithm in simulation with respect to its measured performance in real data are presented, together with a subset of results among those measured for use in CMS physics analyses.

Large detector-systems performance↗
DOE OSTI2021

Differentiable programming for online training of a neural artificial viscosity function within a staggered grid Lagrangian hydrodynamics scheme

Lagrangian methods to solve the inviscid Euler equations produce numerical oscillations near shock waves. A common approach to reducing these oscillations is to add artificial viscosity (AV) to the discrete equations. The AV term acts as a dissipative mechanism that attenuates oscillations by smearing the shock across a finite number of computational cells. However, AV introduces several control parameters that are not determined by the underlying physical model, and hence, in practice are tuned to the characteristics of a given problem. We seek to improve the standard quadratic-linear AV form by replacing it with a learned neural function that reduces oscillations relative to exact solutions of the Euler equations, resulting in a hybrid numerical-neural hydrodynamic solver. Because AV is an artificial construct that exists solely to improve the numerical properties of a hydrodynamic code, there is no offline ‘viscosity data’ against which a neural network can be trained before inserting into a numerical simulation, thus requiring online training. We achieve this via differentiable programming, i.e. end-to-end backpropagation or adjoint solution through both the neural and differential equation code, using automatic differentiation of the hybrid code in the Julia programming language to calculate the necessary loss function gradients. A novel offline pre-training step accelerates training by initializing the neural network to the default numerical AV scheme, which can be learned rapidly by space-filling sampling over the AV input space. We find that online training over early time steps of simulation is sufficient to learn a neural AV function that reduces numerical oscillations in long-term hydrodynamic shock simulations. These results offer an early proof-of-principle that online differentiable training of hybrid numerical schemes with novel neural network components can improve certain performance aspects existing in purely numerical schemes.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

High-resolution source imaging and moment tensor estimation of acoustic emissions during brittle creep of basalt undergoing carbonation

SUMMARY As the high-frequency analogue to field-scale earthquakes, acoustic emissions (AEs) provide a valuable complement to study rock deformation mechanisms. During the load-stepping creep experiments with CO2-saturated water injection into a basaltic sample from Carbfix site in Iceland, 8791 AE events are detected by at least one of the seven piezoelectric sensors. Here, we apply a cross-correlation-based source imaging method, called geometric-mean reverse-time migration (GmRTM) to locate those AE events. Besides the attractive picking-free feature shared with other waveform-based methods (e.g. time-reversal imaging), GmRTM is advantageous in generating high-resolution source images with reduced imaging artefacts, especially for experiments with relatively sparse receivers. In general, the imaged AE locations are found to be scattered across the sample, suggesting a complicated fracture network rather than a well-defined major shear fracture plane, in agreement with X-ray computed tomography imaging results after retrieval of samples from the deformation apparatus. Clustering the events in space and time using the nearest-neighbour approach revealed a group of ‘repeaters’, which are spatially co-located over an elongated period of time and likely indicate crack, or shear band growth. Furthermore, we select 2196 AE events with high signal-to-noise-ratio (SNR) and conduct moment tensor estimation using the adjoint (backpropagated) strain tensor fields at the locations of AE sources. The resulting AE locations and focal mechanisms support our previously assertion that creep of basalt at the experimental conditions is accommodated dominantly by distributed microcracking.

58 GEOSCIENCES↗
DOE OSTI2021

Inverse Design of Plasma Metamaterial Devices for Optical Computing

We apply inverse-design methods to produce two-dimensional plasma metamaterial (PMM) devices. Backpropagated finite-difference frequency-domain (FDFD) simulations are used to design waveguides and demultiplexers operating under both TE and TM modes. Demultiplexing and waveguiding are demonstrated for devices composed of plasma elements with reasonable plasma densities of approximately 7 GHz, allowing for future in situ training and experimental realization of these designs. We also explore the possible applicability of PMMs to nonlinear boolean operations for use in optical computing. Functionally complete logical connectives (or and and) are achieved in the TM mode.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2021

A Visual Designer of Layer‐wise Relevance Propagation Models

Abstract Layer‐wise Relevance Propagation (LRP) is an emerging and widely‐used method for interpreting the prediction results of convolutional neural networks (CNN). LRP developers often select and employ different relevance backpropagation rules and parameters, to compute relevance scores on input images. However, there exists no obvious solution to define a “best” LRP model. A satisfied model is highly reliant on pertinent images and designers' goals. We develop a visual model designer, named as VisLRPDesigner, to overcome the challenges in the design and use of LRP models. Various LRP rules are unified into an integrated framework with an intuitive workflow of parameter setup. VisLRPDesigner thus allows users to interactively configure and compare LRP models. It also facilitates relevance‐based visual analysis with two important functions: relevance‐based pixel flipping and neuron ablation. Several use cases illustrate the benefits of VisLRPDesigner. The usability and limitation of the visual designer is evaluated by LRP users.

Huang, Xinyi↗
DOE OSTI2024

Projective Integral Updates for High-Dimensional Variational Inference

Variational inference is an approximation framework for Bayesian inference that seeks to improve quantified uncertainty in predictions by optimizing a simplified distribution over parameters to stand in for the full posterior. Capturing model variations that remain consistent with training data enables more robust predictions by reducing parameter sensitivity. This work introduces a fixed-point optimization for variational inference that is applicable when every feasible log density can be expressed as a linear combination of functions from a given basis. In such cases, the optimizer becomes a fixed-point of projective integral updates. When the basis spans univariate quadratics in each parameter, the feasible distributions are Gaussian mean-fields and the projective integral updates yield quasi-Newton variational Bayes (QNVB). Other bases and updates are also possible. Since these updates require high-dimensional integration, this work begins by proposing an efficient quasirandom sequence of quadratures for mean-field distributions. Each iterate of the sequence contains two evaluation points that combine to correctly integrate all univariate quadratic functions and, if the mean-field factors are symmetric, all univariate cubics. More importantly, averaging results over short subsequences achieves periodic exactness on a much larger space of multivariate polynomials of quadratic total degree. The corresponding variational updates require four loss evaluations with standard (not second-order) backpropagation to eliminate error terms from over half of all multivariate quadratic basis functions. Furthermore, this integration technique is motivated by first proposing stochastic blocked mean-field quadratures, which may be useful in other contexts. A PyTorch implementation of QNVB allows for better control over model uncertainty during training than competing methods. Experiments demonstrate superior generalizability for multiple learning problems and architectures.

Gaussian mean-field↗
DOE OSTI2024

Semantic Stealth: Crafting Covert Adversarial Patches for Sentiment Classifiers Using Large Language Models

Deep learning models have been shown to be vulnerable to adversarial attacks, in which perturbations to their inputs cause the model to produce incorrect predictions. As opposed to adversarial attacks in computer vision, where small changes introduced to pixel values can drastically alter a model's output while remaining imperceptible to humans, text-based attacks are difficult to conceal due to the discrete nature of tokens. Consequently, unconstrained gradient-based attacks often produce adversarial examples that lack semantic meaning, rendering them detectable through visual inspection or perplexity filters. In contrast to methods that rely on gradient-based optimization in the embedding space, we propose an approach that leverages a Large Language Model's ability to generate grammatically correct and semantically meaningful text to craft adversarial patches that seamlessly blend in with the original input text. These patches can be used to alter the behavior of a target model, such as a text classifier. Since our approach does not rely on gradient backpropagation, it only requires access to the target model's confidence scores, making it a grey-box attack. We demonstrate the feasibility of our approach using open-source LLMs, including Intel's Neural Chat, Llama2, and Mistral-Instruct, to generate adversarial patches capable of altering the predictions of a distilBERT model fine-tuned on the IMDB reviews dataset for sentiment classification.

Roa Carvajal, Maria↗
DOE OSTI2025

Triangle Method for Dense ReLU Layers [SWR-25-72]

This software is an implementation of the methods for initializing and training neural networks to be more efficient per parameter, described more fully below and in the related publication: In theory, depth should make a ReLU network EXPONENTIALLY more efficient by enabling it to produce an exponential number of piecewise linear sections in its output. This reasoning is largely based on the work of mathematicians that have hand-constructed networks that make good use of depth. In practice however, even very deep ReLU networks that have been randomly initialized will behave identically to their shallow counterparts - missing an entire exponential dimension of efficiency. The triangle method is a first attempt at realizing the exponential potential of deep networks. Instead of randomly setting weights, we force pairs of neurons in each layer learn to build triangles (i.e. functions from [0,1] -> [0,1] that look like triangles). This is a very efficient pattern for generating lots of linear pieces because composing two triangular functions doubles the number of pieces with each composition. The triangle method is more than just a different initialization, it is a new paradigm of training. Instead of making direct updates to the matrix weights, we do an extra step of backpropagation to collect the derivatives of the loss function with respect to the shapes of the triangles, training them to tilt left or right. This process essentially holds the networks hand throughout the loss landscape and forces it to always use depth effectively by producing triangular shapes internally. This can produce several orders of magnitude of improvement on convex one-dimensional regression problems. Much more theoretical work is needed to realize its full potential beyond this context, but the implementation in this repository will still work in arbitrary numbers of dimensions. The file Triangle_Method.py is a generalized form of the method that will build each neuron its own custom 1-d convex activation function (with exponential efficiency). Example usage on one dimensional problems can be found in Example_Usage.ipynb and an example of using this in a real neural network can be found in Example_VGG16_CIFAR10.ipynb.

Milkert, Max [National Renewable Energy Laboratory↗
DOE OSTI2025

Graph-based Reversible Evaluation and Tangents Library

GRETL is a C++ library for evaluation, re-evaluation and algorithmic differentiation of functional operations on an arbitrary computational graph with limited memory usage. Similar to popular machine learning frameworks in Python, like PyTorch and JAX, it tracks and stores both operations and output data as functions are evaluated. Once this composition of functions is built up, the entire chain of operations can be back propagated to compute sensitivities of the final result with respect to any number of inputs. In contrast to most machine learning applications, memory usage becomes the bottleneck for back propagation in many physics applications, especially for time-dependent PDEs. Dynamic check pointing becomes essential. An important distinguishing feature of GRETL is its ability to limit the maximum memory usage by automatically dynamic checkpointing the data output for each graph operation (see Wang, Moin, Iaccarino, 2009). During backpropagation, parts of the graph that are no longer in memory are automatically re-evaluated from upstream checkpointed states as needed for derivative sensitivity calculations (or more precisely, for vector-Jacobian products). GRETL is particularly beneficial for applications, such as coupled multi-physics, where deriving adjoint-based sensitivities and managing checkpoint memory across modules becomes onerous. Cases which can be readily handled by the GRETL library include: different time-integration algorithms per physics (e.g., coupled predictor-corrector algorithms, IMEX, etc.), sub-cycling, asynchronous integrators, state dependent timestep sizes, iterative solvers and coupling algorithms, controller algorithms, and more.

Tupek, MichaelR [Lawrence Livermore National Labor↗
DOE OSTI2022

Deep Neural Network Algorithm for CMC Microstructure Characterization and Variability Quantification

Microstructure characterization and variability quantification are crucial for understanding ceramic matrix composites (CMCs) mechanical behavior and deformation mechanisms across length scales. Traditionally, analyses of the micrographs obtained from microscopy are labor-intensive. However, with the vast improvement in computer vision (CV) and deep learning (DL), an automated algorithm can be designed to extract essential microstructure variability from micrographs which can then be used to construct a statistically representative volume element (SRVE). The DL-based algorithm spans the taxonomy of microstructure analyses, including semantic segmentation of microstructure constituents, secondary phases, matrix/fiber interface, and defects, and quantifying the microstructure variability in terms of probability distributions. In this work, C/SiNC and SiC/SiNC CMCs microstructures are semantically segmented through a deep convolutional neural network, followed by variability quantification through the implementation of a fully connected regression layer, hence forming a deep regression network. The deep regression network operates in a feedforward regime, in which the neuron output signal traverses through the network in a unidirectional manner. The weight tensor associated with each layer is updated through a backpropagation stochastic gradient descent approach. The input gray-scale image obtained through in-house scanning electron microscope and confocal microscope micrographs is augmented through affine transformations to increase the training set size, which is then processed through four strided convolutional layers. This compresses the image resolution by half at each layer while increasing the image depth by applying different filters (image encoding). The class activation maps (CAMs) corresponding to the applied filters highlight the key architectural features and assist with the semantic segmentation of the microstructure.

Hamza, Mohamed H.↗
DOE OSTI2024

xesn: Echo state networks powered by Xarray and Dask

Xesn is a Python package that allows scientists to easily design Echo State Networks (ESNs) for forecasting problems. ESNs are a Recurrent Neural Network architecture introduced by Jaeger (2001) that are part of a class of techniques termed Reservoir Computing. One defining characteristic of these techniques is that all internal weights are determined by a handful of global, scalar parameters, thereby avoiding problems during backpropagation and reducing training time significantly. Because this architecture is conceptually simple, many scientists implement ESNs from scratch, leading to questions about computational performance. Xesn offers a straightforward, standard implementation of ESNs that operates efficiently on CPU and GPU hardware. The package leverages optimization tools to automate the parameter selection process, so that scientists can reduce the time finding a good architecture and focus on using ESNs for their domain application. Importantly, the package flexibly handles forecasting tasks for out-of-core, multi-dimensional datasets, eliminating the need to write parallel programming code. Xesn was initially developed to handle the problem of forecasting weather dynamics, and so it integrates naturally with Python packages that have become familiar to weather and climate scientists such as Xarray (Hoyer & Hamman, 2017). However, the software is ultimately general enough to be utilized in other domains where ESNs have been useful, such as in signal processing (Jaeger & Haas, 2004).

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Probabilistic Nanomagnetic Memories for Uncertain and Robust Machine Learning

This project evaluated the use of emerging spintronic memory devices for robust and efficient variational inference schemes. Variational inference (VI) schemes, which constrain the distribution for each weight to be a Gaussian distribution with a mean and standard deviation, are a tractable method for calculating posterior distributions of weights in a Bayesian neural network such that this neural network can also be trained using the powerful backpropagation algorithm. Our project focuses on domain-wall magnetic tunnel junctions (DW-MTJs), a powerful multi-functional spintronic synapse design that can achieve low power switching while also opening the pathway towards repeatable, analog operation using fabricated notches. Our initial efforts to employ DW-MTJs as an all-in-one stochastic synapse with both a mean and standard deviation didn’t end up meeting the quality metrics for hardware-friendly VI. In the future, new device stacks and methods for expressive anisotropy modification may make this idea still possible. However, as a fall back that immediately satisfies our requirements, we invented and detailed how the combination of a DW-MTJ synapse encoding the mean and a probabilistic Bayes-MTJ device, programmed via a ferroelectric or ionically modifiable layer, can robustly and expressively implement VI. This design includes a physics-informed small circuit model, that was scaled up to perform and demonstrate rigorous uncertainty quantification applications, up to and including small convolutional networks on a grayscale image classification task, and larger (Residual) networks implementing multi-channel image classification. Lastly, as these results and ideas all depend upon the idea of an inference application where weights (spintronic memory states) remain non-volatile, the retention of these synapses for the notched case was further interrogated. These investigations revealed and emphasized the importance of both notch geometry and anisotropy modification in order to further enhance the endurance of written spintronic states. In the near future, these results will be mapped to effective predictions for room temperature and elevated operation DW-MTJ memory retention, and experimentally verified when devices become available.

77 NANOSCIENCE AND NANOTECHNOLOGY↗
DOE OSTI2023

Journey Over Destination: Differentiable Sensor Placement Enhances Generalization [Poster]

The challenge of reconstructing spatial fields that change over time from limited sensor data has been a focal point for many research studies. Various machine learning methods have been used in attempts to address this complex issue, including convolutional neural networks. All of the proposed methods share a common requirement that the user needs to manually determine the sensor positions. This requirement remains a limiting factor in the ongoing quest for efficient learning and accurate field reconstruction. This study aims to present a method that enables a model to optimize sensor positions via backpropagation, thereby facilitating the model’s exploration of the spatial domain and enhancing sensor positioning effectively. Indexing naturally incorporates discrete decisions. This operation is nondifferentiable which is a requirement for the application of gradient-based optimization methods. We showcased its effectiveness by training an attention-based neural network, which achieved top-tier performance on two separate datasets. To our knowledge, this represents the first fully end-to-end differentiable workflow for enhancing sensor placement within a neural network model.

58 GEOSCIENCES↗
DOE OSTI2025

Technical Report on Subsurface Monitoring of the Brady Hot Spring Geothermal Site, Nevada, based upon Full Waveform Inversion

Abilities to accurately characterize the subsurface in a geothermal setting is key to assess and support production. An important element of geothermal reservoir monitoring is also the ability to investigate fluid transport within fracture network. This report focuses on improving subsurface imaging and monitoring in geothermal settings using full waveform inversion based on the adjoint method and time-lapse imaging. To assess our method, we rely on a dense seismic dataset collected in 2016 at the Brady Hot Springs geothermal site in Nevada for the DOE-funded project Poroelastic Tomography by Adjoint Inverse Modeling of Data from Seismology, Geodesy, and Hydrology. This dataset captures subsurface changes across four stages of geothermal power plant operations, which involve varying rates of fluid injection and extraction. Two velocity models were previously derived from this dataset using different methods: one based on travel times and another on sweep interferometry. Our first step is to refine these models using adjoint tomography, which has been applied successfully at global and regional-scales but is less common at the reservoir-scale. Two approaches are then explored for time-lapse analysis: directly comparing refined tomographic models from different stages or backpropagating waveform differences relative to a baseline tomographic model. The main take away is that both approaches highlight similar reservoir behaviors, but the latter approach is more computationally effective in capturing small-scale changes in subsurface properties. For this work, we leverage the use of Salvus (www.mondaic.com), an end-to-end seismic imaging solution, relying on the spectral element method to compute forward and adjoint simulations, and developed by Mondaic Ltd. It includes integrated workflow management that handles waveform and metadata, launches simulations, computes waveform misfits and adjoint sources, and iterates for model updates by nonlinear optimization.

15 GEOTHERMAL ENERGY↗
DOE OSTI2025

CRCNS22 Learning Rules in the Hippocampus and their Mapping to Neuromorphic Systems (Final Technical Report)

Large scale biologically-realistic computational models are key to investigating the interplay between structure and function in nervous systems, thus paving the way to new clinical methods and neuro-inspired computing solutions. This project focuses on the hippocampus, in particular the CA3-CA1 regions, due to their role in associative learning and memory, pattern separation and completion, and spatial navigation. Investigations into the neuronal organization and learning rule(s) of this circuit can shed light into how declarative memories are formed, stored, recalled and forgotten and inform computational, experimental and clinical neuroscience work. Our project aims at developing a novel data-driven methodology supported by a broad heterogeneous base of neuroscience experimental knowledge and inspired from advances in computer science and engineering. Specifically, this work will benchmark existing and new learning rules within a full-scale spiking neural network simulation of the CA3-CA1 region. The model will be based on an open-source repository, called the Hippocampome, which contains neuronal morphologies, firing patterns, synapse probabilities, and most other required parameters for all known neuron types in the rodent hippocampal formation. The model will be first trained in a supervised fashion for associative memory tasks using backpropagation through time traditionally used in computer science, enhanced with a new technique called the surrogate gradient method. This optimization method will be used to obtain a global loss minimization, but it is not biologically inspired as it assumes the use of data not locally available to the synapses. However, we propose its use as a benchmarking tool, to compare the training performance of local biologically plausible and hardware-mappable learning rules at scale. New rules or combinations will be proposed and tested as needed, based on the obtained results. Progress in this area will also drive the development of novel hardware-mappable algorithms for continual lifelong learning and categorization of new events from few presented examples. This project goes beyond the existing state-of-the-art by looking at large scale realistic neuronal circuits as networks trainable via global optimization methods such as surrogate gradient descent. The objective function of the brain that supports learning is largely unknown, but it is likely that it operates through local learning rules. Studying network trajectories around local minima as proposed in this work represents a useful strategy for understanding whether a network is training by using a specific (set of) learning rule(s). Starting from a completely untrained network is a challenging test since it is difficult to determine how the learning rule affects the trajectory of the network. This interdisciplinary project will help understand what rule governs learning in these regions or if multiple learning rules are involved. The work will develop a robust methodology to measure if the network is converging to the target solution, oscillating around it, or diverging away.

59 BASIC BIOLOGICAL SCIENCES↗
NASA NTRS1989

Automatic voice recognition using traditional and artificial neural network approaches

The main objective of this research is to develop an algorithm for isolated-word recognition. This research is focused on digital signal analysis rather than linguistic analysis of speech. Features extraction is carried out by applying a Linear Predictive Coding (LPC) algorithm with order of 10. Continuous-word and speaker independent recognition will be considered in future study after accomplishing this isolated word research. To examine the similarity between the reference and the training sets, two approaches are explored. The first is implementing traditional pattern recognition techniques where a dynamic time warping algorithm is applied to align the two sets and calculate the probability of matching by measuring the Euclidean distance between the two sets. The second is implementing a backpropagation artificial neural net model with three layers as the pattern classifier. The adaptation rule implemented in this network is the generalized least mean square (LMS) rule. The first approach has been accomplished. A vocabulary of 50 words was selected and tested. The accuracy of the algorithm was found to be around 85 percent. The second approach is in progress at the present time.

Botros, Nazeih M.↗
NASA NTRS1991

Space shuttle main engine fault detection using neural networks

A method for on-line Space Shuttle Main Engine (SSME) anomaly detection and fault typing using a feedback neural network is described. The method involves the computation of features representing time-variance of SSME sensor parameters, using historical test case data. The network is trained, using backpropagation, to recognize a set of fault cases. The network is then able to diagnose new fault cases correctly. An essential element of the training technique is the inclusion of randomly generated data along with the real data, in order to span the entire input space of potential non-nominal data.

Bishop, Thomas↗