Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “neural encoding”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Statistically-informed deep learning for gravitational wave parameter estimation

We introduce deep learning models to estimate the masses of the binary components of black hole mergers, $(m_1,m_2)$, and three astrophysical properties of the post-merger compact remnant, namely, the final spin, $a_\mathrm f$, and the frequency and damping time of the ringdown oscillations of the fundamental $\ell = m = 2$ bar mode, $(\omega_\mathrm R, \omega_\mathrm I)$. Our neural networks combine a modified WaveNet architecture with contrastive learning and normalizing flow. We validate these models against a Gaussian conjugate prior family whose posterior distribution is described by a closed analytical expression. Upon confirming that our models produce statistically consistent results, we used them to estimate the astrophysical parameters $(m_1,m_2, a_\mathrm f, \omega_\mathrm R, \omega_\mathrm I)$ of five binary black holes: GW150914, GW170104, GW170814, GW190521 and GW190630. We use PyCBC Inference to directly compare traditional Bayesian methodologies for parameter estimation with our deep learning based posterior distributions. Our results show that our neural network models predict posterior distributions that encode physical correlations, and that our data-driven median results and 90% confidence intervals are similar to those produced with gravitational wave Bayesian analyses. This methodology requires a single V100 NVIDIA GPU to produce median values and posterior distributions within two milliseconds for each event. Furthermore, this neural network, and a tutorial for its use, are available at the Data and Learning Hub for Science.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

Learning Constitutive Relations From Soil Moisture Data via Physically Constrained Neural Networks

Abstract The constitutive relations of the Richardson‐Richards equation encode the macroscopic properties of soil water retention and conductivity. These soil hydraulic functions are commonly represented by models with a handful of parameters. The limited degrees of freedom of such soil hydraulic models constrain our ability to extract soil hydraulic properties from soil moisture data via inverse modeling. We present a new free‐form approach to learning the constitutive relations using physically constrained neural networks. We implemented the inverse modeling framework in a differentiable modeling framework, JAX, to ensure scalability and extensibility. For efficient gradient computations, we implemented implicit differentiation through a nonlinear solver for the Richardson‐Richards equation. We tested the framework against synthetic noisy data and demonstrated its robustness against varying magnitudes of noise and degrees of freedom of the neural networks. We applied the framework to soil moisture data from an upward infiltration experiment and demonstrated that the neural network‐based approach was better fitted to the experimental data than a parametric model and that the framework can learn the constitutive relations.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

A unified neural-network framework for nucleon imaging from numerical simulations of QCD

Parton distributions encode the momentum-space structure and, in their generalizations, the spatial tomography of quarks and gluons inside hadrons, the building blocks of visible matter. We present a unified neural-network approach that learns these distributions directly from matrix elements calculated via numerical simulations of quantum chromodynamics (QCD) on the lattice by fitting two complementary inputs simultaneously: data matched to physical quantities via known momentum-space and coordinate-space formalisms. Utilizing data from both methods stabilizes the extraction and mitigates biases that can arise when either is used alone. We validate the method on controlled mock data and apply it to lattice-QCD matrix elements to extract parton distribution functions (PDFs). We show benefits of such an approach for determining the physical quantities. We further extend the framework to zero-skewness generalized parton distributions and demonstrate nucleon tomography within the same neural-network parameterization. Our results provide an adaptable and systematically improvable approach for extracting partonic distributions from Euclidean correlators. It can incorporate polarization, additional channels, and future experimental constraints from current and future facilities, such as the Electron-Ion Collider.

Hadronic Spectroscopy↗
DOE OSTI2023

Machine learning for detection of 3D features using sparse x-ray tomographic reconstruction

In many inertial confinement fusion (ICF) experiments, the neutron yield and other parameters cannot be completely accounted for with one and two dimensional models. This discrepancy suggests that there are three dimensional effects that may be significant. Sources of these effects include defects in the shells and defects in shell interfaces, the fill tube of the capsule, and the joint feature in double shell targets. Due to their ability to penetrate materials, x rays are used to capture the internal structure of objects. Methods such as computational tomography use x-ray radiographs from hundreds of projections, in order to reconstruct a three dimensional model of the object. In experimental environments, such as the National Ignition Facility and Omega-60, the availability of these views is scarce, and in many cases only consists of a single line of sight. Mathematical reconstruction of a 3D object from sparse views is an ill-posed inverse problem. These types of problems are typically solved by utilizing prior information. Neural networks have been used for the task of 3D reconstruction as they are capable of encoding and leveraging this prior information. We utilize half a dozen, different convolutional neural networks to produce different 3D representations of ICF implosions from the experimental data. Deep supervision is utilized to train a neural network to produce high-resolution reconstructions. These representations are used to track 3D features of the capsules, such as the ablator, inner shell, and the joint between shell hemispheres. Machine learning, supplemented by different priors, is a promising method for 3D reconstructions in ICF and x-ray radiography, in general.

Wolfe, Bradley T. (ORCID:0000000268301614)↗
DOE OSTI2025

Learning PDFs through interpretable latent representations in Mellin space

Representing the parton distribution functions (PDFs) of the proton and other hadrons through flexible, high-fidelity parametrizations has been a long-standing goal of particle physics phenomenology. This is particularly true since the chosen parametrization methodology can play an influential role in the ultimate PDF uncertainties as extracted in QCD global analyses; these, in turn, are often determinative of the reach of experiments at the LHC and other facilities to nonstandard physics, including at large 𝑥, where parametrization effects can be significant. In this study, we explore a series of encoder-decoder machine-learning (ML) models with various neural-network topologies as efficient means of reconstructing PDFs from meaningful information stored in an interpretable latent space. Given recent effort to pioneer synergies between QCD analyses and lattice-gauge calculations, we formulate a latent representation based on the behavior of PDFs in Mellin space, i.e., their integrated moments, and test the ability of various models to decode PDFs from this information faithfully. We introduce a numerical package, PDFdecoder, which implements several encoder-decoder models to reconstruct PDFs with high fidelity and use this end-to-end tool to explore how such neural-network-based models might connect PDF parametrizations to underlying properties like their Mellin moments. We additionally dissect patterns of learned correlations between encoded Mellin moments and reconstructed PDFs that suggest opportunities for further improvements to ML-based approaches to PDF parametrizations and uncertainty quantification.

Machine learning↗
DOE OSTI2025

Learning nuclear cross sections across the chart of nuclides with graph neural networks

We explore the use of deep learning techniques to learn how nuclear cross sections change as we add or remove protons and neutrons. As a proof of principle, we focus on the neutron-induced reactions in the fast energy regime. Our approach follows a two-stage learning framework. First, we apply representation learning to encode cross section data into a latent space using either variational autoencoders (VAEs) or implicit neural representations (INRs). Then, we train graph neural networks (GNNs) on the resulting embeddings to predict missing values across the nuclear chart by leveraging the topological structure of neighboring isotopes. We demonstrate accurate cross section predictions within a 9 × 9 block of missing nuclei. We also find that the optimal GNN training strategy depends on the type of latent representation used, with VAE embeddings performing best under end-to-end optimization in the original space, while INR embeddings achieve better results when the GNN is trained only in the latent space. Furthermore, using clustering algorithms, we map groups of latent vectors into regions of the nuclear chart and show that VAEs and INRs can discover some of the neutron magic numbers. These findings suggest that deep-learning models based on the representation encoding of cross sections combined with graph neural networks hold significant potential in augmenting nuclear theory models, e.g., by providing reliable estimates of covariances of cross sections, including cross-material covariances.

Machine learning↗
DOE OSTI2024

Learning protocols for the fast and efficient control of active matter

Exact analytic calculation shows that optimal control protocols for passive molecular systems often involve rapid variations and discontinuities. However, similar analytic baselines are not generally available for active-matter systems, because it is more difficult to treat active systems exactly. Here we use machine learning to derive efficient control protocols for active-matter systems, and find that they are characterized by sharp features similar to those seen in passive systems. We show that it is possible to learn protocols that effect fast and efficient state-to-state transformations in simulation models of active particles by encoding the protocol in the form of a neural network. We use evolutionary methods to identify protocols that take active particles from one steady state to another, as quickly as possible or with as little energy expended as possible. Our results show that protocols identified by a flexible neural-network ansatz, which allows the optimization of multiple control parameters and the emergence of sharp features, are more efficient than protocols derived recently by constrained analytical methods. Our learning scheme is straightforward to use in experiment, suggesting a way of designing protocols for the efficient manipulation of active matter in the laboratory.

74 ATOMIC AND MOLECULAR PHYSICS↗
DOE OSTI2024

Robust errant beam prognostics with conditional modeling for particle accelerators

Abstract Particle accelerators are complex and comprise thousands of components, with many pieces of equipment running at their peak power. Consequently, they can fault and abort operations for numerous reasons, lowering efficiency and science output. To avoid these faults, we apply anomaly detection techniques to predict unusual behavior and perform preemptive actions to improve the total availability. Supervised machine learning (ML) techniques such as siamese neural network models can outperform the often-used unsupervised or semi-supervised approaches for anomaly detection by leveraging the label information. One of the challenges specific to anomaly detection for particle accelerators is the data’s variability due to accelerator configuration changes within a production run of several months. ML models fail at providing accurate predictions when data changes due to changes in the configuration. To address this challenge, we include the configuration settings into our models and training to improve the results. Beam configurations are used as a conditional input for the model to learn any cross-correlation between the data from different conditions and retain its performance. We employ conditional siamese neural network (CSNN) models and conditional variational auto encoder (CVAE) models to predict errant beam pulses at the spallation neutron source under different system configurations and compare their performance. We demonstrate that CSNNs outperform CVAEs in our application.

43 PARTICLE ACCELERATORS↗
DOE OSTI2023

Automatic Crack Segmentation and Feature Extraction in Electroluminescence Images of Solar Modules

The effect of cracks in solar cells on the long-term degradation of photovoltaic (PV) modules remains to be determined. To investigate this effect in future studies, it is necessary to quantitatively describe the crack features (e.g., length) and correlate them with module power loss. Electroluminescence (EL) imaging is a common technique for identifying cracks. However, it is currently challenging and time-consuming to identify cracks in a large number of EL images and quantify complex crack features by human inspection. This article introduces a fast semantic segmentation method (~0.18 s/cell) to automatically segment cracks from EL images and algorithms to extract crack features. Here we fine-tuned a UNet neural network model using pretrained VGG16 as the encoder and obtained an average F1 score of 0.875 and an intersection over union score of 0.782 on the testing set. With cracks and busbars segmented, we developed algorithms for extracting crack features, including the crack-isolated area, the brightness inside the isolated area, and the crack length. We also developed an automatic preprocessing tool for cropping individual cell images from EL images of PV modules (~0.72 s/module). Our codes are published as open-source an software, and our annotated dataset composed of various types of cells is published as a benchmark for crack segmentation in EL images.

14 SOLAR ENERGY↗
DOE OSTI2022

A New Evaluation Metric for Demand Response-Driven Real-Time Price Prediction Towards Sustainable Manufacturing

Abstract The increasing industry energy demand highlights the urgency of demand response management, while the emerging smart manufacturing technologies pave the way for the implementation of real-time price (RTP)-based demand response management towards sustainable manufacturing. The demand response management requires scheduling of manufacturing systems based on RTP predictions, and thus the prediction quality can directly alter the effectiveness of demand response. However, since the general price prediction algorithms and prediction evaluation metrics are not specifically designed for RTP in demand response problems, a good RTP prediction obtained and evaluated by these algorithms and metrics may not be suitable for demand response scheduling. Therefore, in this study, the relationships between the effectiveness of demand response for manufacturing systems and evaluation results from six commonly used metrics are investigated. Meanwhile, a new metric called k-peak distance (KPD), considering the characteristics of the demand response problem, is proposed and compared with the other six metrics. Furthermore, an encoder-decoder long short-term memory recurrent neural network with KPD is proposed to provide better RTP prediction for manufacturing demand response problems. The case studies indicate that the proposed KPD metric shows a 1.8–3.6 times higher correlation with the demand response effectiveness compared to the other metrics. In addition, the production schedule based on the RTP prediction obtained from the proposed algorithm can improve the effectiveness of demand response by 23.4% on average.

Engineering↗
DOE OSTI2023

A survey of techniques for optimizing transformer inference

Recent years have seen a phenomenal rise in the performance and applications of transformer neural networks. The family of transformer networks, including Bidirectional Encoder Representations from Transformer (BERT), Generative Pretrained Transformer (GPT) and Vision Transformer (ViT), have shown their effectiveness across Natural Language Processing (NLP) and Computer Vision (CV) domains. Transformer-based networks such as ChatGPT have impacted the lives of common men. However, the quest for high predictive performance has led to an exponential increase in transformers' memory and compute footprint. Researchers have proposed techniques to optimize transformer inference at all levels of abstraction. Further, this paper presents a comprehensive survey of techniques for optimizing the inference phase of transformer networks. We survey techniques such as knowledge distillation, pruning, quantization, neural architecture search and lightweight network design at the algorithmic level. We further review hardware-level optimization techniques and the design of novel hardware accelerators for transformers. We summarize the quantitative results on the number of parameters/FLOPs and the accuracy of several models/techniques to showcase the tradeoff exercised by them. We also outline future directions in this rapidly evolving field of research. We believe that this survey will educate both novice and seasoned researchers and also spark a plethora of research efforts in this field.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Stochastic machine learning via sigma profiles to build a digital chemical space

This work establishes a different paradigm on digital molecular spaces and their efficient navigation by exploiting sigma profiles. To do so, the remarkable capability of Gaussian processes (GPs), a type of stochastic machine learning model, to correlate and predict physicochemical properties from sigma profiles is demonstrated, outperforming state-of-the-art neural networks previously published. The amount of chemical information encoded in sigma profiles eases the learning burden of machine learning models, permitting the training of GPs on small datasets which, due to their negligible computational cost and ease of implementation, are ideal models to be combined with optimization tools such as gradient search or Bayesian optimization (BO). Gradient search is used to efficiently navigate the sigma profile digital space, quickly converging to local extrema of target physicochemical properties. While this requires the availability of pretrained GP models on existing datasets, such limitations are eliminated with the implementation of BO, which can find global extrema with a limited number of iterations. A remarkable example of this is that of BO toward boiling temperature optimization. Holding no knowledge of chemistry except for the sigma profile and boiling temperature of carbon monoxide (the worst possible initial guess), BO finds the global maximum of the available boiling temperature dataset (over 1,000 molecules encompassing more than 40 families of organic and inorganic compounds) in just 15 iterations (i.e., 15 property measurements), cementing sigma profiles as a powerful digital chemical space for molecular optimization and discovery, particularly when little to no experimental data is initially available.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2022

Learning functional priors and posteriors from data and physics

In this work, we develop a new Bayesian framework based on deep neural networks to be able to extrapolate in space-time using historical data and to quantify uncertainties arising from both noisy and gappy data in physical problems. Specifically, the proposed approach has two stages: (1) prior learning and (2) posterior estimation. At the first stage, we employ the physics-informed Generative Adversarial Networks (PI-GAN) to learn a functional prior either from a prescribed function distribution, e.g., Gaussian process, or from historical data and physics. At the second stage, we employ the Hamiltonian Monte Carlo (HMC) method to estimate the posterior in the latent space of PI-GANs. In addition, we use two different approaches to encode the physics: (1) automatic differentiation, used in the physicsinformed neural networks (PINNs) for scenarios with explicitly known partial differential equations (PDEs), and (2) operator regression using the deep operator network (DeepONet) for PDE-agnostic scenarios. We then test the proposed method for (1) meta-learning for one-dimensional regression, and forward/inverse PDE problems (combined with PINNs); (2) PDE-agnostic physical problems (combined with DeepONet), e.g., fractional diffusion as well as saturated stochastic (100-dimensional) flows in heterogeneous porous media; and (3) spatial-temporal regression problems, i.e., inference of a marine riser displacement field using experimental data from the Norwegian Deepwater Programme (NDP). The results demonstrate that the proposed approach can provide accurate predictions as well as uncertainty quantification given very limited scattered and noisy data, since historical data could be available to provide informative priors. In summary, the proposed method is capable of learning flexible functional priors, e.g., both Gaussian and non-Gaussian process, and can be readily extended to big data problems by enabling mini-batch training using stochastic HMC or normalizing flows since the latent space is generally characterized as low dimensional.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2021

Machine learning enables national assessment of wind plant controls with implications for land use

Summary As deployment of wind energy continues to expand, computationally efficient tools for predicting wind plant performance over a wide range of layout designs, technology innovations, and spatial locations are increasingly important for policy and investment decisions. We demonstrate two approaches to training a surrogate model to predict annual energy production (AEP) of parameterized wind plant layouts: one using a Gaussian process (GP) and the other using a fully convolutional neural network (FCNN). We leverage the powerful FCNN architecture by encoding wind plant design parameters and output response surface as an image. The FCNN produces more accurate results than the GP with mean absolute errors equivalent to 1% and 1.9% of plant rated power, respectively, although the GP performs well under limited training data and provides useful uncertainty information. We also evaluate a surrogate model for wake steering, enabling a nationwide assessment of the impact of plant control strategies and plant layout decisions. Across two million locations, we find that wake steering strategies boost AEP with relative gains upwards of 3%. Gains are most pronounced at sites without a dominant wind direction and where layout optimization is less fruitful. Additionally, we perform a nationwide sensitivity analysis showing that wake steering can mitigate wake losses from higher density plant layouts. Our results suggest that regions which have not been previously viable for wind deployment due to moderate wind resources are especially well enhanced by wake steering strategies that could help overcome land constraints and inflexible layout options, potentially identifying new deployment opportunities.

17 WIND ENERGY↗
DOE OSTI2022

PyZebrascope: An Open-Source Platform for Brain-Wide Neural Activity Imaging in Zebrafish

Understanding how neurons interact across the brain to control animal behaviors is one of the central goals in neuroscience. Recent developments in fluorescent microscopy and genetically-encoded calcium indicators led to the establishment of whole-brain imaging methods in zebrafish, which record neural activity across a brain-wide volume with single-cell resolution. Pioneering studies of whole-brain imaging used custom light-sheet microscopes, and their operation relied on commercially developed and maintained software not available globally. Hence it has been challenging to disseminate and develop the technology in the research community. Here, we present PyZebrascope, an open-source Python platform designed for neural activity imaging in zebrafish using light-sheet microscopy. PyZebrascope has intuitive user interfaces and supports essential features for whole-brain imaging, such as two orthogonal excitation beams and eye damage prevention. Its camera module can handle image data throughput of up to 800 MB/s from camera acquisition to file writing while maintaining stable CPU and memory usage. Its modular architecture allows the inclusion of advanced algorithms for microscope control and image processing. As a proof of concept, we implemented a novel automatic algorithm for maximizing the image resolution in the brain by precisely aligning the excitation beams to the image focal plane. PyZebrascope enables whole-brain neural activity imaging in fish behaving in a virtual reality environment. Thus, PyZebrascope will help disseminate and develop light-sheet microscopy techniques in the neuroscience community and advance our understanding of whole-brain neural dynamics during animal behaviors.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2023

Long–short-term memory encoder–decoder with regularized hidden dynamics for fault detection in industrial processes

The ability of recurrent neural networks (RNN) to model nonlinear dynamics of high dimensional process data has enabled data-driven RNN-based fault detection algorithms. Previous studies have focused on detecting faults by identifying the discrepancies in data distribution between the faulty and normal data, as reflected in prediction errors generated by RNN models. However, in industrial processes, variations in data distribution can also result from changes in normal control setpoints and compensatory control adjustments in response to disturbances, making it hard to differentiate between normal and faulty conditions. This paper proposes a fault detection method utilizing a long short-term memory (LSTM) encoder–decoder structure with regularized hidden dynamics and reversible instance normalization (RevIN) to compactly represent high-dimensional measurements for effective monitoring. During training, the hidden states of the model are regularized to form a low-dimensional latent space representation of the original multivariate time series data. As a result, the prediction errors of the latent states can be used to monitor the abnormal dynamic variations, while the reconstruction errors of the measured variables are used to monitor the abnormal static variations. Furthermore, the proposed indices can reflect operating conditions, even when the distribution of test data changes, which helps distinguish faults from normal adjustments and disturbances that controllers can settle. Here, data from numerical simulation and the Tennessee Eastman process are used to illustrate the effectiveness of the proposed fault detection method.

42 ENGINEERING↗
DOE OSTI2024

Modeling the Impacts of Material Properties on Oscillatory Neuron Behavior

In this study, neuromorphic computing, which mimics the functions of biological brains, offers improvements in both latency and energy efficiency over typical von Neumann computing architectures. Spiking neural networks can be especially power-efficient because they encode information temporally and can use more sparse electrical inputs. Here, we study the design of volatile memristors (variable resistors with memory) for neuronal devices, with particular consideration toward the feasibility of all-on-chip oscillation using built-in capacitance. We use circuit simulations to model the behavior of oscillator neurons with a range of realistic material properties. We find that energy inputs increase with insulating-phase resistivity, thermal conductivity, and device aspect ratio. However, we also find that the minimum capacitance needed for oscillation decreases with increasing insulating-phase resistivity, which opposes the constraints for power efficiency. Based on published data on NbO 2 , VO 2 , and EuNiO 3 , we find that existing materials can be engineered for all-on-chip spiking using their parasitic capacitance.

36 MATERIALS SCIENCE↗
DOE OSTI2025

PruningAMR

PruningAMR is an algorithm that reads in data stored as a neural network and outputs an adaptive mesh with values of the neural network stored at its vertices. Examples of input data include implicit neural representations (INRs) and physics-informed neural networks (PINNs). The output mesh is a grid-based adaptive mesh with larger elements for regions in which the neural network has coarse-scale variation and smaller elements for regions with fine scale variation. The goal of the software is to discretize the neural network to a mesh that faithfully captures the details encoded in the data without resorting a fine scale mesh.

Zvonek, JenniferE [Lawrence Livermore National Lab↗