SEARCH · Engineering Papers

Results for “loss functions”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6

All sources NASA NTRS DOE OSTI

DOE OSTI2026

Deep-learning methods for contrast enhancement and artifact reduction in cryo-electron tomography: a systematic analysis of the state of the art and proposed improvements

Cryo-electron tomography (cryo-ET) has emerged as the preferred technique for visualizing the organization of macromolecular complexes in situ and resolving their structures at subnanometre resolution [Tegunov et al. (2021)View full citation, Nat. Methods, 18, 186–193]. Despite improvements in data quality as a result of advances in detector technology, microscope stability and stage precision, the analysis and interpretation of tomograms remains challenging due to a low signal-to-noise ratio and reconstruction artifacts stemming from experimental constraints in specimen tilt during data collection resulting in a missing wedge in the Fourier space. Recently, self-supervised deep-learning methods have been proposed for contrast enhancement and reduction of resolution anisotropy in reconstructed tomograms. Here, we evaluate several state-of-the-art deep-learning methods which aim to improve the interpretability of cryo-ET reconstructions, with a focus on their performance on downstream tasks of template matching, subtomogram averaging and segmentation. We propose new training architectures and a loss function based on Fourier shell correlation that show improved performance over the standard U-Net with L1/L2 losses. We demonstrate our analysis on four diverse experimental datasets: purified 80S ribosomes, in situ Chlamydomonas reinhardtii, immature HIV-1 virus-like particles and INS-1E cells.

contrast enhancement↗

DOE OSTI2025

Generative Vulnerability Assessment for Cyber-Physical Systems

Cyber-physical systems (CPS) are highly susceptible to malicious attacks due to their complex dynamics and interconnectivity. A comprehensive understanding of their vulnerabilities is essential for designing effective resilience measures. This paper presents a data-driven attack generative system for evaluating the vulnerability of CPS. The proposed approach formulates the vulnerability assessment problem as determining the feasibility of a specific attack set based on two boundary functions that represent the effectiveness and stealthiness of attacks. The attack generative model is trained using a custom loss function, with two universal approximators designed to learn the effectiveness and stealthiness functions simultaneously. Theoretical results for successful generation and asymptotic convergence of the resulting training algorithm are given. As a result, the proposed approach is evaluated via numerical simulation of an IEEE 14-bus system and gas pipeline systems, demonstrating its viability in learning how to attack nonlinear CPS and identify potential vulnerabilities.

Computer systems organization↗

DOE OSTI2024

Collective excitations and low-energy ionization signatures of relativistic particles in silicon detectors

Abstract Solid-state detectors with a low energy threshold have several applications, including searches of non-relativistic halo dark-matter particles with sub-GeV masses. When searching for relativistic, beyond-the-Standard-Model particles with enhanced cross sections for small energy transfers, a small detector with a low energy threshold may have better sensitivity than a larger detector with a higher energy threshold. In this paper, we calculate the low-energy ionization spectrum from high-velocity particles scattering in a dielectric material. We consider the full material response including the excitation of bulk plasmons. We generalize the energy-loss function to relativistic kinematics, and benchmark existing tools used for halo dark-matter scattering against electron energy-loss spectroscopy data. Compared to calculations commonly used in the literature, such as the Photo-Absorption-Ionization model or the free-electron model, including collective effects shifts the recoil ionization spectrum towards higher energies, typically peaking around 4–6 electron-hole pairs. We apply our results to the three benchmark examples: millicharged particles produced in a beam, neutrinos with a magnetic dipole moment produced in a reactor, and upscattered dark-matter particles. Our results show that the proper inclusion of collective effects typically enhances a detector’s sensitivity to these particles, since detector backgrounds, such as dark counts, peak at lower energies.

Physics↗

DOE OSTI2025

Reducing the Parameter Dependency of Phase-Picking Neural Networks with Dice Loss

Training a neural network for picking seismic phase arrivals has been commonly posed as a segmentation problem. It is a highly imbalanced segmentation problem in the sense that the background vastly dominates the foreground because we are trying to pick the optimal single sample point that represents the arrival of a seismic phase in a many seconds long time window. Here, we test the Dice loss, which is a preferred loss function for highly imbalanced image segmentation problems. We show that phase-picking neural networks trained on the Dice loss behave in a binary fashion for which the prediction output is almost always either nearly 1 or nearly 0. This feature removes the strong dependence of data processing workflows on the prediction score threshold, which is an otherwise critical parameter to determine when using neural networks trained on the cross-entropy loss. When strategically used, models trained on the Dice loss can reduce the parameter dependency of machine learning-based seismic monitoring.

58 GEOSCIENCES↗

DOE OSTI2025

Efficient Training of Deep Neural Operator Networks via Randomized Sampling

Neural operators (NOs) employ deep neural networks to learn the mappings between infinitedimensional function spaces. Deep operator network (DeepONet), a popular NO architecture, has demonstrated success in the real-time prediction of complex dynamics across various scientific and engineering applications. In this work, we introduce a random sampling technique to be adopted during the training of DeepONet, aimed at improving the generalization ability of the model, while significantly reducing the computational time. The proposed approach targets the trunk network of the DeepONet model that outputs the basis functions corresponding to the spatiotemporal locations of the bounded domain on which the physical system is defined. While constructing the loss function, DeepONet training traditionally considers a uniform grid of spatiotemporal points at which all the output functions are evaluated for each iteration. This approach leads to a larger batch size, resulting in poor generalization and increased memory demands, due to the limitations of the stochastic gradient descent (SGD) optimizer. The proposed random sampling over the inputs of the trunk net mitigates these challenges, improving generalization and reducing the memory requirements during training, resulting in significant computational gains. We validate our hypothesis through three benchmark examples, demonstrating substantial reductions in training time while achieving comparable or lower overall test errors relative to the traditional training approach. Our results indicate that incorporating randomization in the trunk network inputs during training enhances the efficiency and robustness of DeepONet, offering a promising avenue for improving the framework’s performance in modeling complex physical systems.

Karumuri, Sharmila [Department of Civil & Systems ↗

DOE OSTI2023

On the influence of over-parameterization in manifold based surrogates and deep neural operators

Constructing accurate and generalizable approximators (surrogate models) for complex physico-chemical processes exhibiting highly non-smooth dynamics is challenging. The main question is what type of surrogate models we should construct and should these models be under-parameterized or over-parameterized. In this work, we propose new developments and perform comparisons for two promising approaches: manifold-based polynomial chaos expansion (m-PCE) and the deep neural operator (DeepONet), and we examine the effect of over-parameterization on generalization. While m-PCE enables the construction of a mapping by first identifying low-dimensional embeddings of the input functions, parameters, and quantities of interest (QoIs), a neural operator learns the nonlinear mapping via the use of deep neural networks. Here, we demonstrate the performance of these methods in terms of generalization accuracy by solving the 2D time-dependent Brusselator reaction-diffusion system with uncertainty sources, modeling an autocatalytic chemical reaction between two species. We first propose an extension of the m-PCE by constructing a mapping between latent spaces formed by two separate embeddings of the input functions and the output QoIs. To further enhance the accuracy of the DeepONet, we introduce weight self-adaptivity in the loss function. We demonstrate that the performance of m-PCE and DeepONet is comparable for cases of relatively smooth input-output mappings. However, when highly non-smooth dynamics is considered, DeepONet shows higher approximation accuracy. We also find that for m-PCE, modest over-parameterization leads to better generalization, both within and outside of distribution, whereas aggressive over-parameterization leads to over-fitting. In contrast, an even highly over-parameterized DeepONet leads to better generalization for both smooth and non-smooth dynamics. Furthermore, we compare the performance of the above models with another recently proposed operator learning model, the Fourier Neural Operator, and show that its over-parameterization also leads to better generalization. Taken together, our studies show that m-PCE can provide very good accuracy at very low training cost, whereas a highly over-parameterized DeepONet can provide better accuracy and robustness to noise but at higher training cost. In both methods, the inference cost is negligible.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

Plateau Phenomenon in Gradient Descent Training of RELU Networks: Explanation, Quantification, and Avoidance

The ability of neural networks to provide ‘best in class’ approximation across a wide range of applications is well-documented. Nevertheless, the powerful expressivity of neural networks comes to naught if one is unable to effectively train (choose) the parameters defining the network. In general, neural networks are trained by gradient descent type optimization methods,a stochastic variant thereof. In practice, such methods result in the loss function decreases rapidly at the beginning of training but then, after a relatively small number of steps, significantly slow down. The loss may even appear to stagnate over the period of a large number of epochs, only to then suddenly start to decrease fast again for no apparent reason. This so-called plateau phenomenon manifests itself in many learning tasks. The present work aims to identify and quantify the root causes of plateau phenomenon.analysis is carried out in the setting of univariate ReLU networks. No assumptions are made on the number of neurons relative to the number of training data, and our results hold for both the lazy and adaptive regimes. Here, the main findings are: plateaux correspond to periods during which activation patterns remain constant, where activation pattern refers to the number of data points that activate a given neuron; quantification of convergence of the gradient flow dynamics; and, characterization stationary points in terms solutions of local least squares regression lines over subsets of the training data. Based on these conclusions, we propose a new iterative training method, the Active Neuron Least Squares (ANLS), characterised by the explicit adjustment of the activation pattern at each step, which is designed to enable a quick exit from a plateau. Illustrative numerical examples are included throughout.

97 MATHEMATICS AND COMPUTING↗

NASA NTRSDate not supplied

A Look at the Truths and Misconceptions of the Variational Quantum Eigensolver and the Implications of Overparameterization

In this work, we investigate loss landscapes of the variational quantum eigensolver (VQE) by quantifying the number of local minima through empirical analyses. We focus on minimal models in chemistry and physics so that we can do a complete analysis using more computationally expensive tools. We employ Hessian eigenvalue calculations and the nudged elastic band algorithm to characterize these landscapes. Our results expand upon the existing literature by highlighting the optimization challenges faced by VQE. We find that, as the number of parameters in our ansatz increases, the number of basins increases while the corresponding loss function values converge toward the global minimum value. This observation implies that overparameterization may lead to an ``effective convexity'' in VQE loss landscapes, a phenomenon supported by theoretical and numerical work in classical machine learning.

quantum computing↗

NASA NTRS2016

Development and Provision of Functional Foods to Promote Health on Long-Duration Space Missions

During long-duration NASA space missions, such as proposed missions to Mars, astronauts may experience negative physiological effects such as bone loss. Functional foods such as high-lycopene, high-flavonoids and high-omega-3 products and fruits and vegetables may mitigate the negative effects of spaceflight on physiological factors including the bone health of crewmembers. Previous studies showed that current ISS provisions provide high-lycopene and high-omega-3 food items but the variety is limited, which could promote menu fatigue. Bioactive compounds can degrade like other chemical compounds and lose functionality. The native concentrations and stability of bioactive compounds have never been determined in spaceflight foods, and adequate information is not available for commercial products for the storage durations required for space exploration (5 years). The purpose of this task is to develop new spaceflight foods that are high in omega-3 fatty acids, lycopene, or flavonoids, identify commercial products with these bioactive compounds that meet spaceflight requirements, and define the stability of these nutrients in storage to enable purposeful functional food incorporation into the space food system. The impact of storage temperature on the stability of lutein, lycopene, beta-carotene, omega-3 fatty acids, phenolics, anthocyanins and sterols is being studied in 12 ISS menu items stored at three different temperatures (4, 21, 35 degree C) over 2 years. Additionally, nutrient and quality stability are being assessed on a larger food set stored at 21 degree C over 2 years that contains twelve newly developed foods, 10 commercial products repackaged to spaceflight requirements, and another 5 current ISS menu items expected to be good sources of omega-3 fatty acids, lycopene, or flavonoids. All items were shipped overnight to the Linus Pauling Institute at Oregon State University (Corvalis, OR) after processing and 1-year of storage and analyzed for bioactive compound concentrations. Sensory evaluation was conducted on the newly developed functional foods and commercial products with untrained panelists (n is greater than or equal to 25) using a 9-point Hedonic scale to test sensory attributes and overall acceptability after processing and 1-year of storage (21 degree C). Repeat nutritional and sensory analyses will be conducted in the same foods after the 2-year storage period is completed. The stability of bioactive compounds in the selected foods was dependent on storage temperature and food matrix. Omega-3 showed excellent stability in the analyzed products after 1-year of storage, regardless of the storage temperature; phenolic compounds also showed good stability. Lycopene was more stable in oil-based products rather than water-based products because of the protection that lipids offer to lycopene molecules. Also, lycopene was more stable in freeze-dried products than in high moisture foods. The 12 newly developed functional foods showed good overall acceptability in sensory attributes after processing (average score 7.2 out of 9.0) and maintained sensory quality through 1-year (21 degree C); the overall acceptability was on average 7.1 after storage. Similar behavior was observed for the 10 commercial products after 1 year. The developed products are good sources of omega-3 (both plant and marine), vegetables (7 vegetable-based products), and good sources of carotenoids, such as the Curry Pumpkin Soup and the Sweet and Savory Kale. Nine of the new products, such as Mango Salad, Pickled Beets, and Braised Red Cabbage, are rich in phenolic compounds. Stability of most of the studied nutrients seems to be adequate after 1-year of storage in most of the tested foods. However, storage temperature of the food must be considered during long-duration space missions to achieve stability of all nutrients. Likewise, more information is needed regarding nutrient retention after 2-years of storage to identify nutritional gaps that may be expected over the 5-year shelf life required for a Mars mission. New developed products will be filling a gap in the current space food system to minimize menu fatigue, provide specific nutrients to reduce the negative effects of long-duration space missions and maintain crew members' health. Information about bioactive compounds in developed products after 1-year and 2-year of storage will provide the knowledge base for further product development.

Bermudez-Aguirre, D.↗

DOE OSTI2026

FunDiff: diffusion models over function spaces for physics-informed generative modeling

Recent advances in generative modeling-particularly diffusion models and flow matching-have been widely used for synthesizing discrete data such as images and videos. However, adapting these models to physical applications remains challenging, as the quantities of interest are continuous functions governed by complex physical laws. To address this, we introduce FunDiff, an efficient and robust framework for generative modeling in function spaces. FunDiff combines a latent diffusion process with a function autoencoder architecture to handle input functions with varying discretizations, generates continuous functions that can be evaluated at arbitrary locations, and seamlessly incorporate physical priors. These priors are enforced through architectural constraints or physics-informed loss functions, ensuring that generated samples satisfy fundamental physical laws. We theoretically establish minimax optimality guarantees for density estimation in function spaces, demonstrating that diffusion-based estimators achieve optimal convergence rates under suitable regularity conditions. We further demonstrate the practical effectiveness of FunDiff across diverse applications in fluid dynamics and solid mechanics. Empirical results indicate that our method can generate physically consistent samples with high fidelity to the target distribution, and exhibit robustness to noisy and low-resolution data.

Wang, Sifan [Yale University, New Haven, CT (Unite↗

NASA NTRS1979

Temperature and flow velocity of the interplanetary gases along solar radii

The velocity distributions along solar radii for hydrogen and helium in interplanetary space are calculated by using the Danby-Camm formula modified with a loss function. From these distributions the radial temperature and radial flow velocity of the interplanetary gases are determined. The effects of solar gravitation and ionization loss, due to charge exchange and photoionization, on the gas temperature and velocity are described.

Wu, F. M.↗

DOE OSTI2022

Physics-informed Karhunen-Loeve and Neural Network Approximations for Solving Inverse Differential Equation Problems

Here we present the PI-CKL-NN method for parameter estimation in differential equation (DE) models given sparse measurements of the parameters and states. In the proposed approach, the space- or time-dependent parameters are approximated by Karhunen-Loeve (KL) expansions that are conditioned on the parameters’ measurements, and the states are approximated by deep neural networks (DNNs). The unknown weights in the KL expansions and DNNs are found my minimizing the cost function that enforces the measurements of the states the DE constraint. Regularization is achieved by adding the l2 norm of the conditional KL coefficients into the loss function. Our approach assumes that the parameter fields are correlated in space or time and enforces the statistical knowledge (the mean and the covariance function) in addition to the DE constraints and measurements as opposed to the physics-informed neural network (PINN) and other similar physics-informed machine learning methods where only DE constraints and data are used for parameter estimation. We use the PI-CKL-NN method for parameter estimation in an ordinary differential equation with an unknown time-dependent parameter and the one- and two-dimensional partial differential diffusion equations with unknown space-dependent diffusion coefficients. We also demonstrate that PI-CKL-NN is more accurate than the PINN method, especially when the observations of the parameters are very sparse

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

DOE OSTI2023

New machine learning techniques for simulation-based inference: InferoStatic nets, kernel score estimation, and kernel likelihood ratio estimation

We propose an intuitive, machine-learning approach to multiparameter inference, dubbed the InferoStatic Networks (ISN) method, to model the score and likelihood ratio estimators in cases when the probability density can be sampled but not computed directly. The ISN uses a backend neural network that models a scalar function called the inferostatic potential \varphi φ . In addition, we introduce new strategies, respectively called Kernel Score Estimation (KSE) and Kernel Likelihood Ratio Estimation (KLRE), to learn the score and the likelihood ratio functions from simulated data. We illustrate the new techniques with some toy examples and compare to existing approaches in the literature. We mention en passant some new loss functions that optimally incorporate latent information from simulations into the training procedure.

Kong, Kyoungchul↗

DOE OSTI2023

Codebase release 0.1 for infstat

Kong, Kyoungchul↗

DOE OSTI2022

New Machine Learning Techniques for Simulation-Based Inference: InferoStatic Nets, Kernel Score Estimation, and Kernel Likelihood Ratio Estimation

We propose an intuitive, machine-learning approach to multiparameter inference, dubbed the InferoStatic Networks (ISN) method, to model the score and likelihood ratio estimators in cases when the probability density can be sampled but not computed directly. The ISN uses a backend neural network that models a scalar function called the inferostatic potential $\varphi$. In addition, we introduce new strategies, respectively called Kernel Score Estimation (KSE) and Kernel Likelihood Ratio Estimation (KLRE), to learn the score and the likelihood ratio functions from simulated data. We illustrate the new techniques with some toy examples and compare to existing approaches in the literature. We mention en passant some new loss functions that optimally incorporate latent information from simulations into the training procedure.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2026

Transforming the $v$ World: A New Multivariate Transformer Energy Estimator for NOvA

The NOvA Transformer Energy Estimator (Transformer_EE) is a universal machine learning tool currently used to infer the incoming beam neutrino energy and the outgoing lepton energy in both near andfar detectors. It uses a unique, highly flexible framework for simultaneous multivariate prediction that supports many possible loss functions. A spectral reweighting and flattening scheme lessens training bias. A feature noising subroutine enables adversarial-like training, mitigating sensitivities to certain systematic effects at marginal resolution loss at inference time. The state of the Transformer_EE will be reviewed, and its robustness with respect to several NOvA Near and Far Detector systematics highlighted.

Tong, Leon [Minnesota U.] (ORCID:0000000231625965)↗

DOE OSTI2025

Triangle Method for Dense ReLU Layers [SWR-25-72]

This software is an implementation of the methods for initializing and training neural networks to be more efficient per parameter, described more fully below and in the related publication: In theory, depth should make a ReLU network EXPONENTIALLY more efficient by enabling it to produce an exponential number of piecewise linear sections in its output. This reasoning is largely based on the work of mathematicians that have hand-constructed networks that make good use of depth. In practice however, even very deep ReLU networks that have been randomly initialized will behave identically to their shallow counterparts - missing an entire exponential dimension of efficiency. The triangle method is a first attempt at realizing the exponential potential of deep networks. Instead of randomly setting weights, we force pairs of neurons in each layer learn to build triangles (i.e. functions from [0,1] -> [0,1] that look like triangles). This is a very efficient pattern for generating lots of linear pieces because composing two triangular functions doubles the number of pieces with each composition. The triangle method is more than just a different initialization, it is a new paradigm of training. Instead of making direct updates to the matrix weights, we do an extra step of backpropagation to collect the derivatives of the loss function with respect to the shapes of the triangles, training them to tilt left or right. This process essentially holds the networks hand throughout the loss landscape and forces it to always use depth effectively by producing triangular shapes internally. This can produce several orders of magnitude of improvement on convex one-dimensional regression problems. Much more theoretical work is needed to realize its full potential beyond this context, but the implementation in this repository will still work in arbitrary numbers of dimensions. The file Triangle_Method.py is a generalized form of the method that will build each neuron its own custom 1-d convex activation function (with exponential efficiency). Example usage on one dimensional problems can be found in Example_Usage.ipynb and an example of using this in a real neural network can be found in Example_VGG16_CIFAR10.ipynb.

Milkert, Max [National Renewable Energy Laboratory↗

DOE OSTI2023

Graph Metric Learning Quantifies Morphological Differences between Two Genotypes of Shoot Apical Meristem Cells in Arabidopsis

We present a method for learning “spectrally descriptive” edge weights for graphs. We generalize a previously known distance measure on graphs (Graph Diffusion Distance), thereby allowing it to be tuned to minimize an arbitrary loss function. Because all steps involved in calculating this modified GDD are differentiable, we demonstrate that it is possible for a small neural network model to learn edge weights which minimize loss. We apply this method to discriminate between graphs constructed from shoot apical meristem images of two genotypes of Arabidopsis thaliana specimens: wild-type and trm678 triple mutants with cell division phenotype. Training edge weights and kernel parameters with contrastive loss produces a learned distance metric with large margins between these graph categories. We demonstrate this by showing improved performance of a simple k-nearest-neighbors classifier on the learned distance matrix. We also demonstrate a further application of this method to biological image analysis. Once trained, we use our model to compute the distance between the biological graphs and a set of graphs output by a cell division simulator. Comparing simulated cell division graphs to biological ones allows us to identify simulation parameter regimes which characterize mutant vs. wild-type Arabidopsis cells. We find that trm678 mutant cells are characterized by increased randomness of division planes and decreased ability to avoid previous vertices between cell walls.

59 BASIC BIOLOGICAL SCIENCES↗