AirNet-SNL: Generalized End-to-End Training of Iterative Reconstruction and Deep Neural Network Regularization for Sparse-Data CT.
Abstract not provided.
Lee, Dennis J↗
SEARCH · Engineering Papers
Results for “network regularization”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
AirNet-SNL: End-To-End Training of Iterative Reconstruction and Deep Neural Network Regularization for Sparse-Data XPCI CT.
Abstract not provided.
Lee, Dennis↗
Three-State Majority-vote Model on Scale-Free Networks and the Unitary Relation for Critical Exponents
Abstract We investigate the three-state majority-vote model for opinion dynamics on scale-free and regular networks. In this model, an individual selects an opinion equal to the opinion of the majority of its neighbors with probability 1 − q , and different to it with probability q . The parameter q is called the noise parameter of the model. We build a network of interactions where z neighbors are selected by each added site in the system, a preferential attachment network with degree distribution k − λ , where λ = 3 for a large number of nodes N . In this work, z is called the growth parameter. Using finite-size scaling analysis, we obtain that the critical exponents $$\beta /\bar{\nu }$$ β / ν ¯ and $$\gamma /\bar{\nu }$$ γ / ν ¯ associated with the magnetization and the susceptibility, respectively. Using Monte Carlo simulations, we calculate the critical noise parameter q c as a function of z for the scale-free networks and obtain the phase diagram of the model. We find that the critical exponents add up to unity when using a special volumetric scaling, regardless of the dimension of the network of interactions. We verify this result by obtaining the critical noise and the critical exponents for the two and three-state majority-vote model on cubic lattice networks.
71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
Nonconvex regularization for sparse neural networks
Convex ℓ 1 regularization using an infinite dictionary of neurons has been suggested for constructing neural networks with desired approximation guarantees, but can be affected by an arbitrary amount of over-parametrization. This can lead to a loss of sparsity and result in networks with too many active neurons for the given data, in particular if the number of data samples is large. As a remedy, in this paper, a nonconvex regularization method is investigated in the context of shallow ReLU networks: We prove that in contrast to the convex approach, any resulting (locally optimal) network is finite even in the presence of infinite data (i.e., if the data distribution is known and the limiting case of infinite samples is considered). Moreover, here we show that approximation guarantees and existing bounds on the network size for finite data are maintained.
97 MATHEMATICS AND COMPUTING↗
Noise-Based Selection of Robust Inherited Model for Accurate Continual Learning
There is a growing demand for an intelligent system to continually learn knowledge from a data stream. Continual learning requires both the preservation of previous knowledge (i.e., avoiding catastrophic forgetting) and the acquisition of new knowledge. Different from previous works that focus only on model adaptation (e.g., regularization, network expansion, memory rehearsal, etc.), we propose a novel training scheme named acquisitive learning (AL), which emphasizes both the knowledge inheritance and knowledge acquisition. AL starts from an elaborately selected model with pre-trained knowledge (the inherited model) and then adapts it to new data using segmented training. The selection is achieved by injecting random noise to various inherited models for better model robustness, which promises higher accuracy in further knowledge acquisition. The approach is validated by the visualization of the loss landscape and quantitative roughness measurement. The combination of the selective inherited model and knowledge acquisition reduces catastrophic forgetting by 10X on the CIFAR-100 dataset.
Du, Xiaocong↗
Posterior Regularized Bayesian Neural Network incorporating soft and hard knowledge constraints
Neural Networks (NNs) have been widely used in supervised learning due to their ability to model complex nonlinear patterns, often presented in high-dimensional data such as images and text. However, traditional NNs often lack the ability for uncertainty quantification. Bayesian NNs (BNNS) could help measure the uncertainty by considering the distributions of the NN model parameters. Besides, domain knowledge is commonly available and could improve the performance of BNNs if it can be appropriately incorporated. In this work, we propose a novel Posterior-Regularized Bayesian Neural Network (PR-BNN) model by incorporating different types of knowledge constraints, such as the soft and hard constraints, as a posterior regularization term. Furthermore, we propose to combine the augmented Lagrangian method and the existing BNN solvers for efficient inference. Furthermore, the experiments in simulation and two case studies about aviation landing prediction and solar energy output prediction have shown the knowledge constraints and the performance improvement of the proposed model over traditional BNNs without the constraints.
14 SOLAR ENERGY↗
Structure-preserving neural networks for the regularized entropy-based closure of a linear, kinetic, radiative transport equation
The main challenge of large-scale numerical simulation of radiation transport is the high memory and computation time requirements of discretization methods for kinetic equations. In this work, we derive and investigate a neural network-based approximation to the entropy-based closure method to accurately compute the solution of the multi-dimensional moment system with a low memory footprint and competitive computational time. We extend methods developed for the standard entropy-based closure to the regularized entropy-based closures. The main idea is to interpret structure-preserving neural network approximations of the regularized entropy-based closure as a two-stage approximation to the original entropy-based closure. We conduct a numerical analysis of this approximation and investigate optimal parameter choices. Our numerical experiments demonstrate that the method has a much lower memory footprint than traditional methods with competitive computation times and simulation accuracy. The code and all trained networks are provided on GitHub.
entropy closure↗
Effects of input gradient regularization on neural networks time-series forecasting of thermal power systems
This study proposes using neural networks, specifically gated recurrent unit (GRU), long-short-term memory (LSTM), and transformer networks, to improve control strategies in a 450 MW coal-fired power plant. However, neural networks face issues of becoming overly dependent on just a few variables to make predictions, which negatively impacts control decisions that rely on the model to determine the value of all manipulated variables. The paper introduces regularization techniques, including noise injection and input gradient regularization, during the training phase. Here, the work presents novel contributions in adapting neural networks to control industrial systems and applying regularization techniques from computer vision to industrial process control. Results demonstrate the effectiveness of input gradient regularization in reducing model dependence on subsets of variables, emphasizing the balance between fidelity and controllability. Further exploration is recommended, including the development of recurrent transformers, closed-loop control testing, and a sensitivity analysis on computer models to provide further insight.
20 FOSSIL-FUELED POWER PLANTS↗
Posterior Regularized Bayesian Neural Network
Traditional NNs often lack the ability for uncertainty quantification. Bayesian NNs(BNNs) could help measure the confidence level by using distributions in NNs modeling. Besides, knowledge is commonly available and could improve the performance of BNNs if it can be properly incorporated. In this work, we propose a novel Posterior-Regularized BNN(PR-BNN) model by incorporating soft and hard constraints as a posterior regularization term. We also propose an augmented Lagrangian method and stochastic optimization algorithm for efficient updating via Monte Carlo sampling. The simulations and case studies for solar PV plants have shown the performance improvement of the proposed model over traditional BNNs.
97 MATHEMATICS AND COMPUTING↗
Posterior Regularized Bayesian Neural Network
Traditional NNs often lack the ability for uncertainty quantification. Bayesian NNs(BNNs) could help measure the confidence level by using distributions in NNs modeling. Besides, knowledge is commonly available and could improve the performance of BNNs if it can be properly incorporated. In this work, we propose a novel Posterior-Regularized BNN(PR-BNN) model by incorporating soft and hard constraints as a posterior regularization term. We also propose an augmented Lagrangian method and stochastic optimization algorithm for efficient updating via Monte Carlo sampling. The simulations and case studies in solar energy prediction have shown the performance improvement of the proposed model over traditional BNNs.
14 SOLAR ENERGY↗
Regularizing Invertible Neural Networks for Airfoil Design Through Dimension Reduction
This work examines how dimension reduction can improve the performance of invertible neural networks (INN) for airfoil design. Design workflows are typically expensive, relying on many evaluations of high fidelity computational fluid dynamics (CFD) models. Furthermore, the inverse design problem is typically ill-posed. That is, multiple valid solutions exist that satisfy the design criteria. Regularization can reduce this inverse design space and simplify the problem. We study the use of subspace-based input dimension reduction to act as a regularizer for the INN model and improve the recovery of new airfoil shapes with desired performance characteristics. We find that the dimension reduction identifies two dominant modes, relating to airfoil thickness and camber, that optimally determine the airfoil's aerodynamics. We demonstrate the capability of the proposed INN model to generate 100 airfoils that satisfy the specific aerodynamic and structural characteristics.
airfoil design↗
Epistatic Net allows the sparse spectral regularization of deep neural networks for inferring fitness functions
Abstract Despite recent advances in high-throughput combinatorial mutagenesis assays, the number of labeled sequences available to predict molecular functions has remained small for the vastness of the sequence space combined with the ruggedness of many fitness functions. While deep neural networks (DNNs) can capture high-order epistatic interactions among the mutational sites, they tend to overfit to the small number of labeled sequences available for training. Here, we developed Epistatic Net (EN), a method for spectral regularization of DNNs that exploits evidence that epistatic interactions in many fitness functions are sparse. We built a scalable extension of EN, usable for larger sequences, which enables spectral regularization using fast sparse recovery algorithms informed by coding theory. Results on several biological landscapes show that EN consistently improves the prediction accuracy of DNNs and enables them to outperform competing models which assume other priors. EN estimates the higher-order epistatic interactions of DNNs trained on massive sequence spaces-a computational problem that otherwise takes years to solve.
Aghazadeh, Amirali (ORCID:0000000302230873)↗
Rapid data-driven model reduction of nonlinear dynamical systems including chemical reaction networks using ℓ 1 -regularization
We develop a new data-driven paradigm for efficient model reduction of a broad class of nonlinear dynamical systems. Our model reduction method directly enables the interpretation of key components of the dynamical system, unlike traditional projection-based model reduction methods that focus on reducing computational complexity more than interpretability. Our method is not application specific and is simple to implement on nonlinear dynamical systems arising from a variety of different fields. It requires minimal parameterization using a single parameter to trade-off between model complexity and estimation error. We use a data-driven paradigm to formulate model reduction as an efficient convex optimization problem that scales polynomially in the original size of the complex system, enabling systems with as many as thousands of components to be reduced in a matter of minutes.
37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
Recurrent neural network-based multiaxial plasticity model with regularization for physics-informed constraints
We report a recurrent neural network (RNN) based model is developed as a surrogate to predict nonlinear plastic response under multiaxial loading. The RNN-based model is trained and tested on stress versus strain curves generated using a numerical solution based on the classical radial return method. Besides simply learning the basic constitutive relationship, a novel approach is taken to enforce certain physical conditions. Specifically, regularization is employed to maintain non-negative plastic power density throughout the loading history thereby ensuring monotonically increasing plastic work and thermodynamic consistency. Enforcing physics in this manner permits coupling of the data-driven RNN approach with physics-based knowledge and laws. This has the effect of reducing the necessary amount of data and ensuring known physical laws are not violated. Since, once trained, the model need not perform the expensive task of solving nonlinear equations, its efficiency is orders of magnitude greater than its numerical counterpart. The RNN-based model has been trained on varied sets of data and the accuracy on test datasets validated. The developed model is general and robust and has widespread application such as in the simulation of metal forming, large scale plasticity, and part life prediction.
42 ENGINEERING↗
Deep Neural Networks are Adaptive to Function Regularity and Data Distribution in Approximation and Estimation
Deep learning has exhibited remarkable results across diverse areas. To understand its success, substantial research has been directed towards its theoretical foundations. Nev- ertheless, the majority of these studies examine how well deep neural networks can model functions with uniform regularities. In this paper, we explore a different angle: how deep neural networks can adapt to varying degrees of smoothness in functions and nonuni- form data distributions across different locations and scales. More precisely, we focus on a broad class of functions defined by nonlinear tree-based approximation methods. This class encompasses a range of function types, such as functions with uniform regularities and discontinuous functions. We develop nonparametric approximation and estimation theories for this class using deep ReLU networks. Our results show that deep neural networks are adaptive to the nonuniform smoothness of functions and nonuniform data distributions at different locations and scales. We apply our results to several function classes, and derive the corresponding approximation and generalization errors. The validity of our results is demonstrated through numerical experiments.
97 MATHEMATICS AND COMPUTING↗
Monomer-dimer tensor-network basis for qubit-regularized lattice gauge theories
Traditional SU(𝑁) lattice gauge theories (LGTs) can be formulated using an orthonormal basis constructed from the irreducible representations (irreps) 𝑉 𝜆 of the SU(𝑁) gauge symmetry. On a lattice, the elements of this basis are tensor networks comprising dimer tensors on the links labeled by a set of irreps {𝜆 ℓ } and monomer tensors on sites labeled by {𝜆 𝑠 }. These tensors naturally define a local site Hilbert space, ℋ$^𝑔_𝑠$, on which gauge transformations act. Gauss’s law introduces an additional index 𝛼 𝑠 =1,2,…,𝒟(ℋ$^𝑔_𝑠$) that labels an orthonormal basis of the gauge-invariant subspace of ℋ$^𝑔_𝑠$. This monomer-dimer tensor-network (MDTN) basis, |{𝜆 𝑠 },{𝜆 ℓ },{𝛼 𝑠 }⟩, of the physical Hilbert space enables the construction of new qubit-regularized SU(𝑁) gauge theories that are free of sign problems while preserving key features of traditional LGTs. Here, we investigate finite-temperature confinement-deconfinement transitions in a simple qubit-regularized SU(2) and SU(3) gauge theory in 𝑑 =2 and 𝑑 =3 spatial dimensions, formulated using the MDTN basis, and show that they reproduce the universal results of traditional LGTs at these transitions. Additionally, in 𝑑 =1, we demonstrate using a plaquette chain that the string tension at zero temperature can be continuously tuned to zero by adjusting a model parameter that plays the role of the gauge coupling in traditional LGTs.
72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
Degrees of Rate Control in Interconnected Reaction Networks
Overall reactions in interconnected networks exhibit net, forward, and reverse rates that are governed by both constitutive elementary steps in the pathway of interest and branching elementary steps that lead to alternative products. Accordingly, steps in branching pathways exhibit negative net, forward, and reverse degrees of rate control, as they reduce reaction flux to the desired product. We here contextualize the forward and reverse degrees of rate control in terms of kinetic resistances (inverse of rates) and leverage the additive nature of kinetic resistance to decouple kinetic driving forces contributed by constitutive elementary steps and branching points (nodal species) in interconnected networks. Regardless of the network connectivity, forward and reverse degrees of rate control are shown to converge at equilibrium. Away from equilibrium, we identify two critical features of interconnected networks: stoichiometric regularity─condition where all stoichiometric numbers are unity─and pathway symmetry around nodal species─condition where branching pathways share the same rate constants, stoichiometry, and species concentrations/activities─that result in (i) equal forward, reverse, and consequently net degrees of rate control and (ii) forward and reverse degrees of rate control that exhibit constant offsets, respectively, across all extents of reaction. Furthermore, our discourse further provides a mathematical description for the influence of stoichiometric irregularity and pathway asymmetry on forward and reverse degrees of rate control. Altogether, the presented work details the effects of network (inter)connectivity and stoichiometry on reaction kinetics and, in doing so, establishes general protocols for capturing these effects as additive terms in the formulation of forward and reverse degrees of rate control.
10 SYNTHETIC FUELS↗
Quantification of neural networks uncertainties with applications to SAFARI-1 axial neutron flux profiles
Deep Neural Networks (DNNs) have been widely used as a data-driven modelling tool in nuclear engineering. However, as a Machine Learning model, Artificial Neural Network (ANN) predictions are subjected to uncertainties originating from the noise in training data, incomplete coverage of the domain, and imperfect neural network architectures. In this work, we target at quantifying the prediction/approximation uncertainties of ANNs using Monte Carlo Dropout (MCD), as well as Bayesian Neural Networks (BNNs) which are solved by variational inference. With a demonstration problem in which neural networks are used to predict the assembly axial neutron flux profiles, the results have shown that the three different neural network models (regular DNNs, DNNs solved with MCD and BNNs) can produce results that agree very well among each other and with the measurement data, on cycles that are not used in the training process. Besides the excellent generalization capability, the uncertainty bands produced by MCD and BNN agree very well, and in general, they can fully envelop the noisy measurement data points. (authors)
21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗