Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Activation function”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Logical Activation Functions v.1.1

SAND2024-01501O Logical Activation Functions software is a PyTorch implementation from the paper, "Logical Activation Functions for Training Arbitrary Probabilistic Boolean Logic." The activation functions approximate logit-space marginalization of probabilistic truth tables from probabilistic interpretations of inputs. They also provide a general methodology to approximate logical relationships between abstract antecedents and consequents for machine learning architectures. They do not target any specific application or use-case. By training probabilistic truth tables, these activation functions can capture more expressive relationships in a neural network than typical elementwise activation functions. This code is only designed for a single compute node with a GPU and is limited to machine learning architectures than can fit within the memory of a single GPU. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Duersch, Jed↗
DOE OSTI2020

Locally adaptive activation functions with slope recovery for deep and physics-informed neural networks

Here we propose two approaches of locally adaptive activation functions namely, layer-wise and neuron-wise locally adaptive activation functions, which improve the performance of deep and physics-informed neural networks. The local adaptation of activation function is achieved by introducing a scalable parameter in each layer (layer-wise) and for every neuron (neuron-wise) separately, and then optimizing it using a variant of stochastic gradient descent algorithm. In order to further increase the training speed, an activation slope-based slope recovery term is added in the loss function, which further accelerates convergence, thereby reducing the training cost. On the theoretical side, we prove that in the proposed method, the gradient descent algorithms are not attracted to sub-optimal critical points or local minima under practical conditions on the initialization and learning rate, and that the gradient dynamics of the proposed method is not achievable by base methods with any (adaptive) learning rates. We further show that the adaptive activation methods accelerate the convergence by implicitly multiplying conditioning matrices to the gradient of the base method without any explicit computation of the conditioning matrix and the matrix–vector product. The different adaptive activation functions are shown to induce different implicit conditioning matrices. Furthermore, the proposed methods with the slope recovery are shown to accelerate the training process.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Adaptive Activation Functions Accelerate Convergence in Deep and Physics-informed Neural Networks

We employ adaptive activation functions for regression in deep and physics-informed neural networks (PINNs) to approximate smooth and discontinuous functions as well as solutions of linear and nonlinear partial differential equations. In particular, we solve the nonlinear Klein-Gordon equation, which has smooth solutions, the nonlinear Burgers equation, which can admit high gradient solutions, and the Helmholtz equation. We introduce a scalable hyper-parameter in the activation function, which can be optimized to achieve best performance of the network as it changes dynamically the topology of the loss function involved in the optimization process. The adaptive activation function has better learning capabilities than the traditional one (fixed activation) as it improves greatly the convergence rate, especially at early training, as well as the solution accuracy. To better understand the learning process, we plot the neural network solution in the frequency domain to examine how the network captures successively different frequency bands present in the solution. We consider both forward problems, where the approximate solutions are obtained, as well as inverse problems, where parameters involved in the governing equation are identified. Our simulation results show that the proposed method is a very simple and effective approach to increase the efficiency, robustness and accuracy of the neural network approximation of nonlinear functions as well as solutions of partial differential equations, especially for forward problems. We theoretically prove that in the proposed method, gradient descent algorithms are not attracted to suboptimal critical points or local minima.

machine leaning, Bad minima, Inverse problems, Phy↗
DOE OSTI2021

Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functions

Here we propose a new type of neural networks, Kronecker neural networks (KNNs), that form a general framework for neural networks with adaptive activation functions. KNNs employ the Kronecker product, which provides an efficient way of constructing a very wide network while keeping the number of parameters low. Our theoretical analysis reveals that under suitable conditions, KNNs induce a faster decay of the loss than that by the feed-forward networks. This is also empirically verified through a set of computational examples. Furthermore, under certain technical assumptions, we establish global convergence of gradient descent for KNNs. As a specific case, we propose the Rowdy activation function that is designed to get rid of any saturation region by injecting sinusoidal fluctuations, which include trainable parameters. The proposed Rowdy activation function can be employed in any neural network architecture like feed-forward neural networks, Recurrent neural networks, Convolutional neural networks etc. The effectiveness of KNNs with Rowdy activation is demonstrated through various computational experiments including function approximation using feed-forward neural networks, solution inference of partial differential equations using the physics-informed neural networks, and standard deep learning benchmark problems using convolutional and fully-connected neural networks.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

SineKAN: Kolmogorov-Arnold Networks using sinusoidal activation functions

Recent work has established an alternative to traditional multi-layer perceptron neural networks in the form of Kolmogorov-Arnold Networks (KAN). The general KAN framework uses learnable activation functions on the edges of the computational graph followed by summation on nodes. The learnable edge activation functions in the original implementation are basis spline functions (B-Spline). Here, we present a model in which learnable grids of B-Spline activation functions are replaced by grids of re-weighted sine functions (SineKAN). We evaluate numerical performance of our model on a benchmark vision task. We show that our model can perform better than or comparable to B-Spline KAN models and an alternative KAN implementation based on periodic cosine and sine functions representing a Fourier Series. Further, we show that SineKAN has numerical accuracy that could scale comparably to dense neural networks (DNNs). Compared to the two baseline KAN models, SineKAN achieves a substantial speed increase at all hidden layer sizes, batch sizes, and depths. Current advantage of DNNs due to hardware and software optimizations are discussed along with theoretical scaling. Additionally, properties of SineKAN compared to other KAN implementations and current limitations are also discussed.

Reinhardt, Eric↗
NASA NTRS1972

Methodology for the systems engineering process. Volume 1: System functional activities

Systems engineering is examined in terms of functional activities that are performed in the conduct of a system definition/design, and system development is described in a parametric analysis that combines functions, performance, and design variables. Emphasis is placed on identification of activities performed by design organizations, design specialty groups, as well as a central systems engineering organizational element. Identification of specific roles and responsibilities for doing functions, and monitoring and controlling activities within the system development operation are also emphasized.

Nelson, J. H.↗
DOE OSTI2021

Microbiota functional activity biosensors for characterizing nutrient metabolism in vivo

Methods for measuring gut microbiota biochemical activities in vivo are needed to characterize its functional states in health and disease. To illustrate one approach, an arabinan-containing polysaccharide was isolated from pea fiber, its structure defined, and forward genetic and proteomic analyses used to compare its effects, versus unfractionated pea fiber and sugar beet arabinan, on a human gut bacterial strain consortium in gnotobiotic mice. We produced ‘Microbiota Functional Activity Biosensors’ (MFABs) consisting of glycans covalently linked to the surface of fluorescent paramagnetic microscopic glass beads. Three MFABs, each containing a unique glycan/fluorophore combination, were simultaneously orally gavaged into gnotobiotic mice, recovered from their intestines, and analyzed to directly quantify bacterial metabolism of structurally distinct arabinans in different human diet contexts. Colocalizing pea-fiber arabinan and another polysaccharide (glucomannan) on the bead surface enhanced in vivo degradation of glucomannan. MFABs represent a potentially versatile platform for developing new prebiotics and more nutritious foods.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

GAAF: Searching Activation Functions for Binary Neural Networks Through Genetic Algorithm

Binary neural networks (BNNs) show promising utilization in cost and power-restricted domains such as edge devices and mobile systems. This is due to its significantly less computation and storage demand, but at the cost of degraded performance. To close the accuracy gap, in this paper we propose to add a complementary activation function (AF) ahead of the sign based binarization, and rely on the genetic algorithm (GA) to automatically search for the ideal AFs. These AFs can help extract extra information from the input data in the forward pass, while allowing improved gradient approximation in the backward pass. Fifteen novel AFs are identified through our GA-based search, while most of them show improved performance (up to 2.54% on ImageNet) when testing on different datasets and network models. Interestingly, periodic functions are identified as a key component for most of the discovered AFs, which rarely exist in human designed AFs. Our method offers a novel approach for designing general and application-specific BNN architecture.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2020

A partially disordered region connects gene repression and activation functions of EZH2

Enhancer of Zeste Homolog 2 (EZH2) is the catalytic subunit of Polycomb Repressive Complex 2 (PRC2), which minimally requires two other subunits, EED and SUZ12, for enzymatic activity. EZH2 has been traditionally known to mediate histone H3K27 trimethylation, a hallmark of silent chromatin. Emerging evidence indicates that EZH2 also activates gene expression in cancer cells in a context distinct from canonical PRC2. The molecular mechanism underlying the functional conversion of EZH2 from a gene repressor to an activator is unclear. In this work, we show that EZH2 harbors a hidden, partially disordered transactivation domain (TAD) capable of interacting with components of active transcription machinery, mimicking archetypal acidic activators. The EZH2 TAD comprises the SRM (Stimulation-Responsive Motif) and SANT1 (SWI3, ADA2, N-CoR, and TFIIIB 1) regions that are normally involved in H3K27 methylation. The crystal structure of an EZH2–EED binary complex indicates that the EZH2 TAD mediates protein oligomerization in a noncanonical PRC2 context and is entirely sequestered. The EZH2 TAD can be unlocked by cancer-specific EZH2 phosphorylation events to undergo structural transitions that may enable subsequent transcriptional coactivator binding. The EZH2 TAD directly interacts with the transcriptional coactivator and histone acetyltransferase p300 and activates gene expression in a p300-dependent manner in cells. The corresponding TAD may also account for the gene activation function of EZH1, the paralog of EZH2. Distinct kinase signaling pathways that are known to abnormally convert EZH2 into a gene activator in cancer cells can now be understood in a common structural context of the EZH2 TAD.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2021

The role of thermodynamic features on the functional activity of electron bifurcating enzymes

Electron bifurcation is a biological mechanism to drive a thermodynamically unfavorable redox reaction through direct coupling with an exergonic reaction. This process allows microorganisms to generate high energy reducing equivalents in order to sustain life and is often found in anaerobic metabolism, where the energy economy of the cell is poor. Recent work has revealed details of the redox energy landscapes for a variety of electron bifurcating enzymes, greatly expanding the understanding of how energy is transformed by this unique mechanism. Here we highlight the plasticity of these emerging landscapes, what is known regarding their mechanistic underpinnings, and provide a context for interpreting their biochemical activity within the physiological framework. Here, we conclude with an outlook for propelling the field toward an integrative understanding of the impact of electron bifurcation.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

In Vitro Encapsulation of Functionally Active Abiotic Photosensitizers Inside a Bacterial Microcompartment Shell

Bacterial microcompartments (BMCs) are self-assembling, selectively permeable protein shells that encapsulate enzymes to enhance catalytic efficiency of segments of metabolic pathways through means of confinement. The modular nature of BMC shells' structure and assembly enables programming of shell permeability and underscores their promise in biotechnology engineering efforts for applications in industry, medicine, and clean energy. Realizing this potential requires methods for encapsulation of abiotic molecules, which have been developed here for the first time. We report in vitro cargo loading of BMC shells with ruthenium photosensitizers (RuPS) by two approaches-one involving site-specific covalent labeling and the other driven by diffusion, requiring no specific interactions between cargo molecules and shell proteins. The highly stable shells retain encapsulated cargo over 1 week without egress and preserve RuPS photophysical activity. Finally, this study is an important foundation for further work that will converge biological BMC architecture with synthetic chemistry to facilitate biohybrid photocatalysis.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗