Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “activation function”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Logical Activation Functions v.1.1

SAND2024-01501O Logical Activation Functions software is a PyTorch implementation from the paper, "Logical Activation Functions for Training Arbitrary Probabilistic Boolean Logic." The activation functions approximate logit-space marginalization of probabilistic truth tables from probabilistic interpretations of inputs. They also provide a general methodology to approximate logical relationships between abstract antecedents and consequents for machine learning architectures. They do not target any specific application or use-case. By training probabilistic truth tables, these activation functions can capture more expressive relationships in a neural network than typical elementwise activation functions. This code is only designed for a single compute node with a GPU and is limited to machine learning architectures than can fit within the memory of a single GPU. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Duersch, Jed↗
DOE OSTI2025

SineKAN: Kolmogorov-Arnold Networks using sinusoidal activation functions

Recent work has established an alternative to traditional multi-layer perceptron neural networks in the form of Kolmogorov-Arnold Networks (KAN). The general KAN framework uses learnable activation functions on the edges of the computational graph followed by summation on nodes. The learnable edge activation functions in the original implementation are basis spline functions (B-Spline). Here, we present a model in which learnable grids of B-Spline activation functions are replaced by grids of re-weighted sine functions (SineKAN). We evaluate numerical performance of our model on a benchmark vision task. We show that our model can perform better than or comparable to B-Spline KAN models and an alternative KAN implementation based on periodic cosine and sine functions representing a Fourier Series. Further, we show that SineKAN has numerical accuracy that could scale comparably to dense neural networks (DNNs). Compared to the two baseline KAN models, SineKAN achieves a substantial speed increase at all hidden layer sizes, batch sizes, and depths. Current advantage of DNNs due to hardware and software optimizations are discussed along with theoretical scaling. Additionally, properties of SineKAN compared to other KAN implementations and current limitations are also discussed.

Reinhardt, Eric↗
DOE OSTI2022

GAAF: Searching Activation Functions for Binary Neural Networks Through Genetic Algorithm

Binary neural networks (BNNs) show promising utilization in cost and power-restricted domains such as edge devices and mobile systems. This is due to its significantly less computation and storage demand, but at the cost of degraded performance. To close the accuracy gap, in this paper we propose to add a complementary activation function (AF) ahead of the sign based binarization, and rely on the genetic algorithm (GA) to automatically search for the ideal AFs. These AFs can help extract extra information from the input data in the forward pass, while allowing improved gradient approximation in the backward pass. Fifteen novel AFs are identified through our GA-based search, while most of them show improved performance (up to 2.54% on ImageNet) when testing on different datasets and network models. Interestingly, periodic functions are identified as a key component for most of the discovered AFs, which rarely exist in human designed AFs. Our method offers a novel approach for designing general and application-specific BNN architecture.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

In Vitro Encapsulation of Functionally Active Abiotic Photosensitizers Inside a Bacterial Microcompartment Shell

Bacterial microcompartments (BMCs) are self-assembling, selectively permeable protein shells that encapsulate enzymes to enhance catalytic efficiency of segments of metabolic pathways through means of confinement. The modular nature of BMC shells' structure and assembly enables programming of shell permeability and underscores their promise in biotechnology engineering efforts for applications in industry, medicine, and clean energy. Realizing this potential requires methods for encapsulation of abiotic molecules, which have been developed here for the first time. We report in vitro cargo loading of BMC shells with ruthenium photosensitizers (RuPS) by two approaches-one involving site-specific covalent labeling and the other driven by diffusion, requiring no specific interactions between cargo molecules and shell proteins. The highly stable shells retain encapsulated cargo over 1 week without egress and preserve RuPS photophysical activity. Finally, this study is an important foundation for further work that will converge biological BMC architecture with synthetic chemistry to facilitate biohybrid photocatalysis.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2024

Systems and methods for dynamically reconfigurable artificial synapses and neurons with tunable activation functions

An ionic redox transistor comprises a solid channel, a solid reservoir layer, and a solid electrolyte layer disposed between the channel and the reservoir layer. The channel exhibits a substantially linear current-voltage relationship in a first range of voltages, and a nonlinear current-voltage relationship in a second range of voltages that is greater than the first range of voltages. One or both of the substantially linear current-voltage relationship or the nonlinear current-voltage relationship of the channel is varied by changing the concentration of ions such as oxygen vacancies in the channel. Ion or vacancy transport between the channel and the reservoir layer across the electrolyte layer occurs in response to applying a voltage between the channel and the reservoir layer. Subject to the first range of voltages, the channel can function as a synapse device. Subject to the second range of voltages, the channel can function as a neuron device.

Talin, Albert Alec↗
DOE OSTI2023

A unified and constructive framework for the universality of neural networks

Abstract One of the reasons why many neural networks are capable of replicating complicated tasks or functions is their universal approximation property. Though the past few decades have seen tremendous advances in theories of neural networks, a single constructive and elementary framework for neural network universality remains unavailable. This paper is an effort to provide a unified and constructive framework for the universality of a large class of activation functions including most of the existing ones. At the heart of the framework is the concept of neural network approximate identity (nAI). The main result is as follows: any nAI activation function is universal in the space of continuous functions on compacta. It turns out that most of the existing activation functions are nAI, and thus universal. The framework induces several advantages over the contemporary counterparts. First, it is constructive with elementary means from functional analysis, probability theory, and numerical analysis. Second, it is one of the first unified and constructive attempts that is valid for most of the existing activation functions. Third, it provides new proofs for most activation functions. Fourth, for a given activation and error tolerance, the framework provides precisely the architecture of the corresponding one-hidden neural network with a predetermined number of neurons and the values of weights/biases. Fifth, the framework allows us to abstractly present the first universal approximation with a favorable non-asymptotic rate. Sixth, our framework also provides insights into the developments, and hence providing constructive derivations, of some of the existing approaches.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Adaptive Interface-PINNs (AdaI-PINNs) for inverse problems: Determining material properties for heterogeneous systems

Here, we determine spatially varying discontinuous material properties using a domain-decomposition based physics-informed neural networks (PINNs) framework named the Adaptive Interface-PINNs or AdaI-PINNs (Roy et al., 2024). We propose the use of distinct neural networks for the field variables and material properties within each material, utilizing adaptive activation functions. While the neural networks across different materials share the same weights and biases, their activation functions are uniquely tailored using a hyperparameter that influences the slope of the activation function. The proposed framework is tested on several one-dimensional and two-dimensional benchmark examples, and its performance is compared with conventional PINNs and existing domain-decomposition PINNs frameworks, namely, the Multi-domain physics-informed neural network (M-PINN), and the eXtended physics-informed neural networks (XPINNs). The results demonstrate that the proposed approach can determine randomly distributed discontinuous material properties with an L 2 error of $\mathscr{O}$ (10 -3 ) for the material property and the root-mean-square error of $\mathscr{O}$ (10 -3 ) for the primary variable while the other approaches yield errors that are approximately two orders of magnitude larger (that is, $\mathscr{O}$ (10 -1 )). Moreover, the spatial distribution of material properties obtained using the proposed framework is in close agreement with the true distribution, whereas the other approaches fare much worse. Additionally, the proposed approach is approximately 40% faster than its competitors, indicating its potential as a robust alternative for solving inverse problems in heterogeneous materials.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Adaptive Interface-PINNs (AdaI-PINNs): An Efficient Physics-Informed Neural Networks Framework for Interface Problems

Here, we present an efficient physics-informed neural networks (PINNs) framework, termed Adaptive Interface-PINNs (AdaI-PINNs), to improve the modeling of interface problems with discontinuous coefficients and/or interfacial jumps. This framework is an enhanced version of its predecessor, Interface PINNs or I-PINNs (Sarma et al.; https://doi.org/10.1016/j.cma.2024.117135), which involves domain decomposition and assignment of different predefined activation functions to the neural networks in each subdomain across a sharp interface, while keeping all other parameters of the neural networks identical. In AdaI-PINNs, the activation functions vary solely in their slopes, which are trained along with the other parameters of the neural networks. This makes the AdaI-PINNs framework fully automated without requiring preset activation functions. Comparative studies on one-dimensional, two-dimensional, and three-dimensional benchmark elliptic interface problems reveal that AdaI-PINNs outperform I-PINNs, reducing computational costs by 2-6 times while producing similar or better accuracy.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Synthesis and Exploratory Catalysis of 3d Metals: Atom and Group-Transfer Reactions and the Activation and Functionalization of Small Molecules Including Greenhouse Gases

Determining ways to convert natural gas, with zero emissions, and to more value-added materials such as olefins is one of the main goals in my research group. Over this funding period, we explored the chemistry of early-transition metals with metal-nitrogen multiple bonds, specifically titanium and zirconium nitrides, and explored their redox properties, basicity and reactivity with small molecules including greenhouses gases such as carbon dioxide. Some of these work serve as inspiration for the chemistry of important materials such as uranium nitride along with its unprecedented basicity and ability to activate C-H bonds. Using robust chelating templates we also examined rare examples of trivalent group 4 transition metal ions, and studied these spectroscopically. Using early transition metal ions we also explored synthetic routes to new phosphide based products using the phosphaethynolate salt. In addition, we also explored the chemistry of ferrous systems, and redox active ligand that can allow us to reversibly break and make C-S as well as N-N bonds, activate dinitrogen forming unusual MNNM topologies. Our final component describes how we activate methane and dehydrocouple if with a carbene source to form ethylene. Using this information, we also discovered a simple to make iridium catalyst that activate and functionalize methane with 9:1 selectivity for mono-functionalization. We have established a pathway that leads to poisoning of the catalyst and have found optimal conditions for higher selectivity and with over 170 Turnovers.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2023

Integration of Ag-CBRAM crossbars and Mott ReLU neurons for efficient implementation of deep neural networks in hardware

In-memory computing with emerging non-volatile memory devices (eNVMs) has shown promising results in accelerating matrix-vector multiplications. However, activation function calculations are still being implemented with general processors or large and complex neuron peripheral circuits. Here, we present the integration of Ag-based conductive bridge random access memory (Ag-CBRAM) crossbar arrays with Mott rectified linear unit (ReLU) activation neurons for scalable, energy and area-efficient hardware (HW) implementation of deep neural networks. We develop Ag-CBRAM devices that can achieve a high ON/OFF ratio and multi-level programmability. Compact and energy-efficient Mott ReLU neuron devices implementing ReLU activation function are directly connected to the columns of Ag-CBRAM crossbars to compute the output from the weighted sum current. We implement convolution filters and activations for VGG-16 using our integrated HW and demonstrate the successful generation of feature maps for CIFAR-10 images in HW. Our approach paves a new way toward building a highly compact and energy-efficient eNVMs-based in-memory computing system.

Mott insulators↗
DOE OSTI2023

Activity‐Based Protein Profiling – Enabling Phenotyping of Host‐Associated and Environmental Microbiomes

Abstract Host‐associated and environmental microbiomes play central roles in human health, biogeochemical cycling, response to ecosystem change, and agriculture. Scientific approaches that can reveal the functional activities that contribute to observed phenotypes are needed in microbiome research. Broad characterization of the functional activity of microbes within microbiomes is currently hampered by approaches that rely on inference of function from metagenomes or indirect measurements. Activity‐based protein profiling is uniquely positioned to overcome these challenges and reveal the protein‐level mechanisms associated with microbiome phenotypes. In this review we describe the progress made to date using ABPP in gut microbiome, plant‐microbe interaction, and soil microbiome research, and suggest how ABPP data can be coupled with advanced computational methods to enable phenotype prediction and bioengineering of microbiomes for applied purposes.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

ReLU, Sparseness, and the Encoding of Optic Flow in Neural Networks

Accurate self-motion estimation is critical for various navigational tasks in mobile robotics. Optic flow provides a means to estimate self-motion using a camera sensor and is particularly valuable in GPS- and radio-denied environments. The present study investigates the influence of different activation functions—ReLU, leaky ReLU, GELU, and Mish—on the accuracy, robustness, and encoding properties of convolutional neural networks (CNNs) and multi-layer perceptrons (MLPs) trained to estimate self-motion from optic flow. Our results demonstrate that networks with ReLU and leaky ReLU activation functions not only achieved superior accuracy in self-motion estimation from novel optic flow patterns but also exhibited greater robustness under challenging conditions. The advantages offered by ReLU and leaky ReLU may stem from their ability to induce sparser representations than GELU and Mish do. Our work characterizes the encoding of optic flow in neural networks and highlights how the sparseness induced by ReLU may enhance robust and accurate self-motion estimation from optic flow.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

EPR and 31 P ENDOR Characterization of Pseudo-Jahn–Teller Dynamics and N 2 Activation in Functional Nitrogenase Models, P 3 E M(N 2 ) (M = Fe, Co; E = Si, B, C)

Here, the nominally trigonal, pseudo-Jahn-Teller (PJT)-active, S = ½ N 2 -bound transition-metal complexes, P 3 E M(N 2 ), M = Fe, Co, with three in-plane phosphine-ligands and axial donors, E = Si, B, C, include functional nitrogenase models that catalyze reduction of N 2 to NH 3 . We applied EPR, 31 P ENDOR spectroscopy and DFT computations to characterize the PJT-induced distortions of four selected P 3 E M(N 2 ), revealing how the metal-ion and axial ligand E together tune both PJT dynamics and N 2 activation for reduction. Comparisons reveal an unrecognized correlation between PJT distortion, M-E bond elasticity, and N 2 activation, providing guidelines for designing bioinspired N 2 -reduction catalysts.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2024

Waning immunity and IgG4 responses following bivalent mRNA boosting

Messenger RNA (mRNA) vaccines were highly effective against the ancestral SARS-CoV-2 strain, but the efficacy of bivalent mRNA boosters against XBB variants was substantially lower. Here, we show limited durability of neutralizing antibody (NAb) responses against XBB variants and isotype switching to immunoglobulin G4 (IgG4) responses following bivalent mRNA boosting. Bivalent mRNA boosting elicited modest XBB.1-, XBB.1.5-, and XBB.1.16-specific NAbs that waned rapidly within 3 months. In contrast, bivalent mRNA boosting induced more robust and sustained NAbs against the ancestral WA1/2020 strain, suggesting immune imprinting. Following bivalent mRNA boosting, serum antibody responses were primarily IgG2 and IgG4 responses with poor Fc functional activity. In contrast, a third monovalent mRNA immunization boosted all isotypes including IgG1 and IgG3 with robust Fc functional activity. These data show substantial immune imprinting for the ancestral spike and isotype switching to IgG4 responses following bivalent mRNA boosting, with important implications for future booster designs and boosting strategies.

Science & Technology - Other Topics↗