Skip to content
Free planNo credit card requiredStart writing
AI for Research home
AI for Research home
Dashboard
Sign inGet started

Search 250 million research papers

Read the abstract, see who wrote it and who cites it, and get the citation in APA, MLA, Chicago or Harvard.

About 10,000 papers for “activation functions”

Noisy Activation Functions

Çaǧlar Gülçehre, Marcin Moczulski, Misha Denil et al. · arXiv (Cornell University) · 2016

Common nonlinear activation functions used in neural networks can cause training difficulties due to the saturation behavior of the activation function, which may hide dependencies that are not visible to vanilla-SGD (using first order gradients only). Gating mechanisms that use softly saturating activation functions to emulate the discrete switching of digital logic circuits are good examples of this. We propose to exploit the injection of appropriate noise so that the gradients may flow easily, even if the noiseless application of the activation function would yield zero gradient. Large noise will dominate the noise-free gradient and allow stochastic gradient descent toexplore more. By adding noise only to the problematic parts of the activation function, we allow the optimization procedure to explore the boundary between the degenerate (saturating) and the well-behaved parts of the activation function. We also establish connections to simulated annealing, when the amount of noise is annealed down, making it easier to optimize hard objective functions. We find experimentally that replacing such saturating activation functions by noisy variants helps training in many contexts, yielding state-of-the-art or competitive results on different datasets and task, especially when training seems to be the most difficult, e.g., when curriculum learning is necessary to obtain good results.

Cited by 82Open access

Conic Activation Functions

Changqing Fu, Laurent D. Cohen · SPIRE - Sciences Po Institutional REpository · 2024

International audience

Cited by 0Open access

Properties of activation functions.

Osvaldo Matias Velarde (17373753), Hernán A. Makse (8763285), Lucas C. Parra (12840692) · Figshare · 2023

<div><p>In the visual system of primates, image information propagates across successive cortical areas, and there is also local feedback within an area and long-range feedback across areas. Recent findings suggest that the resulting temporal dynamics of neural activity are crucial in several vision tasks. In contrast, artificial neural network models of vision are typically feedforward and do not capitalize on the benefits of temporal dynamics, partly due to concerns about stability and computational costs.</p><p>In this study, we focus on recurrent networks with feedback connections for visual tasks with static input corresponding to a single fixation. We demonstrate mathematically that a network’s dynamics can be stabilized by four key features of biological networks: layer-ordered structure, temporal delays between layers, longer distance feedback across layers, and nonlinear neuronal responses. Conversely, when feedback has a fixed distance, one can omit delays in feedforward connections to achieve more efficient artificial implementations.</p><p>We also evaluated the effect of feedback connections on object detection and classification performance using standard benchmarks, specifically the COCO and CIFAR10 datasets. Our findings indicate that feedback connections improved the detection of small objects, and classification performance became more robust to noise. We found that performance increased with the temporal dynamics, not unlike what is observed in core vision of primates.</p><p>These results suggest that delays and layered organization are crucial features for stability and performance in both biological and artificial recurrent neural networks.</p></div>

Cited by 0Open access

Mining Generalizable Activation Functions

Alex Vitvitskyi, Michael Boratko, Matej Grcić et al. · arXiv (Cornell University) · 2026

The choice of activation function is an active area of research, with different proposals aimed at improving optimization, while maintaining expressivity. Additionally, the activation function can significantly alter the implicit inductive bias of the architecture, controlling its non-linear behavior. In this paper, in line with previous work, we argue that evolutionary search provides a useful framework for finding new activation functions, while we also make two novel observations. The first is that modern pipelines, such as AlphaEvolve, which relies on frontier LLMs as a mutator operator, allows for a much wider and flexible search space; e.g., over all possible python functions within a certain FLOP budget, eliminating the need for manually constructed search spaces. In addition, these pipelines will be biased towards meaningful activation functions, given their ability to represent common knowledge, leading to a potentially more efficient search of the space. The second observation is that, through this framework, one can target not only performance improvements but also activation functions that encode particular inductive biases. This can be done by using performance on out-of-distribution data as a fitness function, reflecting the degree to which the architecture respects the inherent structure in the data in a manner independent of distribution shifts. We carry an empirical exploration of this proposal and show that relatively small scale synthetic datasets can be sufficient for AlphaEvolve to discover meaningful activations.

Cited by 0Open access

Activation and Penalty Functions

Robert Dunne · 2006

This chapter contains sections titled: Introduction Interpreting Outputs as Probabilities The “Universal Approximator” and Consistency Variance and Bias Binary Variables and Logistic Regression MLP Models and Cross—Entropy A Derivation of the Softmax Activation Function The “Natural” Pairing and ΔQ A Comparison of Least Squares and Cross—Entropy Conclusion Complements and Exercises

Cited by 1

Mining Generalizable Activation Functions

Alex Vitvitskyi, Michael Boratko, Matej Grcić et al. · arXiv (Cornell University) · 2026

The choice of activation function is an active area of research, with different proposals aimed at improving optimization, while maintaining expressivity. Additionally, the activation function can significantly alter the implicit inductive bias of the architecture, controlling its non-linear behavior. In this paper, in line with previous work, we argue that evolutionary search provides a useful framework for finding new activation functions, while we also make two novel observations. The first is that modern pipelines, such as AlphaEvolve, which relies on frontier LLMs as a mutator operator, allows for a much wider and flexible search space; e.g., over all possible python functions within a certain FLOP budget, eliminating the need for manually constructed search spaces. In addition, these pipelines will be biased towards meaningful activation functions, given their ability to represent common knowledge, leading to a potentially more efficient search of the space. The second observation is that, through this framework, one can target not only performance improvements but also activation functions that encode particular inductive biases. This can be done by using performance on out-of-distribution data as a fitness function, reflecting the degree to which the architecture respects the inherent structure in the data in a manner independent of distribution shifts. We carry an empirical exploration of this proposal and show that relatively small scale synthetic datasets can be sufficient for AlphaEvolve to discover meaningful activations.

Cited by 0Open access

Activation functions with learnable amplitude

TRENTIN EDMONDO · Use Siena air (University of Siena) · 1999

Network training algorithms have heavily concentrated on the learning of connection weights. Little effort has been made to learn the amplitude of the activation functions, which defines the range of values that the function can take. This paper introduces novel algorithms to learn the amplitudes of non-linear activations in layered networks, without any assumption on their analytical form. Three instances of the algorithms are developed: (i) a common amplitude is shared among all the nonlinear units; (ii) each layer has its own amplitude; (iii) neuron-specific amplitudes are allowed. Experimental results validate the approach to a large extent, showing a dramatic improvement in performance over the nets with fixed amplitudes.

Cited by 0

Activation Functions: Experimentation and Comparison

Disha Gangadia · 2021

Activation functions are mathematical functions that are used to activate the neurons of an Artificial Neural Network. Non-linear activation functions mainly help a neural network to converge faster while learning and finding patterns in the complex input data. A neural network learns by updating the weights, which is done using the Back Propagation algorithm, which uses first-order derivatives of the activation functions to calculate the gradient descent. This paper tests various existing and proposed activation functions against Minst and Cifar10 datasets for image classification using a shallow Convolutional Neural Network (CNN) Architecture. Based on the results, some of the proposed activation functions: SMod = x <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">*</sub> tanh ( x), the Absolute/Mod Function, a scaled version of Swish, and some other activation functions, are found to be promising. Some of these are then tested against Deeper Neural Networks for various datasets, and it is observed that the average error rate is improved by 2.77. Along with that, suggestions on which activation functions to be used for shallow and deep layers of a Deep Neural Network are provided, resulting in better performance.

Cited by 4

Deriving Activation Functions Using Integration

Allen Hao Huang, Schlag, Imanol · arXiv (Cornell University) · 2024

Our work proposes a novel approach to designing activation functions by focusing on their gradients and deriving the corresponding activation functions using integration. We introduce the Expanded Integral of the Exponential Linear Unit (xIELU), a trainable piecewise activation function derived by integrating trainable affine transformations applied to the Exponential Linear Unit (ELU). xIELU combines two key properties for the gradient: (1) a trainable and linearly increasing gradient for positive inputs, similar to Squared ReLU (ReLU$^2$), and (2) a trainable gradient that can take negative values for negative inputs, inspired by Expanded SiLU (xSiLU). Conceptually, xIELU can be viewed as an extension of ReLU$^2$ to handle negative inputs. The trainable parameters in xIELU allow it to adaptively reduce its nonlinearity for higher-level representations deeper in the network. In experiments with 1.1B and 3B parameter Llama models trained on 125B tokens of FineWeb Edu, xIELU achieves lower perplexity compared to popular activation functions like ReLU$^2$ and SwiGLU when matched for the same compute cost and parameter count. A reference implementation is available at https://github.com/Anonymous5823/xielu.

Cited by 0Open access

Synaptic efficacy and activation functions.

Elif Köksal-Ersöz (20378631), Pascal Benquet (323171), Fabrice Wendling (220532) · Figshare · 2024

&lt;p&gt;(A) Calcium dependent synaptic efficacy function, (2), and (B) activation function, &lt;i&gt;H&lt;/i&gt;(&lt;i&gt;V&lt;/i&gt;&lt;sub&gt;&lt;i&gt;post&lt;/i&gt;&lt;/sub&gt;) (5) for the parameter set given in &lt;a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1012666#pcbi.1012666.t001" target="_blank"&gt;Table 1&lt;/a&gt;.&lt;/p&gt;

Cited by 0Open access