Kunal Banerjee, Vishak Prasad C, Rishi R. Gupta et al. · 2021
Softmax function is widely used in artificial neural networks for multiclass classification, multilabel classification, attention mechanisms, etc. However, its efficacy is often questioned in literature. The log-softmax loss has been shown to belong to a more generic class of loss functions, called spherical family, and its member log-Taylor softmax loss is arguably the best alternative in this class. In another approach which tries to enhance the discriminative nature of the softmax function, soft-margin softmax (SM-softmax) has been proposed to be the most suitable alternative. In this work, we investigate Taylor softmax, SM-softmax and our proposed SM-Taylor softmax, an amalgamation of the earlier two functions, as alternatives to softmax function. Furthermore, we explore the effect of expanding Taylor softmax up to ten terms (original work proposed expanding only to two terms) along with the ramifications of considering Taylor softmax to be a finite or infinite series during backpropagation. Our experiments for the image classification task on different datasets reveal that there is always a configuration of the SM-Taylor softmax function that outperforms the normal softmax function and its other alternatives.
Cited by 1Open access
Rodolfo Zunino, Paolo Gastaldo · 2003
The paper describes the analog implementation of the SoftMax function by using CMOS circuitry. First, an optimization-based strategy is presented, which fits technology requirements to the desired accuracy in the SoftMax-mapping. Then circuit solutions are described to both the approximation of the exp(x) function and the normalizing ratio. These optimized design aspects improve overall effectiveness, reduce VLSI complexity by exploiting inherent parallelism, and ultimately limit overall power consumption.
Cited by 25
Sergios Theodoridis · IEEE RESOURCE CENTERS · 2024
Cited by 0Open access
Michael Franke, Judith Degen · 2023
The softmax function is a ubiquitous helper function, frequently used as a probabilistic link function for unordered categorical data, in different kinds of models, such as regression, artificial neural networks, or probabilistic cognitive models. To fully understand the models in which the softmax function occurs, different levels of understanding of the softmax function itself are necessary. For input-output oriented models, like regression or neural network models, mathematical properties are crucial. For models with interpretable and meaningful internal representations like probabilistic cognitive models, we also require a thorough conceptual understanding of the motivation for using the softmax function (instead of something else). This tutorial provides an in- depth exposition of the informal, mathematical and conceptual properties of the softmax function. It also provides two mathematical derivations (as a stochastic choice model, and as maximum entropy distribution), together with three conceptual interpretations that can serve as rationale for using the softmax function in models that require explainability of modeling choices.
Cited by 18Open access
Makkena Vaishnavi, Anusha KS · 2025
The IEEE 754 floating-point standard is central to digital computing, facilitating numerical computation in different applications efficiently. In this research, a new VLSI-based hardware architecture for the Softmax activation function, a key building block of deep neural networks (DNNs), is proposed. The Softmax function transforms raw neural network outputs to probability distributions used in classification tasks. Conventional hardware realizations, on the other hand, are hampered by power and area inefficiencies, high latencies resulting from the inherent complication of floating-point division and exponentials. Addressing all the above difficulties, we hereunder suggest high-speed VLSI realization making use of pipelined clock-based division mechanism, low overhead exponential calculation component. Pipelined division drastically cuts down computation time at acceptable precision. Experimental results show that our proposed design need 98% On chip power while the actual design need 99%. This work contributes to enhancing neural network hardware accelerators by reducing energy consumption, improving execution speed, ensuring optimal performance in AI systems.
Cited by 0
Yuvarani S, Yuvaraj V, Sakthi Siddharth S et al. · Zenodo (CERN European Organization for Nuclear Research) · 2025
The IEEE 754 floating-point standard is fundamental in digital computing, enabling efficient numerical operations across various applications. This study presents a novel VLSI-based hardware architecture for the Softmax activation function, an essential component in deep neural networks (DNNs). The Softmax function converts raw neural network outputs into probability distributions for classification tasks. However, traditional hardware implementations suffer from high power consumption, latency, and area inefficiency due to the complexity of floating-point division and exponential computations. To address these challenges, we propose a highperformance VLSI design incorporating a pipelined clockbased division technique and an optimized exponential computation unit. The architecture consists of three key modules: (i) an Exponent Module that efficiently computes exponentials using IEEE 754 floating-point representation, (ii) an Adder Tree that rapidly sums exponentials with minimized delay and area, and (iii) a Pipelined Divider that performs normalization efficiently. The pipelined division approach significantly reduces execution time while maintaining high precision. Additionally, we explore an approximate Softmax implementation to achieve further improvements in speed and resource utilization. Comparative analysis with traditional Softmax implementations highlights improvements in power efficiency, accuracy, and latency. Experimental results demonstrate that our proposed design effectively balances computational complexity with hardware efficiency, making it suitable for real-time deep learning applications. This work contributes to enhancing neural network hardware accelerators by reducing energy consumption and improving execution speed, ensuring optimal performance in edge and embedded AI systems.
Cited by 0Open access
Aishwarya Kagalkar, S Raghuram · 2020
Interest has been renewed in the domain of neuromorphic circuits, with the emergence of Deep Neural Networks as the leading model for various classification tasks. In this work, we present a CORDIC based implementation for one of the most important layers, the Softmax layer. The layer is particularly complex for CMOS implementations due to the requirement for multiple evaluations of the exponential function in calculating the output of this layer. We apply the CORDIC method for computing the exponent and the binary division algorithm for calculating the output. It is shown via experiments that our approach for realizing the Softmax function simultaneously out-performs existing implementations using McLaurin series approximations and Padé polynomial approximations from literature in three key metrics: timing, area, and accuracy. We also investigate an inverse form of the Softmax function that avoids the division operation. The implementation can be used to realize the Softmax activation in both ASIC and FPGA implementations of large scale neuromorphic circuits.
Cited by 29
Ioannis Kouretas, Vassilis Paliouras · 2019
In this paper a simplified hardware implementation of a CNN softmax layer is proposed. Initially the softmax activation function is analyzed in terms of required accuracy and certain optimizations are proposed. Subsequently the proposed hardware architecture is evaluated in terms of the introduced approximation error. Finally the proposed circuits are synthesized in a 90-nm 1.0 V CMOS standard-cell library using Synopsys Design Compiler. Comparisons reveal significant reduction up to 47% and 43% for certain cases, in terms of area × delay product over prior art. Area savings are achieved with no performance penalty.
Cited by 95
Yuvarani S, Yuvaraj V, Sakthi Siddharth S et al. · Zenodo (CERN European Organization for Nuclear Research) · 2025
The IEEE 754 floating-point standard is fundamental in digital computing, enabling efficient numerical operations across various applications. This study presents a novel VLSI-based hardware architecture for the Softmax activation function, an essential component in deep neural networks (DNNs). The Softmax function converts raw neural network outputs into probability distributions for classification tasks. However, traditional hardware implementations suffer from high power consumption, latency, and area inefficiency due to the complexity of floating-point division and exponential computations. To address these challenges, we propose a highperformance VLSI design incorporating a pipelined clockbased division technique and an optimized exponential computation unit. The architecture consists of three key modules: (i) an Exponent Module that efficiently computes exponentials using IEEE 754 floating-point representation, (ii) an Adder Tree that rapidly sums exponentials with minimized delay and area, and (iii) a Pipelined Divider that performs normalization efficiently. The pipelined division approach significantly reduces execution time while maintaining high precision. Additionally, we explore an approximate Softmax implementation to achieve further improvements in speed and resource utilization. Comparative analysis with traditional Softmax implementations highlights improvements in power efficiency, accuracy, and latency. Experimental results demonstrate that our proposed design effectively balances computational complexity with hardware efficiency, making it suitable for real-time deep learning applications. This work contributes to enhancing neural network hardware accelerators by reducing energy consumption and improving execution speed, ensuring optimal performance in edge and embedded AI systems.
Cited by 0Open access
Seiya Satoh (16877237), Kenta Yamagishi (16877240), Tatsuji Takahashi (5713172) · Figshare · 2023
<div><p>Neural networks are widely used for classification and regression tasks, but they do not always perform well, nor explicitly inform us of the rationale for their predictions. In this study we propose a novel method of comparing a pair of different feedforward neural networks, which draws on independent components obtained by independent component analysis (ICA) on the hidden layers of these networks. It can compare different feedforward neural networks even when they have different structures, as well as feedforward neural networks that learned partially different datasets, yielding insights into their functionality or performance. We evaluate the proposed method by conducting three experiments with feedforward neural networks that have one hidden layer, and verify whether a pair of feedforward neural networks can be compared by the proposed method when the numbers of hidden units in the layer are different, when the datasets are partially different, and when activation functions are different. The results show that similar independent components are extracted from two feedforward neural networks, even when the three circumstances above are different. Our experiments also reveal that mere comparison of weights or activations does not lead to identifying similar relationships. Through the extraction of independent components, the proposed method can assess whether the internal processing of one neural network resembles that of another. This approach has the potential to help understand the performance of neural networks.</p></div>
Cited by 0Open access
Hao Shao, Shunfang Wang · Entropy · 2023
Recently, there has been a rapid increase in deep classification tasks, such as image recognition and target detection. As one of the most crucial components in Convolutional Neural Network (CNN) architectures, softmax arguably encourages CNN to achieve better performance in image recognition. Under this scheme, we present a conceptually intuitive learning objection function: Orthogonal-Softmax. The primary property of the loss function is to use a linear approximation model that is designed by Gram-Schmidt orthogonalization. Firstly, compared with the traditional softmax and Taylor-Softmax, Orthogonal-Softmax has a stronger relationship through orthogonal polynomials expansion. Secondly, a new loss function is advanced to acquire highly discriminative features for classification tasks. At last, we present a linear softmax loss to further promote the intra-class compactness and inter-class discrepancy simultaneously. The results of the widespread experimental discussion on four benchmark datasets manifest the validity of the presented method. Besides, we want to explore the non-ground truth samples in the future.
Cited by 19Open access
Jiamu Xu, Xixiang Liu, Xinyuan Zhang et al. · arXiv (Cornell University) · 2023
Learning the discriminative features of different faces is an important task in face recognition. By extracting face features in neural networks, it becomes easy to measure the similarity of different face images, which makes face recognition possible. To enhance the neural network's face feature separability, incorporating an angular margin during training is common practice. State-of-the-art loss functions CosFace and ArcFace apply fixed margins between weights of classes to enhance the inter-class separation of face features. Since the distribution of samples in the training set is imbalanced, similarities between different identities are unequal. Therefore, using an inappropriately fixed angular margin may lead to the problem that the model is difficult to converge or the face features are not discriminative enough. It is more in line with our intuition that the margins are angular adaptive, which could increase with the angles between classes growing. In this paper, we propose a new angular margin loss named X2-Softmax. X2-Softmax loss has adaptive angular margins, which provide the margin that increases with the angle between different classes growing. The angular adaptive margin ensures model flexibility and effectively improves the effect of face recognition. We have trained the neural network with X2-Softmax loss on the MS1Mv3 dataset and tested it on several evaluation benchmarks to demonstrate the effectiveness and superiority of our loss function.
Cited by 0Open access
Michael Wasef, Nader Rafla · 2021
The SoftMax activation function is a normalized exponential function that is usually used as an activation function of the last layer of a fully connected neural network. The number of neurons in this layer represents the number of classes. The SoftMax activation function is used to normalize the network outputs to a probability distribution over predicted output classes. In this paper, a multi-rate input SoftMax activation function has been designed and built on FPGA. The unit can read 4 or 2 consecutive inputs or one input, every predefined number of cycles. A ROM design has been utilized to determine the exponential part of the function, while the Coordinate Rotation Digital Computer (CORDIC) reciprocal algorithm has been used to calculate the reciprocal of the sum of the input exponential. Hardware multipliers have been used to calculate the SoftMax output. Unit optimization is achieved by pipelining on the input and output stages. The unit can be configured and controlled by an ARM microcontroller as a complete System-on-Chip (SoC) built on Field Programmable Gate Array (FPGA).
Cited by 8
Jae‐Min Lee, Tae-Young Choe · The Journal of the Korea Contents Association · 2024
분류 문제를 인공지능 방식으로 해결할 때 앙상블은 그 성능을 높이는 기법으로 자주 활용된다. 하지만, 선택된 클래스의 비중을 확률분포로 바꾸는 일반적인 softmax의 확률분포를 그대로 앙상블에 적용하면 softmax내 지수함수의 영향으로 가장 높은 비중을 가진 클래스의 확률이 상대적으로 너무 높아지는 문제가 발생하여 앙상블의 효과를 감소시킬 수 있다. 본 논문에서는 softmax의 이러한 문제점을 완화하기 위해 스무딩 인자를 도입한 smoothed softmax를 제안하며, 이를 통해 앙상블 성능이 개선될 수 있음을 보여준다. 또한 논문에서는 coarse label로 학습한 신경망을 앙상블에 포함하고, 앙상블을 교사 모델로 사용하여 작은 크기의 학생 모델 신경망에 지식 증류할 때 smoothed softmax가 신경망의 정확도를 높임을 보인다. 논문에서 제안한 smoothed softmax는 기존의 softmax를 적용한 경우와 비교해서 fine label로 학습한 신경망으로 앙상블했을 때는 정확도가 1.38% 증가했으며, coarse label로 학습한 신경망이 앙상블에 추가되었을 때는 정확도가 5.21% 증가하였고, 앙상블 신경망을 지식 증류를 통해 그 크기를 감소시켰을 때는 정확도가 최대 7.0%까지 증가하였다.
Cited by 0Open access
Nhat Ho, Huy Nguyen, TrungTin Nguyen · 2023
Cited by 0
Jiahao Xu, Xixiang Liu, Xinyuan Zhang et al. · Expert Systems with Applications · 2024
Cited by 14
Hao Peng, Jianxin Li, Yangqiu Song et al. · Proceedings of the AAAI Conference on Artificial Intelligence · 2017
Neural network language models (NNLMs) have attracted a lot of attention recently. In this paper, we present a training method that can incrementally train the hierarchical softmax function for NNMLs. We split the cost function to model old and update corpora separately, and factorize the objective function for the hierarchical softmax. Then we provide a new stochastic gradient based method to update all the word vectors and parameters, by comparing the old tree generated based on the old corpus and the new tree generated based on the combined (old and update) corpus. Theoretical analysis shows that the mean square error of the parameter vectors can be bounded by a function of the number of changed words related to the parameter node. Experimental results show that incremental training can save a lot of time. The smaller the update corpus is, the faster the update training process is, where an up to 30 times speedup has been achieved. We also use both word similarity/relatedness tasks and dependency parsing task as our benchmarks to evaluate the correctness of the updated word vectors.
Cited by 78Open access
Ioannis Kouretas, Vassilis Paliouras · Technologies · 2020
In this paper a simplified hardware implementation of a CNN softmax-like layer is proposed. Initially the softmax activation function is analyzed in terms of required numerical accuracy and certain optimizations are proposed. A proposed adaptable hardware architecture is evaluated in terms of the introduced error due to the proposed softmax-like function. The proposed architecture can be adopted to the accuracy required by the application by retaining or eliminating certain terms of the approximation thus allowing to explore accuracy for complexity trade-offs. Furthermore, the proposed circuits are synthesized in a 90 nm 1.0 V CMOS standard-cell library using Synopsys Design Compiler. Comparisons reveal that significant reduction is achieved in area × delay and power × delay products for certain cases, respectively, over prior art. Area and power savings are achieved with respect to performance and accuracy.
Cited by 39Open access
David Svitov, Sergey Alyamkin · Automation and Remote Control · 2022
The use of convolutional neural networks trained with the margin-based softmax function allows achieving the highest accuracy in the face recognition problem. The development of embedded systems such as smart intercoms has increased interest in lightweight neural networks. Thus, lightweight neural network models, trained using the margin-based softmax function, were proposed for the face identification problem. In the present paper, we propose a distillation method that allows obtaining greater accuracy than other methods for the face recognition problem on LFW, AgeDB-30, and Megaface datasets. The main idea of our approach is to use the class centers of the teacher network to initialize the student network. Then the student network is trained to produce biometric vectors the angles from which to the class centers are equal to the angles in the teacher network.
Cited by 2
Park, Hanjun, Min, Byeong-Seo, Woo, Jiheon et al. · arXiv (Cornell University) · 2026
Cited by 0Open access