Hanjun Park, Byeong-Seo Min, Jiheon Woo et al. · ArXiv.org · 2026
Homomorphic encryption (HE) is a prominent framework for privacy-preserving machine learning, enabling inference directly on encrypted data. However, evaluating softmax, a core component of transformer architectures, remains particularly challenging in HE due to its multivariate structure, the large dynamic range induced by exponential functions, and the need for accurate division during normalization. In this paper, we propose MGF-softmax, a novel softmax reformulation based on the moment generating function (MGF) that replaces the softmax denominator with its moment-based counterpart. This reformulation substantially reduces multiplicative depth while preserving key properties of softmax and asymptotically converging to the exact softmax as the number of input tokens increases. Extensive experiments on Vision Transformers and large language models show that MGF-softmax provides an efficient and accurate approximation of softmax in encrypted inference. In particular, it achieves inference accuracy close to that of high-depth exact methods, while requiring substantially lower computational cost through reduced multiplicative depth.
Cited by 0Open access
Xuhang Li, Qianzi Shen, Haitao Wang et al. · Preprints.org · 2024
Studies on integrating Spiking Neural Networks (SNNs) with the Transformer architecture holds promise for enabling models to achieve ultra-low energy consumption while possessing the performance of the Transformer architecture. Currently, studies on ANN-to-SNN conversion of integrating Spiking Neural Networks (SNNs) with the Transformer architecture mainly focuses on simple activation functions in MLPs, and has not yet addressed the mismatch between the Softmax activation function in the self-attention mechanism and the computation rules of SNNs. Consequently, the ANN-to-SNN conversion efforts have consistently failed to make the Transformer architecture directly applicable to SNNs. To address this challenge, we propose the Spiking-Softmax method, which integrates Spiking Exponential Neuron (SI-exp) and Spiking Collaboration Normalized Neuron (SI-norm). The Spiking-Softmax method accurately simulates the Softmax activation function with only 12 time steps. Building upon this, we propose the Spike Integrated Transformer conversion (SIT-conversion) method, which enables the conversion of the Transformer architecture to SNNs. The SNNs generated by the SIT-conversion of Transformer models of various sizes achieve accuracy nearly identical to their ANNs counterparts, achieving nearly lossless and ultra-low-latency ANN-to-SNN conversion. This work represents the first implementation of simulating the Softmax activation function and fully converting the Transformer architecture into SNNs through spike firing.
Cited by 0Open access
Hao, Peng, Li, Jianxin, Song, Yangqiu et al. · Rare & Special e-Zone (The Hong Kong University of Science and Technology) · 2017
Neural network language models (NNLMs) have attracted a lot of attention recently. In this paper, we present a training method that can incrementally train the hierarchical softmax function for NNMLs. We split the cost function to model old and update corpora separately, and factorize the objective function for the hierarchical softmax. Then we provide a new stochastic gradient based method to update all the word vectors and parameters, by comparing the old tree generated based on the old corpus and the new tree generated based on the combined (old and update) corpus. Theoretical analysis shows that the mean square error of the parameter vectors can be bounded by a function of the number of changed words related to the parameter node. Experimental results show that incremental training can save a lot of time. The smaller the update corpus is, the faster the update training process is, where an up to 30 times speedup has been achieved. We also use both word similarity/relatedness tasks and dependency parsing task as our benchmarks to evaluate the correctness of the updated word vectors. © Copyright 2017, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.
Cited by 1
Dirmanto Jap, Yoo-Seung Won, Shivam Bhasin · 2021
Softmax is commonly used activation function in neural networks to normalize the output to probability distribution over predicted classes. Being often deployed in the output layer, it can potentially be targeted by fault injection attacks to create misclassification. In this extended abstract, we perform a preliminary fault analysis of Softmax against single bit faults.
Cited by 10
Zheng Zhai, Jiali Zhang, haiyu wang et al. · SSRN Electronic Journal · 2024
Cited by 0Open access
Hongyang Gu, Yao Ding, Xiaogang Yang et al. · 2026
Cited by 0
Muhammad Kamran Chughtai, Muhammad Bilal Babar, Muhammad Yasir Qadri et al. · 2019
This work focuses primarily on the successful design and implementation of a high speed and a resource efficient approximation of Softmax loss function. The implementation explores system generator tool together with Simulink as a viable alternative for rapid prototyping, evaluation, and deployment of Field Programmable Gate Array (FPGA) based solutions. Linear look-up table (LUT) based interpolation is used for approximation of non-linear functions such as exponential as well as the implementations of dividers. They are usually very costly when designed through conventional approaches. After rigorous analysis, a dynamic quantization scheme is developed which offers the best compromise between accuracy and speed of execution. These results are then further expanded to extract rules for the implementation of softmax approximators of varying sizes. The results show that for an architecture with the OEM claimed maximum frequency of 600 MHz the proposed system achieves a synthesis frequency of 400 MHz. These results show that design using Xilinx system generator provides designer access to a vast variety of data visualization and processing tools that come packed with Matlab Simulink without significant performance degradation.
Cited by 6
Dunne Campbell, R. A. Dunne, N. A. Campbell · 2007
It is suggested in the literature [2, 1] that there is a natural pairing between the softmax activation function and the cross--entropy penalty function. We clarify a reason for this pairing and give an improved derivation of the softmax activation function. In addition, we empirically compare some penalty/activation function pairs. 1 Introduction The standard MLP model with two layers of adjustable parameters, p inputs, h hidden layer units and q output units, and no skip-- or intra--layer connections, is described by mlp(x i ; \\Upsilon; \\Omega\\Gamma = f q (\\Upsilon[1; ff h(\\Omega x i )g t ] t ); where \\Upsilon (of size q \\Theta h + 1) and\\Omega (of size h \\Theta p + 1) are the two matrices of adjustable parameters and f h : R h ! (0; 1) h applies the same 1--variable "squashing " function to each of its coordinates. We take the "squashing" function to be the standard sigmoid f 1 (x) = 1=f1 + exp(\\Gammax)g. We also have a set of training data T = fxn ; t ng N n=1 , where ...
Cited by 0
G.C. Cardarilli, Luca Di Nunzio, Rocco Fazzolari et al. · Scientific Reports · 2021
Cited by 75Open access
Yuan Zhang, Zhang Yong-gang, Lele Peng et al. · IEEE Transactions on Circuits and Systems I Regular Papers · 2022
The softmax function is widely used in deep neural networks (DNNs), its hardware performance plays an important role in the training and inference of DNN accelerators. However, due to the complexity of the traditional softmax, the existing hardware architectures are resource-consuming or have low precision. In order to address the challenges, we study a base-2 softmax function in terms of its suitability for neural network training and efficient hardware implementation. Compared to the classical base- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$e$ </tex-math></inline-formula> softmax function, the base-2 softmax function is a new softmax function that uses 2 as the exponential base instead of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$e$ </tex-math></inline-formula> . From the aspects of mathematical derivation and software simulation, we first demonstrate the feasibility and good accuracy of the base-2 softmax function in the application of neural network training. Then, we use the symmetric-mapping lookup table (SM-LUT) method to design a low-complexity architecture but with high precision to implement it. Under TSMC 28nm CMOS technology, an example design of our architecture has the area of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$5676 ~\mu m^{2}$ </tex-math></inline-formula> and the power consumption of 13.12 mW for circuit synthesis at the frequency of 3 GHz. Compared with the latest works, our architecture achieves the best performance and efficiency.
Cited by 53
Liam Collins, Aryan Mokhtari, Advait Parulekar et al. · 2024
Cited by 2
Guillaume Bouchard · 2007
The softmax link is used in many probabilistic model dealing with both discrete and continuous data. However, efficient Bayesian inference for this type of model is still an open problem due to the lack of efficient upper bound for the sum of exponentials. We propose three different bounds for this function and study their approximation properties. We give a direct application to the Bayesian treatment of multiclass logistic regression and discuss its generalization to deterministic approximate inference in hybrid probabilistic graphical models.
Cited by 0
Vladimir Bratischenko · Современные наукоемкие технологии (Modern High Technologies) · 2025
SOFTMAX .. , ,
Cited by 0Open access
Hongyang Gu, Yao Ding, Xiaogang Yang et al. · 2026
Cited by 0
Premananth Gowtham, John Sahaya Rani Alex · Journal of Circuits Systems and Computers · 2025
Deep Neural Networks (DNNs) are one of the prominent and state-of-the-art methods for implementing artificial intelligence algorithms. A noticeable amount of effort has been put into hardware acceleration of DNN, with significantly less attention given to the SoftMax layer, which uses expensive hardware for exponentiation and division. The SoftMax function must be implemented as accurately as possible since it is crucial in multiclass classification training and inference tasks. This paper describes an efficient implementation of the SoftMax function for error-sensitive application of DNNs. We propose a Lookup Table (LUT) based on linear polynomial approximation for exponential and log units. The proposed architecture is modeled in Verilog and synthesized to gate-level netlist in GPDK180[Formula: see text]nm using Cadence[Formula: see text]. Results after the placement and routing stage show a delay of 7.15[Formula: see text]ns while consuming 5.3 mW of total power when operated at 100[Formula: see text]MHz. Multiclass classification is coded in Python with the proposed SoftMax layer to evaluate the accuracy in real-time applications. The results show that the neural network trained with the proposed SoftMax reaches almost the same accuracy as traditional SoftMax of 86.76%. This proves that low-form factor platforms can implement the same accuracy achieved with the proposed SoftMax function with less power.
Cited by 0
G.C. Cardarilli, Luca Di Nunzio, Rocco Fazzolari et al. · Scientific Reports · 2021
Cited by 1Open access
Yuan Zhang, Lele Peng, Lianghua Quan et al. · IEEE Transactions on Circuits and Systems I Regular Papers · 2023
Softmax is a common and complex activation function in Deep Neural Networks (DNN). However, it is a challenge to apply it efficiently in DNN training hardware accelerator. Therefore, we propose a high precision calculation method and architecture based on base-2 softmax, which has low hardware complexity than base- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$e$ </tex-math></inline-formula> softmax but can still be useful in DNN training. First, we simplify the hardware implementation complexity of calculating base-2 softmax. Second, we use the base-2 hyperbolic COordinate Rotation Digital Computer (CORDIC) to implement the core computation. Finally, we show that the proposed method can be used in DNN training through experiments. Moreover, with the same order of the magnitude of high precision, our hardware cost is lower than traditional base- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$e$ </tex-math></inline-formula> softmax or other alternative design methods. Under TSMC 28nm CMOS technology, an example design of our architecture has the area of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$98787.43\mu m^{2}$ </tex-math></inline-formula> and the power consumption of 24.72mW for circuit synthesis at the frequency of 1GHz.
Cited by 16
Ashish Deopa, Abhishek Sinha, Aditya Prakash et al. · 2019
Facial Expression Recognition is an extremely interesting topic of research owing to the uniqueness attached to emotions of different humans. Deep Learning is a novel zone within the domain of machine learning which is exceedingly efficient in image classification problems. Methods of Deep Learning, (CNN) Convolutional Neural Networks in particular, have been used with great precision for the purpose of feature extraction. This paper categorizes each facial image into one of the seven human emotion classes by making use of a specially designed Convolutional Neural Network which employs four subsequent sets of layers and a loss function. The model has been trained and tested on the FER2013 data set from the Kaggle Facial Expression Recognition Challenge, which consists of 35,887, 48-by-48-pixel pictures of human faces, which are grayscale in nature, each with a label of one of the 7 emotion categories. The model gives an accuracy of about 64%.
Cited by 3
Chih-Chi Ni, Zhao Zhang · 2025
In recent years, with the rapid development of deep learning technology, the Transformer model shows superior performance and is widely used in many fields such as natural language processing and computer vision. However, the algorithmic model has high demands in terms of accuracy, real-time and resources, which poses challenges to its hardware implementation. This paper proposes a hardware optimization scheme for the Softmax function widely used in Transformer. By analyzing the computational process of the function, we optimize the computational architecture using mathematical transformations and linear fitting, eliminating complex logic units including division and exponential operations, and significantly reducing the resource consumption. The computational architecture reduces resource consumption by 90% with an average error of less than 5%, and specific parameters for optimization are proposed.
Cited by 0
X. L. Ji, Yitong Zhou, He Zhang et al. · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · 2026
Analog computing-in-memory (ACIM) has garnered widespread attention due to its advantage of high energy efficiency. However, it faces large power and hardware costs to handle sophisticated nonlinear functions, such as the softmax, due to costly exponentiation and division. Existing digital-domain approaches often rely on dedicated modules to carry out these operations, leading to a cost expensive area and high-power consumption. To address the issues, we propose a self-calibrating analog circuitry for a softmax-scaled function with ACIM. By exploiting transistor subthreshold properties, the work eliminates expensive digital operations while mapping exponentiation and division to successive analog circuits. A self-calibration module further mitigates partial mismatch-induced deviations by dynamically tuning bias voltages, improving overall fitting accuracy and system robustness. The proposed softmax-enabled ACIM work achieves energy efficiency of 55.06–60.08TOPS/W and 684.15 GOPS/mm<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> at 4-bit precision. In comparison with the state-of-the-art ACIMs with softmax implications, our proposed work shows higher energy efficiency and area efficiency.
Cited by 0