article · 2007
On The Pairing Of The Softmax Activation And Cross--Entropy Penalty Functions And The Derivation Of The Softmax Activation Function
Dunne Campbell, R. A. Dunne, N. A. Campbell
Cited by 0Abstract and details only
Abstract
It is suggested in the literature [2, 1] that there is a natural pairing between the softmax activation function and the cross--entropy penalty function. We clarify a reason for this pairing and give an improved derivation of the softmax activation function. In addition, we empirically compare some penalty/activation function pairs. 1 Introduction The standard MLP model with two layers of adjustable parameters, p inputs, h hidden layer units and q output units, and no skip-- or intra--layer connections, is described by mlp(x i ; \\Upsilon; \\Omega\\Gamma = f q (\\Upsilon[1; ff h(\\Omega x i )g t ] t ); where \\Upsilon (of size q \\Theta h + 1) and\\Omega (of size h \\Theta p + 1) are the two matrices of adjustable parameters and f h : R h ! (0; 1) h applies the same 1--variable "squashing " function to each of its coordinates. We take the "squashing" function to be the standard sigmoid f 1 (x) = 1=f1 + exp(\\Gammax)g. We also have a set of training data T = fxn ; t ng N n=1 , where ...
Cite this paper
Campbell, D., Dunne, R. A., & Campbell, N. A. (2007). On The Pairing Of The Softmax Activation And Cross--Entropy Penalty Functions And The Derivation Of The Softmax Activation Function.
Campbell, Dunne, et al. “On The Pairing Of The Softmax Activation And Cross--Entropy Penalty Functions And The Derivation Of The Softmax Activation Function.” 2007.
Campbell, Dunne, R. A. Dunne, and N. A. Campbell. 2007. “On The Pairing Of The Softmax Activation And Cross--Entropy Penalty Functions And The Derivation Of The Softmax Activation Function.”
Campbell, D., Dunne, R.A. and Campbell, N.A. (2007) ‘On The Pairing Of The Softmax Activation And Cross--Entropy Penalty Functions And The Derivation Of The Softmax Activation Function’.
@article{campbell2007pairing,
title = {{On The Pairing Of The Softmax Activation And Cross--Entropy Penalty Functions And The Derivation Of The Softmax Activation Function}},
author = {Campbell, Dunne and Dunne, R. A. and Campbell, N. A.},
year = {2007}
}Made from the index record. Check names, capitalisation and page numbers against the paper. Need another style? AI for Research has over 10,000.