Skip to content
Free planNo credit card requiredStart writing
AI for Research home
AI for Research home
Dashboard
Sign inGet started

preprint · arXiv (Cornell University) · 2019

Static Activation Function Normalization

Pierre H. Richemond, Yike Guo

Cited by 0Open access

Abstract

Recent seminal work at the intersection of deep neural networks practice and random matrix theory has linked the convergence speed and robustness of these networks with the combination of random weight initialization and nonlinear activation function in use. Building on those principles, we introduce a process to transform an existing activation function into another one with better properties. We term such transform \emph{static activation normalization}. More specifically we focus on this normalization applied to the ReLU unit, and show empirically that it significantly promotes convergence robustness, maximum training depth, and anytime performance. We verify these claims by examining empirical eigenvalue distributions of networks trained with those activations. Our static activation normalization provides a first step towards giving benefits similar in spirit to schemes like batch normalization, but without computational cost.

Cite this paper

APA 7

Richemond, P. H., & Guo, Y. (2019). Static Activation Function Normalization. arXiv (Cornell University). https://doi.org/10.48550/arxiv.1905.01369

MLA 9

Richemond, Pierre H., and Yike Guo. “Static Activation Function Normalization.” arXiv (Cornell University), 2019, https://doi.org/10.48550/arxiv.1905.01369.

Chicago

Richemond, Pierre H., and Yike Guo. 2019. “Static Activation Function Normalization.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.1905.01369.

Harvard

Richemond, P.H. and Guo, Y. (2019) ‘Static Activation Function Normalization’, arXiv (Cornell University). Available at: https://doi.org/10.48550/arxiv.1905.01369.

BibTeX
@misc{richemond2019static,
  title = {{Static Activation Function Normalization}},
  author = {Richemond, Pierre H. and Guo, Yike},
  journal = {arXiv (Cornell University)},
  year = {2019},
  doi = {10.48550/arxiv.1905.01369}
}

Made from the index record. Check names, capitalisation and page numbers against the paper. Need another style? AI for Research has over 10,000.

Related papers