preprint · arXiv (Cornell University) · 2021

Optimal Binary Classification Beyond Accuracy

Cited by 4Open access

Abstract

The vast majority of statistical theory on binary classification characterizes performance in terms of accuracy. However, accuracy is known in many cases to poorly reflect the practical consequences of classification error, most famously in imbalanced binary classification, where data are dominated by samples from one of two classes. The first part of this paper derives a novel generalization of the Bayes-optimal classifier from accuracy to any performance metric computed from the confusion matrix. Specifically, this result (a) demonstrates that stochastic classifiers sometimes outperform the best possible deterministic classifier and (b) removes an empirically unverifiable absolute continuity assumption that is poorly understood but pervades existing results. We then demonstrate how to use this generalized Bayes classifier to obtain regret bounds in terms of the error of estimating regression functions under uniform loss. Finally, we use these results to develop some of the first finite-sample statistical guarantees specific to imbalanced binary classification. Specifically, we demonstrate that optimal classification performance depends on properties of class imbalance, such as a novel notion called Uniform Class Imbalance, that have not previously been formalized. We further illustrate these contributions numerically in the case of $k$-nearest neighbor classification

Cite this paper

APA 7

Singh, S. K., & Khim, J. (2021). Optimal Binary Classification Beyond Accuracy. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2107.01777

MLA 9

Singh, Shashank Kumar, and Justin Khim. “Optimal Binary Classification Beyond Accuracy.” arXiv (Cornell University), 2021, https://doi.org/10.48550/arxiv.2107.01777.

Chicago

Singh, Shashank Kumar, and Justin Khim. 2021. “Optimal Binary Classification Beyond Accuracy.” arXiv (Cornell University). https://doi.org/10.48550/arxiv.2107.01777.

Harvard

Singh, S.K. and Khim, J. (2021) ‘Optimal Binary Classification Beyond Accuracy’, arXiv (Cornell University). Available at: https://doi.org/10.48550/arxiv.2107.01777.

BibTeX

@misc{singh2021optimal,
  title = {{Optimal Binary Classification Beyond Accuracy}},
  author = {Singh, Shashank Kumar and Khim, Justin},
  journal = {arXiv (Cornell University)},
  year = {2021},
  doi = {10.48550/arxiv.2107.01777}
}

Made from the index record. Check names, capitalisation and page numbers against the paper. Need another style? AI for Research has over 10,000.