Test Error Bounds for Classifiers: A Survey of Old and New Results

In this paper, we focus the attention on one of the oldest problems in pattern recognition and machine learning: the estimation of the generalization error of a classifier through a test set. Despite this problem has been addressed for several decades, the last word has not yet been written, as new proposals continue to appear in the literature. Our objective is to survey and compare old and new techniques, in terms of quality of the estimation, easiness of use, and rigorousness of the approach.