On the Role of Regularization in Machine Learning: Classical Theory, Computational Aspects and Modern Regimes

In this work we study performances of different machine learning models by focusing on regularization properties in order to explain different phenomena that are observed in practice. We consider linear models on possibly infinite dimensionale feature space that are trained by optimizing an empirical mean squared errror. We study regularization properties of accelerated method like Nesterov or $ u$-method and properties of interpolating estimator where the main regularization sources vanish to zero and explain different behaviour which can be seen in practical applications.