Asymptotics of Ridge(less) Regression under General Source Condition

We analyze the prediction error of ridge re- gression in an asymptotic regime where the sample size and dimension go to infinity at a proportional rate. In particular, we consider the role played by the structure of the true re- gression parameter. We observe that the case of a general deterministic parameter can be re- duced to the case of a random parameter from a structured prior. The latter assumption is a natural adaptation of classic smoothness as- sumptions in nonparametric regression, which are known as source conditions in the the context of regularization theory for inverse problems. Roughly speaking, we assume the large coefficients of the parameter are in cor- respondence to the principal components. In this setting a precise characterisation of the test error is obtained, depending on the inputs covariance and regression parameter structure. We illustrate this characterisation in a sim- plified setting to investigate the influence of the true parameter on optimal regularisation for overparameterized models. We show that interpolation (no regularisation) can be opti- mal even with bounded signal-to-noise ratio (SNR), provided that the parameter coeffi- cients are larger on high-variance directions of the data, corresponding to a more regular function than posited by the regularization term. This contrasts with previous work con- sidering ridge regression with isotropic prior, in which case interpolation is only optimal in the limit of infinite SNR.