Learning from Examples as an Inverse Problem

Many works related learning from examples to regularization techniques for inverse prob- lems, emphasizing the strong algorithmic and conceptual analogy of certain learning algo- rithms with regularization algorithms. In particular it is well known that regularization schemes such as Tikhonov regularization can be effectively used in the context of learning and are closely related to algorithms such as support vector machines. Nevertheless the connection with inverse problem was considered only for the discrete (finite sample) prob- lem which is solved in practice and the probabilistic aspects of learning from examples were not taken into account. In this paper we provide a natural extension of such analysis to the continuous (population) case and analyse the interplay between the discrete and continuous problems. From a theoretical point of view, this allows to draw a clear connection between the consistency approach imposed in learning theory, and the stability convergence prop- erty used in ill-posed inverse problems. The main mathematical result of the paper is a new probabilistic bound for the regularized least-squares algorithm. By means of standard results on the approximation term, the consistency of the algorithm easily follows. .