Entropy Based Estimators in the Presence of Multicollinearity and Outliers

The concept and the mathematical properties of entropy play an im- portant role in statistics, cybernetics and information sciences. Indeed many al- gorithms and statistical data processing tools, with a wide range of targets and scopes, have been designed based on entropy. The paper describes two estima- tors inspired by the concept of entropy that allow to robustly cope with multi- collinearity, in one case, and outliers, in the other. The Generalized Maximum Entropy (GME) estimator optimizes the Shannon’s entropy function subject to consistency and normality constraints. In regression applications GME allows, for example, to estimate model coefficients in the presence of multicollinearity. The Least Entropy-Like (LEL) estimator is a novel prediction error model co- efficient identification algorithm that minimizes a nonlinear cost function of the fitting residuals. As the cost function that is minimized shares the same mathe- matical properties of entropy, it allows to compute an estimate of the model co- efficients corresponding to a positively skewed distribution of the residuals. The resulting estimator exhibits higher robustness to outliers with respect to standard, as ordinary least squares (OLS) model coefficient approaches. Both the GME and LEL estimation methods are applied to a common case study to illustrate their respective properties.