Why NRMSE is not completely reliable for forecast/hindcast model test performances

We investigate the reliability of the statistical error indicator NRMSE (Nor- malized Root Mean Square Error) as an index of the performances of numerical simulation for wave forecasting. This widespread indicator, also known as Scat- ter Index, is defined as (S i − O i ) 2 O i 2 NRMSE = (1) where O i are observed values and S i are simulated values. A small value of NRMSE identifies a numerical simulation in good agreement with the field ob- servations. We show that NRMSE tends to be systematically smaller (better) for simulations affected by negative bias. This behaviour is verified comparying forty three different parameterizations of the source terms within a process of validation of the wave model Wavewatch III (WWIII) in the Mediterranean sea. In our analysis we show that NRMSE can be orthogonally decomposed in two contributions related to the average bias and to the scatter of simulated values around the observed values, respectively. Results form numercial sim- ulations show how these contributions appear statistically dependent on each other because positive or negative amplifications of the simulation average gen- erally involve a corresponding amplification in the scatter of simulated values. An almost linear dependency between bias and scatter can be found and it implies that minimum NRMSE simulation and unbiased simulation do not co- incide, since minimum NRMSE requires a minimum squared sum of its bias and scatter contributions. This finding suggests that a lower value of NRMSE is not always associated to the best results, and that this indicator is not always reliable if used to find the best simulation. This flaw of NRMSE indicator is already known in literature, and some authors proposed the usage of corrected indicators to overcome it (e.g. Hanna and Heinolds, 1985[1]).