On the limitations of comparing mean square forecast errors

Abstract

Linear models are invariant under non‐singular, scale‐preserving linear transformations, whereas mean square forecast errors (MSFEs) are not. Different rankings may result across models or methods from choosing alternative yet isomorphic representations of a process. One approach can dominate others for comparisons in levels, yet lose to another for differences, to a second for cointegrating vectors and to a third for combinations of variables. The potential for switches in ranking is related to criticisms of the inadequacy of MSFE against encompassing criteria, which are invariant under linear transforms and entail MSFE dominance. An invariant evaluation criterion which avoids misleading outcomes is examined in a Monte Carlo study of forecasting methods.