A generalisation of the signal-to-noise ratio using proper scoring rules

Abstract

A generalised concept of the signal-to-noise ratio (or equivalently the ratio of predictable components, or RPC) is provided, based on proper scoring rules. This definition is the natural generalisation of the classical RPC, yet it allows one to define and analyse the signal-to-noise properties of any type of forecast that is amenable to scoring, thus drastically widening the applicability of these concepts. The methodology is illustrated through numerical examples of ensemble forecasts, scored using the continuous ranked probability score (CRPS), and of probability forecasts of a binary event, scored using the logarithmic score. Numerical examples are carried out using both synthetic data with prescribed signal-to-noise ratios as well as seasonal ensemble hindcasts of the North Atlantic Oscillation (NAO) index. The latter have previously been interpreted as having a signal-to-noise "paradox", or anomalous signal-to-noise ratio, using the RPC statistic. For the synthetic data, the RPC statistic as well as the scoring rule-based ones agree regarding which data sets exhibit anomalous signal-to-noise ratios, but exhibit different variance, indicating different statistical properties. For the NAO data, on the other hand, the different statistics are more equivocal on whether the signal-to-noise ratio is anomalous.