Estimating ensemble-mean anomaly correlations and the ratio of predictable components using an unbiased estimate of the predictable signal variance

Abstract

The correlation ($r$) between ensemble mean forecasts and observations is a frequently-used metric of forecast quality in the weather and climate prediction communities. However, uncorrected estimates of $r$ are biased towards zero at finite ensemble size due to the influence of unpredictable noise that contaminates the sample ensemble mean. This note summarises established ensemble-size adjustment methods and characterises the behaviour of $r_{\infty}$, a correlation estimator that is constructed from an unbiased and ensemble-size-independent estimate of the signal variance. The only requirement is that forecast members are exchangeable with one another such that they can be interpreted as independent draws from a case-specific predictive distribution. The estimator $r_{\infty}$ does not require forecasts and observations to be statistically exchangeable, only that the unpredictable component of each member is uncorrelated with the observed truth, and thus works for both reliable and unreliable forecasts. In some situations, $r_{\infty}$ may be undefined (i.e. the estimated signal variance is negative) or fall outside the admissible range $-1 \leq r_{\infty} \leq 1$ when the predictable signal is small relative to the sampling uncertainty. These inadmissible values can be interpreted as evidence that the true predictable signal cannot be resolved with the available ensemble size and number of cases. The same unbiased signal variance can also be used to estimate the correlation between the predictable signal and individual members to yield an ensemble-size-invariant estimator of the ratio of predictable components (RPC). In contrast, variance-based RPC estimates, including forms that correct only the model predictable component, are biased low by a factor that depends only on the ensemble size and model signal-to-noise ratio.