Validation of Space Weather Forecasting at Leonid Space

Abstract

We validate Leonid Space's space weather forecasts through comprehensive cross-validation of retroforecasts scored against the entire F10.7 and Ap historical record. We built a unified forecast model that jointly forecasts F10.7 and Ap on all timescales from 3 hours up to multi-decade chains of future solar cycles. This was used to generate an archive of historical 15-year retroforecasts, after performing a leave-one-out fit of the model on held-out data and conditioning only on historical data available at the time. The accuracy of this archive was scored against measured space weather indices at all lookahead horizons from 1 day to 15 years, using the continuous ranked probability score (CRPS) as the primary quality metric. All other publicly available F10.7 and Ap forecast archives were scored using the same approach, including NASA's MSAFE and Schatten forecasts, SWPC's 27-day, 45-day, and Solar Cycle predictions, SWPC/USAF's RSGA, SIDC's Ursigram, CLS, Athens, KSO, BGS, SEPC's 3-day and 27-day forecasts, AFRL/NSO's SIFT, and BoM. Taken together, these 15 flux forecasts and 12 Ap forecasts compared on common lookahead metrics make up the most comprehensive cross-comparison of space weather forecasts we are aware of. Over a common forecast interval since 2000, Leonid had the lowest CRPS prediction error of any F10.7 or Ap prediction model: strictly lowest for Ap at all lookaheads and for F10.7 up to 5 years, and parity with MSAFE after 5 years. Much of this performance comes from our well-calibrated uncertainty bands, and we urge space weather forecasters to include calibrated uncertainties with their predictions and to issue retroforecast archives that can be used to rigorously evaluate their generating models.