Information leakage from data revisions in retrospective forecasts
Johannes Bracher, Sebastian Funk
Abstract
Aygün et al (2026, https://doi.org/10.1038/s41586-026-10658-6) claim that their AI-driven Empirical Research Assistance (ERA) system produces COVID-19 hospitalisation forecasts which outperform the state-of-the-art CDC ensemble by a considerable margin for the 2024/25 season. We demonstrate that the observed performance gain is attributable to information leakage in the retrospective forecasting setup, which resulted because data revisions were not taken into account. As similar mechanisms are at play in many other forecasting fields, our cautionary tale applies not just to epidemic forecasting, but is relevant to the entire emerging field of AI-assisted predictive modelling.
Create a lesson
Related papers
Characterising mortality dynamics across countries and time using a multi-stage clustering approach
Pedro Menezes de Araújo, Ugofilippo Basellini, Thomas Brendan Murphy et al.
Combining Weather Forecast Aggregation and State-Space Models for Adaptive Probabilistic Electricity Load Forecasting
Joseph de Vilmarest, Jonathan Dumas, Jean Thorey
Anthropogenic Forcing, Climate Change, and the Shape of Warming: Statistical Inference for Distributional Cointegration
Won-Ki Seo, Kyungsik Nam
Observational constraints on net radiative forcing confirm aviation contrail warming
Aaron Sonabend-W, Scott Geraedts, Nita Goyal et al.
Nationally Consistent, Locally Incomplete: A Bayesian Remote-Sensing Audit of Rooftop Photovoltaic Registries
Gabriel Kasmi, Yves-Marie Saint-Drenan, Laurent Dubus et al.
Temporal Seam Score for Assessing Continuity at Known Transitions in Time Series
Hongxiao Jin