Identifying good forecasters via adaptive cognitive tests

Abstract

Assessing forecasting performance is a time intensive activity, often requiring months or years before we know whether or not the reported forecasts were accurate. Cognitive tests can be quickly administered and are predictive of forecasting performance, but it is unclear which and how many tests are optimal. In this study, we develop adaptive cognitive tests that optimize the selection and efficiency of cognitive tests to assess forecasters of different skill levels. The tests are based on item response models and the adaptive testing procedures commonly used in educational testing. We show how the procedures can select highly informative cognitive tests from a larger battery of tests, thereby reducing the time taken to administer the tests. We use a second, independent dataset to show that the selected tests yield scores that are highly related to out-of-sample forecasting performance. The approach enables real-time, adaptive testing, providing immediate insights into forecasting talent in practical contexts.