Overfitting the literature to one set of stimuli and data

Abstract

The fast-growing field of Computational Cognitive Neuroscience is on track to meet its first crisis. A large number of papers in this nascent field are developing and testing novel analysis methods using the same stimuli and neuroimaging datasets. Publication bias and confirmatory exploration will result in overfitting to the limited available data. The field urgently needs to collect more good quality open neuroimaging data using a variety of experimental stimuli, to test the generalisability of current published results, and allow for more robust results in future work.