The Probability of Improved Prediction: a new concept in statistical inference

Abstract

In an attempt to provide an answer to the increasing criticism against p-values and to bridge the gap between statistical inference and prediction modelling, we introduce the probability of improved prediction (PIP). In general, the PIP is a probabilistic measure for comparing two competing models. Three versions of the PIP and several estimators are introduced and the relationships between them, p-values and the mean squared error are investigated. The performance of the estimators is assessed in a simulation study. An application shows how the PIP can support p-values to strengthen the conclusions or possibly point at issues with e.g. replicability.