Second-order accuracy metrics for scoring models and their practical use

Abstract

The paper proposes new second-order accuracy metrics for scoring or rating models, which show the target preference of the model, it is better to diagnose good objects or better to diagnose bad ones for a constant generally accepted predictive power determined by the first order metric that is known as the Gini index. There are two metrics, they have both an integral representation and a numerical one. The numerical representation of metrics is of two types, the first of which is based on binary events to evaluate the model, the second on the default probability given by the model. Comparison of the results of calculating the metrics allows you to validate the calibration settings of the scoring or rating model and reveals its distortions. The article provides examples of calculating second-order accuracy metrics for ratings of several rating agencies, as well as for the well known approach to calibration based on van der Burg's ROC curves.