Evaluating the influence of treatment-effect heterogeneity on discrimination

Abstract

Analyzing the heterogeneity of treatment effects is crucial in personalized medicine to identify which patients will benefit from specific treatments. The performance of a conditional average treatment effects model to guide treatment decisions can be assessed in different ways, with an important one being the model's ability to effectively discriminate between individuals who benefit from the treatment and those who do not. While many methods and algorithms have been proposed to develop conditional average treatment effects models and individualized treatment rules, little is known about the discriminative ability that can be achieved according to the population's underlying distribution of treatment effects. In this work, we computed the discrimination that can be achieved under oracle CATE for a panel of 20 distributions with varying average treatment effects and levels of heterogeneity. The assessment included the following discrimination metrics: the c-statistic for benefit, the concentration of benefit, and the population average prescriptive effect (PAPE). Results showed that the three metrics employed in this study did not require the same levels of treatment effect heterogeneity to lead to high discrimination results. Notably, achieving high c-statistic for benefit and PAPE values required greater heterogeneity than obtaining high concentration of benefit values. The three metrics considered behave very differently across the distributions. For instance, the concentration of benefit can indicate perfect discrimination in settings with negligible treatment-effects heterogeneity.