Cohen's f or Mean Standardized Differences? Assessing Covariate Balance with Multivalued Treatments

Abstract

Assessing covariate balance across more than two treatment groups has no established omnibus standard: the prevailing practice averages, or takes the maximum of, pairwise standardized mean differences (SMD), while Cohen's f - the classical generalization of Cohen's d to more than two groups - offers an alternative grounded in an established effect-size framework, but the two have not been formally compared. We extend both to arbitrary weighted and covariate-adjusted models via a community-contributed Stata command, esizereg, and validate them in a simulation study of three, four, and six treatment groups under correctly specified and misspecified generalized-propensity-score weighting, correlating each statistic against downstream treatment-effect bias. Cohen's f tracks estimation bias comparably to mean absolute SMD, both pooled (r = 0.93) and within weighting arm (r approximately 0.79); maximum absolute SMD is generally weakest. This ranking was unchanged under a deliberately adversarial nonlinear/interaction outcome model. Cohen's f and the SMD-based statistics are not numerically comparable: we derive an exact representation of f as a size-weighted quadratic function of the pairwise SMDs and prove the minimum attainable ratio between f and mean absolute SMD under equal group weights, so conventional SMD thresholds should not apply directly to f. We recommend reporting f alongside its per-level and pairwise decomposition.