The AUGUST Two-Sample Test: Powerful, Interpretable, and Fast

Abstract

Two-sample testing is a fundamental problem in statistics, and many famous two-sample tests are designed to be fully non-parametric. These existing methods perform well with location and scale shifts but are less robust when faced with more exotic classes of alternatives, and rejections from these tests can be difficult to interpret. Here, we propose a new univariate non-parametric two-sample test, AUGUST, designed to improve on these aspects. AUGUST tests for inequality in distribution up to a predetermined resolution using symmetry statistics from binary expansion. The AUGUST statistic is exactly distribution-free and has a well-understood asymptotic distribution, permitting fast p-value computation. In empirical studies, we show that AUGUST has power comparable to that of the best existing methods in every context, as well as greater power in some circumstances. We illustrate the clear interpretability of AUGUST on NBA shooting data.