Don't Drop the Singletons: Efficient Inference for Pairwise Experiments with Independent Attrition

Abstract

Pairwise randomization can yield substantial efficiency gains in experiments. Yet methodological guidance cautions against pairwise randomization, especially in settings with attrition, partly because common practices for estimation (i.e., pair fixed effects) imply discarding data from incomplete pairs thus exacerbating data loss from attrition. This practice of dropping incomplete pairs reduces statistical power of tests as well as precision of estimates, in paired experiments, compared to designs with less finely stratified treatment assignment. We argue that this concern is misplaced if attrition is independent of treatment status and potential outcomes, and that these issues follow from an inefficient use of the data that remains post-attrition. First, we show how, by using a specific permutation test, it is possible to use all observed units for inference (complete pairs and incomplete pairs where one unit attrits) while still exploiting the pairwise randomization design structure. The test procedure we suggest provides exact size control under the sharp null. Second, we study an optimally weighted estimator that efficiently combines within-pair and across-pair comparisons. Finally, we show that combining these two insights yields a test procedure that dominates the two commonly used inference methods (a paired $t$-test and the two-sample $t$-test) in power, for any level of attrition. Usefully for applied researchers, we show that the efficient procedure can be implemented via a weighted fixed effects regression, straightforward in standard software. In sum, our results provide researchers with practical tools for conducting experiments with pairwise randomization without sacrificing observations or statistical power when facing independent attrition.