Studentized inference in randomized experiments when a few units dominate the outcome variation

Abstract

In completely randomized experiments with heavy-tailed outcomes, a few units can carry a share of the residual variation that does not vanish as the number of units grows. We study the studentized difference in means and Lin's regression-adjusted estimator, with covariates whose leverage vanishes, when the finite population is fixed, and the treatment assignment is the only source of randomness. The studentized statistic then has a nonnormal limit in general. Populations with the same marginal distributions of potential outcomes can have different limits, as the limit depends on the unobserved pairing mechanism of the two potential outcomes of each dominant unit. When the outcome errors are drawn from a heavy-tailed law, the limit is random and, outside a degenerate case, varies with the population. Under balanced allocation, the conventional Wald interval has an explicit asymptotic coverage floor. At the nominal level $0.95$, the floor is about $0.42$ for every population and about $0.84$ when the two potential residuals of each unit are equal. Both floors are close to the best possible, and without balanced allocation, the coverage can fall to the smaller of the proportions of treated and control units.