Demonstration Experiments

Abstract

Adaptive experiments are used extensively in online platforms, healthcare and biotechnology, and the social sciences. Often, the primary goal is not to precisely estimate a treatment effect but to demonstrate that at least one candidate intervention yields a positive effect, for some subpopulation and on some measured outcome. We formalize this objective as testing the global null in a threshold bandit framework, and develop two inference procedures that are valid under general adaptive sampling: one that pools information across promising arms, and one based on time-uniform multiple testing of individual arm means. To support the latter, we establish a moderate-deviations principle for the sequential $t$-statistic, justifying asymptotic confidence sequences in settings where the number of arms is large relative to the sample size. To illustrate how adaptive designs can target the proposed statistics, we recast experimental design as bandit optimization with an arm's reward given by its signal-to-noise ratio, and analyze an allocation rule for which we establish a logarithmic regret bound. We apply the methods in a simulation study of targeting unconditional cash transfer programs.