Small Samples and Short Panels: Evaluating Policy Evaluation Methods with Realistic Data

Abstract

Methods for estimating causal effects in longitudinal, quasi-experimental settings are widely used in economics, public health, political science, and other fields. However, studies evaluating effects of health policies often rely on limited sample sizes, both in terms of study units and time periods analyzed. Synthetic difference-in-differences (SDiD) and the augmented synthetic control method (ASCM) are recently developed methods for evaluating the effects of policy interventions. Although SDiD and ASCM generally rely on weaker assumptions than both DiD and SCM, they lack theoretical performance guarantees with respect to bias or coverage in relevant settings with small sample sizes or short panel lengths. To evaluate the performance of SDiD and ASCM in realistic, small-sample settings, we employ a calibrated simulation strategy that allows the injection of a known treatment effect into existing data in a setting of interest. Drawing on findings from these empirical investigations, we offer practical guidance for researchers and policymakers on when SDiD and ASCM are likely to yield reliable estimates and inferences under realistic scenarios.