Beyond Arbitrary Replications: A Principled Approach to Simulation Design in Causal Inference

Abstract

Evaluation of novel treatment effect estimators frequently relies on simulation studies lacking formal statistical comparisons and using arbitrary numbers of replications ($J$). This hinders reproducibility and efficiency. We propose the Test-Informed Simulation Count Algorithm (TISCA) to address these shortcomings. TISCA integrates Welch's t-tests with power analysis, iteratively running simulations until a pre-specified power (e.g., 0.8) is achieved for detecting a user-defined minimum detectable effect size (MDE) at a given significance level ($\alpha$). This yields a statistically justified simulation count ($J$) and rigorous model comparisons. Our bibliometric study confirms the heterogeneity of current practices regarding $J$. A case study revisiting McJames et al. (2024) demonstrates TISCA identifies sufficient simulations ($J=500$ vs. original $J=1000$), saving computational resources while providing statistically sound evidence. TISCA promotes rigorous, efficient, and sustainable simulation practices in causal inference and beyond.