False Discovery Rate Adjustments for Average Significance Level Controlling Tests

Abstract

Multiple testing adjustments, such as the Benjamini & Hochberg (1995) step-up procedure for controlling the false discovery rate (FDR), are typically applied to families of tests that control significance level in the classical sense: for each individual test, the probability of false rejection is no greater than the nominal level. In this paper, we consider tests that satisfy only a weaker notion of significance level control, in which the probability of false rejection need only be controlled on average over the hypotheses. We find that the Benjamini & Hochberg (1995) step-up procedure still controls FDR in the asymptotic regime with many weakly dependent p-values and an increasing number of rejections, and that certain adjustments for dependent p-values such as the Benjamini & Yekutieli (2001) procedure continue to yield FDR control in finite samples. Our results open the door to FDR controlling procedures in nonparametric and high dimensional settings where weakening the notion of inference may allow for power improvements.