A Bayesian Approach to Estimating Effect Sizes in Educational Research

Abstract

In this paper, we demonstrate a purely Bayesian approach for estimating within-group and between-group effect sizes for learning outcomes encountered in educational research, taking naturally into account the multilevel structure of the data, as well as heterogeneous residual variances among time points and conditions. We provide a detailed implementation using the brms package in R serving as a wrapper for the probabilistic programming language Stan. We recommend that for a pooled design, one computes an effect size $d_s$ similar to a Cohen's $d$, and for a paired design, one should compute two possibly different quantities $d_s$ and $d_z$ to correct for correlations in within-group designs and allow for comparability across different studies. All these effect sizes are based on ideas coming from Hedge's total effect size $\delta_t$ introduced in 2007. Ultimately, these estimates allow us to study the differential effectiveness of educational interventions with respect to classes.