Standardized mean difference is the effect size typically used to compare the difference between a treatment and control group on continuous outcomes. However, data provided in primary studies to calculate effect sizes vary. Sometimes, multiple options are available, and it’s not always clear which method is best to use.
In an article from 2021, Joseph Taylor and colleagues provided guidelines for reporting data in primary studies to calculate effect sizes, as well as recommendations on which data should be prioritized for meta-analyses.
The authors’ key recommendations for meta-analysts are:
- Use effect sizes that adjust for baseline covariates, at the very least the pretest scores of the outcome measure, and possibly demographic variables as well. This produces an effect size estimate that is more interpretable and precise.
- Avoid using unadjusted means when covariate-adjusted means are available, because effect sizes from unadjusted means introduce imprecision and artificially increase effect size heterogeneity in meta-analyses.
- In cluster studies that assigned schools or classes, adjust effect sizes and variances for baseline covariates and for clustering.
Following these recommendations requires better reporting of necessary data in primary studies. This includes covariate-adjusted means, unadjusted standard deviations and standard error of the adjusted mean difference for individual level studies. For cluster studies, it also requires reporting the intraclass correlation (ICC) and the standard error of the adjusted mean difference from a model that accounts for clustering. The authors provided an online tool to perform all calculations.
Source: Taylor, J. A., Pigott, T., & Williams, R. (2022). Promoting knowledge accumulation about intervention effects: Exploring strategies for standardizing statistical approaches and effect size reporting. Educational Researcher, 51(1), 72–80. https://doi.org/10.3102/0013189X211051319