Encouraging educators and policymakers to use evidence to guide their decisions relies on high-quality evidence. A recent paper by Pigott and Polanin, published in Review of Educational Research, has identified guidelines for modern, rigorous systematic reviews including meta-analyses.
The guidelines are organized into three sections: elements for the systematic review, practices for the statistical synthesis of findings (meta-analysis), and presentation of the methods and results. When conducting the actual systematic review, researchers should ensure that they have clearly specified procedures, documented in advance, that identify what question they hope to answer, where and how they are searching for studies, and a process for screening, reviewing, and extracting information from those studies. These procedures should be made publicly available.
When conducting the actual meta-analysis, researchers should utilize the most up-to-date methods, such as strategies to handle multiple outcomes from a single study, adequately addressing missing data, and explore variation in impacts. This helps to answer not just how well a particular approach works on average, but additional questions, such as if it works better for certain students or in particular settings.
Finally, when researchers report the results of their meta-analyses, they must clearly describe how they arrived at their results, including making their data available. They should also strive to interpret their results so they are relevant to a wide audience. This may include translating findings into easily understood metrics, including converting effect sizes into “natural metrics,” such as gains on commonly used tests, and producing plain language summaries.
These guidelines highlight the “gold-standard” for meta-analyses.