In recent research studies, concerns have arisen regarding the potential inflation of intervention effects in reported findings, a phenomenon known as the ‘winner’s curse’. This raises questions about whether clearinghouses inadvertently contribute to this issue by prioritizing studies with statistically significant results for replication or funding. The latest article by Valentine, who has experience as an investigator at the What Works Clearinghouse (WWC), explores the challenges faced by these organizations in this context.
Clearinghouses have the challenging task of evaluating the credibility of claims relevant to their consumers, distinguishing between more and less credible ones. This complexity arises due to the intricate nature of studies and the limited empirical evidence available regarding the importance of various aspects of study conduct. Despite these challenges, the author argues that establishing evaluation standards can benefit the research community by serving as a communication tool to emphasize crucial aspects of research design and implementation. The article addresses the limitations of clearinghouses in determining ‘what works’, because: (I) probability values from studies should not be directly interpreted as effectiveness, (II) study results are context-dependent, and (III) individual studies may not represent the broader body of research.
To mitigate the winner’s curse, two strategies are proposed for clearinghouses. Firstly, they should avoid presenting information in a way that may lead to misunderstandings; this could involve using Bayesian statistics to account for uncertainty in effect estimation. Secondly, the author suggests that clearinghouses should prioritize systematic reviews as their main mission, drawing attention away from individual studies and towards the body of credible research.
Source: Valentine, J. C. (2023). How evidence clearinghouses can avoid the winner’s curse. Journal of Research on Educational Effectiveness, 0(0), 1–5. https://doi.org/10.1080/19345747.2023.2194298