While randomized studies are often touted as the “gold standard” for determining the efficacy of an intervention, it is important to highlight that there is a spectrum of study designs that can provide valuable insight into program effects. Randomized studies are highly valued in educational research for good reason; however, they often require significant funding, personnel, and infrastructure which may not always be feasible for all educational research endeavors. Recognizing that the building of an educational research base can start with lower barrier to entry research designs, including the use of virtual control groups, ultimately will contribute to a richer understanding of what works.
A recent report from the RAND Corporation employed a matched comparison group methodology with a virtual control group (VCG) to explore the effects of Lexia Core5 Reading on student achievement. Core5 is a blended learning program that addresses key literacy skills, including fluency, vocabulary, and comprehension. It uses online and paper-based activities and offers teacher support materials. Pane and colleagues compared grade 3-5 students in a district using Core5 with similar students across the U.S. who didn’t use Core5, using the NWEA service that creates a virtual matched comparison group from their large database of MAP assessment results nationwide. The VCG is constructed to mirror the characteristics of Core5 students and allows researchers to estimate the program’s impact by comparing the reading achievement of Core5 users to a group of similar students who did not have access to the program during the study.
Results indicated that Core5 students, on average, outperformed their comparison group counterparts in reading achievement during the 2021–2022 academic year. The standardized effect size of +0.11 indicates that about 54% of Core5 students surpassed the comparison group median. The effects of Core5 were consistent across grade levels and genders but varied among racial/ethnic subgroups. White students experienced larger gains (effect size =+0.15), while Black and Hispanic students had smaller gains (effect sizes of +0.05 and +0.06, respectively). High-usage students, meeting Core5’s usage targets for at least 20 weeks, outperformed their peers (effect size = +0.16). While utilizing a VCG design does not offer the same level of causal certainty as randomized trials, it still provides insight into program effectiveness in real-world conditions. VCGs allow researchers to work with existing data, making them more practical in certain situations and more cost-effective. They can include a large number of students from diverse backgrounds, which can enhance the generalizability of study findings. However, even though VCG’s closely resemble the treatment group on observable characteristics, there does exist the risk of groups being different on unobservable factors. Being aware that selection bias can play a part in outcome differences is important to keep in mind when interpreting results. There are benefits and disadvantages associated with the different research designs, but they all have a role in contributing to a comprehensive and robust understanding of the impact of an intervention in various educational settings.
Source (Open Access): Pane, J. F., Seaman, D., & Doss, C. J. (2023). Students Using Lexia® Core5® Reading Show Greater Reading Gains Than Matched Comparison Students. RAND Corporation. https://www.rand.org/pubs/research_reports/RRA2859-1.html