Although integrating technology into education has been found to be effective means in helping students, it may fail to produce impacts because of poor implementation. A recent cluster-random controlled trial study conducted by Mo and colleagues investigated the effect of a computer-assisted learning (CAL) program on English achievement and compared the differences in implementation of the program between institutions. The CAL program was designed to provide remedial tutoring for learning English during computer class sessions organized by a computer teacher. A balanced cohort of 120 primary schools in one of China’s poorest provinces was randomized into three groups: CAL implemented by a government agency, the same program implemented by a non-government organization (NGO), and no CAL program (control schools). A total of 5,253 Grade 4 students completed the program.
Both treatment groups adopted the same software packages for the tutoring, and both the government agency and the NGO conducted implementation, training, and monitoring of the program’s progress. The NGO, a university-based education-oriented entity, designed the implementation protocol and trained designated computer teachers to organize CAL sessions. The government agency assigned local education officers, county program managers, to carry out the program in addition to their other duties. The effectiveness of the program was assessed by standardized English language test scores after one academic year (2013-14), with baseline test scores and students’ characteristics controlled. The results indicated:
- Students in the NGO CAL program performed significantly better than those in the control group (E.S. = +0.16)
- The government CAL program did not improve students’ English achievement (E.S. = – 0.07) compared to the control group.
The authors conducted mechanism analysis to explore potential causes for two identical CAL protocols producing different outcomes. The findings showed the following:
- In 38% of the government CAL schools, regular English classes were replaced with CAL classes in comparison to 18% of the NGO CAL schools. Breaking the protocol may diminish regular English time.
- Instead of computer teachers, 48% of government CAL schools assigned English teachers to run the CAL sessions while 18% NGO schools did the same. If protocol is not followed, the workload of English teachers may increase.
- NGO staff were more likely to call or visit their program schools and monitor the progress than government officials (different by 50% points).
- Correlation analysis indicated that replacement of class time was negatively associated with English test scores with a small effect size.
- Direct monitoring was positively correlated with student scores.
In view of the finding that violation of protocol may diminish program impact, effective monitoring is one of important factors for successful intervention. Cost effectiveness analyses for NGO CAL project indicated that the monitor fee was .91 USD/student/year.
Source: Mo, D., Bai, Y., Shi, Y., Abbey, C., Zhang, L., Rozelle, S., & Loyalka, P. (2020). Institutions, implementation, and program effectiveness: Evidence from a randomized evaluation of computer-assisted learning in rural China. Journal of Development Economics. https://doi.org/10.1016/j.jdeveco.2020.102487