A study published in Evaluation Review evaluated the effect of a teacher professional development program on student math achievement in two experiments. The evaluation was conducted through repeated randomized control trials on two consecutive cohorts of teachers. Since replication is rare in education, the authors wanted to evaluate the relevance of program implementation on its effectiveness.

Two cohorts of 730 teachers and their 13,000 sixth grade students participated in the study in 2008 and 2010. In both cohorts, teachers were randomly assigned to the intervention or to the control group for three years of program implementation.

The PD program, named PON M@t.abel+, was promoted by the Italian Ministry of Education to train math teachers in using strategies close to the students’ everyday life and learning by doing. Training was provided to groups of teachers (15-20 each group) by a tutor using both in-person and online sessions. The training covered different subject areas, such as arithmetic and geometry, and units. During the training, teachers had to implement some of the units in their class and discuss their experiences with the tutor and their colleagues.

To observe the differences in the program effectiveness across the two cohorts, the protocol followed was the same for the two studies and the characteristics of the schools similar.

• Cohort 1 results showed no significant effects on student math achievement after three years (ES = -0.05).
• Cohort 2 results showed significant positive effects (ES = +0.21).

The authors investigated the reasons for the difference in program effectiveness across the cohorts and concluded that it relied on the implementation of the program. Although the protocol followed was the same, some differences were found in the way the program was implemented.

• In cohort 2 there was less time lapse between the enrollment and the beginning of the training, resulting in a lower drop-out rate.
• In cohort 2 a higher compliance was observed with 70% of teachers receiving the full attendance certificate compared to 47% in Cohort 1.
• Teachers in cohort 2 reported more positive impressions about the program compared to teachers in cohort 1.

Based on the results, the study suggests the importance of replicating randomized studies to have robust results and to better study the impact of the program implementation.

Source: Abbiati, G., Argentin, G., Caputo, A., & Pennisi, A. (2021). Repetita Iuvant? A Repeated Randomized Controlled Trial on the Effectiveness of an At-Scale Teacher Professional Development Program. Evaluation Review. https://doi.org/10.1177/0193841X211055354