Evaluation administrators within the Boston Public School (BPS) system received a total of 15 hours training in one of four semesters across 2013-14 to 2014-15 school years. The content focused training aimed on improving the quality and frequency of providing teacher feedback.  BPS worked in conjunction with Kraft and Christian, who recently published the results of their randomized field experiment, to evaluate the effectiveness of the training.  The researchers utilized randomized block design to assign school-based evaluation teams to one of the four training sessions. Schools assigned to fall or spring of the 2013-2014 school year served as treatment group while those trained during the fall or spring of the 2014-2015 school year served as control group.

While a survey of the evaluators attending the training revealed favorable reviews of the content and an intent to implement the content in practice, surveys of teachers receiving the feedback indicated only about 25% of teachers felt that evaluators feedback positively affected their instruction.  Comparisons of teacher surveys from the treatment and control groups revealed that:

• No effects were found on feedback quality, number of observations, number of meetings, and student achievement.
• The only significant improvement noted was a decrease of 1.3 days in the time between a teacher observation and the follow-up meeting.
• However, significant negative effects were noted on teacher self-efficacy for classroom management (ES = – 0.20) and on teacher self-efficacy for instructional strategies (ES = – 0.19).

Given the null and negative effects resulting from the training, the researchers provide possible explanations.

• Overloaded schedules, which on average required evaluators to perform between 50 and 60 formal observations per year, may be partially responsible for a lack of implementation of the techniques promoted in the training.
• The perceived effectiveness of the feedback appears to be dependent upon the characteristics of evaluators: teachers reported higher quality feedback (ES = + 0.19) from more experienced administrators (6-8 years of experience) than less experienced administrators (0-2 years of experience);
• and higher quality feedback from racially matched evaluators (ES = +0.30 for African-Americans; ES = +0.29 for Hispanics; ES = +0.35 for Asians).

The findings of this study may help to inform future efforts to improve teacher effectiveness both within BPS and other districts.

Source: Kraft, M. A., & Christian, A. (2021). Can teacher evaluation systems produce high-quality feedback? An administrator training field experiment. American Educational Research Journal. https://doi.org/10.3102/00028312211024603