Category: Programme Evaluation

Categories
Educational Administration and Leadership Primary School Education Programme Evaluation Secondary School Education

Strategies to promote teacher effectiveness

The Institute of Education Sciences has released a new evaluation brief that synthesizes findings from two impact studies conducted by the National Center for Education Evaluation (NCEE). One study focused on a strategy of providing teachers with feedback on their performance for two years (performance feedback), and the other study focused on a strategy of providing teachers with bonuses for four years based on their performance (pay-for-performance). Both strategies were supported by the Teacher Incentive Fund, which provided competitive grants to help US states and districts implement a multi-strategy approach to enhancing teacher effectiveness.

In each study, elementary and middle schools were randomly assigned to implement the strategy (the treatment group) or not (the control group). The performance feedback study included approximately 29,000 pupils and 1,000 teachers in grades 4–8, while the pay-for-performance study included approximately 38,000 pupils and 3,500 teachers in grades 3–8. Pupil outcomes were measured using end-of-year reading and math scores.

Key findings were as follows:

  • Providing teachers with feedback on their performance for two years improved pupils’ math achievement after the first year with a difference in scores that corresponds to an effect size of +0.05. The cumulative effect after two years of implementation was similar in magnitude but not statistically significant. The effect on reading in both years was positive but not statistically significant.
  • Providing teachers with bonuses based on their performance for four years improved pupils’ reading achievement after one, two and three years of implementation and pupils’ math　achievement after three years. After each of those periods of implementation, the effect size was +0.04 for reading and +0.06 for math. However, as noted in the evaluation report, the impacts of pay-for-performance on classroom observation ratings did not appear to explain the impacts on pupil achievement, and in treatment schools, as many as 40% of teachers were unaware that they could earn a performance bonus.

The brief was prepared for NCEE by Andrew Wayne and Michael Garet of American Institutes for Research and Alison Wellington and Hanley Chiang of Mathematica Policy Research

 

Source (Open Access)：National Center for Education Evaluation (2018). Promoting educator effectiveness- The effects of two key strategies. Retrieved from https://ies.ed.gov/ncee/pubs/20184009/pdf/20184009.pdf… Read the rest

Categories
Kindergarten Primary School Education Programme Evaluation Secondary School Education

How powerful is your evidence?

To offer a beacon through the often muddy waters of interpreting evidence, The Abdul Latif Jameel Poverty Action Lab has released an informal guide describing factors that affect randomised evaluations’ statistical power, the sensitivity of an evaluation to detect any change brought about by the programme. Six Rules of Thumb for Determining Sample Size and Statistical Power describes how relationships between these factors affect a study’s design and results.

The main points outlined in the guide are:

  • Larger sample sizes, which are the amount of subjects in a study, increase statistical power.
  • If effect sizes are small, larger sample sizes can help achieve a given level of power.
  • Evaluations of small programmes need larger sample sizes.
  • If outcomes vary drastically among study subjects, a larger sample size is needed.
  • Study subjects should be divided equally between experimental and control groups.
  • In randomised evaluations, randomising in “clusters,” or groups, is less powerful than individual random assignment.

A companion piece on the dangers of performing under-powered evaluations can be found here.

 

Source (Open Access) : The Abdul Latif Jameel Poverty Action Lab (2018). Six rules of thumb for determining sample size and statistical power. Retrieved from https://www.povertyactionlab.org/sites/default/files/resources/2018.03.21-Rules-of-Thumb-for-Sample-Size-and-Power.pdf… Read the rest

Categories
Primary School Education Programme Evaluation Social and Motivational Outcomes

Social skills intervention programme shows small positive effects

Results from a study published in the Journal of Education Psychology suggest that a classroom social skills programme, The Social Skills Improvement System Classwide Intervention Program (SSIS-CIP), generally has small positive effects on social skills and approaches to learning.

James Clyde DiPerna and colleagues from the Pennsylvania State University evaluated the effects of SSIS-CIP on the social, behavioural and academic outcomes of Year 2 pupils from six primary schools in the mid-Atlantic region of the US. Classrooms were randomly assigned to either treatment or business-as-usual control groups. Teachers assigned to the treatment group implemented the SSIS-CIP over a 12-week period. Outcomes were assessed via teacher ratings and direct observations of classroom behaviour as well as computer-adaptive tests of reading and maths.

Results showed that:

  • SSIS-CIP has a small positive effect on social skills across all social skills subscales (effect sizes ranged from +0.13 to +0.31), with empathy and social engagement showing the largest positive effects (+0.31 and +0.21).
  • The direct observation measure, however, yielded the smallest effect size (+0.05).
  • Students in the treatment group also demonstrated positive effects on academic motivation and engagement (+0.17).
  • However, SSIS-CIP did not demonstrate any substantial effects for problem behaviours, with effect sizes across subscales ranging from +0.01 to +0.07.

Larger effects were also found among second-grade students. The authors suggest that if this pattern could be replicated in the future studies, educators should consider prioritizing second grade for implementation of the SSIS-CIP.

 

DiPerna, J. C., Lei, P., Cheng, W., Hart, S. C., & Bellinger, J. (2018). A cluster randomized trial of the Social Skills Improvement System-Classwide Intervention Program (SSIS-CIP) in first grade. Journal of Educational Psychology, 110(1), 1–16.… Read the rest

Categories
Primary School Education Programme Evaluation

Can schools help prevent childhood obesity?

A study published in The BMJ tests the effectiveness of a school and family based healthy lifestyle intervention (WAVES) in preventing childhood obesity in England.

Almost 1,500 pupils, aged five- and six-years-old, from 54 primary schools in the West Midlands took part in a randomised controlled trial of the WAVES programme. The twelve-month intervention encouraged healthy eating and physical activity, and included an additional 30 minutes of daily physical activity at school and a six-week programme with a local premiership football club.

Children’s measurements – including weight, height, percentage body fat, waist circumference, skinfold thickness and blood pressure – were taken when they started the trial. These measurements were taken again 15 months and 30 months later and were compared with children in a control group.

The results were:

  • At the first follow-up at 15 months, the mean body mass index (BMI) score was not significantly lower for the intervention group compared with the control group.
  • At 30 months, the mean difference was smaller and remained non-significant.

The results suggest that schools alone may not be effective in preventing childhood obesity.

 

Adab, P., Pallan, M. J., Lancashire, E. R., Hemming, K., Frew, E., Barrett, T., … Cheng, K. K. (2018). Effectiveness of a childhood obesity prevention programme delivered through schools, targeting 6 and 7 year olds: cluster randomised controlled trial (WAVES study). BMJ, 360, k211.… Read the rest

Categories
Programme Evaluation Secondary School Education Social and Motivational Outcomes

Preventing depression in secondary school pupils

In Australia, Helen Christensen and colleagues conducted a cluster randomised trial to investigate the effectiveness of an intervention for the prevention of depression in secondary school pupils.

The study, published in the Journal of Medical Internet Research reported on the results of a trial of the SPARX-R programme, a gamified online cognitive behaviour intervention that is delivered to pupils prior to facing a significant stressor – in this case final secondary school exams.

A total of 540 final-year pupils from 10 secondary schools in Sydney, Australia, took part and clusters at the school level were randomly allocated to SPARX-R or the control intervention (lifeSTYLE, an online interactive control programme). Interventions were delivered weekly in class under teacher supervision, in seven 20- to 30-minute modules. Symptoms of depression were measured by the Major Depression Inventory (MDI).

  • Pupils in the SPARX-R group showed a greater reduction in MDI scores than those in the control group, both post-intervention and at the 6-month follow-up.
  • Effect sizes were small post-intervention (+0.29) and at the 6-month (+0.21) and 18-month follow-ups (+0.33).

The study highlights the potentials of using an engaging interactive tool for developing cognitive behaviour therapy skills and depression prevention in advance of a key stressor.

 

Perry, Y., Werner-Seidler, A., Calear, A., Mackinnon, A., King, C., Scott, J., … Batterham, P. J. (2017). Preventing Depression in Final Year Secondary Students: School-Based Randomized Controlled Trial. Journal of Medical Internet Research, 19(11), e369.… Read the rest

Categories
Kindergarten Language Development Maths and Science Learning Programme Evaluation Social and Motivational Outcomes

Self-regulation intervention improves school readiness

Adding a self-regulation intervention to a school readiness programme can improve self-regulation, early academic skills and school readiness in children at higher risk for later school difficulties, according to the results of a study published in Early Childhood Research Quarterly.

Robert J Duncan and colleagues looked at the effect of adding a self-regulation intervention to the Bridge to Kindergarten (B2K) programme – a three-week summer school-readiness programme – in the US state of Oregon. The B2K programme is aimed at children with no prior preschool experience, and therefore considered to be at risk for later school difficulties.

Children from three to five years old were randomly assigned to either a control group (B2K only) or the intervention group (B2K plus intervention). Children in the intervention group received two 20- to 30-minute sessions per week, involving movement and music-based games that encouraged them to practise self-regulation skills.

Results from this randomised controlled trial indicated that

  • Children who received the intervention scored higher on measures of self-regulation than children who participated in the B2K programme alone.
  • There were no significant effects on maths or literacy at the end of the programme.
  • However, four months into kindergarten, children from the intervention group showed increased growth in self-regulation, maths and literacy compared to expected development.

The study concluded that policies and programs aimed at children without schooling experience before attending kindergarten might reduce the school readiness gaps and improve their achievement.

 

Duncan, R. J., Schmitt, S. A., Burke, M., & McClelland, M. M. (2018). Combining a kindergarten readiness summer program with a self-regulation intervention improves school readiness. Early Childhood Research Quarterly, 42, 291–300.… Read the rest