Tag: RCT

Categories
Achievement Educational Administration and Leadership Primary School Education Programme Evaluation

Evidence supports Air Reading

Research consistently highlights the benefits of tutoring for improving student learning and closing achievement gaps. However, scaling up in-person tutoring can be challenging in certain settings. As a result, an increasing number of studies are exploring the efficacy of virtual tutoring as a potential solution. A recent study conducted in spring 2024 of the Air Reading program adds valuable insight to this discussion.

Air Reading is a virtual tutoring program providing skills-based instruction in reading, delivered via paid tutors using a virtual synchronous instruction platform. With a 1:3 tutor-to-student ratio, the program is intended to strike a balance between scalability and individualization.

A randomized controlled trial of Air Reading was conducted in a rural Texas district with 418 first-sixth grade students across six schools. Tutored students received four 40-minute sessions per week for one semester. At post-test, Air Reading students outperformed control students on the NWEA MAP reading assessment (ES = + 0.12), equivalent to an average of 1.6 additional months of learning. Treatment students attended an average of 39.8 sessions of tutoring, and those who completed 40 or more sessions demonstrated significantly higher gains than those with fewer sessions (ES = + 0.17).

 

Source (Open Access): Neitzel, A. J., & Storey, N. (2024). Air reading: A randomized evaluation of a virtual tutoring model. https://jscholarship.library.jhu.edu/handle/1774.2/70119… Read the rest

Categories
Maths and Science Learning Programme Evaluation Secondary School Education

Generative AI’s impact on student learning: A double-edged sword

Generative AI tools, like OpenAI’s GPT-4, are increasingly integrated into educational settings, promising enhanced learning and productivity. However, their long-term impact on skill acquisition remains under scrutiny. A recent research paper from the Wharton School of the University of Pennsylvania details a large-scale randomized controlled trial (RCT) designed to assess how generative AI affects student learning, focusing specifically on math classes in a high school setting. The study involved nearly 1,000 students across three grades, evaluating two GPT-4-based tutors: GPT Base and GPT Tutor. The GPT Base provided a standard ChatGPT interface, while GPT Tutor incorporated safeguards to support learning without providing direct answers.

The results revealed that while GPT-4 significantly improved immediate performance on practice problems—by 48% for GPT Base and 127% for GPT Tutor—these gains did not translate into long-term learning. When access to GPT-4 was removed during exams, students who had used GPT Base performed 17% worse than those who never had access, indicating a detrimental effect on learning. However, GPT Tutor mitigated this negative impact, with performance differences becoming statistically insignificant.

The study underscores the potential of generative AI to enhance short-term performance but also highlights the risk of overreliance on these tools, which can inhibit the development of essential problem-solving skills. As educational institutions increasingly turn to AI-driven tools, the findings stress the need for carefully designed safeguards to ensure that students continue to learn and retain critical skills over time.

 

Source (Open Access): Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö., & Mariman, R. (2024). Generative AI can harm learning. The Wharton School Research Paper. https://www.ssrn.com/abstract=4895486… Read the rest

Categories
Effective Teaching Approach Kindergarten Language Development

Enhancing early literacy: Evaluating the impact of the TELL Curriculum in preschool settings

High-quality preschool programs are pivotal in enhancing literacy outcomes and bridging the achievement gap among children from diverse socioeconomic backgrounds. Early childhood education not only supports cognitive and social development but can also play a crucial role in preparing children for future schooling. A recent study published by Gray and colleagues evaluated the efficacy of Teaching Early Literacy and Language (TELL), a universal whole-class curriculum, in enhancing the oral language and early literacy skills of preschool children from low-SES backgrounds.

The study employed a three-cohort randomized controlled trial design with preschool teachers from the Phoenix area randomly assigned to either the TELL or business-as-usual (BAU) condition, stratified by agency/school district. The TELL curriculum included a year-long sequence of instruction, books, lesson plans, and language-rich activities. Despite challenges including COVID-19 disruptions, 58 teachers (30 TELL, 28 BAU) completed the study, with over 300 students (174 TELL, 150 BAU) included in the final analysis.

The findings showed significant effects in favor of TELL on experimenter-created curriculum-based measures, particularly on print awareness, letter naming, and phonological awareness. These foundational skills are critical predictors of future reading success and were systematically taught throughout the curriculum. However, for more distal measures, such as the Woodcock–Johnson IV Tests of Oral Language and PELI assessments of vocabulary and comprehension, the effects were not significant. This discrepancy highlights ongoing debates regarding the appropriateness of different assessment tools in early childhood education and intervention research.

 

Source: Gray, S. I., Wilcox, M. J., & Reiser, M. (2024). Efficacy of the teaching early literacy and language curriculum with preschoolers from low-income families. Language, Speech, and Hearing Services in Schools, 55(3), 696–713. https://doi.org/10.1044/2024_LSHSS-23-00140… Read the rest

Categories
Educational Administration and Leadership Kindergarten Programme Evaluation

Can the LfL PD program enhance teaching quality in Shenzhen kindergartens?

Professional development (PD) programs in China prioritize content knowledge transfer from teachers to students and tend to lack emphasis on enhancing teacher-child interaction. A cluster RCT study by Huang and colleagues investigated whether an adapted western PD intervention will improve both domain-specific instruction quality and teacher-child interaction strategies in Chinese kindergartens.

The LfL program covers diverse content knowledge, intentional and relational instruction, and the use of responsive teacher–child interactions to meet students’ needs. The adapted program included three PD phases: a 1-day intensive training on quality teaching, followed by six 3.5-hour sessions focusing on domain-specific content knowledge in literacy, math, science, self-regulation, diversity, and assessment. The final phase consisted of biweekly onsite mentoring from March to May 2020 to help teachers apply their new skills. All training was conducted face-to-face.

Stratified random sampling was used to select 24 kindergartens in Shenzhen, a developed area in China, dividing them into 12 control and 12 intervention schools comprising 95 classrooms 202 teachers and 547 children aged 3 to 6 years (K1 and K2). Teachers in the intervention group received the adapted LfL training, while those in the control group continued their regular teaching practice. The Early Childhood Environment Rating Scale-Extension (ECERS-E) measured content-related instruction quality and the Sustained Shared Thinking and Emotional Well-being (SSTEW) evaluated the interaction quality of supporting children’s socioemotional well-being and cognitive domain. Classroom observations were conducted once before (fall 2020) and once after the intervention (summer 2021). IDELA was used to assess children’s literacy, numeracy, socioemotional skills, and executive functions. Data collection was conducted by research assistants blind to participant status.

Multilevel analysis revealed that LfL-teachers outperformed control group in delivering knowledge (ECERS-E, ES = +0.81)) and interaction with students (SSTEW, ES = +0.72). Compared to the control group, children in intervention classrooms showed better performance in literacy (ES = +0.16) and executive function (ES = +0.27), but not in numeracy and socioemotional scores.

The authors suggest policymakers provide national PD programs based on an evidence-based approach for more effective resources’ allocation. An effective PD program could be cross-culturally and contextually adapted to strengthen education quality.

 

Source: Huang, R., Siraj, I., & Melhuish, E. (2024). Promoting effective teaching and learning through a professional development program: A randomized controlled trial. Journal of Educational Psychology. https://doi.org/10.1037/edu0000851… Read the rest

Categories
Educational Administration and Leadership Maths and Science Learning Secondary School Education

Effective and economical: Cost analysis of the ASSISTments online math program

With the growing interest in AI-supported and technology-focused interventions, there is a need for more information on the relative cost-effectiveness of interventions to help policymakers choose the ones that maximize student learning but remain within school budgets, which are often tight.

A WestEd study from Feng and colleagues analyzed the cost and impact of the ASSISTments program, an online, 1-to-1 homework support program for math which can be used in and out of the classroom to provide students with instant feedback and teachers with real-time progress reporting on their students. The original impact study (a randomized control trial of 5,912 7th grade students from 63 schools in 41 districts in North Carolina) showed a positive impact of ES=+0.10, which was sustained one year after the program ended.  

The current study builds on this evidence by estimating the cost required to produce these gains. Specifically, they estimated that it cost $46.23 per student for this 0.1 standard deviation increase in math achievement. This estimate (and cost-effectiveness ratio) is quite low compared to many other similar interventions and is more impressive given the rigorous research design that produced the estimated impacts (with randomization; large, diverse sample; and long-term impact estimate).

This study provides a guide for a quick application of cost-analysis methods, including a breakdown of each component’s contribution to the cost (Table 1). Additionally, the authors discuss the potential for alternate implementation approaches and how to integrate them into the cost-analysis.

Source (Open Access): Feng, M., Weiser, G., & Collins, K. (2024). Cost and cost effectiveness of ASSISTments online math support: Analysis from a randomized controlled study in middle school. WestEd. Report_Cost-and-Cost-Effectiveness-of-ASSISTments-Online-Math-Support_FINAL-ADA-v2… Read the rest