Tag: Meta-analysis

Categories
Effective Teaching Approach Language Development Primary School Education

Reviewing ed tech’s impact on literacy: It’s a win!

Educational technology tools and programs proliferate the elementary classrooms where children are learning to read, but the evidence supporting these tools lags behind their implementation. Silverman and Colleagues collected and reviewed the results from 119 studies that examined the impact of educational technology interventions on literacy outcomes between 2010 and 2023.

The study found medium to large positive impacts of educational technology on literacy across four outcome domains: decoding (ES = + 0.33), language comprehension (ES = +0.30), reading comprehension (ES = +0.23), and writing proficiency (ES = +0.81). Looking closer at these impacts, effects were smaller on standardized outcome measures (such as state tests), compared to researcher-created measures. Most of the analysis of which intervention and design features mattered was inconclusive, but the authors did find constructivist approaches were more beneficial for language comprehension. One notable limitation of the research was the inclusion of only peer-reviewed studies (including dissertations) – which may have led to more positive results than is observed in studies overall given the bias in publishing for both positive and significant findings.

 

Source: Silverman, R. D., Keane, K., Darling-Hammond, E., & Khanna, S. (2024). The effects of educational technology interventions on literacy in elementary school: A meta-analysis. Review of Educational Research, 00346543241261073. https://doi.org/10.3102/00346543241261073… Read the rest

Categories
Educational Administration and Leadership K-12 Education Language Development

Chatbot interaction shows positive effects on language learning

Chatbots are reminiscent of a gossipy co-worker—full of chatter, not all of it credible, yet nonetheless useful now and then. In a meta-analysis of 28 studies comparing Chatbot users to non-users, Wang, Cheung, Neitzel, & Chai recently examined the effects of chatbots on users’ language learning, and the moderators that affected this interaction in terms of 18 characteristics of the humans and chatbots.

Results showed that chatbot users demonstrated greater gains in language learning than comparable non-users (ES=+0.48). Other findings included that language gains were greatest for secondary school students and adult learners; second language learners at the intermediate level (versus beginning or advanced), particularly those learning English; and for those who used a mobile-based program instead of a laptop or PC. Chatbot-driven interaction showed greater gains than user-driven interactions, and were most effective when used 1:1 in-class instead of out-of-class. Voice-based and voice-plus-text-based interactions both demonstrated positive effects on language learning. Implications for the classroom are that chatbots are a valid in-class language learning tool for older students, especially those learning English as a second language.

 

Source: Wang, F., Cheung, A. C. K., Neitzel, A. J., & Chai, C. S. (2024). Does chatting with chatbots improve language learning performance? A meta-analysis of chatbot-assisted language learning. Review of Educational Research, 00346543241255621. https://doi.org/10.3102/00346543241255621… Read the rest

Categories
K-12 Education Maths and Science Learning Programme Evaluation

Factors influencing the impact of math interventions in K-12 education

It is commonly understood that educational interventions are not universally effective; their impact varies depending on specific contexts and conditions. A systematic review with meta-analysis by Megan Rojo and her colleagues examined the influence of intervention and study characteristics in determining the effects of K-12 programs for students with mathematical difficulties. They focused on grade level, group size, content area, and dosage for the intervention characteristics, and on research design, implementation fidelity, year of study, type of measure, and study quality for study characteristics.

The review included 286 studies, 119 of which were randomized trials, 16 quasi-experiments, and 86 single-subject designs. In the model excluding extreme values, the authors found the following findings for intervention characteristics.

  • Problem-solving interventions and interventions involving operations were found to be less effective than those focusing on fractions. The authors suggested the complex nature of fractions and the specialized knowledge of research teams dedicated to this specific area as potential explanations.
  • No clear pattern was found for duration. Longer interventions (more than 22 hours) seemed to have a larger impact, but further research is needed.
  • Grade level (K–2 vs. 3–5, vs. 6–12) was not found to influence the impact of the interventions.
  • Similar effects were observed in interventions delivered either in small groups or individually, which supports findings from previous reviews.

Study characteristics were considered by the authors because the way the studies were designed and conducted may affect their effect sizes (read related BEiB summaries on this topic here). They found that single-subject designs had larger effects compared to quasi-experimental and randomized studies. Measures created by the researchers produced larger effects than independent measures. They concluded that those features should be addressed in future research.

Source: Rojo, M., Gersib, J., Powell, S. R., Shen, Z., King, S. G., Akther, S. S., Arsenault, T. L., Bos, S. E., Lariviere, D. O., & Lin, X. (2024). A meta-analysis of mathematics interventions: Examining the impacts of intervention characteristics. Educational Psychology Review, 36(1), 9. https://doi.org/10.1007/s10648-023-09843-0… Read the rest

Categories
Educational Administration and Leadership K-12 Education Programme Evaluation

School dropout interventions: Assessing what works

School dropout is a problem of global concern due to the negative personal and social consequences it can lead to. Numerous intervention programs, targeting either all students or specifically those identified as at-risk, have been developed and implemented to reduce the dropout rate, although they have shown mixed results regarding their effectiveness. To investigate whether these inconsistent results are due to intervention characteristics and/or study methods, Wang and colleagues conducted a meta-analysis of 26 experimental or quasi-experimental studies related to school completion in K-12 US education.

The meta-analysis found a positive and statistically significant effect of dropout programs (ES= +0.19). Findings revealed that using randomly assigned methods or quasi-experimental matched methods produced a larger effect size than non-random methods. Moreover, compared to interventions that exclusively targeted academics (ES = +0.12), behavioral interventions (that address emotional and/or behavioral factors contributing to a student’s decision to leave school) and combined interventions produced greater effect sizes (respectively, ES = +0.33 and ES = +0.31). These results suggest integrating both behavioral and multicomponent interventions to tackle school dropout, and ensuring equivalent baseline participant characteristics between treatment and control groups in school dropout studies.

 

Source: Wang, Q., Hsiao, Y.-Y., Hushman, C., & Armstrong, J. (n.d.). The effectiveness of dropout intervention programs among K-12 students: A meta-analysis. Journal of Education for Students Placed at Risk (JESPAR), 0(0), 1–31. https://doi.org/10.1080/10824669.2024.2342779… Read the rest

Categories
Achievement Educational Administration and Leadership K-12 Education

Effectiveness of collaborative teaching models on academic achievement

A recent meta-analysis published in Review of Educational Research examined the effects of co-teaching. Co-teaching and related collaborative teaching models are defined as interventions delivered by two or more teachers or professionals in the same physical space. The authors included randomized and quasi-experimental studies that investigated the effects of collaborative teaching models on student outcomes in arts, social science, and STEM subjects in K-12, published between 1984 and 2020. After excluding studies that did not meet the criteria, as well as another 52 studies for low methodological quality, 76 studies were included in the review.

The mean effect size across the studies was +0.11, with a large variability. Characteristics highlighted in the literature as relevant factors (e.g., duration, teacher training) for the impact of co-teaching did not play a role in the results. Similar results were found for interventions conducted by trained teachers and by teaching assistants without a qualification, suggesting that co-teaching may be implemented in a more cost-effective way.

 

Source: Vembye, M. H., Weiss, F., & Hamilton Bhat, B. (2024). The effects of co-teaching and related collaborative models of instruction on student achievement: A systematic review and meta-analysis. Review of Educational Research, 94(3), 376–422. https://doi.org/10.3102/00346543231186588… Read the rest