Tag: Meta-analysis

Categories
Achievement Educational Administration and Leadership K-12 Education Social and Motivational Outcomes

Effectiveness of flipped classrooms on student outcomes

Flipped classroom refers to an instructional approach where students first learn new material through videos outside of class, and then use class time for reviewing the content, hands-on activities, and deeper learning.

A meta-analysis conducted by Li and colleagues examined the effects of flipped classrooms on K–12 student outcomes. The authors included randomized and quasi-experimental studies with more than ten students in which flipped classrooms were compared to regular teacher practice. After excluding the studies that did not meet the criteria, 129 studies were included in the review.

The average effect sizes after excluding outliers were +0.53 for student performance, +0.56 for cognitive outcomes, and +0.46 for affective outcomes. Due to the high variability in effects between studies, the authors examined which characteristics are factors (e.g., duration, teacher training) that moderate the effects of flipped classrooms. Region was found to be a significant moderator, with a larger effect in low- and middle-income countries compared to high-income countries. Considering methodological characteristics, the authors found that it is more effective when the same instructor teaches consistently than when instructors change during an intervention. In addition, outcomes that include a wide range of tests show smaller effect sizes than those that use more specific measures, such as a unit test.

Overall, the characteristics related to educational context and research methodology explained a substantial portion of the variability in the overall effects.

 

Source: Li, S., Fu, W., Liu, X., & Hwang, G.-J. (2024). Effectiveness of flipped classrooms for k–12 students: Evidence from a three-level meta-analysis. Review of Educational Research, 00346543241261732. https://doi.org/10.3102/00346543241261732… Read the rest

Categories
Educational Administration and Leadership K-12 Education

The impact of COVID on learning

As federal relief funds for education are about to expire just as schools are opening for the new school year, the effects of COVID-19 on education and student learning remain significant. NWEA MAP assessments have shown that students in the United States have still not returned to pre-COVID learning levels. A recent study by Storey & Zhang investigated the extent of learning losses caused by the pandemic and the impact of school closures and disruptions on various student groups.

Using meta-analytic research methods, the authors identified 30 studies from the U.S. and similar countries that used standardized assessments to measure student achievement before, during, and after the pandemic (with a 2022 cutoff). The analysis illustrated that students lost 0.21 (p= .006) standard deviations of learning during the pandemic school closures, with losses of 0.43 standard deviations in reading and 0.41 standard deviations in mathematics. 

These results highlight the significant and statistically meaningful impact of COVID-19 on education, providing a clearer understanding of the extent of the disruptions. The findings underscore the need for ongoing federal and state support to help educators and school systems address and recover from these losses.

 

Source: Storey, N., & Zhang, Q. (2024). A meta-analysis of the impact of COVID-19 on student achievement. Educational Research Review, 44, 100624. https://doi.org/10.1016/j.edurev.2024.100624… Read the rest

Categories
Achievement Educational Administration and Leadership K-12 Education

Can metacognition predict academic achievement? A meta-analysis of longitudinal studies

Metacognition refers to students’ self-regulation abilities in their learning process, involving planning, monitoring, and assessing one’s understanding and performance. For instance, the degree to which students monitor their progress during preparing for exams and recognize necessary adjustment for academic success. Previous studies indicated early metacognitive predicts subsequent academic achievement. Though meta-analyses examined the relationships between metacognition and academic outcomes, relying primarily on cross-sectional studies would risk bias from common method variance. To address this, He and colleagues conducted a three-level meta-analysis utilizing only longitudinal studies.

The meta-analysis included 28 studies, yielding 66 effect sizes, published between 1990 and 2023. These studies featured time-lag between measures of metacognition and academic outcomes ranging from 3 to 24 months, with an average of 10.3 months. Results revealed a small but significant positive correlation between initial metacognition and subsequent academic achievement (r = +0.22). Moderator analyses found on-line measures (r = +0.32), which utilized behavioral measure (e.g., on-going task performance), were stronger predictors than off-line measures (r=+0.17), which mainly used self-report questionnaire, after controlling other moderators. Effects also varied by metacognition components with largest effect for metacognitive skill (r = +0.34), followed by metacognitive awareness (r=+0.22), metacognitive strategy study (r = +0.21), and metacognitive knowledge (r = +0.17).

Additionally, examining the reversal link, results found a small positive correlation between initial academic achievement and subsequent metacognition (r=+0.12). Furthermore, autoregressive cross-lagged meta-analytic structural equation modelling revealed significant small bidirectional associations: students’ metacognition predicted their subsequent academic performance (beta = +0.038), and their academic outcomes also predicted future metacognition (beta = +0.079).

While longitudinal data shed light on dynamic relationships, the authors caution that causal inferences cannot be definitively drawn from fitting hypothesized model. Overall, the meta-analysis provides evidence that metacognition and achievement reciprocally and weakly predict each other over time.

 

Source: He, G., Chen, S., Lin, H., & Su, A. (2024). The association between initial metacognition and subsequent academic achievement: A meta-analysis of longitudinal studies. Educational Psychology Review, 36(3), 81. https://doi.org/10.1007/s10648-024-09922-w… Read the rest

Categories
Primary School Education Secondary School Education Social and Motivational Outcomes

The fadeout effect: Unpacking the durability of educational interventions

Although educational interventions play a pivotal role in fostering a broad range of student skills, their sustained impact often wanes as time progresses, prompting a deeper exploration into their long-term efficacy. Hart and colleagues conducted a comprehensive meta-analysis, examining the long-term effectiveness of educational interventions aimed at improving cognitive and social-emotional skills in children and adolescents. The study drew on 86 randomized controlled trials (RCTs), incorporating data from 56,662 participants, to determine whether the gains in social-emotional skills were more persistent than those in cognitive skills. The research included studies from eight previous meta-analyses and aimed to overcome the methodological limitations of earlier works by focusing solely on RCTs and outcomes consistently measured across different time points.

The analysis revealed that both cognitive and social-emotional skills demonstrated a significant fadeout in their effects over time. Initial post-test improvements did not persist strongly at follow-up assessments, challenging the expectation that social-emotional skills would endure more robustly than cognitive skills. The study found similar rates of conditional persistence for both types of skills in the short term (6-12 months). However, cognitive skills exhibited slightly greater persistence at longer follow-up intervals (1-2 years).

This meta-analysis highlights the critical need for educational programs that not only focus on short-term gains but also address strategies to maintain these improvements over time. The findings suggest that without sustained intervention, both cognitive and social-emotional skills are subject to significant declines, which calls into question the long-term efficacy of current educational interventions in altering developmental trajectories. The results also emphasize the importance of designing future educational programs with a focus on long-lasting impact, a challenge that remains central to educational research and practice.

 

Source (Open Access): Hart, Emma R., Drew H. Bailey, Sha Luo, Pritha Sengupta, and Tyler W. Watts. (2024). Fadeout and persistence of intervention impacts on social-emotional and cognitive skills in children and adolescents: A meta-analytic review of randomized controlled trials. (EdWorkingPaper: 23-782). Retrieved from Annenberg Institute at Brown University: https://doi.org/10.26300/7J8S-DY98… Read the rest

Categories
Effective Teaching Approach Language Development Primary School Education

Reviewing ed tech’s impact on literacy: It’s a win!

Educational technology tools and programs proliferate the elementary classrooms where children are learning to read, but the evidence supporting these tools lags behind their implementation. Silverman and Colleagues collected and reviewed the results from 119 studies that examined the impact of educational technology interventions on literacy outcomes between 2010 and 2023.

The study found medium to large positive impacts of educational technology on literacy across four outcome domains: decoding (ES = + 0.33), language comprehension (ES = +0.30), reading comprehension (ES = +0.23), and writing proficiency (ES = +0.81). Looking closer at these impacts, effects were smaller on standardized outcome measures (such as state tests), compared to researcher-created measures. Most of the analysis of which intervention and design features mattered was inconclusive, but the authors did find constructivist approaches were more beneficial for language comprehension. One notable limitation of the research was the inclusion of only peer-reviewed studies (including dissertations) – which may have led to more positive results than is observed in studies overall given the bias in publishing for both positive and significant findings.

 

Source: Silverman, R. D., Keane, K., Darling-Hammond, E., & Khanna, S. (2024). The effects of educational technology interventions on literacy in elementary school: A meta-analysis. Review of Educational Research, 00346543241261073. https://doi.org/10.3102/00346543241261073… Read the rest

Categories
Educational Administration and Leadership K-12 Education Language Development

Chatbot interaction shows positive effects on language learning

Chatbots are reminiscent of a gossipy co-worker—full of chatter, not all of it credible, yet nonetheless useful now and then. In a meta-analysis of 28 studies comparing Chatbot users to non-users, Wang, Cheung, Neitzel, & Chai recently examined the effects of chatbots on users’ language learning, and the moderators that affected this interaction in terms of 18 characteristics of the humans and chatbots.

Results showed that chatbot users demonstrated greater gains in language learning than comparable non-users (ES=+0.48). Other findings included that language gains were greatest for secondary school students and adult learners; second language learners at the intermediate level (versus beginning or advanced), particularly those learning English; and for those who used a mobile-based program instead of a laptop or PC. Chatbot-driven interaction showed greater gains than user-driven interactions, and were most effective when used 1:1 in-class instead of out-of-class. Voice-based and voice-plus-text-based interactions both demonstrated positive effects on language learning. Implications for the classroom are that chatbots are a valid in-class language learning tool for older students, especially those learning English as a second language.

 

Source: Wang, F., Cheung, A. C. K., Neitzel, A. J., & Chai, C. S. (2024). Does chatting with chatbots improve language learning performance? A meta-analysis of chatbot-assisted language learning. Review of Educational Research, 00346543241255621. https://doi.org/10.3102/00346543241255621… Read the rest