Generative AI tools, like OpenAI’s GPT-4, are increasingly integrated into educational settings, promising enhanced learning and productivity. However, their long-term impact on skill acquisition remains under scrutiny. A recent research paper from the Wharton School of the University of Pennsylvania details a large-scale randomized controlled trial (RCT) designed to assess how generative AI affects student learning, focusing specifically on math classes in a high school setting. The study involved nearly 1,000 students across three grades, evaluating two GPT-4-based tutors: GPT Base and GPT Tutor. The GPT Base provided a standard ChatGPT interface, while GPT Tutor incorporated safeguards to support learning without providing direct answers.
The results revealed that while GPT-4 significantly improved immediate performance on practice problems—by 48% for GPT Base and 127% for GPT Tutor—these gains did not translate into long-term learning. When access to GPT-4 was removed during exams, students who had used GPT Base performed 17% worse than those who never had access, indicating a detrimental effect on learning. However, GPT Tutor mitigated this negative impact, with performance differences becoming statistically insignificant.
The study underscores the potential of generative AI to enhance short-term performance but also highlights the risk of overreliance on these tools, which can inhibit the development of essential problem-solving skills. As educational institutions increasingly turn to AI-driven tools, the findings stress the need for carefully designed safeguards to ensure that students continue to learn and retain critical skills over time.
Source (Open Access): Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö., & Mariman, R. (2024). Generative AI can harm learning. The Wharton School Research Paper. https://www.ssrn.com/abstract=4895486