Effect of AI powered virtual agents on learning outcomes

Decades of research and technological progress have facilitated more sophisticated natural language-based interactions and adaptive learning through artificial intelligence. A recent meta-analysis by Dai and colleagues examined the impact of AI-powered virtual agents in computer-based simulations on learning performance. Two sample studies regarding virtual agents were presented: in research conducted by Shiban and colleagues, virtual agents equipped with AI capabilities were designed as a human figure, offering feedback through both text and gestures. In another study by Le and Wartschinski, an AI-powered virtual agent employed a humanlike name and engaged in text-based interactions, assuming the role of a mentor.

The meta-analysis included experimental studies comparing learning outcomes between groups using computer-based simulations featuring AI-powered versus non-AI virtual agents. Performance-based measures rather than self-reports were analyzed from 22 studies published mainly between 2018-2021 (90% of studies), yielding 49 effect sizes. Random effects modeling revealed a moderate overall effect (Hedges’ g = 0.43).

Moderation analysis uncovered several insights:

• Module-based AI technology achieved the highest impact (g = 0.50), followed by natural language processing/machine learning (g = 0.42) and rule-based designs (g = 0.23).
• Human-like with text agents produced the greatest gains (g = 0.78) relative to fictional characters (g. cartoon; g = 0.55) or human avatars (g = 0.35).
• Interaction modality (text, voice, multi-modal) and agent role (e.g., guidance, feedback) had no significant differential impact.
• While intervention duration significantly moderated outcomes, the findings were indeterminate as to whether longer and shorter intervention lengths would produce better results.

Though limited in number of studies, this work provides initial empirical support for AI-powered virtual characters to augment learning via computer simulations. Continued research employing varied methodologies can help elucidate design principles to maximize their instructional potential.

 

Source: Dai, C.-P., Ke, F., Pan, Y., Moon, J., & Liu, Z. (2024). Effects of artificial intelligence-powered virtual agents on learning outcomes in computer-based simulations: A meta-analysis. Educational Psychology Review, 36(1), 31. https://doi.org/10.1007/s10648-024-09855-4