Synthetic Students: A Comparative Study of Bug Distribution Between Large Language Models and Computing Students
Large language models (LLMs) present an exciting opportunity for generating synthetic classroom data. Such data could include code containing a typical distribution of errors, simulated student behaviour to address the cold start problem when developing education tools, and synthetic user data when access to authentic data is restricted due to privacy reasons. In this research paper, we conduct a comparative study examining the distribution of bugs generated by LLMs in contrast to those produced by computing students. Leveraging data from two previous large-scale analyses of student-generated bugs, we investigate whether LLMs can be coaxed to exhibit bug patterns that are similar to authentic student bugs when prompted to inject errors into code. The results suggest that unguided, LLMs do not generate plausible error distributions, and many of the generated errors are unlikely to be generated by real students. However, with guidance including descriptions of common errors and typical frequencies, LLMs can be shepherded to generate realistic distributions of errors in synthetic code.
Sat 7 DecDisplayed time zone: (UTC) Coordinated Universal Time change
15:00 - 16:30 | |||
15:00 30mPaper | Integrating AI Tutors in a Programming Course Conference Iris Ma University of California, Irvine, Alberto Krone-Martins University of California, Irvine, Crista Lopes University of California, Irvine | ||
15:30 30mPaper | Integrating Natural Language Prompting Tasks in Introductory Programming Courses Conference Chris Kerslake Simon Fraser University, Paul Denny The University of Auckland, David Smith University of Illinois at Urbana-Champaign, James Prather Abilene Christian University, Juho Leinonen Aalto University, Andrew Luxton-Reilly The University of Auckland, Stephen MacNeil Temple University | ||
16:00 30mPaper | Synthetic Students: A Comparative Study of Bug Distribution Between Large Language Models and Computing Students Conference Stephen MacNeil Temple University, Magdalena Rogalska Temple University, Juho Leinonen Aalto University, Paul Denny The University of Auckland, Arto Hellas Aalto University, Xandria Crosland Western Governors University |