SIGCSE Virtual 2024
Thu 5 - Sun 8 December 2024
Sat 7 Dec 2024 16:00 - 16:30 at Track 1 - Papers 1: AI (1)

Large language models (LLMs) present an exciting opportunity for generating synthetic classroom data. Such data could include code containing a typical distribution of errors, simulated student behaviour to address the cold start problem when developing education tools, and synthetic user data when access to authentic data is restricted due to privacy reasons. In this research paper, we conduct a comparative study examining the distribution of bugs generated by LLMs in contrast to those produced by computing students. Leveraging data from two previous large-scale analyses of student-generated bugs, we investigate whether LLMs can be coaxed to exhibit bug patterns that are similar to authentic student bugs when prompted to inject errors into code. The results suggest that unguided, LLMs do not generate plausible error distributions, and many of the generated errors are unlikely to be generated by real students. However, with guidance including descriptions of common errors and typical frequencies, LLMs can be shepherded to generate realistic distributions of errors in synthetic code.

Sat 7 Dec

Displayed time zone: (UTC) Coordinated Universal Time change

15:00 - 16:30
Papers 1: AI (1)Conference at Track 1
15:00
30m
Paper
Integrating AI Tutors in a Programming Course
Conference
Iris Ma University of California, Irvine, Alberto Krone-Martins University of California, Irvine, Crista Lopes University of California, Irvine
15:30
30m
Paper
Integrating Natural Language Prompting Tasks in Introductory Programming Courses
Conference
Chris Kerslake Simon Fraser University, Paul Denny The University of Auckland, David Smith University of Illinois at Urbana-Champaign, James Prather Abilene Christian University, Juho Leinonen Aalto University, Andrew Luxton-Reilly The University of Auckland, Stephen MacNeil Temple University
16:00
30m
Paper
Synthetic Students: A Comparative Study of Bug Distribution Between Large Language Models and Computing Students
Conference
Stephen MacNeil Temple University, Magdalena Rogalska Temple University, Juho Leinonen Aalto University, Paul Denny The University of Auckland, Arto Hellas Aalto University, Xandria Crosland Western Governors University