pipette
ESEspañol

Can LLMs Simulate Novice Programmers' Misconceptions?

Batuhan Yeltekin, Daniel Bauer

Preprint

In the authors' words

We evaluate 13 LLMs on generating, solving, simulating, and diagnosing novice programming misconceptions on code-tracing problems. While frontier models reliably perform these tasks, small models (14B) struggle with dynamic execution. Notably, code-tuned models fail during misconception simulation by reverting to correct execution. Finally, misconception diagnosis is significantly easier in True/False formats than in open-ended generation tasks.

Main resultThe abstract does not state a limitation.

Appeared: Friday, October 9. arXiv. Preprint, not yet peer-reviewed.