Can LLMs Simulate Novice Programmers' Misconceptions?
In the authors' words
We evaluate 13 LLMs on generating, solving, simulating, and diagnosing novice programming misconceptions on code-tracing problems. While frontier models reliably perform these tasks, small models (14B) struggle with dynamic execution. Notably, code-tuned models fail during misconception simulation by reverting to correct execution. Finally, misconception diagnosis is significantly easier in True/False formats than in open-ended generation tasks.
Main resultThe abstract does not state a limitation.
Appeared: Friday, October 9. arXiv. Preprint, not yet peer-reviewed.