pipette
ESEspañol

Calibrated large language models demonstrate a consistent advantage in a randomized longitudinal medical education intervention

J. Khurana, H. A. Zaki, A. Dawer, E. Pavlick, S. Gupta, S. S. P. Datla, S. Eldeeb, S. Chaudhry, S. A. Rizvi

PreprintBold claims, read criticallyReal-world use

In the authors' words

Medical trainees increasingly use large language models to review lecture material, but the educational value of that output depends on how it is generated, with prompt and model choice rarely measured. We tested a calibrated pipeline converting Neurology Grand Rounds recordings into summaries and quizzes in a year-long randomized study. In Phase 1, six summary and six question prompts across two models were scored with 2,692 rubric annotations. Claude Prompt 0 produced the best summaries (composite 76.9%) and GPT Prompt 1 the best questions (79.6%); quality varied widely by prompt, and neither model won both tasks. In Phase 2, students in 11 clerkship cohorts over a year were randomized within cohort to receive a summary before each quiz. Summary recipients reported reviewing more and studying faster and scored higher in 9 of 11 cohorts (p = 0.026), although the between-student difference was small (+0.28 of 10 points; P = 0.28).

Main resultLimitation the authors admit

Appeared: Saturday, October 31. medRxiv. Preprint, not yet peer-reviewed.

DOI: 10.64898/2026.10.30.26366878