From examinations to consultations: a physician-reviewed Japanese benchmark for large language models in traditional Kampo medicine
In the authors' words
Objectives: Patients and clinicians increasingly ask large language models (LLMs) about Kampo, Japan's traditional medicine. Yet evaluations of LLMs have measured accuracy on examination questions, which ask what a model knows; a consultation asks whether it can apply that knowledge to the patient at hand. No benchmark had measured that. Materials and Methods: KampoBench comprises 74 consultation scenarios and 310 conversation-specific rubric criteria, following the design of HealthBench. Scenarios, criteria and reference answers were drafted by an LLM and revised by board-certified Kampo physicians. Ten current models from three vendors were evaluated. Because LLM graders are known to favor their own family, each response was graded by a model from each vendor. Results: The best model reached 74.8-86.3% across graders, the physician-revised reference answers 87.9-92.4%. Rankings were stable across graders (Spearman's rho 0.952-0.976), and the family-favoring deviation was 0.4 to 0.6 percentage points, below the bootstrap standard deviation of one measurement; grader leniency varied by 14.3 points. Models were most reliable on criteria about advising that care be sought (83.6%) and least reliable on those requiring a known crude-drug risk to be tied to the case (17.5%), though 79.2% of those responses named the drug. Discussion: The shortfall lay in applying knowledge to the case rather than in holding it - a distinction examination-format evaluations cannot make. Conclusion: Evaluation in traditional medicine should measure the judgments a consultation requires, not recall alone. Quoted alongside the grader that produced it, a KampoBench score is a usable measure of those judgments.
In plain words AI summary
The best AI model scored 74.8-86.3%, while physician answers reached 87.9-92.4%. Models did well at advising patients to seek care (83.6%) but poorly at linking specific drug risks to individual cases (17.5%). The authors conclude that testing should measure consultation judgment, not just knowledge recall.
Written by Kimi K2.5 using only the abstract above. Jev checked that every sentence is supported by the abstract sentences it cites, and removed any that were not. Point at a sentence to highlight its source. 2 sentences were removed for not being supported by the abstract.
Appeared: Monday, October 12. medRxiv. Preprint, not yet peer-reviewed.