pipette
ENEnglish

From examinations to consultations: a physician-reviewed Japanese benchmark for large language models in traditional Kampo medicine

H. Takata, Y. Harada, H. Inoue, K. Uneda, T. Yoshino, T. Nogami

PreprintUso en el mundo real

En palabras de los autores

Objectives: Patients and clinicians increasingly ask large language models (LLMs) about Kampo, Japan's traditional medicine. Yet evaluations of LLMs have measured accuracy on examination questions, which ask what a model knows; a consultation asks whether it can apply that knowledge to the patient at hand. No benchmark had measured that. Materials and Methods: KampoBench comprises 74 consultation scenarios and 310 conversation-specific rubric criteria, following the design of HealthBench. Scenarios, criteria and reference answers were drafted by an LLM and revised by board-certified Kampo physicians. Ten current models from three vendors were evaluated. Because LLM graders are known to favor their own family, each response was graded by a model from each vendor. Results: The best model reached 74.8-86.3% across graders, the physician-revised reference answers 87.9-92.4%. Rankings were stable across graders (Spearman's rho 0.952-0.976), and the family-favoring deviation was 0.4 to 0.6 percentage points, below the bootstrap standard deviation of one measurement; grader leniency varied by 14.3 points. Models were most reliable on criteria about advising that care be sought (83.6%) and least reliable on those requiring a known crude-drug risk to be tied to the case (17.5%), though 79.2% of those responses named the drug. Discussion: The shortfall lay in applying knowledge to the case rather than in holding it - a distinction examination-format evaluations cannot make. Conclusion: Evaluation in traditional medicine should measure the judgments a consultation requires, not recall alone. Quoted alongside the grader that produced it, a KampoBench score is a usable measure of those judgments.

Resultado principalLimitación que admiten los autores

En palabras simples Resumen con IA

El mejor modelo de IA obtuvo 74.8-86.3%, mientras que las respuestas de médicos alcanzaron 87.9-92.4%. Los modelos funcionaron bien al aconsejar a pacientes que buscaran atención (83.6%) pero mal al vincular riesgos específicos de medicamentos a casos individuales (17.5%). Los autores concluyen que las pruebas deberían medir el juicio de consulta, no solo el recuerdo de conocimiento.

Escrito por Kimi K2.5 usando solo el abstract de arriba. Jev verificó que cada oración esté respaldada por las oraciones del abstract que cita y eliminó las que no. Pasá el cursor por una oración para ver de dónde sale. Se eliminaron 2 oraciones por no estar respaldadas por el abstract.

Apareció: lunes, 12 de octubre. medRxiv. Preprint, todavía sin revisión por pares.

DOI: 10.64898/2026.10.07.26365005