pipette
ENEnglish

Using Vision and Large Language Models for Psychological Intervention Competency Assessment: A Pilot Study of Agreement with Clinician Ratings

Y. Liu, W. Leever, M. Stanco, Z. Shook, P. Addico, S. Gleeson, F. Nugroho, G. Varney, E. Youngstrom, E. Sezgin

PreprintUso en el mundo real

En palabras de los autores

Introduction: Supervision in psychological intervention training is limited by the time needed to evaluate simulated sessions. We tested a Vision Language Model (VLM) and Large Language Model (LLM) pipeline that assesses competency from verbal and nonverbal communication. Method: Gemini-3.1-Pro was used to annotate behaviors in 25 simulated therapy videos and incorporate them into transcripts, and ChatGPT-6.1-Sol rated them against the ENACT scale. Two clinicians verified VLM-identified behaviors and four independently rated each video. Agreement was estimated with Gwet's AC1 and ICC. Results: Of 1,558 VLM-identified behaviors, 70% were confirmed by both clinicians and 93% by at least one. AI-clinician agreement on therapeutic behavior attributes (AC1=0.73, 95% CI 0.70-0.76) approached between-clinician agreement (0.75, 0.72-0.78). AI-clinician ICCs on therapeutic domain levels (median 0.57) were modestly lower than between-clinician ICCs (0.67). Discussion: The pipeline agreed with clinicians nearly as well as clinicians agreed with each other and shows promise as a supplementary evaluator in therapist training.

Resultado principalLimitación que admiten los autores

Apareció: miércoles, 4 de noviembre. medRxiv. Preprint, todavía sin revisión por pares.

DOI: 10.64898/2026.11.02.26367108