pipette
ENEnglish

Development and Preliminary Evaluation of the Patient-Artificial Intelligence Rating Instrument

A. Rodman, E. Lowe, E. J. Thomas, Y. Benarroch, L. Ngo, T. Salant, F. C. Bourgeois, M. Newton, C. Walton, S. K. Bell

Preprint

En palabras de los autores

Importance: Patients increasingly use artificial intelligence (AI) for health-related conversations, but evaluation measures largely reflect clinician judgment. Objective: To develop and preliminarily evaluate a patient instrument assessing the quality of patient-AI health conversations. Design: Cross-sectional survey conducted June 5-14, 2026, following patient-partnered instrument development and cognitive interviews. Setting: An anonymous online survey. Participants: A convenience sample of adults with prior use of large language models for personal health-related queries, recruited from an academic medical center-affiliated Insight Panel and patient peers. Exposures: Participants' own prior health-related conversations with AI tools of their choosing. Main Outcomes and Measures: The Patient-AI Rating instrument comprised 12 communication-quality and patient engagement items rated from 1 to 5 and a separately scored patient-perceived-harm item. Internal consistency was assessed using Cronbach alpha, with sensitivity analyses using first ratings only for those using the tool to rate multiple conversations and participant-mean profiles. We examined item-total correlations, inter-item structure, and between- and within-participant variation. Results: Fourteen participants contributed 46 complete ratings; among 13 providing demographics, median age was 63 years (IQR, 53-70 years), and 8 (62%) reported female gender. Participants provided a median of 2 ratings (range, 1-17). Cronbach alpha was 0.88 (95% CI, 0.83-0.93) across all ratings, 0.85 using first ratings only, and 0.92 using participant-mean profiles. Corrected item-total correlations ranged from 0.42 to 0.72; mean inter-item correlation was 0.40. No item deletion improved Cronbach alpha. The mean quality score was 4.43 (SD, 0.45). Among 9 participants contributing multiple ratings, the intraclass correlation coefficient was 0.45, suggesting inter-conversation variability. No perceived harm was reported in 44 of 46 ratings (96%); the harm item correlated minimally with the instrument total (r = 0.09). Conclusions and Relevance: In this small cross-sectional study of engaged patients, the Patient-AI Rating instrument showed promising internal consistency and variation across rated conversations. These findings support further evaluation of patient-reported communication quality alongside clinician-based measures. Validation in larger, more diverse populations, including assessment of construct and criterion validity, is needed before broader use.

Resultado principalLimitación que admiten los autores

Apareció: viernes, 23 de octubre. medRxiv. Preprint, todavía sin revisión por pares.

DOI: 10.64898/2026.10.20.26366053