pipette
ESEspañol

A clinical multimodal vision language foundation model with fine-grain explainability

Yang Bai, Haoran Cheng, Yang Zhou, Jun Zhou, Arun James Thirunavukarasu, Yuhe Ke, Jie Yao, Kanae Fukutsu, Chrystie Wan Ning Quek, Ashley Shuen Ying Hong, Laura Gutiérrez, Alessandro Berni, Zhen Ling Teo, Darren Shu Jeng Ting, Brian T. Soetikno, Christopher S Nielsen, Tobias Elze, Zengxiang Li, Sumit Sharma, Nicholas P. Ruthmann, Le Dinh Linh, Hiok Hong Chan, Victor Koh, Marcus Chun Jin Tan, Kelvin Z. Li, Leonard Yip, Ching Yu Cheng, Yih Chung Tham, Gavin Siew Wei Tan, Leopold Schmetterer, Rae Young Kim, Damon Wing Kee Wong, Marcus Ang, Rahat Husain, Jodhbir Singh Mehta, Tin Aung, Lionel Tim‐Ee Cheng, Nguyen Tuan Anh Tran, Chee Leong Cheng, Tien Yin Wong, Darius M. Moshfeghi, Nan Liu, Iain Beehuat Tan, Soon Thye Lim, Eyal Klang, Tony Kiat Hon Lim, Rick Siow Mong Goh, Yong Liu, Daniel Shu Wei Ting

Peer-reviewed journalReal-world use

In the authors' words

Despite their considerable promise, many current medical foundation models remain modality specific and lack transparent reasoning, limiting clinical adoption. To address this gap, we introduce Explainable Vision Language Foundation Model for Medicine (EVLF-FM), a multimodal vision language foundation model that combines broad diagnostic capabilities with fine-grained explainability. The development and testing of EVLF-FM encompassed more than 1.3 million total samples from 18 global datasets across 10 imaging modalities related to six clinical specialties: dermatology, hepatology, ophthalmology, pathology, pulmonology and radiology. External validation employed 13,443 independent test samples from 17 additional datasets across 10 imaging modalities. EVLF-FM supports disease diagnosis, visual question answering and pixel-level visual grounding with reasoning capabilities. In internal validation for disease diagnostics, EVLF-FM achieved the highest average accuracy, 0.895, outperforming leading generalist and specialist models. In medical visual grounding, EVLF-FM also achieved reasonable performance across nine modalities with average accuracy at 0.5 intersection over union (Acc@0.5) of 0.842. External testing further confirmed consistent zero-shot and few-shot performance, with competitive F1 scores despite a smaller model size. A hybrid training strategy combining supervised and visual reinforcement fine-tuning enables EVLF-FM to generate clinically interpretable explanations aligned with visual evidence. Human evaluation of these reasoning outputs using the S.C.O.R.E. framework yielded a high overall mean score of 4.25/5, supporting their clinical interpretability. Current limitations include the need for additional validation across diverse healthcare settings. Prospective clinical studies are required to evaluate the safety, effectiveness and clinical utility of EVLF-FM before routine clinical deployment. A multimodal vision language foundation model, EVLF-FM, integrates extensive clinical data and advanced training techniques, achieving superior diagnostic accuracy and explainability across diverse imaging modalities, enhancing clinical decision-making through robust reasoning and interpretability.

Main resultLimitation the authors admit

Appeared: Saturday, October 31. Nature Medicine. Peer-reviewed journal.

DOI: 10.1038/s41591-026-04655-5