pipette
ENEnglish

Image-Based Dietary Assessment Using Multimodal Large Language Models: Performance, Limitations, and Clinical Applicability in Diabetes

L. Gravellier, C. Zhang, Y. Wu, V. Parameswaran, A. Scheideman, M. Shao, E. Zhang, D. Perelman, C. D. Gardner, D. C. Klonoff, H. Raeder, M. P. Snyder, T. Wang

PreprintUso en el mundo real

En palabras de los autores

Background Image-based dietary assessment is useful for precision diabetes care. Multimodal Large Language Models (mLLMs) may greatly facilitate dietary self-monitoring. However, comprehensive evaluation of their real-world accuracy remains limited. Methods We developed SeeFood, an open-source benchmark, and evaluated eight commercial and open-source mLLMs on 80 real-world meal photographs from the SNAPMe dataset for nutrient and energy estimation (calories, carbohydrates, protein, fat, fiber), food-item identification, and day-level diet quality (Healthy Eating Index, HEI-2020). Three finalists were evaluated on the complete dataset (1680 photographs, 275 participant-days), and the best model was analyzed by meal type, portion size, sizing-marker use, ingredient visibility and meal complexity. Results GPT-4o mini achieved the lowest errors, with a significant advantage for carbohydrates; the three finalists performed similarly on other outcomes and on HEI-2020. Mean absolute percentage errors (MAPE) remained high (52%--90%), and the direction of bias was model-specific. The carbohydrate mean absolute error was 13.3 g (95% CI 12.4--14.3), and only 57% of meals were within +/- 10 g, a clinically relevant threshold for insulin dosing in type 1 diabetes. Calorie errors for lunch and dinner were in the range reported for human estimation on other images (~55% MAPE), whereas drink-only images had the highest relative errors (312% MAPE) because of dissolved sugars. Food items were reliably identified when fully visible (Dice = 0.74), and HEI-2020 estimates showed moderate agreement with reference values (RMSE = 12.5 points; r = 0.61). Larger portions were increasingly underestimated, and sizing references did not improve accuracy. Newer or larger models and chain-of-thought prompting did not consistently improve performance. Conclusions Current mLLMs can extract meaningful dietary information from real-world meal photographs, but substantial errors in nutrient and portion-size estimation currently limit their use for precision diabetes management. Integrating visual assessment with additional contextual information may enable more reliable, lower-burden dietary monitoring.

Resultado principalLimitación que admiten los autores

Apareció: domingo, 25 de octubre. medRxiv. Preprint, todavía sin revisión por pares.

DOI: 10.64898/2026.10.21.26365949