pipette
ESEspañol

Image-Based Dietary Assessment Using Multimodal Large Language Models: Performance, Limitations, and Clinical Applicability in Diabetes

L. Gravellier, C. Zhang, Y. Wu, V. Parameswaran, A. Scheideman, M. Shao, E. Zhang, D. Perelman, C. D. Gardner, D. C. Klonoff, H. Raeder, M. P. Snyder, T. Wang

PreprintReal-world use

In the authors' words

Background Image-based dietary assessment is useful for precision diabetes care. Multimodal Large Language Models (mLLMs) may greatly facilitate dietary self-monitoring. However, comprehensive evaluation of their real-world accuracy remains limited. Methods We developed SeeFood, an open-source benchmark, and evaluated eight commercial and open-source mLLMs on 80 real-world meal photographs from the SNAPMe dataset for nutrient and energy estimation (calories, carbohydrates, protein, fat, fiber), food-item identification, and day-level diet quality (Healthy Eating Index, HEI-2020). Three finalists were evaluated on the complete dataset (1680 photographs, 275 participant-days), and the best model was analyzed by meal type, portion size, sizing-marker use, ingredient visibility and meal complexity. Results GPT-4o mini achieved the lowest errors, with a significant advantage for carbohydrates; the three finalists performed similarly on other outcomes and on HEI-2020. Mean absolute percentage errors (MAPE) remained high (52%--90%), and the direction of bias was model-specific. The carbohydrate mean absolute error was 13.3 g (95% CI 12.4--14.3), and only 57% of meals were within +/- 10 g, a clinically relevant threshold for insulin dosing in type 1 diabetes. Calorie errors for lunch and dinner were in the range reported for human estimation on other images (~55% MAPE), whereas drink-only images had the highest relative errors (312% MAPE) because of dissolved sugars. Food items were reliably identified when fully visible (Dice = 0.74), and HEI-2020 estimates showed moderate agreement with reference values (RMSE = 12.5 points; r = 0.61). Larger portions were increasingly underestimated, and sizing references did not improve accuracy. Newer or larger models and chain-of-thought prompting did not consistently improve performance. Conclusions Current mLLMs can extract meaningful dietary information from real-world meal photographs, but substantial errors in nutrient and portion-size estimation currently limit their use for precision diabetes management. Integrating visual assessment with additional contextual information may enable more reliable, lower-burden dietary monitoring.

Main resultLimitation the authors admit

Appeared: Sunday, October 25. medRxiv. Preprint, not yet peer-reviewed.

DOI: 10.64898/2026.10.21.26365949