The investigators built an open-source benchmark named SeeFood to evaluate image-based dietary assessment. They used photographs from the SNAPMe dataset. An initial evaluation used 80 real-world meal images; subsequent testing examined three finalist models across the full dataset of 1,680 photographs representing 275 participant-days. The source reported that SeeFood code will be released at publication and that curated metadata linkages to SNAPMe image IDs were used for analysis.
Eight commercial and open-source multimodal large language models were compared, with three advancing to full-dataset evaluation. The authors reported GPT-4o mini as having the lowest overall errors, with a clear advantage on carbohydrates. On other nutrients and on day-level diet quality (Healthy Eating Index, HEI-2020) the three finalists performed similarly. The study team emphasized model-specific directions of bias rather than a consistent over- or underestimation pattern across all models.
Absolute error rates remained substantial. The paper reported mean absolute percentage errors (MAPE) in the range of 52% to 90% across nutrients. For carbohydrates, the carbohydrate mean absolute error (MAE) was 13.3 g (95% CI 12.4–14.3). Only 57% of meals fell within ±10 g of the reference carbohydrate value — a threshold the authors flagged as clinically relevant for insulin dosing in type 1 diabetes. Calorie errors for lunch and dinner were comparable to previously published human visual-estimation results, at about 55% MAPE according to the report. Drink-only images produced the largest relative errors: the source reported a 312% MAPE, which the authors attributed to sugars dissolved in liquids and therefore challenging to detect visually.
Food-item identification depended heavily on item visibility. When items were fully visible, the models produced reliable segmentation and detection (Dice = 0.74 as reported). Day-level diet quality assessment using Healthy Eating Index (HEI-2020) showed moderate agreement with reference values; the authors reported an RMSE of 12.5 points and a correlation r = 0.61. These signals indicate that mLLMs can capture broad dietary patterns even while individual nutrient estimates were noisy.
Portion-size estimation emerged as a systematic source of error. The investigators observed progressive underestimation with increasing portion size. The study tested the inclusion of sizing references (sizing markers) and found that markers did not improve accuracy in their experiments, according to the source text. Drink-only photographs were a distinct failure mode because visual cues do not capture dissolved sugars; this produced the highest measured MAPE.
The authors evaluated whether newer or larger models, and chain-of-thought prompting, would improve results. They reported that neither larger/newer models nor chain-of-thought prompting consistently improved nutrient or portion-estimation performance. This finding suggested that incremental model scale or a particular prompting strategy did not solve the main limitations identified in the visual dietary-assessment pipeline.
The study team framed results against clinical thresholds relevant to diabetes management. The carbohydrate MAE and the proportion of meals within ±10 g are presented with direct clinical relevance for insulin dosing in type 1 diabetes. While models could identify food items and estimate broader diet quality, the authors explained that mean absolute percentage errors in the 52%–90% range currently limit the use of these tools for precise insulin dosing or fine-grained nutrient prescriptions. The investigators suggested combining photographic analysis with patient-provided contextual information (for example, recipe details, portion confirmation, or timing relative to insulin) to potentially lower error and burden.
The authors explicitly reported that substantial errors remain and that these errors limit immediate use for precision diabetes management. They also noted that SeeFood and curated metadata linkages will be shared, but the source did not report details of any prospective clinical validation or regulatory pathway. Open questions recorded by the team include whether integration of contextual patient data can consistently reduce errors, how models perform in more diverse real-world settings, and whether human-in-the-loop workflows would achieve clinically acceptable accuracy. The report stressed that this work used de-identified, publicly available SNAPMe images and that the preprint has not been peer reviewed; the authors cautioned against using these findings to guide clinical practice without further validation.