The authors established a normalized benchmark to evaluate vision-language models (VLMs) on multi-sign point-of-care lung ultrasound (LUS) interpretation. They selected 150 POCUS Atlas clips and produced 125 evaluable cases for analysis. The benchmark comprised three clinically relevant tasks: pleural sliding detection (T1); lung-rocket (B-line) and consolidation classification (T2); and Posterolateral Alveolar and/or Pleural Syndrome (PLAPS) assessment, defined as posterolateral consolidation and/or pleural effusion (T3).
For each case the models received 10 uniformly sampled static frames as inputs. For pleural sliding (T1), the authors additionally provided synthetic M-mode images intended to encode temporal motion information from the sampled frames.
All benchmarked tasks used the same sampling strategy: 10 uniformly sampled frames extracted from each clip. Static-frame inputs served as the baseline for morphology-based assessment of signs such as B-lines and consolidation. To capture motion-dependent features, the investigators generated synthetic M-mode representations from the frames and supplied them as an additional input modality specifically for the sliding detection task (T1).
The benchmark compared multiple VLMs, including open-weight models and closed-weight Claude-family models. The study focused on relative performance across tasks and models rather than introducing new architectures.
Pleural sliding—an inherently motion-dependent sign—was the most challenging task for static-frame VLMs. The study reports that supplying synthetic M-mode improved model performance on sliding detection over static frames for some models. However, absolute performance on T1 remained modest with wide uncertainty ranges, and the authors conclude that sliding accuracy did not reach clinically sufficient levels.
This finding indicates that the temporal dynamics crucial for pleural sliding are incompletely recovered by synthetic M-mode derived from static-frame sampling, and that current VLM implementations evaluated here are unreliable for motion-dependent LUS interpretation.
Pathologies that are primarily morphological—specifically B-lines (lung rockets) and anterior consolidation—were discriminable from single frames. On a balanced cohort, models achieved discrimination above chance. Notably, open-weight VLMs obtained stronger F1 scores for lung-rocket and anterior-consolidation labels compared with closed-weight Claude models in these tasks.
A statistically significant between-model difference was observed for lung rockets, although such a significant difference was not reported across all T2 labels. These results suggest that morphology-driven signs on single frames are better suited to current static-frame VLM approaches than motion-dependent signs.
For the composite PLAPS outcome, many models reported high F1 scores. The authors caution that these elevated performance metrics primarily reflected a high positive-class prevalence in the test cohort rather than robust discrimination by the models. In other words, PLAPS F1 was inflated by class imbalance, reducing confidence that the models were truly accurate across balanced clinical populations.
Inter-model agreement was low: the task-averaged Cohen’s kappa across models ranged from 0.02 to 0.29. This near-chance agreement contrasts with each model’s internal self-consistency, indicating that different models made distinct and largely non-overlapping errors. The authors interpret this as evidence of non-redundant failure modes across VLMs and underscore the difficulty of relying on any single model without broader validation.
From the benchmark results the authors draw two practical implications. First, morphology-assisted use of VLMs may be viable for LUS tasks that rely on static-frame features, such as detecting B-lines and anterior consolidation, where discrimination from single frames was observed. Second, tasks that depend on temporal motion (pleural sliding) or outcomes affected by class prevalence (PLAPS) require substantially larger and more balanced validation cohorts before any clinical deployment can be considered safe or effective.
The study highlights that synthetic M-mode can partially recover temporal information but did not restore clinically sufficient accuracy for sliding. The low inter-model agreement also argues for careful multi-model evaluation and error analysis prior to implementation in clinical workflows.
The benchmark dataset and annotations are publicly available at the cited Hugging Face repository. The authors also made the source code, inference and evaluation pipelines, and replication materials available on GitHub. The underlying lung ultrasound cases were sourced from the publicly accessible POCUS Atlas.
As a preprint, this work has not undergone peer review and the authors explicitly note that findings should not guide clinical practice without further validation. The manuscript reports no competing interests. Specific numeric performance metrics beyond the qualitative descriptions and the reported Cohen’s kappa range are presented in the original preprint; any additional numeric detail not provided in this summary should be taken from the source document.
The authors identify key limitations relevant to interpretation: reliance on static-frame sampling that incompletely encodes temporal motion; class imbalance effects inflating some performance measures (notably PLAPS); and relatively small evaluable case counts derived from the available POCUS Atlas clips. They recommend larger, balanced validation cohorts and further work to integrate true temporal imaging or clinically validated motion encodings if VLMs are to be used for motion-dependent LUS signs.
Overall, the benchmark establishes baseline strengths and weaknesses of current VLMs on multi-sign lung ultrasound: morphology-based detection shows promise, but motion-dependent interpretation and prevalence-influenced outcomes remain unresolved.