The study team submitted 17 synthetic emergency department vignettes to five consumer-facing large language models on 11–12 August 2026. Models tested were Claude Fable 5, GPT-5.6 Sol, Grok 4.5, Gemini 3.6 Flash and Kimi K3. Two emergency physicians independently reviewed the outputs while blinded to which model produced each response. Reviewers evaluated a total of 85 responses (five models × 17 vignettes) against predefined reference answers and triage levels. Primary outcomes included agreement with the reference diagnoses, agreement with Canadian Triage and Acuity Scale (CTAS) levels, ratings of appropriateness and overall response quality on 1–5 scales, any flagged potential harm, and a preferred response designation when applicable. The preprint states that all data are available on request; the manuscript does not provide additional vignette text, patient demographics, or the exact prompts used in the consumer interfaces.
Every model correctly named the reference diagnosis for all 17 vignettes. This uniform result held across the five tested systems and across the full case set. The investigators reported 17/17 diagnostic matches per model. That finding is direct and unambiguous in the manuscript: diagnostic label agreement was complete in this selected sample. The report did not offer subgroup analyses by clinical scenario type, nor did it provide further granularity about differential diagnoses offered alongside the primary label.
Agreement with the reference CTAS level varied only slightly between models. Exact concordance ranged from 14/17 to 15/17 depending on the model, and the overall statistical comparison produced p=0.938. Across all models there were 13 discordant triage assignments in total; the cohort of discordances included nine instances of undertriage and four instances of overtriage relative to the reference CTAS levels. Two independent emergency physicians judged triage level for each response; model identity was concealed during that process. The authors did not report whether discordances clustered by specific vignette themes or by any particular model beyond the aggregate counts described.
Clinician-rated measures diverged more markedly than diagnostic naming or triage concordance. The study team observed statistically significant differences across models for both appropriateness and overall quality (both p<0.001). Reported mean scores on the 1–5 scales placed Kimi K3 highest for appropriateness (4.82) and overall quality (4.74). Grok 4.5 registered the lowest mean scores (3.91 for appropriateness; 3.71 for overall quality). After Holm adjustment for multiple comparisons, Grok 4.5 scored significantly lower than every other model on both clinician-rated scales.
Reviewers also flagged responses with potential to cause harm. Nine responses were marked as potentially harmful: four originated from Grok 4.5, two from Gemini 3.6 Flash, two from GPT-5.6 Sol, one from Claude Fable 5 and none from Kimi K3. The authors report a p value of 0.119 for the harm comparison across models. The preprint does not list the specific clinical content of these flagged responses in the abstract; therefore the present summary does not invent the underlying omissions or commissions that led to harm flags. Clinician preference between model responses was recorded as an outcome in the study methods, but the abstract provides aggregated preference-related results only insofar as they relate to the quality and appropriateness comparisons described above.
The authors emphasized that these findings arise from a selected set of synthetic vignettes and from consumer interfaces used on two specific dates in August 2026. They stated explicitly that these results do not establish any model's clinical safety. The report discloses that two emergency physicians performed blinded ratings and that all data are available on reasonable request to the authors. The preprint includes statements that relevant ethical guidelines were followed, that participant consent and approvals were obtained where applicable, and that no competing interests were declared.
Limitations described or implied by the source include the selected nature of the vignette set, absence of peer review for the preprint, and lack of publicly provided full vignette text and prompt details in the abstract. The manuscript does not report additional demographic or real-world patient data, nor does it provide prospective clinical testing outcomes. The final lines of the abstract reiterate that, despite perfect diagnostic naming in this case series, undertriage events, potentially harmful omissions and measurable differences in response quality occurred. The authors stop short of asserting model safety, and they do not recommend clinical deployment based on the data reported in the preprint.
Questions that remain open from the preprint include whether similar performance would be seen with broader or real-world case mixes, how defined prompt engineering might alter triage or harm rates, and whether reviewer disagreement occurred beyond the aggregated statistics presented. Readers should treat these results as preliminary; the document is a preprint and has not been certified by peer review.