---
title: "Performance of five LLMs on emergency vignettes: diagnosis, CTAS triage and response quality"
id: "medrxiv-4-diagnosis-triage-and-response-quality-of-five-large-language-models-on"
canonical_url: "https://medichelpline.com/clinical-feed/medrxiv-4-diagnosis-triage-and-response-quality-of-five-large-language-models-on"
content_type: "clinical_feed_article"
specialty: "Critical Care"
source_name: "medRxiv (Clinical Preprints)"
source_url: "https://www.medrxiv.org/content/10.64898/2026.10.21.26366076v1?rss=1"
published_at: "2026-10-23T12:00:00.000Z"
evidence_level: "Verified Feed"
license: "CC-BY-NC-4.0 / Informational Use"
---
# Performance of five LLMs on emergency vignettes: diagnosis, CTAS triage and response quality
## Provenance & Clinical Metadata
- **Canonical URL:** https://medichelpline.com/clinical-feed/medrxiv-4-diagnosis-triage-and-response-quality-of-five-large-language-models-on
- **Specialty:** [Critical Care](https://medichelpline.com/clinical-feed/critical-care.md)
- **Primary Source:** medRxiv (Clinical Preprints)
- **Source URL:** [Original Journal Publication](https://www.medrxiv.org/content/10.64898/2026.10.21.26366076v1?rss=1)
- **Published At:** 2026-10-23T12:00:00.000Z
- **Evidence Rating:** Verified Feed
## Executive GIST (TL;DR)
- Investigators submitted 17 synthetic **emergency** department vignettes to five consumer large language models on 11–12 August 2026; the models were Claude Fable 5, GPT-5.6 Sol, Grok 4.5, Gemini 3.6 Flash and Kimi K3. - Two emergency physicians independently assessed 85 model responses (five models × 17 vignettes) while blinded to model identity. - Primary outcomes included agreement with reference diagnoses, agreement with Canadian Triage and Acuity Scale (**CTAS**) levels, clinician-rated appropriateness and overall response quality on 1–5 scales, flags for potential harm, and pairwise preference. - Every model correctly named the reference diagnosis in all 17 vignettes; diagnostic naming accuracy was 17/17 per model. - Exact **CTAS** agreement per model ranged from 14/17 to 15/17; the overall comparison yielded p=0.938. - Among 13 discordant triage assignments, nine were undertriage and four were overtriage relative to reference **CTAS**. - Appropriateness and overall quality differed across models (both p<0.001). The study team reported mean appropriateness and quality scores: Kimi K3 scored highest (4.82 and 4.74) and Grok 4.5 lowest (3.91 and 3.71); Grok 4.5 scored significantly lower than every other model after Holm adjustment. - Nine responses were flagged for potential harm: four from Grok 4.5, two from Gemini 3.6 Flash, two from GPT-5.6 Sol, one from Claude Fable 5 and none from Kimi K3; harm comparison p=0.119. - Authors concluded that while diagnostic naming matched references across this selected vignette set, undertriage, potentially harmful omissions, and intermodel differences in response quality occurred; they noted these findings do not establish clinical safety for any model. - The preprint reports methods and aggregated outcomes; the source did not report additional trial-level patient demographics, vignette content details, or specific prompts used beyond the statement that vignettes were synthetic and submitted via consumer interfaces.
## Clinical Analysis & Structured Key Points
[Skip to main content](https://www.medrxiv.org/content/10.64898/2026.10.21.26366076v1?rss=1#main-content) [![medRxiv](https://www.medrxiv.org/sites/default/files/medrxiv_inner_logo.png)](https://www.medrxiv.org/) * [Home](https://www.medrxiv.org/) * [Submit](https://www.medrxiv.org/submit-a-manuscript) * [FAQ](https://www.medrxiv.org/about/FAQ) * [Blog](https://connect.medrxiv.org/news/) * [ALERTS / RSS](https://www.medrxiv.org/content/alertsrss) * [Resources](https://connect.medrxiv.org/resources/) * [About](https://www.medrxiv.org/content/about-medrxiv) Search for this keyword [Advanced Search](https://www.medrxiv.org/search) Follow this preprint # Diagnosis, triage and response quality of five large language models on emergency medicine vignettes: a blinded comparison [ View ORCID Profile](http://orcid.org/0009-0001-3647-5556)Gurinder (Sunny) Johal, [ View ORCID Profile](http://orcid.org/0000-0002-6377-7920)Li (Danny) Liang doi: https://doi.org/10.64898/2026.10.21.26366076 [This article is a preprint and has not been certified by peer review [](https://www.medrxiv.org/about/FAQ#unrefereed)[what does this mean?](https://www.medrxiv.org/about/FAQ#unrefereed)]. It reports new medical research that has yet to be evaluated and so should _not_ be used to guide clinical practice. Gurinder (Sunny) Johal 1 University of British Columbia & Simon Fraser University; * [Find this author on Google Scholar](https://www.medrxiv.org/lookup/google-scholar?link_type=googlescholar&gs_type=author&author%5B0%5D=Gurinder%2B%28Sunny%29%2BJohal%2B "Open in new tab") * [Find this author on PubMed](https://www.medrxiv.org/lookup/external-ref?access_num=Johal%20GS&link_type=AUTHORSEARCH "Open in new tab") * [Search for this author on this site](https://www.medrxiv.org/search/author1%3AGurinder%2B%2528Sunny%2529%2BJohal%2B) * [ORCID record for Gurinder (Sunny) Johal](http://orcid.org/0009-0001-3647-5556 "Open in new tab") * For correspondence: sunnyjohal@gmail.com Li (Danny) Liang 2 University of British Columbia * [Find this author on Google Scholar](https://www.medrxiv.org/lookup/google-scholar?link_type=googlescholar&gs_type=author&author%5B0%5D=Li%2B%28Danny%29%2BLiang%2B "Open in new tab") * [Find this author on PubMed](https://www.medrxiv.org/lookup/external-ref?access_num=Liang%20LD&link_type=AUTHORSEARCH "Open in new tab") * [Search for this author on this site](https://www.medrxiv.org/search/author1%3ALi%2B%2528Danny%2529%2BLiang%2B) * [ORCID record for Li (Danny) Liang](http://orcid.org/0000-0002-6377-7920 "Open in new tab") * [Abstract](https://www.medrxiv.org/content/10.64898/2026.10.21.26366076v1)[](https://www.medrxiv.org/panels_ajax_tab/biorxiv_tab_art/node:1328580/1) * [Info/History](https://www.medrxiv.org/content/10.64898/2026.10.21.26366076v1.article-info)[](https://www.medrxiv.org/panels_ajax_tab/biorxiv_tab_info/node:1328580/1) * [Metrics](https://www.medrxiv.org/content/10.64898/2026.10.21.26366076v1.article-metrics)[](https://www.medrxiv.org/panels_ajax_tab/article_tab_metrics/node:1328580/1) * [Data/Code](https://www.medrxiv.org/content/10.64898/2026.10.21.26366076v1.external-links)[](https://www.medrxiv.org/panels_ajax_tab/article_tab_data_code/node:1328580/1) * [ Preview PDF](https://www.medrxiv.org/content/10.64898/2026.10.21.26366076v1.full.pdf+html)[](https://www.medrxiv.org/panels_ajax_tab/biorxiv_tab_pdf/node:1328580/1) ![Loading](https://www.medrxiv.org/sites/all/modules/contrib/panels_ajax_tab/images/loading.gif) ## Abstract Background: Emergency care requires appropriate triage and management as well as diagnosis. We compared five large language models on these tasks using the same clinical vignettes. Methods: Seventeen synthetic emergency department vignettes were submitted to Claude Fable 5, GPT-5.6 Sol, Grok 4.5, Gemini 3.6 Flash and Kimi K3 through consumer interfaces on 11-12 August 2026. Two emergency physicians independently assessed the 85 responses with model identities concealed. Outcomes were agreement with reference diagnoses and Canadian Triage and Acuity Scale (CTAS) levels, appropriateness and overall quality (1-5 scales), potential harm and preferred response. Results: Every model correctly named the reference diagnosis in all 17 cases. CTAS exact agreement ranged from 14/17 to 15/17 per model (p=0.938). Of 13 discordant assignments, nine were undertriage and four overtriage. Appropriateness and quality differed across models (both p The Chan Zuckerberg Initiative, Cold Spring Harbor Laboratory, the Sergey Brin Family Foundation, California Institute of Technology, Centre National de la Recherche Scientifique, Fred Hutchinson Cancer Center, Imperial College London, Massachusetts Institute of Technology, Stanford University, The University of Edinburgh, University of Washington, and Vrije Universiteit Amsterdam. [Donate to openRxiv ](https://www.zeffy.com/en-US/donation-form/donate-to-make-a-difference-10981) [ Back to top](https://www.medrxiv.org/content/10.64898/2026.10.21.26366076v1?rss=1#page) [ Previous](https://www.medrxiv.org/content/10.64898/2026.10.20.26366068v1 "Data collection and analysis toolkit for maternal and neonatal health research and surveillance during epidemic threats: a pilot testing and mixed-methods evaluation")[Next ](https://www.medrxiv.org/content/10.64898/2026.10.21.26366147v1 "Pharmacist-Led Comprehensive Medication Management and Patient Outcomes in Pediatric Intensive Care Unit Patients: A Multicenter Cohort Study") Posted October 23, 2026. [ Download PDF](https://www.medrxiv.org/content/10.64898/2026.10.21.26366076v1.full.pdf) Print/Save Options [Download PDF](https://www.medrxiv.org/content/medrxiv/early/2026/10/23/2026.10.21.26366076.full.pdf)Full Text & In-line FiguresXML [More Info](https://www.medrxiv.org/about/FAQ#PrintOptions "More Information on Print/Save Options") [ Author Declarations](https://www.medrxiv.org/content/10.64898/2026.10.21.26366076v1#p-5) [ Data/Code](https://www.medrxiv.org/content/early/2026/10/23/2026.10.21.26366076.external-links) [ Email](https://www.medrxiv.org/ "Email this Article") [ Share](https://www.medrxiv.org/) Diagnosis, triage and response quality of five large language models on emergency medicine vignettes: a blinded comparison Gurinder (Sunny) Johal, Li (Danny) Liang medRxiv 2026.10.21.26366076; doi: https://doi.org/10.64898/2026.10.21.26366076 [This article is a preprint and has not been certified by peer review [](https://www.medrxiv.org/about/FAQ#unrefereed)[what does this mean?](https://www.medrxiv.org/about/FAQ#unrefereed)]. It reports new medical research that has yet to be evaluated and so should _not_ be used to guide clinical practice. Share This Article: Copy [![Twitter logo](https://www.medrxiv.org/sites/all/modules/highwire/highwire/images/twitter.png)](https://www.medrxiv.org/highwire_log/share/twitter?link=http%3A%2F%2Ftwitter.com%2Fshare%3Furl%3Dhttps%253A%2F%2Fwww.medrxiv.org%2Fcontent%2F10.64898%2F2026.10.21.26366076v1%26text%3DDiagnosis%252C%2520triage%2520and%2520response%2520quality%2520of%2520five%2520large%2520language%2520models%2520on%2520emergency%2520medicine%2520vignettes%253A%2520a%2520blinded%2520comparison "Share this on Twitter") [![Facebook logo](https://www.medrxiv.org/sites/all/modules/highwire/highwire/images/fb-blue.png)](https://www.medrxiv.org/highwire_log/share/facebook?link=http%3A%2F%2Fwww.facebook.com%2Fsharer.php%3Fu%3Dhttps%253A%2F%2Fwww.medrxiv.org%2Fcontent%2F10.64898%2F2026.10.21.26366076v1%26t%3DDiagnosis%252C%2520triage%2520and%2520response%2520quality%2520of%2520five%2520large%2520language%2520models%2520on%2520emergency%2520medicine%2520vignettes%253A%2520a%2520blinded%2520comparison "Share on Facebook") [![LinkedIn logo](https://www.medrxiv.org/sites/all/modules/highwire/highwire/images/linkedin-32px.png)](https://www.medrxiv.org/highwire_log/share/linkedin?link=http%3A%2F%2Fwww.linkedin.com%2FshareArticle%3Fmini%3Dtrue%26url%3Dhttps%253A%2F%2Fwww.medrxiv.org%2Fcontent%2F10.64898%2F2026.10.21.26366076v1%26title%3DDiagnosis%252C%2520triage%2520and%2520response%2520quality%2520of%2520five%2520large%2520language%2520models%2520on%2520emergency%2520medicine%2520vignettes%253A%2520a%2520blinded%2520comparison%26summary%3D%26source%3DmedRxiv "Publish this post to LinkedIn") [![Mendeley logo](https://www.medrxiv.org/sites/all/modules/highwire/highwire/images/mendeley.png)](https://www.medrxiv.org/highwire_log/share/mendeley?link=http%3A%2F%2Fwww.mendeley.com%2Fimport%2F%3Furl%3Dhttps%253A%2F%2Fwww.medrxiv.org%2Fcontent%2F10.64898%2F2026.10.21.26366076v1%26title%3DDiagnosis%252C%2520triage%2520and%2520response%2520quality%2520of%2520five%2520large%2520language%2520models%2520on%2520emergency%2520medicine%2520vignettes%253A%2520a%2520blinded%2520comparison "Share on Mendeley") [ Citation Tools](https://www.medrxiv.org/ "Citation Tools") [ Get QR code](https://connect.medrxiv.org/qr/2026.10.21.26366076) * [Tweet Widget](http://twitter.com/share?url=https%3A//www.medrxiv.org/content/10.64898/2026.10.21.26366076v1&count=horizontal&via=&text=Diagnosis%2C%20triage%20and%20response%20quality%20of%20five%20large%20language%20models%20on%20emergency%20medicine%20vignettes%3A%20a%20blinded%20comparison&counturl=www.medrxiv.org/content/10.64898/2026.10.21.26366076v1 "Tweet This") Reviews and Context 0 Comment 0 TRIP Peer Reviews 0 Community Reviews 0 Automated Services 0 Blogs/Media 0 Author Videos **Subject Areas** [**All Articles**](https://www.medrxiv.org/content/early/recent) * [Addiction Medicine](https://www.medrxiv.org/collection/addiction-medicine) (634) * [Allergy and Immunology](https://www.medrxiv.org/collection/allergy-and-immunology) (914) * [Anesthesia](https://www.medrxiv.org/collection/anesthesia) (347) * [Cardiovascular Medicine](https://www.medrxiv.org/collection/cardiovascular-medicine) (4944) * [Dentistry and Oral Medicine](https://www.medrxiv.org/collection/dentistry-and-oral-medicine) (490) * [Dermatology](https://www.medrxiv.org/collection/dermatology) (425) * [Emergency Medicine](https://www.medrxiv.org/collection/emergency-medicine) (672) * [Endocrinology (including Diabetes Mellitus and Metabolic Disease)](https://www.medrxiv.org/collection/endocrinology-including-diabetes-mellitus-and-metabolic-disease) (1668) * [Epidemiology](https://www.medrxiv.org/collection/epidemiology) (16164) * [Forensic Medicine](https://www.medrxiv.org/collection/forensic-medicine) (36) * [Gastroenterology](https://www.medrxiv.org/collection/gastroenterology) (1236) * [Genetic and Genomic Medicine](https://www.medrxiv.org/collection/genetic-and-genomic-medicine) (7214) * [Geriatric Medicine](https://www.medrxiv.org/collection/geriatric-medicine) (747) * [Health Economics](https://www.medrxiv.org/collection/health-economics) (1091) * [Health Informatics](https://www.medrxiv.org/collection/health-informatics) (5225) * [Health Policy](https://www.medrxiv.org/collection/health-policy) (1456) * [Health Systems and Quality Improvement](https://www.medrxiv.org/collection/health-systems-and-quality-improvement) (1813) * [Hematology](https://www.medrxiv.org/collection/hematology) (597) * [HIV/AIDS](https://www.medrxiv.org/collection/hiv-aids) (1370) * [Infectious Diseases (except HIV/AIDS)](https://www.medrxiv.org/collection/infectious_diseases) (16429) * [Intensive Care and Critical Care Medicine](https://www.medrxiv.org/collection/intensive-care-and-critical-care-medicine) (1192) * [Medical Education](https://www.medrxiv.org/collection/medical-education) (691) * [Medical Ethics](https://www.medrxiv.org/collection/medical-ethics) (156) * [Nephrology](https://www.medrxiv.org/collection/nephrology) (739) * [Neurology](https://www.medrxiv.org/collection/neurology) (7501) * [Nursing](https://www.medrxiv.org/collection/nursing) (375) * [Nutrition](https://www.medrxiv.org/collection/nutrition) (1114) * [Obstetrics and Gynecology](https://www.medrxiv.org/collection/obstetrics-and-gynecology) (1281) * [Occupational and Environmental Health](https://www.medrxiv.org/collection/occupational-and-environmental-health) (1031) * [Oncology](https://www.medrxiv.org/collection/oncology) (3710) * [Ophthalmology](https://www.medrxiv.org/collection/ophthalmology) (1079) * [Orthopedics](https://www.medrxiv.org/collection/orthopedics) (401) * [Otolaryngology](https://www.medrxiv.org/collection/otolaryngology) (462) * [Pain Medicine](https://www.medrxiv.org/collection/pain-medicine) (493) * [Palliative Medicine](https://www.medrxiv.org/collection/palliative-medicine) (142) * [Pathology](https://www.medrxiv.org/collection/pathology) (717) * [Pediatrics](https://www.medrxiv.org/collection/pediatrics) (1845) * [Pharmacology and Therapeutics](https://www.medrxiv.org/collection/pharmacology-and-therapeutics) (749) * [Primary Care Research](https://www.medrxiv.org/collection/primary-care-research) (781) * [Psychiatry and Clinical Psychology](https://www.medrxiv.org/collection/psychiatry-and-clinical-psychology) (6065) * [Public and Global Health](https://www.medrxiv.org/collection/public-and-global-health) (9918) * [Radiology and Imaging](https://www.medrxiv.org/collection/radiology-and-imaging) (2499) * [Rehabilitation Medicine and Physical Therapy](https://www.medrxiv.org/collection/rehabilitation-medicine-and-physical-therapy) (1500) * [Respiratory Medicine](https://www.medrxiv.org/collection/respiratory-medicine) (1263) * [Rheumatology](https://www.medrxiv.org/collection/rheumatology) (656) * [Sexual and Reproductive Health](https://www.medrxiv.org/collection/sexual-and-reproductive-health) (794) * [Sports Medicine](https://www.medrxiv.org/collection/sports-medicine) (594) * [Surgery](https://www.medrxiv.org/collection/surgery) (795) * [Toxicology](https://www.medrxiv.org/collection/toxicology) (109) * [Transplantation](https://www.medrxiv.org/collection/transplantation) (315) * [Urology](https://www.medrxiv.org/collection/urology) (303) Evaluation/discussion of this paper x Comments 0TRiP 0Community 0Automated 0Blogs/Media 0Video 0 Comments medRxiv aims to provide a venue for anyone to comment on a medRxiv preprint. Comments are moderated for offensive or irrelevant content (this can take ~24 h). Please avoid duplicate submissions. The content of a comment is not endorsed by medRxiv. medRxiv is now using a new commenting interface powered by Hypothes.is. All comments previously submitted via the legacy platform (Disqus) have been transferred to the new platform using the medRxiv_admin account. Please read our [commenting guide](https://connect.biorxiv.org/news/2022/03/21/commenting_on_preprints) for more information. Share this comments tab (click to copy link)Copied! TRiP medRxiv partners with journals and review services to enable posting of peer reviews and editorial decisions related to preprints they are evaluating. Reviews are posted with the consent of the authors. TRiP reviews are part of a peer review pilot project in which participating organizations post peer reviews of manuscripts they are evaluating _There is no TRiP material for this paper._ Community Reviews medRxiv aims to inform readers about online discussion of this preprint occurring elsewhere. The content at the links below is not endorsed by either medRxiv or the preprint's authors. Community reviews for this article: _There are no community reviews for this paper._ Automated Evaluations Certain services provide automated analysis of preprints. Analyses invited by the authors are displayed at the top of this tab. Those done independently of authors are shown underneath . None of these analyses is endorsed by medRxiv. Automated Evaluations: _There are no automated evaluations for this paper._ Blog/Media Links medRxiv aims to inform readers about online discussion of this preprint occurring elsewhere. The content at the links below is not endorsed by either medRxiv or the preprint's authors. Video medRxiv partners with conferences and institutions to display recordings of talks and seminars related to preprints. These are posted with the consent of the authors. Video: _There are no videos for this paper._ Powered by [![Hypothes.is logo](https://connect.biorxiv.org/eval/hypothesis.png)](https://web.hypothes.is/) Powered by [![Altmetric logo](https://connect.biorxiv.org/eval/altmetric.png)](https://www.altmetric.com/) ## Follow this preprint X You can now receive automatic notifications when a preprint is revised, withdrawn, commented on, peer reviewed, or published in a journal. Select the events you would like to follow below and click "Submit". To see all of the preprints you are currently following, please go to the [medRxiv Alerts Page](https://medrxiv.org/alerts). Sign In to Follow this Preprint Email * Email this Articleclose Thank you for your interest in spreading the word about medRxiv. NOTE: Your email address is requested solely to identify you as the sender of this article. Your Email * Your Name * Send To * Enter multiple addresses on separate lines or separate them with commas. You are going to email the following [Diagnosis, triage and response quality of five large language models on emergency medicine vignettes: a blinded comparison](https://www.medrxiv.org/content/10.64898/2026.10.21.26366076v1) Message Subject (Your Name) has forwarded a page to you from medRxiv Message Body (Your Name) thought you would like to see this page from the medRxiv website. Your Personal Message CAPTCHA This question is for testing whether or not you are a human vis
## Related Clinical Research

- [Telemedicine for Gestational Diabetes: Meta-analysis of Randomized Trials](https://medichelpline.com/clinical-feed/pubmed-42913437.md) (DOI: 10.2196/103569)
- [Machine learning predicts in-hospital mortality for ICU patients with diabetes and sepsis](https://medichelpline.com/clinical-feed/pubmed-42907983.md) (DOI: 10.1186/s12911-026-03580-4)
- [Clinicians’ Views on “Fighting the Ventilator”: Survey of ICU Practice and Gaps](https://medichelpline.com/clinical-feed/medrxiv-23-clinicians-perspectives-on-fighting-the-ventilator.md)
- [Psychosocial profile in adults with congenital heart disease by documented anxiety disorder history](https://medichelpline.com/clinical-feed/plos-one-21-psychosocial-profile-of-adults-with-congenital-heart-disease-according-to.md)
- [Serum IL-6, IL-10 and IL6/IL10 Genotypes in Bangladeshi Patients with Diabetic Neuropathy](https://medichelpline.com/clinical-feed/plos-one-14-association-of-il-6-and-il-10-serum-levels-with-genetic-variants-in-diabetic.md)

## Navigation
- [← Back to Critical Care Feed](https://medichelpline.com/clinical-feed/critical-care.md)
- [← All Clinical Specialties](https://medichelpline.com/clinical-feed.md)
## Medical & Regulatory Disclaimer

> [!CAUTION]
> MedicHelpline content is structured for research, educational, and professional discovery purposes. It does not constitute individual medical advice, clinical diagnosis, or treatment recommendations.
> Always verify dosing, contraindications, and regulatory alerts against official product labeling and primary regulatory sources before clinical decision-making.