Large language models (LLMs) have demonstrated growing capability on medical knowledge tasks, but their performance in extracting structured clinical information from real-world clinical documentation is not well characterised. This study developed an open-source framework to evaluate LLM performance relative to medical professionals when extracting SNOMED-CT coded clinical facts from Ear, Nose and Throat (ENT) electronic health records (EHRs) sourced from the publicly available MTSamples dataset.
The investigation focused on reliability and accuracy across multiple clinical content categories, with the objective of informing potential roles for LLMs in clinical documentation workflows.
Dataset and annotations
Ninety-eight ENT clinical documents from MTSamples were used. Fourteen medical professionals who were also study authors independently annotated each document. Annotators extracted information across seven predefined clinical categories: socio-demographics, symptoms, signs, diagnoses, treatments, risk factors, and test results. All extracted items were assigned SNOMED-CT codes to enforce standardised terminology and enable direct comparison across annotators and models.
Models evaluated
Seven LLMs were evaluated. The models named in the source include GPT-4o, Claude 3.5, Gemini 1.5 Pro, Gemma 3, and three LLAMA variants. Model outputs were mapped to SNOMED-CT codes to align with human annotations for comparison.
Agreement and statistical analysis
Inter-rater reliability was quantified using Cohen's Kappa. To compare clinician–clinician (medic–medic) agreement with clinician–model (medic–LLM) agreement and to assess non-inferiority of medic–LLM agreement, the authors used Bayesian hierarchical modelling. Beta-distributed likelihoods with weakly informative priors were specified. Non-inferiority margins of 0.05, 0.10, and 0.15 were tested, each using a 95% posterior probability threshold to determine non-inferiority.
Additional performance metrics reported for models included precision, recall, and false positive rate for at least one model (GPT-4o).
Inter-rater reliability
Cohen's Kappa for agreement among medical professionals (medic–medic) was reported as 0.752 with a 95% confidence interval of 0.710 to 0.794. Cohen's Kappa for agreement between LLMs and medical professionals (medic–LLM) was 0.391 with a 95% confidence interval of 0.362 to 0.420. These point estimates indicate substantially higher inter-rater reliability among human annotators compared with model-to-clinician agreement.
Bayesian comparison and non-inferiority testing
Posterior estimates from the Bayesian hierarchical model showed medic–medic agreement had a posterior mean of 0.813 (95% credible interval 0.755–0.860), while medic–LLM agreement had a posterior mean of 0.659 (95% credible interval 0.633–0.684). The posterior difference (medic–medic minus medic–LLM) was 0.154 with a 95% credible interval of 0.091 to 0.209.
Non-inferiority testing rejected medic–LLM non-inferiority at all three prespecified margins (delta = 0.05, 0.10, 0.15) when using a 95% posterior probability threshold, indicating the models did not meet the criterion for being within these margins of clinician-level agreement.
Variation by clinical category
Agreement between LLMs and clinicians varied by clinical category. The smallest differences between medic–medic and medic–LLM agreement were observed for test results, while the largest differences occurred for diagnoses.
Model-level metrics
At least one model-level performance summary is reported: GPT-4o achieved 97.0% precision and 84.9% recall, with a 7.5% false positive rate. The source reports aggregate LLM performance comparisons but does not provide a full breakdown for every model in the abstract.
In this evaluation using ENT EHR notes annotated with SNOMED-CT, current LLMs did not reach inter-rater reliability levels comparable to medical professionals for clinical information extraction. Bayesian non-inferiority testing indicated medic–LLM agreement was meaningfully lower than medic–medic agreement under the tested margins.
The authors interpret these results as evidence that LLMs are better suited for initial extraction tasks that are followed by human verification rather than for fully autonomous operation in clinical documentation workflows. The findings are positioned to guide cautious deployment of LLM-assisted extraction in healthcare settings, emphasising the need for human oversight.
The study used openly available ENT clinical notes from MTSamples. The authors provided open-source code and resources to reproduce the evaluation framework. Source data are available at the MTSamples ENT page (as reported), and code is available in the linked GitHub repository (repository details are provided in the source). Exact URLs and repository names are reported in the article's Data/Code statements.
The authors declared that only openly available human data were used, and that appropriate ethical considerations were followed. They stated that all necessary patient/participant consent processes and oversight declarations relevant to use of publicly available data were completed as described. The authors declared no competing interests.
Funding support was acknowledged from the NIHR Biomedical Research Centre at University College London Hospitals, Hearing Health Theme (IS-BRC-1215-20016) as reported. The preprint is made available under a CC-BY 4.0 International license, and the medRxiv posting date reported in the source is August 24, 2026.