Acute coronary syndrome (ACS) requires rapid recognition and prompt treatment, but identifying instances of management delay in routine clinical care can be challenging. This study aimed to develop and evaluate a large language model (LLM)–based system to detect ACS management delays from clinical documentation and to characterize cases confirmed as delayed.
The analysis included admissions supervised by internal medicine residents at a single institution from July 2022 through June 2025 (n = 4,642). The authors used resident admission notes and initial cardiology consult notes from this cohort to develop and validate prompts for the LLM system.
Prompts were designed and validated to determine whether the resident admission note documented initiation of ACS management and whether the initial cardiology consult note documented initiation of ACS management. For validation, the authors used sets of notes with ground-truth labels: 161 resident admission notes and 161 initial cardiology consult notes.
The LLM was run on the dataset to flag cases with potential management delays. Performance reporting in the abstract focuses on the positive predictive value: the LLM identified management delays with a 52% positive predictive value (n = 35 confirmed delays out of 67 cases flagged by the model).
When the LLM output was discordant with ground-truth labels, cases were reviewed by three physicians. These reviewers applied a validated tool to confirm whether a management delay had occurred. This adjudication process established the reference for the model’s positive predictive value and enabled further characterization of confirmed delay cases.
The study compared demographics and key clinical findings between patients with confirmed management delays and those without. Reported differences include:
These differences indicate demographic subgroups that, within this cohort, experienced higher rates of confirmed management delay.
Confirmed management delay cases had substantially longer average times to receiving core ACS therapies and procedures:
These reported mean differences quantify the clinical impact of confirmed delays on receipt of antithrombotic therapy and definitive coronary evaluation.
The authors conclude that an LLM-based detection system can identify ACS management delays at scale. By flagging cases for review, such a system could support targeted individual clinician feedback and inform system-level quality improvement efforts aimed at reducing delays in ACS diagnosis and treatment.
The abstract reports core cohort size, validation set sizes, and main performance and comparative outcome metrics, but does not provide detailed methods for model training, model architecture, or sensitivity/specificity metrics beyond the reported positive predictive value. Additional methodological details and external validation results were not reported in the abstract.
The authors declared no known competing financial interests or personal relationships that might have influenced the reported work. Keywords provided by the authors include: acute coronary syndrome; diagnostic errors; large language models; management delay.