Early diagnosis and timely intervention are critical in Multiple Sclerosis (MS), yet diagnostic delays remain common. The authors evaluated whether a structured decision framework that combines a decision tree based on the 2017 McDonald criteria with reasoning by a large language model (LLM, specifically GPT-4) can derive MS diagnosis status from the first neurology clinic note. The aim was to extract diagnostic signals from clinical documentation while preserving a transparent node-by-node reasoning trace.
The analysis used first neurology notes from 125 randomly selected patients enrolled in a clinic cohort between 2017 and 2023. The cohort included patients with MS, related disorders, and controls representative of the clinic population. Reported demographics for the study group were mean age 40 ± 13 years and 81% women. For input to the algorithm, authors included the clinical history and diagnostic testing sections of the notes and redacted the assessment and plan sections to avoid exposing final clinician conclusions.
Investigators converted the 2017 McDonald criteria into a computable decision tree. Each decision node represented a discrete diagnostic question derived from the criteria. Expert-curated clinical knowledge was provided at nodes to guide the LLM’s reasoning. The tree produced terminal nodes corresponding to classification outcomes for MS diagnosis status, allowing transparent traversal from input findings to final label.
At each decision node, GPT-4 was prompted to generate a binary (yes/no) decision based on the note content and the provided expert guidance. The model’s binary outputs determined the path through the decision tree until a terminal node produced a final MS classification. This node-level design was intended to structure the model’s reasoning and make intermediate decisions auditable.
Model outputs were evaluated against neurologist-assessed diagnoses (the reference standard used in the study). In this cohort, the LLM-guided decision-tree achieved 84% accuracy, 79% precision, 74% recall (sensitivity), and 91% specificity for predicting MS diagnosis using first neurology notes. The authors report these aggregate performance metrics for the 125-note sample drawn from the clinic population.
The investigators systematically evaluated nonfactual or erroneous model behaviors, which they categorized as hallucinations. Hallucinations were observed in 32 cases (26% of notes). Among these hallucination instances, the most commonly identified types were incoherence (75% of hallucination cases) and overreliance (47% of hallucination cases). Other hallucination categories described in the study included incongruent, irrelevant, and logical reasoning errors, though detailed counts for those subtypes beyond the two most common were not reported in the abstract.
The authors note that the work is presented as a preprint and has not undergone peer review; they caution that results should not be used to guide clinical practice without further validation. The study used a modest, clinic-based sample of 125 notes; the authors recommend larger-scale studies to validate the approach and to mitigate hallucinations before routine application. Institutional review was reported: the University of Pittsburgh Institutional Review Board approved the study (STUDY19080007), and all participants provided written informed consent for access to records. Funding sources declared include National Institutes of Health support (R01NS098023, UL1TR001857) and the University of Pittsburgh Clinical Scientist Training Program. The authors declared no competing interests.
A structured decision-tree approach that integrates the LLM GPT-4 to perform node-level reasoning can flag MS diagnoses from early clinical documentation with promising performance metrics in this sample. However, hallucinations occurred in a meaningful fraction of cases, and the authors emphasize the need for larger studies to validate findings, refine safeguards against nonfactual outputs, and assess practical implementation in clinical workflows. Details on deployment, user interfaces, and real-world workflow integration were not reported in the source and therefore are not summarized here.