Much of the symptom burden experienced by patients with colorectal cancer (CRC) is recorded in unstructured clinical text such as discharge notes. Manual extraction of these symptoms is labor-intensive and not scalable across large electronic health record repositories. This study benchmarked automated approaches for symptom extraction from clinical notes to determine whether modern zero-shot LLMs outperform traditional rule-based and named entity recognition (NER) methods when applied to CRC discharge documentation.
The authors analyzed 2,704 discharge notes from CRC patients contained in the MIMIC-IV clinical notes dataset (version 2.2). A target list of 46 cancer-related symptoms was created by combining items from the Memorial Symptom Assessment Scale and the EORTC QLQ-CR29. These 46 symptom concepts constituted the extraction targets used across all evaluated methods.
Four primary approaches were compared:
Two hybrid variants were also tested, in which LLM outputs were post-processed using a rule-based negation filter intended to remove negated mentions.
A 200-note gold standard was created and adjudicated by two human raters. Inter-rater agreement was reported as pooled kappa = 0.71 and macro kappa = 0.49. Models were evaluated against this adjudicated set using standard classification metrics: Macro F1, Micro F1, precision, and recall. These metrics were used to compare performance across the symptom concepts and to quantify overall extraction accuracy.
Zero-shot LLMs substantially outperformed the rule-based and NER approaches on this task. Key reported results include:
Gemini 3.5 Flash: Macro F1 = 0.70; Micro F1 = 0.86; Macro Precision = 0.74. This model achieved the best overall performance across evaluated metrics.
Claude Haiku: Macro F1 = 0.63; Macro Recall = 0.71. Claude Haiku ranked second in the reported comparisons.
Rule-based matching: Macro F1 = 0.44. The dictionary approach performed markedly worse than the top-performing LLMs.
Pretrained clinical NER: Macro F1 = 0.38. The NER pipeline also lagged behind LLM methods.
These results indicate that zero-shot application of contemporary LLMs can extract a broad symptom set from unstructured oncology notes more accurately than conventional NLP pipelines that depend on handcrafted dictionaries or entity taggers.
The authors evaluated hybrid approaches that applied a post-hoc rule-based negation filter to the raw LLM outputs. Contrary to the intended effect, this post-processing step degraded LLM performance. Reported declines included:
The authors attribute the performance drop to rigid, fixed-window matching in the negation filter that overrode correct LLM predictions by misclassifying contexts where negation scope was syntactically complex. The finding highlights that applying naive, rigid negation correction to LLM outputs can be harmful unless the syntactic or semantic scope of negation is validated for the specific extraction context.
On the dataset and symptom list studied, zero-shot LLMs demonstrated substantially higher extraction accuracy than both rule-based and pretrained NER approaches. The authors conclude that zero-shot LLM extraction presents a scalable and accurate alternative to manual chart review and to traditional rule- or model-training–dependent NLP pipelines for oncology symptom surveillance. Notably, these benefits were observed without institution-specific rule development or supervised fine-tuning of models.
The study also advises caution: post-hoc, rule-based negation corrections should not be applied to LLM outputs without careful validation of negation scope, as such filters can reduce performance by overturning correct model inferences.
This work is reported as a preprint and has not undergone peer review; findings should not be used to guide clinical practice without validation. The gold standard comprised 200 adjudicated notes with reported inter-rater agreement (pooled kappa = 0.71, macro kappa = 0.49). The study used the de-identified MIMIC-IV notes database under an approved data use agreement; access to the underlying clinical notes requires credentialing and completion of PhysioNet training. The authors provided a code repository link in the source for reproducibility. Additional methodological details, error analyses, and per-symptom performance breakdowns were reported in the source but are not reproduced here beyond the aggregated metrics above.
These results support further evaluation of zero-shot LLM approaches for large-scale symptom surveillance in oncology registries and EHR systems, while emphasizing careful handling of linguistic phenomena such as negation when combining rule-based components with generative language model outputs.