Patient-reported and clinically documented narratives often contain informal, fragmented and linguistically heterogeneous expressions that complicate medical concept normalization (MCN). Such variation frequently requires manual or semi-automated pre-processing before mapping to standard terminologies. Despite advances in biomedical language modeling, the relative performance of representation models for MCN and the value of integrating instruction-tuned large language models (LLMs) as upstream text-correction modules have been underexplored.
This study aimed to provide a systematic, comparative benchmark of transformer-based representation models and instruction-tuned LLMs for MCN on real-world short medical expressions, and to demonstrate a framework that integrates automatic text refinement with semantic retrieval and terminology mapping.
The evaluation used 12,713 instances drawn from five established real-world datasets: TAC2017_ADR, TwADR-L, TwiMed, CADEC and SMM4H2017. These datasets contain heterogeneous short medical expressions originating from patient-reported text and clinical documents, chosen to represent the noisy language and abbreviation use that challenge automated normalization workflows.
Two model families were benchmarked separately:
Transformer-based representation models (15 models total) spanning general-domain, biomedical and clinical pretraining. These models were evaluated using embedding-based semantic retrieval to return candidate standardized concepts.
Instruction-tuned LLMs (12 models) assessed as upstream text-correction modules. Their role was to refine raw input expressions before embedding and retrieval, with the aim of improving final normalization performance.
Performance was measured using retrieval accuracy metrics, with emphasis on Top-1 and Top-5 accuracy for mapping to standardized terminologies.
Among the representation models benchmarked, SapBERT achieved the highest Top-5 accuracy without any upstream correction. Specifically, SapBERT reached 63.8% Top-5 accuracy for SNOMED CT and 58.0% Top-5 accuracy for MedDRA. These results indicate that a purpose-built biomedical representation model provided the strongest baseline retrieval performance on the pooled real-world instances.
Instruction-tuned LLMs were evaluated for their utility as automatic text-correction modules that operate upstream of semantic retrieval. Models differed in both normalization impact and computational efficiency. The benchmark identified Qwen 2 Instruct as the preferred corrector because it provided a favorable balance between improved Top-1 normalization performance and computational cost compared with substantially larger instruction-tuned models (for example, Llama 3.1 Instruct with 70B parameters).
When Qwen 2 Instruct was incorporated to refine inputs prior to embedding-based retrieval, Top-5 accuracy rose to 69.4% for SNOMED CT and 63.2% for MedDRA, relative to the no-correction baseline. These gains demonstrate that upstream text refinement with an instruction-tuned LLM can materially improve downstream terminology mapping.
The authors present a scalable pipeline that accepts heterogeneous short medical expressions without manual pre-processing and automates three stages: text refinement using an instruction-tuned LLM, semantic retrieval using a representation model, and mapping to standardized vocabulary codes. The combined approach—using Qwen 2 Instruct for correction and SapBERT for embeddings—yielded the best reported Top-5 retrieval performance for the terminologies evaluated.
This framework is designed to be plug-and-play for terminology standardization tasks in which input text is noisy and diverse, enabling end-to-end automated normalization from raw expression to vocabulary code.
The benchmark provides guidance for selecting model combinations in applied MCN workflows. Key implications include:
Purpose-built biomedical representation models (exemplified by SapBERT) remain highly effective for embedding-based retrieval on noisy, real-world expressions.
Instruction-tuned LLMs can serve as effective automatic correctors that improve downstream normalization metrics; however, trade-offs between performance and computational cost should inform model choice.
A modular pipeline—automatic refinement, semantic retrieval, and mapping—can reduce reliance on manual pre-processing and scale to heterogeneous sources of clinical and patient-generated text.
This article is a preprint and has not been peer reviewed. The abstract reports the evaluation datasets, the number of instances (12,713), the models compared (15 representation models and 12 instruction-tuned LLMs), and specific performance outcomes for SapBERT and Qwen 2 Instruct. Additional experimental details, full model lists, exact Top-1 results, computational-resource measurements, and other per-model metrics were not reported in the abstract and should be consulted in the full preprint and supplementary material for reproducibility and implementation guidance.
Competing interests were declared as none by the authors. The preprint includes links to supplementary material and data/code repositories; users should refer to those sources for implementation details and to verify the full experimental setup.