Adverse event (AE) coding using the Medical Dictionary for Regulatory Activities (MedDRA) is critical for safety monitoring in oncology, particularly in acute myeloid leukemia (AML) where intensive therapies cause frequent and heterogeneous toxicities. Manual assignment of Low-Level Terms (LLTs) is labor-intensive, subjective, and difficult to scale. Although large language models (LLMs) show promise as decision-support tools for automated coding, unguided zero-shot generation has been found insufficient for reliable, fine-grained MedDRA coding.
The authors aimed to develop and evaluate a retrieval-augmented reasoning pipeline to produce clinically aligned LLT-level MedDRA coding from free-text AE descriptions collected in prospective AML clinical trials. The work focused on producing constrained candidate sets for the model and combining retrieval with structured LLM reasoning and deterministic hierarchical mapping.
The pipeline implemented a retrieval-augmented paradigm inspired by retrieval-augmented generation (RAG). Key components were:
Dense semantic retrieval to generate a constrained top-100 candidate set of LLTs for each free-text AE.
Structured LLM reasoning to select a single best-matching LLT from the candidate set.
Deterministic mapping of the selected LLT to the corresponding Preferred Term (PT) and System Organ Class (SOC) levels.
LLaMA-3.3-70B-Instruct was used as the primary backbone model, and the framework was benchmarked across multiple open instruction-tuned LLMs to evaluate the effect of backbone choice. The pipeline was evaluated retrospectively on AE datasets from three prospective AML clinical trials: MOSAIC, DELTA, and DaunoDouble. Evaluation metrics included automated string-agreement measures at LLT, PT, and SOC levels, plus expert-assessed Clinical Correctness Rate (CCR) to assess clinical acceptability.
The authors compared retrieval-augmented reasoning against zero-shot generation and random candidate selection baselines. They also contrasted dense semantic retrieval with lexical similarity–based retrieval in terms of including coder-assigned LLTs among candidates.
Automated evaluation across the three AML datasets produced the following ranges of accuracy:
LLT exact-string accuracy: 50–58%.
PT accuracy: 78–85%.
SOC accuracy: 90–93%.
Under expert clinical review, the retrieval-augmented reasoning pipeline demonstrated high clinical acceptability with Clinical Correctness Rates (CCR) between 91% and 97% across datasets.
Zero-shot generation and random candidate selection performed substantially worse than the retrieval-augmented approach under the reported automated and clinical evaluations. Dense semantic retrieval more frequently included the coder-assigned LLT among the candidate set compared with lexical similarity retrieval.
Benchmarking multiple backbone models indicated that the choice of LLM primarily affected LLT-level exact agreement, whereas PT and SOC performance remained comparatively stable across models.
The study shows that retrieval-augmented reasoning can support clinically aligned MedDRA coding of free-text AEs in AML clinical trials when realistic candidate constraints are applied. The authors highlight that strict LLT-level string agreement underestimates clinical appropriateness; hierarchical automated metrics (LLT/PT/SOC) should be paired with clinical expert validation to assess AI-assisted coding.
Key implications include:
Constraining candidate LLTs via dense semantic retrieval improves downstream selection and coding compared with unconstrained zero-shot approaches.
Deterministic hierarchical mapping from LLT to PT and SOC yields high-level coding stability even when exact LLT string agreement is imperfect.
Multi-model benchmarking is useful because model choice influences fine-grained LLT agreement more than higher-level PT/SOC metrics.
For implementation in hematology trials, combining automated retrieval-augmented pipelines with expert review can offer scalable and clinically acceptable MedDRA coding.
The study was conducted in accordance with the Declaration of Helsinki and approved by the Ethics Committee of TU Dresden (INSPRIME; BO-EK-400092023). The article lists competing interests for specific authors: one author declared consulting services and industry relationships, and another disclosed shareholdings and research grants; the remaining authors declared no competing interests. The underlying patient-level clinical trial data are not publicly available due to institutional, ethical, and patient privacy restrictions. Aggregated results and methodological details are included in the article and Supplementary Information; additional data may be available from the corresponding author upon reasonable request and subject to institutional and regulatory approval.