Rare diseases collectively affect an estimated 300 million people worldwide, yet research that informs diagnosis and treatment is often scattered across many unstructured publications. Natural history studies (NHS) are an important evidence source for disease progression and outcomes, but manually extracting structured data from NHS publications is time-consuming and does not scale. The authors developed a proof-of-concept information extraction pipeline to evaluate whether locally deployed open-source LLMs can extract key NHS characteristics from PubMed abstracts and thereby accelerate evidence synthesis for rare disease research.
The analysis used PubMed abstracts curated from a Chan Zuckerberg Initiative disease research state model corpus. The corpus included 302 gold-standard abstracts (used for evaluation and curation) and a larger full corpus of 8,338 abstracts. The study focused on extracting structured NHS characteristics from these abstracts using automated model-based extraction.
Three open-source large language models were integrated into the pipeline and compared: Athena-v3-AWQ, Google's Gemma3-27B, and Meta's Llama-3.1-70B-Instruct. The pipeline was implemented for local deployment of these models to process abstracts, extract target fields, and produce structured outputs suitable for downstream evidence synthesis. The code and implementation artifacts are available in a public GitHub repository.
Models were compared on several dimensions: processing efficiency (runtime), extraction completeness as measured by an automated Token F1 metric, and human expert–rated accuracy of extracted outputs. Expert raters assessed outputs and assigned quality labels; the study reports the proportion of outputs rated as “good.” Automated metrics and human ratings were compared to identify concordance or divergence between quantitative and qualitative assessments.
All three models processed abstracts with high success rates, exceeding 99% for the corpus. Reported runtimes included Gemma completing processing of a subset of the full corpus (3,547 abstracts) in approximately 16 minutes, indicating practical throughput for large-scale extraction tasks when models are locally deployed.
The automated Token F1 metric was used to quantify extraction completeness. According to this metric, Llama scored higher (Token F1 = 0.874) than Gemma (Token F1 = 0.723). These automated numbers reflect token-level overlap with reference annotations in the evaluation set; however, automated overlap does not necessarily equate to clinically useful or well-synthesized outputs.
Human expert review yielded a different ranking than the automated metric. Gemma produced the highest expert-rated accuracy: 68.0% of its outputs were rated “good.” Llama’s outputs were rated “good” 36.0% of the time, and Athena’s outputs were rated “good” 10.0% of the time. This divergence highlights that automated token-based metrics and human judgment of extraction quality may not align.
Model-specific behaviors were noted in the evaluation. Athena’s lower expert-rated performance was largely attributed to a tendency for verbatim copying from source abstracts rather than synthesizing and normalizing extracted content into structured fields. Llama achieved high automated overlap with reference tokens but lower expert-rated quality, suggesting that lexical overlap can occur without producing coherent or clinically usable outputs. Gemma combined comparatively strong runtime performance with the highest human-rated output quality in this study.
The proof-of-concept demonstrates that locally deployed open-source LLMs can extract structured NHS characteristics at scale from PubMed abstracts. Results indicate that model selection matters: automated metrics alone may not predict human-assessed usefulness, and models that synthesize information rather than copy text may produce outputs better rated by experts. These capabilities can support accelerated evidence synthesis in rare disease research by enabling retrieval of structured NHS data from unstructured literature sources.
The authors declare that all data and code produced for this project are available online at the project's GitHub repository: https://github.com/ncats/RDAS/tree/nhs_extraction_dockerization/NHS_info_extraction. The study used publicly available abstracts curated with assistance from the Chan Zuckerberg Initiative. The authors state that they followed relevant ethical guidelines and that any necessary IRB or oversight approvals were obtained or not required. The authors declared no competing interests. The preprint was posted on September 20, 2026, and the work is designated as a US Government work and available under a CC0 license.