---
title: "Using Open-Source LLMs to Extract Natural History Study Data for Rare Disease Research"
id: "medrxiv-21-llm-enabled-natural-history-study-analysis-to-support-rare-disease-research"
canonical_url: "https://medichelpline.com/clinical-feed/medrxiv-21-llm-enabled-natural-history-study-analysis-to-support-rare-disease-research"
content_type: "clinical_feed_article"
specialty: "General"
source_name: "medRxiv (Clinical Preprints)"
source_url: "https://www.medrxiv.org/content/10.64898/2026.09.17.26363303v1?rss=1"
published_at: "2026-09-20T12:00:00.000Z"
evidence_level: "Verified Feed"
license: "CC-BY-NC-4.0 / Informational Use"
---
# Using Open-Source LLMs to Extract Natural History Study Data for Rare Disease Research
## Provenance & Clinical Metadata
- **Canonical URL:** https://medichelpline.com/clinical-feed/medrxiv-21-llm-enabled-natural-history-study-analysis-to-support-rare-disease-research
- **Specialty:** [General](https://medichelpline.com/clinical-feed/general.md)
- **Primary Source:** medRxiv (Clinical Preprints)
- **Source URL:** [Original Journal Publication](https://www.medrxiv.org/content/10.64898/2026.09.17.26363303v1?rss=1)
- **Published At:** 2026-09-20T12:00:00.000Z
- **Evidence Rating:** Verified Feed
## Executive GIST (TL;DR)
- Rare diseases affect an estimated 300 million people worldwide, but evidence from Natural History Studies (NHS) is dispersed across unstructured literature and difficult to extract at scale. - The authors developed a proof-of-concept information extraction pipeline using three open-source **large language models (LLMs)**: Athena-v3-AWQ, Google's Gemma3-27B, and Meta's Llama-3.1-70B-Instruct. - Input data were PubMed abstracts curated from a Chan Zuckerberg Initiative disease research state model corpus comprising 302 gold-standard abstracts and 8,338 full-corpus abstracts. - Models were compared on processing efficiency, extraction completeness (automated Token F1 reported), and expert-rated accuracy. - All three models processed abstracts successfully >99% of the time. - Gemma produced the highest expert-rated accuracy: 68.0% of outputs rated “good,” compared with 36.0% for Llama and 10.0% for Athena. - Llama scored higher on the automated Token F1 metric (0.874) than Gemma (0.723), indicating divergence between automated metrics and human judgment. - Athena underperformed primarily because it tended to copy text verbatim rather than synthesize extracted content. - Gemma also had the fastest runtime on the full corpus (~16 minutes for 3,547 abstracts reported). - The study demonstrates that locally deployed open-source LLMs can extract structured NHS characteristics at scale and could accelerate evidence synthesis in rare disease research. - Data and code are publicly available in a GitHub repository for the NHS information extraction pipeline.
## Clinical Analysis & Structured Key Points
LLM-enabled Natural History Study Analysis to Support Rare Disease Research | medRxiv Skip to main content LLM-enabled Natural History Study Analysis to Support Rare Disease Research View ORCID Profile Kevin Li , View ORCID Profile Eric Sid , View ORCID Profile Qian Zhu doi: https://doi.org/10.64898/2026.09.17.26363303 Kevin Li National Center for Advancing Translational Sciences Find this author on Google Scholar Find this author on PubMed Search for this author on this site ORCID record for Kevin Li Eric Sid National Center for Advancing Translational Sciences Find this author on Google Scholar Find this author on PubMed Search for this author on this site ORCID record for Eric Sid Qian Zhu National Center for Advancing Translational Sciences Find this author on Google Scholar Find this author on PubMed Search for this author on this site ORCID record for Qian Zhu For correspondence: qian.zhu{at}nih.gov Abstract Info/History Metrics Data/Code Preview PDF Abstract Background: Rare diseases affect an estimated 300 million people worldwide, yet the research needed to guide diagnosis and treatment is often fragmented across multiple unstructured literature sources. Natural history studies (NHS) are a key source of this evidence, but manually extracting structured information from NHS publications can be tedious and does not scale. Methods: We have developed a proof-of-concept for an information extraction pipeline testing three opensource large-language models (LLMs) -- Athena-v3-AWQ, Google's Gemma3-27B, and Meta's Llama-3.1-70B-Instruct, to extract key NHS characteristics from PubMed abstracts curated from a Chan Zuckerberg Initiative disease research state model corpus (302 gold-standard and 8,338 full-corpus abstracts), and compared the models on efficiency, extraction completeness, and expert-rated accuracy. Results: All three models processed abstracts with success rates exceeding 99%. However, Gemma achieved the best overall performance, with the highest expert-rated accuracy (68.0% of outputs rated "good" vs. 36.0% for Llama and 10.0% for Athena) and the fastest runtime on the full corpus (~16 minutes for 3,547 abstracts), despite Llama scoring higher on the automated Token F1 metric (0.874 vs. 0.723), highlighting a divergence between automated and human evaluation. Athena's lower performance was largely attributable to verbatim copying rather than synthesis of extracted content. Conclusions: These findings illustrate how locally deployed open-source LLMs can extract structured NHS characteristics at scale, thus supporting their use to accelerate evidence synthesis in rare disease research. Competing Interest Statement The authors have declared no competing interest. Author Declarations I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained. Yes The details of the IRB/oversight body that provided approval or exemption for the research described are given below: This study is based on publicly available abstracts labeled with help from the Chan Zuckerberg Initiative. Paper containing data found at: Chen K, Ao M, Moon S, Burns G, Zhu Q. Machine Learning-Based Identification of Natural History Studies in Rare Diseases: A Step toward Understanding Disease Development and Outcome. J Rare Dis (Berlin). 2025;4(1). doi:10.1007/s44162-025-00115-9 I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals. Yes I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance). Yes I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable. Yes Data Availability All data and code produced are available online at GitHub in https://github.com/ncats/RDAS/tree/nhs_extraction_dockerization/NHS_info_extraction. https://github.com/ncats/RDAS/tree/nhs_extraction_dockerization/NHS_info_extraction Funder Information Declared National Institutes of Health , ZIA TR000548 Copyright The copyright holder for this preprint is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. This article is a US Government work. It is not subject to copyright under 17 USC 105 and is also made available for use under a CC0 license. Back to top Previous Next Posted September 20, 2026. Download PDF Data/Code Email Thank you for your interest in spreading the word about medRxiv. NOTE: Your email address is requested solely to identify you as the sender of this article. Your Email * Your Name * Send To * Enter multiple addresses on separate lines or separate them with commas. You are going to email the following LLM-enabled Natural History Study Analysis to Support Rare Disease Research Message Subject (Your Name) has forwarded a page to you from medRxiv Message Body (Your Name) thought you would like to see this page from the medRxiv website. Your Personal Message CAPTCHA This question is for testing whether or not you are a human visitor and to prevent automated spam submissions. Share LLM-enabled Natural History Study Analysis to Support Rare Disease Research Kevin Li , Eric Sid , Qian Zhu medRxiv 2026.09.17.26363303; doi: https://doi.org/10.64898/2026.09.17.26363303 Share This Article: Copy Citation Tools LLM-enabled Natural History Study Analysis to Support Rare Disease Research Kevin Li , Eric Sid , Qian Zhu medRxiv 2026.09.17.26363303; doi: https://doi.org/10.64898/2026.09.17.26363303 Citation Manager Formats BibTeX Bookends EasyBib EndNote (tagged) EndNote 8 (xml) Medlars Mendeley Papers RefWorks Tagged Ref Manager RIS Zotero Tweet Widget Facebook Like Google Plus One Subject Area Health Informatics Subject Areas All Articles Addiction Medicine (619) Allergy and Immunology (903) Anesthesia (334) Cardiovascular Medicine (4828) Dentistry and Oral Medicine (479) Dermatology (419) Emergency Medicine (651) Endocrinology (including Diabetes Mellitus and Metabolic Disease) (1633) Epidemiology (15974) Forensic Medicine (33) Gastroenterology (1206) Genetic and Genomic Medicine (7065) Geriatric Medicine (734) Health Economics (1066) Health Informatics (5060) Health Policy (1437) Health Systems and Quality Improvement (1775) Hematology (587) HIV/AIDS (1347) Infectious Diseases (except HIV/AIDS) (16307) Intensive Care and Critical Care Medicine (1174) Medical Education (671) Medical Ethics (155) Nephrology (727) Neurology (7291) Nursing (368) Nutrition (1084) Obstetrics and Gynecology (1243) Occupational and Environmental Health (1005) Oncology (3614) Ophthalmology (1055) Orthopedics (398) Otolaryngology (457) Pain Medicine (472) Palliative Medicine (139) Pathology (709) Pediatrics (1805) Pharmacology and Therapeutics (739) Primary Care Research (768) Psychiatry and Clinical Psychology (5919) Public and Global Health (9770) Radiology and Imaging (2432) Rehabilitation Medicine and Physical Therapy (1456) Respiratory Medicine (1250) Rheumatology (642) Sexual and Reproductive Health (773) Sports Medicine (579) Surgery (782) Toxicology (107) Transplantation (305) Urology (293)
## Related Clinical Research

- [Barriers to Primary Healthcare for the Rural Bangladeshi Elderly: A Social Ecological Model Review](https://medichelpline.com/clinical-feed/plos-one-19-using-the-social-ecological-model-to-identify-barriers-to-accessing-and-using.md)
- [Meditation Practice and Long-Term Mortality in US Adults: Pooled NHIS Cohorts Analysis](https://medichelpline.com/clinical-feed/medrxiv-6-meditation-practice-and-long-term-mortality-in-us-adults-a-doubly-robust.md)
- [Cigarette Filters Reduce Appeal and Alter Harm Perceptions: Randomised Online Study in Great Brita](https://medichelpline.com/clinical-feed/medrxiv-20-examining-the-appeal-of-cigarette-filters-an-online-randomised-experiment-among.md)
- [Jehovah’s Witnesses permit blood-derived products but keep ban on whole-blood transfusions](https://medichelpline.com/clinical-feed/stat-news-0-jehovah-s-witnesses-allow-blood-derived-products-but-keep-ban-on-whole-blood.md)
- [8-hour time-restricted eating yields greater weight loss but not consistent blood pressure benefits](https://medichelpline.com/clinical-feed/medical-news-today-0-16-8-intermittent-fasting-best-for-weight-loss-but-not-for-blood-pressure.md)

## Navigation
- [← Back to General Feed](https://medichelpline.com/clinical-feed/general.md)
- [← All Clinical Specialties](https://medichelpline.com/clinical-feed.md)
## Medical & Regulatory Disclaimer

> [!CAUTION]
> MedicHelpline content is structured for research, educational, and professional discovery purposes. It does not constitute individual medical advice, clinical diagnosis, or treatment recommendations.
> Always verify dosing, contraindications, and regulatory alerts against official product labeling and primary regulatory sources before clinical decision-making.