---
title: "Large language model to automate risk-of-bias (QUIPS) assessments in neurology prognosis reviews"
id: "medrxiv-5-a-large-language-model-for-risk-of-bias-assessment-in-systematic-reviews-of"
canonical_url: "https://medichelpline.com/clinical-feed/medrxiv-5-a-large-language-model-for-risk-of-bias-assessment-in-systematic-reviews-of"
content_type: "clinical_feed_article"
specialty: "Neurology"
source_name: "medRxiv (Clinical Preprints)"
source_url: "https://www.medrxiv.org/content/10.64898/2026.09.09.26362643v1?rss=1"
published_at: "2026-09-13T12:00:00.000Z"
evidence_level: "Verified Feed"
license: "CC-BY-NC-4.0 / Informational Use"
---
# Large language model to automate risk-of-bias (QUIPS) assessments in neurology prognosis reviews
## Provenance & Clinical Metadata
- **Canonical URL:** https://medichelpline.com/clinical-feed/medrxiv-5-a-large-language-model-for-risk-of-bias-assessment-in-systematic-reviews-of
- **Specialty:** [Neurology](https://medichelpline.com/clinical-feed/neurology.md)
- **Primary Source:** medRxiv (Clinical Preprints)
- **Source URL:** [Original Journal Publication](https://www.medrxiv.org/content/10.64898/2026.09.09.26362643v1?rss=1)
- **Published At:** 2026-09-13T12:00:00.000Z
- **Evidence Rating:** Verified Feed
## Executive GIST (TL;DR)
- This preprint reports development and testing of a prompt-engineered **large language model (LLM)** pipeline to automate **risk-of-bias (ROB)** assessment using the **QUIPS** (Quality in Prognosis Studies) tool for prognosis studies in **neurology**. - The authors implemented a zero-shot prompting approach to mimic a human reviewer and applied the pipeline to articles from previously published systematic reviews across three neurological domains: epilepsy, traumatic brain injury, and stroke. - The evaluation dataset comprised 298 articles from 15 systematic reviews. - Agreement was measured between the LLM and the original human ROB assessments, and between the original human raters themselves. - LLM-human agreement across overall risk scores produced a Cohen’s weighted kappa of 0.22 (95% CI, 0.12–0.33), which the authors describe as limited. - A small sample comparison (n = 5) suggested LLM-human agreement may not be inferior to human-human agreement; human-human Cohen’s weighted kappa reported as −0.25 (95% CI, −1.04–0.54), though this is based on few cases. - Wilcoxon signed-rank tests were statistically significant (p < 0.05) across four bias domains and for overall risk scores, and rank-biserial correlations indicated human raters tended to assign higher risk scores than the LLM. - The authors present feasibility evidence that a tailored, prompt-engineered **LLM** pipeline can automate QUIPS-based ROB assessment and provide a public GitHub repository with data, code, and the pipeline. - They conclude that with methodological refinements — including standardized QUIPS implementation and validation against expert ratings — automated ROB assessment may reduce time and cost of systematic reviews of prognosis studies in **neurology** and beyond. - Competing interests were declared as none; data and code are available at the provided GitHub links.
## Clinical Analysis & Structured Key Points
[](https://www.medrxiv.org/cdn-cgi/content?id=crYBhl6wou1Jr6xc1hMEhaghvr_J_Key1CxZq68TW2g-1789340316.7173784-1.2.1.1-fE_xbtJ5vhNmLE1RbmMIBWkBmZNsR2QPRzOyRCQbBYQjKCtT3vg0Z3Kz6tanNUhn) [Skip to main content](https://www.medrxiv.org/content/10.64898/2026.09.09.26362643v1?rss=1#main-content) [![medRxiv](https://www.medrxiv.org/sites/default/files/medrxiv_inner_logo.png)](https://www.medrxiv.org/) * [Home](https://www.medrxiv.org/) * [Submit](https://www.medrxiv.org/submit-a-manuscript) * [FAQ](https://www.medrxiv.org/about/FAQ) * [Blog](https://connect.medrxiv.org/news/) * [ALERTS / RSS](https://www.medrxiv.org/content/alertsrss) * [Resources](https://connect.medrxiv.org/resources/) * [About](https://www.medrxiv.org/content/about-medrxiv) Search for this keyword [Advanced Search](https://www.medrxiv.org/search) Follow this preprint # A large language model for risk-of-bias assessment in systematic reviews of prognosis studies in clinical neurology [ View ORCID Profile](http://orcid.org/0009-0004-2093-9389)Sem L Kampman, [ View ORCID Profile](http://orcid.org/0000-0003-3631-0003)Kees PJ Braun, [ View ORCID Profile](http://orcid.org/0000-0003-1511-6834)Willem M Otte doi: https://doi.org/10.64898/2026.09.09.26362643 [This article is a preprint and has not been certified by peer review [](https://www.medrxiv.org/about/FAQ#unrefereed)[what does this mean?](https://www.medrxiv.org/about/FAQ#unrefereed)]. It reports new medical research that has yet to be evaluated and so should _not_ be used to guide clinical practice. Sem L Kampman Department of Child Neurology, UMC Utrecht Brain Center, University Medical Center Utrecht and Utrecht University, Utrecht, The Netherlands, member of ERN EpiCARE * [Find this author on Google Scholar](https://www.medrxiv.org/lookup/google-scholar?link_type=googlescholar&gs_type=author&author%5B0%5D=Sem%2BL%2BKampman%2B "Open in new tab") * [Find this author on PubMed](https://www.medrxiv.org/lookup/external-ref?access_num=Kampman%20SL&link_type=AUTHORSEARCH "Open in new tab") * [Search for this author on this site](https://www.medrxiv.org/search/author1%3ASem%2BL%2BKampman%2B) * [ORCID record for Sem L Kampman](http://orcid.org/0009-0004-2093-9389 "Open in new tab") Kees PJ Braun Department of Child Neurology, UMC Utrecht Brain Center, University Medical Center Utrecht and Utrecht University, Utrecht, The Netherlands, member of ERN EpiCARE * [Find this author on Google Scholar](https://www.medrxiv.org/lookup/google-scholar?link_type=googlescholar&gs_type=author&author%5B0%5D=Kees%2BPJ%2BBraun%2B "Open in new tab") * [Find this author on PubMed](https://www.medrxiv.org/lookup/external-ref?access_num=Braun%20KP&link_type=AUTHORSEARCH "Open in new tab") * [Search for this author on this site](https://www.medrxiv.org/search/author1%3AKees%2BPJ%2BBraun%2B) * [ORCID record for Kees PJ Braun](http://orcid.org/0000-0003-3631-0003 "Open in new tab") Willem M Otte Department of Child Neurology, UMC Utrecht Brain Center, University Medical Center Utrecht and Utrecht University, Utrecht, The Netherlands, member of ERN EpiCARE * [Find this author on Google Scholar](https://www.medrxiv.org/lookup/google-scholar?link_type=googlescholar&gs_type=author&author%5B0%5D=Willem%2BM%2BOtte%2B "Open in new tab") * [Find this author on PubMed](https://www.medrxiv.org/lookup/external-ref?access_num=Otte%20WM&link_type=AUTHORSEARCH "Open in new tab") * [Search for this author on this site](https://www.medrxiv.org/search/author1%3AWillem%2BM%2BOtte%2B) * [ORCID record for Willem M Otte](http://orcid.org/0000-0003-1511-6834 "Open in new tab") * For correspondence: w.m.otte@umcutrecht.nl * [Abstract](https://www.medrxiv.org/content/10.64898/2026.09.09.26362643v1)[](https://www.medrxiv.org/panels_ajax_tab/biorxiv_tab_art/node:1299353/1) * [Info/History](https://www.medrxiv.org/content/10.64898/2026.09.09.26362643v1.article-info)[](https://www.medrxiv.org/panels_ajax_tab/biorxiv_tab_info/node:1299353/1) * [Metrics](https://www.medrxiv.org/content/10.64898/2026.09.09.26362643v1.article-metrics)[](https://www.medrxiv.org/panels_ajax_tab/article_tab_metrics/node:1299353/1) * [Supplementary material](https://www.medrxiv.org/content/10.64898/2026.09.09.26362643v1.supplementary-material)[](https://www.medrxiv.org/panels_ajax_tab/biorxiv_tab_data/node:1299353/1) * [Data/Code](https://www.medrxiv.org/content/10.64898/2026.09.09.26362643v1.external-links)[](https://www.medrxiv.org/panels_ajax_tab/article_tab_data_code/node:1299353/1) * [ Preview PDF](https://www.medrxiv.org/content/10.64898/2026.09.09.26362643v1.full.pdf+html)[](https://www.medrxiv.org/panels_ajax_tab/biorxiv_tab_pdf/node:1299353/1) ![Loading](https://www.medrxiv.org/sites/all/modules/contrib/panels_ajax_tab/images/loading.gif) ## Abstract Background: Risk-of-bias (ROB) assessments represent an integral component of systematic reviews. However, this task is often highly repetitive, time-consuming, and may lack inter-rater consistency. Large language models (LLMs) offer opportunities for automation in systematic reviews, which may expedite and enhance the quality and consistency of research synthesis. Methods: Using zero-shot prompting, we designed an LLM-based pipeline as a virtual mimic of a human reviewer for the Quality in Prognosis Studies (QUIPS) framework. Then, focusing on prognostic research in a single discipline (neurology), we applied this pipeline to articles included in previously published systematic reviews. We studied inter-rater agreement between both (1) the LLM and the original human ROB assessments and (2) between original human ROB assessments. Results: 298 articles from 15 reviews across three domains (epilepsy, traumatic brain injury, stroke) were included. We demonstrate the feasibility of a tailored, prompt-engineered LLM pipeline for automating ROB assessments with the QUIPS tool. While LLM-human agreement was limited (Cohen's weighted kappa; = 0.22, 95% CI, 0.12 - 0.33), our data tentatively suggest, based on a small sample (n=5), that it may not be inferior to human-human agreement (Cohen's weighted kappa; = -0.25, 95% CI, -1.04 - 0.54). Wilcoxon signed-rank tests were statistically significant (p ## Funder Information Declared SK, KB and WO are supported by the MING fonds. Copyright The copyright holder for this preprint is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. It is made available under a [CC-BY 4.0 International license](http://creativecommons.org/licenses/by/4.0/). bioRxiv and medRxiv thank the following for their generous financial support: > The Chan Zuckerberg Initiative, Cold Spring Harbor Laboratory, the Sergey Brin Family Foundation, California Institute of Technology, Centre National de la Recherche Scientifique, Fred Hutchinson Cancer Center, Imperial College London, Massachusetts Institute of Technology, Stanford University, The University of Edinburgh, University of Washington, and Vrije Universiteit Amsterdam. [Donate to openRxiv ](https://www.zeffy.com/en-US/donation-form/donate-to-make-a-difference-10981) [ Back to top](https://www.medrxiv.org/content/10.64898/2026.09.09.26362643v1?rss=1#page) [ Previous](https://www.medrxiv.org/content/10.64898/2026.09.09.26361541v1 "Resolving the Genomic Context of Clinically Relevant Antibiotic Resistance Genes in Wastewater with Ligation-Mediated PCR")[Next ](https://www.medrxiv.org/content/10.64898/2026.09.09.26362574v1 "Integrative Genetic and Single-Cell Analysis Reveals Macrophages as Key Mediators Linking Aging and Osteoporosis") Posted September 13, 2026. [ Download PDF](https://www.medrxiv.org/content/10.64898/2026.09.09.26362643v1.full.pdf) Print/Save Options [Download PDF](https://www.medrxiv.org/content/medrxiv/early/2026/09/13/2026.09.09.26362643.full.pdf)Full Text & In-line FiguresXML [More Info](https://www.medrxiv.org/about/FAQ#PrintOptions "More Information on Print/Save Options") [ Author Declarations](https://www.medrxiv.org/content/10.64898/2026.09.09.26362643v1#p-5) [Supplementary Material ](https://www.medrxiv.org/content/10.64898/2026.09.09.26362643v1.supplementary-material) [ Data/Code](https://www.medrxiv.org/content/early/2026/09/13/2026.09.09.26362643.external-links) [ Email](https://www.medrxiv.org/ "Email this Article") [ Share](https://www.medrxiv.org/) A large language model for risk-of-bias assessment in systematic reviews of prognosis studies in clinical neurology Sem L Kampman, Kees PJ Braun, Willem M Otte medRxiv 2026.09.09.26362643; doi: https://doi.org/10.64898/2026.09.09.26362643 [This article is a preprint and has not been certified by peer review [](https://www.medrxiv.org/about/FAQ#unrefereed)[what does this mean?](https://www.medrxiv.org/about/FAQ#unrefereed)]. It reports new medical research that has yet to be evaluated and so should _not_ be used to guide clinical practice. Share This Article: Copy [![Twitter logo](https://www.medrxiv.org/sites/all/modules/highwire/highwire/images/twitter.png)](https://www.medrxiv.org/highwire_log/share/twitter?link=http%3A%2F%2Ftwitter.com%2Fshare%3Furl%3Dhttps%253A%2F%2Fwww.medrxiv.org%2Fcontent%2F10.64898%2F2026.09.09.26362643v1%26text%3DA%2520large%2520language%2520model%2520for%2520risk-of-bias%2520assessment%2520in%2520systematic%2520reviews%2520of%2520prognosis%2520studies%2520in%2520clinical%2520neurology "Share this on Twitter") [![Facebook logo](https://www.medrxiv.org/sites/all/modules/highwire/highwire/images/fb-blue.png)](https://www.medrxiv.org/highwire_log/share/facebook?link=http%3A%2F%2Fwww.facebook.com%2Fsharer.php%3Fu%3Dhttps%253A%2F%2Fwww.medrxiv.org%2Fcontent%2F10.64898%2F2026.09.09.26362643v1%26t%3DA%2520large%2520language%2520model%2520for%2520risk-of-bias%2520assessment%2520in%2520systematic%2520reviews%2520of%2520prognosis%2520studies%2520in%2520clinical%2520neurology "Share on Facebook") [![LinkedIn logo](https://www.medrxiv.org/sites/all/modules/highwire/highwire/images/linkedin-32px.png)](https://www.medrxiv.org/highwire_log/share/linkedin?link=http%3A%2F%2Fwww.linkedin.com%2FshareArticle%3Fmini%3Dtrue%26url%3Dhttps%253A%2F%2Fwww.medrxiv.org%2Fcontent%2F10.64898%2F2026.09.09.26362643v1%26title%3DA%2520large%2520language%2520model%2520for%2520risk-of-bias%2520assessment%2520in%2520systematic%2520reviews%2520of%2520prognosis%2520studies%2520in%2520clinical%2520neurology%26summary%3D%26source%3DmedRxiv "Publish this post to LinkedIn") [![Mendeley logo](https://www.medrxiv.org/sites/all/modules/highwire/highwire/images/mendeley.png)](https://www.medrxiv.org/highwire_log/share/mendeley?link=http%3A%2F%2Fwww.mendeley.com%2Fimport%2F%3Furl%3Dhttps%253A%2F%2Fwww.medrxiv.org%2Fcontent%2F10.64898%2F2026.09.09.26362643v1%26title%3DA%2520large%2520language%2520model%2520for%2520risk-of-bias%2520assessment%2520in%2520systematic%2520reviews%2520of%2520prognosis%2520studies%2520in%2520clinical%2520neurology "Share on Mendeley") [ Citation Tools](https://www.medrxiv.org/ "Citation Tools") [ Get QR code](https://connect.medrxiv.org/qr/2026.09.09.26362643) * [Tweet Widget](http://twitter.com/share?url=https%3A//www.medrxiv.org/content/10.64898/2026.09.09.26362643v1&count=horizontal&via=&text=A%20large%20language%20model%20for%20risk-of-bias%20assessment%20in%20systematic%20reviews%20of%20prognosis%20studies%20in%20clinical%20neurology&counturl=www.medrxiv.org/content/10.64898/2026.09.09.26362643v1 "Tweet This") ## Subject Area * [Neurology ](https://www.medrxiv.org/collection/neurology) Reviews and Context 0 Comment 0 TRIP Peer Reviews 0 Community Reviews 0 Automated Services 0 Blogs/Media 0 Author Videos **Subject Areas** [**All Articles**](https://www.medrxiv.org/content/early/recent) * [Addiction Medicine](https://www.medrxiv.org/collection/addiction-medicine) (617) * [Allergy and Immunology](https://www.medrxiv.org/collection/allergy-and-immunology) (900) * [Anesthesia](https://www.medrxiv.org/collection/anesthesia) (330) * [Cardiovascular Medicine](https://www.medrxiv.org/collection/cardiovascular-medicine) (4808) * [Dentistry and Oral Medicine](https://www.medrxiv.org/collection/dentistry-and-oral-medicine) (478) * [Dermatology](https://www.medrxiv.org/collection/dermatology) (414) * [Emergency Medicine](https://www.medrxiv.org/collection/emergency-medicine) (648) * [Endocrinology (including Diabetes Mellitus and Metabolic Disease)](https://www.medrxiv.org/collection/endocrinology-including-diabetes-mellitus-and-metabolic-disease) (1626) * [Epidemiology](https://www.medrxiv.org/collection/epidemiology) (15929) * [Forensic Medicine](https://www.medrxiv.org/collection/forensic-medicine) (32) * [Gastroenterology](https://www.medrxiv.org/collection/gastroenterology) (1202) * [Genetic and Genomic Medicine](https://www.medrxiv.org/collection/genetic-and-genomic-medicine) (7039) * [Geriatric Medicine](https://www.medrxiv.org/collection/geriatric-medicine) (733) * [Health Economics](https://www.medrxiv.org/collection/health-economics) (1064) * [Health Informatics](https://www.medrxiv.org/collection/health-informatics) (5035) * [Health Policy](https://www.medrxiv.org/collection/health-policy) (1433) * [Health Systems and Quality Improvement](https://www.medrxiv.org/collection/health-systems-and-quality-improvement) (1760) * [Hematology](https://www.medrxiv.org/collection/hematology) (584) * [HIV/AIDS](https://www.medrxiv.org/collection/hiv-aids) (1344) * [Infectious Diseases (except HIV/AIDS)](https://www.medrxiv.org/collection/infectious_diseases) (16285) * [Intensive Care and Critical Care Medicine](https://www.medrxiv.org/collection/intensive-care-and-critical-care-medicine) (1171) * [Medical Education](https://www.medrxiv.org/collection/medical-education) (667) * [Medical Ethics](https://www.medrxiv.org/collection/medical-ethics) (153) * [Nephrology](https://www.medrxiv.org/collection/nephrology) (722) * [Neurology](https://www.medrxiv.org/collection/neurology) (7263) * [Nursing](https://www.medrxiv.org/collection/nursing) (366) * [Nutrition](https://www.medrxiv.org/collection/nutrition) (1080) * [Obstetrics and Gynecology](https://www.medrxiv.org/collection/obstetrics-and-gynecology) (1240) * [Occupational and Environmental Health](https://www.medrxiv.org/collection/occupational-and-environmental-health) (1005) * [Oncology](https://www.medrxiv.org/collection/oncology) (3603) * [Ophthalmology](https://www.medrxiv.org/collection/ophthalmology) (1049) * [Orthopedics](https://www.medrxiv.org/collection/orthopedics) (397) * [Otolaryngology](https://www.medrxiv.org/collection/otolaryngology) (453) * [Pain Medicine](https://www.medrxiv.org/collection/pain-medicine) (470) * [Palliative Medicine](https://www.medrxiv.org/collection/palliative-medicine) (139) * [Pathology](https://www.medrxiv.org/collection/pathology) (706) * [Pediatrics](https://www.medrxiv.org/collection/pediatrics) (1797) * [Pharmacology and Therapeutics](https://www.medrxiv.org/collection/pharmacology-and-therapeutics) (737) * [Primary Care Research](https://www.medrxiv.org/collection/primary-care-research) (766) * [Psychiatry and Clinical Psychology](https://www.medrxiv.org/collection/psychiatry-and-clinical-psychology) (5887) * [Public and Global Health](https://www.medrxiv.org/collection/public-and-global-health) (9744) * [Radiology and Imaging](https://www.medrxiv.org/collection/radiology-and-imaging) (2415) * [Rehabilitation Medicine and Physical Therapy](https://www.medrxiv.org/collection/rehabilitation-medicine-and-physical-therapy) (1449) * [Respiratory Medicine](https://www.medrxiv.org/collection/respiratory-medicine) (1247) * [Rheumatology](https://www.medrxiv.org/collection/rheumatology) (640) * [Sexual and Reproductive Health](https://www.medrxiv.org/collection/sexual-and-reproductive-health) (771) * [Sports Medicine](https://www.medrxiv.org/collection/sports-medicine) (578) * [Surgery](https://www.medrxiv.org/collection/surgery) (776) * [Toxicology](https://www.medrxiv.org/collection/toxicology) (107) * [Transplantation](https://www.medrxiv.org/collection/transplantation) (305) * [Urology](https://www.medrxiv.org/collection/urology) (291) Evaluation/discussion of this paper x Comments 0TRiP 0Community 0Automated 0Blogs/Media 0Video 0 Comments medRxiv aims to provide a venue for anyone to comment on a medRxiv preprint. Comments are moderated for offensive or irrelevant content (this can take ~24 h). Please avoid duplicate submissions. The content of a comment is not endorsed by medRxiv. medRxiv is now using a new commenting interface powered by Hypothes.is. All comments previously submitted via the legacy platform (Disqus) have been transferred to the new platform using the medRxiv_admin account. Please read our [commenting guide](https://connect.biorxiv.org/news/2022/03/21/commenting_on_preprints) for more information. Share this comments tab (click to copy link)Copied! TRiP medRxiv partners with journals and review services to enable posting of peer reviews and editorial decisions related to preprints they are evaluating. Reviews are posted with the consent of the authors. TRiP reviews are part of a peer review pilot project in which participating organizations post peer reviews of manuscripts they are evaluating _There is no TRiP material for this paper._ Community Reviews medRxiv aims to inform readers about online discussion of this preprint occurring elsewhere. The content at the links below is not endorsed by either medRxiv or the preprint's authors. Community reviews for this article: _There are no community reviews for this paper._ Automated Evaluations Certain services provide automated analysis of preprints. Analyses invited by the authors are displayed at the top of this tab. Those done independently of authors are shown underneath . None of these analyses is endorsed by medRxiv. Automated Evaluations: _There are no automated evaluations for this paper._ Blog/Media Links medRxiv aims to i
## Related Clinical Research

- [Outcome Measure Feasibility and Validity in STXBP1-Related Disorders: Bayley-4, PDMS-3, GMFM-66 Fi](https://medichelpline.com/clinical-feed/medrxiv-7-feasibility-and-validity-of-clinical-outcome-measures-in-a-prospective-natural.md)
- [Bupropion Hydrochloride for Improving Recovery in Acute Ischaemic Stroke: BASE Trial Protocol](https://medichelpline.com/clinical-feed/bmj-open-4-protocol-for-base-a-multicentre-double-blind-randomised-controlled-trial-of.md)
- [Serum vitamin D (25[OH]D) and Morse Fall Scale risk in hospitalized adults](https://medichelpline.com/clinical-feed/plos-one-18-serum-vitamin-d-levels-and-fall-risk-in-adults-a-cross-sectional-study-using.md)
- [Neurocysticercosis in California: Prevalence, Demographics, and Neuropsychiatric Comorbidities in](https://medichelpline.com/clinical-feed/medrxiv-0-neurocysticercosis-in-california-clinical-characteristics-and-associated.md)
- [Sabzevar PERSIAN Cohort Study: First Northeastern Iran NCD Profile](https://medichelpline.com/clinical-feed/bmj-open-14-first-population-based-study-on-non-communicable-diseases-and-risk-factors-in-northeastern-iran-sabzevar-cohort-profile.md)

## Navigation
- [← Back to Neurology Feed](https://medichelpline.com/clinical-feed/neurology.md)
- [← All Clinical Specialties](https://medichelpline.com/clinical-feed.md)
## Medical & Regulatory Disclaimer

> [!CAUTION]
> MedicHelpline content is structured for research, educational, and professional discovery purposes. It does not constitute individual medical advice, clinical diagnosis, or treatment recommendations.
> Always verify dosing, contraindications, and regulatory alerts against official product labeling and primary regulatory sources before clinical decision-making.