---
title: "Expert Disagreement and Ground Truth in Pathology AI: Impact on Clinical Relevance"
id: "medrxiv-4-beyond-consensus-human-ground-truth-expert-disagreement-and-clinical-relevance"
canonical_url: "https://medichelpline.com/clinical-feed/medrxiv-4-beyond-consensus-human-ground-truth-expert-disagreement-and-clinical-relevance"
content_type: "clinical_feed_article"
specialty: "Pulmonology"
source_name: "medRxiv (Clinical Preprints)"
source_url: "https://www.medrxiv.org/content/10.64898/2026.11.18.26368395v1?rss=1"
published_at: "2026-11-20T12:00:00.000Z"
evidence_level: "Verified Feed"
license: "CC-BY-NC-4.0 / Informational Use"
---
# Expert Disagreement and Ground Truth in Pathology AI: Impact on Clinical Relevance
## Provenance & Clinical Metadata
- **Canonical URL:** https://medichelpline.com/clinical-feed/medrxiv-4-beyond-consensus-human-ground-truth-expert-disagreement-and-clinical-relevance
- **Specialty:** [Pulmonology](https://medichelpline.com/clinical-feed/pulmonology.md)
- **Primary Source:** medRxiv (Clinical Preprints)
- **Source URL:** [Original Journal Publication](https://www.medrxiv.org/content/10.64898/2026.11.18.26368395v1?rss=1)
- **Published At:** 2026-11-20T12:00:00.000Z
- **Evidence Rating:** Verified Feed
## Executive GIST (TL;DR)
- The study compared how expert-panel size, composition, and consensus formation affect reference standards across three independent pathology cohorts: cryobiopsy cases for **interstitial lung disease**, resected **lung adenocarcinoma**, and images of **pulmonary granulomatous lesions**. - Investigators evaluated 50 transbronchial lung cryobiopsy cases scored by 27 pathologists; 95 resected lung adenocarcinomas scored by 11 pathologists; and 596 granuloma images scored by 12 pathologists. - Larger panels yielded more stable collective judgments; the cohort-level classifications converged as panel size increased, producing reproducible consensus labels. - The study team observed that increased reproducibility did not translate into stronger associations with clinical outcomes (disease progression or survival) in the cryobiopsy and lung adenocarcinoma cohorts. - In the adenocarcinoma cohort, a distinct cluster of pathologists produced consensus classifications that differed from the all-rater consensus; examination of those cases revealed morphological patterns that may explain disagreement and carry clinical signal. - A modified Delphi process in the granuloma image cohort increased interrater agreement (Krippendorff’s α rising from 0.351 to 0.539), but disease–morphology associations changed only modestly after consensus refinement. - Authors noted that reproducibility (agreement) and clinical relevance (outcome association) are separate properties of reference standards; therefore, disagreement should not be presumed to be mere noise. - The data availability statement indicates underlying data can be requested from the corresponding author, subject to institutional and ethical restrictions. - The Ethics Committee of Nagasaki University approved the work (Approval No.17051510-4).
## Clinical Analysis & Structured Key Points
Beyond Consensus: Human Ground Truth, Expert Disagreement, and Clinical Relevance in Pathology AI | medRxiv Skip to main content Beyond Consensus: Human Ground Truth, Expert Disagreement, and Clinical Relevance in Pathology AI Makoto Kawamoto , View ORCID Profile Kris Lami , View ORCID Profile Frederico Gaia , View ORCID Profile Sosuke Ishijima , Shintaro Sato , View ORCID Profile Rosane Duarte Achcar , Amna R. Almutrafi , View ORCID Profile Richard Attanoos , Jaroslaw Augustyniak , View ORCID Profile Mary Beth Beasley , View ORCID Profile Sabina Berezowska , View ORCID Profile Luka Brcic , View ORCID Profile Alberto Cavazza , Wei-Chin Chang , Andrew Churg , Wendy A. Cooper , View ORCID Profile Alexandre Todorovic Fabro , Akira Hebisawa , Chutima Jiragawasan , Izidor Kern , Yuka Kitamura , Kristine E. Konopka , Shinobu Kunugi , View ORCID Profile Brandon T. Larsen , Tereza Losmanova , Mai Matsumura , View ORCID Profile Alberto Marchevsky , Jijgee Munkhdelger , View ORCID Profile Ellen Caroline Toledo do Nascimento , Koji Okudera , Noriah Binti Othman , Ryoko Saito-Koyama , Tomoe Sawazumi , Frank Schneider , View ORCID Profile Yuri Tachibana , Angela Takano , View ORCID Profile Fabio Tavora , Yukio Tsushima , Lee Fong Wan , Joanne L. Wright , Masayo Yoshimura , View ORCID Profile Wataru Uegami , View ORCID Profile Anil Parwani , View ORCID Profile Andrey Bychkov , View ORCID Profile Junya Fukuoka doi: https://doi.org/10.64898/2026.11.18.26368395 Makoto Kawamoto 1 Department of Pathology Informatics, Nagasaki University Graduate School of Biomedical Sciences, Nagasaki, Japan; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Kris Lami 1 Department of Pathology Informatics, Nagasaki University Graduate School of Biomedical Sciences, Nagasaki, Japan; Find this author on Google Scholar Find this author on PubMed Search for this author on this site ORCID record for Kris Lami Frederico Gaia 1 Department of Pathology Informatics, Nagasaki University Graduate School of Biomedical Sciences, Nagasaki, Japan; Find this author on Google Scholar Find this author on PubMed Search for this author on this site ORCID record for Frederico Gaia Sosuke Ishijima 1 Department of Pathology Informatics, Nagasaki University Graduate School of Biomedical Sciences, Nagasaki, Japan; Find this author on Google Scholar Find this author on PubMed Search for this author on this site ORCID record for Sosuke Ishijima Shintaro Sato 2 Department of Respiratory Medicine, Saitama Red Cross Hospital, Japan; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Rosane Duarte Achcar 3 Department of Pathology, Cleveland Clinic, Cleveland, Ohio, USA; Find this author on Google Scholar Find this author on PubMed Search for this author on this site ORCID record for Rosane Duarte Achcar Amna R. Almutrafi 4 Department of Anatomical Pathology, King Fahad Medical City, Riyadh, Saudi Arabia; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Richard Attanoos 5 Department of Cellular Pathology, University Hospital of Wales, Cardiff, UK; Find this author on Google Scholar Find this author on PubMed Search for this author on this site ORCID record for Richard Attanoos Jaroslaw Augustyniak 6 Laborteam Pathology, Goldach, Switzerland; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Mary Beth Beasley 7 Department of Pathology, Icahn School of Medicine at Mount Sinai, New York, USA; Find this author on Google Scholar Find this author on PubMed Search for this author on this site ORCID record for Mary Beth Beasley Sabina Berezowska 8 Institute of Pathology, Lausanne University Hospital and University of Lausanne, Lausanne, Switzerland; Find this author on Google Scholar Find this author on PubMed Search for this author on this site ORCID record for Sabina Berezowska Luka Brcic 9 Department of Pathology, Medical University of Vienna, Vienna, Austria; Find this author on Google Scholar Find this author on PubMed Search for this author on this site ORCID record for Luka Brcic Alberto Cavazza 10 Pathology Service, Azienda USL/IRCCS di Reggio Emilia, Reggio Emilia, Italy; Find this author on Google Scholar Find this author on PubMed Search for this author on this site ORCID record for Alberto Cavazza Wei-Chin Chang 11 Department of Pathology, Taipei Medical University Hospital, Taipei, Taiwan/ Department of Pathology, School of Medicine, Taipei Medical University, Taipei, Taiwan; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Andrew Churg 12 Department of Pathology, Botucatu Medical School, Sao Paulo State University (UNESP), Botucatu, Brazil; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Wendy A. Cooper 13 Tissue Pathology and Diagnostic Oncology, NSW Health Pathology, Royal Prince Alfred Hospital, Camperdown, NSW, Australia/ School of Medicine, Western Sydney University, Sydney, NSW, Australia/ Faculty of Health and Medicine, University of Sydney, Sydney; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Alexandre Todorovic Fabro 12 Department of Pathology, Botucatu Medical School, Sao Paulo State University (UNESP), Botucatu, Brazil; Find this author on Google Scholar Find this author on PubMed Search for this author on this site ORCID record for Alexandre Todorovic Fabro Akira Hebisawa 14 Division of Clinical Pathology, National Hospital Organization Tokyo National Hospital, Tokyo, Japan; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Chutima Jiragawasan 1 Department of Pathology Informatics, Nagasaki University Graduate School of Biomedical Sciences, Nagasaki, Japan; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Izidor Kern 15 Cytology and Pathology Laboratory, University Clinic of Respiratory and Allergic Diseases, Golnik, Slovenia; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Yuka Kitamura 16 Department of Pathology Informatics, Nagasaki University Graduate School of Biomedical Sciences, Nagasaki, Japan/ N Lab Inc., Nagasaki, Japan; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Kristine E. Konopka 17 Department of Pathology, University of Michigan, Ann Arbor, Michigan, USA; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Shinobu Kunugi 18 Department of Analytic Human Pathology, Nippon Medical School, Tokyo, Japan; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Brandon T. Larsen 19 Department of Laboratory Medicine and Pathology, Mayo Clinic, Phoenix, Arizona, USA; Find this author on Google Scholar Find this author on PubMed Search for this author on this site ORCID record for Brandon T. Larsen Tereza Losmanova 20 Institute of Tissue Medicine and Pathology, University of Bern, Bern, Switzerland; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Mai Matsumura 21 Department of Pathology, Saitama Medical University, Saitama, Japan; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Alberto Marchevsky 22 Department of Pathology and Laboratory Medicine, Cedars-Sinai Medical Center, Los Angeles, California, USA; Find this author on Google Scholar Find this author on PubMed Search for this author on this site ORCID record for Alberto Marchevsky Jijgee Munkhdelger 23 Department of Pathology, Kameda Medical Center, Kamogawa, Japan; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Ellen Caroline Toledo do Nascimento 24 Department of Pathology, Hospital das Clinicas, University of Sao Paulo, Sao Paulo, Brazil; Find this author on Google Scholar Find this author on PubMed Search for this author on this site ORCID record for Ellen Caroline Toledo do Nascimento Koji Okudera 21 Department of Pathology, Saitama Medical University, Saitama, Japan; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Noriah Binti Othman 25 Pathology Department, Hospital Sultan Idris Shah Serdang, Selangor, Malaysia; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Ryoko Saito-Koyama 26 Department of Pathology, NHO Sendai Medical Center, Sendai, Japan; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Tomoe Sawazumi 27 Department of Pathology, Kanagawa Cardiovascular and Respiratory Center, Yokohama, Japan; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Frank Schneider 28 Department of Pathology and Laboratory Medicine, Emory University, Atlanta, Georgia, USA; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Yuri Tachibana 23 Department of Pathology, Kameda Medical Center, Kamogawa, Japan; Find this author on Google Scholar Find this author on PubMed Search for this author on this site ORCID record for Yuri Tachibana Angela Takano 29 Department of Anatomical Pathology, Singapore General Hospital and Duke-NUS Medical School, Singapore; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Fabio Tavora 30 Argos Laboratory, Federal University of Ceara, Fortaleza, Brazil; Find this author on Google Scholar Find this author on PubMed Search for this author on this site ORCID record for Fabio Tavora Yukio Tsushima 31 Department of Thoracic Surgery, Shinmatsudo Central General Hospital, Matsudo, Japan; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Lee Fong Wan 32 Sarawak General Hospital, Sarawak, Malaysia; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Joanne L. Wright 33 Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Masayo Yoshimura 34 Department of Pathology, Fukuoka University School of Medicine and Hospital, Fukuoka, Japan; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Wataru Uegami 23 Department of Pathology, Kameda Medical Center, Kamogawa, Japan; Find this author on Google Scholar Find this author on PubMed Search for this author on this site ORCID record for Wataru Uegami Anil Parwani 35 Department of Pathology, The Ohio State University, Columbus, Ohio, USA; Find this author on Google Scholar Find this author on PubMed Search for this author on this site ORCID record for Anil Parwani Andrey Bychkov 23 Department of Pathology, Kameda Medical Center, Kamogawa, Japan; Find this author on Google Scholar Find this author on PubMed Search for this author on this site ORCID record for Andrey Bychkov Junya Fukuoka 36 Nagasaki University / Kameda Medical Center Find this author on Google Scholar Find this author on PubMed Search for this author on this site ORCID record for Junya Fukuoka For correspondence: fukuokaj{at}nagasaki-u.ac.jp Abstract Info/History Metrics Data/Code Preview PDF Abstract Human consensus is often treated as ground truth in clinical artificial intelligence (AI), yet greater agreement does not necessarily ensure greater clinical relevance. We investigated how expert panel size, composition, and consensus formation influence the reproducibility and clinical relevance of reference standards across three independent pathology cohorts evaluated by international expert pathologists: 50 transbronchial lung cryobiopsy cases from patients with interstitial lung disease assessed by 27 pathologists, 95 resected lung adenocarcinomas evaluated by 11 pathologists, and 596 images of pulmonary granulomatous lesions evaluated by 12 pathologists. Larger panels produced more stable collective judgments but did not improve associations with disease progression or survival in the first two cohorts, instead converging toward consensus classifications with weaker outcome associations. Groups of pathologists identified solely by their diagnostic behavior generated consensus classifications with differing clinical relevance. In lung adenocarcinoma, examination of cases graded differently by the all-rater consensus and a distinct pathologist cluster revealed specific morphological patterns potentially contributing to these differences, suggesting that expert disagreement may contain clinically informative signals. In the granuloma cohort, a modified Delphi process increased agreement, with Krippendorff's α rising from 0.351 to 0.539, while disease-morphology associations changed only modestly. These findings demonstrate that reproducibility and clinical relevance are distinct properties of expert-derived reference standards. Expert disagreement should therefore not be regarded solely as noise to be eliminated, but also as a potential source of clinically meaningful information. Identifying and independently validating outcome-associated morphological features may enable refinement of existing consensus frameworks, supporting the development of more clinically relevant, task-specific ground truth for pathology AI. Competing Interest Statement Yuka Kitamura serves as the Chief Executive Officer (CEO) of N Lab Inc., Nagasaki, Japan. Author Declarations I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained. Yes The details of the IRB/oversight body that provided approval or exemption for the research described are given below: The Ethics Committee of Nagasaki University Graduate School of Biomedical Sciences gave ethical approval for this work (Approval No.17051510-4). I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals. Yes I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance). Yes I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable. Yes Data Availability The data supporting the findings of this study are available from the corresponding author upon reasonable request, subject to institutional and ethical restrictions concerning patient confidentiality. Copyright The copyright holder for this preprint is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY-NC-ND 4.0 International license . Back to top Previous Next Posted November 20, 2026. Download PDF Data/Code Email Thank you for your interest in spreading the word about medRxiv. NOTE: Your email address is requested solely to identify you as the sender of this article. Your Email * Your Name * Send To * Enter multiple addresses on separate lines or separate them with commas. You are going to email the following Beyond Consensus: Human Ground Truth, Expert Disagreement, and Clinical Relevance in Pathology AI Message Subject (Your Name) has forwarded a page to you from medRxiv Message Body (Your Name) thought you would like to see this page from the medRxiv website. Your Personal Message CAPTCHA This question is for testing whether or not you are a human visitor and to prevent automated spam submissions. Share Beyond Consensus: Human Ground Truth, Expert Disagreement, and Clinical Relevance in Pathology AI Makoto Kawamoto , Kris Lami , Frederico Gaia , Sosuke Ishijima , Shintaro Sato , Rosane Duarte Achcar , Amna R. Almutrafi , Richard Attanoos , Jaroslaw Augustyniak , Mary Beth Beasley , Sabina Berezowska , Luka Brcic , Alberto Cavazza , Wei-Chin Chang , Andrew Churg , Wendy A. Cooper , Alexandre Todorovic Fabro , Akira Hebisawa , Chutima Jiragawasan , Izidor Kern , Yuka Kitamura , Kristine E. Konopka , Shinobu Kunugi , Brandon T. Larsen , Tereza Losmanova , Mai Matsumura , Alberto Marchevsky , Jijgee Munkhdelger , Ellen Caroline Toledo do Nascimento , Koji Okudera , Noriah Binti Othman , Ryoko Saito-Koyama , Tomoe Sawazumi , Frank Schneider , Yuri Tachibana , Angela Takano , Fabio Tavora , Yukio Tsushima , Lee Fong Wan , Joanne L. Wright , Masayo Yoshimura , Wataru Uegami , Anil Parwani , Andrey Bychkov , Junya Fukuoka medRxiv 2026.11.18.26368395; doi: https://doi.org/10.64898/2026.11.18.26368395 Share This Article: Copy Citation Tools Beyond Consensus: Human Ground Truth, Expert Disagreement, and Clinical Relevance in Pathology AI Makoto Kawamoto , Kris Lami , Frederico Gaia , Sosuke Ishijima , Shintaro Sato , Rosane Duarte Achcar , Amna R. Almutrafi , Richard Attanoos , Jaroslaw Augustyniak , Mary Beth Beasley , Sabina Berezowska , Luka Brcic , Alberto Cavazza , Wei-Chin Chang , Andrew Churg , Wendy A. Cooper , Alexandre Todorovic Fabro , Akira Hebisawa , Chutima Jiragawasan , Izidor Kern , Yuka Kitamura , Kristine E. Konopka , Shinobu Kunugi , Brandon T. Larsen , Tereza Losmanova , Mai Matsumura , Alberto Marchevsky , Jijgee Munkhdelger , Ellen Caroline Toledo do Nascimento , Koji Okudera , Noriah Binti Othman , Ryoko Saito-Koyama , Tomoe Sawazumi , Frank Schneider , Yuri Tachibana , Angela Takano , Fabio Tavora , Yukio Tsushima , Lee Fong Wan , Joanne L. Wright , Masayo Yoshimura , Wataru Uegami , Anil Parwani , Andrey Bychkov , Junya Fukuoka medRxiv 2026.11.18.26368395; doi: https://doi.org/10.64898/2026.11.18.26368395 Citation Manager Formats BibTeX Bookends EasyBib EndNote (tagged) EndNote 8 (xml) Medlars Mendeley Papers RefWorks Tagged Ref Manager RIS Zotero Tweet Widget Facebook Like Google Plus One Subject Areas All Articles Addiction Medicine (642) Allergy and Immunology (928) Anesthesia (355) Cardiovascular Medicine (5060) Dentistry and Oral Medicine (496) Dermatology (433) Emergency Medicine (681) Endocrinology (including Diabetes Mellitus and Metabolic Disease) (1704) Epidemiology (16353) Forensic Medicine (37) Gastroenterology (1250) Genetic and Genomic Medicine (7334) Geriatric Medicine (761) Health Economics (1101) Health Informatics (5352) Health Policy (1480) Health Systems and Quality Improvement (1847) Hematology (604) HIV/AIDS (1389) Infectious Diseases (except HIV/AIDS) (16537) Intensive Care and Critical Care Medicine (1210) Medical Education (702) Medical Ethics (157) Nephrology (757) Neurology (7669) Nursing (380) Nutrition (1144) Obstetrics and Gynecology (1308) Occupational and Environmental Health (1046) Oncology (3769) Ophthalmology (1105) Orthopedics (414) Otolaryngology (474) Pain Medicine (508) Palliative Medicine (144) Pathology (727) Pediatrics (1877) Pharmacology and Therapeutics (759) Primary Care Research (793) Psychiatry and Clinical Psychology (6187) Public and Global Health (10051) Radiology and Imaging (2562) Rehabilitation Medicine and Physical Therapy (1540) Respiratory Medicine (1280) Rheumatology (666) Sexual and Reproductive Health (814) Sports Medicine (601) Surgery (809) Toxicology
## Related Clinical Research

- [Mavacamten inhibits diaphragm myosin in vitro, with greater potency on slow fibers](https://medichelpline.com/clinical-feed/biorxiv-0-beta-myosin-inhibition-in-skeletal-muscle-impact-of-mavacamten-on-the-diaphragm.md)
- [Dual C5aR1 and C5aR2 loss separates neutrophil influx from antibacterial function in lung infection](https://medichelpline.com/clinical-feed/biorxiv-10-dual-c5ar1-and-c5ar2-deficiency-uncouples-neutrophilic-inflammation-from.md)
- [Hyperbaric Oxygen Therapy Did Not Alter Endocannabinoid Markers or LPS After 10 Sessions](https://medichelpline.com/clinical-feed/pubmed-43059161.md) (DOI: 10.12659/MSM.953923)
- [Liability-threshold rare-variant test (LTBT) links HTRA3 to idiopathic pulmonary fibrosis](https://medichelpline.com/clinical-feed/medrxiv-9-a-new-liability-based-rare-variant-burden-test-identifies-a-novel-genetic.md)
- [Circadian Rhythms Modulate Biologic Targets in Asthma: Transcriptome Analysis](https://medichelpline.com/clinical-feed/medrxiv-1-timing-the-target-exploring-the-circadian-transcriptome-of-biologic-therapies.md)

## Navigation
- [← Back to Pulmonology Feed](https://medichelpline.com/clinical-feed/pulmonology.md)
- [← All Clinical Specialties](https://medichelpline.com/clinical-feed.md)
## Medical & Regulatory Disclaimer

> [!CAUTION]
> MedicHelpline content is structured for research, educational, and professional discovery purposes. It does not constitute individual medical advice, clinical diagnosis, or treatment recommendations.
> Always verify dosing, contraindications, and regulatory alerts against official product labeling and primary regulatory sources before clinical decision-making.