Large Language Models for Detecting Colorectal Polyps in Endoscopic Images

Message Commercial, regulatory-approved computer-aided detection (CADe) tools are widely used in colonoscopy; however, open-source multimodal large language models may present a promising alternative. This study compared the polyp detection performance of two such models - GPT-4o and Gemini 1.5 Pro - with ENDO-AID CADe, a commercial artificial intelligence (AI) system, using colonoscopy videos.

Per-lesion sensitivities were 75% for GPT-4o, 50% for Gemini and 87% for ENDO-AID. While the large language models were less effective at detecting polyps ≤5 mm, they performed comparably for larger lesions, suggesting their potential as alternative AI tools in endoscopic practice.

In more detail Commercial CADe systems have shown up to a 24% increase in adenoma detection rates by providing real-time visual assistance during colonoscopy. 1 However, widespread implementation is limited by high costs and the labour-intensive process of image annotation required for system development.

Recently, multimodal large language models (MLLMs) capable of processing both text...