Abstract

Colorectal cancer (CRC) ranks among the most prevalent malignancies globally, with early detection and removal of precancerous polyps during colonoscopy being the most effective preventive strategy. However, conventional manual detection is hindered by operator fatigue and subjective assessment, resulting in clinically significant polyp miss rates of up to 27%. Existing computer-aided detection (CAD) systems face a fundamental trade-off: detection-only systems offer real-time speed but lack pixel-level precision, while segmentation-focused systems provide accurate delineation but are computationally prohibitive for live procedures. This paper presents DetSEG, a novel hybrid coarse-to-fine deep learning framework that resolves this trade-off through a two-stage architecture. In the first stage, a YOLOv8 object detection model performs rapid full-frame polyp localization, generating bounding box coordinates. In the second stage, cropped regions of interest are passed to a U-Net segmentation network with a pretrained ResNet-34 encoder for pixel-precise boundary delineation. We further propose an enhanced detection mode integrating Slicing Aided Hyper Inference (SAHI) with a dedicated small-polyp YOLOv8 model and an upgraded U-Net with EfficientNet-B4 encoder for improved detection of diminutive and flat polyps. The Kvasir-SEG dataset is utilized for model training and evaluation, supplemented by cross-dataset validation on CVC-ClinicDB. A comparative study against Faster R-CNN, SSD, and RT-DETR validates the superiority of YOLOv8 for the localization task. Experimental results demonstrate that DetSEG achieves a detection mAP_50 of 0.952, segmentation Dice coefficient of 0.891, and mean IoU of 0.842, with end-to-end inference under 100ms per frame. Additionally, a web-based CAD application with cloud deployment on Hugging Face Spaces is developed, providing an accessible clinical support tool. The proposed system substantially enhances colonoscopy procedures by offering both real-time detection and precise segmentation in a unified, deployable pipeline.

Authors

S. Devu, H. Vishnu, K. Varsha, Athira Viju, K. Sabeena, Chinchu M. Pillai, S. Geetha
College of Engineering Chengannur, India

Keywords

Polyp Localization, Semantic Segmentation, YOLOv8, U-Net, SAHI