Abstract

Histopathological image analysis plays a crucial role in diagnosing cancer by examining tissue specimens under a microscope. Traditional manual methods are limited by their inability to scale efficiently for large datasets. With the rise of digital pathology, automated image analysis has become essential for handling large volumes of tissue samples, enabling faster, more accurate cancer detection. Nuclei segmentation and tissue classification are fundamental tasks, but existing methods struggle with complex tissue structures, particularly overlapping or clumped nuclei. The primary challenge in histopathology image analysis is accurately segmenting individual nuclei, especially in cases where nuclei are clumped or overlapping. Existing segmentation techniques like thresholding or conventional deep learning models often fail to address these challenges, leading to poor segmentation quality. Consequently, this affects the accuracy of classification models and, ultimately, the reliability of cancer diagnosis. This study proposes a novel approach that combines deep learning-based segmentation with improved watershed algorithms to enhance the accuracy of nuclei detection and separation. The method begins with a convolutional neural network (CNN) model for initial blob detection, followed by an improved watershed segmentation to separate clumped nuclei. A refined deep learning model (U-Net or Mask R-CNN) is then employed to further improve the segmentation results. Morphological and statistical features are extracted from the segmented nuclei, which are subsequently used in a machine learning classifier (e.g., Random Forest, SVM) to classify tissue patches as tumour or non-tumour. The proposed method is evaluated on a dataset of annotated histopathology images. The proposed method outperformed existing techniques in both training and test phases. On the training set, it achieved an accuracy of 96.2%, precision of 94.7%, recall of 97.1%, and F-measure of 95.9%. On the test set, the accuracy was 92.4%, precision was 90.3%, recall was 94.1%, and F-measure was 92.2%. Compared to traditional methods (e.g., thresholding + SVM), the proposed method demonstrated superior performance, particularly in handling clumped nuclei and producing more reliable classifications.