Improving the Segmentation of Colorectal Cancer from Histopathological Images Using a Hybrid Deep Learning Pipeline: A Case Study

Fahima Idiri; Farid MEZIANE; Hakim BOUCHAL

doi:10.35882/jeeemi.v8i1.1158

Fahima Idiri University of Béjaia, Faculty of Exact Sciences, Laboratory of Medical Informatics and Intelligent and Dynamic Environments (LIMED), Béjaia 06000, Algeria https://orcid.org/0009-0000-3385-7914
Farid MEZIANE University of Derby, Data Science Research Center, United Kingdom https://orcid.org/0000-0001-9811-6914
Hakim BOUCHAL University of Béjaia, Faculty of Technology, Laboratory of Medical Informatics and Intelligent and Dynamic Environments (LIMED), Béjaia 06000, Algeria https://orcid.org/0000-0002-6421-9672

DOI: https://doi.org/10.35882/jeeemi.v8i1.1158

Keywords: Colorectal cancer (CRC), Deep Learning, DeepLabV3, intelligent decision‑support tools, Histopathological Hematoxylin and Eosin Images

Abstract

Early and precise diagnosis of colorectal cancer plays a crucial role in enhancing patients' outcomes. Although histopathological assessment remains the reference standard for diagnosis, it is often lengthy and subject to variability between pathologists. This study aims to develop and evaluate a hybrid deep learning-based approach for the automated segmentation of Hematoxylin and Eosin-stained colorectal histopathology images. The work investigates how preprocessing strategies and architectural design choices influence the model’s ability to identify meaningful tissue patterns while preserving computational efficiency. Furthermore, it demonstrates the integration of a deep learning-based segmentation module into colorectal cancer diagnostic workflows. Several deep learning–based segmentation models with varying architectural configurations were trained and evaluated using a publicly available endoscopic biopsy histopathological hematoxylin and eosin image dataset. Preprocessing procedures were applied to generate computationally efficient image representations, thereby improving training stability and overall segmentation performance. The best-performing configuration achieved a segmentation accuracy of 0.97, reflecting consistent and reliable performance across samples. It accurately delineated cancerous tissue boundaries and effectively distinguished benign from malignant regions, demonstrating sensitivity to fine morphological details relevant to diagnosis. Strong agreement between predicted and expert-annotated regions confirmed the model’s reliability and alignment with expert assessments. Minimal overfitting was observed, indicating stable training behavior and robust generalization across different colorectal tissue types. In comparative evaluations, the model maintained high accuracy across all cancer categories and outperformed existing state-of-the-art approaches. Overall, these findings demonstrate the model’s robustness, efficiency, and adaptability, confirming that careful architectural and preprocessing optimization can substantially enhance segmentation quality and diagnostic reliability. The proposed approach can support pathologists by providing accurate tissue segmentation, streamlining diagnostic procedures, and improving clinical decision-making. This study underscores the value of optimized deep learning models as intelligent decision-support tools for efficient and consistent colorectal cancer diagnosis

Downloads

Download data is not yet available.

References

J.-M. Bokhorst et al., “Deep learning for multi-class semantic segmentation enables colorectal cancer detection and classification in digital pathology images,” Sci. Rep., vol. 13, no. 1, Dec. 2023, doi: 10.1038/s41598-023-35491-z.

A. S. N. Raju et al., “Colorectal cancer detection with enhanced precision using a hybrid supervised and unsupervised learning approach,” Sci. Rep., vol. 15, no. 1, p. 3180, 2025, doi: 10.1038/s41598-025-86590-y.

B. Á. Pataki et al., “HunCRC: annotated pathological slides to enhance deep learning applications in colorectal cancer screening,” Sci. Data, vol. 9, no. 1, Dec. 2022, doi: 10.1038/s41597-022-01450-y.

S. Sengupta et al., “Assessment of different U-Net backbones in segmenting colorectal adenocarcinoma from H&E histopathology,” Pathol. Res. Pract., vol. 266, p. 155820, Feb. 2025, doi: 10.1016/j.prp.2025.155820.

R. Feng, X. Liu, J. Chen, D. Z. Chen, H. Gao, and J. Wu, “A deep learning approach for colonoscopy pathology WSI analysis: accurate segmentation and classification,” IEEE J. Biomed. Health Inform., vol. 25, no. 10, pp. 3700–3708, Oct. 2021, doi: 10.1109/JBHI.2020.3040269.

S. R. Alotaibi, M. A. Alohali, M. Maashi, H. Alqahtani, M. Alotaibi, and A. Mahmud, “Advances in colorectal cancer diagnosis using optimal deep feature fusion approach on biomedical images,” Sci. Rep., vol. 15, no. 1, Dec. 2025, doi: 10.1038/s41598-024-83466-5.

S. Qiu, H. Lu, J. Shu, T. Liang, and T. Zhou, “Colorectal cancer segmentation algorithm based on deep features from enhanced CT images,” Comput. Mater. Contin., vol. 80, no. 2, pp. 2495–2510, Aug. 2024, doi: 10.32604/CMC.2024.052476.

J. Cheng et al., “EBHI-Seg: a novel enteroscope biopsy histopathological hematoxylin and eosin image dataset for image segmentation tasks,” Front. Med., vol. 10, Jan. 2023, doi: 10.3389/fmed.2023.1114673.

L. Sun and V. S. Sheng, “DDViT: double-level fusion domain adapter vision transformer,” in Proc. AAAI Conf. Artif. Intell., Mar. 2024, pp. 23661–23663. doi: 10.1609/aaai.v38i21.30516.

H. Le Xuan, M. H. Le, V. V. Truong, H. P. Quang, and H. V. Huy, “MASK2TASKS: leveraging segmentation to enhance classification performance in histopathological colorectal images,” in Proc. 2nd Tiny Papers Track at ICLR, 2024.

T. Liu, H. Yang, J. Huang, and Y. Zhou, “A review of colorectal cancer histopathology image segmentation using deep learning Methods,” in Proc. 2024 6th Int. Conf. Control Robot. (ICCR), 2024, pp. 281–291. doi: 10.1109/ICCR64365.2024.10927570.

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2016, pp. 770–778. doi: 10.1109/CVPR.2016.90.

A. Howard et al., “Searching for MobileNetV3,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2019, pp. 1314–1324. doi: 10.1109/ICCV.2019.00140.

L. C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs,” in IEEE Trans. Pattern Anal. Mach. Intell., IEEE Computer Society, Apr. 2018, pp. 834–848. doi: 10.1109/TPAMI.2017.2699184.

M. Yeung, L. Rundo, Y. Nan, E. Sala, C. B. Schönlieb, and G. Yang, “Calibrating the dice loss to handle neural network overconfidence for biomedical image segmentation,” J. Digit. Imaging, vol. 36, no. 2, p. 739, Apr. 2022, doi: 10.1007/S10278-022-00735-3.

S. A. Aula and T. A. Rashid, “Foxtsage vs. Adam: revolution or evolution in optimization?,” Cogn. Syst. Res., vol. 92, p. 101373, Sep. 2025, doi: 10.1016/J.COGSYS.2025.101373.

M. Xiao et al., “Addressing overfitting problem in deep learning‐based solutions for next generation data‐driven networks,” Wirel. Commun. Mob. Comput., Aug. 2021, doi: 10.1155/2021/8493795.

K. Lee et al., “Deep learning of histopathology images at the single cell level,” Front. Artif. Intell., vol. 4, Sep. 2021, doi: 10.3389/frai.2021.754641.

C. Shorten and T. M. Khoshgoftaar, “A survey on image data augmentation for deep learning,” J. Big Data, vol. 6, Jul. 2019, doi: doi.org/10.1186/s40537-019-0197-0.

M. Abbas, M. Arslan, R. Abid Bhatty, F. Yousaf, A. Ahmad Khan, and A. Rafay, “Enhanced skin disease diagnosis through convolutional neural networks and data augmentation techniques,” J. Comput. Biomed. Inf., vol. 7, pp. 87–106, Jun. 2024, doi: 10.56979.

A. Mumuni and F. Mumuni, “Data augmentation: a comprehensive survey of moderne approaches,” Array, vol. 16, p. 100258, Dec. 2022, doi: 10.1016/j.array.2022.100258.

M. Xu, S. Yoon, A. Fuentes, and D. S. Park, “A comprehensive survey of image augmentation techniques for deep learning,” Pattern Recognit., vol. 137, p. 109347, May 2023, doi: 10.1016/j.patcog.2023.109347.

T. Kumar, R. Brennan, A. Mileo, and M. Bendechache, “Image data augmentation approaches: a comprehensive survey and future directions,” IEEE Access, vol. 12, pp. 187536–187571, 2024.

E. Goceri, “Medical image data augmentation: techniques, comparisons and interpretations,” Artif. Intell. Rev., vol. 56, no. 11, pp. 12561–12605, Mar. 2023, doi: 10.1007/s10462-023-10453-z.

P. Chlap, H. Min, N. Vandenberg, J. Dowling, L. Holloway, and A. Haworth, “A review of medical image data augmentation techniques for deep learning applications,” J. Med. Imaging Radiat. Oncol., vol. 65, no. 5, pp. 545–563, 2021, doi: 10.1111/1754-9485.13261.

R. J and P. A, “Analysis of lung cells with a novel segmentation methodology using FCN and Deeplab V3,” in Proc. 2024 3rd Int. Conf. Distributed Comput. Electr. Circuits Electron. (ICDCECE), 2024, pp. 1–8. doi: 10.1109/ICDCECE60827.2024.10548209.

S. Pak et al., “Application of deep learning for semantic segmentation in robotic prostatectomy: comparison of convolutional neural networks and visual transformers,” Investig. Clin. Urol., vol. 65, no. 6, pp. 551–558, Nov. 2024, doi: 10.4111/icu.20240159.

R. Archana and P. S. E. Jeevaraj, “Deep learning models for digital image processing: a review,” Artif Intell Rev, vol. 57, no. 1, p. 11, 2024, doi: 10.1007/s10462-023-10631-z.

T. Jo, “Convolutional neural networks,” in Deep Learning Foundations, Springer Int. Publ., 2023, pp. 303–326. doi: 10.1007/978-3-031-32879-4_12.

S. S. Kshatri and D. Singh, “Convolutional neural network in medical image analysis: a review,” Arch. Comput. Methods Eng., vol. 30, no. 4, pp. 2793–2810, 2023.

D. R. Sarvamangala and R. V Kulkarni, “Convolutional neural networks in medical image understanding: a survey,” Evol. Intell., vol. 15, no. 1, pp. 1–22, 2022.

G. Hille, P. Tummala, L. Spitz, and S. Saalfeld, “Transformers for colorectal cancer segmentation in CT imaging,” Int. J. Comput. Assist. Radiol. Surg., vol. 19, pp. 2079--2087, 2024, doi: 10.1007/s11548-024-03217-9.

H. Xiao, L. Li, Q. Liu, X. Zhu, and Q. Zhang, “Transformers in medical image segmentation: a review,” Biomed. Signal Process. Control, vol. 84, p. 104791, 2023.

Z. Yin, C. Yao, L. Zhang, and S. Qi, “Application of artificial intelligence in diagnosis and treatment of colorectal cancer: a novel Prospect,” Front. Med. (Lausanne), vol. 10, p. 1128084, 2023.

P. C. Neto et al., “An interpretable machine learning system for colorectal cancer diagnosis from pathology slides,” NPJ Precis. Oncol., vol. 8, no. 1, Dec. 2024, doi: 10.1038/s41698-024-00539-4.

M. Cooper, Z. Ji, and R. G. Krishnan, “Machine learning in computational histopathology: challenges and opportunities,” Sep. 01, 2023, John Wiley and Sons Inc. doi: 10.1002/gcc.23177.

A. Santhoshi and A. Muthukumaravel, “Texture and shape-based feature extraction for colorectal tumor segmentation,” in Proc. 2024 10th Int. Conf. Adv. Comput. Commun. Syst. (ICACCS), IEEE, 2024, pp. 315–320.

C. Ho et al., “A promising deep learning-assistive algorithm for histopathological screening of colorectal cancer,” Sci. Rep., vol. 12, no. 1, Dec. 2022, doi: 10.1038/s41598-022-06264-x.

H. M. Selby, Y. A. Son, V. R. Sheth, T. H. Wagner, E. L. Pollom, and A. M. Morris, “AI-ready rectal cancer MR imaging: a workflow for tumor detection and segmentation,” BMC Med. Imaging, vol. 25, no. 1, Dec. 2025, doi: 10.1186/s12880-025-01614-3.

M. F. Avram, D. C. Lazăr, M. I. Mariş, and S. Olariu, “Artificial intelligence in improving the outcome of surgical treatment in colorectal cancer,” Front. Oncol., vol. 13, Jan. 2023, doi: 10.3389/fonc.2023.1116761.

K. S. Wang et al., “Accurate diagnosis of colorectal cancer based on histopathology images using artificial intelligence,” BMC Med., vol. 19, no. 1, Dec. 2021, doi: 10.1186/s12916-021-01942-5.