Hybrid CNN–ViT Model for Breast Cancer Classification in Mammograms: A Three-Phase Deep Learning Framework

Vandana Saini; Meenu Khurana; Rama Krishna Challa

doi:10.35882/jeeemi.v7i4.920

Vandana Saini Chitkara University School of Engineering and Technology, Chitkara University, Himachal Pradesh, India https://orcid.org/0009-0000-6338-0504
Meenu Khurana Chitkara University School of Engineering and Technology, Chitkara University, Himachal Pradesh, India https://orcid.org/0000-0001-6515-7939
Rama Krishna Challa Department of Computer Science and Engineering, NITTTR, Chandigarh, India https://orcid.org/0000-0002-4647-2842

DOI: https://doi.org/10.35882/jeeemi.v7i4.920

Keywords: Mammogram; MIAS; Classification; INbreast; DDSM; CNN

Abstract

Breast cancer is one of the leading causes of death among women worldwide. Early and accurate detection plays a vital role in improving survival rates and guiding effective treatment. In this study, we propose a deep learning-based model for automatic breast cancer detection using mammogram images. The model is divided into three phases: preprocessing, segmentation, and classification. The first two phases, image enhancement and segmentation, were developed and validated in our previous works. Both phases were designed in a robust manner using learning networks; the usage of VGG-16 in preprocessing and U-net in segmentation helps in enhancing the overall classification performance. In this paper, we focus on the classification phase and introduce a novel hybrid deep learning based model that combines the strengths of Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). This model captures both fine-grained image details and the broader global context, making it highly effective for distinguishing between benign and malignant breast tumors. We also include attention-based feature fusion and Grad CAM visualizations to make predictions more explainable for clinical use and reference. The model was tested on multiple benchmark datasets, DDSM, INbreast, and MIAS, and a combination of all three datasets, and achieved excellent results, including 100% accuracy on MIAS and over 99% accuracy on other datasets. Compared to recent deep learning models, our method outperforms existing approaches in both accuracy and reliability. This research offers a promising step toward supporting radiologists with intelligent tools that can improve the speed and accuracy of breast cancer diagnosis.

Downloads

Download data is not yet available.

References

B. A. Mohamed, N. Salem, M. M. A. Hadhoud, and A. Seddik, “Automatic segmentation and classification of masses from digital mammograms,” Artif. Intell. Vis. Process., vol. 4, no. 4, pp. 17–17, 2016, doi: 10.14738/aivp.44.2151

M. Karunya and K. Rahimunnisa, “Breast cancer segmentation and classification using adaptive clustering technique,” in Proc. Int. Conf. Signal Process. Commun., 2017, pp. 1–6, doi: 10.1109/IICETA54559.2022.9888432

P. B. Chanda and S. Sarkar, “Detection and classification of breast cancer in mammographic images using efficient image segmentation technique,” Lect. Notes Electr. Eng., vol. 569, pp. 99–106, 2019.

W. Mustafa, A. A. Azmi, M. A. Jamlos, H. Alquran, W. Khairunizam, S. Ismail, A. Alkhayyat, and J. Haron, “Breast cancer detection and classification on mammogram images using morphological approach,” in Proc. 5th Int. Conf. Eng. Technol. Appl., 2022, pp. 260–264, doi: 10.1109/IICETA54559.2022.9888432

D. G. Chanda, “Detection and classification of tumors in a digital mammogram,” in Proc. Int. Conf. Comput. Sci. Inf. Technol., 2020, pp. 24–28.

R. Suresh, A. N. Rao, and B. E. Reddy, “Detection and classification of normal and abnormal patterns in mammograms using deep neural network,” Concurrency Computat. Pract. Exper., vol. 31, no. 2, pp. e5293, 2019. doi: 10.1002/cpe.5293.

S. D. Tzikopoulos, M. Mavroforakis, H. Georgiou, N. Dimitropoulos, and S. Theodoridis, “A fully automated scheme for mammographic segmentation and classification based on breast density and asymmetry,” Comput. Methods Programs Biomed., vol. 102, no. 1, pp. 47–63, Apr. 2011, doi: 10.1016/j.cmpb.2010.11.016.

Jafari, Z.; Karami, E. Breast Cancer Detection in Mammography Images: A CNN-Based Approach with Feature Selection. Information 2023, 14, 410, doi: 10.3390/info14070410.

T. Umamaheswari and Y. M. Mohanbabu, “ViT MAENB7: An innovative breast cancer diagnosis model from 3D mammograms using advanced segmentation and classification process,” Comput. Methods Programs Biomed., vol. 257, no. 1, pp. 108373, Jan. 2024, doi: 10.1016/j.cmpb.2024.108373

A. P. Charate and S. B. Jamge, “Mammogram image analysis for breast cancer detection,” in Proc. Natl. Conf. Adv. Comput. Technol., 2016, pp. 35–40.

R. K. Rajashekar, “Detection and classification of tumors in a digital mammogram,” Int. J. Comput. Appl., vol. 1, no. 1, pp. 24–28, 2012.

R. Pawar, S. Saraf, U. Dixit, and A. Jadhav, “Diagnosis of mammographic images for breast cancer detection using FF CSO algorithm,” in Proc. Adv. Comput. Commun. Technol. High Perform. Appl., 2023, pp. 1–5, doi: 10.1109/ACCTHPA57160.2023.10083387

S. H. Manishkumar and P. Saranya, "Detection and Classification of Breast Cancer from Mammogram Images Using Adaptive Deep Learning Technique," 2022 6th International Conference on Devices, Circuits and Systems (ICDCS), Coimbatore, India, 2022, pp. 327-331, doi: 10.1109/ICDCS54290.2022.9780770.

M. Sreevani and R. Latha, "A Deep Learning with Metaheuristic Optimization Driven Breast Cancer Segmentation and Classification Model," Eng. Technol. Appl. Sci. Res., vol. 15, no. 1, 2025, doi: 10.48084/etasr

M. J. J. Ghrabat et al., "Fully Automated Model on Breast Cancer Classification Using Deep Learning Classifiers," Indones. J. Electr. Eng. Comput. Sci., vol. 28, no. 1, pp. 183–191, 2022, doi: 10.11591/ijeecs.v28.i1.pp183-191.

R. Khdhir et al., "Breast Cancer Segmentation in Mammograms Using Antlion Optimization and CNN/GRU Architectures," in Proc. IWCMC, pp. 1030–1035, 2024, doi: 10.1109/IWCMC61514.2024.10592614.

A. Gerbasi et al., "DeepMiCa: Automatic Segmentation and Classification of Breast Microcalcifications," Comput. Methods Programs Biomed., vol. 235, pp. 107483, 2023, doi: 10.1016/j.cmpb.2023.10748.

V. Tiryaki, "Mass Segmentation and Classificationfrom Film Mammograms Using Cascaded Deep Transfer Learning," Biomed. Signal Process. Control., vol. 84, pp. 104819, 2023, doi: 10.1016/j.bspc.2023.104819.

A. Sinha et al., "ROI Segmentation for Breast Cancer Classification: Deep Learning Perspective," in Proc. IEEE INDISCON, pp. 1–7, 2023, doi: 10.1007/978-981-97-8526-1_39.

S. M’Rabet, A. Fnaiech and H. Sahli, "Heightened Breast Cancer Segmentation in Mammogram Images," 2024 International Conference on Control, Automation and Diagnosis (ICCAD), Paris, France, 2024, pp. 1-6, doi: 10.1109/ICCAD60883.2024.10553930.

V. Rathinam, R. Sasireka, and K. Valarmathi, "Adaptive Fuzzy C Means and Deep Learning for Mammogram Classification," Biomed. Signal Process. Control., vol. 88, pp. 105617, 2024,.

A. Islam et al., "Localization, Segmentation, and Classification of Mammographic Abnormalities Using Deep Learning," Proc. SPIE Med. Imaging, vol. 13174, pp. 131741Q, 2024, doi: 10.1016/j.bspc.2023.105617.

Sinha et al., "Segmentation Based Classification Deep Learning Model for Breast Cancer Detection," in Proc. IEEE MysuruCon, pp. 1–8, 2023, doi: 10.1109/59703.2023.10397015.

R. Remya and N. H. Rajini, "Transfer Learning Based Breast Cancer Detection and Classification," in Proc. ICEARS, pp. 1060–1065, 2022.

Remya and N. Hema Rajini, "Transfer Learning Based Breast Cancer Detection and Classification using Mammogram Images," 2022 International Conference on Electronics and Renewable Systems (ICEARS), Tuticorin, India, 2022, pp. 1060-1065, doi: 10.1109/ICEARS53579.2022.9751974

S. Almutairi et al., "An Efficient USE Net Deep Learning Model for Cancer Detection," Int. J. Intell. Syst., vol. 2023, pp. 1–12, 2023, doi: 10.1155/2023/8509433 .

Singh and R. Mishra, "Design and Development of Deep Learning Algorithm for Breast Cancer Classification,"International Symposium on Wireless Personal Multimedia Communications (WPMC)WPMC, pp. 297–302, 2022, doi: 10.1109/WPMC55625.2022.10014847.

C. K. Leung and H. H. Nguyen, "A Novel Deep Learning Approach for Breast Cancer Detection on Screening Mammography," in Proc. IEEE BIBE, pp. 277–284, 2023, doi: 10.1109/BIBE60311.2023.00052 .

Bouzar Benlabiod et al., " A novel breast cancer detection architecture based on a CNN-CBR system for mammogram classification," Comput. Biol. Med., vol. 163, pp. 107133, 2023, doi: 10.1016/j.compbiomed.2023.107133.

G. Kaur et al., "Patch Based All Convolutional Neural Network for Benign and Malignant Mammogram Classification," in Proc. IEEE SPICES, vol. 1, pp. 448–455, 2022, doi: 0.1109/SPICES52834.2022.9774096.

V. Saini, M. Khurana, and R. K. Challa, “VGG Inspired Convolutional Neural Network Denoiser for the Enhancement of Mammogram Images,” in Proc. ICMLA, vol. CCIS 2238, pp. 457–465, 2025, doi: 10.1007/978-3-031-75861-4_40.

Saini V, Khurana M, Challa R. K. A Hybrid Model for the Segmentation of Mammogram Images using Otsu Thresholding, Morphology and U Net. Biomed Pharmacol J 2025;18(1), doi: 10.13005/bpj/3130.

X. Zhao, Q. Zhu, and J. Wu, “AResNet ViT: A Hybrid CNN Transformer Network for Benign and Malignant Breast Nodule Classification in Ultrasound Images,” ArXiv, vol. abs/2407.19316, 2024, doi: 10.48550/arXiv.2407.19316

S. Kollem, C. Sirigiri, and S. Peddakrishna, “A novel hybrid deep CNN model for breast cancer classification using Lipschitz based image augmentation and recursive feature elimination,” Biomed. Signal Process. Control., vol. 95, pp. 106406, 2024, doi: 10.1016/j.bspc.2024.106406

Annepu, M. Abbas, H. R. Bitra, N. Vaegae, and K. Bagadi, “Advanced Breast Cancer Diagnostics With PolyBreastVit: A Combined PolyNet and Vision Transformer Approach,” Appl. Comput. Intell. Soft Comput., 2024, doi: 10.1155/2024/5574638.

E. V. J. Pulvera and D. M. Lao, "Enhancing Deep Learning-Based Breast Cancer Classification in Mammograms: A Multi-Convolutional Neural Network with Feature Concatenation, and an Applied Comparison of Best-Worst Multi-Attribute Decision-Making and Mutual Information Feature Selections," 2024 9th International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS), Okinawa, Japan, 2024, pp. 1-8, doi: 10.1109/ICIIBMS62405.2024.10792816.

A. F. A. Alshamrani and F. S. Z. Alshomrani, “Optimizing Breast Cancer Mammogram Classification Through a Dual Approach: A Deep Learning Framework Combining ResNet50, SMOTE, and Fully Connected Layers for Balanced and Imbalanced Data,” IEEE Access, vol. 13, pp. 4815–4826, 2025, doi: 10.1109/ACCESS.2024.3524633.

O. Tanimola, O. Shobayo, O. Popoola, and O. Okoyeigbo, “Breast Cancer Classification Using Fine Tuned SWIN Transformer Model on Mammographic Images,” Analytics, vol. 3, no. 4, 2024, doi: 10.3390/analytics3040026 .

A. Anbumani and P. Jayanthi, “Classification of Mammogram Breast Cancer Using Customized Deep Learning Model,” J. Intell. Fuzzy Syst., 2024, doi: 10.3233/jifs 232896.

Zebari, D.A.; Zeebaree, D.Q.; Abdulazeez, A.M.; Haron, H.; Abdul Hamed, H.N. Improved Threshold Based and Trainable Fully Automated Segmentation for Breast Cancer Boundary and Pectoral Muscle in Mammogram Images. IEEE Access 2020, 8, 203097–203116, doi: doi: 10.1109/ACCESS.2020.3036072 .