Abstract

While most computer-aided diagnosis (CAD) systems for breast cancer use image features largely from mammograms, structured clinical information has an important complementary diagnostic value and model reasoning remains largely unaddressed. The aim of this study was to explore the possibility of combining mammography with categorical clinical features to provide a more accurate pathologist classification of breast lesions, and to determine if explainable AI (XAI) can be used to make such models interpretable for clinical use. We propose three strategies for multimodal deep-learning fusion: feature concatenation, co-attention, and cross-attention; the latter is a novel technique that has not yet been explored in this context. We compare these three strategies in a systematic manner, and integrate a ResNet-50 based mammographic embedding with a one-hot-encoded clinical feature vector, taking into account different missing-modality scenarios that are present in clinical practice, using modality dropout. The interpretation of reasoning is done at the image level (Grad- CAM) and the feature level (SHAP & LIME). The proposed crossattention fusion framework was evaluated based on the publicly available CBIS-DDSM benchmark, and the results were compared with the best unimodal baseline and simple fusion methods. The proposed fusion method outperforms the unimodal baseline and simple fusion methods, achieving an AUC-ROC of 0.98, accuracy of 0.96, F1-score of 0.94, precision of 0.92, and recall of 0.95, with the improvement confirmed by McNemar’s test (p < 0.01) and stable bootstrap confidence intervals. The three types of explanations (Grad-CAM, SHAP, and LIME) agreed on clinically relevant areas of the images and clinically relevant variables and the case-level analysis of the correct and incorrect predictions demonstrated the value of the explanations and limitations of the framework. The results suggest that crossattention- based multimodal fusion and multi-level explainability can meaningfully benefit the accuracy and the clinical trustworthiness of using AI assistance for breast cancer diagnosis, and provide a template for multimodal CAD systems that are interpretable and applicable to real-world, missing-data clinical workflows.

References

• M. Abavisani, L. Wu, S. Hu, J. Tetreault, and A. Jaimes, “Multimodal categorization of crisis events in social media,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), 2020, pp. 14679–14689.
• M. Ahmed, T. Bibi, R. A. Khan, and S. Nasir, “Enhancing breast cancer diagnosis in mammography: Evaluation and integration of convolutional neural networks and explainable AI,” arXiv preprint, 2024. [Online]. Available: http://arxiv.org/abs/2404.03892
• Z. A. Ansari, M. M. Tripathi, and R. Ahmed, “The role of explainable AI in enhancing breast cancer diagnosis using machine learning and deep learning models,” Discover Artificial Intelligence, vol. 5, art. 75, 2025. doi: 10.1007/s44163-025- 00307-8.
• A. B. Arrieta et al., “Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI,” Information Fusion, vol. 58, pp. 82–115, 2020.
• F. Bray, M. Laversanne, H. Sung, J. Ferlay, R. L. Siegel, I. Soerjomataram, and A. Jemal, “Global cancer statistics 2022: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries,” CA: A Cancer Journal for Clinicians, vol. 74, no. 3, pp. 229–263, 2024. doi: 10.3322/caac.21834.
• H. Chougrad, H. Zouaki, and O. Alheyane, “Multi-label transfer learning for the early diagnosis of breast cancer,” Neurocomputing, vol. 392, pp. 168–180, 2020.
• F. Doshi-Velez and B. Kim, “Towards a rigorous science of interpretable machine learning,” arXiv preprint arXiv:1702.08608, 2017.
• S. Eskandari, A. Eslamian, and Q. Cheng, “Comparative analysis of deep learning models for breast cancer classification,” arXiv preprint, 2024. doi: 10.48550/arXiv.2408.16859.
• T.W. Freer and M. J. Ulissey, “Screening mammography with computer-aided detection: Prospective study of 12,860 patients in a community breast center,” Radiology, vol. 220, no. 3, pp. 781–786, 2001.
• L. Garrucho et al., “A large-scale multicenter breast cancer DCE-MRI benchmark dataset with expert segmentations,” Scientific Data, vol. 12, art. 453, 2025. doi: 10.1038/s41597- 025-04707-4.
• A. Ghasemi, S. Hashtarkhani, D. L. Schwartz, and A. Shaban- Nejad, “Explainable artificial intelligence in breast cancer detection and risk prediction: A systematic scoping review,” Cancer Innovation, vol. 3, no. 5, art. e136, 2024. doi: 10.1002/cai2.136.
• D. Hao, D. Arefan, M. Zuley, W. Berg, and S. Wu, “Adversarially robust feature learning for breast cancer diagnosis,” arXiv preprint, 2024. doi: 10.48550/arXiv.2402.08768.
• K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778.
• S. Hussain, M. A. Teevno, U. Naseem, D. B. Avendano Avalos, S. Cardona-Huerta, and J. G. Tamez-Pe˜na, “Multiview multimodal feature fusion for breast cancer classification using deep learning,” IEEE Access, vol. 13, pp. 9265–9275, 2025. doi: 10.1109/ACCESS.2024.3524203.
• S. Iniyan, M. Senthil Raja, R. Poonguzhali, A. Vikram, J. V. N. Ramesh, S. N. Mohanty, and K. V. Dudekula, “Enhanced breast cancer diagnosis through integration of computer vision with fusion based joint transfer learning using multi modality medical images,” Scientific Reports, vol. 14, art. 28376, 2024. doi: 10.1038/s41598-024-79363-6.
• K. Kerlikowske, A. J. Cook, D. S. M. Buist, S. R. Cummings, C. Vachon, P. Vacek, and D. L. Miglioretti, “Breast cancer risk by breast density, menopause, and postmenopausal hormone therapy use,” Journal of Clinical Oncology, vol. 28, no. 24, p. 3830, 2010.
• J. Kim, A. Harper, V. McCormack, H. Sung, N. Houssami, E. Morgan, et al., “Global patterns and trends in breast cancer incidence and mortality across 185 countries,” Nature Medicine, published online 24 February 2025. doi: 10.1038/s41591-025-03502-3.
• A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” Commun. ACM, vol. 60, no. 6, pp. 84–90, 2017.
• R. S. Lee, F. Gimenez, A. Hoogi, K. K. Miyake, M. Gorovoy, and D. L. Rubin, “A curated mammography data set for use in computer-aided detection and diagnosis research,” Scientific Data, vol. 4, no. 1, pp. 1–9, 2017.
• R. S. Lee, F. Gimenez, A. Hoogi, and D. L. Rubin, “Curated Breast Imaging Subset of DDSM (CBISDDSM),” The Cancer Imaging Archive, 2017. doi: 10.7937/K9/TCIA.2016.7O02S9CY.
• M. Li, Y. Fang, J. Shao, Y. Jiang, G. Xu, X. W. Cui, and X. Wu, “Vision transformer-based multimodal fusion network for classification of tumor malignancy on breast ultrasound: A retrospective multicenter study,” International Journal of Medical Informatics, vol. 196, art. 105793, 2025. doi: 10.1016/j.ijmedinf.2025.105793.
• J. Lu, D. Batra, D. Parikh, and S. Lee, “ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-andlanguage tasks,” in Advances in Neural Information Processing Systems (NeurIPS), 2019.
• T. P. Matthews et al., “A multi-site study of a breast density deep learning model for full-field digital mammography images and synthetic mammography images,” Radiology: Artificial Intelligence, art. e200015, 2020.
• G. Murphy and R. Singh, “Comparative analysis and ensemble enhancement of leading CNN architectures for breast cancer classification,” arXiv preprint, 2024. doi: 10.48550/arXiv.2410.03333.
• National Cancer Institute, “SEER stat fact sheets: Breast cancer stats,” 2023.
• H. M. Rai, J. Yoo, S. Agarwal, and N. Agarwal, “LightweightUNet: Multimodal deep learning with GAN-augmented imaging data for efficient breast cancer detection,” Bioengineering, vol. 12, no. 1, art. 73, 2025. doi: 10.3390/bioengineering12010073.
• O. Rezaeian, A. E. Bayrak, and O. Asan, “The impact of AI explanations on clinicians’ trust and diagnostic accuracy in breast cancer,” Applied Ergonomics, vol. 129, art. 104577, 2025. doi: 10.1016/j.apergo.2025.104577.
• D. Ribli, A. Horv´ath, Z. Unger, P. Pollner, and I. Csabai, “Detecting and classifying lesions in mammograms with deep learning,” Scientific Reports, vol. 8, no. 1, pp. 1–7, 2018.
• S. Saharan, S. Bawa, and N. Kumar, “Dynamic pricing techniques for intelligent transportation system in smart cities: A systematic review,” Computer Communications, vol. 150, pp. 603–625, 2020.
• S. Saharan, S. Bawa, and N. Kumar, “OP3S: On-street occupancy based parking prices prediction system for ITS,” in 2020 IEEE Globecom Workshops (GC Wkshps), 2020, pp. 1– 6.
• S. K. B. Sangeetha, S. K. Mathivanan, P. Karthikeyan, H. Rajadurai, B. D. Shivahare, S. Mallik, and H. Qin, “An enhanced multimodal fusion deep learning neural network for lung cancer classification,” Systems and Soft Computing, vol. 6, art. 200068, 2024. doi: 10.1016/j.sasc.2023.200068.
• L. Shen, L. R. Margolies, J. H. Rothstein, E. Fluder, R. McBride, and W. Sieh, “Deep learning to improve breast cancer detection on screening mammography,” Scientific Reports, vol. 9, no. 1, pp. 1–12, 2019.
• E. J. Topol, “High-performance medicine: The convergence of human and artificial intelligence,” Nature Medicine, vol. 25, no. 1, pp. 44–56, 2019.
• M. Van Lent, W. Fisher, and M. Mancuso, “An explainable artificial intelligence system for small-unit tactical behavior,” in Proc. National Conf. on Artificial Intelligence, 2004, pp. 900–907.
• G.-I. Verras, L. Tchabashvili, D.-D. Chlorogiannis, F. Mulita, and M.-I. Argentou, “Updated clinical evidence on the role of adipokines and breast cancer: A review,” Cancers, vol. 15, no. 5, art. 1572, 2023.
• H. Q. Vo, P. Yuan, T. He, S. T. C. Wong, and H. V. Nguyen, “Multimodal breast lesion classification using cross-attention deep networks,” in 2021 IEEE EMBS Int. Conf. Biomedical and Health Informatics (BHI), 2021. doi: 10.1109/BHI50953.2021.9508604.
• Z. Wang, R. Lin, Y. Li, J. Zeng, Y. Chen, W. Ouyang, H. Li, X. Jia, Z. Lai, Y. Yu, H. Yao, andW. Su, “Deep learning-based multi-modal data integration enhancing breast cancer diseasefree survival prediction,” Precision Clinical Medicine, vol. 7, no. 2, art. pbae012, 2024. doi: 10.1093/pcmedi/pbae012.
• World Health Organization, “Breast cancer,” WHO Fact Sheets, 2023.