Breast Cancer Pathological Image Classification Model via Combining Multi-View Transformer Coding and Online Fusion Mutual Learning

LI Guang-li; YE Yi-yuan; WU Guang-ting; LI Chuan-xiu; LÜ Jing-qin; ZHANG Hong-bin

doi:10.12263/DZXB.20230305

您当前的位置：

首页 >

文章列表页 >

Breast Cancer Pathological Image Classification Model via Combining Multi-View Transformer Coding and Online Fusion Mutual Learning

PAPERS | 更新时间：2025-12-24

- Breast Cancer Pathological Image Classification Model via Combining Multi-View Transformer Coding and Online Fusion Mutual Learning
- ACTA ELECTRONICA SINICA Vol. 52, Issue 7, Pages: 2369-2381(2024)
- 作者机构：
  
  1.华东交通大学信息工程学院，江西南昌 330013
  2.华东交通大学软件学院，江西南昌 330013
- 作者简介：
- 基金信息：
  
  National Natural Science Foundation of China(62161011);Key Research and Development Plan of Jiangxi Province (Key Project)(20223BBE51036);Natural Science Foundation of Jiangxi Province(20212BAB202006);Humanity and Social Science Foundation of Jiangxi Province(22TQ01);Science and Technology Projects of Jiangxi Provincial Department of Education(GJJ200628;GJJ2200639);Graduate Innovation Fundation Project of Jiangxi Province(YC2022-s546)
- DOI：10.12263/DZXB.20230305
  CLC： TP391;
- Received：04 April 2023，
  
  Revised：2024-03-12，
  
  Published：25 July 2024
- 稿件说明：
移动端阅览
李广丽, 叶艺源, 吴光庭, 等. 联合多视角Transformer编码与在线融合互学习的乳腺癌病理图像分类模型[J]. 电子学报, 2024, 52(07): 2369-2381.

LI Guang-li, YE Yi-yuan, WU Guang-ting, et al. Breast Cancer Pathological Image Classification Model via Combining Multi-View Transformer Coding and Online Fusion Mutual Learning[J]. Acta Electronica Sinica, 2024, 52(07): 2369-2381.
李广丽, 叶艺源, 吴光庭, 等. 联合多视角Transformer编码与在线融合互学习的乳腺癌病理图像分类模型[J]. 电子学报, 2024, 52(07): 2369-2381. DOI：10.12263/DZXB.20230305

LI Guang-li, YE Yi-yuan, WU Guang-ting, et al. Breast Cancer Pathological Image Classification Model via Combining Multi-View Transformer Coding and Online Fusion Mutual Learning[J]. Acta Electronica Sinica, 2024, 52(07): 2369-2381. DOI：10.12263/DZXB.20230305

摘要

乳腺癌是女性最常见的癌症.单一网络在乳腺癌病理图像分类中存在缺陷，卷积神经网络无法提取全局上下文，而Transformer不能准确描述局部细节.本文提出联合多视角Transformer编码与在线融合互学习的乳腺癌病理图像分类模型（Multi-View Transformer Online Fusion Mutual Learning，MVT-OFML）.采用ResNet-50（Residual Network-50）提取图像局部特征，设计多视角Transformer编码模块，捕获图像中全局上下文；联合Logits和中间特征层构建OFML框架，实现ResNet-50与多视角Transformer编码模块间双向传递知识，使2个网络优势互补以完成乳腺癌病理图像分类.实验表明，在BreakHis和BACH数据集上，MVT-OFML的准确率比最强基线分别提升0.90%和2.26%，

均值比最强基线分别提升4.75%和3.21%.

Abstract

Breast cancer is the most common cancer in women. The single neural network used in breast cancer pathological image classification has the following defects: the convolutional neural network (CNN) lacks the ability to extract global context information while the Transformer lacks the ability to depict local lesion details. To alleviate the problem

a novel model

named multi-view Transformer coding and online fusion mutual learning (MVT-OFML)

is proposed for breast cancer pathological image classification. First

ResNet-50 is employed to extract local features in images. Then

a new multi-view Transformer (MVT) coding module is designed to capture the global context information. Finally

a novel online fusion mutual learning (OFML) framework based on the Logits and middle feature layers is designed to implement the bi-directional knowledge transfer between ResNet-50 and the MVT coding module. This makes the two networks complement each other to complete breast cancer pathological image classification. Experiments validated on BreakHis and BACH show that compared to the best baseline

the performance improvements of accuracy are 0.90% and 2.26%

respectively

whereas the correspondin

g improvements of average

score are 4.75% and 3.21%

respectively.

关键词

Keywords

references

SUNG H , FERLAY J , SIEGEL R L , et al . Global cancer statistics 2020: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries [J ] . A Cancer Journal for Clinicians , 2021 , 71 ( 3 ): 209 - 249 .

宋杰 , 肖亮 , 练智超 , 等 . 基于深度学习的数字病理图像分割综述与展望 [J ] . 软件学报 , 2021 , 32 ( 5 ): 1427 - 1460 .

SONG J , XIAO L , LIAN Z C , et al . Overview and prospect of deep learning for image segmentation in digital pathology [J ] . Journal of Software , 2021 , 32 ( 5 ): 1427 - 1460 . (in Chinese)

KRITHIGA R , GEETHA P . Breast cancer detection, segmentation and classification on histopathology images analysis: A systematic review [J ] . Archives of Computational Methods in Engineering , 2021 , 28 ( 4 ): 2607 - 2619 .

SRINIDHI C L , CIGA O , MARTEL A L . Deep neural network models for computational histopathology: A survey [J ] . Medical Image Analysis , 2021 , 67 : 101813 .

DOSOVITSKIY A , BEYER L , KOLESNIKOV A , et al . An image is worth 16 × 16 words: Transformers for image recognition at scale[EB/OL ] .( 2021-06-03 )[ 2023-03-02 ] . http://arxiv.org/pdf/2010.11929 http://arxiv.org/pdf/2010.11929 .

SANSONE M , FUSCO R , GRASSI F , et al . Machine learning approaches with textural features to calculate breast density on mammography [J ] . Current Oncology , 2023 , 30 ( 1 ): 839 - 853 .

WANG P , HU X L , LI Y M , et al . Automatic cell nuclei segmentation and classification of breast cancer histopathology images [J ] . Signal Processing , 2016 , 122 : 1 - 13 .

XUE H , YANG Q , CHEN S . SVM:Support vector machines [M ] // The Top Ten Algorithms in Data Mining . New York : Chapman and Hall/CRC , 2009 : 51 - 74 .

范虹 , 张程程 , 侯存存 , 等 . 结合双树复小波变换和改进密度峰值快速搜索聚类的乳腺MR图像分割 [J ] . 电子学报 , 2019 , 47 ( 10 ): 2149 - 2157 .

FAN H , ZHANG C C , HOU C C , et al . Dual-tree complex wavelet transform and improved density peak fast search and clustering method for breast MR image segmentation [J ] . Acta Electronica Sinica , 2019 , 47 ( 10 ): 2149 - 2157 . (in Chinese)

蒲秀娟 , 刘浩伟 , 韩亮 , 等 . 使用深度学习与海马体异构特征融合的阿尔茨海默病分类方法 [J ] . 电子学报 , 2023 , 51 ( 11 ): 3305 - 3319 .

PU X J , LIU H W , HAN L , et al . Method on Alzheimer's disease classification utilizing deep learning and hippocampus heterogeneous feature fusion [J ] . Acta Electronica Sinica , 2023 , 51 ( 11 ): 3305 - 3319 . (in Chinese)

ZOU Y , ZHANG J X , HUANG S , et al . Breast cancer histopathological image classification using attention high-order deep network [J ] . International Journal of Imaging Systems and Technology , 2022 , 32 ( 1 ): 266 - 279 .

CRUZ-ROA A , BASAVANHALLY A , GONZÁLEZ F , et al . Automatic detection of invasive ductal carcinoma in whole slide images with convolutional neural networks‍ [C ] // SPIE Proceedings of Medical Imaging 2014: Digital Pathology . SanDiego : SPIE , 2014 : 904103 .

ZHOU Y P , ZHANG C , GAO S S . Breast cancer classification from histopathological images using resolution adaptive network [J ] . IEEE Access , 1809 , 10 : 35977 - 35991 .

WANG Z N , MA Y T , ZHENG Y L , et al . Weakly-supervised learning using pretraining for classification in HER2 immunohistochemistry image of breast cancer [C ] // Proceedings of the 2021 6th International Conference on Mathematics and Artificial Intelligence . New York : ACM , 2021 : 66 - 71 .

田永林 , 王雨桐 , 王建功 , 等 . 视觉Transformer研究的关键问题: 现状及展望 [J ] . 自动化学报 , 2022 , 48 ( 4 ): 957 - 979 .

TIAN Y L , WANG Y T , WANG J G , et al . Key problems and progress of vision transformers: The state of the art and prospects [J ] . Acta Automatica Sinica , 2022 , 48 ( 4 ): 957 - 979 . (in Chinese)

PARMAR N , VASWANI A , USZKOREIT J , et al . Image transformer [EB/OL ] . ( 2018-06-15 )[ 2023-03-02 ] . http:// arxiv.org/pdf/1802.05751 http://arxiv.org/pdf/1802.05751 .

ALOTAIBI A , ALAFIF T , ALKHILAIWI F , et al . ViT-DeiT: An ensemble model for breast cancer histopathological images classification [EB/OL ] . ( 2022-11-01 )[ 2023-03-02 ] . http://arxiv.org/abs/2211.00749 http://arxiv.org/abs/2211.00749 .

ZOU Y , CHEN S N , SUN Q L , et al . DCET-net: Dual-stream convolution expanded transformer for breast cancer histopathological image classification [C ] // 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) . Piscataway : IEEE , 2021 : 1235 - 1240 .

TUMMALA S , KADRY S , BUKHARI S A C , et al . Classification of brain tumor from magnetic resonance imaging using vision transformers ensembling [J ] . Current Oncology , 2022 , 29 ( 10 ): 7498 - 7511 .

SUDHARSHAN P J , PETITJEAN C , SPANHOL F , et al . Multiple instance learning for histopathological breast cancer image classification [J ] . Expert Systems with Applications , 2019 , 117 : 103 - 111 .

ILSE M , TOMCZAK J M , WELLING M . Attention-based deep multiple instance learning [EB/OL ] . ( 2018-06-28 )[ 2023-03-02 ] . http://arxiv.org/abs/1802.04712 http://arxiv.org/abs/1802.04712 .

SHAO Z C , BIAN H , CHEN Y , et al . TransMIL: Transformer based correlated multiple instance learning for whole slide image classication [EB/OL ] .( 2021-06-02 )[ 2023-03-02 ] . http://arxiv.org/abs/2106.00908 http://arxiv.org/abs/2106.00908 .

LI G L , LI C X , WU G T , et al . Multi-view attention-guided multiple instance detection network for interpretable breast cancer histopathological image diagnosis [J ] . IEEE Access , 2021 , 9 : 79671 - 79684 .

ZHANG H Y , CISSE M , DAUPHIN Y N , et al . Mixup: Beyond empirical risk minimization [EB/OL ] . ( 2018-04-27 ) [ 2023-03-02 ] . http://arxiv.org/abs/1710.09412 http://arxiv.org/abs/1710.09412 .

LI X , SHEN X , ZHOU Y X , et al . Classification of breast cancer histopathological images using interleaved DenseNet with SENet (IDSNet) [J ] . PLoS One , 2020 , 15 ( 5 ): e0232127 .

MAN R , YANG P , XU B W . Classification of breast cancer histopathological images using discriminative patches screened by generative adversarial networks [J ] . IEEE Access , 2020 , 8 : 155362 - 155377 .

JIANG Y , CHEN L , ZHANG H , et al . Breast cancer histopathological image classification using convolutional neural networks with small SE-ResNet module [J ] . PLoS One , 2019 , 14 ( 3 ): e0214587 .

TUMMALA S , KIM J , KADRY S . BreaST-net: Multi-class classification of breast cancer from histopathological images using ensemble of swin transformers [J ] . Mathematics , 2022 , 10 ( 21 ): 4109 .

ZEROUAOUI H , IDRI A . Deep hybrid architectures for binary classification of medical breast cancer images [J ] . Biomedical Signal Processing and Control , 2022 , 71 : 103226 .

YE H C , ZHANG X F , HUANG Z Z , et al . HybridDNN: A framework for high-performance hybrid DNN accelerator design and implementation [C ] // 2020 57th ACM/IEEE Design Automation Conference (DAC) . Piscataway : IEEE , 2020 : 1 - 6 .

SENOUSY Z , ABDELSAMEA M M , MOHAMED M M , et al . 3E-net: Entropy-based elastic ensemble of deep convolutional neural networks for grading of invasive breast carcinoma histopathological microscopic images [J ] . Entropy , 2021 , 23 ( 5 ): 620 .

LIU L X , FENG W X , CHEN C , et al . Classification of breast cancer histology images using MSMV-PFENet [J ] . Scientific Reports , 2022 , 12 ( 1 ): 17447 .

VAN DER MAATEN L , HINTON G . Visualizing data using t-SNE [J ] . Journal of Machine Learning Research , 2008 , 9 : 2579 - 2625 .

SELVARAJU R R , DAS A , VEDANTAM R , et al . Grad-CAM: Why did you say that? [EB/OL ] . ( 2017-01-25 )[ 2023-03-02 ] . http://arxiv.org/abs/1611.07450 http://arxiv.org/abs/1611.07450 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Unified Global Spatial Representation for EEG Subject-Independent Emotion Recognition

Research on Event‑Driven Lane Recognition Algorithms

A Neuronal Classification Approach with Adaptive Projection Using Deep Learning Networks

Related Author

LV Jing-qin

REN Yong-gong

WANG Yi-xin

ZHANG Jing

CHEN Yi-jie

XU Pin-jie

LI Zhi-nan

ZHAO Di

Related Institution

School of Computer and Information Technology， Liaoning Normal University

Institute of Computing Technology， Chinese Academy of Sciences

School of Information and Communication Engineering， Beijing University of Posts and Telecommunications

International School， Beijing University of Posts and Telecommunications

University of Chinese Academy of Sciences

⁰