联合多视角Transformer编码与在线融合互学习的乳腺癌病理图像分类模型

李广丽; 叶艺源; 吴光庭; 李传秀; 吕敬钦; 张红斌

doi:10.12263/DZXB.20230305

您当前的位置：

首页 >

文章列表页 >

联合多视角Transformer编码与在线融合互学习的乳腺癌病理图像分类模型

学术论文 | 更新时间：2025-12-24

- 联合多视角Transformer编码与在线融合互学习的乳腺癌病理图像分类模型
- Breast Cancer Pathological Image Classification Model via Combining Multi-View Transformer Coding and Online Fusion Mutual Learning
- 电子学报 2024年52卷第7期页码：2369-2381
- 作者机构：
  
  1.华东交通大学信息工程学院，江西南昌 330013
  2.华东交通大学软件学院，江西南昌 330013
- 作者简介：
  
  [ "李广丽女，1977年11月出生，广西壮族自治区玉林人.华东交通大学信息工程学院教授，硕士生导师.主要研究方向为医学图像处理、数据挖掘、跨模态分析、推荐系统等.E-mail：1333@ecjtu.edu.cn" ]
  [ "叶艺源女，1998年12月出生，江西上饶人.华东交通大学信息工程学院硕士研究生.主要研究方向为医学图像处理、肿瘤图像识别、深度学习等.E-mail：1043549290@qq.com" ]
  [ "吴光庭男，1996年3月出生，江西抚州人.华东交通大学信息工程学院硕士.主要研究方向为医学图像处理、肿瘤图像识别、深度学习等.E-mail：1394692438@qq.com" ]
  [ "李传秀男，1995年9月出生，山东青岛人.华东交通大学信息工程学院硕士.主要研究方向为医学图像处理、肿瘤图像识别、深度学习等.E-mail：523536220@qq.com" ]
  [ "吕敬钦男，1984年11月出生，江西九江人.华东交通大学软件学院副教授.主要研究方向为行人检测、机器视觉等.E-mail：jingqinlv@ecjtu.edu.cn" ]
  [ "张红斌男，1979年10月出生，江苏如皋人.华东交通大学软件学院教授，硕士生导师.主要研究方向为医学影像分析、自然语言处理、跨模态分析等.E-mail：zhanghongbin@whu.edu.cn" ]
- 基金信息：
  
  国家自然科学基金(62161011);江西省重点研发计划重点项目（揭榜挂帅）(20223BBE51036);江西省自然科学基金(20212BAB202006);江西省社科规划项目(22TQ01);江西省教育厅科技项目(GJJ200628;GJJ2200639);江西省研究生创新基金(YC2022-s546)
- DOI：10.12263/DZXB.20230305
  中图分类号： TP391;
- 收稿：2023-04-04，
  
  修回：2024-03-12，
  
  纸质出版：2024-07-25
- 稿件说明：
移动端阅览
李广丽, 叶艺源, 吴光庭, 等. 联合多视角Transformer编码与在线融合互学习的乳腺癌病理图像分类模型[J]. 电子学报, 2024, 52(07): 2369-2381.

LI Guang-li, YE Yi-yuan, WU Guang-ting, et al. Breast Cancer Pathological Image Classification Model via Combining Multi-View Transformer Coding and Online Fusion Mutual Learning[J]. Acta Electronica Sinica, 2024, 52(07): 2369-2381.
李广丽, 叶艺源, 吴光庭, 等. 联合多视角Transformer编码与在线融合互学习的乳腺癌病理图像分类模型[J]. 电子学报, 2024, 52(07): 2369-2381. DOI：10.12263/DZXB.20230305

LI Guang-li, YE Yi-yuan, WU Guang-ting, et al. Breast Cancer Pathological Image Classification Model via Combining Multi-View Transformer Coding and Online Fusion Mutual Learning[J]. Acta Electronica Sinica, 2024, 52(07): 2369-2381. DOI：10.12263/DZXB.20230305

摘要

乳腺癌是女性最常见的癌症.单一网络在乳腺癌病理图像分类中存在缺陷，卷积神经网络无法提取全局上下文，而Transformer不能准确描述局部细节.本文提出联合多视角Transformer编码与在线融合互学习的乳腺癌病理图像分类模型（Multi-View Transformer Online Fusion Mutual Learning，MVT-OFML）.采用ResNet-50（Residual Network-50）提取图像局部特征，设计多视角Transformer编码模块，捕获图像中全局上下文；联合Logits和中间特征层构建OFML框架，实现ResNet-50与多视角Transformer编码模块间双向传递知识，使2个网络优势互补以完成乳腺癌病理图像分类.实验表明，在BreakHis和BACH数据集上，MVT-OFML的准确率比最强基线分别提升0.90%和2.26%，

均值比最强基线分别提升4.75%和3.21%.

Abstract

Breast cancer is the most common cancer in women. The single neural network used in breast cancer pathological image classification has the following defects: the convolutional neural network (CNN) lacks the ability to extract global context information while the Transformer lacks the ability to depict local lesion details. To alleviate the problem

a novel model

named multi-view Transformer coding and online fusion mutual learning (MVT-OFML)

is proposed for breast cancer pathological image classification. First

ResNet-50 is employed to extract local features in images. Then

a new multi-view Transformer (MVT) coding module is designed to capture the global context information. Finally

a novel online fusion mutual learning (OFML) framework based on the Logits and middle feature layers is designed to implement the bi-directional knowledge transfer between ResNet-50 and the MVT coding module. This makes the two networks complement each other to complete breast cancer pathological image classification. Experiments validated on BreakHis and BACH show that compared to the best baseline

the performance improvements of accuracy are 0.90% and 2.26%

respectively

whereas the correspondin

g improvements of average

score are 4.75% and 3.21%

respectively.

关键词

Keywords

references

SUNG H , FERLAY J , SIEGEL R L , et al . Global cancer statistics 2020: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries [J ] . A Cancer Journal for Clinicians , 2021 , 71 ( 3 ): 209 - 249 .

宋杰 , 肖亮 , 练智超 , 等 . 基于深度学习的数字病理图像分割综述与展望 [J ] . 软件学报 , 2021 , 32 ( 5 ): 1427 - 1460 .

SONG J , XIAO L , LIAN Z C , et al . Overview and prospect of deep learning for image segmentation in digital pathology [J ] . Journal of Software , 2021 , 32 ( 5 ): 1427 - 1460 . (in Chinese)

KRITHIGA R , GEETHA P . Breast cancer detection, segmentation and classification on histopathology images analysis: A systematic review [J ] . Archives of Computational Methods in Engineering , 2021 , 28 ( 4 ): 2607 - 2619 .

SRINIDHI C L , CIGA O , MARTEL A L . Deep neural network models for computational histopathology: A survey [J ] . Medical Image Analysis , 2021 , 67 : 101813 .

DOSOVITSKIY A , BEYER L , KOLESNIKOV A , et al . An image is worth 16 × 16 words: Transformers for image recognition at scale[EB/OL ] .( 2021-06-03 )[ 2023-03-02 ] . http://arxiv.org/pdf/2010.11929 http://arxiv.org/pdf/2010.11929 .

SANSONE M , FUSCO R , GRASSI F , et al . Machine learning approaches with textural features to calculate breast density on mammography [J ] . Current Oncology , 2023 , 30 ( 1 ): 839 - 853 .

WANG P , HU X L , LI Y M , et al . Automatic cell nuclei segmentation and classification of breast cancer histopathology images [J ] . Signal Processing , 2016 , 122 : 1 - 13 .

XUE H , YANG Q , CHEN S . SVM:Support vector machines [M ] // The Top Ten Algorithms in Data Mining . New York : Chapman and Hall/CRC , 2009 : 51 - 74 .

范虹 , 张程程 , 侯存存 , 等 . 结合双树复小波变换和改进密度峰值快速搜索聚类的乳腺MR图像分割 [J ] . 电子学报 , 2019 , 47 ( 10 ): 2149 - 2157 .

FAN H , ZHANG C C , HOU C C , et al . Dual-tree complex wavelet transform and improved density peak fast search and clustering method for breast MR image segmentation [J ] . Acta Electronica Sinica , 2019 , 47 ( 10 ): 2149 - 2157 . (in Chinese)

蒲秀娟 , 刘浩伟 , 韩亮 , 等 . 使用深度学习与海马体异构特征融合的阿尔茨海默病分类方法 [J ] . 电子学报 , 2023 , 51 ( 11 ): 3305 - 3319 .

PU X J , LIU H W , HAN L , et al . Method on Alzheimer's disease classification utilizing deep learning and hippocampus heterogeneous feature fusion [J ] . Acta Electronica Sinica , 2023 , 51 ( 11 ): 3305 - 3319 . (in Chinese)

ZOU Y , ZHANG J X , HUANG S , et al . Breast cancer histopathological image classification using attention high-order deep network [J ] . International Journal of Imaging Systems and Technology , 2022 , 32 ( 1 ): 266 - 279 .

CRUZ-ROA A , BASAVANHALLY A , GONZÁLEZ F , et al . Automatic detection of invasive ductal carcinoma in whole slide images with convolutional neural networks‍ [C ] // SPIE Proceedings of Medical Imaging 2014: Digital Pathology . SanDiego : SPIE , 2014 : 904103 .

ZHOU Y P , ZHANG C , GAO S S . Breast cancer classification from histopathological images using resolution adaptive network [J ] . IEEE Access , 1809 , 10 : 35977 - 35991 .

WANG Z N , MA Y T , ZHENG Y L , et al . Weakly-supervised learning using pretraining for classification in HER2 immunohistochemistry image of breast cancer [C ] // Proceedings of the 2021 6th International Conference on Mathematics and Artificial Intelligence . New York : ACM , 2021 : 66 - 71 .

田永林 , 王雨桐 , 王建功 , 等 . 视觉Transformer研究的关键问题: 现状及展望 [J ] . 自动化学报 , 2022 , 48 ( 4 ): 957 - 979 .

TIAN Y L , WANG Y T , WANG J G , et al . Key problems and progress of vision transformers: The state of the art and prospects [J ] . Acta Automatica Sinica , 2022 , 48 ( 4 ): 957 - 979 . (in Chinese)

PARMAR N , VASWANI A , USZKOREIT J , et al . Image transformer [EB/OL ] . ( 2018-06-15 )[ 2023-03-02 ] . http:// arxiv.org/pdf/1802.05751 http://arxiv.org/pdf/1802.05751 .

ALOTAIBI A , ALAFIF T , ALKHILAIWI F , et al . ViT-DeiT: An ensemble model for breast cancer histopathological images classification [EB/OL ] . ( 2022-11-01 )[ 2023-03-02 ] . http://arxiv.org/abs/2211.00749 http://arxiv.org/abs/2211.00749 .

ZOU Y , CHEN S N , SUN Q L , et al . DCET-net: Dual-stream convolution expanded transformer for breast cancer histopathological image classification [C ] // 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) . Piscataway : IEEE , 2021 : 1235 - 1240 .

TUMMALA S , KADRY S , BUKHARI S A C , et al . Classification of brain tumor from magnetic resonance imaging using vision transformers ensembling [J ] . Current Oncology , 2022 , 29 ( 10 ): 7498 - 7511 .

SUDHARSHAN P J , PETITJEAN C , SPANHOL F , et al . Multiple instance learning for histopathological breast cancer image classification [J ] . Expert Systems with Applications , 2019 , 117 : 103 - 111 .

ILSE M , TOMCZAK J M , WELLING M . Attention-based deep multiple instance learning [EB/OL ] . ( 2018-06-28 )[ 2023-03-02 ] . http://arxiv.org/abs/1802.04712 http://arxiv.org/abs/1802.04712 .

SHAO Z C , BIAN H , CHEN Y , et al . TransMIL: Transformer based correlated multiple instance learning for whole slide image classication [EB/OL ] .( 2021-06-02 )[ 2023-03-02 ] . http://arxiv.org/abs/2106.00908 http://arxiv.org/abs/2106.00908 .

LI G L , LI C X , WU G T , et al . Multi-view attention-guided multiple instance detection network for interpretable breast cancer histopathological image diagnosis [J ] . IEEE Access , 2021 , 9 : 79671 - 79684 .

ZHANG H Y , CISSE M , DAUPHIN Y N , et al . Mixup: Beyond empirical risk minimization [EB/OL ] . ( 2018-04-27 ) [ 2023-03-02 ] . http://arxiv.org/abs/1710.09412 http://arxiv.org/abs/1710.09412 .

LI X , SHEN X , ZHOU Y X , et al . Classification of breast cancer histopathological images using interleaved DenseNet with SENet (IDSNet) [J ] . PLoS One , 2020 , 15 ( 5 ): e0232127 .

MAN R , YANG P , XU B W . Classification of breast cancer histopathological images using discriminative patches screened by generative adversarial networks [J ] . IEEE Access , 2020 , 8 : 155362 - 155377 .

JIANG Y , CHEN L , ZHANG H , et al . Breast cancer histopathological image classification using convolutional neural networks with small SE-ResNet module [J ] . PLoS One , 2019 , 14 ( 3 ): e0214587 .

TUMMALA S , KIM J , KADRY S . BreaST-net: Multi-class classification of breast cancer from histopathological images using ensemble of swin transformers [J ] . Mathematics , 2022 , 10 ( 21 ): 4109 .

ZEROUAOUI H , IDRI A . Deep hybrid architectures for binary classification of medical breast cancer images [J ] . Biomedical Signal Processing and Control , 2022 , 71 : 103226 .

YE H C , ZHANG X F , HUANG Z Z , et al . HybridDNN: A framework for high-performance hybrid DNN accelerator design and implementation [C ] // 2020 57th ACM/IEEE Design Automation Conference (DAC) . Piscataway : IEEE , 2020 : 1 - 6 .

SENOUSY Z , ABDELSAMEA M M , MOHAMED M M , et al . 3E-net: Entropy-based elastic ensemble of deep convolutional neural networks for grading of invasive breast carcinoma histopathological microscopic images [J ] . Entropy , 2021 , 23 ( 5 ): 620 .

LIU L X , FENG W X , CHEN C , et al . Classification of breast cancer histology images using MSMV-PFENet [J ] . Scientific Reports , 2022 , 12 ( 1 ): 17447 .

VAN DER MAATEN L , HINTON G . Visualizing data using t-SNE [J ] . Journal of Machine Learning Research , 2008 , 9 : 2579 - 2625 .

SELVARAJU R R , DAS A , VEDANTAM R , et al . Grad-CAM: Why did you say that? [EB/OL ] . ( 2017-01-25 )[ 2023-03-02 ] . http://arxiv.org/abs/1611.07450 http://arxiv.org/abs/1611.07450 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

统一全局空间表达的脑电信号跨被试情感识别

基于事件驱动的车道线识别算法研究

基于深度学习网络的神经元自适应投影分类方法