基于图表征知识蒸馏的图像分类方法

杨传广; 陈路明; 赵二虎; 安竹林; 徐勇军

doi:10.12263/DZXB.20230976

您当前的位置：

首页 >

文章列表页 >

基于图表征知识蒸馏的图像分类方法

学术论文 | 更新时间：2026-05-07

- 基于图表征知识蒸馏的图像分类方法
- Graph-Based Representation Knowledge Distillation for Image Classification
- 电子学报 2024年52卷第10期页码：3435-3447
- 作者机构：
  
  1.中国科学院计算技术研究所，北京 100190
  2.93114部队，北京 100080
- 作者简介：
  
  [ "杨传广男，1996年4月出生，山东潍坊人.中国科学院计算技术研究所特别研究助理.主要研究方向为人工智能、计算机视觉和模型轻量化与加速.E-mail: yangchuanguang@ict.ac.cn" ]
  [ "陈路明男，1991年4月出生，河北石家庄人.93114部队工程师.主要研究方向为机器视觉、深度学习. E-mail: 295170692@qq.com" ]
  [ "赵二虎男，1985年9月出生，河北邢台人.中国科学院计算技术研究所高级工程师、硕士生导师、特聘研究骨干.主要研究方向为嵌入式智能计算系统. E-mail: zhaoerhu@ict.ac.cn" ]
  [ "安竹林男，1980年4月出生，山东日照人.中国科学院计算技术研究所副研究员.主要研究方向为人工智能、计算机视觉和模型轻量化与加速. E-mail: anzhulin@ict.ac.cn" ]
  [ "徐勇军男，1979年7月出生，安徽安庆人.中国科学院计算技术研究所正研级高级工程师、博士生导师、专项技术研究中心主任.主要研究方向为数据智能. E-mail: xyj@ict.ac.cn" ]
- 基金信息：
  
  国家自然科学基金(62072434);北京市自然科学基金(4212027)
- DOI：10.12263/DZXB.20230976
  中图分类号： TP391.42;
- 收稿：2023-10-19，
  
  修回：2023-12-19，
  
  纸质出版：2024-10-25
- 稿件说明：
移动端阅览
杨传广, 陈路明, 赵二虎, 等. 基于图表征知识蒸馏的图像分类方法[J]. 电子学报, 2024, 52(10): 3435-3447.

YANG Chuan-guang, CHEN Lu-ming, ZHAO Er-hu, et al. Graph-Based Representation Knowledge Distillation for Image Classification[J]. Acta Electronica Sinica, 2024, 52(10): 3435-3447.
杨传广, 陈路明, 赵二虎, 等. 基于图表征知识蒸馏的图像分类方法[J]. 电子学报, 2024, 52(10): 3435-3447. DOI：10.12263/DZXB.20230976

YANG Chuan-guang, CHEN Lu-ming, ZHAO Er-hu, et al. Graph-Based Representation Knowledge Distillation for Image Classification[J]. Acta Electronica Sinica, 2024, 52(10): 3435-3447. DOI：10.12263/DZXB.20230976

摘要

知识蒸馏的核心思想是利用1个作为教师网络的大型模型来指导1个作为学生网络的小型模型，提升学生网络在图像分类任务上的性能.现有知识蒸馏方法通常从单一的输入样本中提取类别概率或特征信息作为知识，并没有对样本间关系进行建模，造成网络的表征学习能力下降.为解决此问题，本文引入图卷积神经网络，将输入样本集视为图结点构建关系图，图中的每个样本都可以聚合其他样本信息，提升样本的表征能力.本文从图结点和图关系2个角度构建图表征知识蒸馏误差，利用元学习引导学生网络自适应学习教师网络更佳的图表征，提升学生网络的图建模能力.相比于基线方法，本文提出的图表征知识蒸馏方法在加拿大高等研究院（Canadian Institute For Advanced Research，CIFAR）发布的100种分类数据集上提升了3.70%的分类准确率，表明本文方法引导学生网络学习到了更具有判别性的特征空间，提升了图像分类能力.

Abstract

The core idea of knowledge distillation is to use a large model as the teacher network to guide a small model as the student network

improving the performance of the student network in image classification tasks. Existing knowledge distillation methods often extract category probability or feature information as knowledge from a single input sample. They could not model the relationships between samples

decreasing the network’s representation learning ability. To solve this problem

this paper introduces a graph convolutional neural network

which treats the input sample set as graph nodes to construct a relationship graph. Each sample in the graph could aggregate information from other samples

improving its own representation ability. This paper constructs the distillation loss of graph representation knowledge from the perspectives of graph nodes and relationships. It uses meta-learning to guide the student network to adaptively learn better graph representations from a teacher network

thereby improving the graph modeling ability of the student network. Compared to the baseline method

the graph-based representation knowledge distillation method improves the classification accuracy by 3.70% on the 100-classification dataset published by Canadian Institute For Advanced Research. The result indicates that the proposed method makes the student network learn a more discriminative feature space

thereby improving its image classification ability.

关键词

Keywords

references

王波 , 黄冕 , 刘利军 , 等 . 基于多层聚焦Inception-V3卷积网络的细粒度图像分类 [J ] . 电子学报 , 2022 , 50 ( 1 ): 72 - 78 .

WANG B , HUANG M , LIU L J , et al . Multi-layer focused Inception-V3 models for fine-grained visual recognition [J ] . Acta Electronica Sinica , 2022 , 50 ( 1 ): 72 - 78 . (in Chinese)

HE K , ZHANG X , REN S , et al . Deep residual learning for image recognition [C ] // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE Computer Society , 2016 : 770 - 778 .

ZHU H , AN Z L , YANG C G , et al . EENA: Efficient evolution of neural architecture [C ] // 2019 International Conference on Computer Vision Workshop (ICCVW) . Piscataway : IEEE , 2019 : 1891 - 1899 .

YANG C G , AN Z L , LI C , et al . Multi-objective pruning for CNNs using genetic algorithm [C ] // International Conference on Artificial Neural Networks . Cham : Springer , 2019 : 299 - 305 .

郑云飞 , 王晓兵 , 张雄伟 , 等 . 基于金字塔知识的自蒸馏HRNet目标分割方法 [J ] . 电子学报 , 2023 , 51 ( 3 ): 746 - 756 .

ZHENG Y F , WANG X B , ZHANG X W , et al . The self-distillation HRNet object segmentation based on the pyramid knowledge [J ] . Acta Electronica Sinica , 2023 , 51 ( 3 ): 746 - 756 . (in Chinese)

HINTON G , VINYALS O , DEAN J . Distilling the knowledge in a neural network [EB/OL ] . ( 2015-03-05 )[ 2023-09-02 ] . https://arxiv.org/abs/1503.02531 https://arxiv.org/abs/1503.02531 .

ROMERO A , BALLAS N , KAHOU S E , et al . Fitnets: Hints for thin deep nets [EB/OL ] . ( 2014-12-19 )[ 2023-09-05 ] . http://arxiv.org/abs/1412.6550 http://arxiv.org/abs/1412.6550 .

ZAGORUYKO S , KOMODAKIS N . Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer [EB/OL ] . ( 2016-12-12 )[ 2023-09-05 ] . https://arxiv.org/abs/1612.03928 https://arxiv.org/abs/1612.03928 .

AHN S , HU S X , DAMIANOU A , et al . Variational information distillation for knowledge transfer [C ] // 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2019 : 9155 - 9163 .

TIAN Y L , KRISHNAN D , ISOLA P . Contrastive representation distillation [EB/OL ] . ( 2019-10-23 )[ 2023-9-16 ] . http://arxiv.org/abs/1910.10699 http://arxiv.org/abs/1910.10699 .

PARK W , KIM D , LU Y , et al . Relational knowledge distillation [C ] // IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2019 : 3962 - 3971 .

KIPF T N , WELLING M . Semi-supervised classification with graph convolutional networks [EB/OL ] . ( 2016-09-23 )[ 2023-09-06 ] . https://arxiv.org/abs/1609.02907 https://arxiv.org/abs/1609.02907 .

KRIZHEVSKY A . Learning multiple layers of features from tiny images [EB/OL ] . ( 2009-04-08 )[ 2023-10-08 ] . http://www.cs.utoronto.ca/~kriz/learning-features-2009-TR.pdf http://www.cs.utoronto.ca/~kriz/learning-features-2009-TR.pdf .

ZHAO B R , CUI Q , SONG R J , et al . Decoupled knowledge distillation [C ] // 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2022 : 11943 - 11952 .

COATES A , NG A Y , LEE H . An analysis of single-layer networks in unsupervised feature learning [C ] // International Conference on Artificial Intelligence and Statistics . Fort Lauderdale : JMLR.org , 2011 : 215 - 223 .

DENG J , DONG W , SOCHER R , et al . ImageNet: A large-scale hierarchical image database [C ] // 2009 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE Computer Society , 2009 : 248 - 255 .

FAN W Q , MA Y , LI Q , et al . Graph neural networks for social recommendation [C ] // The World Wide Web Conference . New York : ACM , 2019 : 417 - 426 .

THEKUMPARAMPIL K K , WANG C , OH S , et al . Attention-based graph neural network for semi-supervised learning [EB/OL ] . ( 2018-03-10 )[ 2023-10-08 ] . http://arxiv.org/abs/1803.03735 http://arxiv.org/abs/1803.03735 .

车向北 , 康文倩 , 邓彬 , 等 . 一种基于图神经网络的SDN路由性能预测模型 [J ] . 电子学报 , 2021 , 49 ( 3 ): 484 - 491 .

CHE X B , KANG W Q , DENG B , et al . A prediction model of SDN routing performance based on graph neural network [J ] . Acta Electronica Sinica , 2021 , 49 ( 3 ): 484 - 491 . (in Chinese)

WANG X L , GUPTA A . Videos as space-time region graphs [C ] // European Conference on Computer Vision (ECCV) . Cham : Springer , 2018 : 413 - 431 .

HU H Z , JI D Y , GAN W H , et al . Class-wise dynamic graph convolution for semantic segmentation [EB/OL ] . ( 2020-07-19 )[ 2023-10-08 ] . http://arxiv.org/abs/2007.09690 http://arxiv.org/abs/2007.09690 .

袁冠 , 邴睿 , 刘肖 , 等 . 基于时空图神经网络的手势识别‍ [J ] . 电子学报 , 2022 , 50 ( 4 ): 921 - 931 .

YUAN G , BING R , LIU X , et al . Spatial-temporal graph neural network based hand gesture recognition [J ] . Acta Electronica Sinica , 2022 , 50 ( 4 ): 921 - 931 . (in Chinese)

YANG C G , AN Z L , CAI L H , et al . Hierarchical self-supervised augmented knowledge distillation [C ] // Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence . California : International Joint Conferences on Artificial Intelligence Organization , 2021 : 1217 - 1223 .

YANG C G , AN Z L , CAI L H , et al . Mutual contrastive learning for visual representation learning [J ] . Proceedings of the AAAI Conference on Artificial Intelligence , 2022 , 36 ( 3 ): 3045 - 3053 .

YANG C G , ZHOU H L , AN Z L , et al . Cross-image relational knowledge distillation for semantic segmentation ‍ [C ] // 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2022 : 12309 - 12318 .

YANG T , ZHU S J , CHEN C , et al . Mutualnet: Adaptive convnet via mutual learning from network width and resolution [EB/OL ] . ( 2019-09-27 )[ 2023-10-10 ] . http://arxiv.org/abs/1909.12978 http://arxiv.org/abs/1909.12978 .

AGARAP A F . Deep learning using rectified linear units (relu) [EB/OL ] . ( 2018-03-22 )[ 2023-10-10 ] . http://arxiv.org/abs/1803.08375 http://arxiv.org/abs/1803.08375 .

FINN C , ABBEEL P , LEVINE S . Model-agnostic meta-learning for fast adaptation of deep networks [C ] // 34th International Conference on Machine Learning . New York : ACM , 2017 : 1126 - 1135 .

ZAGORUYKO S , KOMODAKIS N . Wide residual networks [C ] // British Machine Vision Conference 2016 . York : BMVA Press , 2016 : 1 - 15 .

SIMONYAN K , ZISSERMAN A . Very deep convolutional networks for large-scale image recognition [EB/OL ] . ( 2014-09-04 )[ 2023-10-12 ] . https://arxiv.org/abs/1409.1556 https://arxiv.org/abs/1409.1556 .

SANDLER M , HOWARD A , ZHU M , et al . Mobilenetv2: Inverted residuals and linear bottlenecks [C ] // 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE Computer Society , 2018 : 4510 - 4520 .

ZHANG X Y , ZHOU X Y , LIN M X , et al . Shufflenet: An extremely efficient convolutional neural network for mobile devices [C ] // 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE Computer Society , 2018 : 6848 - 6856 .

VAN DER MAATEN L , HINTON G . Visualizing data using t-SNE [J ] . Journal of Machine Learning Research , 2008 , 9 ( 11 ): 2579 - 2605 .

SELVARAJU R R , COGSWELL M , DAS A , et al . Grad-CAM: Visual explanations from deep networks via gradient-based localization [J ] . International Journal of Computer Vision , 2020 , 128 ( 2 ): 336 - 359 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于坐标重要性池化和解耦类别对齐蒸馏的图像分类算法

衰减解耦机制的图像分类网络

全局依赖引导的特征重构图像分类网络