Graph-Based Representation Knowledge Distillation for Image Classification

YANG Chuan-guang; CHEN Lu-ming; ZHAO Er-hu; AN Zhu-lin; XU Yong-jun

doi:10.12263/DZXB.20230976

您当前的位置：

首页 >

文章列表页 >

Graph-Based Representation Knowledge Distillation for Image Classification

PAPERS | 更新时间：2026-05-07

- Graph-Based Representation Knowledge Distillation for Image Classification
- ACTA ELECTRONICA SINICA Vol. 52, Issue 10, Pages: 3435-3447(2024)
- 作者机构：
  
  1.中国科学院计算技术研究所，北京 100190
  2.93114部队，北京 100080
- 作者简介：
- 基金信息：
  
  National Natural Science Foundation of China(62072434);Beijing Natural Science Foundation(4212027)
- DOI：10.12263/DZXB.20230976
  CLC： TP391.42;
- Received：19 October 2023，
  
  Revised：2023-12-19，
  
  Published：25 October 2024
- 稿件说明：
移动端阅览
杨传广, 陈路明, 赵二虎, 等. 基于图表征知识蒸馏的图像分类方法[J]. 电子学报, 2024, 52(10): 3435-3447.

YANG Chuan-guang, CHEN Lu-ming, ZHAO Er-hu, et al. Graph-Based Representation Knowledge Distillation for Image Classification[J]. Acta Electronica Sinica, 2024, 52(10): 3435-3447.
杨传广, 陈路明, 赵二虎, 等. 基于图表征知识蒸馏的图像分类方法[J]. 电子学报, 2024, 52(10): 3435-3447. DOI：10.12263/DZXB.20230976

YANG Chuan-guang, CHEN Lu-ming, ZHAO Er-hu, et al. Graph-Based Representation Knowledge Distillation for Image Classification[J]. Acta Electronica Sinica, 2024, 52(10): 3435-3447. DOI：10.12263/DZXB.20230976

摘要

知识蒸馏的核心思想是利用1个作为教师网络的大型模型来指导1个作为学生网络的小型模型，提升学生网络在图像分类任务上的性能.现有知识蒸馏方法通常从单一的输入样本中提取类别概率或特征信息作为知识，并没有对样本间关系进行建模，造成网络的表征学习能力下降.为解决此问题，本文引入图卷积神经网络，将输入样本集视为图结点构建关系图，图中的每个样本都可以聚合其他样本信息，提升样本的表征能力.本文从图结点和图关系2个角度构建图表征知识蒸馏误差，利用元学习引导学生网络自适应学习教师网络更佳的图表征，提升学生网络的图建模能力.相比于基线方法，本文提出的图表征知识蒸馏方法在加拿大高等研究院（Canadian Institute For Advanced Research，CIFAR）发布的100种分类数据集上提升了3.70%的分类准确率，表明本文方法引导学生网络学习到了更具有判别性的特征空间，提升了图像分类能力.

Abstract

The core idea of knowledge distillation is to use a large model as the teacher network to guide a small model as the student network

improving the performance of the student network in image classification tasks. Existing knowledge distillation methods often extract category probability or feature information as knowledge from a single input sample. They could not model the relationships between samples

decreasing the network’s representation learning ability. To solve this problem

this paper introduces a graph convolutional neural network

which treats the input sample set as graph nodes to construct a relationship graph. Each sample in the graph could aggregate information from other samples

improving its own representation ability. This paper constructs the distillation loss of graph representation knowledge from the perspectives of graph nodes and relationships. It uses meta-learning to guide the student network to adaptively learn better graph representations from a teacher network

thereby improving the graph modeling ability of the student network. Compared to the baseline method

the graph-based representation knowledge distillation method improves the classification accuracy by 3.70% on the 100-classification dataset published by Canadian Institute For Advanced Research. The result indicates that the proposed method makes the student network learn a more discriminative feature space

thereby improving its image classification ability.

关键词

Keywords

references

王波 , 黄冕 , 刘利军 , 等 . 基于多层聚焦Inception-V3卷积网络的细粒度图像分类 [J ] . 电子学报 , 2022 , 50 ( 1 ): 72 - 78 .

WANG B , HUANG M , LIU L J , et al . Multi-layer focused Inception-V3 models for fine-grained visual recognition [J ] . Acta Electronica Sinica , 2022 , 50 ( 1 ): 72 - 78 . (in Chinese)

HE K , ZHANG X , REN S , et al . Deep residual learning for image recognition [C ] // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE Computer Society , 2016 : 770 - 778 .

ZHU H , AN Z L , YANG C G , et al . EENA: Efficient evolution of neural architecture [C ] // 2019 International Conference on Computer Vision Workshop (ICCVW) . Piscataway : IEEE , 2019 : 1891 - 1899 .

YANG C G , AN Z L , LI C , et al . Multi-objective pruning for CNNs using genetic algorithm [C ] // International Conference on Artificial Neural Networks . Cham : Springer , 2019 : 299 - 305 .

郑云飞 , 王晓兵 , 张雄伟 , 等 . 基于金字塔知识的自蒸馏HRNet目标分割方法 [J ] . 电子学报 , 2023 , 51 ( 3 ): 746 - 756 .

ZHENG Y F , WANG X B , ZHANG X W , et al . The self-distillation HRNet object segmentation based on the pyramid knowledge [J ] . Acta Electronica Sinica , 2023 , 51 ( 3 ): 746 - 756 . (in Chinese)

HINTON G , VINYALS O , DEAN J . Distilling the knowledge in a neural network [EB/OL ] . ( 2015-03-05 )[ 2023-09-02 ] . https://arxiv.org/abs/1503.02531 https://arxiv.org/abs/1503.02531 .

ROMERO A , BALLAS N , KAHOU S E , et al . Fitnets: Hints for thin deep nets [EB/OL ] . ( 2014-12-19 )[ 2023-09-05 ] . http://arxiv.org/abs/1412.6550 http://arxiv.org/abs/1412.6550 .

ZAGORUYKO S , KOMODAKIS N . Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer [EB/OL ] . ( 2016-12-12 )[ 2023-09-05 ] . https://arxiv.org/abs/1612.03928 https://arxiv.org/abs/1612.03928 .

AHN S , HU S X , DAMIANOU A , et al . Variational information distillation for knowledge transfer [C ] // 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2019 : 9155 - 9163 .

TIAN Y L , KRISHNAN D , ISOLA P . Contrastive representation distillation [EB/OL ] . ( 2019-10-23 )[ 2023-9-16 ] . http://arxiv.org/abs/1910.10699 http://arxiv.org/abs/1910.10699 .

PARK W , KIM D , LU Y , et al . Relational knowledge distillation [C ] // IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2019 : 3962 - 3971 .

KIPF T N , WELLING M . Semi-supervised classification with graph convolutional networks [EB/OL ] . ( 2016-09-23 )[ 2023-09-06 ] . https://arxiv.org/abs/1609.02907 https://arxiv.org/abs/1609.02907 .

KRIZHEVSKY A . Learning multiple layers of features from tiny images [EB/OL ] . ( 2009-04-08 )[ 2023-10-08 ] . http://www.cs.utoronto.ca/~kriz/learning-features-2009-TR.pdf http://www.cs.utoronto.ca/~kriz/learning-features-2009-TR.pdf .

ZHAO B R , CUI Q , SONG R J , et al . Decoupled knowledge distillation [C ] // 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2022 : 11943 - 11952 .

COATES A , NG A Y , LEE H . An analysis of single-layer networks in unsupervised feature learning [C ] // International Conference on Artificial Intelligence and Statistics . Fort Lauderdale : JMLR.org , 2011 : 215 - 223 .

DENG J , DONG W , SOCHER R , et al . ImageNet: A large-scale hierarchical image database [C ] // 2009 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE Computer Society , 2009 : 248 - 255 .

FAN W Q , MA Y , LI Q , et al . Graph neural networks for social recommendation [C ] // The World Wide Web Conference . New York : ACM , 2019 : 417 - 426 .

THEKUMPARAMPIL K K , WANG C , OH S , et al . Attention-based graph neural network for semi-supervised learning [EB/OL ] . ( 2018-03-10 )[ 2023-10-08 ] . http://arxiv.org/abs/1803.03735 http://arxiv.org/abs/1803.03735 .

车向北 , 康文倩 , 邓彬 , 等 . 一种基于图神经网络的SDN路由性能预测模型 [J ] . 电子学报 , 2021 , 49 ( 3 ): 484 - 491 .

CHE X B , KANG W Q , DENG B , et al . A prediction model of SDN routing performance based on graph neural network [J ] . Acta Electronica Sinica , 2021 , 49 ( 3 ): 484 - 491 . (in Chinese)

WANG X L , GUPTA A . Videos as space-time region graphs [C ] // European Conference on Computer Vision (ECCV) . Cham : Springer , 2018 : 413 - 431 .

HU H Z , JI D Y , GAN W H , et al . Class-wise dynamic graph convolution for semantic segmentation [EB/OL ] . ( 2020-07-19 )[ 2023-10-08 ] . http://arxiv.org/abs/2007.09690 http://arxiv.org/abs/2007.09690 .

袁冠 , 邴睿 , 刘肖 , 等 . 基于时空图神经网络的手势识别‍ [J ] . 电子学报 , 2022 , 50 ( 4 ): 921 - 931 .

YUAN G , BING R , LIU X , et al . Spatial-temporal graph neural network based hand gesture recognition [J ] . Acta Electronica Sinica , 2022 , 50 ( 4 ): 921 - 931 . (in Chinese)

YANG C G , AN Z L , CAI L H , et al . Hierarchical self-supervised augmented knowledge distillation [C ] // Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence . California : International Joint Conferences on Artificial Intelligence Organization , 2021 : 1217 - 1223 .

YANG C G , AN Z L , CAI L H , et al . Mutual contrastive learning for visual representation learning [J ] . Proceedings of the AAAI Conference on Artificial Intelligence , 2022 , 36 ( 3 ): 3045 - 3053 .

YANG C G , ZHOU H L , AN Z L , et al . Cross-image relational knowledge distillation for semantic segmentation ‍ [C ] // 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2022 : 12309 - 12318 .

YANG T , ZHU S J , CHEN C , et al . Mutualnet: Adaptive convnet via mutual learning from network width and resolution [EB/OL ] . ( 2019-09-27 )[ 2023-10-10 ] . http://arxiv.org/abs/1909.12978 http://arxiv.org/abs/1909.12978 .

AGARAP A F . Deep learning using rectified linear units (relu) [EB/OL ] . ( 2018-03-22 )[ 2023-10-10 ] . http://arxiv.org/abs/1803.08375 http://arxiv.org/abs/1803.08375 .

FINN C , ABBEEL P , LEVINE S . Model-agnostic meta-learning for fast adaptation of deep networks [C ] // 34th International Conference on Machine Learning . New York : ACM , 2017 : 1126 - 1135 .

ZAGORUYKO S , KOMODAKIS N . Wide residual networks [C ] // British Machine Vision Conference 2016 . York : BMVA Press , 2016 : 1 - 15 .

SIMONYAN K , ZISSERMAN A . Very deep convolutional networks for large-scale image recognition [EB/OL ] . ( 2014-09-04 )[ 2023-10-12 ] . https://arxiv.org/abs/1409.1556 https://arxiv.org/abs/1409.1556 .

SANDLER M , HOWARD A , ZHU M , et al . Mobilenetv2: Inverted residuals and linear bottlenecks [C ] // 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE Computer Society , 2018 : 4510 - 4520 .

ZHANG X Y , ZHOU X Y , LIN M X , et al . Shufflenet: An extremely efficient convolutional neural network for mobile devices [C ] // 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE Computer Society , 2018 : 6848 - 6856 .

VAN DER MAATEN L , HINTON G . Visualizing data using t-SNE [J ] . Journal of Machine Learning Research , 2008 , 9 ( 11 ): 2579 - 2605 .

SELVARAJU R R , COGSWELL M , DAS A , et al . Grad-CAM: Visual explanations from deep networks via gradient-based localization [J ] . International Journal of Computer Vision , 2020 , 128 ( 2 ): 336 - 359 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Image Classification Algorithm Based on Coordinate Importance Pooling and Decoupled Class Alignment Distillation

Image Classification Network with Attenuation Disentangling Mechanism

Global Dependency Guided Feature Reconstruction for Image Classification

Related Author

LIU Ying

XUE Jia-hao

ZHANG Wei-dong

XU Zhi-jie

YUAN Heng

YANG Jizhen

ZHANG Shengchong

WU Jingrui

Related Institution

Center for Image and Information Processing, Xi’an University of Posts and Telecommunications

International Joint-Research Center for Wireless Communication and Information Processing

University of Huddersfield, West Yorkshire HD13DH, United Kingdom of Great Britain and Northern Ireland

School of Software, Liaoning Technical University

Key Laboratory of Optoelectronic Information Control and Security Technology

⁰