Convolutional Neural Network Compression Based on Adaptive Layer Entropy

WEI Yu-xuan; CHEN Ying

doi:10.12263/DZXB.20201372

您当前的位置：

首页 >

文章列表页 >

Convolutional Neural Network Compression Based on Adaptive Layer Entropy

PAPERS | 更新时间：2025-12-08

- Convolutional Neural Network Compression Based on Adaptive Layer Entropy
- ACTA ELECTRONICA SINICA Vol. 50, Issue 10, Pages: 2398-2408(2022)
- 作者机构：
  
  江南大学轻工过程先进控制教育部重点实验室，江苏无锡 214122
- 作者简介：
- 基金信息：
- DOI：10.12263/DZXB.20201372
  CLC： TP391.41;
- Received：01 December 2020，
  
  Revised：2021-01-29，
  
  Published：25 October 2022
- 稿件说明：
移动端阅览
魏钰轩,陈莹.基于自适应层信息熵的卷积神经网络压缩[J].电子学报,2022,50(10):2398-2408.

WEI Yu-xuan,CHEN Ying.Convolutional Neural Network Compression Based on Adaptive Layer Entropy[J].ACTA ELECTRONICA SINICA,2022,50(10):2398-2408.
魏钰轩,陈莹.基于自适应层信息熵的卷积神经网络压缩[J].电子学报,2022,50(10):2398-2408. DOI： 10.12263/DZXB.20201372.

WEI Yu-xuan,CHEN Ying.Convolutional Neural Network Compression Based on Adaptive Layer Entropy[J].ACTA ELECTRONICA SINICA,2022,50(10):2398-2408. DOI： 10.12263/DZXB.20201372.

摘要

网络剪枝是一种有效的卷积神经网络压缩方法.多数现有压缩方法因迭代剪枝了“不重要”的网络结构，一方面破坏了网络结构的信息整体性，另一方面其迭代操作耗费了大量的计算资源与时间.为了解决上述问题，论文从网络结构全局考虑，提出基于自适应层信息熵的卷积神经网络压缩方法.首先，在获取压缩网络结构的过程中，本文设计了一种端到端的结构化网络剪枝方案，将卷积层看作一个整体，利用层信息熵之间的关联性直接确定各卷积层过滤器的保留率，避免迭代剪枝训练造成的信息损失.其次，对剪裁后的网络进行重训练时，综合考虑压缩过程中使用的层信息熵指标，通过对卷积层与批归一化（Batch Normalization，BN）层进行自适应联合嫁接，让网络学习到更多的信息，提升网络性能.针对3种主流网络在不同的数据集上进行了实验，验证了所提方法的有效性与优越性.例如在CIFAR-10上，针对ResNet-56，相比于基线网络，在计算量压缩36.2%的情况下，本文方法准确率提升了1%；针对ResNet-110，在计算量压缩52.4%的情况下，本文方法准确率提升了1.42%；针对轻量型网络MobileNetV2，在计算量压缩55.2%的情况下，本文方法准确率提升了1.29%.

Abstract

Network pruning has proven to be an effective approach to compress convolutional neural network (CNN). However

most existing CNN compression methods iteratively prune the "least important" filters

which not only destroys the information integrity of network structures

but also results in significant computation cost due to the iterative operation. To solve the problems

a convolutional neural network compression method based on adaptive layer entropy(ALE) is proposed

considering a global network structure. Firstly

an end-to-end network pruning strategy is designed

in which the retention rate of each convolutional layer filter is directly determined based on the entropy correlation between layers. The pruning strategy takes the convolutional layer as a whole

which decreases the information loss and computation cost of iterative pruning. Then

an adaptive joint grafting method

in which both convolutional and batch normalization(BN) layers are considered

is presented for the pruned network retraining to learn more information from the network. The layer entropies used in the compression are also utilized for the grafting. Experiments are conducted on different benchmarks and three popular networks

which demonstrate the efficiency and superiority of the proposed ALE over other methods. For the experiments on CIFAR-10

ALE achieves 36.2%

52.4% and 55.2% pruned rate in FLOPs for ResNet-56

ResNet-110 and MobileNetV2 while with increase of 1%

1.42%

1.29% accuracy respectively.

关键词

Keywords

references

ZHU F , ZHU L , YANG Y . Sim-real joint reinforcement transfer for 3d indoor navigation [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . California : IEEE , 2019 : 11388 - 11397 .

权宇 , 李志欣 , 张灿龙 , 等 . 融合深度扩张网络和轻量化网络的目标检测模型 [J]. 电子学报 , 2020 , 48 ( 2 ): 390 - 397 .

QUAN Y , LI Z X , ZHANG C L , et al . Fusing deep dilated convolutions network and light-weight network for object detection [J]. Acta Electronica Sinica , 2020 , 48 ( 2 ): 390 - 397 . (in Chinese)

周涛 , 霍兵强 , 陆惠玲 , 等 . 残差神经网络及其在医学图像处理中的应用研究 [J]. 电子学报 , 2020 , 48 ( 7 ): 1436 - 1447 .

ZHOU T , HUO B Q , LU H L , et al . Research on residual neural network and its application on medical image processing [J]. Acta Electronica Sinica , 2020 , 48 ( 7 ): 1436 - 1447 . (in Chinese)

DING X , DING G , GUO Y , et al . Centripetal sgd for pruning very deep convolutional networks with complicated structure [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . California : IEEE , 2019 : 4943 - 4953 .

饶川 , 陈靓影 , 徐如意 , 等 . 一种基于动态量化编码的深度神经网络压缩方法 [J]. 自动化学报 , 2019 , 45 ( 10 ): 1960 - 1968 .

RAO C , CHEN J Y , XU R Y , et al . A dynamic quantization coding based deep neural network compression method [J]. Acta Automatica Sinica , 2019 , 45 ( 10 ): 1960 - 1968 . (in Chinese)

ADRIANA R , NICOLAS B , SAMIRA E K , et al . Fitnets: Hints for thin deep nets [C]// International Conference on Learning Representations . California : OpenReview.net , 2015 : 1 - 13 .

ZHANG X , ZOU J , HE K , et al . Accelerating very deep convolutional networks for classification and detection [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2015 , 38 ( 10 ): 1943 - 1955 .

WANG Y , XU C , XU C , et al . Beyond filters: Compact feature map for portable deep model [C]// International Conference on Machine Learning . Sydney : ACM , 2017 : 3703 - 3711 .

HE K , ZHANG X , REN S , et al . Deep residual learning for image recognition [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Las Vegas : IEEE , 2016 : 770 - 778 .

SZEGEDY C , LIU W , JIA Y , et al . Going deeper with convolutions [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Boston : IEEE , 2015 : 1 - 9 .

CARREIRA-PERPINAN M A , IDELBAYEV Y . Learning-compression algorithms for neural net pruning [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Salt Lake City : IEEE , 2018 : 8532 - 8541 .

HAN S , POOL J , TRAN J , et al . Learning both weights and connections for efficient neural network [C]// Advances in Neural Information Processing Systems . Quebec : MIT Press , 2015 : 1135 - 1143 .

WEN W , WU C , WANG Y , et al . Learning structured sparsity in deep neural networks [C]// Advances in Neural Information Processing Systems . Barcelona : MIT Press , 2016 : 2074 - 2082 .

LUO J H , WU J , LIN W . Thinet: A filter level pruning method for deep neural network compression [C]// Proceedings of the IEEE International Conference on Computer Vision . Venice : IEEE , 2017 : 5058 - 5066 .

WANG D , ZHOU L , ZHANG X , et al . Exploring Linear Relationship in Feature Map Subspace for Convnets Compression [EB/OL]. ( 2018-03-15 )[ 2020-12-01 ]. https://arxiv.org/abs/1803.05729 https://arxiv.org/abs/1803.05729 .

LI H , KADAV A , DURDANOVIC I , et al . Pruning filters for efficient convnets [C]// International Conference on Learning Representations . Toulon : OpenReview.net , 2017 : 1 - 13 .

YE J , LU X , LIN Z , et al . Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers [C]// International Conference on Learning Representations . Vancouver : OpenReview.net , 2018 : 1 - 11 .

ZHUO H , QIAN X , FU Y , et al . Scsp: Spectral Clustering Filter Pruning with Soft Self-Adaption Manners [EB/OL]. ( 2018-07-14 )[ 2020-12-01 ]. https://arxiv.org/abs/1806.05320 https://arxiv.org/abs/1806.05320 .

HE Y , LIU P , WANG Z , et al . Filter pruning via geometric median for deep convolutional neural networks acceleration [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . California : IEEE , 2019 : 4340 - 4349 .

LIU Z , SUN M , ZHOU T , et al . Rethinking the value of network pruning [C]// International Conference on Learning Representations . New Orleans : OpenReview.net , 2019 : 1 - 21 .

LIN S , JI R , YAN C , et al . Towards optimal structured cnn pruning via generative adversarial learning [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . California : IEEE , 2019 : 2790 - 2799 .

LIN M , JI R , ZHANG Y , et al . Channel pruning via automatic structure search [C]// International Joint Conference on Artificial Intelligence . Yokohama : Morgan Kaufmann , 2020 : 1 - 7 .

LIU Z , ZHANG X , SHEN Z , et al . Joint Multi-Dimension Pruning [EB/OL]. ( 2020-05-18 )[ 2020-12-01 ]. https://arxiv.org/abs/2005.08931 https://arxiv.org/abs/2005.08931 .

MENG F , CHENG H , LI K , et al . Filter grafting for deep neural networks [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle : IEEE , 2020 : 6599 - 6607 .

LUO J H , WU J . An Entropy-Based Pruning Method for CNN Compression [EB/OL]. ( 2017-07-19 )[ 2020-12-01 ]. https://arxiv.org/abs/1706.05791 https://arxiv.org/abs/1706.05791 .

LI Y , LIN S , ZHANG B , et al . Exploiting kernel sparsity and entropy for interpretable CNN compression [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . California : IEEE , 2019 : 2800 - 2809 .

WANG L , XIONG Y , WANG Z , et al . Temporal segment networks: Towards good practices for deep action recognition [C]// Proceedings of the European Conference on Computer Vision , Amsterdam . Holland : Springer , 2016 : 20 - 36 .

KRIZHEVSKY A , HINTON G . Learning multiple layers of features from tiny images [J]. Technical Report , 2009 , 1 ( 1 ): 1 - 60 .

NETZER Y , WANG T , COATES A , et al . Reading digits in natural images with unsupervised feature learning [J]. NIPS Workshop on Deep Learning and Unsupervised Feature Learning , 2011 , 1 ( 1 ): 1 - 9 .

ZHENG L , SHEN L , TIAN L , et al . Scalable person re-identification: A benchmark [C]// Proceedings of the IEEE International Conference on Computer Vision . Santiago : IEEE , 2015 : 1116 - 1124 .

RISTANI E , SOLERA F , ZOU R , et al . Performance measures and a data set for multi-target, multi-camera tracking [C]// Proceedings of the European Conference on Computer Vision . Amsterdam : Springer , 2016 : 17 - 35 .

HUANG Z , WANG N . Data-driven sparse structure selection for deep neural networks [C]// Proceedings of the European Conference on Computer Vision . Munich : Springer , 2018 : 304 - 320 .

LIN M , JI R , WANG Y , et al . HRank: Filter pruning using high-rank feature map [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle : IEEE , 2020 : 1529 - 1538 .

CHIN T W , DING R , ZHANG C , et al . Towards efficient model compression via learned global ranking [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle : IEEE , 2020 : 1518 - 1528 .

YU R , LI A , CHEN C F , et al . Nisp: Pruning networks using neuron importance score propagation [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Salt Lake City : IEEE , 2018 : 9194 - 9203 .

LI Y , GU S , MAYER C , et al . Group sparsity: The hinge between filter pruning and decomposition for network compression [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle : IEEE , 2020 : 8018 - 8027 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Lightweight Fully-Connected Tensorial Mapping Network for Hyperspectral Image Classification

Operator Fusion Method and Hardware Architecture Design Based on Non-Standard Operators

On-Device Intelligence Acceleration Technologies: A Survey

Low-Rank Adaptation Based Flexibility-Aware Distillation Method

Related Author

LIN Zhi-xin

ZHENG Yu-bang

MA Tian-yu

WANG Rui

LI Heng-chao

WANG Ying

GAO Lan

ZHANG Zhe

Related Institution

School of Information Science and Technology, Southwest Jiaotong University

College of Information Engineering, Capital Normal University

School of Mathematical Science, Capital Normal University

Faculty of Software Technologics, Shanxi Agricultural University

Department of Computer Science and Technology, Tsinghua University

⁰