基于自适应层信息熵的卷积神经网络压缩

魏钰轩; 陈莹

doi:10.12263/DZXB.20201372

您当前的位置：

首页 >

文章列表页 >

基于自适应层信息熵的卷积神经网络压缩

学术论文 | 更新时间：2025-12-08

- 基于自适应层信息熵的卷积神经网络压缩
- Convolutional Neural Network Compression Based on Adaptive Layer Entropy
- 电子学报 2022年50卷第10期页码：2398-2408
- 作者机构：
  
  江南大学轻工过程先进控制教育部重点实验室，江苏无锡 214122
- 作者简介：
  
  [ "魏钰轩男，1997年10月出生，江苏徐州人.江南大学研究生.主要研究方向为图像处理、模型压缩." ]
  [ "陈莹（通讯作者）女，1976年12月出生，浙江丽水人.江南大学教授，博士生导师.主要研究方向为图像处理、信息融合、模式识别." ]
- 基金信息：
  
  国家自然科学基金(61573168)
- DOI：10.12263/DZXB.20201372
  中图分类号： TP391.41;
- 收稿：2020-12-01，
  
  修回：2021-01-29，
  
  纸质出版：2022-10-25
- 稿件说明：
移动端阅览
魏钰轩,陈莹.基于自适应层信息熵的卷积神经网络压缩[J].电子学报,2022,50(10):2398-2408.

WEI Yu-xuan,CHEN Ying.Convolutional Neural Network Compression Based on Adaptive Layer Entropy[J].ACTA ELECTRONICA SINICA,2022,50(10):2398-2408.
魏钰轩,陈莹.基于自适应层信息熵的卷积神经网络压缩[J].电子学报,2022,50(10):2398-2408. DOI： 10.12263/DZXB.20201372.

WEI Yu-xuan,CHEN Ying.Convolutional Neural Network Compression Based on Adaptive Layer Entropy[J].ACTA ELECTRONICA SINICA,2022,50(10):2398-2408. DOI： 10.12263/DZXB.20201372.

摘要

网络剪枝是一种有效的卷积神经网络压缩方法.多数现有压缩方法因迭代剪枝了“不重要”的网络结构，一方面破坏了网络结构的信息整体性，另一方面其迭代操作耗费了大量的计算资源与时间.为了解决上述问题，论文从网络结构全局考虑，提出基于自适应层信息熵的卷积神经网络压缩方法.首先，在获取压缩网络结构的过程中，本文设计了一种端到端的结构化网络剪枝方案，将卷积层看作一个整体，利用层信息熵之间的关联性直接确定各卷积层过滤器的保留率，避免迭代剪枝训练造成的信息损失.其次，对剪裁后的网络进行重训练时，综合考虑压缩过程中使用的层信息熵指标，通过对卷积层与批归一化（Batch Normalization，BN）层进行自适应联合嫁接，让网络学习到更多的信息，提升网络性能.针对3种主流网络在不同的数据集上进行了实验，验证了所提方法的有效性与优越性.例如在CIFAR-10上，针对ResNet-56，相比于基线网络，在计算量压缩36.2%的情况下，本文方法准确率提升了1%；针对ResNet-110，在计算量压缩52.4%的情况下，本文方法准确率提升了1.42%；针对轻量型网络MobileNetV2，在计算量压缩55.2%的情况下，本文方法准确率提升了1.29%.

Abstract

Network pruning has proven to be an effective approach to compress convolutional neural network (CNN). However

most existing CNN compression methods iteratively prune the "least important" filters

which not only destroys the information integrity of network structures

but also results in significant computation cost due to the iterative operation. To solve the problems

a convolutional neural network compression method based on adaptive layer entropy(ALE) is proposed

considering a global network structure. Firstly

an end-to-end network pruning strategy is designed

in which the retention rate of each convolutional layer filter is directly determined based on the entropy correlation between layers. The pruning strategy takes the convolutional layer as a whole

which decreases the information loss and computation cost of iterative pruning. Then

an adaptive joint grafting method

in which both convolutional and batch normalization(BN) layers are considered

is presented for the pruned network retraining to learn more information from the network. The layer entropies used in the compression are also utilized for the grafting. Experiments are conducted on different benchmarks and three popular networks

which demonstrate the efficiency and superiority of the proposed ALE over other methods. For the experiments on CIFAR-10

ALE achieves 36.2%

52.4% and 55.2% pruned rate in FLOPs for ResNet-56

ResNet-110 and MobileNetV2 while with increase of 1%

1.42%

1.29% accuracy respectively.

关键词

Keywords

references

ZHU F , ZHU L , YANG Y . Sim-real joint reinforcement transfer for 3d indoor navigation [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . California : IEEE , 2019 : 11388 - 11397 .

权宇 , 李志欣 , 张灿龙 , 等 . 融合深度扩张网络和轻量化网络的目标检测模型 [J]. 电子学报 , 2020 , 48 ( 2 ): 390 - 397 .

QUAN Y , LI Z X , ZHANG C L , et al . Fusing deep dilated convolutions network and light-weight network for object detection [J]. Acta Electronica Sinica , 2020 , 48 ( 2 ): 390 - 397 . (in Chinese)

周涛 , 霍兵强 , 陆惠玲 , 等 . 残差神经网络及其在医学图像处理中的应用研究 [J]. 电子学报 , 2020 , 48 ( 7 ): 1436 - 1447 .

ZHOU T , HUO B Q , LU H L , et al . Research on residual neural network and its application on medical image processing [J]. Acta Electronica Sinica , 2020 , 48 ( 7 ): 1436 - 1447 . (in Chinese)

DING X , DING G , GUO Y , et al . Centripetal sgd for pruning very deep convolutional networks with complicated structure [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . California : IEEE , 2019 : 4943 - 4953 .

饶川 , 陈靓影 , 徐如意 , 等 . 一种基于动态量化编码的深度神经网络压缩方法 [J]. 自动化学报 , 2019 , 45 ( 10 ): 1960 - 1968 .

RAO C , CHEN J Y , XU R Y , et al . A dynamic quantization coding based deep neural network compression method [J]. Acta Automatica Sinica , 2019 , 45 ( 10 ): 1960 - 1968 . (in Chinese)

ADRIANA R , NICOLAS B , SAMIRA E K , et al . Fitnets: Hints for thin deep nets [C]// International Conference on Learning Representations . California : OpenReview.net , 2015 : 1 - 13 .

ZHANG X , ZOU J , HE K , et al . Accelerating very deep convolutional networks for classification and detection [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2015 , 38 ( 10 ): 1943 - 1955 .

WANG Y , XU C , XU C , et al . Beyond filters: Compact feature map for portable deep model [C]// International Conference on Machine Learning . Sydney : ACM , 2017 : 3703 - 3711 .

HE K , ZHANG X , REN S , et al . Deep residual learning for image recognition [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Las Vegas : IEEE , 2016 : 770 - 778 .

SZEGEDY C , LIU W , JIA Y , et al . Going deeper with convolutions [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Boston : IEEE , 2015 : 1 - 9 .

CARREIRA-PERPINAN M A , IDELBAYEV Y . Learning-compression algorithms for neural net pruning [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Salt Lake City : IEEE , 2018 : 8532 - 8541 .

HAN S , POOL J , TRAN J , et al . Learning both weights and connections for efficient neural network [C]// Advances in Neural Information Processing Systems . Quebec : MIT Press , 2015 : 1135 - 1143 .

WEN W , WU C , WANG Y , et al . Learning structured sparsity in deep neural networks [C]// Advances in Neural Information Processing Systems . Barcelona : MIT Press , 2016 : 2074 - 2082 .

LUO J H , WU J , LIN W . Thinet: A filter level pruning method for deep neural network compression [C]// Proceedings of the IEEE International Conference on Computer Vision . Venice : IEEE , 2017 : 5058 - 5066 .

WANG D , ZHOU L , ZHANG X , et al . Exploring Linear Relationship in Feature Map Subspace for Convnets Compression [EB/OL]. ( 2018-03-15 )[ 2020-12-01 ]. https://arxiv.org/abs/1803.05729 https://arxiv.org/abs/1803.05729 .

LI H , KADAV A , DURDANOVIC I , et al . Pruning filters for efficient convnets [C]// International Conference on Learning Representations . Toulon : OpenReview.net , 2017 : 1 - 13 .

YE J , LU X , LIN Z , et al . Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers [C]// International Conference on Learning Representations . Vancouver : OpenReview.net , 2018 : 1 - 11 .

ZHUO H , QIAN X , FU Y , et al . Scsp: Spectral Clustering Filter Pruning with Soft Self-Adaption Manners [EB/OL]. ( 2018-07-14 )[ 2020-12-01 ]. https://arxiv.org/abs/1806.05320 https://arxiv.org/abs/1806.05320 .

HE Y , LIU P , WANG Z , et al . Filter pruning via geometric median for deep convolutional neural networks acceleration [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . California : IEEE , 2019 : 4340 - 4349 .

LIU Z , SUN M , ZHOU T , et al . Rethinking the value of network pruning [C]// International Conference on Learning Representations . New Orleans : OpenReview.net , 2019 : 1 - 21 .

LIN S , JI R , YAN C , et al . Towards optimal structured cnn pruning via generative adversarial learning [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . California : IEEE , 2019 : 2790 - 2799 .

LIN M , JI R , ZHANG Y , et al . Channel pruning via automatic structure search [C]// International Joint Conference on Artificial Intelligence . Yokohama : Morgan Kaufmann , 2020 : 1 - 7 .

LIU Z , ZHANG X , SHEN Z , et al . Joint Multi-Dimension Pruning [EB/OL]. ( 2020-05-18 )[ 2020-12-01 ]. https://arxiv.org/abs/2005.08931 https://arxiv.org/abs/2005.08931 .

MENG F , CHENG H , LI K , et al . Filter grafting for deep neural networks [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle : IEEE , 2020 : 6599 - 6607 .

LUO J H , WU J . An Entropy-Based Pruning Method for CNN Compression [EB/OL]. ( 2017-07-19 )[ 2020-12-01 ]. https://arxiv.org/abs/1706.05791 https://arxiv.org/abs/1706.05791 .

LI Y , LIN S , ZHANG B , et al . Exploiting kernel sparsity and entropy for interpretable CNN compression [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . California : IEEE , 2019 : 2800 - 2809 .

WANG L , XIONG Y , WANG Z , et al . Temporal segment networks: Towards good practices for deep action recognition [C]// Proceedings of the European Conference on Computer Vision , Amsterdam . Holland : Springer , 2016 : 20 - 36 .

KRIZHEVSKY A , HINTON G . Learning multiple layers of features from tiny images [J]. Technical Report , 2009 , 1 ( 1 ): 1 - 60 .

NETZER Y , WANG T , COATES A , et al . Reading digits in natural images with unsupervised feature learning [J]. NIPS Workshop on Deep Learning and Unsupervised Feature Learning , 2011 , 1 ( 1 ): 1 - 9 .

ZHENG L , SHEN L , TIAN L , et al . Scalable person re-identification: A benchmark [C]// Proceedings of the IEEE International Conference on Computer Vision . Santiago : IEEE , 2015 : 1116 - 1124 .

RISTANI E , SOLERA F , ZOU R , et al . Performance measures and a data set for multi-target, multi-camera tracking [C]// Proceedings of the European Conference on Computer Vision . Amsterdam : Springer , 2016 : 17 - 35 .

HUANG Z , WANG N . Data-driven sparse structure selection for deep neural networks [C]// Proceedings of the European Conference on Computer Vision . Munich : Springer , 2018 : 304 - 320 .

LIN M , JI R , WANG Y , et al . HRank: Filter pruning using high-rank feature map [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle : IEEE , 2020 : 1529 - 1538 .

CHIN T W , DING R , ZHANG C , et al . Towards efficient model compression via learned global ranking [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle : IEEE , 2020 : 1518 - 1528 .

YU R , LI A , CHEN C F , et al . Nisp: Pruning networks using neuron importance score propagation [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Salt Lake City : IEEE , 2018 : 9194 - 9203 .

LI Y , GU S , MAYER C , et al . Group sparsity: The hinge between filter pruning and decomposition for network compression [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle : IEEE , 2020 : 8018 - 8027 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于轻量级全连接张量映射网络的高光谱图像分类方法

基于非一般类算子融合方法及硬件架构设计

端智能推理加速技术综述

基于低秩自适应的伸缩感知蒸馏方法