基于自适应层信息熵的卷积神经网络压缩

魏钰轩; 陈莹

doi:10.12263/DZXB.20201372

PDF(1349 KB)

电子学报 ›› 2022, Vol. 50 ›› Issue (10) : 2398-2408. DOI: 10.12263/DZXB.20201372

学术论文

基于自适应层信息熵的卷积神经网络压缩

作者信息 +

Convolutional Neural Network Compression Based on Adaptive Layer Entropy

Author information +

文章历史 +

本文亮点

网络剪枝是一种有效的卷积神经网络压缩方法.多数现有压缩方法因迭代剪枝了“不重要”的网络结构，一方面破坏了网络结构的信息整体性，另一方面其迭代操作耗费了大量的计算资源与时间.为了解决上述问题，论文从网络结构全局考虑，提出基于自适应层信息熵的卷积神经网络压缩方法.首先，在获取压缩网络结构的过程中，本文设计了一种端到端的结构化网络剪枝方案，将卷积层看作一个整体，利用层信息熵之间的关联性直接确定各卷积层过滤器的保留率，避免迭代剪枝训练造成的信息损失.其次，对剪裁后的网络进行重训练时，综合考虑压缩过程中使用的层信息熵指标，通过对卷积层与批归一化（Batch Normalization，BN）层进行自适应联合嫁接，让网络学习到更多的信息，提升网络性能.针对3种主流网络在不同的数据集上进行了实验，验证了所提方法的有效性与优越性.例如在CIFAR-10上，针对ResNet-56，相比于基线网络，在计算量压缩36.2%的情况下，本文方法准确率提升了1%；针对ResNet-110，在计算量压缩52.4%的情况下，本文方法准确率提升了1.42%；针对轻量型网络MobileNetV2，在计算量压缩55.2%的情况下，本文方法准确率提升了1.29%.

HeighLight

Network pruning has proven to be an effective approach to compress convolutional neural network (CNN). However, most existing CNN compression methods iteratively prune the "least important" filters, which not only destroys the information integrity of network structures, but also results in significant computation cost due to the iterative operation. To solve the problems, a convolutional neural network compression method based on adaptive layer entropy(ALE) is proposed, considering a global network structure. Firstly, an end-to-end network pruning strategy is designed, in which the retention rate of each convolutional layer filter is directly determined based on the entropy correlation between layers. The pruning strategy takes the convolutional layer as a whole, which decreases the information loss and computation cost of iterative pruning. Then, an adaptive joint grafting method, in which both convolutional and batch normalization(BN) layers are considered, is presented for the pruned network retraining to learn more information from the network. The layer entropies used in the compression are also utilized for the grafting. Experiments are conducted on different benchmarks and three popular networks, which demonstrate the efficiency and superiority of the proposed ALE over other methods. For the experiments on CIFAR-10, ALE achieves 36.2%, 52.4% and 55.2% pruned rate in FLOPs for ResNet-56, ResNet-110 and MobileNetV2 while with increase of 1%, 1.42%, 1.29% accuracy respectively.

导出引用

魏钰轩 , 陈莹. 基于自适应层信息熵的卷积神经网络压缩[J]. 电子学报, 2022, 50(10): 2398-2408. https://doi.org/10.12263/DZXB.20201372

WEI Yu-xuan , CHEN Ying. Convolutional Neural Network Compression Based on Adaptive Layer Entropy[J]. Acta Electronica Sinica, 2022, 50(10): 2398-2408. https://doi.org/10.12263/DZXB.20201372

中图分类号： TP391.41

参考文献

原文顺序 | 文献年度倒序 | 文中引用次数倒序

1	ZHUF, ZHUL, YANGY. Sim-real joint reinforcement transfer for 3d indoor navigation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. California: IEEE, 2019: 11388-11397. 本文引用 [1]

2	权宇, 李志欣, 张灿龙, 等. 融合深度扩张网络和轻量化网络的目标检测模型[J]. 电子学报, 2020, 48(2): 390-397. QUANY, LIZ X, ZHANGC L, et al. Fusing deep dilated convolutions network and light-weight network for object detection[J]. Acta Electronica Sinica, 2020, 48(2): 390-397. (in Chinese)

3	周涛, 霍兵强, 陆惠玲, 等. 残差神经网络及其在医学图像处理中的应用研究[J]. 电子学报, 2020, 48(7): 1436-1447. ZHOUT, HUOB Q, LUH L, et al. Research on residual neural network and its application on medical image processing[J]. Acta Electronica Sinica, 2020, 48(7): 1436-1447. (in Chinese) 本文引用 [1]

4	DINGX, DINGG, GUOY, et al. Centripetal sgd for pruning very deep convolutional networks with complicated structure[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. California: IEEE, 2019: 4943-4953. 本文引用 [3]

5	饶川, 陈靓影, 徐如意, 等. 一种基于动态量化编码的深度神经网络压缩方法[J]. 自动化学报, 2019, 45(10):1960-1968. RAOC, CHENJ Y, XUR Y, et al. A dynamic quantization coding based deep neural network compression method[J]. Acta Automatica Sinica, 2019, 45(10): 1960-1968. (in Chinese) 本文引用 [1]

6	ADRIANAR, NICOLASB, SAMIRAE K, et al. Fitnets: Hints for thin deep nets[C]//International Conference on Learning Representations. California: OpenReview.net, 2015: 1-13. 本文引用 [1]

7	ZHANGX, ZOUJ, HEK, et al. Accelerating very deep convolutional networks for classification and detection[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 38(10): 1943-1955. 本文引用 [1]

8	WANGY, XUC, XUC, et al. Beyond filters: Compact feature map for portable deep model[C]//International Conference on Machine Learning. Sydney: ACM, 2017: 3703-3711. 本文引用 [1]

9	HEK, ZHANGX, RENS, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778. 本文引用 [1]

10	SZEGEDYC, LIUW, JIAY, et al. Going deeper with convolutions[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 1-9. 本文引用 [1]

11	CARREIRA-PERPINANM A, IDELBAYEVY. Learning-compression algorithms for neural net pruning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 8532-8541. 本文引用 [1]

12	HANS, POOLJ, TRANJ, et al. Learning both weights and connections for efficient neural network[C]//Advances in Neural Information Processing Systems. Quebec: MIT Press, 2015: 1135-1143. 本文引用 [1]

13	WENW, WUC, WANGY, et al. Learning structured sparsity in deep neural networks[C]//Advances in Neural Information Processing Systems. Barcelona: MIT Press, 2016: 2074-2082. 本文引用 [1]

14	LUOJ H, WUJ, LINW. Thinet: A filter level pruning method for deep neural network compression[C]//Proceedings of the IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 5058-5066. 本文引用 [1]

15	WANGD, ZHOUL, ZHANGX, et al. Exploring Linear Relationship in Feature Map Subspace for Convnets Compression[EB/OL]. (2018-03-15)[2020-12-01]. https://arxiv.org/abs/1803.05729 本文引用 [1]

16	LIH, KADAVA, DURDANOVICI, et al. Pruning filters for efficient convnets[C]//International Conference on Learning Representations. Toulon: OpenReview.net, 2017: 1-13. 本文引用 [6]

17	YEJ, LUX, LINZ, et al. Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers[C]//International Conference on Learning Representations. Vancouver: OpenReview.net, 2018: 1-11. 本文引用 [2]

18	ZHUOH, QIANX, FUY, et al. Scsp: Spectral Clustering Filter Pruning with Soft Self-Adaption Manners[EB/OL]. (2018-07-14)[2020-12-01]. https://arxiv.org/abs/1806.05320 本文引用 [1]

19	HEY, LIUP, WANGZ, et al. Filter pruning via geometric median for deep convolutional neural networks acceleration[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. California: IEEE, 2019: 4340-4349. 本文引用 [4]

20	LIUZ, SUNM, ZHOUT, et al. Rethinking the value of network pruning[C]//International Conference on Learning Representations. New Orleans: OpenReview.net, 2019: 1-21. 本文引用 [1]

21	LINS, JIR, YANC, et al. Towards optimal structured cnn pruning via generative adversarial learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. California: IEEE, 2019: 2790-2799. 本文引用 [11]

22	LINM, JIR, ZHANGY, et al. Channel pruning via automatic structure search[C]//International Joint Conference on Artificial Intelligence. Yokohama: Morgan Kaufmann, 2020: 1-7. 本文引用 [5]

23	LIUZ, ZHANGX, SHENZ, et al. Joint Multi-Dimension Pruning[EB/OL]. (2020-05-18)[2020-12-01]. https://arxiv.org/abs/2005.08931 本文引用 [1]

24	MENGF, CHENGH, LIK, et al. Filter grafting for deep neural networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 6599-6607. 本文引用 [2]

25	LUOJ H, WUJ. An Entropy-Based Pruning Method for CNN Compression[EB/OL]. (2017-07-19)[2020-12-01]. https://arxiv.org/abs/1706.05791 本文引用 [1]

26	LIY, LINS, ZHANGB, et al. Exploiting kernel sparsity and entropy for interpretable CNN compression[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. California: IEEE, 2019: 2800-2809. 本文引用 [3]

27	WANGL, XIONGY, WANGZ, et al. Temporal segment networks: Towards good practices for deep action recognition[C]//Proceedings of the European Conference on Computer Vision, Amsterdam. Holland: Springer, 2016: 20-36. 本文引用 [1]

28	KRIZHEVSKYA, HINTONG. Learning multiple layers of features from tiny images[J]. Technical Report, 2009, 1(1): 1-60. 本文引用 [2]

29	NETZERY, WANGT, COATESA, et al. Reading digits in natural images with unsupervised feature learning[J]. NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011, 1(1): 1-9. 本文引用 [1]

30	ZHENGL, SHENL, TIANL, et al. Scalable person re-identification: A benchmark[C]//Proceedings of the IEEE International Conference on Computer Vision. Santiago: IEEE, 2015: 1116-1124. 本文引用 [1]

31	RISTANIE, SOLERAF, ZOUR, et al. Performance measures and a data set for multi-target, multi-camera tracking[C]//Proceedings of the European Conference on Computer Vision. Amsterdam: Springer, 2016: 17-35. 本文引用 [1]

32	HUANGZ, WANGN. Data-driven sparse structure selection for deep neural networks[C]//Proceedings of the European Conference on Computer Vision. Munich: Springer, 2018: 304-320. 本文引用 [3]

33	LINM, JIR, WANGY, et al. HRank: Filter pruning using high-rank feature map[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 1529-1538. 本文引用 [6]

34	CHINT W, DINGR, ZHANGC, et al. Towards efficient model compression via learned global ranking[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 1518-1528. 本文引用 [4]

35	YUR, LIA, CHENC F, et al. Nisp: Pruning networks using neuron importance score propagation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 9194-9203. 本文引用 [3]

36	LIY, GUS, MAYERC, et al. Group sparsity: The hinge between filter pruning and decomposition for network compression[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 8018-8027. 本文引用 [2]