电子学报 ›› 2022, Vol. 50 ›› Issue (10): 2398-2408.DOI: 10.12263/DZXB.20201372

• 学术论文 • 上一篇    下一篇

基于自适应层信息熵的卷积神经网络压缩

魏钰轩, 陈莹()   

  1. 江南大学轻工过程先进控制教育部重点实验室,江苏 无锡 214122
  • 收稿日期:2020-12-01 修回日期:2021-01-29 出版日期:2022-10-25
    • 通讯作者:
    • 陈莹
    • 作者简介:
    • 魏钰轩 男,1997年10月出生,江苏徐州人.江南大学研究生.主要研究方向为图像处理、模型压缩.
      陈 莹(通讯作者) 女,1976年12月出生,浙江丽水人.江南大学教授,博士生导师.主要研究方向为图像处理、信息融合、模式识别.
    • 基金资助:
    • 国家自然科学基金 (61573168)

Convolutional Neural Network Compression Based on Adaptive Layer Entropy

WEI Yu-xuan, CHEN Ying()   

  1. The Key Laboratory of Advanced Process Control for Light Industry (Ministry of Education),Jiangnan University,Wuxi,Jiangsu 214122,China
  • Received:2020-12-01 Revised:2021-01-29 Online:2022-10-25 Published:2022-10-11
    • Corresponding author:
    • CHEN Ying

摘要:

网络剪枝是一种有效的卷积神经网络压缩方法.多数现有压缩方法因迭代剪枝了“不重要”的网络结构,一方面破坏了网络结构的信息整体性,另一方面其迭代操作耗费了大量的计算资源与时间.为了解决上述问题,论文从网络结构全局考虑,提出基于自适应层信息熵的卷积神经网络压缩方法.首先,在获取压缩网络结构的过程中,本文设计了一种端到端的结构化网络剪枝方案,将卷积层看作一个整体,利用层信息熵之间的关联性直接确定各卷积层过滤器的保留率,避免迭代剪枝训练造成的信息损失.其次,对剪裁后的网络进行重训练时,综合考虑压缩过程中使用的层信息熵指标,通过对卷积层与批归一化(Batch Normalization,BN)层进行自适应联合嫁接,让网络学习到更多的信息,提升网络性能.针对3种主流网络在不同的数据集上进行了实验,验证了所提方法的有效性与优越性.例如在CIFAR-10上,针对ResNet-56,相比于基线网络,在计算量压缩36.2%的情况下,本文方法准确率提升了1%;针对ResNet-110,在计算量压缩52.4%的情况下,本文方法准确率提升了1.42%;针对轻量型网络MobileNetV2,在计算量压缩55.2%的情况下,本文方法准确率提升了1.29%.

关键词: 卷积神经网络, 网络剪枝, 信息熵, 嫁接, 模型压缩

Abstract:

Network pruning has proven to be an effective approach to compress convolutional neural network (CNN). However, most existing CNN compression methods iteratively prune the "least important" filters, which not only destroys the information integrity of network structures, but also results in significant computation cost due to the iterative operation. To solve the problems, a convolutional neural network compression method based on adaptive layer entropy(ALE) is proposed, considering a global network structure. Firstly, an end-to-end network pruning strategy is designed, in which the retention rate of each convolutional layer filter is directly determined based on the entropy correlation between layers. The pruning strategy takes the convolutional layer as a whole, which decreases the information loss and computation cost of iterative pruning. Then, an adaptive joint grafting method, in which both convolutional and batch normalization(BN) layers are considered, is presented for the pruned network retraining to learn more information from the network. The layer entropies used in the compression are also utilized for the grafting. Experiments are conducted on different benchmarks and three popular networks, which demonstrate the efficiency and superiority of the proposed ALE over other methods. For the experiments on CIFAR-10, ALE achieves 36.2%, 52.4% and 55.2% pruned rate in FLOPs for ResNet-56, ResNet-110 and MobileNetV2 while with increase of 1%, 1.42%, 1.29% accuracy respectively.

Key words: convolutional neural network, network pruning, entropy, grafting, model compression

中图分类号: