Research on Multi-Granularity Neural Network Pruning Method with Regularization Mechanism

LIU Qi; CHEN Ying

doi:10.12263/DZXB.20210844

您当前的位置：

首页 >

文章列表页 >

Research on Multi-Granularity Neural Network Pruning Method with Regularization Mechanism

PAPERS | 更新时间：2025-12-08

- Research on Multi-Granularity Neural Network Pruning Method with Regularization Mechanism
- ACTA ELECTRONICA SINICA Vol. 51, Issue 8, Pages: 2202-2212(2023)
- 作者机构：
  
  江南大学轻工过程先进控制教育部重点实验室，江苏无锡 214122
- 作者简介：
- 基金信息：
  
  National Natural Science Foundation of China(62173160)
- DOI：10.12263/DZXB.20210844
  CLC： TP391.41;
- Received：05 July 2021，
  
  Revised：2021-12-10，
  
  Published：25 August 2023
- 稿件说明：
移动端阅览
刘奇,陈莹.正则化机制下多粒度神经网络剪枝方法研究[J].电子学报,2023,51(08):2202-2212.

LIU Qi,CHEN Ying.Research on Multi-Granularity Neural Network Pruning Method with Regularization Mechanism[J].ACTA ELECTRONICA SINICA,2023,51(08):2202-2212.
刘奇,陈莹.正则化机制下多粒度神经网络剪枝方法研究[J].电子学报,2023,51(08):2202-2212. DOI： 10.12263/DZXB.20210844.

LIU Qi,CHEN Ying.Research on Multi-Granularity Neural Network Pruning Method with Regularization Mechanism[J].ACTA ELECTRONICA SINICA,2023,51(08):2202-2212. DOI： 10.12263/DZXB.20210844.

摘要

目前流行的模型压缩剪枝算法裁减的对象通常是整个卷积核.一些网络结构中存在特征图维度匹配的硬性要求，如ResNet中的残差结构主干上最后一个卷积层的卷积核个数以及Inception网络中的级联操作前所有分支上最后一个卷积层的卷积核个数都不能改变，这直接限定了剪枝的空间.本文提出一种正则化机制下的多粒度神经网络剪枝方法，针对维度匹配限制了剪枝空间的问题，设计从粗到细的多粒度剪枝策略，在稀疏化的同时维持了处于维度匹配位置的卷积层中卷积核的数量不变.并且，本文提出一种自适应L1正则化的稀疏方式，可以使网络在更新参数的同时兼顾到网络结构的变化.稀疏化后的卷积核不仅有比原卷积核更少的参数和计算量，而且拥有更加优异的结构性质，使网络具有更高的表达能力.例如，在CIFAR-10上，针对VGG-16，相比基准网络，在计算量压缩了76.73%的情况下，准确率提高了0.19%；针对ResNet-56，在计算量压缩了82.54%的情况下，准确率只下降了0.14%.在ImageNet上，针对ResNet-50，在计算量压缩了56.95%的情况下，准确率只下降了0.48%.本文方法优于现有先进的剪枝方法.

Abstract

At present

the object of pruning algorithm is usually the whole convolution kernel. The rigid requirement of feature graph dimension matching in some network structur

e.g

the number of the last convolution kernel on the backbone of residual structure in ResNet and the number of convolution kernel of all branches before concatenation operation in Inception network cannot be changed

directly limits the pruning space. To solve the problem of dimensional matching that limits the pruning space

a multi-granularity pruning strategy from coarse to fine is designed to maintain dimensional matching

which keeps the number of convolution kernels in the convolution layers positioning for dimensional matching unchanged while increasing the sparsity of the neural network. Moreover

an adaptive L1 regularization sparse method is presented

which enables the network update parameters while taking into account the changes in the network structure. The sparse convolution kernel not only has fewer parameters and calculations than the original convolution kernel

but also has more excellent structural properties

which enables the network better ability for feature representation. For VGG-16 on CIFAR-10

the accuracy is increased by 0.19% when the calculation amount is compressed by 76.73% compared with the baseline network; for ResNet-56

the accuracy rate is reduced by only 0.14% when the calculation amount is compressed by 82.54%. For ResNet-50 on ImageNet

when the calculation amount is compressed by 56.95%

the accuracy rate is only reduced by 0.48%. So the proposed method is better than the existing advanced pruning methods.

关键词

Keywords

references

ZHU F D , ZHU L C , YANG Y . Sim-real joint reinforcement transfer for 3D indoor navigation [C ] // 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2020 : 11380 - 11389 .

权宇 , 李志欣 , 张灿龙 , 等 . 融合深度扩张网络和轻量化网络的目标检测模型 [J ] . 电子学报 , 2020 , 48 ( 2 ): 390 - 397 .

QUAN Y , LI Z X , ZHANG C L , et al . Fusing deep dilated convolutions network and light-weight network for object detection [J ] . Acta Electronica Sinica , 2020 , 48 ( 2 ): 390 - 397 . (in Chinese)

DING X H , DING G G , GUO Y C , et al . Centripetal SGD for pruning very deep convolutional networks with complicated structure [C ] // 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2020 : 4938 - 4948 .

饶川 , 陈靓影 , 徐如意 , 等 . 一种基于动态量化编码的深度神经网络压缩方法 [J ] . 自动化学报 , 2019 , 45 ( 10 ): 1960 - 1968 .

RAO C , CHEN J Y , XU R Y , et al . A dynamic quantization coding based deep neural network compression method [J ] . Acta Automatica Sinica , 2019 , 45 ( 10 ): 1960 - 1968 . (in Chinese)

CHO J H , HARIHARAN B . On the efficacy of knowledge distillation [C ] // 2019 IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway : IEEE , 2020 : 4793 - 4801 .

HOWARD A G , ZHU M , CHEN B , et al . MobileNets: Efficient convolutional neural networks for mobile vision applications [EB/OL ] . ( 2017-04-17 )[ 2021-07-05 ] . https://arxiv.org/abs/1704.04861 https://arxiv.org/abs/1704.04861 .

DAI X , JIANG Z R , WU Z , et al . General instance distillation for object detection [C ] // 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2021 : 7838 - 7847 .

TAI C , XIAO T , ZHANG Y , et al . Convolutional neural networks with low-rank regularization [EB/OL ] . ( 2015-11-19 )[ 2021-07-05 ] . https://arxiv.org/abs/1511.06067 https://arxiv.org/abs/1511.06067 .

DETTMERS T . 8-bit approximations for parallelism in deep learning [EB/OL ] . ( 2025-11-14 )[ 2021-07-05 ] . https://arxiv.org/abs/1511.04561 https://arxiv.org/abs/1511.04561 .

HAN S , POOL J , TRAN J , et al . Learning both weights and connections for efficient neural networks [J ] . Advances in Neural Information Processing Systems , 2015 , 1 : 1135 - 1143 .

LIU Z H , XU J Z , PENG X L , et al . Frequency-domain dynamic pruning for convolutional neural networks [C ] // Proceedings of the 32nd International Conference on Neural Information Processing Systems . New York : ACM , 2018 : 1051 - 1061 .

LIN M , JI R , ZHANG Y , et al . Channel pruning via automatic structure search [EB/OL ] . ( 2021-01-23 )[ 2021-07-05 ] . https://arxiv.org/abs/2001.08565 https://arxiv.org/abs/2001.08565 .

LI H , KADAV A , DURDANOVIC I , et al . Pruning filters for efficient ConvNets [EB/OL ] . ( 2013-08-31 )[ 2021-07-05 ] . https://arxiv.org/abs/1608.08710 https://arxiv.org/abs/1608.08710 .

LIU Z , LI J G , SHEN Z Q , et al . Learning efficient convolutional networks through network slimming [C ] // 2017 IEEE International Conference on Computer Vision (ICCV) . Piscataway : IEEE , 2017 : 2755 - 2763 .

KANG M , HAN B . Operation-aware soft channel pruning using differentiable masks [C ] // Proceedings of the 37th International Conference on Machine Learning . New York : ACM , 2020 : 5122 - 5131 .

ZHUANG T , ZHANG Z X , HUANG Y H , et al . Neuron-level structured pruning using polarization regularizer [C ] // Proceedings of the 34th International Conference on Neural Information Processing Systems . New York : ACM , 2020 : 9865 - 9877 .

MENG F , CHENG H , LI K , et al . Pruning filter in filter [EB/OL ] .( 2020-09-30 )[ 2021-07-05 ] . https://arxiv.org/abs/2009.14410 https://arxiv.org/abs/2009.14410 .

LIN M B , JI R R , WANG Y , et al . HRank: filter pruning using high-rank feature map [C ] // 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2020 : 1526 - 1535 .

HE Y H , ZHANG X Y , SUN J . Channel pruning for accelerating very deep neural networks [C ] // 2017 IEEE International Conference on Computer Vision (ICCV) . Piscataway : IEEE , 2017 : 1398 - 1406 .

LIU Z , SUN M , ZHOU T , et al . Rethinking the value of network pruning [EB/OL ] . ( 2018-10-11 )[ 2021-07-05 ] . https://arxiv.org/abs/1810.05270 https://arxiv.org/abs/1810.05270 .

XU B , WANG N , CHEN T , et al . Empirical evaluation of rectified activations in convolutional network [EB/OL ] . ( 2015-05-05 )[ 2021-07-05 ] . https://arxiv.org/abs/1505.00853 https://arxiv.org/abs/1505.00853 .

DENG J , DONG W , SOCHER R , et al . ImageNet: A large-scale hierarchical image database [C ] // 2009 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2009 : 248 - 255 .

HUANG Z H , WANG N Y . Data-driven sparse structure selection for deep neural networks [C ] // Computer Vision - ECCV 2018 . Cham : Springer International Publishing , 2018 : 317 - 334 .

LIN S H , JI R R , YAN C Q , et al . Towards optimal structured CNN pruning via generative adversarial learning [C ] // 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2020 : 2785 - 2794 .

WEI Y X , CHEN Y . Structured network pruning via adversarial multi-indicator architecture selection [J ] . Circuits, Systems, and Signal Processing , 2021 , 40 ( 8 ): 4127 - 4143 .

CHIN T W , DING R Z , ZHANG C , et al . Towards efficient model compression via learned global ranking [C ] // 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2020 : 1515 - 1525 .

LI Y W , GU S H , MAYER C , et al . Group sparsity: The hinge between filter pruning and decomposition for network compression [C ] // 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2020 : 8015 - 8024 .

Gao S , Huang F , Huang H . Model compression via hyper-structure network [C ] // International Conference on Learning Representations , IEEE , 2021 : 1 - 17 .

LE D H " , VO T N , THOAI N . Paying more attention to snapshots of Iterative pruning: Improving model compression via ensemble distillation [EB/OL ] . ( 2020-06-20 )[ 2021-07-05 ] . https://arxiv.org/abs/2006.11487 https://arxiv.org/abs/2006.11487 .

JOO D , YI E , BAEK S , et al . Linearly replaceable filters for deep network channel pruning [J ] . Proceedings of the AAAI Conference on Artificial Intelligence , 2021 , 35 ( 9 ): 8021 - 8029 .

YU R C , LI A , CHEN C F , et al . NISP: pruning networks using neuron importance score propagation [C ] // 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2018 : 9194 - 9203 .

HE Y , LIU P , WANG Z W , et al . Filter pruning via geometric Median for deep convolutional neural networks acceleration [C ] // 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2020 : 4335 - 4344 .

WANG W , FU C , GUO J , et al . COP: Customized deep model compression via regularized correlation-based filter-level pruning [EB/OL ] . ( 2019-06-25 )[ 2021-07-05 ] . https://arxiv.org/abs/1906.10337 https://arxiv.org/abs/1906.10337 .

LI H , KADAV A , DURDANOVIC I , et al . Pruning filters for efficient ConvNets [EB/OL ] . ( 2016-08-31 )[ 2021-07-05 ] . https://arxiv.org/abs/1608.08710 https://arxiv.org/abs/1608.08710 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Deep Prior-Based Blind Image Deblurring

Operator Fusion Method and Hardware Architecture Design Based on Non-Standard Operators

Shared Super-Resolution Dual-Branch Network for Spatiotemporal Fusion of Remote Sensing Images

Lightweight Fully-Connected Tensorial Mapping Network for Hyperspectral Image Classification

Related Author

BAI Yong-qiang

YU Jing

LI Yi-nong

XIAO Chuang-bai

BAI Yong-qiang

YU Jing

LI Yi-nong

XIAO Chuang-bai

Related Institution

Faculty of Information Technology, Beijing University of Technology

Faculty of Information Technology， Beijing University of Technology

College of Information Engineering, Capital Normal University

School of Mathematical Science, Capital Normal University

Faculty of Software Technologics, Shanxi Agricultural University

⁰