高效还原式二值神经网络

曾凯; 万子鑫; 王铭涛; 沈韬

doi:10.12263/DZXB.20240640

您当前的位置：

首页 >

文章列表页 >

高效还原式二值神经网络

学术论文 | 更新时间：2025-12-08

- 高效还原式二值神经网络
  增强出版
- Efficient Restoration for Binary Neural Networks
- 电子学报 2025年53卷第2期页码：568-580
- 作者机构：
  
  1.昆明理工大学信息工程与自动化学院，云南昆明 650500
  2.云南省计算机技术应用重点实验室，云南昆明 650500
- 作者简介：
  
  [ "曾凯男，1985年生，副教授，昆明理工大学硕士生导师.主要研究方向为模型压缩、边缘计算.E-mail: zengkai@kust.edu.cn" ]
  [ "万子鑫男，2024年取得昆明理工大学硕士学位.主要研究方向为神经网络量化与二值神经网络. E-mail: wanzixin@stu.kust.edu.cn" ]
  [ "王铭涛男，现就读于昆明理工大学硕士研究生.主要研究方向为二值神经网络、计算机视觉. E-mail: wangmingtao@stu.kust.edu.cn" ]
  [ "沈韬男，教授，昆明理工大学博士生导师，云南省杰出基金项目获得者.主要研究方向为边缘计算、多源智能感知. E-mail: shentao@kust.edu.cn" ]
- 基金信息：
  
  云南省杰出青年人才项目(202301AV070003);云南省重大科技专项(202302AG050009);云南省重大科技专项(202202AD080013)
- DOI：10.12263/DZXB.20240640
  中图分类号： TP391;TP301
- 收稿：2024-07-08，
  
  修回：2024-10-22，
  
  纸质出版：2025-02-25
- 稿件说明：
移动端阅览
曾凯, 万子鑫, 王铭涛, 等. 高效还原式二值神经网络[J]. 电子学报, 2025, 53(02): 568-580.

ZENG Kai, WAN Zi-xin, WANG Ming-tao, et al. Efficient Restoration for Binary Neural Networks[J]. Acta Electronica Sinica, 2025, 53(02): 568-580.
曾凯, 万子鑫, 王铭涛, 等. 高效还原式二值神经网络[J]. 电子学报, 2025, 53(02): 568-580. DOI：10.12263/DZXB.20240640

ZENG Kai, WAN Zi-xin, WANG Ming-tao, et al. Efficient Restoration for Binary Neural Networks[J]. Acta Electronica Sinica, 2025, 53(02): 568-580. DOI：10.12263/DZXB.20240640

摘要

将权重分布、激活分布和梯度尽可能地还原为原始全精度网络数据，能够极大提高二值网络的推理能力.然而，现有方法将正向传播中的还原操作直接作用于二值数据，同时用以控制反向传播的梯度近似函数均为固定或手动方式确定，导致二值网络的还原效率有待改进.针对这一问题，构建了高效还原式二值神经网络.首先提出面向信息熵最大的分布恢复方法，通过对原始全精度权重均值平移和模长缩放，使量化后的二值权重直接具备分布最大还原特性，同时采用基于简单统计的平移和缩放因子，极大提高了权重和激活的还原效率；进一步提出基于自适应分布近似的梯度函数，根据当前全精度数据的实际分布，以P分位动态确定当前梯度的更新范围，进而自适应改变近似函数的形状，使训练过程中的梯度得到高效更新，从而提高了模型的收敛能力.在保证执行效率提升的前提下，通过理论分析证实了本文方法能够使二值数据达到最大程度还原.与当前现有的先进二值网络模型相比本文方法实验结果表现优异，其中针对ResNet-18和ResNet-20量化的分布还原操作计算时间开销分别下降了60%和67%；同时在CIFAR-10数据集上针对VGG-Small二值量化取得93.0%的准确率，在ImageNet数据集上针对ResNet-18二值量化取得61.9%的准确率，均为当前二值神经网络的最佳性能表现.相关代码开源在

https：//github.com/sjmp525/IA/tree/ER-BNN

https://github.com/sjmp525/IA/tree/ER-BNN

Abstract

Restoring the weight distribution

activation distribution

and gradient to the original full precision network data as much as possible can greatly improve the inference ability of the binary network. However

existing methods directly apply the restoration operation in forward propagation to binary data

and the gradient approximation functions for backpropagation are fixed or manually determined

resulting in the need for improvement in the restoration efficiency of binary networks. To address this problem

the efficient restoration method is investigated for binary neural networks. Firstly

a distribution recovery method for maximizing information entropy is proposed. By shifting the original full precision weight mean and scaling the modulus

the quantized binary weight directly has the characteristic of maximum distribution restoration. At the same time

a simple statistical translation and scali

ng factor is used to greatly improve the restoration efficiency of weight and activation. Furthermore

it is proposed a gradient function based on adaptive distribution approximation

which dynamically determines the update range of the current gradient in the P-percentile according to the actual distribution of the current full precision data. It adaptively changes the shape of the approximation function to efficiently update the gradient during the training process

thereby improving the convergence ability of the model. On the premise of ensuring the improvement of execution efficiency

theoretical analysis has confirmed that the method proposed in this paper can achieve maximum restoration of binary data. Compared with the existing advanced binary network models

the experimental results of our method show excellent performance

with a 60% and 67% reduction in computational time for the distribution restoration operation quantization of ResNet-18 and ResNet-20

respectively. An accuracy of 93.0% is achieved for VGG-Small binary quantization on the CIFAR-10 dataset

and 61.9% is achieved for ResNet-18 binary quantization on the ImageNet dataset

both of which are the best performance of the current binary neural network. The relevant code is available in

https://github.com/sjmp525/IA/tree/ER-BNN

关键词

Keywords

references

王子为 , 鲁继文 , 周杰 . 基于自适应梯度优化的二值神经网络 [J ] . 电子学报 , 2023 , 51 ( 2 ): 257 - 266 .

WANG Z W , LU J W , ZHOU J . Learning adaptive gradients for binary neural networks [J ] . Acta Electronica Sinica , 2023 , 51 ( 2 ): 257 - 266 . (in Chinese)

YUAN C , AGAIAN S S . A comprehensive review of binary neural network [J ] . Artificial Intelligence Review , 2023 , 56 ( 11 ): 12949 - 13013 .

袁海英 , 成君鹏 , 曾智勇 , 等 . Mobile＿BLNet:基于Big-Little Net的轻量级卷积神经网络优化设计 [J ] . 电子学报 , 2023 , 51 ( 1 ): 180 - 191 .

YUAN H Y , CHENG J P , ZENG Z Y , et al . Mobile_BLNet: Optimization design of lightweight convolutional neural network based on Big-Little Net [J ] . Acta Electronica Sinica , 2023 , 51 ( 1 ): 180 - 191 . (in Chinese)

TU Z , CHEN X , REN P , et al . AdaBin: Improving binary neural networks with adaptive binary sets [C ] // European Conference on Computer Vision (ECCV) . Cham : Springer , 2022 : 379 - 395 .

RASTEGARI M , ORDONEZ V , REDMON J , et al . XNOR-Net: ImageNet classification using binary convolutional neural networks [C ] // European Conference on Computer Vision (ECCV) . Cham : Springer , 2016 : 525 - 542 .

LIU Z , LUO W , WU B , et al . Bi-Real Net: Binarizing deep network towards real-network performance [J ] . International Journal of Computer Vision , 2020 , 128 ( 1 ): 202 - 219 .

MARTINEZ B , YANG J , BULAT A , et al . Training binary neural networks with real-to-binary convolutions [C ] // International Conference on Learning Representations (ICLR) . Piscataway : IEEE , 2020 : 1 - 13 .

LIU Z , SHEN Z , SAVVIDES M , et al . ReActNet: Towards precise binary neural network with generalized activation functions [C ] // European Conference on Computer Vision (ECCV) . Cham : Springer , 2020 : 143 - 159 .

QIN H , GONG R , LIU X , et al . Forward and backward information retention for accurate binary neural networks [C ] // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2020 : 2250 - 2259 .

ZHANG S , GE F , DING R , et al . Learning to binarize convolutional neural networks with adaptive neural encoder [C ] // International Joint Conference on Neural Networks (IJCNN) . Piscataway : IEEE , 2021 : 1 - 8 .

HUBARA I , COURBARIAUX M , SOUDRY D , et al . Binarized neural networks [J ] . Advances in Neural Information Processing Systems , 2016 , 29 : 1 - 14 .

LIN M , JI R , XU Z , et al . Rotated binary neural network [C ] // Neural Information Processing Systems (NeurIPS) . Piscataway : IEEE , 2020 : 7474 - 7485 .

LU H , YAO Q , KWOK J T . Loss-aware binarization of deep networks [C ] // International Conference on Learning Representations (ICLR) . Piscataway : IEEE , 2017 : 24 - 26 .

WANG P S , HE X Y , LI G , et al . Sparsity-inducing binarized neural networks [J ] . Proceedings of the AAAI Conference on Artificial Intelligence , 2020 , 34 ( 7 ): 12192 - 12199 .

XUE P , LU Y , CHANG J , et al . Self-distribution binary neural networks [J ] . Applied Intelligence , 2022 , 52 ( 12 ): 13870 - 13882 .

QIN H , ZHANG X , GONG R , et al . Distribution-sensitive information retention for accurate binary neural network [J ] . International Journal of Computer Vision , 2023 , 131 ( 1 ): 26 - 47 .

GONG R , LIU X , JIANG S , et al . Differentiable soft quantization: Bridging full-precision and low-bit neural networks [C ] // IEEE International Conference on Computer Vision (ICCV) . Piscataway : IEEE , 2019 : 4852 - 4861 .

YANG Z , WANG Y , HAN K , et al . Searching for low-bit weights in quantized neural networks [J ] . Advances in Neural Information Processing Systems , 2020 , 33 : 4091 - 4102 .

HAN K , WANG Y , XU Y , et al . Training binary neural networks through learning with noisy supervision [C ] // International Conference on Machine Learning (ICML) . Piscataway : IEEE , 2020 : 4017 - 4026 .

WU L J , LIN X , CHEN Z C , et al . An efficient binary convolutional neural network with numerous skip connections for fog computing [J ] . IEEE Internet of Things Journal , 2021 , 8 ( 14 ): 11357 - 11367 .

KIM D , SINGH K P , CHOI J . Learning architectures for binary networks [C ] // European Conference on Computer Vision (ECCV) . Cham : Springer , 2020 : 575 - 591 .

KIM H , PARK J , LEE C , et al . Improving accuracy of binary neural networks using unbalanced activation distribution [C ] // Computer Vision and Pattern Recognition (CVPR) . Cham : Springer , 2021 : 7862 - 7871 .

YE J , WANG J , ZHANG S . Distillation-guided residual learning for binary convolutional neural networks [J ] . IEEE Transactions on Neural Networks and Learning Systems , 2021 , 33 ( 12 ): 7765 - 7777 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

暂无数据