面向灵活并行度的稀疏卷积神经网络加速器

doi:10.12263/DZXB.20211514

摘要/Abstract

摘要：

大规模卷积神经网络计算复杂度高且资源开销大，这极大提高了深度学习算法的硬件部署成本.在模型推理过程中充分利用层间稀疏激活的信息冗余，以较低资源开销和几乎无损的网络精度降低推理时延和功耗提供高效的加速器解决方案.针对稀疏卷积神经加速器中控制粒度过大导致运算模块利用率过低问题，本文提出基于FPGA具有灵活并行度的稀疏卷积神经网络加速器架构.基于运算簇思想对卷积运算模块实现灵活调度，根据卷积层结构在线调整输入通道和输出激活的并行度；根据输出激活并行运算的数据一致性设计了一种输入数据的并行传播方式.本文在Xilinx VC709目标设备上实现了提出的加速器硬件架构，它包含1 024个乘累加单元，提供409.6GOP/s理论峰值算力；实际运算速度在VGG-16模型中达到325.8GOP/s，等效于稀疏激活优化前加速器的794.63GOP/s，运算性能达到baseline模型4.6倍以上.

关键词: FPGA, 卷积神经网络, 硬件加速, 稀疏感知, 并行计算

Abstract:

Convolutional neural network involves in high computational complexity and excessive hardware resources, which greatly increases hardware deployment cost of deep learning algorithm. It is a promising scheme to make full use of the information redundancy of sparsity activation between layers can reduce the inference delay and power consumption with low resource overhead and almost lossless network accuracy. To solve low utilization problem of operation module caused by coarse-grained control in sparse convolution neural network accelerator, a sparsity-aware accelerator with flexible parallelism based on FPGA is designed. Convolution operation module is flexibly scheduled based on operation clustering idea,and the parallelism of input channel and output activation is adjusted online.In addition, a parallel propagation mode of input data is designed according to the data consistency during output activated parallel operation. The proposed hardware architecture is implemented on Xilinx VC709. It contains up to 1 024 multiplication and accumulation units and provides 409.6GOP/s peak computing power, and the operation speed is up to 325.8GOP/ s in VGG-16 model, which is equivalent to 794.63GOP/s of accelerator without sparse activation optimization. Its performance is 4.6 times more than that of baseline model.

Key words: field programmable gate array, convolutional neural network, hardware accelerator, sparse perception, parallel computation

中图分类号:

TN47

袁海英, 曾智勇, 成君鹏. 面向灵活并行度的稀疏卷积神经网络加速器[J]. 电子学报, 2022, 50(8): 1811-1818.

YUAN Hai-ying, ZENG Zhi-yong, CHENG Jun-peng. A Sparsity-Aware Convolutional Neural Network Accelerator with Flexible Parallelism[J]. Acta Electronica Sinica, 2022, 50(8): 1811-1818.

图/表 15

图1 卷积映射示意图

图2 任务划分

图3 计算任务并行度Tp 的伪代码段

图4 加速器架构顶层视图

图5 CU结构

图6 片上缓存数据存储方式

图7 不同Tp 情况下的运算路径

图8 输出数据路由模块

图9 Tp =2时输出缓存的链接关系

表1 资源消耗

类型	LUT	FF	BRAM	DSP
使用量	322 745	537 191	768	1 089
利用率	74.5%	62.00%	52.00%	30.25%

图10 在AlexNet中各个卷积层的运算时间占比

图11 在VGG-16中不同层的运算性能

图12 在VGG-16中不同层的资源利用率

图13 本文加速器的Roofline模型

表2 加速器性能对比

Design	文献[10]	LACS^[16]	文献[17]	OMNI^[18]	本文
Precision	Fixed-16	Fixed-8	Fixed-16	Fixed-16	Fixed-16
Density	—	—	23.5%	12%	41%	39%
Device	Arria-10GX 1150	XC7Z030	ZCU102	ZCU102	VC709
CNN Module	VGG-16	VGG-16	VGG-16	VGG-16	VGG-16	VGG-19
Frequency(MHz)	150	250	200	200	200
DSP Utilization	1 518	256	1 144	—	1 089
Power(W)	—	—	23.6	23.6	17.4
Performance (GOP/s)	645.25	127.5	309.0	127.1	325.8	332.7

参考文献 18

1	BAI L, LYU Y, HUANG X. RoadNet-RT: High throughput CNN architecture and SoC design for real-time road segmentation[J]. IEEE Transactions on Circuits and Systems I, 2021, 68(2): 704-714.
2	KRIZHEVSKY A, SUTSKEVER I, HINTON G. ImageNet classification with deep convolutional neural networks[J]. Advances in Neural Information Processing Systems, 2012, 25(2): 1097-1105.
3	HE K M, ZHANG X Y, REN S Q, et al. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification[C]//IEEE International Conference on Computer Vision. Santiago: IEEE, 2015: 1026-1034.
4	刘杰, 葛一凡, 田明, 马力强. 基于ZYNQ的可重构卷积神经网络加速器[J]. 电子学报, 2021, 49(4): 729-735.
	LIU Jie, GE Yi-fan, TIAN Ming, MA Li-qiang. Reconfigurable convolutional network accelerator based on ZYNQ[J]. Acta Electronica Sinica, 2021, 49(4): 729-735. (in Chinese)
5	LIANG S, YIN S, LIU L, et al. Acoarse-grained reconfigurable architecture for compute-intensive mapreduce acceleration[J]. IEEE Computer Architecture Letters, 2016, 15 (2): 69-72.
6	YU Y, WU C, ZHAO T, et al.OPU: An FPGA-based overlay processor for convolutional neural networks[J]. IEEE Transactions on Very Large-Scale Integration(VLSI) Systems, 2020, 28 (1): 35-47.
7	ZHANG C, ZHENMAN F, PEIPEI Z, et al. Caffeine: Towards uniformed representation and acceleration for deep convolutional neural networks[C]//IEEE/ACM International Conference on Computer-Aided Design(ICCAD). Austin: IEEE, 2016: 1-8.
8	GUO J, YIN S, OUYANG P, et al. Bit-width based resource partitioning for CNN acceleration on FPGA[C]//IEEE 25th Annual International Symposium on Field-Programmable Custom Computing Machines(FCCM).Napa:IEEE, 2017: 31-31.
9	ALBERICIO J, JUDD P, HETHERINGTON T, et al. Cnvlutin: Ineffectual-neuron-freedeep neural network computing[C]//IEEE 43th International Symposium on Computer Architecture. Seoul: IEEE, 2016: 1-13.
10	MA Y, CAOY, VRUDHULA S, et al. Optimizing loop operation and dataflow in FPGA acceleration of deep convolutional neural networks[C]//ACM/Sigda International Symposium on Field-programmable Gate Arrays. Monterey: ACM, 2017:45-54.
11	LEE H, GROSSE R, RANGANATH R, et al. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations[C]//26th International Conference on Machine Learning. Montreal: ACM, 2009: 609-616.
12	KARPATHY A, TODERICI G, SHETTY S, et al. Large-scale video classification with convolutional neural networks[C]//Computer Vision & Pattern Recognition. Columbus: IEEE, 2014: 1725-1732.
13	YU D, DENG L. Deep learning and its applications to signal and information processing[J]. IEEE Signal Processing Magazine,2011, 28 (1): 145-154.
14	CONG J, XIAO B. Minimizing computation in convolutional neural networks[C]//International Conference on Artificial Neural Networks. Hamburg: Springer, Cham, 2014: 281-290.
15	LI Y, MA S, GUO Y, et al. Configurable CNN accelerator based on tiling dataflow[C]//2018 IEEE 9th International Conference on Software Engineering and Service Science(ICSESS). Beijing: IEEE, 2018: 309-313.
16	SHANG J W, QIAN L, ZHANG Z, et al.LACS: A high-computational-efficiency accelerator for CNNs[J]. IEEE Access, 2020, 8: 6045-6059.
17	ZHU C, HUANG K, YANG S, et al. Anefficient hardware accelerator for structured sparse convolutional neural networks on FPGAs[J]. IEEE Transactions on Very Large-Scale Integration(VLSI) Systems,2020, 28 (9): 1953-1965.
18	LIANG Y, LU L Q, XIE J M. OMNI: A framework for integrating hardware and software optimizations for sparse CNNs[J]. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2021, 40 (8): 1648-1661.

[1]	吴靖, 叶晓晶, 黄峰, 陈丽琼, 王志锋, 刘文犀. 基于深度学习的单帧图像超分辨率重建综述[J]. 电子学报, 2022, 50(9): 2265-2294.
[2]	毛国君, 王者浩, 黄山, 王翔. 基于剪边策略的图残差卷积深层网络模型[J]. 电子学报, 2022, 50(9): 2205-2214.
[3]	陈晓杰, 李斌, 周清雷. RTL级可扩展高性能数据压缩方法实现[J]. 电子学报, 2022, 50(7): 1548-1557.
[4]	王相海, 赵晓阳, 王鑫莹, 赵克云, 宋传鸣. 非抽取小波边缘学习深度残差网络的单幅图像超分辨率重建[J]. 电子学报, 2022, 50(7): 1753-1765.
[5]	张志文, 刘天歌, 聂鹏举. 基于实景数据增强和双路径融合网络的实时街景语义分割算法[J]. 电子学报, 2022, 50(7): 1609-1620.
[6]	丁毅, 沈薇, 李海生, 钟琼慧, 田明宇, 李洁. 面向CNN的区块链可信隐私服务计算模型[J]. 电子学报, 2022, 50(6): 1399-1409.
[7]	张波, 陆云杰, 秦东明, 邹国建. 一种卷积自编码深度学习的空气污染多站点联合预测模型[J]. 电子学报, 2022, 50(6): 1410-1427.
[8]	许皓文, 陆浩, 王振占. 高光谱微波辐射计系统中2GHz带宽数字谱仪设计[J]. 电子学报, 2022, 50(6): 1472-1479.
[9]	曲立国, 陈国豪, 胡俊, 陈鹏. 单次扫描连通域分析算法研究综述[J]. 电子学报, 2022, 50(6): 1521-1536.
[10]	田春生, 陈雷, 王源, 王硕, 周婧, 张瑶伟, 庞永江, 周冲, 马筱婧, 杜忠, 薛钰. 面向FPGA的布局与布线技术研究综述[J]. 电子学报, 2022, 50(5): 1243-1254.
[11]	廖勇, 李玉杰. 一种轻量化低复杂度的FDD大规模MIMO系统CSI反馈方法[J]. 电子学报, 2022, 50(5): 1211-1217.
[12]	张云, 化青龙, 姜义成, 徐丹. 基于混合型复数域卷积神经网络的三维转动舰船目标识别[J]. 电子学报, 2022, 50(5): 1042-1049.
[13]	崔亚奇, 何友, 唐田田, 熊伟. 一种深度学习航迹关联方法[J]. 电子学报, 2022, 50(3): 759-763.
[14]	伍邦谷, 张苏林, 石红, 朱鹏飞, 王旗龙, 胡清华. 基于多分支结构的不确定性局部通道注意力机制[J]. 电子学报, 2022, 50(2): 374-382.
[15]	陈文俊, 杨春玲. 图像压缩感知的特征域优化及自注意力增强神经网络重构算法[J]. 电子学报, 2022, 50(11): 2629-2637.