基于视频先验信息的轻量化去噪卷积神经网络

doi:10.12263/DZXB.20210679

摘要/Abstract

摘要：

提出了一种基于视频先验信息的轻量化去噪卷积神经网络.先验信息从近邻的多帧视频图像中获取，采用了基于预去噪的视频运动补偿方法消除噪声和运动偏移对信息获取准确度的影响.为降低卷积神经网络复杂度，构建了基于双路处理的卷积神经网络用于去除视频噪声，特别是设计了双路稠密连接单元，实现了网络的轻量化.双路稠密连接单元通过高、低分辨率特征分解和特征拼接，有效降低了网络复杂度.实验结果表明：采用本文方法去除视频噪声能够获得较好的客观评价结果和主观视觉结果.此外，在减少网络参数、降低浮点运算次数和提升运行速度方面均体现出了良好性能.

关键词: 视频去噪, 卷积神经网络, 先验信息, 轻量化网络, 特征

Abstract:

A priori information-based lightweight convolutional neural network (CNN) for video denoising is proposed in this paper. More specifically, the priori information is obtained from the adjacent frames and the pre-denosing-based motion compensation is applied to effectively collect the priori information. Meanwhile, the dual-path processing-based CNN is designed to remove the video noise. Specifically, the dual-path cross connection unit (DCU) is proposed to simplify the feature extraction for the design of a lightweight network. With DCU, the high-resolution features and the low-resolution features are generated and they are concatenated by shorter connections, which achieve low complexity. The experimental results demonstrate that our proposed method offers good objective results as well as subjective results. Moreover, it effectively reduces network parameters and floating point operations, and achieves a fast CNN-based video denosing.

Key words: video denoising, convolutional neural network, priori information, lightweight network, feature

中图分类号:

TP391

申屠敏健, 朱强, 朱树元, 等. 基于视频先验信息的轻量化去噪卷积神经网络[J]. 电子学报, DOI: 10.12263/DZXB.20210679.

Min-jian SHENTU, Qiang ZHU, Shu-yuan ZHU, et al. A Priori Information-Based Lightweight Convolutional Neural Network for Video Denoising[J]. Acta Electronica Sinica, DOI: 10.12263/DZXB.20210679.

图/表 9

参考文献 23

1	DABOV K, FOI A, KATKOVNIK V, et al. Image denoising by sparse 3-D transform-domain collaborative filtering[J]. IEEE Transactions on Image Processing: A Publication of the IEEE Signal Processing Society, 2007, 16(8): 2080-2095.
2	GU S H, ZHANG L, ZUO W M, et al. Weighted nuclear norm minimization with application to image denoising[C]//2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus: IEEE, 2014: 2862-2869.
3	MAIRAL J, BACH F, PONCE J, et al. Non-local sparse models for image restoration[C]//2009 IEEE 12th International Conference on Computer Vision. Kyoto: IEEE, 2009: 2272-2279.
4	DONG W S, ZHANG L, SHI G M, et al. Nonlocally centralized sparse representation for image restoration[J]. IEEE Transactions on Image Processing, 2013, 22(4): 1620-1630.
5	BUADES A, COLL B, MOREL J M. Nonlocal image and movie denoising[J]. International Journal of Computer Vision, 2008, 76(2): 123-139.
6	WEISS Y, FREEMAN W T. What makes a good model of natural images?[C]//2007 IEEE Conference on Computer Vision and Pattern Recognition. Minneapolis: IEEE, 2007: 1-8.
7	DABOV K, FOI A, EGIAZARIAN K. Video denoising by sparse 3D transform-domain collaborative filtering[C]//Proceedings of the 2007 European Signal Processing Conference. Poznan: IEEE, 2007.145-149.
8	MAGGIONI M, BORACCHI G, FOI A, et al. Video denoising, deblocking, and enhancement through separable 4-D nonlocal spatiotemporal transforms[J]. IEEE Transactions on Image Processing, 2012, 21(9): 3952-3966.
9	ARIAS P, MOREL J M. Towards a Bayesian Video Denoising Method[M]//Advanced Concepts for Intelligent Vision Systems. Cham: Springer International Publishing, 2015: 107-117.
10	ZHANG K, ZUO W M, CHEN Y J, et al. Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising[J]. IEEE Transactions on Image Processing, 2017, 26(7): 3142-3155.
11	DAVY A, EHRET T, MOREL J M, et al. A non-local CNN for video denoising[C]//2019 IEEE International Conference on Image Processing. Taipei: IEEE, 2019: 2409-2413.
12	IANDOLA F N, MOSKEWICZ M W, ASHRAF K, et al. SqueezeNet: AlexNet-Level Accuracy with 50x Fewer Parameters and <0.5MB Model Size[EB/OL]. (2016)[2021]. .
13	HOWARD A, ZHU M, CHEN B, et al. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications[EB/OL]. (2017)[2021]. .
14	CHOLLET F. Xception: deep learning with depthwise separable convolutions[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 1800-1807.
15	ZHANG X, ZHOU X, LIN M, et al. ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices[EB/OL]. (2017)[2021]. .
16	HAN K, WANG Y H, TIAN Q, et al. GhostNet: more features from cheap operations[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Seattle: IEEE, 2020: 1577-1586.
17	孟琭, 徐磊, 郭嘉阳. 一种基于改进的MobileNetV2网络语义分割算法[J]. 电子学报, 2020, 48(9): 1769-1776.
	MENG L, XU L, GUO J Y. Semantic segmentation algorithm based on improved MobileNetV2[J]. Acta Electronica Sinica, 2020, 48(9): 1769-1776. (in Chinese)
18	IOFFE S, SZEGEDY C. Batch normalization: Accelerating deep network training by reducing internal covariate shift[C]//Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37. New York: ACM, 2015: 448-456.
19	KRIZHEVSKY A, SUTSKEVER I, HINTON G. ImageNet classification with deep convolutional neural networks[C]//Proceedings of the 2012 International Conference on Neural Information Processing Systems. Doha: Curran Associates Inc., 2012: 1097-1105.
20	TASSANO M, DELON J, VEIT T. FastDVDnet: towards real-time deep video denoising without flow estimation[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Seattle: IEEE, 2020: 1351-1360.
21	PONT-TUSET J, PERAZZI F, CAELLES S, et al. The 2017 DAVIS Challenge on Video Object Segmentation[EB/OL]. (2018)[2021]. .
22	KINGMA D, BA J. Adam: A Method for Stochastic Optimization[EB/OL]. (2017)[2021]. .
23	HE K M, ZHANG X Y, REN S Q, et al. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification[C]//2015 IEEE International Conference on Computer Vision. Santiago: IEEE, 2015: 1026-1034.

标准差	对比方法	Five-Seq	Set8	DAVIS
10	BM3D	35.37/0.943 3	36.07/0.948 7	38.18/0.961 1
	DnCNN	35.45/0.945 2	36.32/0.951 4	38.41/0.962 6
	VNLnet	36.20/0.954 8	36.66/0.955 1	38.94/0.966 5
	FastDVDnet	35.87/0.956 5	36.21/0.953 4	38.56/0.965 5
	本文方法	36.62/0.958 6	37.00/0.958 6	39.38/0.969 9
20	BM3D	31.57/0.881 0	32.36/0.898 9	34.48/0.919 3
	DnCNN	31.73/0.886 5	32.76/0.907 1	34.80/0.924 7
	VNLnet	32.73/0.907 8	33.20/0.914 3	35.49/0.933 8
	FastDVDnet	32.97/0.919 6	33.24/0.918 9	35.60/0.931 2
	本文方法	33.17/0.914 8	33.30/0.914 3	36.10/0.942 2
30	BM3D	29.47/0.828 5	30.27/0.854 9	32.39/0.880 9
	DnCNN	29.71/0.833 8	30.70/0.907 1	32.72/0.889 3
	VNLnet	30.77/0.864 0	31.21/0.876 4	33.47/0.902 3
	FastDVDnet	31.07/0.879 0	31.49/0.888 1	33.86/0.909 8
	本文方法	31.21/0.875 2	31.75/0.890 0	34.20/0.916 6
40	BM3D	27.81/0.770 2	28.61/0.803 8	30.57/0.833 8
	DnCNN	28.31/0.786 6	29.34/0.830 4	31.36/0.857 5
	VNLnet	29.41/0.822 1	29.80/0.841 5	32.07/0.874 2
	FastDVDnet	29.80/0.845 6	30.27/0.860 0	32.67/0.890 3
	本文方法	29.87/0.839 1	30.41/0.860 3	32.83/0.891 9

标准差	对比方法	Five-Seq	Set8	DAVIS
10	BM3D	35.37/0.943 3	36.07/0.948 7	38.18/0.961 1
	DnCNN	35.45/0.945 2	36.32/0.951 4	38.41/0.962 6
	VNLnet	36.20/0.954 8	36.66/0.955 1	38.94/0.966 5
	FastDVDnet	35.87/0.956 5	36.21/0.953 4	38.56/0.965 5
	本文方法	36.62/0.958 6	37.00/0.958 6	39.38/0.969 9
20	BM3D	31.57/0.881 0	32.36/0.898 9	34.48/0.919 3
	DnCNN	31.73/0.886 5	32.76/0.907 1	34.80/0.924 7
	VNLnet	32.73/0.907 8	33.20/0.914 3	35.49/0.933 8
	FastDVDnet	32.97/0.919 6	33.24/0.918 9	35.60/0.931 2
	本文方法	33.17/0.914 8	33.30/0.914 3	36.10/0.942 2
30	BM3D	29.47/0.828 5	30.27/0.854 9	32.39/0.880 9
	DnCNN	29.71/0.833 8	30.70/0.907 1	32.72/0.889 3
	VNLnet	30.77/0.864 0	31.21/0.876 4	33.47/0.902 3
	FastDVDnet	31.07/0.879 0	31.49/0.888 1	33.86/0.909 8
	本文方法	31.21/0.875 2	31.75/0.890 0	34.20/0.916 6
40	BM3D	27.81/0.770 2	28.61/0.803 8	30.57/0.833 8
	DnCNN	28.31/0.786 6	29.34/0.830 4	31.36/0.857 5
	VNLnet	29.41/0.822 1	29.80/0.841 5	32.07/0.874 2
	FastDVDnet	29.80/0.845 6	30.27/0.860 0	32.67/0.890 3
	本文方法	29.87/0.839 1	30.41/0.860 3	32.83/0.891 9

比较指标	VNLnet	本文方法
FLOPs(G)	226.85	52.64
Parameters(K)	4 518.05	2 340.94
FPS(GPU)	38.46	84.67
FPS(CPU)	1.38	3.71
Memory(MB)	1 654.32	378.55

比较指标	VNLnet	本文方法
FLOPs(G)	226.85	52.64
Parameters(K)	4 518.05	2 340.94
FPS(GPU)	38.46	84.67
FPS(CPU)	1.38	3.71
Memory(MB)	1 654.32	378.55

比较指标	消融实验1	消融实验2	消融实验3	本文方法
PSNR	31.13	31.18	29.87	31.21
SSIM	0.873 8	0.874 1	0.841 1	0.875 2
FLOPs(G)	86.53	88.01	52.27	52.64
Parameters(K)	2 758.75	3 307.82	2 329.27	2 340.94
Memory(MB)	766.15	442.29	378.55	378.55
FPS(GPU)	68.37	62.46	86.54	84.67
FPS(CPU)	3.58	2.47	3.79	3.71