一种轻量级的多尺度通道注意图像超分辨率重建网络

doi:10.12263/DZXB.20201089

摘要/Abstract

摘要：

近年来，基于深度卷积神经网络的图像超分辨率技术取得了突出进展，并主导了当前的超分辨率技术的研究.但是，性能的改进，往往以参数量的急剧增加为代价，这限制了超分辨率方法的实际应用.本文设计了一个轻量级单图像超分辨率深度卷积网络，主要贡献包括：提出了一个多尺度的特征融合模块，使用不同感受野的卷积核，提取多种尺度的特征；提出了一个通道搅乱注意力模块，促进特征通道之间的信息流动，并增强特征选择能力；提出了一个全局特征融合连接模块，提高浅层特征的利用率.实验证明，本文方法与当前代表性的方法MSRN（Multi-Scale Residual Network）相比，参数量减少了3/4，重建的高分辨率图像的主观和客观质量均显著更好.

关键词: 超分辨率, 深度学习, 卷积神经网络, 注意力机制, 多尺度特征

Abstract:

Recently, image super-resolution technology based on deep convolutional neural network has made remarkable achievements and has become popular in the current super-resolution technology. However, superior performance is often at the expense of the large number of parameter amounts, which limits the real-world applications for single image super-resolution. In this paper, a lightweight single image super-resolution deep convolutional network is proposed. The main contributions of this paper are as follows: a multi-scale feature fusion block is proposed to extract multiple features via convolution kernels with different receptive fields; the channel shuffle attention mechanism we designed promotes the flow of the information across feature channels, which enhances the ability of feature selection; a global feature fusion connection is proposed to improve the feature utilization. Extensive experiments demonstrate that the parameter amounts of our method reduced by 3/4 compared with the current state-of-the-art MSRN method, while subjective visual and objective quality of the reconstructed high-resolution image are perform significantly better.

Key words: super-resolution, deep learning, convolutional neural network, attention mechanism, multi-scale feature

中图分类号:

TP391

周登文, 李文斌, 李金新, 等. 一种轻量级的多尺度通道注意图像超分辨率重建网络[J]. 电子学报, 2022, 50(10): 2336-2346.

Deng-wen ZHOU, Wen-bin LI, Jin-xin LI, et al. Image Super-Resolution Reconstruction Based on Lightweight Multi-Scale Channel Attention Network[J]. Acta Electronica Sinica, 2022, 50(10): 2336-2346.

图/表 17

参考文献 42

1	THORNTON M W, ATKINSON P M, HOLLAND D A. Sub-pixel mapping of rural land cover objects from fine spatial resolution satellite sensor imagery using super-resolution pixel-swapping[J]. International Journal of Remote Sensing, 2006, 27(3): 473-491.
2	ZOU W W W, YUEN P C. Very low resolution face recognition problem[J]. IEEE Transactions on Image Processing, 2012, 21(1): 327-340.
3	SHI W Z, CABALLERO J, LEDIG C, et al. Cardiac image super-resolution with global correspondence using multi-atlas patchmatch[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention. Berlin, Heidelberg: Springer, 2013: 9-16.
4	LEDIG C, THEIS L, HUSZÁR F, et al. Photo-realistic single image super-resolution using a generative adversarial network[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 105-114.
5	WANG F, JIANG M Q, QIAN C, et al. Residual attention network for image classification[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 6450-6458.
6	LI Z, YANG J L, LIU Z, et al. Feedback network for image super-resolution[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Long Beach: IEEE, 2019: 3862-3871.
7	LIM B, SON S, KIM H, et al. Enhanced deep residual networks for single image super-resolution[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Honolulu: IEEE, 2017: 1132-1140.
8	ZHANG Y L, TIAN Y P, KONG Y, et al. Residual dense network for image super-resolution[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 2472-2481.
9	DONG C, LOY C C, HE K M, et al. Learning a Deep Convolutional Network for Image Super-Resolution[C]//European Conference on Computer Vision. Cham: Springer, 2014: 184-199.
10	KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1646-1654.
11	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778.
12	TAI Y, YANG J, LIU X M. Image super-resolution via deep recursive residual network[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 2790-2798.
13	TAI Y, YANG J, LIU X M, et al. MemNet: A persistent memory network for image restoration[C]//2017 IEEE International Conference on Computer Vision. Honolulu: IEEE, 2017: 4549-4557.
14	AHN N, KANG B, SOHN K A. Fast, accurate, and lightweight super-resolution with cascading residual network[C]//European Conference on Computer Vision. Cham: Springer International Publishing, 2018: 256-272.
15	ZHU F Y, ZHAO Q J. Efficient single image super-resolution via hybrid residual feature learning with compact back-projection network[C]//2019 IEEE/CVF International Conference on Computer Vision Workshop. Seoul: IEEE, 2019: 2453-2460.
16	HUI Z, GAO X B, YANG Y C, et al. Lightweight image super-resolution with information multi-distillation network[C]//MM'19: Proceedings of the 27th ACM International Conference on Multimedia. New York: ACM, 2019: 2024-2032.
17	LI J C, FANG F M, MEI K F, et al. Multi-scale residual network for image super-resolution[C]//European Conference on Computer Vision. Cham: Springer International Publishing, 2018: 527-542.
18	ITTI L, KOCH C, NIEBUR E. A model of saliency-based visual attention for rapid scene analysis[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1998, 20(11): 1254-1259.
19	HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 7132-7141.
20	ZHANG Y L, LI K P, LI K, et al. Image super-resolution using very deep residual channel attention networks[C]//European Conference on Computer Vision. Cham: Springer, 2018: 294-310.
21	KIM J, LEE J K, LEE K M. Deeply-recursive convolutional network for image super-resolution[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1637-1645.
22	SHI W Z, CABALLERO J, HUSZÁR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1874-1883.
23	ZHANG Y L, LI K P, LI K, et al. Image super-resolution using very deep residual channel attention networks[C]//European Conference on Computer Vision. Cham: Springer, 2018: 294-310.
24	ZHANG X Y, ZHOU X Y, LIN M X, et al. ShuffleNet: An extremely efficient convolutional neural network for mobile devices[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 6848-6856.
25	AGUSTSSON E, TIMOFTE R. NTIRE 2017 challenge on single image super-resolution: Dataset and study[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Honolulu: IEEE, 2017: 1122-1131.
26	BEVILACQUA M, ROUMY A, GUILLEMOT C, et al. Low-complexity single-image super-resolution based on nonnegative neighbor embedding[C]//Proceedings British Machine Vision Conference. Surrey: BMVA Press, 2012: 135.1-135.10.
27	ZEYDE R, ELAD M, PROTTER M. On Single Image Scale-up Using Sparse-Representations[M]//Curves and Surfaces. Berlin, Heidelberg: Springer, 2012: 711-730.
28	MARTIN D, FOWLKES C, TAL D, et al. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[C]//Proceedings Eighth IEEE International Conference on Computer Vision. Vancouver: IEEE, 2001: 416-423.
29	HUANG J B, SINGH A, AHUJA N. Single image super-resolution from transformed self-exemplars[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 5197-5206.
30	MATSUI Y, ITO K, ARAMAKI Y, et al. Sketch-based manga retrieval using manga109 dataset[J]. Multimedia Tools and Applications, 2017, 76(20): 21811-21838.
31	WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: From error visibility to structural similarity[J]. IEEE Transactions on Image Processing: A Publication of the IEEE Signal Processing Society, 2004, 13(4): 600-612.
32	KINGMA D P, BA J. Adam: A method for stochastic optimization[EB/OL]. (2014)[2020]. .
33	PASZKE A, GROSS S, CHINTALA S, et al. Automatic differentiation in pytorch[OL]. (2017)[2020]. .
34	ZHANG K, ZUO W M, ZHANG L. Learning a single convolutional super-resolution network for multiple degradations[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 3262-3271.
35	DONG C, LOY C C, TANG X O. Accelerating the super-resolution convolutional neural network[C]//European Conference on Computer Vision. Cham: Springer, 2016: 391-407.
36	LAI W S, HUANG J B, AHUJA N, et al. Deep Laplacian pyramid networks for fast and accurate super-resolution[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 5835-5843.
37	YANG W M, WANG W, ZHANG X C, et al. Lightweight feature fusion network for single image super-resolution[J]. IEEE Signal Processing Letters, 2019, 26(4): 538-542.
38	DAI T, CAI J R, ZHANG Y B, et al. Second-order attention network for single image super-resolution[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 11057-11066.
39	QIU Y J, WANG R X, TAO D P, et al. Embedded block residual network: A recursive restoration model for single-image super-resolution[C]//2019 IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 4179-4188.
40	HE X Y, MO Z T, WANG P S, et al. ODE-inspired network design for single image super-resolution[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Long Beach: IEEE, 2019: 1732-1741.
41	LAN R S, SUN L, LIU Z B, et al. Cascading and enhanced residual networks for accurate single-image super-resolution[J]. IEEE Transactions on Cybernetics, 2020, 51(1): 115-125.
42	TIMOFTE R, ROTHE R, VAN GOOL L. Seven ways to improve example-based single image super resolution[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1865-1873.

层名称	输入尺寸	输出尺寸	卷积核大小
1×1卷积层	H×W×64	H×W×48	1×1
组卷积层	H×W×48	H×W×48	1×1
通道搅乱层	H×W×48	H×W×48	—
1×1卷积层	H×W×48	H×W×64	1×1
平均池化层	1×1×64	1×1×64	—

层名称	输入尺寸	输出尺寸	卷积核大小
1×1卷积层	H×W×64	H×W×48	1×1
组卷积层	H×W×48	H×W×48	1×1
通道搅乱层	H×W×48	H×W×48	—
1×1卷积层	H×W×48	H×W×64	1×1
平均池化层	1×1×64	1×1×64	—

层名称	输入尺寸	输出尺寸	卷积核大小
3×3卷积层	H×W×64	H×W×64	3×3
Relu激活	H×W×64	H×W×64	5×5
5×5卷积层	H×W×64	H×W×64	—
Relu激活	H×W×64	H×W×64	1×1
合并(concat)	H×W×64	H×W×128	—
1×1卷积层	H×W×128	H×W×64	1×1

层名称	输入尺寸	输出尺寸	卷积核大小
3×3卷积层	H×W×64	H×W×64	3×3
Relu激活	H×W×64	H×W×64	5×5
5×5卷积层	H×W×64	H×W×64	—
Relu激活	H×W×64	H×W×64	1×1
合并(concat)	H×W×64	H×W×128	—
1×1卷积层	H×W×128	H×W×64	1×1

GFFC	×	√	×	×	√	√	×	√
CSAM	×	×	√	×	√	×	√	√
MSFFB	×	×	×	√	×	√	√	√
PSNR	32.14	32.17	32.15	32.20	32.22	32.26	32.27	32.32