基于深度学习的单帧图像超分辨率重建综述

doi:10.12263/DZXB.20220091

电子学报 ›› 2022, Vol. 50 ›› Issue (9): 2265-2294.DOI: 10.12263/DZXB.20220091

基于深度学习的单帧图像超分辨率重建综述

吴靖¹^,², 叶晓晶¹^,², 黄峰¹^,², 陈丽琼¹^,², 王志锋¹^,², 刘文犀²^,³

^1.福州大学机械工程及自动化学院，福建福州 350116
^2.福州大学先进技术创新研究院，福建福州 350116
^3.福州大学计算机与大数据学院，福建福州 350116

收稿日期:2022-01-17 修回日期:2022-05-13 出版日期:2022-09-25
- 作者简介:
- 吴靖男，1986年生.博士.福州大学机械工程及自动化学院副教授、硕士生导师.主要研究方向为计算机视觉、光机电算一体化.E-mail: wujing@fzu.edu.cn
  叶晓晶女，1998年生.福州大学机械工程及自动化学院硕士研究生.主要研究方向为图像处理、图像超分辨率重建.E-mail: yexiaojingyxj@163.com
  黄峰（通讯作者）男，1979年生.博士.福州大学机械工程及自动化学院研究员、博士生导师.主要研究方向为机器视觉、光学精密测量与成像、智能光学感知.
  陈丽琼女，1994年生.博士.福州大学机械工程及自动化学院讲师、硕士生导师.主要研究方向为计算机视觉、超分辨率重建、目标检测.E-mail: liqiongchen@fzu.edu.cn
  王志锋男，1996年生.福州大学机械工程及自动化学院硕士研究生.主要研究方向为深度学习、图像超分辨重建、图像增强处理.E-mail: 15880277679@139.com
  刘文犀男，1986年生.博士.福州大学计算机与大数据学院教授、博士生导师.主要研究方向为计算机视觉.E-mail: wenxi.liu@hotmail.com
- 基金资助:
- 国家自然科学基金 (62072110)

A Review of Single Image Super-Resolution Reconstruction Based on Deep Learning

WU Jing¹^,², YE Xiao-jing¹^,², HUANG Feng¹^,², CHEN Li-qiong¹^,², WANG Zhi-feng¹^,², LIU Wen-xi²^,³

^1.School of Mechanical Engineering and Automation，Fuzhou University，Fuzhou，Fujian 350116，China
^2.Advanced Technology Innovation Institute，Fuzhou University，Fuzhou，Fujian 350116，China
^3.College of Computer and Data Science，Fuzhou University，Fuzhou，Fujian 350116，China

Received:2022-01-17 Revised:2022-05-13 Online:2022-09-25 Published:2022-10-26

摘要/Abstract

摘要：

图像超分辨率重建是计算机视觉中的基本图像处理技术之一，不仅可以提高图像分辨率改善图像质量，还可以辅助其他计算机视觉任务. 近年来，随着人工智能浪潮的兴起，基于深度学习的图像超分辨率重建也取得了显著进展. 本文在简述图像超分辨率重建方法的基础上，全面综述了基于深度学习的单帧图像超分辨率重建的技术架构及研究历程，包括数据集构建方式、网络模型基本框架以及用于图像质量评估的主、客观评价指标，重点介绍了根据网络结构及图像重建效果划分的基于卷积神经网络的方法、基于生成对抗网络的方法以及基于Transformer的方法，并对相关网络模型加以评述和对比，最后依据网络模型和超分辨率重建挑战赛相关内容，展望了图像超分辨率重建未来的发展趋势.

关键词: 超分辨率重建, 深度学习, 单帧图像, 卷积神经网络, 生成对抗网络, Transformer, 挑战赛

Abstract:

Image super-resolution reconstruction is one of the basic image processing techniques in computer vision, which can not only improve image resolution and image quality, but also assist other computer vision tasks. In recent years, with the rise of artificial intelligence, deep-learning-based image super-resolution reconstruction has also made remarkable progress. Based on a brief description of the image super-resolution reconstruction methodology, this paper comprehensively reviews the technical architecture and research process of deep-learning-based single image super-resolution reconstruction, including the method of datasets construction, the basic framework of the network model, the subjective and objective evaluation metrics for image quality evaluation. The methods based on convolutional neural networks, generative adversarial networks and Transformer, which are divided according to network structure and image reconstruction effect are mainly introduced, and related network models are reviewed and compared. Finally, the future development trend of image super-resolution reconstruction is prospected according to the related content of network model and super-resolution reconstruction challenges.

Key words: super-resolution reconstruction, deep learning, single image, convolutional neural network, generative adversarial network, transformer, challenges

中图分类号:

TP751

吴靖, 叶晓晶, 黄峰, 陈丽琼, 王志锋, 刘文犀. 基于深度学习的单帧图像超分辨率重建综述[J]. 电子学报, 2022, 50(9): 2265-2294.

WU Jing, YE Xiao-jing, HUANG Feng, CHEN Li-qiong, WANG Zhi-feng, LIU Wen-xi. A Review of Single Image Super-Resolution Reconstruction Based on Deep Learning[J]. Acta Electronica Sinica, 2022, 50(9): 2265-2294.

图/表 26

参考文献 158

1	DAI D X, WANG Y J, CHEN Y H, et al. Is image super-resolution helpful for other vision tasks?[C]//2016 IEEE Winter Conference on Applications of Computer Vision. Lake Placid: IEEE, 2016: 1-9.
2	BAI Y C, ZHANG Y Q, DING M L, et al. SOD-MTGAN: Amall object detection via multi-task generative adversarial network[C]//Proceedings of the European Conference on Computer Vision(ECCV 2018). Munich: Springer, 2018: 210-226.
3	BEI Y J, DAMIAN A, HU S J, et al. New techniques for preserving global structure and denoising with low information loss in single-image super-resolution[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops(CVPRW). Salt Lake City: IEEE, 2018: 987-9877.
4	吴秀秀, 肖珊, 张煜. 基于配准的肺4D-CT图像超分辨率重建[J]. 电子学报, 2015, 43(2): 383-386.
	WU X X, XIAO S, ZHANG Y. Registration based super-resolution reconstruction for lung 4D-CT image[J]. Acta Electronica Sinica, 2015, 43(2): 383-386. (in Chinese)
5	YIN Y, ROBINSON J, ZHANG Y L, et al. Joint super-resolution and alignment of tiny faces[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(7): 12693-12700.
6	HUANG Y W, ZHENG F, WANG D Y, et al. Super-resolution and inpainting with degraded and upgraded generative adversarial networks[C]//Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence. Yokohama: ACM, 2021: 645-651.
7	WANG Z Y, JIANG K, YI P, et al. Ultra-dense GAN for satellite imagery super-resolution[J]. Neurocomputing, 2020, 398: 328-337.
8	GAO G W, YU Y, XIE J, et al. Constructing multilayer locality-constrained matrix regression framework for noise robust face super-resolution[J]. Pattern Recognition, 2021, 110: 107539.
9	李云红, 王珍, 张凯兵, 等. 基于学习的图像超分辨重建方法综述[J]. 计算机工程与应用, 2018, 54(15): 13-21.
	LI Y H, WANG Z, ZHANG K B, et al. Survey on example learning-based single image super-resolution technique[J]. Computer Engineering and Applications, 2018, 54(15): 13-21. (in Chinese)
10	孙旭, 李晓光, 李嘉锋, 等. 基于深度学习的图像超分辨率复原研究进展[J]. 自动化学报, 2017, 43(5): 697-709.
	SUN X, LI X G, LI J F, et al. Review on deep learning based image super-resolution restoration algorithms[J]. Acta Automatica Sinica, 2017, 43(5): 697-709. (in Chinese)
11	刘颖, 朱丽, 林庆帆, 等. 图像超分辨率技术的回顾与展望[J]. 计算机科学与探索, 2020, 14(2): 181-199.
	LIU Ying, ZHU Li, LIM K P, et al. Review and prospect of image super-resolution technology[J]. Journal of Frontiers of Computer Science and Technology, 2020, 14(2): 181-199. (in Chinese)
12	HARRIS J L. Diffraction and resolving power[J]. Journal of the Optical Society of America, 1964, 54(7): 931-936.
13	GOODMAN J W, COX M E. Introduction to fourier optics[J]. Physics Today, 1969, 22(4): 97-101.
14	TSAI R Y, HUANG T S. Multiframe image restoration and registration[J]. Advances in Computer Vision and Image Processing, 1984, 1(2): 317-339.
15	DUCHON C E. Lanczos filtering in one and two dimensions[J]. Journal of Applied Meteorology, 1979, 18(8): 1016-1022.
16	KEYS R. Cubic convolution interpolation for digital image processing[J]. IEEE Transactions on Acoustics, Speech, and Signal Processing, 1981, 29(6): 1153-1160.
17	STARK H, OSKOUI P. High-resolution image recovery from image-plane arrays, using convex projections[J]. Journal of the Optical Society of America. A, Optics and Image Science, 1989, 6(11): 1715-1726.
18	IRANI M, PELEG S. Improving resolution by image registration[J]. CVGIP: Graphical Models and Image Processing, 1991, 53(3): 231-239.
19	SCHULTZ R R, STEVENSON R L. A Bayesian approach to image expansion for improved definition[J]. IEEE Transactions on Image Processing, 1994, 3(3): 233-242.
20	伍政华, 孙明健, 顾宗山, 等. 基于二阶广义方向性全变分的图像超分辨率重建方法[J]. 电子学报, 2017, 45(11): 2625-2632.
	WU Z H, SUN M J, GU Z S, et al. Second-order directional total generalized variation regularization for image super-resolution[J]. Acta Electronica Sinica, 2017, 45(11): 2625-2632. (in Chinese)
21	王相海, 赵晓阳, 毕晓昀, 等. 小波域多角度轮廓模板变分模型的单幅图像超分辨率重建[J]. 电子学报, 2018, 46(9): 2256-2262.
	WANG X H, ZHAO X Y, BI X Y, et al. Single image super-resolution reconstruction approach based on multi-angle contour templates variational calculus model in wavelet domain[J]. Acta Electronica Sinica, 2018, 46(9): 2256-2262. (in Chinese)
22	FREEMAN W T, PASZTOR E C, CARMICHAEL O T. Learning low-level vision[J]. International Journal of Computer Vision, 2000, 40: 25-47.
23	CHANG H, YEUNG D Y, XIONG Y M. Super-resolution through neighbor embedding[C]//Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington: IEEE, 2004: 275-282.
24	YANG J C, WRIGHT J, HUANG T, et al. Image super-resolution as sparse representation of raw image patches[C]//2008 IEEE Conference on Computer Vision and Pattern Recognition. Anchorage: IEEE, 2008: 1-8.
25	TIMOFTE R, DE V, GOOL L V. Anchored neighborhood regression for fast example-based super-resolution[C]//2013 IEEE International Conference on Computer Vision. Sydney: IEEE, 2013: 1920-1927.
26	潘宗序, 禹晶, 肖创柏, 等. 基于自适应多字典学习的单幅图像超分辨率算法[J]. 电子学报, 2015, 43(2): 209-216.
	PAN Z X, YU J, XIAO C B, et al. Single image super resolution based on adaptive multi-dictionary learning[J]. Acta Electronica Sinica, 2015, 43(2): 209-216. (in Chinese)
27	PARK S C, PARK M K, KANG M G. Super-resolution image reconstruction: A technical overview[J]. IEEE Signal Processing Magazine, 2003, 20(3): 21-36.
28	江静, 张雪松. 图像超分辨率重建算法综述[J]. 红外技术, 2012, 34(1): 24-30.
	JIANG J, ZHANG X S. A review of super-resolution reconstruction algorithms[J]. Infrared Technology, 2012, 34(1): 24-30. (in Chinese)
29	苏衡, 周杰, 张志浩. 超分辨率图像重建方法综述[J]. 自动化学报, 2013, 39(8): 1202-1213.
	SU H, ZHOU J, ZHANG Z H. Survey of super-resolution image reconstruction methods[J]. Acta Automatica Sinica, 2013, 39(8): 1202-1213. (in Chinese)
30	YANG W M, ZHANG X C, TIAN Y P, et al. Deep learning for single image super-resolution: A brief review[J]. IEEE Transactions on Multimedia, 2019, 21(12): 3106-3121.
31	史振威, 雷森. 图像超分辨重建算法综述[J]. 数据采集与处理, 2020, 35(1): 1-20.
	SHI Z W, LEI S. Review of image super-resolution reconstruction[J]. Journal of Data Acquisition and Processing, 2020, 35(1): 1-20. (in Chinese)
32	张德, 林青宇, 郭茂祖. 单幅图像超分辨重建的深度学习方法综述[J]. 计算机工程与应用, 2021, 57(22): 28-41.
	ZHANG D, LIN Q Y, GUO M Z. Review of single image super-resolution based on deep learning[J]. Computer Engineering and Applications, 2021, 57(22): 28-41. (in Chinese)
33	黄健, 赵元元, 郭苹, 等. 深度学习的单幅图像超分辨率重建方法综述[J]. 计算机工程与应用, 2021, 57(18): 13-23.
	HUANG J, ZHAO Y Y, GUO P, et al. Survey of single image super-resolution based on deep learning[J]. Computer Engineering and Applications, 2021, 57(18): 13-23. (in Chinese)
34	YANG Z Y, SHI P, PAN D. A survey of super-resolution based on deep learning[C]//2020 International Conference on Culture-oriented Science & Technology(ICCST). Beijing: IEEE, 2020: 514-518.
35	李佳星, 赵勇先, 王京华. 基于深度学习的单幅图像超分辨率重建算法综述[J]. 自动化学报, 2021, 47(10): 2341-2363.
	LI J X, ZHAO Y X, WANG J H. A review of single image super-resolution reconstruction algorithms based on deep learning[J]. Acta Automatica Sinica, 2021, 47(10): 2341-2363. (in Chinese)
36	唐艳秋, 潘泓, 朱亚平, 等. 图像超分辨率重建研究综述[J]. 电子学报, 2020, 48(7): 1407-1420.
	TANG Y Q, PAN H, ZHU Y P, et al. A survey of image super-resolution reconstruction[J]. Acta Electronica Sinica, 2020, 48(7): 1407-1420. (in Chinese)
37	WANG Z H, CHEN J, HOI S C H. Deep learning for image super-resolution: A survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(10): 3365-3387.
38	ANWAR S, KHAN S, BARNES N. A deep journey into super-resolution[J]. ACM Computing Surveys, 2021, 53(3): 1-34.
39	CHEN H G, HE X H, QING L B, et al. Real-world single image super-resolution: A brief review[J]. Information Fusion, 2022, 79: 124-145.
40	AGUSTSSON E, TIMOFTE R. NTIRE 2017 challenge on single image super-resolution: Dataset and study[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Honolulu: IEEE, 2017: 1122-1131.
41	GU S H, LUGMAYR A, DANELLJAN M, et al. DIV8K: DIVerse 8K resolution image dataset[C]//2019 IEEE/CVF International Conference on Computer Vision Workshop(ICCVW). Seoul: IEEE, 2019: 3512-3516.
42	CAI J R, ZENG H, YONG H W, et al. Toward real-world single image super-resolution: A new benchmark and a new model[C]//2019 IEEE/CVF International Conference on Computer Vision(ICCV). Seoul: IEEE, 2019: 3086-3095.
43	WEI P X, XIE Z W, LU H N, et al. Component divide-and-conquer for real-world image super-resolution[C]//Computer Vision-ECCV 2020. Glasgow: Springer, 2020: 101-117.
44	ZHANG K, ZUO W M, ZHANG L. Learning a single convolutional super-resolution network for multiple degradations[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 3262-3271.
45	GU J J, LU H N, ZUO W M, et al. Blind super-resolution with iterative kernel correction[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Long Beach: IEEE, 2019: 1604-1613.
46	LUO Z X, HUANG Y, LI S, et al. Unfolding the alternating optimization for blind super resolution[EB/OL]. (2020-11-25)[2022-01-16]. .
47	ZHANG K, LIANG J Y, VAN GOOL L, et al. Designing a practical degradation model for deep blind image super-resolution[C]//2021 IEEE/CVF International Conference on Computer Vision(ICCV). Montreal: IEEE, 2021: 4771-4780.
48	WANG X T, XIE L B, DONG C, et al. Real-ESRGAN: Training real-world blind super-resolution with pure synthetic data[EB/OL]. (2021-08-17)[2022-01-16]. .
49	BELL-KLIGLER S, SHOCHER A, IRANI M. Blind super-resolution kernel estimation using an Internal-GAN[C]//Proceedings of the 33rd International Conference on Neural Information Processing Systems(NeurIPS). Vancouver: Curran Associates, Inc., 2019: 284-293.
50	FRITSCHE M, GU S H, TIMOFTE R. Frequency separation for real-world super-resolution[C]//2019 IEEE/CVF International Conference on Computer Vision Workshop(ICCVW). Seoul: IEEE, 2019: 3599-3608.
51	JI X Z, TAO G P, CAO Y, et al. Frequency consistent adaptation for real world super resolution[EB/OL]. (2020-12-18)[2022-01-16]. .
52	YANG J C, WRIGHT J, HUANG T S, et al. Image super-resolution via sparse representation[J]. IEEE Transactions on Image Processing, 2010, 19(11): 2861-2873.
53	TIMOFTE R, DE SMET V, VAN GOOL L. A+: Adjusted anchored neighborhood regression for fast super-resolution[C]//Computer Vision-ACCV 2014. Singapore: Springer, 2015: 111-126.
54	SCHULTER S, LEISTNER C, BISCHOF H. Fast and accurate image upscaling with super-resolution forests[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 3791-3799.
55	KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1646-1654.
56	DONG C, LOY C C, TANG X O. Accelerating the super-resolution convolutional neural network[C]//Computer Vision-ECCV 2016. Amsterdam: Springer, 2016: 391-407.
57	LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft COCO: Common objects in context[C]//Computer Vision-ECCV 2014. Zurich: Springer, 2014: 740-755.
58	DENG J, DONG W, SOCHER R, et al. ImageNet: A large-scale hierarchical image database[C]//2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami: IEEE, 2009: 248-255.
59	TIMOFTE R, AGUSTSSON E, Van GOOL L, et al. NTIRE 2017 challenge on single image super-resolution: Methods and results[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops(CVPRW). Honolulu: IEEE, 2017: 1110-1121.
60	JI X Z, CAO Y, TAI Y, et al. Real-world super-resolution via kernel estimation and noise injection[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops(CVPRW). Seattle: IEEE, 2020: 1914-1923.
61	IGNATOV A, KOBYSHEV N, TIMOFTE R, et al. DSLR-quality photos on mobile devices with deep convolutional networks[C]//2017 IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 3297-3305.
62	WANG X T, YU K, DONG C, et al. Recovering realistic texture in image super-resolution by deep spatial feature transform[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 606-615.
63	BEVILACQUA M, ROUMY A, GUILLEMOT C, et al. Low-complexity single-image super-resolution based on nonnegative neighbor embedding[C]//Proceedings of the British Machine Vision Conference 2012. Guildford: BMVA Press, 2012: 135.1-135.10
64	ZEYDE R, ELAD M, PROTTER M. On single image scale-up using sparse-representations[C]//Proceedings of the 7th International Conference on Curves and Surfaces. Avignon: Springer, 2012: 711-730.
65	MARTIN D, FOWLKES C, TAL D, et al. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[C]//Proceedings Eighth IEEE International Conference on Computer Vision(ICCV). Vancouver: IEEE, 2001: 416-423.
66	HUANG J B, SINGH A, AHUJA N. Single image super-resolution from transformed self-exemplars[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 5197-5206.
67	FUJIMOTO A, OGAWA T, YAMAMOTO K, et al. Manga109 dataset and creation of metadata[C]//Proceedings of the 1st International Workshop on coMics ANalysis, Processing and Understanding. Cancun: ACM, 2016: (2) 1-5.
68	BLAU Y, MECHREZ R, TIMOFTE R, et al. The 2018 PIRM challenge on perceptual image super-resolution[C]//Lecture Notes in Computer Science. Munich: Springer, 2019: 334-355.
69	DONG C, LOY C C, HE K M, et al. Learning a deep convolutional network for image super-resolution[C]//Computer Vision-ECCV 2014. Zurich: Springer, 2014: 184-199.
70	DONG C, LOY C C, HE K M, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295-307.
71	KIM J, LEE J K, LEE K M. Deeply-recursive convolutional network for image super-resolution[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1637-1645.
72	HU X C, MU H Y, ZHANG X Y, et al. Meta-SR: A magnification-arbitrary network for super-resolution[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Long Beach: IEEE, 2019: 1575-1584.
73	SHI W Z, CABALLERO J, HUSZÁR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1874-1883.
74	FAN Y C, SHI H H, YU J H, et al. Balanced two-stage residual networks for image super-resolution[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Honolulu: IEEE, 2017: 1157-1164.
75	ZHANG Y, LI K, LI K, et al. Residual non-local attention networks for image restoration[C]//Proceedings of the Seventh International Conference on Learning Representations(ICLR 2019). New Orleans: ICLR Press, 2019: 1-18.
76	LAI W S, HUANG J B, AHUJA N, et al. Deep laplacian pyramid networks for fast and accurate super-resolution[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Honolulu: IEEE, 2017: 5835-5843.
77	LAI W S, HUANG J B, AHUJA N, et al. Fast and accurate image super-resolution with deep Laplacian pyramid networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(11): 2599-2613.
78	CHUDASAMA V, UPLA K. E-ProSRNet: An enhanced progressive single image super-resolution approach[J]. Computer Vision and Image Understanding, 2020, 200: 103038.
79	HARIS M, SHAKHNAROVICH G, UKITA N. Deep back-projection networks for super-resolution[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 1664-1673.
80	HARIS M, SHAKHNAROVICH G, UKITA N. Deep back-Projection Networks for single image super-resolution[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(12): 4323-4337.
81	HAN W, CHANG S Y, LIU D, et al. Image super-resolution via dual-state recurrent networks[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 1654-1663.
82	LI Z, YANG J L, LIU Z, et al. Feedback network for image super-resolution[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Long Beach: IEEE, 2019: 3862-3871.
83	LUGMAYR A, DANELLJAN M, TIMOFTE R, et al. NTIRE 2020 challenge on real-world image super-resolution: Methods and results[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops(CVPRW). Seattle: IEEE, 2020: 2058-2076.
84	STREIJL R C, WINKLER S, HANDS D S. Mean opinion score(MOS) revisited: Methods and applications, limitations and alternatives[J]. Multimedia Systems, 2016, 22(2): 213-227.
85	张凯兵, 朱丹妮, 王珍, 等. 超分辨图像质量评价综述[J]. 计算机工程与应用, 2019, 55(4): 31-40, 47.
	ZHANG K B, ZHU D N, WANG Z, et al. Survey of super-resolution images quality assessment[J]. Computer Engineering and Applications, 2019, 55(4): 31-40, 47. (in Chinese)
86	高敏娟, 党宏社, 魏立力, 等. 全参考图像质量评价回顾与展望[J]. 电子学报, 2021, 49(11): 2261-2272.
	GAO M J, DANG H S, WEI L L, et al. Review and prospect of full reference image quality assessment[J]. Acta Electronica Sinica, 2021, 49(11): 2261-2272. (in Chinese)
87	WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612.
88	SHEIKH H R, BOVIK A C, DE VECIANA G. An information fidelity criterion for image quality assessment using natural scene statistics[J]. IEEE Transactions on Image Processing, 2005, 14(12): 2117-2128.
89	ZHANG R, ISOLA P, EFROS A A, et al. The unreasonable effectiveness of deep features as a perceptual metric[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 586-595.
90	MITTAL A, SOUNDARARAJAN R, BOVIK A C. Making a “completely blind” image quality analyzer[J]. IEEE Signal Processing Letters, 2013, 20(3): 209-212.
91	VENKATANATH N, PRANEETH D, BH M C, et al. Blind image quality evaluation using perception based features[C]//2015 Twenty First National Conference on Communications(NCC). Mumbai: IEEE, 2015: 1-6.
92	MA C, YANG C Y, YANG X K, et al. Learning a no-reference quality metric for single-image super-resolution[J]. Computer Vision and Image Understanding, 2017, 158: 1-16.
93	LEDIG C, THEIS L, HUSZÁR F, et al. Photo-realistic single image super-resolution using a generative adversarial network[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 105-114.
94	CHEN H T, WANG Y H, GUO T Y, et al. Pre-trained image processing transformer[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Nashville: IEEE, 2021: 12294-12305.
95	HUBEL D H, WIESEL T N. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex[J]. The Journal of Physiology, 1962, 160(1): 106-154.
96	LI Z W, LIU F, YANG W J, et al. A survey of convolutional neural networks: Analysis, applications, and prospects[J]. IEEE Transactions on Neural Networks and Learning Systems, 2021: 1-21.
97	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778.
98	MAO X, SHEN C, YANG Y. Image restoration using convolutional auto-encoders with symmetric skip connections[C]//Proceedings of the Advances in Neural Information Processing Systems 29(NIPS 2016)[C]. Barcelona: Curran Associates, Inc., 2016.
99	LIM B, SON S, KIM H, et al. Enhanced deep residual networks for single image super-resolution[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops(CVPRW). Honolulu: IEEE, 2017: 1132-1140.
100	LI J, FANG F, MEI K, et al. Multi-scale residual network for image super-resolution[C]//Proceedings of the European Conference on Computer Vision. Munich: Springer, 2018: 527-542.
101	LAN R S, SUN L, LIU Z B, et al. Cascading and enhanced residual networks for accurate single-image super-resolution[J]. IEEE Transactions on Cybernetics, 2021, 51(1): 115-125.
102	南方哲, 钱育蓉, 行艳妮, 等. 基于深度学习的单图像超分辨率重建研究综述[J]. 计算机应用研究, 2020, 37(2): 321-326.
	NAN F Z, QIAN Y R, XING Y N, et al. Survey of single image super resolution based on deep learning[J]. Application Research of Computers, 2020, 37(2): 321-326. (in Chinese)
103	杨祎玥, 伏潜, 万定生. 基于深度循环神经网络的时间序列预测模型[J]. 计算机技术与发展, 2017, 27(3): 35-38, 43.
	YANG Y Y, FU Q, WAN D S. A prediction model for time series based on deep recurrent neural network[J]. Computer Technology and Development, 2017, 27(3): 35-38, 43. (in Chinese)
104	TAI Y, YANG J, LIU X M. Image super-resolution via deep recursive residual network[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Honolulu: IEEE, 2017: 2790-2798.
105	HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely connected convolutional networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Honolulu: IEEE, 2017: 2261-2269.
106	TONG T, LI G, LIU X J, et al. Image super-resolution using dense skip connections[C]//2017 IEEE International Conference on Computer Vision(ICCV). Venice: IEEE, 2017: 4809-4817.
107	TAI Y, YANG J, LIU X M, et al. MemNet: A persistent memory network for image restoration[C]//2017 IEEE International Conference on Computer Vision(ICCV). Venice: IEEE, 2017: 4549-4557.
108	SHAMSOLMOALI P, ZAREAPOOR M, ZHANG J H, et al. Image super resolution by dilated dense progressive network[J]. Image and Vision Computing, 2019, 88: 9-18.
109	PAN Z, LI B, XI T, et al. Real image super resolution via heterogeneous model ensemble using GP-NAS[C]//Proceedings of the European Conference on Computer Vision(ECCV). Glasgow: Springer, 2020: 423-436.
110	JIANG K, WANG Z Y, YI P, et al. Hierarchical dense recursive network for image super-resolution[J]. Pattern Recognition, 2020, 107: 107475.
111	CHAUDHARI S, MITHAL V, POLATKAN G, et al. An attentive survey of attention models[J]. ACM Transactions on Intelligent Systems and Technology, 2021, 12(5): 53(1-32.
112	HU J, SHEN L, ALBANIE S, et al. Squeeze-and-excitation networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020,42(8): 2011-2023.
113	ZHANG Y, LI K, LI K, et al. Image super-resolution using very deep residual channel attention networks[C]//Proceedings of the European Conference on Computer Vision(ECCV). Munich: Springer, 2018: 294-310.
114	DAI T, CAI J R, ZHANG Y B, et al. Second-order attention network for single image super-resolution[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Long Beach: IEEE, 2019: 11057-11066.
115	AHN N, KANG B, Fast SOHN K., accurate, and lightweight super-resolution with cascading residual network[C]//Proceedings of the European Conference on Computer Vision(ECCV). Munich: Springer, 2018: 256-272.
116	HUI Z, WANG X M, GAO X B. Fast and accurate single image super-resolution via information distillation network[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 723-731.
117	HUI Z, GAO X, YANG Y, et al. Lightweight image super-resolution with information multi-distillation network[C]//Proceedings of the 27th ACM International Conference on Multimedia. Nice: ACM, 2019: 2024-2032.
118	LIU J, TANG J, WU G. Residual feature distillation network for lightweight image super-resolution[C]//Proceedings of the European Conference on Computer Vision(ECCV). Glasgow: Springer, 2020: 41-55.
119	CHU X X, ZHANG B, MA H L, et al. Fast, accurate and lightweight super-resolution with neural architecture search[C]//2020 25th International Conference on Pattern Recognition(ICPR). Milan: IEEE, 2021: 59-64.
120	LI W B, ZHOU K, QI L, et al. LAPAR: Linearly-assembled pixel-adaptive regression network for single image super-resolution and beyond[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. Vancouver: ACM, 2020: 20343-20355.
121	WANG L G, DONG X Y, WANG Y Q, et al. Exploring sparsity in image super-resolution for efficient inference[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Nashville: IEEE, 2021: 4915-4924.
122	RADOSAVOVIC I, KOSARAJU R P, GIRSHICK R, et al. Designing network design spaces[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Seattle: IEEE, 2020: 10425-10433.
123	ZHANG K, DANELLJAN M, LI Y, et al. AIM 2020 challenge on efficient super-resolution: methods and results[C]//Proceedings of the European Conference on Computer Vision(ECCV). Glasgow: Springer, 2020: 5-40.
124	GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems(NIPS 2014). Montreal: MIT Press, 2014: 2672-2680.
125	SAJJADI M S M, SCHÖLKOPF B, HIRSCH M. EnhanceNet: single image super-resolution through automated texture synthesis[C]//2017 IEEE International Conference on Computer Vision(ICCV). Venice: IEEE, 2017: 4501-4510.
126	WANG X, YU K, WU S, et al. ESRGAN: enhanced super-resolution generative adversarial networks[C]//Proceedings of the European Conference on Computer Vision(ECCV). Munich: Springer, 2018: 63-79.
127	WANG X T, YU K, DONG C, et al. Deep network interpolation for continuous imagery effect transition[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Long Beach: IEEE, 2019: 1692-1701.
128	SHANG T Z, DAI Q J, ZHU S C, et al. Perceptual extreme super resolution network with receptive field block[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops(CVPRW). Seattle: IEEE, 2020: 1778-1787.
129	ZHANG W L, LIU Y H, DONG C, et al. RankSRGAN: generative adversarial networks with ranker for image super-resolution[C]//2019 IEEE/CVF International Conference on Computer Vision(ICCV). Seoul: IEEE, 2019: 3096-3105.
130	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach: Curran Associates Inc., 2017: 6000-6010.
131	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: Transformers for image recognition at scale[C]//Proceedings of the International Conference on Learning Representations(ICLR 2021). London: ICLR Press, 2021: 1-22.
132	CARION N, MASSA F, SYNNAEVE G, et al. End-to-end object detection with transformers[C]//Proceedings of the European Conference on Computer Vision(ECCV 2020). Glasgow: Springer, 2020: 213-229.
133	ARNAB A, DEHGHANI M, HEIGOLD G, et al. ViViT: A video vision transformer[C]//2021 IEEE/CVF International Conference on Computer Vision(ICCV). Montreal: IEEE, 2021: 6816-6826.
134	BI J R, ZHU Z L, MENG Q L. Transformer in computer vision[C]//2021 IEEE International Conference on Computer Science, Electronic Information Engineering and Intelligent Control Technology(CEI). Fuzhou: IEEE, 2021: 178-188.
135	LIANG J Y, CAO J Z, SUN G L, et al. SwinIR: image restoration using swin transformer[C]//2021 IEEE/CVF International Conference on Computer Vision Workshops(ICCVW). Montreal: IEEE, 2021: 1833-1844.
136	LIU Z, LIN Y T, CAO Y, et al. Swin transformer: hierarchical vision transformer using shifted windows[C]//2021 IEEE/CVF International Conference on Computer Vision(ICCV). Montreal: IEEE, 2021: 9992-10002.
137	LU Z, LIU H, LI J, et al. Efficient transformer for single image super-resolution[EB/OL]. (2021-10-28)[2022-01-16]. .
138	SHOCHER A, COHEN N, IRANI M. Zero-shot super-resolution using deep internal learning[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Salt Lake City: IEEE, 2018: 3118-3126.
139	BULAT A, YANG J, TZIMIROPOULOS G. To learn image super-resolution, use a GAN to learn how to do image degradation first[C]//Proceedings of the 15th European Conference on Computer Vision(ECCV). Munich: Springer, 2018: 187-202.
140	YUAN Y, LIU S Y, ZHANG J W, et al. Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops(CVPRW). Salt Lake City: IEEE, 2018: 814-81409.
141	ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//2017 IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 2242-2251.
142	ZHANG Y B, LIU S Y, DONG C, et al. Multiple cycle-in-cycle generative adversarial networks for unsupervised image super-resolution[J]. IEEE Transactions on Image Processing, 2020, 29: 1101-1112.
143	WU S, DONG C, QIAO Y. Blind image restoration based on cycle-consistent network[J/OL]. IEEE Transactions On Multimedia, 2022. DOI: 10.1109/TMM.2021 .
	3139209.
144	TIMOFTE R, ROTHE R, VAN GOOL L. Seven ways to improve example-based single image super resolution[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Las Vegas: IEEE, 2016: 1865-1873.
145	TIMOFTE R, GU S H, WU J Q, et al. NTIRE 2018 challenge on single image super-resolution: Methods and results[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops(CVPRW). Salt Lake City: IEEE, 2018: 965-96511.
146	CAI J R, GU S H, TIMOFTE R, et al. NTIRE 2019 challenge on real image super-resolution: Methods and results[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops(CVPRW). Long Beach: IEEE, 2019: 2211-2223.
147	LUGMAYR A, DANELLJAN M, TIMOFTE R, et al. AIM 2019 challenge on real-world image super-resolution: Methods and results[C]//2019 IEEE/CVF International Conference on Computer Vision Workshop(ICCVW). Seoul: IEEE, 2019: 3575-3583.
148	ZHANG K, GU S H, TIMOFTE R, et al. AIM 2019 challenge on constrained super-resolution: Methods and results[C]//2019 IEEE/CVF International Conference on Computer Vision Workshop(ICCVW). Seoul: IEEE, 2019: 3565-3574.
149	GU S H, DANELLJAN M, TIMOFTE R, et al. AIM 2019 challenge on image extreme super-resolution: Methods and results[C]//2019 IEEE/CVF International Conference on Computer Vision Workshop(ICCVW). Seoul: IEEE, 2019: 3556-3564.
150	ZHANG K, GU S, TIMOFTE R, et al. NTIRE 2020 challenge on perceptual extreme super-resolution: Methods and results[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops(CVPRW). Seattle: IEEE, 2020: 2045-2057.
151	WEI P, LU H, TIMOFTE R, et al. AIM 2020 challenge on real image super-resolution: methods and results[C]//Proceedings of the European Conference on Computer Vision(ECCV). Berlin: Springer, 2020: 392-422.
152	BHAT G, DANELLJAN M, TIMOFTE R, et al. NTIRE 2021 challenge on burst super-resolution: Methods and results[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops(CVPRW). Nashville: IEEE, 2021: 613-626.
153	IGNATOV A, TIMOFTE R, DENNA M, et al. Real-time quantized image super-resolution on mobile NPUs, mobile AI 2021 challenge: Report[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops(CVPRW). Nashville: IEEE, 2021: 2525-2534.
154	FAN Y C, YU J H, LIU D, et al. Scale-wise convolution for image restoration[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(7): 10770-10777.
155	YOO J, CHEN Q. SinIR: Efficient general image manipulation with single image reconstruction[C]//Proceedings of the 38th International Conference on Machine Learning(ICML). Champaign: PMLR, 2021: 12040-12050.
156	KONG X T, ZHAO H Y, QIAO Y, et al. ClassSR: A general framework to accelerate super-resolution networks by data characteristic[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Nashville: IEEE, 2021: 12011-12020.
157	WANG L G, WANG Y Q, LIN Z P, et al. Learning a single network for scale-arbitrary super-resolution[C]//2021 IEEE/CVF International Conference on Computer Vision(ICCV). Montreal: IEEE, 2021: 4781-4790.
158	AYAZOGLU M. Extremely lightweight quantization robust real-time single-image super resolution for mobile devices[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops(CVPRW). Nashville: IEEE, 2021: 2472-2479.

数据集名称	图像数量/张	合成/真实数据集 (训练集/验证集/测试集)	图像格式	图像内容
T91^[52]	91	合成数据集(训练集)	PNG	包括动植物、人、车等局部纹理图像
Timofte^[53]	110	合成数据集(训练集)	PNG	包括T91, Set5和Set14三个数据集的图像
291-images^[54,55]	291	合成数据集(训练集)	PNG, JPG	包括T91和BSD两个数据集的图像
General-100^[56]	100	合成数据集(训练集)	BMP	包括建筑纹理、动植物、人、食物、日用品等图像
MSCOCO^[57]	328 000	合成数据集(训练集)	JPG	包括91种易识别的物体类型在自然环境中的复杂日常场景的图像
ImageNet^[58]	3 200 000	合成数据集(训练集)	JPG	包括动植物、交通工具、家具、乐器、地质构造、工具等图像
DIV2K^[40]	1 000	合成数据集(800张训练集、100张验证集、100张测试集)	PNG	包括人、手工艺品、环境(城市、村庄)、动植物、自然风景等图像
Flickr2K^[59]	2 650	合成数据集(训练集)	PNG	包括人、车、动植物、食物、建筑和风景等图像
DF2K^[60]	3 450	合成数据集(训练集)	PNG	包括DIV2K和Flickr2K两个数据集的图像
DPED^[61]	5 827	真实数据集(5 614张训练集、113张验证集、100张测试集)	PNG	包括各种道路交通场景(建筑、植物、道路等)等图像
OutdoorScene(OST)^[62]	10 624	合成数据集(10 324张训练集、300张测试集,即OST300)	PNG	包括动物、建筑、草、山、植物、天空和水等7类纹理丰富的图像
DIV8K^[41]	1 504	真实数据集(1 304张训练集、100张验证集、100张测试集)	PNG	包括人、动物、建筑和风景等各种场景和物体的图像
RealSR^[42]	595	真实数据集(550张训练集、45张测试集)	PNG	包括建筑、风景、动植物、海报、物体等纹理丰富的室内外场景图像
DRealSR^[43]	884(×2) 783(×3) 840(×4)	真实数据集(对于尺度因子2, 3, 4分别用83, 84, 93张图像进行测试,其余用于训练)	PNG	包括广告海报、植物、办公室、建筑物等室内外场景的图像
Set5^[63]	5	合成数据集(测试集)	PNG	包括婴儿、鸟、蝴蝶、头部和女士等5张图像
Set14^[64]	14	合成数据集(测试集)	PNG	包括动植物、风景(船和桥)、PPT和人等图像
Berkeley Segmentation Dataset(BSD)/BSD500^[65]	500	合成数据集(测试集)	JPG	包括动物、建筑、食物、风景、人和植物等图像.BSD100和BSD300分别是BSD500中常用的100或300张图片
Urban100^[66]	100	合成数据集(测试集)	PNG	包括不同类型的建筑图像
Manga109^[67]	109	合成数据集(测试集)	PNG	包括来自日本漫画书的图像
PIRM^[68]	200	合成数据集(100张验证集、100张测试集)	PNG	包括人、物、环境、植物、自然风景等图像
DIV2KRK^[49]	100	合成数据集(测试集)	PNG	包括对DIV2K验证集的100张图像进行更复杂退化操作的图像
DIV2K4D^[47]	400	合成数据集(测试集)	PNG	包括对100张DIV2K验证集采取四种退化的图像

数据集名称	图像数量/张	合成/真实数据集 (训练集/验证集/测试集)	图像格式	图像内容
T91^[52]	91	合成数据集(训练集)	PNG	包括动植物、人、车等局部纹理图像
Timofte^[53]	110	合成数据集(训练集)	PNG	包括T91, Set5和Set14三个数据集的图像
291-images^[54,55]	291	合成数据集(训练集)	PNG, JPG	包括T91和BSD两个数据集的图像
General-100^[56]	100	合成数据集(训练集)	BMP	包括建筑纹理、动植物、人、食物、日用品等图像
MSCOCO^[57]	328 000	合成数据集(训练集)	JPG	包括91种易识别的物体类型在自然环境中的复杂日常场景的图像
ImageNet^[58]	3 200 000	合成数据集(训练集)	JPG	包括动植物、交通工具、家具、乐器、地质构造、工具等图像
DIV2K^[40]	1 000	合成数据集(800张训练集、100张验证集、100张测试集)	PNG	包括人、手工艺品、环境(城市、村庄)、动植物、自然风景等图像
Flickr2K^[59]	2 650	合成数据集(训练集)	PNG	包括人、车、动植物、食物、建筑和风景等图像
DF2K^[60]	3 450	合成数据集(训练集)	PNG	包括DIV2K和Flickr2K两个数据集的图像
DPED^[61]	5 827	真实数据集(5 614张训练集、113张验证集、100张测试集)	PNG	包括各种道路交通场景(建筑、植物、道路等)等图像
OutdoorScene(OST)^[62]	10 624	合成数据集(10 324张训练集、300张测试集,即OST300)	PNG	包括动物、建筑、草、山、植物、天空和水等7类纹理丰富的图像
DIV8K^[41]	1 504	真实数据集(1 304张训练集、100张验证集、100张测试集)	PNG	包括人、动物、建筑和风景等各种场景和物体的图像
RealSR^[42]	595	真实数据集(550张训练集、45张测试集)	PNG	包括建筑、风景、动植物、海报、物体等纹理丰富的室内外场景图像
DRealSR^[43]	884(×2) 783(×3) 840(×4)	真实数据集(对于尺度因子2, 3, 4分别用83, 84, 93张图像进行测试,其余用于训练)	PNG	包括广告海报、植物、办公室、建筑物等室内外场景的图像
Set5^[63]	5	合成数据集(测试集)	PNG	包括婴儿、鸟、蝴蝶、头部和女士等5张图像
Set14^[64]	14	合成数据集(测试集)	PNG	包括动植物、风景(船和桥)、PPT和人等图像
Berkeley Segmentation Dataset(BSD)/BSD500^[65]	500	合成数据集(测试集)	JPG	包括动物、建筑、食物、风景、人和植物等图像.BSD100和BSD300分别是BSD500中常用的100或300张图片
Urban100^[66]	100	合成数据集(测试集)	PNG	包括不同类型的建筑图像
Manga109^[67]	109	合成数据集(测试集)	PNG	包括来自日本漫画书的图像
PIRM^[68]	200	合成数据集(100张验证集、100张测试集)	PNG	包括人、物、环境、植物、自然风景等图像
DIV2KRK^[49]	100	合成数据集(测试集)	PNG	包括对DIV2K验证集的100张图像进行更复杂退化操作的图像
DIV2K4D^[47]	400	合成数据集(测试集)	PNG	包括对100张DIV2K验证集采取四种退化的图像

级别	绝对主观评价指标	相对主观评价指标
1	很好(第1名)	一组中最好(5分)
2	较好(第2名)	比该组的平均水平好(4分)
3	一般(第3名)	该组中平均水平(3分)
4	较差(第4名)	比该组的平均水平差(2分)
5	很差(第5名)	一组中最差(1分)

级别	绝对主观评价指标	相对主观评价指标
1	很好(第1名)	一组中最好(5分)
2	较好(第2名)	比该组的平均水平好(4分)
3	一般(第3名)	该组中平均水平(3分)
4	较差(第4名)	比该组的平均水平差(2分)
5	很差(第5名)	一组中最差(1分)

网络名称-- 发表时间(类型)	网络框架 (上采样方法)	训练集	LR图像获取方式	数据增强	测试集	损失函数	评价指标
SRCNN--2014 (基于S-CNN)	预上采样 (双三次插值)	91-images, ImageNet	Bicubic+GB	—	Set5, Set14	L₂	PSNR, SSIM, runtime
FSRCNN--2016 (基于S-CNN)	后上采样 (反卷积)	91-images, General-100	下采样	缩放、旋转	Set5, Set14, BSD200	L₂	PSNR, SSIM, IFC, runtime
ESPCN--2016 (基于S-CNN)	后上采样 (亚像素卷积)	ImageNet	下采样+GB	—	91-images, Set5, Set14, BSD300, BSD500, super texture	L₂	PSNR, runtime
VDSR--2016 (基于ResNet)	预上采样 (双三次插值)	291-images	下采样	旋转、翻转	Set5, Set14, Urban100, B100	L₂	PSNR, SSIM, runtime
RED-Net--2016 (基于ResNet)	预上采样 (双三次插值)	BSD300	下采样	旋转、翻转	Set5, Set14, BSD100	L₂	PSNR, SSIM
DRCN--2016 (基于RNN)	预上采样 (双三次插值)	91-images	下采样	—	Set5, Set14, B100, Urban100	L₂	PSNR, SSIM
LapSRN--2017 (基于ResNet)	渐进式上采样 (转置卷积)	291-images	Bicubic	缩放、旋转、翻转	Set5, Set14, BSDS100, Urban100, Manga109	L_Charbonnier	PSNR, SSIM, IFC
EDSR--2017 (基于ResNet)	后上采样 (亚像素卷积)	DIV2K	Bicubic	翻转、旋转	Set5, Set14, B100, Urban100, DIV2K	L₁	PSNR, SSIM
DRRN--2017 (基于RNN)	预上采样 (双三次插值)	291-images	下采样	翻转、旋转	Set5, Set14, BSD100, Urban100	L₂	PSNR, SSIM, IFC
SRDenseNet--2017 (基于DenseNet)	后上采样 (反卷积)	ImageNet	Bicubic	—	Set5, Set14, B100, Urban100	L₂	PSNR, SSIM
MemNet--2017 (基于DenseNet)	预上采样 (双三次插值)	291-images	Bicubic	翻转、旋转	Set5, Set14, BSD100, Urban100	L₂	PSNR, SSIM
SRGAN--2017 (基于GAN)	后上采样 (亚像素卷积)	ImageNet	Bicubic	—	Set5, Set14, BSD100	L_percep, L_GAN	PSNR, SSIM, MOS
EnhanceNet--2017 (基于GAN)	后上采样 (最近邻插值)	MSCOCO	Bicubic	—	Set5, Set14, BSD100, Urban100	L_percep, L_GAN, L_texture	PSNR, SSIM, IFC, MOS
DBPN--2018 (基于ResNet)	迭代式上下采样(反卷积)	DIV2K, Flickr2K, ImageNet	Bicubic	—	Set5, Set14, BSDS100, Urban100, Manga109	L₂	PSNR,SSIM
MSRN--2018 (基于ResNet)	后上采样 (亚像素卷积)	DIV2K	Bicubic	缩放、旋转、翻转	Set5, Set14, BSDS100, Urban100, Manga109	L₁	PSNR, SSIM
RCAN--2018 (基于AM)	后上采样 (亚像素卷积)	DIV2K	①Bicubic ②模糊下采样	旋转、翻转	Set5, Set14, B100, Urban100, Manga109	L₁	PSNR, SSIM
ESRGAN--2018 (基于GAN)	后上采样 (最近邻插值)	DIV2K, Flickr2K, OST	Bicubic	翻转、旋转	Set5, Set14, BSD100, Urban100, PIRM	L_percep, L_GAN, L₁	PSNR, PI
LP-KPN--2019 (基于ResNet)	渐进式上采 (shuffle upsample)	RealSR	—	旋转、翻转	RealSR	L₂	PSNR, SSIM
SRFBN--2019 (基于RNN)	后上采样 (反卷积)	DIV2K, Flickr2K	①Bicubic ②下采样+GB ③Bicubic+GN	翻转、旋转	Set5, Set14, B100, Urban100, Manga109	L₁	PSNR, SSIM
Meta-SR--2019 (基于DenseNet)	后上采样 (元上采样)	DIV2K	下采样	翻转、旋转	Set14, B100, Manga109, DIV2K	L₁	PSNR, SSIM
SAN--2019 (基于AM)	后上采样 (亚像素卷积)	DIV2K	①Bicubic ②模糊下采样	旋转、翻转	Set5, Set14, BSD100, Urban100, Manga109	L₁	PSNR, SSIM
DAN--2020 (基于ResNet)	后上采样 (亚像素卷积)	DIV2K, Flickr2K	①GB+下采样 ②GB(旋转、噪声、归一化)+ 下采样	—	①Set5, Set14, Urban100, BSD100, Manga109 ②DIV2KRK	L₁	PSNR, SSIM
HDRN--2020 (基于DenseNet)	后上采样 (亚像素卷积)	DIV2K	Bicubic	翻转、旋转	Set5, Set14, BSD100, Urban100, Manga109, Jilin-1 satellite imagery, Kaggle	L_Charbonnier	PSNR, SSIM
CDC-2020 (基于AM)	渐进式上采样 (亚像素卷积)	RealSR, DRealSR	—	—	RealSR, DRealSR	L_GW(L₁)	PSNR, SSIM, LPIPS
RealSR--2020 (无监督式)	后上采样 (最近邻插值)	①DF2K ②DPED	无监督式退化模型	—	①DF2K ②DPED	L₁, L_percep, L_GAN	①PSNR, SSIM, LPIPS ②MOR
RFB-ESRGAN--2020 (基于GAN)	后上采样 (交替使用最近邻插值和亚像素卷积)	DIV8K, DIV2K, Flickr2K, OST	Bicubic	翻转、旋转	DIV8K	L₁, L_VGG, L_GAN	PSNR, SSIM, LPIPS, PI
CRN--2021 (基于ResNet)	后上采样 (亚像素卷积)	DIV2K	Bicubic	—	Set5, Set14, B100, Urban100	L₁	PSNR, SSIM
SMSR--2021 (基于LN)	后上采样 (亚像素卷积+双三次插值)	DIV2K	下采样	旋转、翻转	Set5, Set14, B100, Urban100, Manga109	L₁, L_reg	PSNR, SSIM, Params, FLOPs
BSRGAN--2021 (基于GAN)	后上采样 (最近邻插值)	DIV2K, Flick2K,WED, FFHQ	B+下采样+N	—	①合成的DIV2K4D, ②真实的RealSRSet	L₁, L_percep,L_PatchGAN	①PSNR, LPIPS ②NIQE, NRQM, PI
Real-ESRGAN--2021 (基于GAN)	后上采样 (最近邻插值)	DIV2K, Flickr2K, OST	Bicubic+B+N+J	—	RealSR, DRealSR, OST300, DPED, ADE20K	L₁, L_percep, L_GAN	视觉效果
IPT--2021 (基于Transformer)	—	ImageNet	Bicubic	—	Set5, Set14, B100, Urban100	L₁, L_contrastive	PSNR
SwinIR--2021 (基于Transformer)	后上采样 (亚像素卷积)	DIV2K, Flickr2K	①Bicubic ②B+下采样+N	—	Set5, Set14, Urban100, BSD100, Manga109	①L₁ ②L₁, L_GAN, L_percep	PSNR, SSIM, Params, Mult-Adds

基于深度学习的单帧图像超分辨率重建综述

A Review of Single Image Super-Resolution Reconstruction Based on Deep Learning

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 26

参考文献 158

相关文章 15

编辑推荐

Metrics

挑战赛	赛道设置(尺度因子)	数据集	评价指标
NTIRE 2017--单图像超分辨率重建挑战^[59]	赛道1:经典的双三次下采样(×2、×3、×4) 赛道2:未知下采样(×2、×3、×4)	DIV2K 数据集	PSNR, SSIM
NTIRE 2018--单图像超分辨率重建挑战^[145]	赛道1:经典的双三次(×8) 赛道2:现实温和不利条件(×4) 赛道3:现实困难不利条件(×4) 赛道4:现实野生不利条件(×4)	DIV2K 数据集	PSNR, SSIM
PIRM 2018--感知图像超分辨率重建的挑战^[68]	赛道:采用双三次核下采样的单幅图像SR(×4)	PIRM数据集	PI, RMSE
NTIRE 2019--真实图像超分辨率重建挑战^[146]	赛道:真实世界SR(未知的尺度因子和退化算子)	RealSR 数据集	PSNR, SSIM
AIM 2019--真实世界图像超分辨率重建挑战^[147]	赛道1:源域(×4) 赛道2:目标域(×4)	DIV2K, Flickr2K 数据集	PSNR, SSIM, LPIPS, MOS
AIM 2019--受限超分辨率重建挑战^[148]	赛道1:参数(×4) 赛道2:推理时间(×4) 赛道3:保真度(PSNR)(×4)	DIV2K 数据集	参数个数、平均运行时间和平均PSNR
AIM 2019--图像极端超分辨率重建挑战^[149]	赛道1:保真度(×16) 赛道2:感知质量(×16)	DIV8K 数据集	赛道1:PSNR, SSIM 赛道2:PSNR, SSIM, MOS
NTIRE 2020--真实世界图像超分辨率重建挑战^[83]	赛道1:图像处理伪影(×4) 赛道2:智能手机图像(×4)	赛道1:Flickr2K, DIV2K数据集赛道2:DPED数据集	赛道1:PSNR, SSIM, LPIPS, MOS 赛道2:NIQE, BRISQUE, PIQE, NRQM, PI, IQA-Rank, MOR
NTIRE 2020--感知极端超分辨率重建挑战^[150]	赛道:感知极端SR(×16)	DIV8K 数据集	PSNR, SSIM, LPIPS, PI
AIM 2020--真实图像超分辨率重建挑战^[151]	赛道1:上采样×2 赛道2:上采样×3 赛道3:上采样×4	DRealSR 数据集	PSNR, SSIM
AIM 2020--高效超分辨率重建挑战^[123]	赛道:保持PSNR的同时,减少运行时间、参数、FLOPs、激活和内存消耗等一个或者几个方面(×4)	DIV2K 数据集	验证和测试PSNR、运行时间、参数数量、FLOPs数量、激活数量和推断期间消耗的最大GPU内存
NTIRE 2021--突发超分辨率重建挑战^[152]	赛道1:合成数据集(×4) 赛道2:真实世界数据集(×4)	赛道1:合成突发数据集赛道2:BurstSR数据集	赛道1:PSNR, SSIM, LPIPS 赛道2:PSNR, SSIM, LPIPS, MOR
Mobile AI 2021--基于移动NPU的实时量化图像超分辨率重建挑战^[153]	赛道:开发一个有效实际的解决方案用于移动任务	DIV2K 数据集	PSNR, SSIM, 运行时间

[1]	毛国君, 王者浩, 黄山, 王翔. 基于剪边策略的图残差卷积深层网络模型[J]. 电子学报, 2022, 50(9): 2205-2214.
[2]	李雪莹, 王田路, 梁鹏, 王翀. 基于系统模型的用户评论中非功能需求的自动分类[J]. 电子学报, 2022, 50(9): 2079-2089.
[3]	琚长瑞, 秦晓燕, 袁广林, 李豪, 朱虹. 尺度敏感损失与特征融合的快速小目标检测方法[J]. 电子学报, 2022, 50(9): 2119-2126.
[4]	袁海英, 曾智勇, 成君鹏. 面向灵活并行度的稀疏卷积神经网络加速器[J]. 电子学报, 2022, 50(8): 1811-1818.
[5]	张志昌, 于沛霖, 庞雅丽, 朱林, 曾扬扬. SMGN：用于对话状态跟踪的状态记忆图网络[J]. 电子学报, 2022, 50(8): 1851-1858.
[6]	张亚洲, 俞洋, 朱少林, 陈锐, 戎璐, 梁辉. 一种量子概率启发的对话讽刺识别网络模型[J]. 电子学报, 2022, 50(8): 1885-1893.
[7]	王飞扬, 冀鹏欣, 孙笠, 危倩, 李根, 张忠宝. 一种基于深度学习的动态社交网络用户对齐方法[J]. 电子学报, 2022, 50(8): 1925-1936.
[8]	徐兴荣, 刘聪, 李婷, 郭娜, 任崇广, 曾庆田. 基于双向准循环神经网络和注意力机制的业务流程剩余时间预测方法[J]. 电子学报, 2022, 50(8): 1975-1984.
[9]	张志文, 刘天歌, 聂鹏举. 基于实景数据增强和双路径融合网络的实时街景语义分割算法[J]. 电子学报, 2022, 50(7): 1609-1620.
[10]	欧阳与点, 谢鲲, 谢高岗, 文吉刚. 面向大规模网络测量的数据恢复算法：基于关联学习的张量填充[J]. 电子学报, 2022, 50(7): 1653-1663.
[11]	裴炤, 邱文涛, 王淼, 马苗, 张艳宁. 基于Transformer动态场景信息生成对抗网络的行人轨迹预测方法[J]. 电子学报, 2022, 50(7): 1537-1547.
[12]	刘文杰, 赵胶胶, 张颖, 葛业波. 一种量子条件生成对抗网络算法[J]. 电子学报, 2022, 50(7): 1586-1593.
[13]	王相海, 赵晓阳, 王鑫莹, 赵克云, 宋传鸣. 非抽取小波边缘学习深度残差网络的单幅图像超分辨率重建[J]. 电子学报, 2022, 50(7): 1753-1765.
[14]	彭闯, 王伦文, 胡炜林. 融合深度特征的电磁频谱异常检测算法[J]. 电子学报, 2022, 50(6): 1359-1369.
[15]	张波, 陆云杰, 秦东明, 邹国建. 一种卷积自编码深度学习的空气污染多站点联合预测模型[J]. 电子学报, 2022, 50(6): 1410-1427.