基于多级解码网络的图像修复

doi:10.12263/DZXB.20201451

电子学报 ›› 2022, Vol. 50 ›› Issue (3): 625-636.DOI: 10.12263/DZXB.20201451

基于多级解码网络的图像修复

刘微容¹, 米彦春¹, 杨帆¹, 张彦¹, 郭宏林², 刘仲民¹

^1.兰州理工大学电气工程与信息工程学院, 甘肃兰州 730050
^2.大型电气传动系统与装备技术国家重点实验室, 甘肃天水 741000

收稿日期:2020-12-17 修回日期:2021-08-05 出版日期:2022-03-25 发布日期:2022-03-25
作者简介:刘微容男，1976年12月出生于湖南祁阳，兰州理工大学教授，主要研究方向为机器视觉与人工智能、复杂系统先进控制理论与应用等.E-mail:liu_weirong@163.com
米彦春女，1994年6月出生于甘肃定西，兰州理工大学硕士研究生，主要研究方向为机器视觉与人工智能.
杨帆男，1996年7月出生于湖北宜昌，兰州理工大学硕士研究生，主要研究方向为机器视觉与人工智能.
张彦女，1992年6月出生于甘肃白银，兰州理工大学硕士研究生，主要研究方向为机器视觉与人工智能.
郭宏林男，1977年4月出生于甘肃天水，天水电气传动研究所集团有限公司高级工程师，主要从事电气传动系统、高精度电源及数字控制系统、智能装备等领域研究.
刘仲民男，1978年5月出生于甘肃白银，兰州理工大学副教授，主要研究方向为机器视觉、模式识别与图像处理.
基金资助:
国家自然科学基金(61861027)

Generative Image Inpainting with Multi-Stage Decoding Network

LIU Wei-rong¹, MI Yan-chun¹, YANG Fan¹, ZHANG Yan¹, GUO Hong-lin², LIU Zhong-min¹

^1.College of Electrical and Information Engineering，Lanzhou University of Technology，Lanzhou，Gansu 730050，China
^2.State Key Laboratory of Large Electric Drive System and Equipment Technology，Tianshui，Gansu 741000，China

Received:2020-12-17 Revised:2021-08-05 Online:2022-03-25 Published:2022-03-25

摘要/Abstract

摘要：

当前主流的图像修复方法重点依赖于自动编解码网络，此类方法试图利用编码阶段压缩后的信息在解码阶段恢复出原始图像.然而自编码网络在压缩过程中必然存在信息丢失，仅利用压缩后的信息难以得到细节丰富的修复结果，主要表现为模糊和修复区域周围明显的边缘响应.本文针对图像信息利用不完备的问题，提出多级解码网络（Multi-Stage Decoding Network，MSDN），由多个解码器对编码阶段各层特征进行解码并聚合，增大对编码器不同尺度特征的利用率，进而得到更能反映缺损区域内容的特征映射.在国际公认数据集上组织的对比实验结果表明，MSDN修复的图像视觉效果有一定提升.

关键词: 图像修复, 编解码器, 多级解码网络（MSDN）

Abstract:

Current image inpainting methods mainly rely on automatic encoding and decoding networks. These methods try to use the information compressed in the encoding stage to restore an original image in the decoding stage. While, it is difficult to reconstruct detailed inpainting results by using only compressed information. Due to the loss of information during compression, there are visual artifacts in the results, such as blurring and obvious edge response around the reconstructed area. Aimed at the problem of incomplete utilization of image information, this manuscript proposed a multi-stage decoding network (MSDN). The MSDN decodes and aggregates features of each layer in the encoder by multiple decoders successively, which can increase utilization of features from different layers in the encoding stage and obtain better feature maps to reflect the defected area. The experiment results, which are conducted on internationally recognized datasets, show that visual effects of images generated by MSDN have been improved.

Key words: image inpainting, automatic encoding and decoding networks, multi-stage decoding network

中图分类号:

TP391

刘微容, 米彦春, 杨帆, 张彦, 郭宏林, 刘仲民. 基于多级解码网络的图像修复[J]. 电子学报, 2022, 50(3): 625-636.

LIU Wei-rong, MI Yan-chun, YANG Fan, ZHANG Yan, GUO Hong-lin, LIU Zhong-min. Generative Image Inpainting with Multi-Stage Decoding Network[J]. Acta Electronica Sinica, 2022, 50(3): 625-636.

图/表 15

图1 是否充分利用编码部分信息修复结果对比

图2 带有跳转连接的编解码网络

图3 多级解码网络的生成器和判别器体系架构.生成器G中，红色曲线表示一个或多个反卷积操作，称之为副解码器，黑色曲线表示跳转连接.

图4 不同尺度特征汇聚示意图，其中黑色箭头表示主解码过程，红色箭头表示副解码过程

图5 ATN的连接形式

图6 注意力转移网络（ATN）

表1 训练集、测试集划分

数据集	训练集	测试集	总数
Celeba-HQ^［28］	25000	500	30000
Facade^［29］	556	50	606
Places2^［30］	8020000	6628	8026628
Mural	1000	92	1092

图7 不同方法在CelebA-HQ数据集上的重建效果图

图8 不同方法在Facade数据集上的重建效果图，放大观察效果更佳

图9 不同方法在Places2数据集上的重建效果图

图10 MSDN在Mural数据集上的重建效果图

图11 Pen-Net（b）及MSDN（c）在Celeba-HQ数据集上的重建效果图

表2 各种方法的定量评价结果（不规则掩码）.

	掩码面积	CA	GMCNN	MSDN
	掩码面积	CelebA-HQ/Facade/Places2	CelebA-HQ/Facade/Places2	CelebA-HQ/Facade/Places2
PSNR^↑	10~20%	25.585/24.151/22.492	29.729/27.431/27.318	31.076/27.887/27.559
	30~40%	22.948/23.151/18.502	26.651/24.934/22.725	28.375/25.362/23.316
	50~60%	21.337/21.536/17.553	24.785/23.318/21.417	26.545/23.819/22.202
	均值	23.29/22.946/19.516	27.055/25.227/23.82	28.665/25.689/24.359
SSIM^↑	10~20%	0.9239/0.9230/0.9025	0.9486/0.9398/0.9336	0.9578/0.9443/0.9330
	30~40%	0.8786/0.8762/0.7896	0.9152/0.9020/0.8510	0.9322/0.9087/0.8528
	50~60%	0.8406/0.8328/0.7477	0.8875/0.8669/0.8170	0.9110/0.8767/0.8216
	均值	0.8810/0.8773/0.8132	0.9171/0.9029/0.8672	0.9336/0.9099/0.8691
FID^↓	10~20%	34.036/16.880/30.179	14.594/11.055/15.841	10.057/11.137/15.582
	30~40%	53.223/27.261/50.728	27.849/17.323/27.026	16.974/17.366/24.963
	50~60%	73.466/37.111/66.768	37.526/22.782/38.028	21.841/22.313/34.569
	均值	53.575/27.084/49.225	26.656/17.053/26.965	16.290/16.938/25.038
$L 1 -$ Loss^↓	10~20%	0.0406/0.0399/0.0594	0.0228/0.0291/0.0307	0.0196/0.0280/0.0299
	30~40%	0.0669/0.0647/0.1313	0.0382/0.0475/0.0711	0.0319/0.0459/0.0671
	50~60%	0.0901/0.0883/0.1589	0.0522/0.0645/0.0889	0.0434/0.0623/0.0823
	均值	0.0658/0.0634/0.1165	0.0377/0.047/0.0635	0.0316/0.0454/0.0576

表2 各种方法的定量评价结果（不规则掩码）.

	掩码面积	CA	GMCNN	MSDN
	掩码面积	CelebA-HQ/Facade/Places2	CelebA-HQ/Facade/Places2	CelebA-HQ/Facade/Places2
PSNR^↑	10~20%	25.585/24.151/22.492	29.729/27.431/27.318	31.076/27.887/27.559
	30~40%	22.948/23.151/18.502	26.651/24.934/22.725	28.375/25.362/23.316
	50~60%	21.337/21.536/17.553	24.785/23.318/21.417	26.545/23.819/22.202
	均值	23.29/22.946/19.516	27.055/25.227/23.82	28.665/25.689/24.359
SSIM^↑	10~20%	0.9239/0.9230/0.9025	0.9486/0.9398/0.9336	0.9578/0.9443/0.9330
	30~40%	0.8786/0.8762/0.7896	0.9152/0.9020/0.8510	0.9322/0.9087/0.8528
	50~60%	0.8406/0.8328/0.7477	0.8875/0.8669/0.8170	0.9110/0.8767/0.8216
	均值	0.8810/0.8773/0.8132	0.9171/0.9029/0.8672	0.9336/0.9099/0.8691
FID^↓	10~20%	34.036/16.880/30.179	14.594/11.055/15.841	10.057/11.137/15.582
	30~40%	53.223/27.261/50.728	27.849/17.323/27.026	16.974/17.366/24.963
	50~60%	73.466/37.111/66.768	37.526/22.782/38.028	21.841/22.313/34.569
	均值	53.575/27.084/49.225	26.656/17.053/26.965	16.290/16.938/25.038
$L 1 -$ Loss^↓	10~20%	0.0406/0.0399/0.0594	0.0228/0.0291/0.0307	0.0196/0.0280/0.0299
	30~40%	0.0669/0.0647/0.1313	0.0382/0.0475/0.0711	0.0319/0.0459/0.0671
	50~60%	0.0901/0.0883/0.1589	0.0522/0.0645/0.0889	0.0434/0.0623/0.0823
	均值	0.0658/0.0634/0.1165	0.0377/0.047/0.0635	0.0316/0.0454/0.0576

图12 用户研究结果统计图

图13 注意力转移网络消融对比实验效果图

参考文献 35

1	ELHARROUSS O, ALMAADEED N, et al. Image inpainting: a review[J]. Neural Processing Letters, 2020, 51(2):2007‑2028.
2	Ballester Coloma, BERTALMIO M, CASELLES V, et al. Filling-in by joint interpolation of vector fields and gray levels[J]. IEEE Transactions on Image Processing, 2001, 10(2): 1200‑1211.
3	ASHIKHMIN M. Synthesizing natural textures[C]//Proceedings of the 2001 symposium on Interactive 3D graphics - SI3D＇01. New York: ACM Press, 2001: 217‑226.
4	CONNELLY B, ELI S, ADAM F, et al. Patchmatch: a randomized correspondence algorithm for structural image editing[J]. ACM Transactions on Graphics, 2009, 28(3): 1‑11.
5	Tijana Ruzic, Aleksandra Pizurica. Context-aware patch-based image inpainting using markov random field modeling[J]. IEEE Trans Image Process, 2015, 24(1): 444‑456.
6	李志丹, 和红杰, 尹忠科, 等. 基于curvelet方向特征的样本块图像修复算法[J]. 电子学报, 2016, 44(1): 150‑154.
	LI Zhi-dan,HE Hong-jie,YIN Zhong-ke,et al, Exemplar based image inpainting algorithm using direction features of curvelet transform[J]. Acta Electronica Sinica, 2016, 44(1): 150‑154. (in Chinese)
7	吴晓军, 李功清. 基于样本和线性结构信息的大范围图像修复算法[J]. 电子学报, 2012, 40(8): 1509‑1514.
	WU Xiao-jun, LI Gong-qing.Large scale image inpainting based on exemplar and structure information[J]. Acta Electronica Sinica, 2012, 40(8): 1509‑1514. (in Chinese)
8	强振平, 何丽波, 陈旭, 等. 深度学习图像修复方法综述[J]. 中国图象图形学报, 2019, 24: 447‑463.
9	曹承瑞. 基于密集判别与注意力特征传播的细粒度图像修复[D].兰州：兰州理工大学, 2020.
10	王贵杭. 基于几何信息的图像修复[D]. 合肥：中国科学技术大学, 2015.
11	YANG C, LU X, LIN Z, et al. High-resolution image inpainting using multi-scale neural patch synthesis[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2017: 4076‑4084.
12	Goodfellow Ian J, Pouget-Abadie Jean, Mirza Mehdi, et al. Generative adversarial nets[C]// Proceedings of the 27th International Conference on Neural Information Processing Systems-Volume 2. Cambridge, MA, US: MIT Press, 2014: 2672‑2680
13	PATHAK D, KRÄHENBÜHL P, DONAHUE J, et al. Context encoders: feature learning by inpainting[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2016: 2536‑2544.
14	IIZUKA S, SIMO-SERRA E, ISHIKAWA H. Globally and locally consistent image completion[J]. ACM Transactions on Graphics, 2017, 36(4): 1‑14.
15	YU J H, LIN Z, YANG J M, et al. Generative image inpainting with contextual attention[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2018: 5505‑5514.
16	罗会兰, 敖阳, 袁璞. 一种生成对抗网络用于图像修复的方法[J]. 电子学报, 2020, 48(10): 1891‑1898.
	LUO H L, AO Y, YUAN P. Image inpainting using generative adversarial networks[J]. Acta Electronica Sinica, 2020, 48(10): 1891‑1898. (in Chinese)
17	王斌, 胡辽林, 曹京京, 等. 基于小波域稀疏最优的图像修复方法[J]. 电子学报, 2016, 44(3): 600‑606.
	WANG B, HU L L, CAO J J, et al. Image restoration based on sparse-optimal strategy in wavelet domain[J]. Acta Electronica Sinica, 2016, 44(3): 600‑606.(in Chinese)
18	熊锐, 张雷洪, 蒋周杰, 等. 基于编码-解码对称神经网络的高分辨率图像重构机理研究[J]. 光学仪器, 2019, 41(4): 36‑41.
	XIONG R, ZHANG L H, JIANG Z J, et al. Research on high-resolution image reconstruction mechanism based on coding-decoding symmetric neural network[J]. Optical Instruments, 2019, 41(4): 36‑41. (in Chinese)
19	魏域林. 层间特征融合与多注意力的图像修复算法研究[D]; 兰州理工大学, 2020.
20	李志丹, 程吉祥, 刘家伟. 基于结构偏移映射统计和多方向特征的MRF图像修复算法[J]. 电子学报, 2020, 48(5): 985‑989.
	LI Z D, CHENG J X, LIU J W. MRF image inpainting algorithm based on structure offsets statistics and multi-direction features[J]. Acta Electronica Sinica, 2020, 48(5): 985‑989. (in Chinese)
21	LIU G L, REDA F A, SHIH K J, et al. Image inpainting for irregular holes using partial convolutions[M]//Computer Vision-ECCV 2018. Cham: Springer International Publishing, 2018: 89‑105.
22	WANG Yi, TAO Xin, QI Xiao-juan, et al. Image inpainting via generative multi-column convolutional neural networks[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems. Red Hook, NY: Curran Associates Inc, 2018: 329‑338.
23	YU J H, LIN Z, YANG J M, et al. Free-form image inpainting with gated convolution[C]//2019 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway, NJ: IEEE, 2019: 4470‑4479.
24	ZENG Y H, FU J L, CHAO H Y, et al. Learning pyramid-context encoder network for high-quality image inpainting[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ: IEEE, 2019: 1486‑1494.
25	LI J Y, WANG N, ZHANG L F, et al. Recurrent feature reasoning for image inpainting[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ: IEEE, 2020: 7757‑7765.
26	鲍建敏. 基于生成对抗网络的图像合成[D]. 合肥: 中国科学技术大学, 2019.
	BAO J M. Image Synthesis Based on Generative Adversarial Networks[D]. Hefei: University of Science and Technology of China, 2019. (in Chinese)
27	Nazeri Kamyar, Ng Eric, Joseph Tony, et al. Edgeconnect: generative image inpainting with adversarial edge learning[EB/OL].(2019-01-11). .
28	KARRAS T, AILA T, LAINE S, et al. Progressive growing of GANs for improved quality, stability, and variation[EB/OL]. (2018-02-26). .
29	Tyleček Radim, Šára Radim. Spatial pattern templates for recognition of objects with regular structure[C]//German Conference on Pattern Recognition. Berlin: Springer, 2013: 364‑374.
30	Zhou Bo-lei, Lapedriza Agata, Khosla Aditya, et al. Places: a 10 million image database for scene recognition[J]. IEEE transactions on pattern analysis and machine intelligence, 2017, 40(6): 1452‑1464.
31	Ronneberger Olaf, Fischer Philipp, Brox Thomas. U-Net: convolutional networks for biomedical image segmentation[C]// Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015, Part Ⅲ. Berlin: Springer,2015: 234‑241.
32	HORÉ A, ZIOU D. Image quality metrics: PSNR vs. SSIM[C]//2010 20th International Conference on Pattern Recognition. Piscataway, NJ: IEEE, 2010: 2366‑2369.
33	WANG Zhou. Image quality assessment : from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600‑612.
34	HEUSEL M, RAMSAUER H, UNTERTHINER T, et al. GANs trained by a two time-scale update rule converge to a local Nash equilibrium[C]//NIPS＇17: Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook, NY, US: Curran Associates Inc, 2017: 6629‑6640.
35	GAO Xiao-li, FANG Yi-xin. A note on the generalized degrees of freedom under the L1 loss function[J]. Journal of Statistical Planning & Inference, 2011, 141(2): 677‑686.

[1]	冀振燕, 韩梦豪, 宋晓军, 冯其波. 面向激光光条图像修复的循环相似度映射网络[J]. 电子学报, 2022, 50(5): 1234-1242.
[2]	李志丹, 程吉祥, 刘家伟. 基于结构偏移映射统计和多方向特征的MRF图像修复算法[J]. 电子学报, 2020, 48(5): 985-989.
[3]	罗会兰, 敖阳, 袁璞. 一种生成对抗网络用于图像修复的方法[J]. 电子学报, 2020, 48(10): 1891-1898.
[4]	李志丹, 和红杰, 尹忠科, 陈帆. 基于Curvelet方向特征的样本块图像修复算法[J]. 电子学报, 2016, 44(1): 150-154.
[5]	余烨, 李冰飞, 叶强强, 刘晓平. 面向球面虚拟实景底部空洞的快速修补算法[J]. 电子学报, 2015, 43(4): 822-828.
[6]	许刚, 马爽. 动态尺度块匹配约束下的链式优化图像修复研究[J]. 电子学报, 2015, 43(3): 529-535.
[7]	李志丹, 和红杰, 尹忠科, 陈帆, 仁青诺布. 基于块结构稀疏度的自适应图像修复算法[J]. 电子学报, 2013, 41(3): 549-554.
[8]	吴晓军, 李功清. 基于样本和线性结构信息的大范围图像修复算法[J]. 电子学报, 2012, 40(8): 1509-1514.
[9]	张岩;孙正兴;姚伟. 基于方向经验模型分解的图像修复方法[J]. 电子学报, 2010, 38(2): 257-262.
[10]	李晋江;张彩明;范辉;原达. 基于分形的图像修复算法[J]. 电子学报, 2010, 38(10): 2430-2435.
[11]	刘春晓, 彭群生, 杨颖振, 王进, 陈为. 利用大位移视图修复单幅图像的透视畸变优化算法[J]. 电子学报, 2009, 37(S1): 112-117.

基于多级解码网络的图像修复

Generative Image Inpainting with Multi-Stage Decoding Network

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 15

参考文献 35

相关文章 11

编辑推荐

Metrics