电子学报 ›› 2022, Vol. 50 ›› Issue (3): 625-636.DOI: 10.12263/DZXB.20201451
刘微容1, 米彦春1, 杨帆1, 张彦1, 郭宏林2, 刘仲民1
收稿日期:
2020-12-17
修回日期:
2021-08-05
出版日期:
2022-03-25
发布日期:
2022-03-25
作者简介:
基金资助:
LIU Wei-rong1, MI Yan-chun1, YANG Fan1, ZHANG Yan1, GUO Hong-lin2, LIU Zhong-min1
Received:
2020-12-17
Revised:
2021-08-05
Online:
2022-03-25
Published:
2022-03-25
摘要:
当前主流的图像修复方法重点依赖于自动编解码网络,此类方法试图利用编码阶段压缩后的信息在解码阶段恢复出原始图像.然而自编码网络在压缩过程中必然存在信息丢失,仅利用压缩后的信息难以得到细节丰富的修复结果,主要表现为模糊和修复区域周围明显的边缘响应.本文针对图像信息利用不完备的问题,提出多级解码网络(Multi-Stage Decoding Network,MSDN),由多个解码器对编码阶段各层特征进行解码并聚合,增大对编码器不同尺度特征的利用率,进而得到更能反映缺损区域内容的特征映射.在国际公认数据集上组织的对比实验结果表明,MSDN修复的图像视觉效果有一定提升.
中图分类号:
刘微容, 米彦春, 杨帆, 张彦, 郭宏林, 刘仲民. 基于多级解码网络的图像修复[J]. 电子学报, 2022, 50(3): 625-636.
LIU Wei-rong, MI Yan-chun, YANG Fan, ZHANG Yan, GUO Hong-lin, LIU Zhong-min. Generative Image Inpainting with Multi-Stage Decoding Network[J]. Acta Electronica Sinica, 2022, 50(3): 625-636.
数据集 | 训练集 | 测试集 | 总数 |
---|---|---|---|
Celeba-HQ[ | 25000 | 500 | 30000 |
Facade[ | 556 | 50 | 606 |
Places2[ | 8020000 | 6628 | 8026628 |
Mural | 1000 | 92 | 1092 |
表1 训练集、测试集划分
数据集 | 训练集 | 测试集 | 总数 |
---|---|---|---|
Celeba-HQ[ | 25000 | 500 | 30000 |
Facade[ | 556 | 50 | 606 |
Places2[ | 8020000 | 6628 | 8026628 |
Mural | 1000 | 92 | 1092 |
掩码面积 | CA | GMCNN | MSDN | |
---|---|---|---|---|
CelebA-HQ/Facade/Places2 | CelebA-HQ/Facade/Places2 | CelebA-HQ/Facade/Places2 | ||
PSNR↑ | 10~20% | 25.585/24.151/22.492 | 29.729/27.431/27.318 | 31.076/27.887/27.559 |
30~40% | 22.948/23.151/18.502 | 26.651/24.934/22.725 | 28.375/25.362/23.316 | |
50~60% | 21.337/21.536/17.553 | 24.785/23.318/21.417 | 26.545/23.819/22.202 | |
均值 | 23.29/22.946/19.516 | 27.055/25.227/23.82 | 28.665/25.689/24.359 | |
SSIM↑ | 10~20% | 0.9239/0.9230/0.9025 | 0.9486/0.9398/0.9336 | 0.9578/0.9443/0.9330 |
30~40% | 0.8786/0.8762/0.7896 | 0.9152/0.9020/0.8510 | 0.9322/0.9087/0.8528 | |
50~60% | 0.8406/0.8328/0.7477 | 0.8875/0.8669/0.8170 | 0.9110/0.8767/0.8216 | |
均值 | 0.8810/0.8773/0.8132 | 0.9171/0.9029/0.8672 | 0.9336/0.9099/0.8691 | |
FID↓ | 10~20% | 34.036/16.880/30.179 | 14.594/11.055/15.841 | 10.057/11.137/15.582 |
30~40% | 53.223/27.261/50.728 | 27.849/17.323/27.026 | 16.974/17.366/24.963 | |
50~60% | 73.466/37.111/66.768 | 37.526/22.782/38.028 | 21.841/22.313/34.569 | |
均值 | 53.575/27.084/49.225 | 26.656/17.053/26.965 | 16.290/16.938/25.038 | |
10~20% | 0.0406/0.0399/0.0594 | 0.0228/0.0291/0.0307 | 0.0196/0.0280/0.0299 | |
30~40% | 0.0669/0.0647/0.1313 | 0.0382/0.0475/0.0711 | 0.0319/0.0459/0.0671 | |
50~60% | 0.0901/0.0883/0.1589 | 0.0522/0.0645/0.0889 | 0.0434/0.0623/0.0823 | |
均值 | 0.0658/0.0634/0.1165 | 0.0377/0.047/0.0635 | 0.0316/0.0454/0.0576 |
表2 各种方法的定量评价结果(不规则掩码).
掩码面积 | CA | GMCNN | MSDN | |
---|---|---|---|---|
CelebA-HQ/Facade/Places2 | CelebA-HQ/Facade/Places2 | CelebA-HQ/Facade/Places2 | ||
PSNR↑ | 10~20% | 25.585/24.151/22.492 | 29.729/27.431/27.318 | 31.076/27.887/27.559 |
30~40% | 22.948/23.151/18.502 | 26.651/24.934/22.725 | 28.375/25.362/23.316 | |
50~60% | 21.337/21.536/17.553 | 24.785/23.318/21.417 | 26.545/23.819/22.202 | |
均值 | 23.29/22.946/19.516 | 27.055/25.227/23.82 | 28.665/25.689/24.359 | |
SSIM↑ | 10~20% | 0.9239/0.9230/0.9025 | 0.9486/0.9398/0.9336 | 0.9578/0.9443/0.9330 |
30~40% | 0.8786/0.8762/0.7896 | 0.9152/0.9020/0.8510 | 0.9322/0.9087/0.8528 | |
50~60% | 0.8406/0.8328/0.7477 | 0.8875/0.8669/0.8170 | 0.9110/0.8767/0.8216 | |
均值 | 0.8810/0.8773/0.8132 | 0.9171/0.9029/0.8672 | 0.9336/0.9099/0.8691 | |
FID↓ | 10~20% | 34.036/16.880/30.179 | 14.594/11.055/15.841 | 10.057/11.137/15.582 |
30~40% | 53.223/27.261/50.728 | 27.849/17.323/27.026 | 16.974/17.366/24.963 | |
50~60% | 73.466/37.111/66.768 | 37.526/22.782/38.028 | 21.841/22.313/34.569 | |
均值 | 53.575/27.084/49.225 | 26.656/17.053/26.965 | 16.290/16.938/25.038 | |
10~20% | 0.0406/0.0399/0.0594 | 0.0228/0.0291/0.0307 | 0.0196/0.0280/0.0299 | |
30~40% | 0.0669/0.0647/0.1313 | 0.0382/0.0475/0.0711 | 0.0319/0.0459/0.0671 | |
50~60% | 0.0901/0.0883/0.1589 | 0.0522/0.0645/0.0889 | 0.0434/0.0623/0.0823 | |
均值 | 0.0658/0.0634/0.1165 | 0.0377/0.047/0.0635 | 0.0316/0.0454/0.0576 |
1 | ELHARROUSS O, ALMAADEED N, et al. Image inpainting: a review[J]. Neural Processing Letters, 2020, 51(2):2007‑2028. |
2 | Ballester Coloma, BERTALMIO M, CASELLES V, et al. Filling-in by joint interpolation of vector fields and gray levels[J]. IEEE Transactions on Image Processing, 2001, 10(2): 1200‑1211. |
3 | ASHIKHMIN M. Synthesizing natural textures[C]//Proceedings of the 2001 symposium on Interactive 3D graphics - SI3D'01. New York: ACM Press, 2001: 217‑226. |
4 | CONNELLY B, ELI S, ADAM F, et al. Patchmatch: a randomized correspondence algorithm for structural image editing[J]. ACM Transactions on Graphics, 2009, 28(3): 1‑11. |
5 | Tijana Ruzic, Aleksandra Pizurica. Context-aware patch-based image inpainting using markov random field modeling[J]. IEEE Trans Image Process, 2015, 24(1): 444‑456. |
6 | 李志丹, 和红杰, 尹忠科, 等. 基于curvelet方向特征的样本块图像修复算法[J]. 电子学报, 2016, 44(1): 150‑154. |
LI Zhi-dan,HE Hong-jie,YIN Zhong-ke,et al, Exemplar based image inpainting algorithm using direction features of curvelet transform[J]. Acta Electronica Sinica, 2016, 44(1): 150‑154. (in Chinese) | |
7 | 吴晓军, 李功清. 基于样本和线性结构信息的大范围图像修复算法[J]. 电子学报, 2012, 40(8): 1509‑1514. |
WU Xiao-jun, LI Gong-qing.Large scale image inpainting based on exemplar and structure information[J]. Acta Electronica Sinica, 2012, 40(8): 1509‑1514. (in Chinese) | |
8 | 强振平, 何丽波, 陈旭, 等. 深度学习图像修复方法综述[J]. 中国图象图形学报, 2019, 24: 447‑463. |
9 | 曹承瑞. 基于密集判别与注意力特征传播的细粒度图像修复[D].兰州:兰州理工大学, 2020. |
10 | 王贵杭. 基于几何信息的图像修复[D]. 合肥:中国科学技术大学, 2015. |
11 | YANG C, LU X, LIN Z, et al. High-resolution image inpainting using multi-scale neural patch synthesis[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2017: 4076‑4084. |
12 | Goodfellow Ian J, Pouget-Abadie Jean, Mirza Mehdi, et al. Generative adversarial nets[C]// Proceedings of the 27th International Conference on Neural Information Processing Systems-Volume 2. Cambridge, MA, US: MIT Press, 2014: 2672‑2680 |
13 | PATHAK D, KRÄHENBÜHL P, DONAHUE J, et al. Context encoders: feature learning by inpainting[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2016: 2536‑2544. |
14 | IIZUKA S, SIMO-SERRA E, ISHIKAWA H. Globally and locally consistent image completion[J]. ACM Transactions on Graphics, 2017, 36(4): 1‑14. |
15 | YU J H, LIN Z, YANG J M, et al. Generative image inpainting with contextual attention[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2018: 5505‑5514. |
16 | 罗会兰, 敖阳, 袁璞. 一种生成对抗网络用于图像修复的方法[J]. 电子学报, 2020, 48(10): 1891‑1898. |
LUO H L, AO Y, YUAN P. Image inpainting using generative adversarial networks[J]. Acta Electronica Sinica, 2020, 48(10): 1891‑1898. (in Chinese) | |
17 | 王斌, 胡辽林, 曹京京, 等. 基于小波域稀疏最优的图像修复方法[J]. 电子学报, 2016, 44(3): 600‑606. |
WANG B, HU L L, CAO J J, et al. Image restoration based on sparse-optimal strategy in wavelet domain[J]. Acta Electronica Sinica, 2016, 44(3): 600‑606.(in Chinese) | |
18 | 熊锐, 张雷洪, 蒋周杰, 等. 基于编码-解码对称神经网络的高分辨率图像重构机理研究[J]. 光学仪器, 2019, 41(4): 36‑41. |
XIONG R, ZHANG L H, JIANG Z J, et al. Research on high-resolution image reconstruction mechanism based on coding-decoding symmetric neural network[J]. Optical Instruments, 2019, 41(4): 36‑41. (in Chinese) | |
19 | 魏域林. 层间特征融合与多注意力的图像修复算法研究[D]; 兰州理工大学, 2020. |
20 | 李志丹, 程吉祥, 刘家伟. 基于结构偏移映射统计和多方向特征的MRF图像修复算法[J]. 电子学报, 2020, 48(5): 985‑989. |
LI Z D, CHENG J X, LIU J W. MRF image inpainting algorithm based on structure offsets statistics and multi-direction features[J]. Acta Electronica Sinica, 2020, 48(5): 985‑989. (in Chinese) | |
21 | LIU G L, REDA F A, SHIH K J, et al. Image inpainting for irregular holes using partial convolutions[M]//Computer Vision-ECCV 2018. Cham: Springer International Publishing, 2018: 89‑105. |
22 | WANG Yi, TAO Xin, QI Xiao-juan, et al. Image inpainting via generative multi-column convolutional neural networks[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems. Red Hook, NY: Curran Associates Inc, 2018: 329‑338. |
23 | YU J H, LIN Z, YANG J M, et al. Free-form image inpainting with gated convolution[C]//2019 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway, NJ: IEEE, 2019: 4470‑4479. |
24 | ZENG Y H, FU J L, CHAO H Y, et al. Learning pyramid-context encoder network for high-quality image inpainting[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ: IEEE, 2019: 1486‑1494. |
25 | LI J Y, WANG N, ZHANG L F, et al. Recurrent feature reasoning for image inpainting[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ: IEEE, 2020: 7757‑7765. |
26 | 鲍建敏. 基于生成对抗网络的图像合成[D]. 合肥: 中国科学技术大学, 2019. |
BAO J M. Image Synthesis Based on Generative Adversarial Networks[D]. Hefei: University of Science and Technology of China, 2019. (in Chinese) | |
27 | Nazeri Kamyar, Ng Eric, Joseph Tony, et al. Edgeconnect: generative image inpainting with adversarial edge learning[EB/OL].(2019-01-11). . |
28 | KARRAS T, AILA T, LAINE S, et al. Progressive growing of GANs for improved quality, stability, and variation[EB/OL]. (2018-02-26). . |
29 | Tyleček Radim, Šára Radim. Spatial pattern templates for recognition of objects with regular structure[C]//German Conference on Pattern Recognition. Berlin: Springer, 2013: 364‑374. |
30 | Zhou Bo-lei, Lapedriza Agata, Khosla Aditya, et al. Places: a 10 million image database for scene recognition[J]. IEEE transactions on pattern analysis and machine intelligence, 2017, 40(6): 1452‑1464. |
31 | Ronneberger Olaf, Fischer Philipp, Brox Thomas. U-Net: convolutional networks for biomedical image segmentation[C]// Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015, Part Ⅲ. Berlin: Springer,2015: 234‑241. |
32 | HORÉ A, ZIOU D. Image quality metrics: PSNR vs. SSIM[C]//2010 20th International Conference on Pattern Recognition. Piscataway, NJ: IEEE, 2010: 2366‑2369. |
33 | WANG Zhou. Image quality assessment : from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600‑612. |
34 | HEUSEL M, RAMSAUER H, UNTERTHINER T, et al. GANs trained by a two time-scale update rule converge to a local Nash equilibrium[C]//NIPS'17: Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook, NY, US: Curran Associates Inc, 2017: 6629‑6640. |
35 | GAO Xiao-li, FANG Yi-xin. A note on the generalized degrees of freedom under the L1 loss function[J]. Journal of Statistical Planning & Inference, 2011, 141(2): 677‑686. |
[1] | 冀振燕, 韩梦豪, 宋晓军, 冯其波. 面向激光光条图像修复的循环相似度映射网络[J]. 电子学报, 2022, 50(5): 1234-1242. |
[2] | 李志丹, 程吉祥, 刘家伟. 基于结构偏移映射统计和多方向特征的MRF图像修复算法[J]. 电子学报, 2020, 48(5): 985-989. |
[3] | 罗会兰, 敖阳, 袁璞. 一种生成对抗网络用于图像修复的方法[J]. 电子学报, 2020, 48(10): 1891-1898. |
[4] | 李志丹, 和红杰, 尹忠科, 陈帆. 基于Curvelet方向特征的样本块图像修复算法[J]. 电子学报, 2016, 44(1): 150-154. |
[5] | 余烨, 李冰飞, 叶强强, 刘晓平. 面向球面虚拟实景底部空洞的快速修补算法[J]. 电子学报, 2015, 43(4): 822-828. |
[6] | 许刚, 马爽. 动态尺度块匹配约束下的链式优化图像修复研究[J]. 电子学报, 2015, 43(3): 529-535. |
[7] | 李志丹, 和红杰, 尹忠科, 陈帆, 仁青诺布. 基于块结构稀疏度的自适应图像修复算法[J]. 电子学报, 2013, 41(3): 549-554. |
[8] | 吴晓军, 李功清. 基于样本和线性结构信息的大范围图像修复算法[J]. 电子学报, 2012, 40(8): 1509-1514. |
[9] | 张岩;孙正兴;姚伟. 基于方向经验模型分解的图像修复方法[J]. 电子学报, 2010, 38(2): 257-262. |
[10] | 李晋江;张彩明;范 辉;原 达. 基于分形的图像修复算法[J]. 电子学报, 2010, 38(10): 2430-2435. |
[11] | 刘春晓, 彭群生, 杨颖振, 王进, 陈为. 利用大位移视图修复单幅图像的透视畸变优化算法[J]. 电子学报, 2009, 37(S1): 112-117. |
阅读次数 | ||||||
全文 |
|
|||||
摘要 |
|
|||||