一种生成对抗网络用于图像修复的方法

doi:10.3969/j.issn.0372-2112.2020.10.003

电子学报 ›› 2020, Vol. 48 ›› Issue (10): 1891-1898.DOI: 10.3969/j.issn.0372-2112.2020.10.003

一种生成对抗网络用于图像修复的方法

罗会兰, 敖阳, 袁璞

江西理工大学信息工程学院, 江西赣州 341000

收稿日期:2019-08-06 修回日期:2020-05-08 出版日期:2020-10-25
- 作者简介:
- 罗会兰 女,1974年9月生于江西上高.2008年获浙江大学工学博士学位.现为江西理工大学图像处理实验室教授、硕士生导师.主要从事机器学习、模式识别等方面的研究.E-mail:luohuilan@sina.com
  敖阳男,1996年6月生于江西赣州.2018年进入江西理工大学.在读硕士研究生,研究方向为图像修复.E-mail:1522173817@qq.com
  袁璞女,1997年5月生于江西吉安.2018年进入江西理工大学.在读硕士研究生,研究方向为图像修复、显著性目标检测.E-mail:emmanuel_97@163.com
- 基金资助:
- 国家自然科学基金 (No.61862031）; 江西省赣州市"科技创新人才计划"

Image Inpainting Using Generative Adversarial Networks

LUO Hui-lan, AO Yang, YUAN Pu

School of Information Engineering, Jiangxi University of Science and Technology, Ganzhou, Jiangxi 341000, China

Received:2019-08-06 Revised:2020-05-08 Online:2020-10-25 Published:2020-10-25
- Supported by:
- National Natural Science Foundation of China (No.61862031); Ganzhou Science and Technology Innovation Talents Project of Jiangxi Province

摘要/Abstract

摘要： 近年来基于深度学习的图像修复方法相比于传统方法，表现出明显优势，前者能更好的生成视觉上合理的图像结构和纹理.但现有的标准卷积神经网络方法，通常会造成颜色差异过大和图像纹理缺失与失真的问题.本文提出了一种新型图像修复深度网络模型，该模型由两个相互独立的生成对抗式网络模块组成.其中，图像修复网络模块旨在解决图像缺失区域的修复问题，其生成器基于部分卷积网络；图像优化网络模块旨在解决修复后图像存在局部色差的问题，其生成器基于深度残差网络.通过两个网络模块的协同作用，图像的视觉效果与图像质量得到提高.与其他先进方法进行定性和定量比较的实验结果表明，本文提出的方法在图像修复质量上表现更好.

关键词: 部分卷积, 生成对抗神经网络, 残差网络, 图像修复

Abstract: In recent years, deep learning based methods have shown preferable results for the task of inpainting corrupted images. However, the existing standard convolutional neural network approaches often cause problems with excessive color discrepancy, image texture loss and distortion. A deep network based image inpainting model is proposed in this paper, consisting of two generative adversarial network modules. One of the modules is used to inpaint the missing area of the image, where the generator is constituted with partial convolutions. The other module is the image optimization network, which is applied to solve the problem of local chromatic aberration after image restoration, and in which the generator is originated from the depth residual network. These two modules cooperated to improve the visual effect and image quality of inpainted images. Using MOS, SSIM and PSRN as the evaluation criteria, the experimental results of qualitative and quantitative comparisons with other state-of-the-art methods have shown that the proposed model performed better.

Key words: partial convolution, generative adversarial networks, residual network, image inpainting

中图分类号:

TP391.9

罗会兰, 敖阳, 袁璞. 一种生成对抗网络用于图像修复的方法[J]. 电子学报, 2020, 48(10): 1891-1898.

LUO Hui-lan, AO Yang, YUAN Pu . Image Inpainting Using Generative Adversarial Networks[J]. Acta Electronica Sinica, 2020, 48(10): 1891-1898.

参考文献

[1] Newson A,Almansa A,Fradet M,et al.Video inpainting of complex scenes[J].Siam Journal on Imaging Sciences,2014,7(4):1993-2019.
[2] Levin A,Zomet A,Peleg S,et al.Seamless image stitching in the gradient domain[A].European Conference on Computer Vision[C].Czech Republic:IEEE,2004.377-389.
[3] Barnes C,et al.Patch match:A randomized correspondence algorithm for structural image editing[J].Acm Transactions on Graphics,2009,28(3):2-11.
[4] Zheng C,Cham T,Cai J.Pluralistic image completion[A].Proceedings of IEEE Conference on Computer Vision and Pattern Recognition[C].USA:IEEE,2019.1438-1447.
[5] Iizuka S,Simo-Serra E,Ishikawa H.Globally and locally consistent image completion[J].ACM Transactions on Graphics,2017,36(4):1-14.
[6] Yu J,Lin Z,Yang J,et al.Generative image inpainting with contextual attention[A].Proceedings of IEEE Conference on Computer Vision and Pattern Recognition[C].USA:IEEE,2018.5505-5514.
[7] Ronneberger O,Fischer P,Brox T,et al.U-Net:convolutional networks for biomedical image segmentation[J].Siam Journal on Imaging Sciences,2015,18(4):55-67.
[8] Liu G,Reda A,Shih K,et al.Image inpainting for irregular holes using partial convolutions[J].ACM Transactions on Graphics,2018,9(3):37-51.
[9] Marcelo Bertalm,Sapiro G,Caselles V,et al.Image inpainting[A].Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques[C].USA:SIGGRAPH,2000.417-424.
[10] Ballester C,Coloma M,Bertalmio,et al.Filling-in by joint interpolation of vector fields and gray levels[J].IEEE Transactions on Image Processing,2001,10(8):1200-1211.
[11] Hays J,Efros A.Scene completion using millions of photographs[J].ACM Transactions on Graphics,2007,26(3):4-12.
[12] Pathak D,Krahenbuhl P,Donahue J,et al.Context encoders:feature learning by inpainting[A].Proceedings of IEEE Conference on Computer Vision and Pattern Recognition[C].USA:IEEE,2016.2536-2544.
[13] Harley A,Derpanis K,Kokkinos I,et al.Segmentation-aware convolutional networks using local attention masks[A].IEEE International Conference on Computer Vision[C].Italy:IEEE,2017.22-29.
[14] Sergey Ioffe,Christian Szegedy.Batch normalization:accelerating deep network training by reducing internal covariate shift[A].International Conference on Machine Learning[C].France:ICML,2015.448-456.
[15] Ledig C,Theis L,Huszar F,et al.Photo-realistic single image super-resolution using a generative adversarial network[A].Conference on Computer Vision and Pattern Recognition[C].USA:IEEE,2017.105-114.
[16] Goodfellow I,Pouget-Abadie J,Mirza M,et al.Generative adversarial networks[J].Advances in Neural Information Processing Systems,2014,3(4):2672-2680.
[17] He K,Zhang X,Ren S,et al.Deep residual learning for image recognition[A].Conference on Computer Vision and Pattern Recognition[C].USA:IEEE,2016.770-778.
[18] Zeng Y,Fu J,Chao H,et al.Learning pyramid-context encoder network for high-quality image inpainting[A].Conference on Computer Vision and Pattern Recognition[C].USA:IEEE,2019.1486-1494.
[19] Yu F,Koltun V.Multi-scale context aggregation by dilated convolutions[A].4th International Conference on Learning Representations[C].Rico:ICLR,2016.23-32.
[20] Johnson J,Alahi A,Fei-Fei L.Perceptual losses for real-time style transfer and super-resolution[A].European Conference on Computer Vision[C].USA:IEEE,2016.694-711.
[21] Zhou B,Lapedriza A,Khosla A,et al.Places:A 10 million image database for scene recognition[J].IEEE Transactions on Pattern Analysis & Machine Intelligence,2018,40(6):1452-1464.
[22] Liu Z,Luo P,Wang X,et al.Deep learning face attributes in the wild[A].IEEE International Conference on Computer Vision[C].Chile:IEEE,2015.3730-3738.
[23] Liu X,Gao J,He X,et al.Representation learning using multi-task deep neural networks for semantic classification and information retrieval[A].Proceedings of 2015 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies[C].USA:NAACL,2015.912-921.
[24] Kingm D,Ba J.Adam:A method for stochastic optimization[A].International Conference on Learning Representations[C].USA:ICLR,2014.58-64.
[25] Zhou W,Alan B,Hamid S,et al.Image quality assessment:from error visibility to structural similarity[J].IEEE Trans Image Process,2004,13(4):600-612.
[26] Korhonen J,Junyong Y.Peak signal-to-noise ratio[A].Fourth International Workshop on Quality of Multimedia Experience Electronics Letters[C].Australia:QoMEX,2012.37-38.

一种生成对抗网络用于图像修复的方法

Image Inpainting Using Generative Adversarial Networks

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	胡向东, 李之涵. 基于胶囊网络的工业互联网入侵检测方法[J]. 电子学报, 2022, 50(6): 1457-1465.
[2]	冀振燕, 韩梦豪, 宋晓军, 冯其波. 面向激光光条图像修复的循环相似度映射网络[J]. 电子学报, 2022, 50(5): 1234-1242.
[3]	廖勇, 李玉杰. 一种轻量化低复杂度的FDD大规模MIMO系统CSI反馈方法[J]. 电子学报, 2022, 50(5): 1211-1217.
[4]	刘微容, 米彦春, 杨帆, 张彦, 郭宏林, 刘仲民. 基于多级解码网络的图像修复[J]. 电子学报, 2022, 50(3): 625-636.
[5]	刘小洋, 刘加苗, 刘超, 张宜浩. 融合字符级滑动窗口和深度残差网络的僵尸网络DGA域名检测方法[J]. 电子学报, 2022, 50(1): 250-256.
[6]	张聿远, 张立民, 闫文君. 基于深度多级残差网络的低信噪比下空频分组码识别方法[J]. 电子学报, 2022, 50(1): 79-88.
[7]	李志丹, 程吉祥, 刘家伟. 基于结构偏移映射统计和多方向特征的MRF图像修复算法[J]. 电子学报, 2020, 48(5): 985-989.
[8]	秦鑫, 黄洁, 查雄, 骆丽萍, 胡德秀. 基于扩张残差网络的雷达辐射源信号识别[J]. 电子学报, 2020, 48(3): 456-462.
[9]	郑兴华, 孙喜庆, 吕嘉欣, 鲜征征, 李磊. 基于深度学习和智能规划的行为识别[J]. 电子学报, 2019, 47(8): 1661-1668.
[10]	查雄, 许漫坤, 彭华, 秦鑫, 李天昀. 基于深度残差网络的特定协议信号识别[J]. 电子学报, 2019, 47(7): 1532-1537.
[11]	江泽涛, 简雄, 刘小艳, 曾聪文. 一种改进的二进制哈希编码方法[J]. 电子学报, 2019, 47(2): 462-469.
[12]	李志丹, 和红杰, 尹忠科, 陈帆. 基于Curvelet方向特征的样本块图像修复算法[J]. 电子学报, 2016, 44(1): 150-154.
[13]	余烨, 李冰飞, 叶强强, 刘晓平. 面向球面虚拟实景底部空洞的快速修补算法[J]. 电子学报, 2015, 43(4): 822-828.
[14]	许刚, 马爽. 动态尺度块匹配约束下的链式优化图像修复研究[J]. 电子学报, 2015, 43(3): 529-535.
[15]	李志丹, 和红杰, 尹忠科, 陈帆, 仁青诺布. 基于块结构稀疏度的自适应图像修复算法[J]. 电子学报, 2013, 41(3): 549-554.