Generative Image Inpainting with Multi-Stage Decoding Network

LIU Wei-rong; MI Yan-chun; YANG Fan; ZHANG Yan; GUO Hong-lin; LIU Zhong-min

doi:10.12263/DZXB.20201451

您当前的位置：

首页 >

文章列表页 >

Generative Image Inpainting with Multi-Stage Decoding Network

PAPERS | 更新时间：2025-12-11

- Generative Image Inpainting with Multi-Stage Decoding Network
- ACTA ELECTRONICA SINICA Vol. 50, Issue 3, Pages: 625-636(2022)
- 作者机构：
  
  1.兰州理工大学电气工程与信息工程学院,甘肃兰州 730050
  2.大型电气传动系统与装备技术国家重点实验室,甘肃天水 741000
- 作者简介：
- 基金信息：
- DOI：10.12263/DZXB.20201451
  CLC： TP391
- Received：17 December 2020，
  
  Revised：2021-08-05，
  
  Published：25 March 2022
- 稿件说明：
移动端阅览
刘微容,米彦春,杨帆等.基于多级解码网络的图像修复[J].电子学报,2022,50(03):625-636.

LIU Wei-rong,MI Yan-chun,YANG Fan,et al.Generative Image Inpainting with Multi-Stage Decoding Network[J].ACTA ELECTRONICA SINICA,2022,50(03):625-636.
刘微容,米彦春,杨帆等.基于多级解码网络的图像修复[J].电子学报,2022,50(03):625-636. DOI： 10.12263/DZXB.20201451.

LIU Wei-rong,MI Yan-chun,YANG Fan,et al.Generative Image Inpainting with Multi-Stage Decoding Network[J].ACTA ELECTRONICA SINICA,2022,50(03):625-636. DOI： 10.12263/DZXB.20201451.

摘要

当前主流的图像修复方法重点依赖于自动编解码网络，此类方法试图利用编码阶段压缩后的信息在解码阶段恢复出原始图像.然而自编码网络在压缩过程中必然存在信息丢失，仅利用压缩后的信息难以得到细节丰富的修复结果，主要表现为模糊和修复区域周围明显的边缘响应.本文针对图像信息利用不完备的问题，提出多级解码网络（Multi-Stage Decoding Network，MSDN），由多个解码器对编码阶段各层特征进行解码并聚合，增大对编码器不同尺度特征的利用率，进而得到更能反映缺损区域内容的特征映射.在国际公认数据集上组织的对比实验结果表明，MSDN修复的图像视觉效果有一定提升.

Abstract

Current image inpainting methods mainly rely on automatic encoding and decoding networks. These methods try to use the information compressed in the encoding stage to restore an original image in the decoding stage. While

it is difficult to reconstruct detailed inpainting results by using only compressed information. Due to the loss of information during compression

there are visual artifacts in the results

such as blurring and obvious edge response around the reconstructed area. Aimed at the problem of incomplete utilization of image information

this manuscript proposed a multi-stage decoding network (MSDN). The MSDN decodes and aggregates features of each layer in the encoder by multiple decoders successively

which can increase utilization of features from different layers in the encoding stage and obtain better feature maps to reflect the defected area. The experiment results

which are conducted on internationally recognized datasets

show that visual effects of images generated by MSDN have been improved.

关键词

Keywords

references

ELHARROUSS O , ALMAADEED N , et al . Image inpainting: a review [J]. Neural Processing Letters , 2020 , 51 ( 2 ): 2007 ‑ 2028 .

Coloma Ballester , BERTALMIO M , CASELLES V , et al . Filling-in by joint interpolation of vector fields and gray levels [J]. IEEE Transactions on Image Processing , 2001 , 10 ( 2 ): 1200 ‑ 1211 .

ASHIKHMIN M . Synthesizing natural textures [C]// Proceedings of the 2001 symposium on Interactive 3D graphics - SI3D＇01 . New York : ACM Press , 2001 : 217 ‑ 226 .

CONNELLY B , ELI S , ADAM F , et al . Patchmatch: a randomized correspondence algorithm for structural image editing [J]. ACM Transactions on Graphics , 2009 , 28 ( 3 ): 1 ‑ 11 .

Ruzic Tijana , Pizurica Aleksandra . Context-aware patch-based image inpainting using markov random field modeling [J]. IEEE Trans Image Process, 2015 , 24 ( 1 ): 444 ‑ 456 .

李志丹 , 和红杰 , 尹忠科 , 等 . 基于curvelet方向特征的样本块图像修复算法 [J]. 电子学报 , 2016 , 44 ( 1 ): 150 ‑ 154 .

LI Zhi-dan,HE Hong-jie,YIN Zhong-ke,et al, Exemplar based image inpainting algorithm using direction features of curvelet transform [J]. Acta Electronica Sinica , 2016 , 44 ( 1 ): 150 ‑ 154 . (in Chinese)

吴晓军 , 李功清 . 基于样本和线性结构信息的大范围图像修复算法 [J]. 电子学报 , 2012 , 40 ( 8 ): 1509 ‑ 1514 .

WU Xiao-jun , LI Gong-qing . Large scale image inpainting based on exemplar and structure information [J]. Acta Electronica Sinica , 2012 , 40 ( 8 ): 1509 ‑ 1514 . (in Chinese)

强振平 , 何丽波 , 陈旭 , 等 . 深度学习图像修复方法综述 [J]. 中国图象图形学报 , 2019 , 24 : 447 ‑ 463 .

曹承瑞 . 基于密集判别与注意力特征传播的细粒度图像修复 [D]. 兰州：兰州理工大学 , 2020 .

王贵杭 . 基于几何信息的图像修复 [D]. 合肥：中国科学技术大学 , 2015 .

YANG C , LU X , LIN Z , et al . High-resolution image inpainting using multi-scale neural patch synthesis [C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway, NJ : IEEE , 2017 : 4076 ‑ 4084 .

Goodfellow Ian J , Jean Pouget-Abadie , Mehdi Mirza , et al . Generative adversarial nets [C]// Proceedings of the 27th International Conference on Neural Information Processing Systems-Volume 2 . Cambridge, MA, US : MIT Press , 2014 : 2672 ‑ 2680

PATHAK D , KRÄHENBÜHL P , DONAHUE J , et al . Context encoders: feature learning by inpainting [C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway, NJ : IEEE , 2016 : 2536 ‑ 2544 .

IIZUKA S , SIMO-SERRA E , ISHIKAWA H . Globally and locally consistent image completion [J]. ACM Transactions on Graphics , 2017 , 36 ( 4 ): 1 ‑ 14 .

YU J H , LIN Z , YANG J M , et al . Generative image inpainting with contextual attention [C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway, NJ : IEEE , 2018 : 5505 ‑ 5514 .

罗会兰 , 敖阳 , 袁璞 . 一种生成对抗网络用于图像修复的方法 [J]. 电子学报 , 2020 , 48 ( 10 ): 1891 ‑ 1898 .

LUO H L , AO Y , YUAN P . Image inpainting using generative adversarial networks [J]. Acta Electronica Sinica , 2020 , 48 ( 10 ): 1891 ‑ 1898 . (in Chinese)

王斌 , 胡辽林 , 曹京京 , 等 . 基于小波域稀疏最优的图像修复方法 [J]. 电子学报 , 2016 , 44 ( 3 ): 600 ‑ 606 .

WANG B , HU L L , CAO J J , et al . Image restoration based on sparse-optimal strategy in wavelet domain [J]. Acta Electronica Sinica , 2016 , 44 ( 3 ): 600 ‑ 606 . (in Chinese)

熊锐 , 张雷洪 , 蒋周杰 , 等 . 基于编码-解码对称神经网络的高分辨率图像重构机理研究 [J]. 光学仪器 , 2019 , 41 ( 4 ): 36 ‑ 41 .

XIONG R , ZHANG L H , JIANG Z J , et al . Research on high-resolution image reconstruction mechanism based on coding-decoding symmetric neural network [J]. Optical Instruments , 2019 , 41 ( 4 ): 36 ‑ 41 . (in Chinese)

魏域林 . 层间特征融合与多注意力的图像修复算法研究 [D]; 兰州理工大学 , 2020 .

李志丹 , 程吉祥 , 刘家伟 . 基于结构偏移映射统计和多方向特征的MRF图像修复算法 [J]. 电子学报 , 2020 , 48 ( 5 ): 985 ‑ 989 .

LI Z D , CHENG J X , LIU J W . MRF image inpainting algorithm based on structure offsets statistics and multi-direction features [J]. Acta Electronica Sinica , 2020 , 48 ( 5 ): 985 ‑ 989 . (in Chinese)

LIU G L , REDA F A , SHIH K J , et al . Image inpainting for irregular holes using partial convolutions [M]// Computer Vision-ECCV 2018 . Cham : Springer International Publishing , 2018 : 89 ‑ 105 .

WANG Yi , TAO Xin , QI Xiao-juan , et al . Image inpainting via generative multi-column convolutional neural networks [C]// Proceedings of the 32nd International Conference on Neural Information Processing Systems . Red Hook, NY : Curran Associates Inc , 2018 : 329 ‑ 338 .

YU J H , LIN Z , YANG J M , et al . Free-form image inpainting with gated convolution [C]// 2019 IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway, NJ : IEEE , 2019 : 4470 ‑ 4479 .

ZENG Y H , FU J L , CHAO H Y , et al . Learning pyramid-context encoder network for high-quality image inpainting [C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway, NJ : IEEE , 2019 : 1486 ‑ 1494 .

LI J Y , WANG N , ZHANG L F , et al . Recurrent feature reasoning for image inpainting [C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway, NJ : IEEE , 2020 : 7757 ‑ 7765 .

鲍建敏 . 基于生成对抗网络的图像合成 [D]. 合肥 : 中国科学技术大学 , 2019 .

BAO J M . Image Synthesis Based on Generative Adversarial Networks [D]. Hefei : University of Science and Technology of China , 2019 . (in Chinese)

Kamyar Nazeri , Eric Ng , Tony Joseph , et al . Edgeconnect: generative image inpainting with adversarial edge learning [EB/OL].( 2019-01-11 ). https://arxiv.org/abs/1901. 00212v3 https://arxiv.org/abs/1901.00212v3 .

KARRAS T , AILA T , LAINE S , et al . Progressive growing of GANs for improved quality, stability, and variation [EB/OL]. ( 2018-02-26 ). https://arxiv.org/abs/1710.10196 https://arxiv.org/abs/1710.10196 .

Radim Tyleček , Radim Šára . Spatial pattern templates for recognition of objects with regular structure [C]// German Conference on Pattern Recognition . Berlin : Springer , 2013 : 364 ‑ 374 .

Zhou Bo-lei , Agata Lapedriza , Aditya Khosla , et al . Places: a 10 million image database for scene recognition [J]. IEEE transactions on pattern analysis and machine intelligence , 2017 , 40 ( 6 ): 1452 ‑ 1464 .

Olaf Ronneberger , Philipp Fischer , Thomas Brox . U-Net: convolutional networks for biomedical image segmentation [C]// Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015 , Part Ⅲ . Berlin : Springer , 2015 : 234 ‑ 241 .

HORÉ A , ZIOU D . Image quality metrics: PSNR vs. SSIM [C]// 2010 20th International Conference on Pattern Recognition . Piscataway, NJ : IEEE , 2010 : 2366 ‑ 2369 .

WANG Zhou . Image quality assessment : from error visibility to structural similarity [J]. IEEE Transactions on Image Processing , 2004 , 13 ( 4 ): 600 ‑ 612 .

HEUSEL M , RAMSAUER H , UNTERTHINER T , et al . GANs trained by a two time-scale update rule converge to a local Nash equilibrium [C]// NIPS＇17: Proceedings of the 31st International Conference on Neural Information Processing Systems . Red Hook, NY, US : Curran Associates Inc , 2017 : 6629 ‑ 6640 .

GAO Xiao-li , FANG Yi-xin . A note on the generalized degrees of freedom under the L1 loss function [J]. Journal of Statistical Planning & Inference , 2011 , 141 ( 2 ): 677 ‑ 686 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Dual-Stream Attention Image Inpainting Method Based on Interacting and Fusing Internal-External Features

Recurrent Similarity Mapping Network Oriented to Laser Stripe Image Inpainting

Adaptive Image Inpainting Algorithm Based on Patch Structure Sparsity

Large Scale Image Inpainting Based on Exemplar and Structure Information

Related Author

JIANG Xue-qin

ZHOU Shu-bo

HUANG Rong

HUANG Guang-yuan

JI Zhen-yan

HAN Meng-hao

SONG Xiao-jun

FENG Qi-bo

Related Institution

Engineering Research Center of Digitized Textile & Apparel Technology, Ministry of Education, Donghua University

College of Information Science and Technology, Donghua University

Department of Software, Beijing Jiaotong University

Department of Science, Beijing Jiaotong University

School of Computer Science and TechnologyShandong Institute of Economic and TechnologyYantaiShandong 264005China

⁰