基于深度网络的图像语义分割综述

doi:10.3969/j.issn.0372-2112.2019.10.024

摘要/Abstract

摘要： 图像语义分割不仅预测一幅图像中的不同类别，同时还定位不同语义类别的位置，具有重要的研究意义和应用价值.本文阐述了图像语义分割最新的研究成果和方法，从三个角度综述了基于深度卷积神经网络的图像语义分割模型，分别是基于候选区域模型、基于全卷积网络模型和基于弱监督学习的语义分割模型，对这三类模型的方法和结构进行了详细的研究和分析.并在PASCAL VOC 2012数据集上对一些代表性的语义分割算法的性能进行了比较分析.

关键词: 图像语义分割, 深度卷积神经网络, 候选区域, 全卷积网络, 弱监督学习, PASCAL VOC 2012数据集

Abstract: Image semantic segmentation not only predicts different categories in an image, but also orientates different semantic categories locations, which has important research significance and application value. This paper expounds the latest research results and methods of image semantic segmentation,and from three perspectives: based candidate region models、based full convolutional network models and based weakly-supervised learning models. The image semantic segmentation model methods and structure based on deep convolutional neural network are deeply overviewed in this paper. This paper compares the performance of some representative semantic segmentation algorithms on the PASCAL VOC 2012 dataset.

Key words: image semantic segmentation, deep convolutional neural network, candidate region, full convolution network, weakly-supervised learning, PASCAL VOC 2012 dataset

中图分类号:

TP391

罗会兰, 张云. 基于深度网络的图像语义分割综述[J]. 电子学报, 2019, 47(10): 2211-2220.

LUO Hui-lan, ZHANG Yun . A Survey of Image Semantic Segmentation Based on Deep Network[J]. Acta Electronica Sinica, 2019, 47(10): 2211-2220.

参考文献

[1] Gould S,Fulton R,Koller D.Decomposing a scene into geometric and semantically consistent regions[A].Proceedings of the IEEE International Conference on Computer Vision[C].Kyoto,Japan:IEEE,2009.1-8.
[2] Gupta A,Efros A A,Hebert M.Blocks world revisited:image understanding using qualitative geometry and mechanics[A].Proceedings of the European Conference on Computer Vision[C].Crete,Greece:ACM,2010.482-496.
[3] Yu S X,Zhang H,Malik J.Inferring spatial layout from a single image via depth-ordered grouping[A].Proceedings of the Computer Vision and Pattern Recognition[C].USA:IEEE,2008.1-7.
[4] Lee D C,Hebert M,Kanade T.Geometric reasoning for single image structure recovery[A].Proceedings of the Computer Vision and Pattern Recognition[C].USA:IEEE,2009.2136-2143.
[5] 李亚峰.一种基于多字典学习的图像分割模糊方法[J].电子学报,2018,46(7):1700-1709. Li Ya-feng.An image segmentation fuzzy method based on multi-dictionary learning[J].Acta Electronica Sinica,2018,46(7):1700-1709.(in Chinese)
[6] Shelhamer E,Long J,Darrell T.Fully convolutional networks for semantic segmentation[J].IEEE Transactions on Pattern Analysis & Machine Intelligence,2015,39(4):640-651.
[7] Hung W C,Tsai Y H,Liou Y T,et al.Adversarial learning for semi-supervised semantic segmentation[J].CORR,2018,57(8):7540-7551.
[8] Souly N,Spampinato C,Shah M.Semi supervised semantic segmentation using generative adversarial network[A].Proceedings of the IEEE International Conference on Computer Vision[C].New York,USA:IEEE,2017.5689-5697.
[9] Ghiasi G,Fowlkes C C.Laplacian Pyramid reconstruction and fefinement for semantic segmentation[A].Proceedings of the European Conference on Computer Vision[C].USA:IEEE,2016.519-534.
[10] Lin G,Shen C,Hengel A V D,et al.Efficient piecewise training of deep structured models for semantic segmentation[A].Proceedings of the Computer Vision and Pattern Recognition[C].USA:IEEE,2016.3194-3203.
[11] Liu Z,Li X,Luo P,et al.Semantic image segmentation via deep parsing network[A].Proceedings of the IEEE International Conference on Computer Vision[C].Santiago,Chile:IEEE,2015.1377-1385.
[12] Papandreou G,Chen L C,Murphy K P,et al.Weakly-and semi-supervised learning of a deep convolutional network for semantic image segmentation[A].Proceedings of the IEEE International Conference on Computer Vision[C].Santiago,Chile:IEEE,2015.1742-1750.
[13] Peng C,Zhang X,Yu G,et al.Large kernel matters-improve semantic segmentation by global convolutional network[J].IEEE Conference on Computer Vision and Pattern Recognition,2017,189(1):1743-1751.
[14] Ning Q,Zhu J,Chen C.Very fast semantic image segmentation using hierarchical dilation and feature refining[J].Cognitive Computation,2017,10(2):1-11.
[15] Jiang Z,Yuan Y,Wang Q.Contour-aware network for semantic segmentation via adaptive depth[J].Neurocomputing,2018,284(1):27-35.
[16] Zhu H,Mng F,Cai J,et al.Beyond pixels:A comprehensive survey from bottom-up to semantic image segmentation and cosegmentation[J].Journal of Visual Communication & Image Representation,2015,34(2):12-27.
[17] Thoma M.A Survey of semantic segmentation[J].CORR,2016,65(41):1-15.
[18] Guo Y,Liu Y,Georgiou T,et al.A review of semantic segmentation using deep neural networks[J].International Journal of Multimedia Information Retrieval,2017,7(2):87-93.
[19] Garcia-garcia A,Orts-Escolano S,Oprea S,et al.A review on deep learning techniques applied to semantic segmentation[J].CORR,2017,75(9):41-65.
[20] Geng Q,Zhou Z,Cao X.Survey of recent progress in semantic image segmentation with CNNs[J].Science China(Information Sciences),2018,61(5):1101-1118.
[21] Shetty S.Application of convolutional neural network for image classification on pascal voc challenge 2012 dataset[J].CORR,2016,21(3):1-6.
[22] Carreira J,Sminchisescu C.CPMC:Automatic object segmentation using constrained parametric min-cuts[J].IEEE Transactions on Pattern Analysis & Machine Intelligence,2012,34(7):1312-1328.
[23] Carreira J,Rui C,Batista J,et al.Semantic segmentation with second-order pooling[J].European Conference on Computer Vision,2012,7578(1):430-443.
[24] Girshick R,Donahue J,Darrell T,et al.Rich feature hierarchies for accurate object detection and semanticsegmentation[A].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition[C].USA:IEEE,2014.580-587.
[25] Uijlings J R R,Van DE Sande K E A,Gevers T,et al.Selective search for object recognition[J].IJCV,2013,104(2):154-171.
[26] Hariharan B,Arbel EZ P,Girshick R,et al.Simultaneous detection and segmentation[A].Proceedings of the European Conference on Computer Vision[C].USA:IEEE,2014.297-312.
[27] Arbel EZ P,Pont-Tuset J,Barron J,et al.Multiscale combinatorial grouping[A].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition[C].USA:IEEE,2014.328-335.
[28] Ren S,Girshick R,et al.Faster R-CNN:Towards real-time object detection with region proposal networks[J].IEEE Transactions on Pattern Analysis & Machine Intelligence,2017,39(6):1137-1149.
[29] Caesar H,Uijlings J,Ferrari V.Region-based semantic segmentation with end-to-end training[A].Proceedings of the European Conference on Computer Vision[C].USA:IEEE,2016.381-397.
[30] Girshick R.Fast R-CNN[A].Proceedings of the IEEE International Conference on Computer Vision[C].Santiago,Chile:IEEE,2015.1-9.
[31] X J,Y G,Z F,et al.An end-to-end human segmentation by region proposed fully convolutional network[A].IEEE Access[C].USA:IEEE,2019.16395-16405.
[32] Noh H,Hong S,Han B.Learning deconvolution network for semantic segmentation[A].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition[C].Santiago,Chile:IEEE,2015.1520-1528.
[33] Simonyan K,Zisserman A.Very deep convolutional networks for large-scale image recognition[A].International Conference on Learing Representation[C].USA:IEEE,2014.1-14
[34] 李宝奇,贺昱曜,何灵蛟,等.基于全卷积神经网络的非对称并行语义分割模型[J].电子学报,2019,47(5):1058-1064. Li Bao-qi,HE Yu-yao,HE ling-jiao,et al.Asymmetric parallel semantic segmentation model based on full convolutional neural network[J].Acta Electronica Sinica,2019,47(5):1058-1064.(in Chinese)
[35] Badrinarayanan V,Kendall A,Cipolla R.SegNet:A deep convolutional encoder-decoder architecture for scene segmentation[J].IEEE Transactions on Pattern Analysis & Machine Intelligence,2015,39(1):2481-2495.
[36] Ronneberger O,Fischer P,Brox T.U-Net:Convolutional networks for biomedical image segmentation[J].Medical Image Computing and Computer-Assisted Intervention,2015,56(9):234-241.
[37] MIilletar F,Navab N,Ahmadi S A.V-Net:Fully convolutional neural networks for volumetric medical image segmentation[A].Proceedings of the International Conference on 3d Vision[C].USA:IEEE,2016.565-571.
[38] Chen L C,Papandreou G,Kokkinos I,et al.Semantic image segmentation with deep convolutional nets and fully connected crfs[J].Computer Science,2014,4):357-361.
[39] Chen L C,Papandreou G,Kokkinos I,et al.DeepLab:Semantic image segmentation with deep convolutional nets,atrous convolution,and fully connected crfs[J].IEEE Transactions on Pattern Analysis & Machine Intelligence,2017,40(4):834-848.
[40] Chen L C,Zhu Y,Papandreou G,et al.Encoder-decoder with atrous separable convolution for semantic image segmentation[A].Proceedings of the European Conference on Computer Vision[C].Germany:IEEE,2018.833-851.
[41] Kr H H P,Koltun V.Efficient inference in fully connected crfs with Gaussian edge potentials[J].CORR,2012,34(2):1-9.
[42] Chandra S,Kokkinos I.Fast,exact and multi-scale inference for semantic image segmentation with deep Gaussian crfs[A].Proceedings of the European Conference on Computer Vision[C].USA:IEEE,2016.402-418.
[43] Pohlen T,Hermans A,Mathias M,et al.Full-fesolution residual networks for semantic segmentation in street scenes[A].Proceedings of the Computer Vision and Pattern Recognition[C].USA:IEEE,2016.3309-3318.
[44] Pinheiro P O,Collobert R.From image-level to pixel-level labeling with convolutional networks[A].Proceedings of the Computer Vision and Pattern Recognition[C].USA:IEEE,2015.1713-1721.
[45] Lin G,Milan A,Shen C,et al.RefineNet:multi-path refinement networks for high-resolution semantic segmentation[A].Proceedings of the Computer Vision and Pattern Recognition[C].USA:IEEE,2016.5168-5177.
[46] Zhao H,Shi J,Qi X,et al.Pyramid scene parsing network[A].IEEE Conference on Computer Vision and Pattern Recognition[C].USA:IEEE,2016.6230-6239.
[47] Luo P,Wang G,Lin L,et al.Deep dual learning for semantic image segmentation[A].Proceedings of the IEEE International Conference on Computer Vision[C].Venice,Italy:IEEE,2017.2737-2745.
[48] Goodfellow I J,Pouget-Abadie J,Mirza M,et al.Generative adversarial nets[A].Proceedings of the International Conference on Neural Information Processing Systems[C].USA:ACM,2014.2672-2680.
[49] Luc P,Couprie C,Chintala S,et al.Semantic segmentation using adversarial networks[J].CORR,2016,56(7):1-12.
[50] Metzen J H,Kumar M C,Brox T,et al.Universal adversarial perturbations against semantic image segmentation[A].Proceedings of the IEEE International Conference on Computer Vision[C].Venice,Italy:IEEE,2017.2774-2783.
[51] Xie C,Wang J,Zhang Z,et al.Adversarial examples for semantic segmentation and object detection[A].Proceedings of the IEEE International Conference on Computer Vision[C].USA:IEEE,2017.1378-1387.
[52] Zhu W,Xiang X,Tran T D,et al.Adversarial deep structural networks for mammographic mass segmentation[J].CORR,2017,43(6):1-9.
[53] Rezaei M,Harmuth K,Gierke W,et al.A conditional adversarial network for semantic segmentation of brain tumor[A].Proceedings of the International MICCAI Brainlesion Workshop[C].USA:IEEE,2017.241-252.
[54] Dai J,He K,Sun J.BoxSup:exploiting bounding boxes to supervise convolutional networks for semantic segmentation[A].IEEE International Conference on Computer Vision[C].USA:IEEE,2015.1635-1643.
[55] Khoreva A,Benenson R,Hosang J,et al.Simple does it:weakly supervised instance and semantic segmentation[A].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition[C].USA:IEEE,2017.1665-1674.
[56] Lin D,Dai J,Jia J,et al.ScribbleSup:scribble-supervised convolutional networks for semantic segmentation[A].Proceedings of the Computer Vision and Pattern Recognition[C].USA:IEEE,2016.3159-3167.
[57] Vernaza P,Chandraker M.Learning random-walk label propagation for weakly-supervised semantic segmentation[A].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition[C].USA:IEEE,2017.2953-2961.
[58] Tang M,Djelouah A,Perazzi F,et al.Normalized cut loss for weakly-supervised cnn segmentation[A].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition[C].USA:IEEE,2018.1818-1827.
[59] Russakovsky O,Deng J,Su H,et al.Imagenet large scale visual recognition challenge[J].International Journal of Computer Vision,2015,115(3):211-252.
[60] Pathak D,Krahenbuhl P,Darrell T.Constrained convolutional neural networks for weakly supervised segmentation[A].Proceedings of the IEEE International Conference on Computer Vision[C].Santiago,Chile:IEEE,2015.1796-1804.
[61] Wei Y,Liang X,Chen Y,et al.STC:A simple to complex framework for weakly-supervised semantic segmentation[J].IEEE Transactions on Pattern Analysis & Machine Intelligence,2016,39(11):2314-2320.
[62] Jin B,Segovia M V O,Susstrunk S.Webly supervised semantic segmentation[A].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition[C].USA:IEEE,2017.1705-1714.
[63] Hong S,Yeo D,Kwak S,et al.Weakly supervised semantic segmentation using web-crawled videos[A].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition[C].USA:IEEE,2018.2224-2232.
[64] Ahn J,Kwak S.Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation[A].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition[C].USA:IEEE,2018.4981-4990.
[65] Cordts M,Omran M,Ramos S,et al.The cityscapes dataset[A].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition[C].USA:IEEE,2015.3213-3223.
[66] Brostow G J,Fauqueur J,Cipolla R.Semantic object classes in video:A high-definition ground truth database[J].Pattern Recognition Letters,2009,30(2):88-97.
[67] Ros G,Sellart L,Materzynska J,et al.The SYNTHIA Dataset:A large collection of synthetic images for semantic segmentation of urban scenes[A].Proceedings of the Computer Vision and Pattern Recognition[C].USA:IEEE,2016.3234-3243.
[68] Lin T Y,Maire M,Belongie S,et al.Microsoft COCO:Common objects in context[A].Proceedings of the European Conference on Computer Vision[C].USA:IEEE,2014.740-755.