协同视觉显著性检测方法综述

doi:10.3969/j.issn.0372-2112.2019.06.024

电子学报 ›› 2019, Vol. 47 ›› Issue (6): 1352-1365.DOI: 10.3969/j.issn.0372-2112.2019.06.024

协同视觉显著性检测方法综述

钱晓亮¹, 白臻¹, 陈渊¹, 张鼎文², 史坤峰¹, 王芳¹, 吴青娥¹, 毋媛媛¹, 王慰¹

1. 郑州轻工业学院电气信息工程学院, 河南郑州 450002;
2. 西安电子科技大学机电工程学院, 陕西西安 710071

收稿日期:2018-05-02 修回日期:2018-08-02 出版日期:2019-06-25
- 通讯作者:
- 毋媛媛, 王慰
- 作者简介:
- 钱晓亮 男,1982年10月生于河南孟州.2013年毕业于西北工业大学,获工学博士学位.现为郑州轻工业学院电气信息工程学院校聘副教授,硕士生导师.主要研究方向为计算机视觉与人工智能,视觉注意力计算,遥感图像分析与处理.发表SCI/EI论文20余篇;授权国家发明专利9项;主持或参与国家自然科学基金、河南省高等学校重点科研项目、河南省科技攻关项目、河南省高校科技创新团队等项目10余项.E-mail:qxl_sunshine@163.com;白臻女,1992年11月生于河南巩义.2016年毕业于武汉华夏理工学院,获学士学位.现为郑州轻工业学院电气信息工程学院硕士研究生.主要研究方向为计算机视觉与机器学习,视觉显著性检测.E-mail:bz532476@163.com
- 基金资助:
- 国家自然科学基金 (No.61501407，No.61603350，No.61703373）; 河南省高校科技创新团队 (No.19IRTSTHN013）; 河南省高等学校重点科研项目 (No.19A413014，No.16A413017）; 郑州轻工业学院博士基金 (No.2014BSJJ016，No.2015BSJJ004，No.2017BSJJ008）

A Review of Co-saliency Detection

QIAN Xiao-liang¹, BAI Zhen¹, CHEN Yuan¹, ZHANG Ding-wen², SHI Kun-feng¹, WANG Fang¹, WU Qing-e¹, WU Yuan-yuan¹, WANG Wei¹

1. School of Electrical and Information Engineering, Zhengzhou University of Light Industry, Zhengzhou, Henan 450002, China;
2. School of Mechano-Electronic Engineering, Xidian University, Xi'an, Shaanxi 710071, China

Received:2018-05-02 Revised:2018-08-02 Online:2019-06-25 Published:2019-06-25
- Supported by:
- National Natural Science Foundation of China (No.61501407, No.61603350, No.61703373); Science and Technology Innovation Team of Colleges and Universities in Henan Province (No.19IRTSTHN013); Key Scientific Research Programs of colleges and universities of Henan Province (No.19A413014, No.16A413017); Doctoral Foundation od Zhengzhou University of Light Industry (No.2014BSJJ016, No.2015BSJJ004, No.2017BSJJ008)

摘要/Abstract

摘要： 协同视觉显著性检测是视觉注意力计算领域中一个快速发展的新兴分支，致力于检测多幅相关场景图像中的公共显著目标，在各种计算机视觉任务中有广泛应用.考虑到特征提取策略的设计是协同视觉显著性检测当前研究的重点，本文首先根据特征提取策略的不同对现有的协同视觉显著检测方法进行了分类介绍和定性分析.其次，通过在5个公开数据库上的主观和定量对比，对各流行算法的性能进行了评估，分析了特征提取策略对算法性能的影响以及各数据库的复杂度，并验证了协同视觉显著性检测和视觉显著性检测的区别.最后，对本文工作进行了总结，并对当前研究中存在的问题和未来的研究工作进行了讨论.

关键词: 视觉注意力, 协同视觉显著性, 特征提取策略, 手工特征, 浅层学习特征, 深度学习特征

Abstract: Co-saliency detection is a new branch with the rapid development in the field of visual attention,which concerns the detection of the common salient objects from multiple relevant scene images,and can be widely used in various computer vision tasks.Considering the key point of current research is the design of feature extraction strategy,the existing co-saliency detection methods are firstly summarized and qualitatively analyzed according to the different feature extraction strategies in this paper.Subsequently,based on the subjective and quantitative comparisons in the five open datasets,the performance of the state-of-the-art algorithms is evaluated,the influence of the feature extraction strategy on the performance of algorithms and the complexity of the datasets is analyzed,and the difference of co-saliency detection and saliency detection is also verified.Finally,the conclusion of this paper are presented,the problems of current research and the future development are also discussed.

Key words: visual attention, co-saliency, feature extraction strategy, hand-designed features, shallow learning features, deep learning feature

中图分类号:

TP391.4

钱晓亮, 白臻, 陈渊, 等. 协同视觉显著性检测方法综述[J]. 电子学报, 2019, 47(6): 1352-1365.

QIAN Xiao-liang, BAI Zhen, CHEN Yuan, et al. A Review of Co-saliency Detection[J]. Acta Electronica Sinica, 2019, 47(6): 1352-1365.

参考文献

[1] QIAN Xiao-liang,ZHANG Huan-long,LIU Yu-cui,et al.Visual Significance Detection Method Based on Self-Learning Characteristics and Matrix Low-Rank Recovery:China ZL 201610926623.0[P].2017.
[2] WANG Wen-guan,SHEN Jian-bing,LI Xue-long,et al.Robust video object cosegmentation[J].IEEE Transactions on Image Processing,2015,24(10):3137-3148.
[3] ZHU Hong-yuan,MENG Fan-man,CAI Jian-fei,et al.Beyond pixels:A comprehensive survey from bottom-up to semantic image segmentation and cosegmentation[J].Journal of Visual Communication & Image Representation,2016,34(2):12-27.
[4] CHO Min-su,SHIN Young-min,LEE Kyoung-mu.Co-recognition of image pairs by data-driven Monte Carlo image exploration[A].Proceedings of the 10th European Conference on Computer Vision[C].Berlin:Springer,2008.144-157.
[5] GOFERMAN S,TAL A,ZELNIK M L.Puzzle-like collage[J].Computer Graphics Forum,2010,29(2):459-468.
[6] 柯圣财,赵永威,李弼程,等.基于卷积神经网络和监督核哈希的图像检索方法[J].电子学报,2017,45(1):157-163. KE Sheng-cai,ZHAO Yong-wei,LI Bi-cheng,et al.Image retrieval based on convolutional neural network and kernel-based supervised hashing[J].Acta Electronica Sinica,2017,45(1):157-163.(in Chinese)
[7] JACOBS D E,DAN B G,SHECHTMAN Eli.Cosaliency:where people look when comparing images[A].Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology[C].New York:ACM,2010.219-228.
[8] CHEN H T.Preattentive co-saliency detection[A].Proceedings of 2010 IEEE International Conference on Image Processing[C].Piscataway,NJ:IEEE,2010.1117-1120.
[9] LI Hong-liang,NGAN K N.A co-saliency model of image pairs[J].IEEE Transactions on Image Processing,2011,20(12):3365-3375.
[10] FU Hua-zhu,CAO Xiao-chun,TU Zhuo-wen.Cluster-based co-saliency detection[J].IEEE Transactions on Image Processing,2013,22(10):3766-3778.
[11] LI Hong-liang,MENG Fan-man,NGAN K N.Co-salient object detection from multiple images[J].IEEE Transactions on Multimedia,2013,15(8):1896-1909.
[12] JERRIPOTHULA K R,CAI Jian-fei,YUAN Jun-song.CATS:co-saliency activated tracklet selection for video co-localization[A].Proceedings of 18th European Conference on Computer Vision[C].Berlin:Springer,2016.187-202.
[13] YAO Xi-wen,HAN Jun-wei,ZHANG Ding-wen,et al.Revisiting co-saliency detection:a novel approach based on two-stage multi-view spectral rotation co-clustering[J].IEEE Transactions on Image Processing,2017,26(7):3196-3209.
[14] GE Chen-jie,FU Ke-ren,LIU Fang-hui,et al.co-saliency detection via inter and intra saliency propagation[J].Signal Processing-image Communication,2016,44:69-83.
[15] ZHANG Ding-wen,FU Hua-zhu,HAN Jun-wei,et al.A Review of co-saliency detection technique:fundamentals,applications,and challenges[A].LNCS:Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition[C].New York:IEEE,2016.arXiv:1604.07090.
[16] ZHANG Ding-wen,MENG Deyu,Han Junwei.co-saliency detection via a self-paced multiple-instance learning framework[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2017,39(5):865-878.
[17] ZHANG Ding-wen,MENG De-yu,LI Chao,et al.A self-paced multiple-instance learning framework for co-saliency detection[A].Proceedings of the 2015 IEEE International Conference on Computer Vision[C].Washington:IEEE,2015.594-602.
[18] CAO Xiao-chun,CHENG Yu-peng,TAO Zhi-qiang,et al.co-saliency detection via base reconstruction[A].Proceedings of the 22nd ACM International Conference on Multimedia[C].New York:ACM,2014.997-1000.
[19] CAO Xiao-chun,TAO Zhi-qiang,ZHANG Bao,et al.Self-adaptively weighted co-saliency detection via rank constraint[J].IEEE Transactions on Image Processing,2014,23(9):4175-4186.
[20] LIU Zhi,ZOU Wen-bin,LI Li-na,et al.Co-saliency detection based on hierarchical segmentation[J].IEEE Signal Processing Letters,2014,21(1):88-92.
[21] 孙晓,潘汀.基于兴趣区域深度神经网络的静态面部表情识别[J].电子学报,2017,45(5):1189-1197. SUN Xiao,PAN Ting.Static facial expression recognition system using ROI deep neural network[J].Acta Electronica Sinica,2017,45(5):1189-1197.(in Chinese)
[22] JUDD Tilke,EHINGER K A,DURAND Fredo,et al.Learning to predict where humans look[A].Proceedings of 2009 IEEE 12th International Conference on Computer Vision[C].Piscataway,NJ:IEEE,2009.2106-2113.
[23] LI Yin,ZHOU Yue,YAN Jun-chi,et al.Visual saliency based on conditional entropy[A].Proceedings of the 9th Asian Conference on Computer Vision[C].Berlin:ACM,2009.246-257.
[24] YAN Jun-chi,LIU Jian,LI Yin,et al.Visual saliency detection via rank-sparsity decomposition[A].Proceedings of 2010 IEEE International Conference on Image Processing[C].Piscataway,NJ:IEEE,2010.1089-1092.
[25] WANG Wei,CHEN Cheng,WANG Yi-zhou,et al.Simulating human saccadic scanpaths on natural images[A].Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition[C].DC:IEEE,2011.441-448.
[26] KUMMERER M,THEIS L,BETHGE M.Deepgaze I:boosting saliency prediction with feature maps trained on imagenet[A].LNCS:Proceedings of Poster Presented at International Conference on Learning Representations[C].Berlin:Springer,2014.arXiv:1411.1045.
[27] KUMMERER M,WALLIS T S,BETHGE M.Deep gaze Ⅱ:reading fixations from deep features trained on object recognition[A].LNCS:Proceedings of the 2016 Computer Vision and Pattern Recognition[C].Berlin:Springer,2016.arXiv:1610.01563.
[28] KRUTHIVENTI S S,GUDISA V,DHOLAKIYA J H,et al.Saliency unified:a deep architecture for simultaneous eye fixation prediction and salient object segmentation[A].Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition[C].Piscataway,NJ:IEEE,2016.5781-5790.
[29] CORNIA M,BARALDI L,SERRA G,et al.A deep multi-level network for saliency prediction[A].Proceedings of 23rd International Conference on Pattern Recognition[C].New York:IEEE,2016.3488-3493.
[30] HOU Xiao-di,ZHANG Liqing.Saliency detection:a spectral residual approach[A].Proceedings of the 2007 Computer Vision and Pattern Recognition[C].Berlin:Springer,2007.1-8.
[31] HOU Xiao-di,HAREL J,KOCH C.Imagesignature:highlighting sparse salient regions[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2012,34(1):194-201.
[32] AVRAHAM T,LINDENBAUM M.Esaliency (extended saliency):meaningful attention using stochastic image modeling[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2010,32(4):693-708.
[33] HAREL J,KOCH C,PERONA P.Graph-based visual saliency[A].Proceedings of the 19th International Conference on Neural Information Processing Systems[C].Cambridge:MIT,2006.545-552.
[34] 郑云飞,张雄伟,曹铁勇,等.基于全卷积网络的语义显著性区域检测方法研究[J].电子学报,2017,45(11):2593-2601. ZHENG Yun-fei,ZHANG Xiong-wei,CAO Tie-yong,et al.The samantic salient region detection algorithm based on the fully convolutional networks[J].Acta Electronica Sinica,2017,45(11):2593-2601.(in Chinese)
[35] LIU Tie,YUAN Ze-jian,SUN Jian,et al.Learning to detect a salient object[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2011,33(2):353-367.
[36] CHENG Ming-ming,MITRA N J,HUANG Xiao-lei,et al.Global contrast based salient region detection[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2015,37(3):569-582.
[37] LI Yin,ZHOU Yue,XU Lei,et al.Incremental sparse saliency detection[A].Proceedings of the 16th IEEE International Conference on Image Processing[C].Piscataway,NJ:IEEE,2009.3093-3096.
[38] JIANG Huai-zu,WANG Jing-dong,YUAN Ze-jian,et al.Salient object detection:a discriminative regional feature integration approach[A].Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition[C].Washington:IEEE,2013.2083-2090.
[39] SHEN Xiao-hui,WU Ying.A unified approach to salient object detection via low rank matrix recovery[A].Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition[C].Washington:IEEE,2012.853-860.
[40] TONG Na,LU Hu-chuan,RUAN Xiang,et al.Salient object detection via bootstrap learning[A].Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition[C].Washington:IEEE,2015.1884-1892.
[41] HAN Jun-wei,ZHANG Ding-wen,HU Xin-tao,et al.Background prior-based salient object detection via deep reconstruction residual[J].IEEE Transactions on Circuits and Systems for Video Technology,2015,25(8):1309-1321.
[42] ITTI L,KOCH C,NIEBUR E.A model of saliency-based visual attention for rapid scene analysis[J].IEEE Transactions on Pattern Analysis & Machine Intelligence,1998,20(11):1254-1259.
[43] LI Li-na,LIU Zhi,ZOU Wen-bin,et al.Co-saliency detection based on region-level fusion and pixel-level refinement[A].Proceedings of 2014 IEEE International Conference on Multimedia and Expo[C].New York:IEEE,2014.1-6.
[44] ACHANTA R,HEMAMI S,ESTRADA F,et al.Frequency-tuned salient region detection[A].Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition[C].Piscataway:IEEE,2009.1597-1604.
[45] HOU Xiao-di,ZHANG Li-qing.Saliency detection:a spectral residual approach[A].Proceedings of 2007 IEEE Conference on Computer Vision and Pattern Recognition[C].New York:IEEE,2007.1-8.
[46] SONG Hang-ke,LIU Zhi,XIE Yu-feng,et al.RGBD Co-saliency detection via bagging-based clustering[J].IEEE Signal Processing Letters,2016,23(12):1722-1726.
[47] HUANG Rui,FENG Wei,SUN Ji-zhou.Saliency and co-saliency detection by low-rank multiscale fusion[A].Proceedings of 2015 IEEE International Conference on Multimedia and Expo[C].New York:IEEE,2015.1-6.
[48] SONG Hang-ke,LIU Zhi,DU Huan,et al.Depth-aware saliency detection using discriminative saliency fusion[A].Proceedings of 2016 IEEE International Conference on Acoustics,Speech and Signal Processing[C].New York:IEEE,2016.1626-1630.
[49] ARBELAEZ P,MAIRE M,FOWLKES C,et al.Contour detection andhierarchical image segmentation[J].IEEE Transactions on Pattern Analysis & Machine Intelligence,2011,33(5):898-916.
[50] YE Lin-wei,LIU Zhi,LI Jun-hao,et al.Co-saliency detection via Co-salient object discovery and recovery[J].IEEE Signal Processing Letters,2015,22(11):2073-2077.
[51] DU Shu-ze,CHEN Shi-feng.Detecting Co-salient objects in large image sets[J].IEEE Signal Processing Letters,2014,22(2):145-148.
[52] CONG Run-min,LEI Jian-jun,FU Hua-zhu,et al.An iterative Co-saliency framework for RGBD images[J].IEEE Transactions on Cybernetics,2017,PP(99):1-14.
[53] ZHANG Zhao-feng,WU Ze-min,JIANG Qing-zhu,et al.Co-saliency detection based on superpixel matching and cellular automata[J].Ksii Transactions on Internet & Information Systems,2017,11(5):2576-2589.
[54] SHEN Ning-min,LI Jing,ZHOU Pei-yun,et al.BSFCoS:block and sparse principal component analysis-based fast Co-saliency detection method[J].International Journal of Pattern Recognition and Artificial Intelligence,2016,30(1):1655003.
[55] 胡正平,陈俊岭.多层融合深度局部PCA子空间稀疏优化特征提取模型[J].电子学报,2017,45(10):2383-2389. HU Zheng-ping,CHEN Juan-ling.Featureextraction model based on multi-layered deep local subspace sparse optimization[J].Acta Electronica Sinica,2017,45(10):2383-2389.(in Chinese)
[56] CHEN Duan-yu,LIN Chuan-yu,YANG NienTzu,et al.Sparse coding-based co-salient object detection with application to video abstraction[A].Proceedings of 2013 International Conference on Machine Learning and Cybernetics[C].New York:IEEE,2013.1474-1479.
[57] 钱晓亮,郭雷,韩军伟,等.一种基于加权稀疏编码的频域视觉显著性检测算法[J].电子学报,2013,41(6):1159-1165. QIAN Xiao-liang,GUO Lei,HAN Jun-wei,et al.Aspetrcal algorithm based on weighted sparse coding for visual saliency detection[J].Acta Electronica Sinica,2013,41(6):1159-1165.(in Chinese)
[58] HINTON G E,SALAKHUTDINOV R.Reducing the dimensionality of data with neural networks[J].Science,2006,313(5786):504-507.
[59] ZHANG Ding-wen,HAN Jun-wei,LI Chao,et al.Co-saliency detection via looking deep and wide[A].Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition[C].New York:IEEE,2015.2994-3002.
[60] ZHANG Ding-wen,HAN Jun-wei,HAN Jun-gong,et al.Cosaliency detection based on intrasaliency prior transfer and deep intersaliency mining[J].IEEE Transactions on Neural Networks & Learning Systems,2016,27(6):1163-1176.
[61] JEONG D J,HWANG I,CHO N I.Co-salient object detection based on deep saliency networks and seed propagation over an integrated graph[A].LNCS:Proceedings of 2017 Computer Vision and Pattern Recognition[C].New York:IEEE,2017.arXiv:1706.09650.
[62] SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[A].LNCS:Proceedings of 2014 Computer Vision and Pattern Recognition[C].New York:IEEE,2014.arXiv:1409.1556.
[63] HAN Jun-wei,CHENG Gong,LI Zhen-peng,et al.A unified metric learning-based framework for co-saliency detection[J].IEEE Transactions on Circuits & Systems for Video Technology,2017,PP(99):1-1.
[64] LECUN Y,BENGIO Y,HINTON G.Deep learning[J].Nature,2015,521(7553):436-444.
[65] BRAHMA P P,WU Da-peng,SHE Yi-yuan.Why deep learning works:a manifold disentanglement perspective[J].IEEE Transactions on Neural Networks & Learning Systems,2016,27(10):1997-2008.
[66] OLSHUAUSEN B A,FIELD D J.Sparse coding with an overcomplete basis set:a strategy employed by V1?[J].Vision Research,1997,37(23):3311-3325.
[67] BATRA D,KOWDEL A,PARIKH D,et al.iCoseg:interactive co-segmentation with intelligent scribble guidance[A].Proceedings of 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition[C].New York:IEEE,2010.3169-3176.
[68] WINN J M,CRIMINISI A,MINKA T.Object categorization by learned universal visual dictionary[A].Proceedings of Tenth IEEE International Conference on Computer Vision[C].New York:IEEE,2005.1800-1807.
[69] RUSSAKOVSKY O,DENG Jia,SU Hao,et al.Image net large scale visual recognition challenge[J].International Journal of Computer Vision,2015,115(3):211-252.
[70] PREST A,LEISTNER C,CIVERA J,et al.Learning object class detectors from weakly annotated video[A].Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition[C].Washington:IEEE,2012.3282-3289.
[71] MARON O,EREZ L P.A framework for multiple-instance learning[J].Advances in Neural Information Processing Systems,1998,200(2):570-576.
[72] FU Hua-zhu,XU Dong,LIN Stephen,et al.Object-based RGBD image co-segmentation with mutex constraint[A].Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition[C].New York:IEEE,2015.4428-4436.
[73] LI Yi-jun,FU Ke-ren,LIU Zhi,et al.Efficient saliency-model-guided visual co-saliency detection[J].IEEE Signal Processing Letters,2015,22(5):588-592.
[74] LOU Jing,XU Feng-lei,XIA Qing-yuan,et al.Hierarchical co-salient object detection via color names[A].Proceedings of the Asian Conference on Pattern Recognition[C].New York:IEEE,2017.718-724.
[75] CHENG Ming-ming,MITRA N J,HUANG Xiao-lei,et al.Global contrast based salient region detection[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2015,37(3):569-582.
[76] HUANG Fang,QI Jin-qing,LU Hu-chuan,et al.Salient object detection via multiple instance learning[J].IEEE Transactions on Image Processing,2017,26(4):1911-1922.
[77] QIN Yao,FENG Meng-yang,LU Hu-chuan,et al.Hierarchical cellular automata for visual saliency[J].International Journal of Computer Vision,2018,126(7):751-770.
[78] ZENG Yu,LU Hu-chuan,ZHANG Li-he,et al.Learning to promote saliency detectors[A].Proceedings of 2018 IEEE Conference on Computer Vision and Pattern Recognition[C].New York:IEEE,2018.1644-1653.
[79] WANG Wen-guan,SHEN Jian-bing,SUN Han-qiu,et al.ViCoS2:video co-saliency guided co-segmentation[J].IEEE Transactions on Circuits & Systems for Video Technology,2018,28(8):1727-1736.

协同视觉显著性检测方法综述

A Review of Co-saliency Detection

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 3

编辑推荐

Metrics

[1]	付利华, 赵宇, 姜涵煦, 赵茹, 吴会贤, 闫绍兴. 基于前景感知视觉注意的半监督视频目标分割[J]. 电子学报, 2022, 50(1): 195-206.
[2]	董力赓;邸慧军;陶霖密;徐光. 一种基于动态贝叶斯网的视觉注意力识别方法[J]. 电子学报, 2011, 39(3A): 140-146.
[3]	郎丛妍, 须德, 李兵. 一种基于模糊信息粒化的视频时空显著单元提取方法[J]. 电子学报, 2007, 35(10): 2023-2028.