深度信号引导学习混合变换器的高性能无监督视频目标分割

doi:10.12263/DZXB.20221162

电子学报

• •

深度信号引导学习混合变换器的高性能无监督视频目标分割

苏天康¹^,², 宋慧慧¹^,²(), 樊佳庆³, 张开华¹^,²

^1.南京信息工程大学江苏省大数据分析技术重点实验室, 江苏南京 210044
^2.南京信息工程大学大气环境与装备技术协同创新中心, 江苏南京 210044
^3.南京航天航空大学计算机与科学技术学院, 江苏南京 211106

收稿日期:2022-10-11 修回日期:2023-01-18 出版日期:2023-03-06
- 通讯作者:
- 宋慧慧
- 作者简介:
- 苏天康男，1999年生，安徽芜湖人，硕士，研究生，主要研究方向为无监督视频目标分割.E-mail: tiankangsu@gmail.com
  宋慧慧（通讯作者）女， 1986年生，山东省聊城市人，博士，教授，主要研究领域为视频目标分割和图像超分.
  樊佳庆男，1994年生，江苏南通人，博士在读，主要研究领域为视频目标分割和视觉跟踪.E-mail: jqfan@nuaa.edu.cn
  张开华男，1983年生，山东省日照市人，博士，教授，主要研究领域为协同显著性检测和视觉跟踪. E-mail: zhkhua@gmail
- 基金资助:
- 科技创新2030-“新一代人工智能”重大项目(2018AAA0100400);国家自然科学基金(62276141)

Learning Depth Signal Guided Mixed Transformer for High-Performance Unsupervised Video Object Segmentation

SU Tian-kang¹^,², SONG Hui-hui¹^,²(), FAN Jia-qing³, ZHANG Kai-hua¹^,²

^1.Jiangsu Key Laboratory of Big Data Analysis Technology，Nanjing University of Information Science and Technology，Nanjing，Jiangsu 210044，China
^2.Collaborative Innovation Center on Atmospheric Environment and Equipment Technology，University of Information Science and Technology，Nanjing，Jiangsu 210044，China
^3.College of Computer Science and Technology，Nanjing University of Aeronautics and Astronautics，Nanjing，Jiangsu 211106，China

Received:2022-10-11 Revised:2023-01-18 Online:2023-03-06
- Corresponding author:
- SONG Hui-hui
- Supported by:
- National Key Research and Development Program of China(2018AAA0100400);National Natural Science Foundation of China(62276141)

摘要/Abstract

摘要：

现存的无监督视频目标分割方法通常使用光流作为运动线索来提升模型性能.然而，光流的估计常存在误差，这将导致双流网络易对噪声过拟合.为此，本文提出一种基于混合变换器的无监督视频目标分割算法，通过引入深度信号引导变换器高效融合不同模态数据，以学习更加鲁棒的特征表达，从而减轻模型对噪声的过拟合.首先，设计一个新颖的混合注意力模块来获得全局感受野并对不同模态的特征进行充分交互，以增强特征的全局语义信息来提升模型的抗干扰能力.接着，为了进一步感知精细化的目标边缘，设计了一个局部-非局部语义增强模块，将局部语义的归纳偏置引入补充学习非局部语义特征，在提升模型抗干扰力的同时突出更精细化的目标区域.最后，增强后的特征输入变换器的解码器，预测得到高质量的分割结果.与最先进的方法相比，本文所提算法在四个标准数据集上都获得了领先的性能，充分表明了本文所提方法的有效性.

关键词: 无监督视频目标分割, 混合变换器, 混合注意力, 多模态, 深度估计, 鲁棒特征

Abstract:

The existing unsupervised video object segmentation methods usually employ optical flow as a motion cue to improve the model performance. However, the estimation of optical flow frequently involves errors, resulting in lots of noise, especially for objects with static or complicated motion interference. The two-stream networks will easily overfit to the noise, which severely degrades the segmentation model. To relieve this, we propose to a novel mixed transformer in unsupervised video object segmentation, which can efficiently fuse different modality data by introducing depth signals to learn more robust feature representation and reduce the model overfitting to noise. In specific, the video frame, optical flow and depth map that are cropped into a set of fixed-size patches and concatenated together, are first composed of a triplet as the transformer input. The linear layer followed by a position-encoding layer is applied on the triplet, producing the features to be encoded. After this, the features are integrated by a novel mixed attention module, which can obtain the global respective field and sufficiently interact with the various modality features, to enhance the global semantic features and improve the anti-interference ability of the model. The local-non-local semantic enhancement module is developed in order to further perceive the refined target edge by introducing the inductive bias of local semantic information into supplementary learning of non-local semantic features. In this way, the target region is more refined while improving the anti-interference capability of the model. In the end, the enhanced features as the transformer decoder input to produce the predicted segmentation mask. Extensive experiments on four standard challenging benchmarks demonstrate that the proposed method achieves favorable performance against state-of-the-art methods.

Key words: unsupervised video object segmentation mixed transformer, mixed attention, multimodality, depth estimation, robust features

中图分类号:

TP391.41

苏天康, 宋慧慧, 樊佳庆, 等. 深度信号引导学习混合变换器的高性能无监督视频目标分割[J]. 电子学报, DOI: 10.12263/DZXB.20221162.

Tian-kang SU, Hui-hui SONG, Jia-qing FAN, et al. Learning Depth Signal Guided Mixed Transformer for High-Performance Unsupervised Video Object Segmentation[J]. Acta Electronica Sinica, DOI: 10.12263/DZXB.20221162.

图/表 8

参考文献 22

1	谢青松, 刘晓庆, 安志勇, 等. 基于前景优化的视觉目标跟踪算法[J]. 电子学报, 2022, 50(7): 1558-1566.
	XIE Q S, LIU X Q, AN Z Y, et al. Visual object tracking algorithm based on foreground optimization[J]. Acta Electronica Sinica, 2022, 50(7): 1558-1566. (in Chinese).
2	付利华, 赵宇, 姜涵煦, 等. 基于前景感知视觉注意的半监督视频目标分割[J]. 电子学报, 2022, 50(1): 195-206.
	FU L H, ZHAO Y, JIANG H X, et al. Semi-supervised video object segmentation based on foreground perception visual attention[J]. Acta Electronica Sinica, 2022, 50(1): 195-206. (in Chinese).
3	付利华, 赵宇, 孙晓威, 等. 基于孪生网络的快速视频目标分割[J]. 电子学报, 2020, 48(4): 625-630.
	FU L H, ZHAO Y, SUN X W, et al. Fast video object segmentation based on Siamese networks[J]. Acta Electronica Sinica, 2020, 48(4): 625-630. (in Chinese).
4	FAN J, ZHANG K, ZHAO Y, et al. Unsupervised video object segmentation via weak user interaction and temporal modulation[J]. Chinese Journal of Electronics, 2022, 32: 1-13.
5	ZHOU T F, LI J W, WANG S Z, et al. Matnet: Motion-attentive transition network for zero-shot video object segmentation[J]. IEEE Transactions on Image Processing, 2020, 29: 8326-8338.
6	JI G P, FU K R, WU Z, et al. Full-duplex strategy for video object segmentation[C]//2021 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway: IEEE, 2022: 4922-4933.
7	ZHANG K H, ZHAO Z C, LIU D, et al. Deep transport network for unsupervised video object segmentation[C]//2021 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway: IEEE, 2021: 8781-8790.
8	REN S, LIU W, LIU Y, et al. Reciprocal transformations for unsupervised video object segmentation[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2021: 15455-15464.
9	TOKMAKOV P, ALAHARI K, SCHMID C. Learning video object segmentation with visual memory[C]//2017 IEEE International Conference on Computer Vision (ICCV). Piscataway: IEEE, 2017: 4481-4490.
10	LU X K, WANG W G, DANELLJAN M, et al. Video object segmentation with episodic graph memory Networks[C]//Computer Vision - ECCV 2020. Cham: Springer International Publishing, 2020: 661-679.
11	Mahadevan S, Athar A, Ošep A, et al. Making a case for 3d convolutions for object segmentation in videos[EB/OL]. arXiv preprint arXiv:, 2020.
12	SCHMIDT C, ATHAR A, MAHADEVAN S, et al. D2conv3d: Dynamic dilated convolutions for object segmentation in videos[C]//2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). Piscataway: IEEE, 2022: 1200-1209.
13	RANFTL R, LASINGER K, HAFNER D, et al. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 44(3): 1623-1637.
14	TEED Z, DENG J. RAFT: Recurrent all-pairs field transforms for optical flow[C]//Computer Vision - ECCV 2020. Cham: Springer International Publishing, 2020: 402-419.
15	PERAZZI F, PONT-TUSET J, MCWILLIAMS B, et al. A benchmark dataset and evaluation methodology for video object segmentation[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2016: 724-732.
16	OCHS P, MALIK J, BROX T. Segmentation of moving objects by long term video analysis[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 36(6): 1187-1200.
17	XU N, YANG L J, FAN Y C, et al. Youtube-vos: Sequence-to-sequence video object segmentation [C]//Computer Vision - ECCV 2018. Cham: Springer International Publishing, 2018: 585-601.
18	FAN D P, WANG W G, CHENG M M, et al. Shifting more attention to video salient object detection[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2019: 8554-8564.
19	WANG W G, SHEN J B, SHAO L. Consistent video saliency using local gradient flow optimization and global refinement[J]. IEEE Transactions on Image Processing, 2015, 24(11): 4185-4196.
20	CHEN Y W, JIN X J, SHEN X H, et al. Video salient object detection via contrastive features and attention modules[C]//2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). Piscataway: IEEE, 2022: 1320-1329.
21	ZHEN M M, LI S W, ZHOU L, et al. Learning discriminative feature with crf for unsupervised video object segmentation[C]//Computer Vision - ECCV 2020. Cham: Springer International Publishing, 2020: 445-462.
22	JI Y Z, ZHANG H J, JIE Z Q, et al. CASNet: A cross-attention Siamese network for video salient object detection[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020, 32(6): 2676-2690.

算法	DAVIS16			FBMS
算法	J&F	J	F	J
MATNet^[5]	81.6	82.4	80.7	76.1
3DC-Seg^[11]	84.5	84.3	84.7	-
FSNet^[6]	83.3	83.4	83.1	-
TransportNet^[7]	84.8	84.5	85.0	78.7
RTNet^[8]	85.2	85.6	84.7	-
CFANet^[20]	82.8	83.5	82.0	-
D2Conv3D^[12]	86.0	85.5	86.5	-
本文方法	86.4	86.1	86.7	84.0

算法	DAVIS16			FBMS
算法	J&F	J	F	J
MATNet^[5]	81.6	82.4	80.7	76.1
3DC-Seg^[11]	84.5	84.3	84.7	-
FSNet^[6]	83.3	83.4	83.1	-
TransportNet^[7]	84.8	84.5	85.0	78.7
RTNet^[8]	85.2	85.6	84.7	-
CFANet^[20]	82.8	83.5	82.0	-
D2Conv3D^[12]	86.0	85.5	86.5	-
本文方法	86.4	86.1	86.7	84.0

算法	DAVIS16			FBMS			DAVSOD			ViSal
算法	S_a↑	F_m↑	MAE↓	S_a↑	F_m↑	MAE↓	S_a↑	F_m↑	MAE↓	S_a↑	F_m↑	MAE↓
SSAV^[18]	0.893	0.861	0.028	0.879	0.865	0.040	0.724	0.603	0.092	0.943	0.939	0.020
DFNet^[21]	-	0.899	0.018	-	0.833	0.054	-	-	-	-	0.927	0.017
3DC-Seg^[11]	-	0.918	0.015	-	0.845	0.048	-	-	-	-	0.922	0.019
CASNet^[22]	0.873	0.860	0.032	0.856	0.863	0.056	0.694	-	0.089	0.820	0.847	0.029
TransportNet^[7]	-	0.928	0.013	-	0.885	0.045	-	-	-	-	0.953	0.012
FSNet^[6]	0.920	0.907	0.020	0.890	0.888	0.041	0.773	0.685	0.072	-	-	-
CFANet^[20]	0.918	0.909	0.015	0.909	0.915	0.026	0.753	0.662	0.083	-	-	-
本文方法	0.937	0.929	0.010	0.916	0.922	0.026	0.790	0.698	0.069	0.953	0.954	0.013

算法	DAVIS16			FBMS			DAVSOD			ViSal
算法	S_a↑	F_m↑	MAE↓	S_a↑	F_m↑	MAE↓	S_a↑	F_m↑	MAE↓	S_a↑	F_m↑	MAE↓
SSAV^[18]	0.893	0.861	0.028	0.879	0.865	0.040	0.724	0.603	0.092	0.943	0.939	0.020
DFNet^[21]	-	0.899	0.018	-	0.833	0.054	-	-	-	-	0.927	0.017
3DC-Seg^[11]	-	0.918	0.015	-	0.845	0.048	-	-	-	-	0.922	0.019
CASNet^[22]	0.873	0.860	0.032	0.856	0.863	0.056	0.694	-	0.089	0.820	0.847	0.029
TransportNet^[7]	-	0.928	0.013	-	0.885	0.045	-	-	-	-	0.953	0.012
FSNet^[6]	0.920	0.907	0.020	0.890	0.888	0.041	0.773	0.685	0.072	-	-	-
CFANet^[20]	0.918	0.909	0.015	0.909	0.915	0.026	0.753	0.662	0.083	-	-	-
本文方法	0.937	0.929	0.010	0.916	0.922	0.026	0.790	0.698	0.069	0.953	0.954	0.013

模型变化			DAVIS16		FBMS
深度图	混合注意力模块	局部-非局部语义增强模块	J	F	J
			80.7	80.6	78.2
√			82.6	82.2	80.8
	√		83.2	83.8	81.4
		√	81.3	81.5	79.8
√	√		84.6	84.8	82.6
√		√	83.6	84.0	81.2
	√	√	84.2	84.6	82.0
√	√	√	86.1	86.7	84.0

深度信号引导学习混合变换器的高性能无监督视频目标分割

Learning Depth Signal Guided Mixed Transformer for High-Performance Unsupervised Video Object Segmentation

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 8

参考文献 22

相关文章 15

编辑推荐

Metrics

[1]	刘玉轩, 张力, 艾海滨, 许彪, 孙钰珊, 樊仲藜. 光场相机三维重建研究进展与展望[J]. 电子学报, 2022, 50(7): 1774-1792.
[2]	徐建博, 魏昕, 周亮. 面向跨模态通信的信息恢复技术[J]. 电子学报, 2022, 50(7): 1631-1642.
[3]	庾骏, 黄伟, 张晓波, 尹贺峰. 基于松弛Hadamard矩阵的多模态融合哈希方法[J]. 电子学报, 2022, 50(4): 909-920.
[4]	周奕涛, 张斌, 刘自豪. 基于多模态深度神经网络的应用层DDoS攻击检测模型[J]. 电子学报, 2022, 50(2): 508-512.
[5]	乔思波, 庞善臣, 王敏, 翟雪, 于世行, 丁桐. 基于残差混合注意力机制的脑部CT图像分类卷积神经网络模型[J]. 电子学报, 2021, 49(5): 984-991.
[6]	石鹏, 张启忠, 张华平, 席旭刚. 基于多元经验模态分解的多元多尺度熵静态平衡能力评估[J]. 电子学报, 2020, 48(4): 670-674.
[7]	赖小波, 许茂盛, 徐小媚. 多分类CNN的胶质母细胞瘤多模态MR图像分割[J]. 电子学报, 2019, 47(8): 1738-1747.
[8]	杨晓东, 陈益强, 于汉超, 张迎伟, 钟习, 胡子昂, 刘弘. 面向帕金森病的多模态异构协同感知方法[J]. 电子学报, 2018, 46(3): 659-664.
[9]	黄殿中, 张静飞, 张茹, 李鹏超, 郭云彪. 基于大数据环境的多模态信息隐藏新体系[J]. 电子学报, 2017, 45(2): 477-484.
[10]	袁红星, 吴少群, 朱仁祥, 安鹏. 加权SIFT流深度迁移的单幅图像2D转3D[J]. 电子学报, 2015, 43(2): 242-247.
[11]	李翔, 李迅波, 陈亮. 超声导波信号形态分量分析方法研究[J]. 电子学报, 2013, 41(3): 444-450.
[12]	管业鹏. 基于多模态视觉特征的自然人机交互[J]. 电子学报, 2013, 41(11): 2223-2229.
[13]	隆志力, 吴运新, 韩雷, 钟掘. 热超声倒装键合换能系统多模态振动与有限元分析[J]. 电子学报, 2008, 36(2): 255-260.
[14]	葛云;舒华忠;罗立民;唐培. 基于距离变换和非线性规划理论的三维图像表皮配准的研究[J]. 电子学报, 2004, 32(4): 684-686.
[15]	葛云;舒华忠;罗立民. 基于Legendre正交矩的配准方法及其在二值图像配准中的应用[J]. 电子学报, 2001, 29(1): 54-56.