基于帧间高级特征差分的跨场景视频前景分割算法

doi:10.12263/DZXB.20200620

电子学报 ›› 2021, Vol. 49 ›› Issue (10): 2032-2040.DOI: 10.12263/DZXB.20200620

基于帧间高级特征差分的跨场景视频前景分割算法

张锦¹^,², 李阳², 任传伦³, 黄炼⁴, 王帅辉², 段晔鑫¹^,², 潘志松², 谢钧²

^1.陆军军事交通学院镇江校区，江苏镇江 212003
^2.陆军工程大学指挥控制工程学院，江苏南京 210007
^3.华北计算技术研究所，北京 100083
^4.海军装备部驻上海地区军事代表局，上海 200129

收稿日期:2020-06-28 修回日期:2020-12-03 出版日期:2021-10-25 发布日期:2021-10-25
作者简介:张锦男，1985年生，湖南邵东人.2009年获海军工程大学核科学与技术专业硕士学位，其后在陆军军事交通学院镇江校区工作.目前在陆军工程大学攻读博士学位.研究方向为人工智能、图像处理. E-mail:zhang_jin_1115@163.com
李阳男，1984年生，河北廊坊人.2007年、2010年、2018年分别在北京航空航天大学、解放军理工大学、陆军工程大学获学士、硕士和博士学位.现为陆军工程大学讲师，主要研究方向为人工智能、机器视觉与图像检索. E-mail:solarleeon@outlook.com
潘志松（通信作者）男，1973年生，江苏南京人.2003年获南京航空航天大学博士学位.现为陆军工程大学教授、博士生导师.主要研究方向为人工智能、模式识别. E-mail:panzs@nuaa.edu.cn
谢钧男，1973年生，四川成都人.2005年获南京大学博士学位.现为陆军工程大学教授、博士生导师.主要研究方向为智能信息处理、无线网络. E-mail:xiejun73@189.cn
基金资助:
国家自然科学基金(61806220)

Cross-Scene Foreground Segmentation Algorithm Based on High-Level Feature Differencing Between Frames

Jin ZHANG¹^,², Yang LI², Chuan-lun REN³, Lian HUANG⁴, Shuai-hui WANG², Ye-xin DUAN¹^,², Zhi-song PAN², Jun XIE²

^1.Zhenjiang Campus，Army Military Transportation University of PLA，Zhenjiang，Jiangsu 212003，China
^2.Command and Control Engineering College，Army Engineering University of PLA，Nanjing，Jiangsu 210007，China
^3.North China Institute of Computer Technology，Beijing 100083，China
^4.Shanghai Military Representative Bureau，Navy Equipment Department of PLA，Shanghai 200129，China

Received:2020-06-28 Revised:2020-12-03 Online:2021-10-25 Published:2021-10-25

摘要/Abstract

摘要：

当前基于深度学习的有监督前景分割方法得益于大量待分割场景的标注信息，其性能大幅超越传统的无监督方法.然而，获取高精度的像素级标注需要耗费大量的人力和时间成本，这严重限制了有监督算法在无标注场景的部署应用.为解决对场景监督信息依赖的问题，设计了一种与传统的帧间差分法相融合的跨场景深度学习架构，即帧间高级特征差分算法.该算法重点围绕时域变化等跨场景共性知识的迁移，在不依赖待分割场景监督信息的前提下实现高精度分割.面向五类不同模式的困难场景开展实验，本文算法的平均F值达到0.8719，超越了当前最高性能的有监督算法FgSegNet_v2（相同的跨场景条件下）和最佳的无监督算法SemanticBS.本文算法对QVGA视频（320×240）的处理速度达到35帧/s，具有较好的实时性.

关键词: 前景分割, 迁移学习, 帧间差分法, 跨场景学习, 深度学习

Abstract:

Benefiting from large amounts of ground-truths of to-be-segmented scenarios, deep-learning based and supervised foreground segmentation algorithms generally outperform conventional unsupervised methods. However, pixel-wise annotation is a tedious task, especially when it comes to the annotation of foreground moving objects. This seriously limits the deployment of a supervised algorithm in a wide range of scenes without ground-truths. To address the dependence on supervised information of to-be-segmented unseen scenes, we design an inter-frame high-level feature differencing algorithm with a deep learning architecture via integrating the traditional frame differencing method. The proposed algorithm leverages the transfer of cross-scene common knowledge, such as temporal changes, so as to achieve high performance for the scene in the absence of supervised information of to-be-segmented scenes. We evaluate our method on five challenging scenes with different patterns. The average F-Measure of our algorithm is 0.8719, which surpasses the current highest-performance (supervised) algorithm (FgSegNet_v2) under the cross-scene learning condition and the best unsupervised algorithm SemanticBS. Our method which can process a QVGA (320 × 240) video at 35 frames per second shows favorable real-time performance.

Key words: foreground segmentation, transfer learning, frame differencing algorithm, cross-scene learning, deep learni

中图分类号:

TP391.4

张锦, 李阳, 任传伦, 黄炼, 王帅辉, 段晔鑫, 潘志松, 谢钧. 基于帧间高级特征差分的跨场景视频前景分割算法[J]. 电子学报, 2021, 49(10): 2032-2040.

Jin ZHANG, Yang LI, Chuan-lun REN, Lian HUANG, Shuai-hui WANG, Ye-xin DUAN, Zhi-song PAN, Jun XIE. Cross-Scene Foreground Segmentation Algorithm Based on High-Level Feature Differencing Between Frames[J]. Acta Electronica Sinica, 2021, 49(10): 2032-2040.

图/表 9

参考文献 30

1	LeCun Y, Bottou L, Bengio Y, et al. Gradient-based learning applied to document recognition[A]. Proceedings of the IEEE[C]. USA: IEEE, 1998. 2278－2324.
2	Fu J, Liu J, Tian H J, et al. Dual attention network for scene segmentation[A]. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)[C]. Long Beach, CA, USA: IEEE, 2019. 3141－3149.
3	Sakkos D, Liu H, Han J G, et al. End-to-end video background subtraction with 3d convolutional neural networks[J]. Multimedia Tools and Applications, 2018, 77(17): 23023－23041.
4	Wang Y, Luo Z M, Jodoin P M. Interactive deep learning method for segmenting moving objects[J]. Pattern Recognition Letters, 2017, 96: 66－75.
5	Lim L A, Yalim Keles H. Foreground segmentation using convolutional neural networks for multiscale feature encoding[J]. Pattern Recognition Letters, 2018, 112: 256－262.
6	Lim L A, Keles H Y. Learning multi-scale features for foreground segmentation[J]. Pattern Analysis and Applications, 2020, 23(3): 1369－1380.
7	Babaee M, Dinh D T, Rigoll G. A deep convolutional neural network for video sequence background subtraction[J]. Pattern Recognition, 2018, 76: 635－649.
8	Wang Y, Jodoin P M, Porikli F, et al. CDnet 2014: An expanded change detection benchmark dataset[A]. 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops[C]. Columbus, OH, USA: IEEE, 2014. 393－400.
9	Maddalena L, Petrosino A. Towards benchmarking scene background initialization[A]. New Trends in Image Analysis and Processing－ICIAP 2015 Workshops[C]. Cham, GER: Springer, 2015. 469－476. DOI:10.1007/978－3－319－23222－5_57.
10	Mandal M, Dhar V, Mishra A, et al. 3DFR: A swift 3D feature reductionist framework for scene independent change detection[J]. IEEE Signal Processing Letters, 2019, 26(12): 1882－1886.
11	Bouwmans T, Javed S, Sultana M, et al. Deep neural network concepts for background subtraction: A systematic review and comparative evaluation[J]. Neural Networks, 2019, 117: 8－66.
12	Braham M, Van Droogenbroeck M. Deep background subtraction with scene-specific convolutional neural networks[A]. 2016 International Conference on Systems, Signals and Image Processing (IWSSIP)[C]. Bratislava, Slovakia: IEEE, 2016. 1－4.
13	Chen M L, Wei X, Yang Q X, et al. Spatiotemporal GMM for background subtraction with superpixel hierarchy[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(6): 1518－1525.
14	Shi G M, Huang T, Dong W S, et al. Robust foreground estimation via structured Gaussian scale mixture modeling[J]. IEEE Transactions on Image Processing, 2018, 27(10): 4810－4824.
15	常侃, 张智勇, 陈诚, 等. 采用低秩与加权稀疏分解的视频前景检测算法[J]. 电子学报, 2017, 45(9): 2272－2280.
	Chang K, Zhang Z Y, Chen C, et al. Video foreground detection by low-rank and reweighted sparse decomposition[J]. Acta Electronica Sinica, 2017, 45(9): 2272－2280. (in Chinese)
16	秦晓燕, 袁广林, 李从利, 等. 一种快速鲁棒的视频序列运动目标检测方法[J]. 电子学报, 2017, 45(10): 2355－2361.
	Qin X Y, Yuan G L, Li C L, et al. An approach to fast and robust detecting of moving target in video sequences[J]. Acta Electronica Sinica, 2017, 45(10): 2355－2361. (in Chinese)
17	Paul N, Singh A, Midya A, et al. Moving object detection using modified temporal differencing and local fuzzy thresholding[J]. The Journal of Supercomputing, 2017, 73(3): 1120－1139.
18	Bilodeau G A, Jodoin J P, Saunier N. Change detection in feature space using local binary similarity patterns[A]. 2013 International Conference on Computer and Robot Vision[C]. Regina, SK, Canada: IEEE, 2013. 106－112.
19	St-Charles P L, Bilodeau G A, Bergevin R. A self-adjusting approach to change detection based on background word consensus[A]. 2015 IEEE Winter Conference on Applications of Computer Vision[C]. Waikoloa, HI, USA: IEEE, 2015. 990－997.
20	St-Charles P L, Bilodeau G A, Bergevin R. SuBSENSE: A universal change detection method with local adaptive sensitivity[J]. IEEE Transactions on Image Processing, 2015, 24(1): 359－373.
21	Braham M, Piérard S, Van Droogenbroeck M. Semantic background subtraction[A]. 2017 IEEE International Conference on Image Processing (ICIP)[C]. Beijing, China: IEEE, 2017. 4552－4556.
22	Sultana M, Mahmood A, Javed S, et al. Unsupervised deep context prediction for background estimation and foreground segmentation[J]. Machine Vision and Applications, 2019, 30(3): 375－395.
23	Zeng D D, Zhu M, Kuijper A. Combining background subtraction algorithms with convolutional neural network[J]. Journal of Electronic Imaging, 2019, 28(1): 013011.
24	Zhang J, Li Y, Chen F Q, et al. X-net: A binocular summation network for foreground segmentation[J]. IEEE Access, 2019, 7: 71412－71422.
25	Chen L C, Zhu Y K, Papandreou G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation[A]. European Conference on Computer Vision[C]. Cham, GER: Springer, 2018. 833－851. DOI:10.1007/978－3－030－01234－2_49.
26	Yang M K, Yu K, Zhang C, et al. DenseASPP for semantic segmentation in street scenes[A]. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition[C]. Salt Lake City, UT, USA: IEEE, 2018. 3684－3692.
27	Lin T Y, Goyal P, Girshick R, et al. Focal loss for dense object detection[A]. 2017 IEEE International Conference on Computer Vision (ICCV)[C]. Venice, Italy: IEEE, 2017. 2999－3007.
28	Bianco S, Ciocca G, Schettini R. Combination of video change detection algorithms by genetic programming[J]. IEEE Transactions on Evolutionary Computation, 2017, 21(6): 914－928.
29	Chen M L, Yang Q X, Li Q, et al. Spatiotemporal background subtraction using minimum spanning tree and optical flow[A]. European Conference on Computer Vision[C]. Cham, GER: Springer, 2014. 521－534. DOI:10.1007/978－3－319－10584－0_34.
30	Sajid H, Cheung S C S. Universal multimode background subtraction[J]. IEEE Transactions on Image Processing, 2017, 26(7): 3249－3260.

实验视频	大小×帧数	描述
PETS.	［576×720］×1200	室内，弱阴影，小尺度前景
boulev.	［240×352］×2500	室外，相机抖动
boats	［240×320］×7999	室外，动态背景
bungal.	［240×360］×1700	室外，硬阴影，大尺度前景
park	［288×352］×600	室外，红外视频，前/背景灰度相似度大

实验视频	大小×帧数	描述
PETS.	［576×720］×1200	室内，弱阴影，小尺度前景
boulev.	［240×352］×2500	室外，相机抖动
boats	［240×320］×7999	室外，动态背景
bungal.	［240×360］×1700	室外，硬阴影，大尺度前景
park	［288×352］×600	室外，红外视频，前/背景灰度相似度大

Methods	Metrics	Overall	PETS.	boulev.	boats	bungal.	park
CS-DFS₃	R	0.8053	0.9221	0.8303	0.6901	0.7523	0.8318
	Sp	0.9985	0.9995	0.9992	0.9998	0.9982	0.9958
	P	0.9346	0.9593	0.9809	0.9628	0.9641	0.8060
	FPN	0.0015	0.0005	0.0008	0.0002	0.0018	0.0042
	FNR	0.1947	0.0779	0.1697	0.3099	0.2477	0.1682
	PWC	0.7344	0.1521	0.8725	0.2111	1.6880	0.7484
CS-DFS₃	F	0.8719	0.9334	0.8452	0.9472	0.8879	0.7457
CS-DFS₂	F	0.8039	0.9307	0.7530	0.8202	0.7844	0.7311
CS-DFS₁	F	0.8155	0.9514	0.8183	0.6660	0.8767	0.7651
CS-DFS₀	F	0.6786	0.9005	0.6175	0.4679	0.7218	0.6852

Methods	Metrics	Overall	PETS.	boulev.	boats	bungal.	park
CS-DFS₃	R	0.8053	0.9221	0.8303	0.6901	0.7523	0.8318
	Sp	0.9985	0.9995	0.9992	0.9998	0.9982	0.9958
	P	0.9346	0.9593	0.9809	0.9628	0.9641	0.8060
	FPN	0.0015	0.0005	0.0008	0.0002	0.0018	0.0042
	FNR	0.1947	0.0779	0.1697	0.3099	0.2477	0.1682
	PWC	0.7344	0.1521	0.8725	0.2111	1.6880	0.7484
CS-DFS₃	F	0.8719	0.9334	0.8452	0.9472	0.8879	0.7457
CS-DFS₂	F	0.8039	0.9307	0.7530	0.8202	0.7844	0.7311
CS-DFS₁	F	0.8155	0.9514	0.8183	0.6660	0.8767	0.7651
CS-DFS₀	F	0.6786	0.9005	0.6175	0.4679	0.7218	0.6852

场景	PETS	boulev	boats	bungal	park	Overall	Speed
IURIS-5^［28］	0.9354	0.7680	0.7532	0.8392	0.7652	0.8122	-
MBS^［30］	0.8648	0.8672	0.9041	0.7475	0.7099	0.8187	-
STBM^［29］	0.8610	0.7423	0.8760	0.7927	0.8323	0.8208	12
PAWCS^［19］	0.9315	0.8444	0.8416	0.8387	0.8286	0.8570	-
SemanticBS^［21］	0.9442	0.7372	0.9795	0.9259	0.7694	0.8712	7
FgSegNet_v2^［6］	0.8833	0.7313	0.5581	0.7896	0.5426	0.7010	38
CS-DFS₃（本文）	0.9334	0.8452	0.9472	0.8879	0.7457	0.8719	35

基于帧间高级特征差分的跨场景视频前景分割算法

Cross-Scene Foreground Segmentation Algorithm Based on High-Level Feature Differencing Between Frames

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 9

参考文献 30

相关文章 15

编辑推荐

Metrics

[1]	廖勇, 李玉杰. 一种轻量化低复杂度的FDD大规模MIMO系统CSI反馈方法[J]. 电子学报, 2022, 50(5): 1211-1217.
[2]	冀振燕, 韩梦豪, 宋晓军, 冯其波. 面向激光光条图像修复的循环相似度映射网络[J]. 电子学报, 2022, 50(5): 1234-1242.
[3]	潘敏婷, 王韫博, 朱祥明, 高思宇, 龙明盛, 杨小康. 基于无标签视频数据的深度预测学习方法综述[J]. 电子学报, 2022, 50(4): 869-886.
[4]	杨曦, 张鑫, 郭浩远, 王楠楠, 高新波. 基于不变特征的多源遥感图像舰船目标检测算法[J]. 电子学报, 2022, 50(4): 887-899.
[5]	崔亚奇, 何友, 唐田田, 熊伟. 一种深度学习航迹关联方法[J]. 电子学报, 2022, 50(3): 759-763.
[6]	张浩, 胡昌华, 杜党波, 裴洪, 张建勋. 多状态影响下基于Bi‑LSTM网络的锂电池剩余寿命预测方法[J]. 电子学报, 2022, 50(3): 619-624.
[7]	李维刚, 甘平, 谢璐, 李松涛. 基于样本对元学习的小样本图像分类方法[J]. 电子学报, 2022, 50(2): 295-304.
[8]	骈纬国, 吴映波, 陈蒙, 蔡俊鹏. 一种基于时空动态图注意力网络的共享出行需求预测方法[J]. 电子学报, 2022, 50(2): 432-439.
[9]	张小恒, 张馨月, 李勇明, 王品, 刘玉川. 面向帕金森病语音诊断的非监督两步式卷积稀疏迁移学习算法[J]. 电子学报, 2022, 50(1): 177-184.
[10]	李居朋, 王颖慧, 李刚. 医学图像关键点检测深度学习方法研究与挑战[J]. 电子学报, 2022, 50(1): 226-237.
[11]	李勇斌, 王彬, 邵高平, 邵帅. 一种基于RCGAN的水声通信信号降噪方法[J]. 电子学报, 2022, 50(1): 54-62.
[12]	张聿远, 张立民, 闫文君. 基于深度多级残差网络的低信噪比下空频分组码识别方法[J]. 电子学报, 2022, 50(1): 79-88.
[13]	席亮, 刘涵, 樊好义, 张凤斌. 基于深度对抗学习潜在表示分布的异常检测模型[J]. 电子学报, 2021, 49(7): 1257-1265.
[14]	谢佳龙, 张波涛, 吕强. 一种基于双流融合3D卷积神经网络的动态头势识别方法[J]. 电子学报, 2021, 49(7): 1363-1369.
[15]	罗会兰, 袁璞, 童康. 基于深度学习的显著性目标检测方法综述[J]. 电子学报, 2021, 49(7): 1417-1427.