基于帧间高级特征差分的跨场景视频前景分割算法

doi:10.12263/DZXB.20200620

电子学报 ›› 2021, Vol. 49 ›› Issue (10): 2032-2040.DOI: 10.12263/DZXB.20200620

基于帧间高级特征差分的跨场景视频前景分割算法

张锦¹^,², 李阳², 任传伦³, 黄炼⁴, 王帅辉², 段晔鑫¹^,², 潘志松², 谢钧²

^1.陆军军事交通学院镇江校区，江苏镇江 212003
^2.陆军工程大学指挥控制工程学院，江苏南京 210007
^3.华北计算技术研究所，北京 100083
^4.海军装备部驻上海地区军事代表局，上海 200129

收稿日期:2020-06-28 修回日期:2020-12-03 出版日期:2021-11-29
- 作者简介:
- 张锦男，1985年生，湖南邵东人.2009年获海军工程大学核科学与技术专业硕士学位，其后在陆军军事交通学院镇江校区工作.目前在陆军工程大学攻读博士学位.研究方向为人工智能、图像处理. E-mail:zhang_jin_1115@163.com
  李阳男，1984年生，河北廊坊人.2007年、2010年、2018年分别在北京航空航天大学、解放军理工大学、陆军工程大学获学士、硕士和博士学位.现为陆军工程大学讲师，主要研究方向为人工智能、机器视觉与图像检索. E-mail:solarleeon@outlook.com
  潘志松（通信作者）男，1973年生，江苏南京人.2003年获南京航空航天大学博士学位.现为陆军工程大学教授、博士生导师.主要研究方向为人工智能、模式识别. E-mail:panzs@nuaa.edu.cn
  谢钧男，1973年生，四川成都人.2005年获南京大学博士学位.现为陆军工程大学教授、博士生导师.主要研究方向为智能信息处理、无线网络. E-mail:xiejun73@189.cn
- 基金资助:
- 国家自然科学基金 (61806220)

Cross-Scene Foreground Segmentation Algorithm Based on High-Level Feature Differencing Between Frames

ZHANG Jin¹^,², LI Yang², REN Chuan-lun³, HUANG Lian⁴, WANG Shuai-hui², DUAN Ye-xin¹^,², PAN Zhi-song², XIE Jun²

^1.Zhenjiang Campus, Army Military Transportation University of PLA, Zhenjiang, Jiangsu 212003, China
^2.Command and Control Engineering College, Army Engineering University of PLA, Nanjing, Jiangsu 210007, China
^3.North China Institute of Computer Technology, Beijing 100083, China
^4.Shanghai Military Representative Bureau, Navy Equipment Department of PLA, Shanghai 200129, China

Received:2020-06-28 Revised:2020-12-03 Online:2021-11-29 Published:2021-10-25
- Supported by:
- National Natural Science Foundation of China (61806220)

摘要/Abstract

摘要：

当前基于深度学习的有监督前景分割方法得益于大量待分割场景的标注信息，其性能大幅超越传统的无监督方法.然而，获取高精度的像素级标注需要耗费大量的人力和时间成本，这严重限制了有监督算法在无标注场景的部署应用.为解决对场景监督信息依赖的问题，设计了一种与传统的帧间差分法相融合的跨场景深度学习架构，即帧间高级特征差分算法.该算法重点围绕时域变化等跨场景共性知识的迁移，在不依赖待分割场景监督信息的前提下实现高精度分割.面向五类不同模式的困难场景开展实验，本文算法的平均F值达到0.8719，超越了当前最高性能的有监督算法FgSegNet_v2（相同的跨场景条件下）和最佳的无监督算法SemanticBS.本文算法对QVGA视频（320×240）的处理速度达到35帧/s，具有较好的实时性.

关键词: 前景分割, 迁移学习, 帧间差分法, 跨场景学习, 深度学习

Abstract:

Benefiting from large amounts of ground-truths of to-be-segmented scenarios, deep-learning based and supervised foreground segmentation algorithms generally outperform conventional unsupervised methods. However, pixel-wise annotation is a tedious task, especially when it comes to the annotation of foreground moving objects. This seriously limits the deployment of a supervised algorithm in a wide range of scenes without ground-truths. To address the dependence on supervised information of to-be-segmented unseen scenes, we design an inter-frame high-level feature differencing algorithm with a deep learning architecture via integrating the traditional frame differencing method. The proposed algorithm leverages the transfer of cross-scene common knowledge, such as temporal changes, so as to achieve high performance for the scene in the absence of supervised information of to-be-segmented scenes. We evaluate our method on five challenging scenes with different patterns. The average F-Measure of our algorithm is 0.8719, which surpasses the current highest-performance (supervised) algorithm (FgSegNet_v2) under the cross-scene learning condition and the best unsupervised algorithm SemanticBS. Our method which can process a QVGA (320 × 240) video at 35 frames per second shows favorable real-time performance.

Key words: foreground segmentation, transfer learning, frame differencing algorithm, cross-scene learning, deep learni

中图分类号:

TP391.4

张锦, 李阳, 任传伦, 黄炼, 王帅辉, 段晔鑫, 潘志松, 谢钧. 基于帧间高级特征差分的跨场景视频前景分割算法[J]. 电子学报, 2021, 49(10): 2032-2040.

ZHANG Jin, LI Yang, REN Chuan-lun, HUANG Lian, WANG Shuai-hui, DUAN Ye-xin, PAN Zhi-song, XIE Jun. Cross-Scene Foreground Segmentation Algorithm Based on High-Level Feature Differencing Between Frames[J]. Acta Electronica Sinica, 2021, 49(10): 2032-2040.

图/表 9

参考文献 30

1	LeCunY, BottouL, BengioY, et al. Gradient-based learning applied to document recognition[A]. Proceedings of the IEEE[C]. USA: IEEE, 1998. 2278－2324.
2	FuJ, LiuJ, TianH J, et al. Dual attention network for scene segmentation[A]. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)[C]. Long Beach, CA, USA: IEEE, 2019. 3141－3149.
3	SakkosD, LiuH, HanJ G, et al. End-to-end video background subtraction with 3d convolutional neural networks[J]. Multimedia Tools and Applications, 2018, 77(17): 23023－23041.
4	WangY, LuoZ M, JodoinP M. Interactive deep learning method for segmenting moving objects[J]. Pattern Recognition Letters, 2017, 96: 66－75.
5	LimL A, Yalim KelesH. Foreground segmentation using convolutional neural networks for multiscale feature encoding[J]. Pattern Recognition Letters, 2018, 112: 256－262.
6	LimL A, KelesH Y. Learning multi-scale features for foreground segmentation[J]. Pattern Analysis and Applications, 2020, 23(3): 1369－1380.
7	BabaeeM, DinhD T, RigollG. A deep convolutional neural network for video sequence background subtraction[J]. Pattern Recognition, 2018, 76: 635－649.
8	WangY, JodoinP M, PorikliF, et al. CDnet 2014: An expanded change detection benchmark dataset[A]. 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops[C]. Columbus, OH, USA: IEEE, 2014. 393－400.
9	MaddalenaL, PetrosinoA. Towards benchmarking scene background initialization[A]. New Trends in Image Analysis and Processing－ICIAP 2015 Workshops[C]. Cham, GER: Springer, 2015. 469－476. DOI:10.1007/978－3－319－23222－5_57.
10	MandalM, DharV, MishraA, et al. 3DFR: A swift 3D feature reductionist framework for scene independent change detection[J]. IEEE Signal Processing Letters, 2019, 26(12): 1882－1886.
11	BouwmansT, JavedS, SultanaM, et al. Deep neural network concepts for background subtraction: A systematic review and comparative evaluation[J]. Neural Networks, 2019, 117: 8－66.
12	BrahamM, Van DroogenbroeckM. Deep background subtraction with scene-specific convolutional neural networks[A]. 2016 International Conference on Systems, Signals and Image Processing (IWSSIP)[C]. Bratislava, Slovakia: IEEE, 2016. 1－4.
13	ChenM L, WeiX, YangQ X, et al. Spatiotemporal GMM for background subtraction with superpixel hierarchy[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(6): 1518－1525.
14	ShiG M, HuangT, DongW S, et al. Robust foreground estimation via structured Gaussian scale mixture modeling[J]. IEEE Transactions on Image Processing, 2018, 27(10): 4810－4824.
15	常侃, 张智勇, 陈诚, 等. 采用低秩与加权稀疏分解的视频前景检测算法[J]. 电子学报, 2017, 45(9): 2272－2280.
	ChangK, ZhangZ Y, ChenC, et al. Video foreground detection by low-rank and reweighted sparse decomposition[J]. Acta Electronica Sinica, 2017, 45(9): 2272－2280. (in Chinese)
16	秦晓燕, 袁广林, 李从利, 等. 一种快速鲁棒的视频序列运动目标检测方法[J]. 电子学报, 2017, 45(10): 2355－2361.
	QinX Y, YuanG L, LiC L, et al. An approach to fast and robust detecting of moving target in video sequences[J]. Acta Electronica Sinica, 2017, 45(10): 2355－2361. (in Chinese)
17	PaulN, SinghA, MidyaA, et al. Moving object detection using modified temporal differencing and local fuzzy thresholding[J]. The Journal of Supercomputing, 2017, 73(3): 1120－1139.
18	BilodeauG A, JodoinJ P, SaunierN. Change detection in feature space using local binary similarity patterns[A]. 2013 International Conference on Computer and Robot Vision[C]. Regina, SK, Canada: IEEE, 2013. 106－112.
19	St-CharlesP L, BilodeauG A, BergevinR. A self-adjusting approach to change detection based on background word consensus[A]. 2015 IEEE Winter Conference on Applications of Computer Vision[C]. Waikoloa, HI, USA: IEEE, 2015. 990－997.
20	St-CharlesP L, BilodeauG A, BergevinR. SuBSENSE: A universal change detection method with local adaptive sensitivity[J]. IEEE Transactions on Image Processing, 2015, 24(1): 359－373.
21	BrahamM, PiérardS, Van DroogenbroeckM. Semantic background subtraction[A]. 2017 IEEE International Conference on Image Processing (ICIP)[C]. Beijing, China: IEEE, 2017. 4552－4556.
22	SultanaM, MahmoodA, JavedS, et al. Unsupervised deep context prediction for background estimation and foreground segmentation[J]. Machine Vision and Applications, 2019, 30(3): 375－395.
23	ZengD D, ZhuM, KuijperA. Combining background subtraction algorithms with convolutional neural network[J]. Journal of Electronic Imaging, 2019, 28(1): 013011.
24	ZhangJ, LiY, ChenF Q, et al. X-net: A binocular summation network for foreground segmentation[J]. IEEE Access, 2019, 7: 71412－71422.
25	ChenL C, ZhuY K, PapandreouG, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation[A]. European Conference on Computer Vision[C]. Cham, GER: Springer, 2018. 833－851. DOI:10.1007/978－3－030－01234－2_49.
26	YangM K, YuK, ZhangC, et al. DenseASPP for semantic segmentation in street scenes[A]. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition[C]. Salt Lake City, UT, USA: IEEE, 2018. 3684－3692.
27	LinT Y, GoyalP, GirshickR, et al. Focal loss for dense object detection[A]. 2017 IEEE International Conference on Computer Vision (ICCV)[C]. Venice, Italy: IEEE, 2017. 2999－3007.
28	BiancoS, CioccaG, SchettiniR. Combination of video change detection algorithms by genetic programming[J]. IEEE Transactions on Evolutionary Computation, 2017, 21(6): 914－928.
29	ChenM L, YangQ X, LiQ, et al. Spatiotemporal background subtraction using minimum spanning tree and optical flow[A]. European Conference on Computer Vision[C]. Cham, GER: Springer, 2014. 521－534. DOI:10.1007/978－3－319－10584－0_34.
30	SajidH, CheungS C S. Universal multimode background subtraction[J]. IEEE Transactions on Image Processing, 2017, 26(7): 3249－3260.

实验视频	大小×帧数	描述
PETS.	［576×720］×1200	室内，弱阴影，小尺度前景
boulev.	［240×352］×2500	室外，相机抖动
boats	［240×320］×7999	室外，动态背景
bungal.	［240×360］×1700	室外，硬阴影，大尺度前景
park	［288×352］×600	室外，红外视频，前/背景灰度相似度大

实验视频	大小×帧数	描述
PETS.	［576×720］×1200	室内，弱阴影，小尺度前景
boulev.	［240×352］×2500	室外，相机抖动
boats	［240×320］×7999	室外，动态背景
bungal.	［240×360］×1700	室外，硬阴影，大尺度前景
park	［288×352］×600	室外，红外视频，前/背景灰度相似度大

Methods	Metrics	Overall	PETS.	boulev.	boats	bungal.	park
CS-DFS₃	R	0.8053	0.9221	0.8303	0.6901	0.7523	0.8318
	Sp	0.9985	0.9995	0.9992	0.9998	0.9982	0.9958
	P	0.9346	0.9593	0.9809	0.9628	0.9641	0.8060
	FPN	0.0015	0.0005	0.0008	0.0002	0.0018	0.0042
	FNR	0.1947	0.0779	0.1697	0.3099	0.2477	0.1682
	PWC	0.7344	0.1521	0.8725	0.2111	1.6880	0.7484
CS-DFS₃	F	0.8719	0.9334	0.8452	0.9472	0.8879	0.7457
CS-DFS₂	F	0.8039	0.9307	0.7530	0.8202	0.7844	0.7311
CS-DFS₁	F	0.8155	0.9514	0.8183	0.6660	0.8767	0.7651
CS-DFS₀	F	0.6786	0.9005	0.6175	0.4679	0.7218	0.6852

Methods	Metrics	Overall	PETS.	boulev.	boats	bungal.	park
CS-DFS₃	R	0.8053	0.9221	0.8303	0.6901	0.7523	0.8318
	Sp	0.9985	0.9995	0.9992	0.9998	0.9982	0.9958
	P	0.9346	0.9593	0.9809	0.9628	0.9641	0.8060
	FPN	0.0015	0.0005	0.0008	0.0002	0.0018	0.0042
	FNR	0.1947	0.0779	0.1697	0.3099	0.2477	0.1682
	PWC	0.7344	0.1521	0.8725	0.2111	1.6880	0.7484
CS-DFS₃	F	0.8719	0.9334	0.8452	0.9472	0.8879	0.7457
CS-DFS₂	F	0.8039	0.9307	0.7530	0.8202	0.7844	0.7311
CS-DFS₁	F	0.8155	0.9514	0.8183	0.6660	0.8767	0.7651
CS-DFS₀	F	0.6786	0.9005	0.6175	0.4679	0.7218	0.6852

场景	PETS	boulev	boats	bungal	park	Overall	Speed
IURIS-5^［28］	0.9354	0.7680	0.7532	0.8392	0.7652	0.8122	-
MBS^［30］	0.8648	0.8672	0.9041	0.7475	0.7099	0.8187	-
STBM^［29］	0.8610	0.7423	0.8760	0.7927	0.8323	0.8208	12
PAWCS^［19］	0.9315	0.8444	0.8416	0.8387	0.8286	0.8570	-
SemanticBS^［21］	0.9442	0.7372	0.9795	0.9259	0.7694	0.8712	7
FgSegNet_v2^［6］	0.8833	0.7313	0.5581	0.7896	0.5426	0.7010	38
CS-DFS₃（本文）	0.9334	0.8452	0.9472	0.8879	0.7457	0.8719	35

基于帧间高级特征差分的跨场景视频前景分割算法

Cross-Scene Foreground Segmentation Algorithm Based on High-Level Feature Differencing Between Frames

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 9

参考文献 30

相关文章 15

编辑推荐

Metrics

[1]	吴靖, 叶晓晶, 黄峰, 陈丽琼, 王志锋, 刘文犀. 基于深度学习的单帧图像超分辨率重建综述[J]. 电子学报, 2022, 50(9): 2265-2294.
[2]	李雪莹, 王田路, 梁鹏, 王翀. 基于系统模型的用户评论中非功能需求的自动分类[J]. 电子学报, 2022, 50(9): 2079-2089.
[3]	琚长瑞, 秦晓燕, 袁广林, 李豪, 朱虹. 尺度敏感损失与特征融合的快速小目标检测方法[J]. 电子学报, 2022, 50(9): 2119-2126.
[4]	张志昌, 于沛霖, 庞雅丽, 朱林, 曾扬扬. SMGN：用于对话状态跟踪的状态记忆图网络[J]. 电子学报, 2022, 50(8): 1851-1858.
[5]	张亚洲, 俞洋, 朱少林, 陈锐, 戎璐, 梁辉. 一种量子概率启发的对话讽刺识别网络模型[J]. 电子学报, 2022, 50(8): 1885-1893.
[6]	王飞扬, 冀鹏欣, 孙笠, 危倩, 李根, 张忠宝. 一种基于深度学习的动态社交网络用户对齐方法[J]. 电子学报, 2022, 50(8): 1925-1936.
[7]	徐兴荣, 刘聪, 李婷, 郭娜, 任崇广, 曾庆田. 基于双向准循环神经网络和注意力机制的业务流程剩余时间预测方法[J]. 电子学报, 2022, 50(8): 1975-1984.
[8]	邵志文, 周勇, 谭鑫, 马利庄, 刘兵, 姚睿. 基于深度学习的表情动作单元识别综述[J]. 电子学报, 2022, 50(8): 2003-2017.
[9]	欧阳与点, 谢鲲, 谢高岗, 文吉刚. 面向大规模网络测量的数据恢复算法：基于关联学习的张量填充[J]. 电子学报, 2022, 50(7): 1653-1663.
[10]	裴炤, 邱文涛, 王淼, 马苗, 张艳宁. 基于Transformer动态场景信息生成对抗网络的行人轨迹预测方法[J]. 电子学报, 2022, 50(7): 1537-1547.
[11]	彭闯, 王伦文, 胡炜林. 融合深度特征的电磁频谱异常检测算法[J]. 电子学报, 2022, 50(6): 1359-1369.
[12]	杨伟超, 杜宇, 文伟, 侯舒维, 徐常志, 张建华. 基于多重分形谱智能分析的卫星信号调制识别研究[J]. 电子学报, 2022, 50(6): 1336-1343.
[13]	李政伟, 李佳树, 尤著宏, 聂茹, 赵欢, 钟堂波. 基于异质图注意力网络的miRNA与疾病关联预测算法[J]. 电子学报, 2022, 50(6): 1428-1435.
[14]	张波, 陆云杰, 秦东明, 邹国建. 一种卷积自编码深度学习的空气污染多站点联合预测模型[J]. 电子学报, 2022, 50(6): 1410-1427.
[15]	廖勇, 李玉杰. 一种轻量化低复杂度的FDD大规模MIMO系统CSI反馈方法[J]. 电子学报, 2022, 50(5): 1211-1217.