基于金字塔块匹配的双目场景流估计

doi:10.12263/DZXB.20210385

电子学报 ›› 2022, Vol. 50 ›› Issue (9): 2164-2171.DOI: 10.12263/DZXB.20210385

基于金字塔块匹配的双目场景流估计

陈震¹, 倪晶晶¹, 张聪炫¹, 葛利跃², 王梓歌¹

^1.南昌航空大学无损检测技术教育部重点实验室，江西南昌 330063
^2.南昌航空大学信息工程学院，江西南昌 330063

收稿日期:2021-03-22 修回日期:2022-05-04 出版日期:2022-09-25
- 通讯作者:
- 张聪炫
- 作者简介:
- 陈　震　男，1969年11月出生于江西省九江市. 博士，教授. 主要研究方向为图像理解与测量.E-mail: dr_chenzhen@163.com
  倪晶晶　女， 1993年09月出生于安徽省桐城市. 硕士研究生. 主要研究方向为图像检测与智能识别.E-mail: njjplhello@163.com
  张聪炫　男， 1984年07月出生于河南省焦作市. 博士，教授. 主要研究方向为图像处理与计算机视觉.
  葛利跃　男， 1992年10月出生于安徽省蚌埠市. 硕士，助理实验师. 主要研究方向为图像处理与计算机视觉.E-mail: lygeah@163.com
  王梓歌　女， 1998年04月出生于河北省晋州市. 硕士研究生. 主要研究方向为图像检测与智能识别．E-mail: wangzggg@163.com
- 基金资助:
- 国家重点研发计划 (2020YFC2003800); 国家自然科学基金 (61866026); 江西省优势科技创新团队计划 (20165BCB19007); 江西省技术创新引导类计划项目 (20212AEI91005); 江西省自然科学基金重点项目 (20202ACB214007); 航空科学基金 (2018ZC56008); 中国博士后科学基金 (2019M650894); 江西省教育厅科学技术研究项目 (GJJ210910); 江西省图像处理与模式识别重点实验室开放基金资助 (ET202104413)

Binocular Scene Flow Estimation Based on Pyramid Block Matching

CHEN Zhen¹, NI Jing-jing¹, ZHANG Cong-xuan¹, GE Li-yue², WANG Zi-ge¹

^1.Key Laboratory of Nondestructive Testing，Ministry of Education，Nanchang Hangkong University，Nanchang，Jiangxi 330063，China
^2.School of Information Engineering，Nanchang Hangkong University，Nanchang，Jiangxi 330063，China

Received:2021-03-22 Revised:2022-05-04 Online:2022-09-25 Published:2022-10-26
- Corresponding author:
- ZHANG Cong-xuan

摘要/Abstract

摘要：

针对现有双目场景流计算方法在大位移、运动遮挡及光照变化等复杂场景下场景流估计的准确性与鲁棒性问题，提出一种基于金字塔块匹配的双目场景流计算方法.首先对双目图像序列进行超像素分割和视差估计，得到图像初始分割结果和视差信息，然后建立基于金字塔块匹配的运动模型并采用Ransac随机一致性算法拟合刚性运动模型和最小化重投影算法估计对象运动参数.最后，本文将金字塔块匹配结果作为约束项，联合对象运动参数和超像素平面参数构建基于金字塔块匹配的双目场景流估计能量函数模型，通过最小化能量函数得到最终场景流.实验分别采用KITTI2015（Karlsruhe Institute of Technology and Toyota technological Institute 2015）和MPI-Sintel（Max-Planck Institute and Sintel）数据集测试图像对本文方法和具有代表性场景流算法进行综合对比分析，结果表明本文方法相对于其他对比方法有效提高大位移、运动遮挡以及光照变化情况下场景流估计精度和鲁棒性.

关键词: 双目场景流, 金字塔块匹配, 运动模型, 大位移运动, 超像素分割

Abstract:

Aiming at the accuracy and robustness of existing binocular scene flow calculation methods in complex scenes such as large displacement, motion occlusion and illumination changes, this paper proposes a binocular scene flow estimation method based on pyramid block matching. Firstly, we apply the superpixel segmentation and disparity estimation to the binocular image sequence to obtain the initial image segmentation results and disparity information. Secondly, we establish a motion model based on pyramid block matching. Then we fit the rigid motion model by using Ransac stochastic consensus algorithm and estimate the object motion parameters by minimizing the reprojection algorithm. Finally, this paper takes the matching result of the pyramid block as a constraint item, then we construct a binocular scene flow estimation energy function model based on the pyramid block matching by combines the object motion parameters and the superpixel plane parameters, and obtains the final scene flow by minimizing the energy function. The image sequences provided by the KITTI2015(Karlsruhe Institute of Technology and Toyota Technological Institute 2015) and MPI-Sintel(Max-Planck Institute and Sintel) databases were adopted to compare and analyze the proposed method in this paper and the existing representative scene flow method. The experimental results show that compared with other comparison methods, the method in this paper has high accuracy and robustness of scene flow estimatin, especially in large displacement, motion occlusion and lighting changes.

Key words: binocular scene flow, pyramid block matching, motion model, large displacement motion, superpixel segmentation

中图分类号:

TP391

陈震, 倪晶晶, 张聪炫, 等. 基于金字塔块匹配的双目场景流估计[J]. 电子学报, 2022, 50(9): 2164-2171.

Zhen CHEN, Jing-jing NI, Cong-xuan ZHANG, et al. Binocular Scene Flow Estimation Based on Pyramid Block Matching[J]. Acta Electronica Sinica, 2022, 50(9): 2164-2171.

图/表 7

参考文献 16

1	常侃, 张智勇, 陈诚, 等. 采用低秩与加权稀疏分解的视频前景检测算法[J]. 电子学报, 2017, 45(9): 2272-2280.
	CHANG Kan, ZHANG Zhi-yong, CHEN Chen, et al. Video foreground detection by low-rank and reweighted sparse decomposition[J]. Acta Electronica Sinica, 2017, 45(9): 2272-2280. (in Chinese)
2	谢佳龙, 张波涛, 吕强. 一种基于双流融合3D卷积神经网络的动态头势识别方法[J]. 电子学报, 2021, 49(7): 1363-1369.
	XIE Jia-long, ZHANG Bo-tao, Qiang LÜ. A dynamic head gesture recognition method based on 3D convolutional two-stream network fusion[J]. Acta Electronica Sinica, 2021, 49(7): 1363-1369. (in Chinese)
3	王伟, 于磊, 任国恒, 等. 城市建筑立面三维"线-面"结构快速重建[J]. 电子学报, 2021, 49(8): 1551-1560.
	WANG Wei, YU Lei, REN Guo-heng, et al. Rapidly reconstructing 3D line-plane structures of urban building facades[J]. Acta Electronica Sinica, 2021, 49(8): 1551-1560. (in Chinese)
4	张聪炫, 裴刘继, 陈震, 等. FRFCM聚类与深度优化的RGBD场景流计算[J]. 电子学报, 2020, 48(7): 1380-1386.
	ZHANG Cong-xuan, PEI Liu-ji, CHEN Zhen, et al. RGBD scene flow estimation based on FRFCM clustering and depth optimization[J]. Acta Electronica Sinica, 2020, 48(7): 1380-1386. (in Chinese)
5	MENZE M, GEIGER A. Object scene flow for autonomous vehicles[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2015: 3061-3070.
6	SCHUSTER R, WASENMULLER O, KUSCHK G, et al. SceneFlowFields: dense interpolation of sparse scene flow correspondences[C]//2018 IEEE Winter Conference on Applications of Computer Vision. Piscataway: IEEE Press, 2018: 1056-1065.
7	REN Z L, SUN D Q, KAUTZ J, et al. Cascaded scene flow prediction using semantic segmentation[C]//2017 International Conference on 3D Vision. Piscataway: IEEE Press, 2017: 225-233.
8	JIANG H Z, SUN D Q, JAMPANI V, et al. SENSE: A shared encoder network for scene-flow estimation[C]//2019 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE Press, 2019: 3194-3203.
9	YANG G S, RAMANAN D. Upgrading optical flow to 3D scene flow through optical expansion[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2020: 1331-1340.
10	SAXENA R, SCHUSTER R, WASENMULLER O, et al. PWOC-3D: Deep occlusion-aware end-to-end scene flow estimation[C]//2019 IEEE Intelligent Vehicles Symposium. Piscataway: IEEE Press, 2019: 324-331.
11	HUR J, ROTH S. Self-supervised monocular scene flow estimation[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2020: 7394-7403.
12	MENZE M, HEIPKE C, GEIGER A. Object scene flow[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018, 140: 60-76.
13	TANIAI T, SINHA S N, SATO Y. Fast multi-frame stereo scene flow with motion segmentation[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 6891-6900.
14	VOGEL C, SCHINDLER K, ROTH S. Piecewise rigid scene flow[C]//2013 IEEE International Conference on Computer Vision. Piscataway: IEEE Press, 2013: 1377-1384.
15	BUTLER D J, WULFF J, STANLEY G B, et al. A Naturalistic Open Source Movie for Optical Flow Evaluation[C]//European Conference on Computer Vision. Berlin, Heidelberg: Springer Press, 2012: 611-625.
16	陈震, 马龙, 张聪炫, 等. 基于语义分割的双目场景流估计[J]. 电子学报, 2020, 48(4): 631-636.
	CHEN Zhen, MA Long, ZHANG Cong-xuan, et al. Binocular scene flow estimation based on semantic segmentation[J]. Acta Electronica Sinica, 2020, 48(4): 631-636. (in Chinese)

类型	对比方法	D1-bg	D1-fg	D1-all	Fl-bg	Fl-fg	Fl-all	SF-bg	SF-fg	SF-all
深度学习方法	SE^[9]	1.48	3.46	1.81	5.83	8.66	6.30	7.06	13.44	8.12
	PWOC-3D^[10]	4.19	9.82	5.13	12.40	15.78	12.96	14.30	22.66	15.69
	SENSE^[8]	2.07	3.01	2.22	7.30	9.33	7.64	8.36	15.49	9.55
	Self_Mono^[11]	20.72	29.41	22.16	15.51	17.96	15.91	31.51	45.77	33.88
传统方法	SSF^[7]	3.55	8.75	4.42	5.63	14.71	7.14	7.18	24.58	10.07
	OSF^[5]	4.54	12.03	5.79	5.62	18.92	7.83	7.01	26.34	10.23
	SFF^[6]	5.12	13.83	6.57	10.58	24.41	12.88	12.48	32.28	15.78
	FSF+MS^[13]	5.72	11.84	6.74	8.48	25.43	11.30	11.17	33.91	14.96
	PRSF^[14]	4.74	13.74	6.24	11.73	24.33	13.83	13.49	31.22	16.44
	本文方法	3.25	9.11	4.22	5.02	15.23	6.72	6.15	21.48	8.70

类型	对比方法	D1-bg	D1-fg	D1-all	Fl-bg	Fl-fg	Fl-all	SF-bg	SF-fg	SF-all
深度学习方法	SE^[9]	1.48	3.46	1.81	5.83	8.66	6.30	7.06	13.44	8.12
	PWOC-3D^[10]	4.19	9.82	5.13	12.40	15.78	12.96	14.30	22.66	15.69
	SENSE^[8]	2.07	3.01	2.22	7.30	9.33	7.64	8.36	15.49	9.55
	Self_Mono^[11]	20.72	29.41	22.16	15.51	17.96	15.91	31.51	45.77	33.88
传统方法	SSF^[7]	3.55	8.75	4.42	5.63	14.71	7.14	7.18	24.58	10.07
	OSF^[5]	4.54	12.03	5.79	5.62	18.92	7.83	7.01	26.34	10.23
	SFF^[6]	5.12	13.83	6.57	10.58	24.41	12.88	12.48	32.28	15.78
	FSF+MS^[13]	5.72	11.84	6.74	8.48	25.43	11.30	11.17	33.91	14.96
	PRSF^[14]	4.74	13.74	6.24	11.73	24.33	13.83	13.49	31.22	16.44
	本文方法	3.25	9.11	4.22	5.02	15.23	6.72	6.15	21.48	8.70

场景	序列	SSF^[7]				OSF^[5]				SE^[9]				本文方法
场景	序列	D1	D2	Fl	SF	D1	D2	Fl	SF	D1	D2	Fl	SF	D1	D2	Fl	SF
大位移	04	4.70	8.81	7.82	11.65	3.28	10.65	14.91	15.15	2.08	3.18	6.06	8.00	2.16	2.76	4.61	4.83
大位移	17	2.73	4.49	7.31	8.54	2.44	3.61	6.55	7.24	0.69	1.98	4.83	5.62	1.85	3.06	5.92	6.35
运动遮挡	07	5.13	5.11	20.66	22.93	4.53	4.55	13.07	15.36	1.00	1.00	4.84	5.28	3.54	3.51	3.10	4.74
运动遮挡	18	9.28	52.55	54.99	55.78	8.84	53.63	54.91	55.68	3.32	50.26	56.16	57.11	5.95	52.69	54.22	54.64
光照变化	02	3.76	4.01	3.92	4.91	4.24	4.45	3.75	4.93	1.89	4.56	6.61	8.32	3.27	3.61	3.48	4.07
光照变化	03	7.28	12.02	15.66	16.60	9.89	14.75	16.49	17.32	1.48	3.93	6.45	10.37	5.01	6.72	11.16	11.81

场景	序列	SSF^[7]				OSF^[5]				SE^[9]				本文方法
场景	序列	D1	D2	Fl	SF	D1	D2	Fl	SF	D1	D2	Fl	SF	D1	D2	Fl	SF
大位移	04	4.70	8.81	7.82	11.65	3.28	10.65	14.91	15.15	2.08	3.18	6.06	8.00	2.16	2.76	4.61	4.83
大位移	17	2.73	4.49	7.31	8.54	2.44	3.61	6.55	7.24	0.69	1.98	4.83	5.62	1.85	3.06	5.92	6.35
运动遮挡	07	5.13	5.11	20.66	22.93	4.53	4.55	13.07	15.36	1.00	1.00	4.84	5.28	3.54	3.51	3.10	4.74
运动遮挡	18	9.28	52.55	54.99	55.78	8.84	53.63	54.91	55.68	3.32	50.26	56.16	57.11	5.95	52.69	54.22	54.64
光照变化	02	3.76	4.01	3.92	4.91	4.24	4.45	3.75	4.93	1.89	4.56	6.61	8.32	3.27	3.61	3.48	4.07
光照变化	03	7.28	12.02	15.66	16.60	9.89	14.75	16.49	17.32	1.48	3.93	6.45	10.37	5.01	6.72	11.16	11.81

场景	图像序列	PRSF^[14]	OSF^[5]	FSF+MS^[13]	SFF^[6]	SS-SF^[16]	SE^[9]	本文方法
大位移	market_2	5.81	10.08	5.17	7.11	5.79	6.06	5.58
	ambush_4	48.60	49.16	45.23	60.03	48.55	54.34	39.23
	ambush_6	49.77	54.75	44.05	57.06	49.37	59.45	36.88
	平均值	34.73	38.00	31.48	41.40	34.57	39.95	27.23
运动遮挡	temple_2	12.61	10.52	9.66	29.58	12.57	13.70	6.88
	ambush_2	66.22	87.37	72.68	90.92	66.33	76.19	40.39
	bamboo_2	5.05	4.86	3.65	5.84	5.06	6.33	4.53
	平均值	27.96	34.25	28.66	42.11	27.99	32.07	17.27
光照变化	market_5	41.33	29.58	26.31	40.77	41.36	39.83	23.48
	alley_2	1.62	1.44	1.20	2.85	1.61	1.71	1.19
	market_6	22.84	16.39	13.13	28.92	22.87	16.97	10.27
	平均值	21.93	15.80	13.55	24.18	21.95	19.50	11.65

基于金字塔块匹配的双目场景流估计

Binocular Scene Flow Estimation Based on Pyramid Block Matching

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 7

参考文献 16

相关文章 7

编辑推荐

Metrics

[1]	陈丹, 姚伯羽. 运动模型引导的自适应核相关目标跟踪方法[J]. 电子学报, 2021, 49(3): 550-558.
[2]	陈震, 马龙, 张聪炫, 黎明, 吴俊劼, 江少锋. 基于语义分割的双目场景流估计[J]. 电子学报, 2020, 48(4): 631-636.
[3]	张聪炫, 陈震, 熊帆, 黎明, 葛利跃, 陈昊. 非刚性稠密匹配大位移运动光流估计[J]. 电子学报, 2019, 47(6): 1316-1323.
[4]	于冲, 赵海, 司帅宗, 彭海霞. 车载网运动模型的复杂网络解析方法[J]. 电子学报, 2017, 45(6): 1449-1455.
[5]	宋涛, 李鸥, 刘广怡. 基于置信区域内多级动态层表达的类贯序蒙特卡洛视觉跟踪方法[J]. 电子学报, 2016, 44(6): 1355-1361.
[6]	刘明亮, 高剑. 基于改进的迭代增量频域反卷积滤波器的图像复原[J]. 电子学报, 2004, 32(12): 2017-2019.
[7]	俞天力, 章毓晋. 基于全局运动信息的视频检索技术[J]. 电子学报, 2001, 29(S1): 1794-1798.