基于语义分割的双目场景流估计

doi:10.3969/j.issn.0372-2112.2020.04.002

电子学报 ›› 2020, Vol. 48 ›› Issue (4): 631-636.DOI: 10.3969/j.issn.0372-2112.2020.04.002

基于语义分割的双目场景流估计

陈震¹, 马龙¹, 张聪炫^1,2, 黎明¹, 吴俊劼¹, 江少锋¹

1. 南昌航空大学无损检测技术教育部重点实验室, 江西南昌 330063;
2. 中国科学院自动化研究所, 北京 100190

收稿日期:2018-12-07 修回日期:2019-05-13 出版日期:2020-04-25
- 通讯作者:
- 张聪炫
- 作者简介:
- 陈震男,1969年11月生,江西九江人.分别于1993、2000和2003年在西北工业大学获得学士、硕士和博士学位.现为南昌航空大学教授,博士生导师,主要研究方向为计算机视觉、图像处理与模式识别.E-mail:dr_chenzhen@163.com;马龙男,1993年9月出生于河南省鹤壁市.现为南昌航空大学测试与光电工程学院硕士研究生.主要研究方向为图像检测与智能识别.E-mail:1007516637@qq.com
- 基金资助:
- 国家自然科学基金 (No.61866026，No.61772255，No.61866025）; 江西省优势科技创新团队计划 (No.20152BCB24004，No.20165BCB19007）; 江西省青年科学基金 (No.20171BAB212012）; 中国博士后科学基金 (No.2019M650894）

Binocular Scene Flow Estimation Based on Semantic Segmentation

CHEN Zhen¹, MA Long¹, ZHANG Cong-xuan^1,2, LI Ming¹, WU Jun-jie¹, JIANG Shao-feng¹

1. Key Laboratory of Nondestructive Testing (Ministry of Education), Nanchang Hangkong University, Nanchang, Jiangxi 330063, China;
2. Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China

Received:2018-12-07 Revised:2019-05-13 Online:2020-04-25 Published:2020-04-25
- Corresponding author:
- ZHANG Cong-xuan
- Supported by:
- National Natural Science Foundation of China (No.61866026, No.61772255, No.61866025); Superior Science and Technology Innovation Team Project of Jiangxi Province (No.20152BCB24004, No.20165BCB19007); Youth Science Fund of Jiangxi Province (No.20171BAB212012); China Postdoctoral Science Foundation (No.2019M650894)

摘要/Abstract

摘要： 针对现有场景流计算方法在复杂场景、大位移和运动遮挡等情况下易产生运动边缘模糊的问题，提出一种基于语义分割的双目场景流估计方法.首先，根据图像中的语义信息类别，通过深度学习的卷积神经网络模型将图像划分为带有语义标签的区域；针对不同语义类别的图像区域分别进行运动建模，利用语义知识计算光流信息并通过双目立体匹配的半全局匹配方法计算图像视差信息.然后，对输入图像进行超像素分割，通过最小二乘法耦合光流和视差信息，分别求解每个超像素块的运动参数.最后，在优化能量函数中添加语义分割边界的约束信息，通过更新像素到超像素块的映射关系和超像素块到移动平面的映射关系得到最终的场景流估计结果.采用KITTI 2015标准测试图像序列对本文方法和代表性的场景流计算方法进行对比分析.实验结果表明，本文方法具有较高的精度和鲁棒性，尤其对于复杂场景、运动遮挡和运动边缘模糊的图像具有较好的边缘保护作用.

关键词: 语义分割, 场景流, 深度学习, 双目立体匹配, 最小二乘法, 超像素分割, 运动遮挡, 边缘保护

Abstract: In order to address the issue of motion boundary blurring caused by the complex scenes, large displacement and motion occlusion, this paper proposes a binocular scene flow estimation method based on semantic segmentation. Firstly, by using the image semantic information, we classify the image regions into several categories with semantic labels through convolutional neural networks. Then we plan the motion models of various image regions according to the different semantic categories and compute the optical flow and disparity under the prior knowledge of semantic information. Secondly, we apply the superpixel segmentation to the input image and couple the optical flow and disparity information via least squares method to solve the motion parameters of each superpixel patch. Finally, we add the boundary information of semantic segmentation constraint to the optimization energy function, and estimate the scene flow by updating the mappings of pixels-to-superpixel and superpixel-to-plane. We evaluate the proposed approach and some state-of-the-art methods on the KITTI 2015 database to conduct a comparison experiment. The experimental results demonstrate that our method has high accuracy and good robustness, and especially has significant benefit of boundary preserving in the areas of complex scene, motion occlusion and motion boundary.

Key words: semantic segmentation, scene flow, deep learning, binocular stereo matching, least squares method, superpixel segmentation, motion occlusion, edge protection

中图分类号:

TP391

陈震, 马龙, 张聪炫, 等. 基于语义分割的双目场景流估计[J]. 电子学报, 2020, 48(4): 631-636.

CHEN Zhen, MA Long, ZHANG Cong-xuan, et al. Binocular Scene Flow Estimation Based on Semantic Segmentation[J]. Acta Electronica Sinica, 2020, 48(4): 631-636.

参考文献

[1] Vedula S,Baker S,Collins R,et al.Three-dimensional scene flow[A].International Conference on Computer Vision[C].Kerkyra:IEEE,1999.722-729.
[2] Huguet F,Devernay F.A variational method for scene flow estimation from stereo sequences[A].IEEE International Conference on Computer Vision[C].Rio de Janeiro:IEEE,2007.1-7.
[3] Basha T,Moses Y,Kiryati N.Multi-view scene flow estimation:a view centered variational approach[J].International Journal of Computer Vision,2013,101(1):6-21.
[4] Schuster R,Wasenmuller O,Kuschk G,et al.Sceneflowfields:dense interpolation of sparse scene flow correspondences[A].IEEE Winter Conference on Applications of Computer Vision[C].Nevada:IEEE,2018.1-10.
[5] Schuster R,Bailer C,Wasenmuller O,et al.Combining stereo disparity and optical flow for basic scene flow[A].International Commercial Vehicle Technology Symposium[C].Kaiserslautern:Springer,2018.90-101.
[6] Vogel C,Schindler K,Roth S.Piecewise rigid scene flow[A].IEEE International Conference on Computer Vision[C].Sydney:IEEE,2013.1377-1384.
[7] Chen L C,Papandreou G,Kokkinos I,et al.Deeplab:semantic image segmentation with deep convolutional nets,atrous convolution,and fully connected CRFs[J].IEEE Transactions on Pattern Analysis & Machine Intelligence,2018,40(4):834-848.
[8] Sevilla-Lara L,Sun D,Jampani V,et al.Optical flow with semantic segmentation and localized layers[A].IEEE Conference on Computer Vision and Pattern Recognition[C].Las Vegas:IEEE,2016.3889-3898.
[9] Lempitsky V,Roth S,Rother C.Fusionflow:discrete-continuous optimization for optical flow estimation[A].IEEE Conference on Computer Vision and Pattern Recognition[C].Anchorage:IEEE,2008.1-8.
[10] Derome M,Plyer A,Sanfourche M,et al.A prediction-correction approach for real-time optical flow computation using stereo[A].German Conference on Pattern Recognition[C].Hannover:Springer,2016.1-7.
[11] Lü Z,Beall C,Alcantarilla P F,et al.A continuous optimization approach for efficient and accurate scene flow[A].European Conference on Computer Vision[C].Amsterdam:Springer,2016.757-773.
[12] Taniai T,Sinha S N,Sato Y.Fast multi-frame stereo scene flow with motion segmentation[A].IEEE Conference on Computer Vision and Pattern Recognition[C].Hawaii:IEEE,2017.729-739.

[1]	李钦, 刘伟, 牛朝阳, 宝音图, 惠周勃. 低信噪比下基于分裂EfficientNet网络的雷达信号调制方式识别[J]. 电子学报, 2023, 51(3): 675-686.
[2]	范兵兵, 何庭建, 张聪炫, 陈震, 黎明. 联合遮挡约束与残差补偿的特征金字塔光流计算方法[J]. 电子学报, 2023, 51(3): 648-657.
[3]	张聿远, 闫文君, 张立民. 基于多模态特征融合网络的空时分组码识别算法[J]. 电子学报, 2023, 51(2): 489-498.
[4]	许新征, 李杉. 基于特征膨胀卷积模块的轻量化技术研究[J]. 电子学报, 2023, 51(2): 355-364.
[5]	张永梅, 孙捷. 基于动静态特征双输入神经网络的咳嗽声诊断COVID-19算法[J]. 电子学报, 2023, 51(1): 202-212.
[6]	袁海英, 成君鹏, 曾智勇, 武延瑞. Mobile_BLNet：基于Big-Little Net的轻量级卷积神经网络优化设计[J]. 电子学报, 2023, 51(1): 180-191.
[7]	王神龙, 雍宇, 吴晨睿. 基于伪孪生神经网络的低纹理工业零件6D位姿估计[J]. 电子学报, 2023, 51(1): 192-201.
[8]	李滔, 董秀成, 林宏伟. 基于深监督跨尺度注意力网络的深度图像超分辨率重建[J]. 电子学报, 2023, 51(1): 128-138.
[9]	郭晓轩, 冯其波, 冀振燕, 郑发家, 杨燕燕. 多线激光光条图像缺陷分割模型研究[J]. 电子学报, 2023, 51(1): 172-179.
[10]	贾童瑶, 卓力, 李嘉锋, 张菁. 基于深度学习的单幅图像去雾研究进展[J]. 电子学报, 2023, 51(1): 231-245.
[11]	何滢婕, 刘月峰, 边浩东, 郭威, 张小燕. 基于Informer的电池荷电状态估算及其稀疏优化方法[J]. 电子学报, 2023, 51(1): 50-56.
[12]	陈震, 倪晶晶, 张聪炫, 葛利跃, 王梓歌. 基于金字塔块匹配的双目场景流估计[J]. 电子学报, 2022, 50(9): 2164-2171.
[13]	吴靖, 叶晓晶, 黄峰, 陈丽琼, 王志锋, 刘文犀. 基于深度学习的单帧图像超分辨率重建综述[J]. 电子学报, 2022, 50(9): 2265-2294.
[14]	李雪莹, 王田路, 梁鹏, 王翀. 基于系统模型的用户评论中非功能需求的自动分类[J]. 电子学报, 2022, 50(9): 2079-2089.
[15]	琚长瑞, 秦晓燕, 袁广林, 李豪, 朱虹. 尺度敏感损失与特征融合的快速小目标检测方法[J]. 电子学报, 2022, 50(9): 2119-2126.

基于语义分割的双目场景流估计

Binocular Scene Flow Estimation Based on Semantic Segmentation

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics