Binocular Scene Flow Estimation Based on Semantic Segmentation
CHEN Zhen1, MA Long1, ZHANG Cong-xuan1,2, LI Ming1, WU Jun-jie1, JIANG Shao-feng1
1. Key Laboratory of Nondestructive Testing(Ministry of Education), Nanchang Hangkong University, Nanchang, Jiangxi 330063, China;
2. Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
Abstract：In order to address the issue of motion boundary blurring caused by the complex scenes,large displacement and motion occlusion,this paper proposes a binocular scene flow estimation method based on semantic segmentation.Firstly,by using the image semantic information,we classify the image regions into several categories with semantic labels through convolutional neural networks.Then we plan the motion models of various image regions according to the different semantic categories and compute the optical flow and disparity under the prior knowledge of semantic information.Secondly,we apply the superpixel segmentation to the input image and couple the optical flow and disparity information via least squares method to solve the motion parameters of each superpixel patch.Finally,we add the boundary information of semantic segmentation constraint to the optimization energy function,and estimate the scene flow by updating the mappings of pixels-to-superpixel and superpixel-to-plane.We evaluate the proposed approach and some state-of-the-art methods on the KITTI 2015 database to conduct a comparison experiment.The experimental results demonstrate that our method has high accuracy and good robustness,and especially has significant benefit of boundary preserving in the areas of complex scene,motion occlusion and motion boundary.
[1] Vedula S,Baker S,Collins R,et al.Three-dimensional scene flow[A].International Conference on Computer Vision[C].Kerkyra:IEEE,1999.722-729.
[2] Huguet F,Devernay F.A variational method for scene flow estimation from stereo sequences[A].IEEE International Conference on Computer Vision[C].Rio de Janeiro:IEEE,2007.1-7.
[3] Basha T,Moses Y,Kiryati N.Multi-view scene flow estimation:a view centered variational approach[J].International Journal of Computer Vision,2013,101(1):6-21.
[4] Schuster R,Wasenmuller O,Kuschk G,et al.Sceneflowfields:dense interpolation of sparse scene flow correspondences[A].IEEE Winter Conference on Applications of Computer Vision[C].Nevada:IEEE,2018.1-10.
[5] Schuster R,Bailer C,Wasenmuller O,et al.Combining stereo disparity and optical flow for basic scene flow[A].International Commercial Vehicle Technology Symposium[C].Kaiserslautern:Springer,2018.90-101.
[6] Vogel C,Schindler K,Roth S.Piecewise rigid scene flow[A].IEEE International Conference on Computer Vision[C].Sydney:IEEE,2013.1377-1384.
[7] Chen L C,Papandreou G,Kokkinos I,et al.Deeplab:semantic image segmentation with deep convolutional nets,atrous convolution,and fully connected CRFs[J].IEEE Transactions on Pattern Analysis & Machine Intelligence,2018,40(4):834-848.
[8] Sevilla-Lara L,Sun D,Jampani V,et al.Optical flow with semantic segmentation and localized layers[A].IEEE Conference on Computer Vision and Pattern Recognition[C].Las Vegas:IEEE,2016.3889-3898.
[9] Lempitsky V,Roth S,Rother C.Fusionflow:discrete-continuous optimization for optical flow estimation[A].IEEE Conference on Computer Vision and Pattern Recognition[C].Anchorage:IEEE,2008.1-8.
[10] Derome M,Plyer A,Sanfourche M,et al.A prediction-correction approach for real-time optical flow computation using stereo[A].German Conference on Pattern Recognition[C].Hannover:Springer,2016.1-7.
[11] Lü Z,Beall C,Alcantarilla P F,et al.A continuous optimization approach for efficient and accurate scene flow[A].European Conference on Computer Vision[C].Amsterdam:Springer,2016.757-773.
[12] Taniai T,Sinha S N,Sato Y.Fast multi-frame stereo scene flow with motion segmentation[A].IEEE Conference on Computer Vision and Pattern Recognition[C].Hawaii:IEEE,2017.729-739.
2020, Vol. 48 
