单目图像序列光流三维重建技术研究综述

doi:10.3969/j.issn.0372-2112.2016.12.033

电子学报 ›› 2016, Vol. 44 ›› Issue (12): 3044-3052.DOI: 10.3969/j.issn.0372-2112.2016.12.033

单目图像序列光流三维重建技术研究综述

张聪炫^1,2, 陈震^1,2, 黎明¹

1. 南昌航空大学无损检测技术教育部重点实验室, 江西南昌 330063;
2. 南昌航空大学测试与光电工程学院, 江西南昌 330063

收稿日期:2015-11-24 修回日期:2016-01-19 出版日期:2016-12-25
- 通讯作者:
- 陈震
- 作者简介:
- 张聪炫,男,1984年7月出生于河南省焦作市.分别于2007年和2014年在南昌航空大学和南京航空航天大学获得学士和博士学位.现为南昌航空大学讲师,硕士生导师.主要研究方向为图像检测与智能识别.E-mail:zcxdsg@163.com;黎明,男,1965年2月出生于江西省樟树市.分别于1985年、1990年和1997年在上海交通大学和南京航空航天大学获得学士、硕士和博士学位.现为南昌航空大学教授,博士生导师.主要研究方向为图像处理与模式识别、智能计算.E-mail:limingniat@hotmail.com
- 基金资助:
- 国家自然科学基金 (No.61401190，No.U1233125，No.61462062）; 江西省主要学科学术带头人培养计划项目 (No.201208421）; 江西省自然科学基金重点项目 (No.20133ACB20004）; 江西省科技落地计划项目 (No.201408083）; 航空科学基金 (No.2015ZC56009）; 江西省优势学科团队计划项目 (No.20152BCB24004）; 无损检测技术教育部重点实验室开放基金 (No.ZD201529001）

Review of the 3D Reconstruction Technology Based on Optical Flow of Monocular Image Sequence

ZHANG Cong-xuan^1,2, CHEN Zhen^1,2, LI Ming¹

1. Key Laboratory of Nondestructive Testing, Ministry of Education, Nanchang Hangkong University, Nanchang, Jiangxi 330063, China;
2. School of Measuring and Optical Engineering, Nanchang Hangkong University, Nanchang, Jiangxi 330063, China

Received:2015-11-24 Revised:2016-01-19 Online:2016-12-25 Published:2016-12-25
- Supported by:
- National Natural Science Foundation of China (No.61401190, No.U1233125, No.61462062); Key Disciplines Academic Leader Cultivation Plan of Jiangxi Province (No.201208421); Key Program of Natural Science Foundation of Jiangxi Province, China (No.20133ACB20004); Science and Technology Program of Jiangxi Province (No.201408083); Aeronautical Science Foundation of China, ASFC (No.2015ZC56009); Preponderant Disciplines Team Project of Jiangxi Province (No.20152BCB24004); Key Laboratory Open Foundation for Non-destructive Testing Technology of Education Ministry (No.ZD201529001)

摘要/Abstract

摘要：

由单目图像序列光流重建物体或场景的三维运动与结构是计算机视觉、图像处理与模式识别等领域的重要研究内容，在机器人视觉、无人机导航、车辆辅助驾驶以及医学影像分析等方面具有重要的应用.本文首先从精度与鲁棒性等方面对单目图像序列光流计算及三维重建技术近年来取得的进展进行综述与分析.然后采用Middlebury测试图像序列对HS、LDOF、CLG-TV、SOF、AOFSCNN和Classic+NL等典型光流算法以及Adiv、RMROF、Sekkati和DMDPOF等基于光流的间接与直接重建方法进行实验对比分析，指出各对比方法的优点与不足，归纳各类方法的性能特点与适用范围.最后对利用分数阶微分模型、非局部约束、立体视觉以及深度线索解决亮度突变、非刚性运动、运动遮挡与模糊情况下光流计算及重建模型的局限性与鲁棒性问题进行总结与展望.

关键词: 单目图像序列, 光流, 运动估计, 三维重建, 困难场景, 鲁棒性

Abstract:

The research on 3D motion estimation and structure reconstruction of the object or scene based on the monocular image sequence optical flow is important in computer vision,image processing and pattern recognition,and the research achievements are applied in many fields.such as robot vision,unmanned aerial vehicle(UAV) navigation,driver assistance system,medical image analysis,and so on.Firstly,the research progress of the technologies of the monocular image sequence optical flow estimation and 3D reconstruction is reviewed and analyzed from the aspects of accuracy and robustness.Secondly,the test image sequences of Middlebury database are employed to make comparison among the optical flow methods of HS,LDOF,CLG-TV,SOF,AOFSCNN and Classic+NL,and the direct and indirect reconstruction methods of Adiv,RMROF,Sekkati and DMDPOF based on optical flow are compared.Through the comparison results,the advantages and disadvantages of the methods are pointed out,and the characteristics and applications of the methods are generalized.Finally,the limitations and robustness of the models of optical flow estimation and 3D reconstruction in brightness changing,non-rigid motion,motion occlusion and blur are summarized,and the solutions by the fractional order differential model,non-local constraint,stereo vision and depth cue are presented.

Key words: monocular image sequence, optical flow, motion estimation, 3D reconstruction, hard scene, robustness

中图分类号:

TP391

张聪炫, 陈震, 黎明. 单目图像序列光流三维重建技术研究综述[J]. 电子学报, 2016, 44(12): 3044-3052.

ZHANG Cong-xuan, CHEN Zhen, LI Ming . Review of the 3D Reconstruction Technology Based on Optical Flow of Monocular Image Sequence[J]. Acta Electronica Sinica, 2016, 44(12): 3044-3052.

参考文献

[1] Roberts R,Dellaert F.Direct superpixel labeling for mobile robot navigation using learned general optical flow templates[A].International Conference on Intelligent Robots and Systems[C].Chicago:IEEE,2014.1032-1037.
[2] Nourani-Vatani N,Borges P V K,Roberts J M,et al.On the use of optical flow for scene change detection and description[J].Journal of Intelligent & Robotic Systems,2014,74(3):817-846.
[3] 罗寰,王芳,陈中起,等.基于对称差分和光流估计的红外弱小目标检测[J].光学学报,2010,30(6):1715-1720. Luo Huan,Wang fang,Chen Zhong-qi,et al.Infrared target detecting based on symmetrical displaced frame difference and optical flow estimation[J].Acta Optica Sinica,2010,30(6):1715-1720.(in Chinese)
[4] Rhudy M B,Chao H,Gu Y.Wide-field optical flow aided inertial navigation for unmanned aerial vehicles[A].International Conference on Intelligent Robots and Systems[C].Chicago:IEEE,2014.674-679.
[5] Zhou L,Kambhamettu C,Goldgof D B,et al.Tracking nonrigid motion and structure from 2D satellite cloud images without correspondences[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2001,23(11):1330-1336.
[6] 许鸿奎,江铭炎,杨明强.基于改进光流场模型的脑部多模医学图像配准[J].电子学报,2012,40(3):525-529. Xu Hong-kui,Jiang Ming-yan,Yang Ming-qiang.Registration of multimodal brain medical images based on improved optical flow model[J].Acta Electronica Sinica,2012,40(3):525-529.(in Chinese)
[7] Horn B K,Schunck B G.Determining optical flow[J].Artificial Intelligence,1981,17(1):185-203.
[8] Lucas B D,Kanade T.An iterative image registration technique with an application to stereo vision[A].Proceedings of the 7th International Joint Conference on Artificial Intelligence[C].Vienna:Springer,1981.674-679.
[9] Bruhn A,Weickert J,Schnörr C.Lucas/Kanade meets Horn/Schunck:Combining local and global optic flow methods[J].International Journal of Computer Vision,2005,61(3):211-231.
[10] Brox T,Bruhn A,Papenberg N,et al.High accuracy optical flow estimation based on a theory for warping[A].European Conference on Computer Vision[C].Prague:Springer,2004.25-36.
[11] 陈震,张聪炫,晏文敬,等.基于图像局部结构的区域匹配变分光流算法[J].电子学报,2015,43(11):2200-2209. Chen Zhen,Zhang Cong-xuan,Yan Wen-jing,et al.Region matching variational optical flow algorithm based on image local structure[J].Acta Electronica Sinica,2015,43(11):2200-2209.(in Chinese)
[12] Papenberg N,Bruhn A,Brox T,et al.Highly accurate optic flow computation with theoretically justified warping[J].International Journal of Computer Vision,2006,67(2):141-158.
[13] 刘骏,祖静,张瑜,等.光照变化条件下的光流估计[J].中国图象图形学报,2014,19(10):1475-1480. Liu Jun,Zu Jing,Zhang Yu,et al.Optical flow estimation method under the condition of illumination change[J].Journal of Image and Graphics,2014,19(10):1475-1480.(in Chinese)
[14] Wedel A,Pock T,Zach C,et al.An improved algorithm for TV-L1 optical flow[A].Proceedings of the Statistical and Geometrical Approaches to Visual Motion Analysis[C].Dagstuhl Castle:Springer,2009.23-45.
[15] Drulea M,Nedevschi S.Total variation regularization of local-global optical flow[A].International Conference on Intelligent Transportation Systems[C].Washington:IEEE,2011.318-323.
[16] 项学智,开湘龙,张磊,等.一种变分偏微分多模型光流求解方法[J].仪器仪表学报,2014,35(1):109-116. Xiang Xue-zhi,Kai Xiang-long,Zhang Lei,et al.A method variational partial differential based multi-model optical flow calculation[J].Chinese Journal of Scientific Instrument,2014,35(1):109-116.(in Chinese)
[17] Nagel H H,Enkelmann W.An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,1986,8(5):565-593.
[18] Werlberger M,Trobin W,Pock T,et al.Anisotropic Huber-L1 optical flow[A].Proceedings of the British machine vision conference[C].London:BMVA,2009.1-11.
[19] Xu L,Jia J,Matsushita Y.Motion detail preserving optical flow estimation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2012,34(9):1744-1757.
[20] Zimmer H,Bruhn A,Weickert J.Optic flow in harmony[J].International Journal of Computer Vision,2011,93(3):368-388.
[21] Brox T,Malik J.Large displacement optical flow:Descriptor matching in variational motion estimation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2011,33(3):500-513.
[22] Ayvaci A,Raptis M,Soatto S.Sparse occlusion detection with optical flow[J].International Journal of Computer Vision,2012,97(3):322-338.
[23] Zhang C X,Chen Z,Li M,et al.Anisotropic optical flow algorithm based on self-adaptive cellular neural network[J].Journal of Electronic Imaging,2013,22(1):013038-1-9.
[24] Sun D,Roth S,Black M J.A quantitative analysis of current practices in optical flow estimation and the principles behind them[J].International Journal of Computer Vision,2014,106(2):115-137.
[25] Baker S,Scharstein D,Lewis J P,et al.A database and evaluation methodology for optical flow[J].International Journal of Computer Vision,2011,92(1):1-31.
[26] Roach J W.Determining the movement of objects from a sequence of images[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,1980,2(6):554-562.
[27] Zhuang X,Huang T S,Abuja N,et al.A simplified linear optic flow-motion algorithm[J].Computer Vision,Graphics,and Image Processing,1988,42(3):334-344.
[28] Chiuso A,Favaro P,Jin H,et al.Structure from motion causally integrated over time[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2002,24(4):523-535.
[29] Ozden K E,Schindler K,Van G L.Multibody structure-from-motion in practice[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2010,32(6):1134-1141.
[30] 王俊,朱利.基于图像匹配-点云融合的建筑物立面三维重建[J].计算机学报,2012,35(10):2072-2079. Wang Jun,Zhu Li.3D building facade reconstruction based on image matching-point cloud fusing[J].Chinese Journal of Computer,2012,35(10):2072-2079.(in Chinese)
[31] Nachimson M,Kovalsky S Z,Shlizerman I,et al.Global motion estimation from point matches[A].International Conference on 3D Imaging,Modeling,Processing,Visualization and Transmission[C].Zurich:IEEE,2012.81-88.
[32] Scaramuzza D.1-point-ransac structure from motion for vehicle-mounted cameras by exploiting non-holonomic constraints[J].International Journal of Computer Vision,2011,95(1):74-85.
[33] Crandall D,Owens A,Snavely N,et al.Discrete-continuous optimization for large-scale structure from motion[A].International Conference on Computer Vision and Pattern Recognition[C].Providence:IEEE,2011.3001-3008.
[34] Sinha S N,Steedly D,Szeliski R.A multi-stage linear approach to structure from motion[A].European Conference on Computer Vision[C].Heraklion:Springer,2010.267-281.
[35] Liu Y,Huang T S.Estimation of rigid body motion using straight line correspondences[J].Computer Vision,Graphics,and Image Processing,1988,43(1):37-52.
[36] Liu Y,Huang T S.A linear algorithm for motion estimation using straight line correspondences[J].Computer Vision,Graphics,and Image Processing,1988,44(1):35-57.
[37] 陈震,许强,闫军,等.基于直线光流场的三维检测新方法[J].仪器仪表学报,2012,33(5):1141-1147. Chen Zhen,Xu Qiang,Yan Jun,et a1.New method of 3D detection based on straight-line optical flow[J].Chinese Journal of Scientific Instrument,2012,33(5):1141-1147.(in Chinese)
[38] Taylor C J,Kriegman D J.Structure and motion from line segments in multiple images[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,1995,17(11):1021-1032.
[39] Mirzaei F M,Roumeliotis S I.Globally optimal pose estimation from line correspondences[A].International Conference on Robotics and Automation[C].Shanghai:IEEE,2011.5581-5588.
[40] Kollmann I R,Reichinger A.3D camera pose estimation using line correspondences and 1D homographies[J].Advances in Visual Computing,2010,64(54):41-52.
[41] Mosaddegh S,Fofi D,Vasseur P.Line based motion estimation and reconstruction of piece-wise planar scenes[A].International Workshop on Applications of Computer Vision[C].Kona:IEEE,2011.658-663.
[42] Liu J,Chen Y,Tang X.Decomposition of complex line drawings with hidden lines for 3D planar-faced manifold object reconstruction[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2011,33(1):3-15.
[43] Elqursh A,Elgammal A.Line-based relative pose estimation[A].International Conference on Computer Vision and Pattern Recognition[C].Providence:IEEE,2011.3049-3056.
[44] Bruss R,Horn K P.Passive navigation[J].Computer Vision,Graphics,and Image Processing,1983,21(1):3-20.
[45] Kolev K,Klodt M,Brox T,et al.Continuous global optimization in multiview 3D reconstruction[J].International Journal of Computer Vision,2009,84(1):80-96.
[46] 张聪炫,陈震,黎明.基于稠密光流的鲁棒运动估计与表面结构重建[J].仪器仪表学报,2014,35(1):218-225. Zhang Cong-xuan,Chen Zhen,Li Ming.Robust motion estimation and surface structure reconstruction based on dense optical flow[J].Chinese Journal of Scientific Instrument,2014,35(1):218-225.(in Chinese)
[47] Chessa M,Solari F,Sabatini S P.Adjustable linear models for optic flow based obstacle avoidance[J].Computer Vision and Image Understanding,2013,117(6):603-619.
[48] Adiv G.Determining three-dimensional motion and structure from optical flow generated by several moving objects[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,1985,7(4):384-401.
[49] Meinhardt L E,Hondt O,Facciolo G,et al.Relative depth from monocular optical flow[A].International Conference on Image Processing[C].Brussels:IEEE,2011.2085-2088.
[50] Calway A.Recursive estimation of 3D motion and surface structure from local affine flow parameters[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2005,27(4):562-574.
[51] Lee D J,Merrell P,Wei Z,et al.Two-frame structure from motion using optical flow probability distributions for unmanned air vehicle obstacle avoidance[J].Machine Vision and Applications,2010,21(3):229-240.
[52] Horn B K P,Weldon J E J.Direct methods for recovering motion[J].International Journal of Computer Vision,1988,2(1):51-76.
[53] Mitiche A,Hadjres S.MDL estimation of a dense map of relative depth and 3D motion from a temporal sequence of images[J].Pattern Analysis & Applications,2003,6(1):78-87.
[54] Sekkati H,Mitiche A.A variational method for the recovery of dense 3D structure from motion[J].Robotics and Autonomous Systems,2007,55(7):597-607.
[55] Perriollat M,Hartley R,Bartoli A.Monocular template-based reconstruction of inextensible surfaces[J].International Journal of Computer Vision,2011,95(2):124-137.
[56] Mitiche A,Sekkati H.Optical flow 3d segmentation and interpretation:A variational method with active curve evolution and level sets[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2006,28(11):1818-1829.
[57] Zhang C X,Chen Z,Li M,et al.Direct method for motion estimation and structure reconstruction based on optical flow[J].Optical Engineering,2012,51(6):067004-1-14.
[58] Ummenhofer B,Brox T.Dense 3D reconstruction with a hand-held camera[A].Proceedings of the 34th Symposium of the German Association for Pattern Recognition[C].Graz:Springer,2012.103-112.
[59] 张聪炫,陈震,黎明.金字塔光流三维运动估计与深度重建直接方法[J].仪器仪表学报,2015,36(5):1093-1105. Zhang Cong-xuan,Chen Zhen,Li Ming.Direct method for motion estimation and depth reconstruction based on pyramid optical flow[J].Chinese Journal of Scientific Instrument,2015,36(5):1093-1105.(in Chinese)
[60] Cubber G,Sahli H.Partial differential equation-based dense 3D structure and motion estimation from monocular image sequences[J].IET Computer Vision,2012,6(3):174-185.

单目图像序列光流三维重建技术研究综述

Review of the 3D Reconstruction Technology Based on Optical Flow of Monocular Image Sequence

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	范兵兵, 何庭建, 张聪炫, 陈震, 黎明. 联合遮挡约束与残差补偿的特征金字塔光流计算方法[J]. 电子学报, 2023, 51(3): 648-657.
[2]	刘玉轩, 张力, 艾海滨, 许彪, 孙钰珊, 樊仲藜. 光场相机三维重建研究进展与展望[J]. 电子学报, 2022, 50(7): 1774-1792.
[3]	林志, 林敏, 黄清泉, 赵柏, 顾晨伟, 许拔. 能效最大化准则下的星地融合网络的安全波束成形算法[J]. 电子学报, 2022, 50(1): 124-134.
[4]	李燕, 严永财. 基于同心圆阵列的结构光主动视觉系统标定算法[J]. 电子学报, 2021, 49(3): 536-541.
[5]	禤韵怡, 杨春玲. 基于帧间组稀疏的两阶段递归增强视频压缩感知重构网络[J]. 电子学报, 2021, 49(3): 435-442.
[6]	张聪炫, 周仲凯, 陈震, 葛利跃, 黎明, 江少锋, 陈昊. 深度学习光流计算技术研究进展[J]. 电子学报, 2020, 48(9): 1841-1849.
[7]	吴立强, 韩益亮, 杨晓元, 张敏情, 杨凯. 基于理想格的鲁棒门限代理重加密方案[J]. 电子学报, 2020, 48(9): 1786-1794.
[8]	孟祥新, 柳桃荣, 笪敏, 刘泽鑫. 一种高精度的毫米波稀疏平面阵列频域类成像算法[J]. 电子学报, 2020, 48(8): 1479-1485.
[9]	井美, 惠小静, 王蓉. 基于相似度的直觉模糊推理反向三I算法的鲁棒性[J]. 电子学报, 2020, 48(2): 265-271.
[10]	曾伟辉, 李淼, 李增, 熊焰. 基于高阶残差和参数共享反馈卷积神经网络的农作物病害识别[J]. 电子学报, 2019, 47(9): 1979-1986.
[11]	张聪炫, 陈震, 熊帆, 黎明, 葛利跃, 陈昊. 非刚性稠密匹配大位移运动光流估计[J]. 电子学报, 2019, 47(6): 1316-1323.
[12]	王春东, 杨蕾, 万福金, 刘哲理, 朱立坤. 数据库水印模型及其算法综述[J]. 电子学报, 2019, 47(4): 946-954.
[13]	葛利跃, 张聪炫, 陈震, 黎明, 陈昊. 相互结构引导滤波TV-L1变分光流估计[J]. 电子学报, 2019, 47(3): 707-713.
[14]	惠小静, 井美, 王蓉. 基于直觉模糊推理(1,2,2)-a型泛三I算法的鲁棒性[J]. 电子学报, 2019, 47(2): 410-416.
[15]	黄翔东, 杨琳, 杨孟凯, 黄光明. 抗噪鲁棒性可分级的稀疏阵列频率和到达角估计[J]. 电子学报, 2019, 47(1): 122-128.