深度学习光流计算技术研究进展

doi:10.3969/j.issn.0372-2112.2020.09.023

电子学报 ›› 2020, Vol. 48 ›› Issue (9): 1841-1849.DOI: 10.3969/j.issn.0372-2112.2020.09.023

深度学习光流计算技术研究进展

张聪炫^1,2, 周仲凯¹, 陈震¹, 葛利跃¹, 黎明¹, 江少锋¹, 陈昊¹

1. 南昌航空大学无损检测技术教育部重点实验室, 江西南昌 330063;
2. 中国科学院自动化研究所, 北京 100190

收稿日期:2019-08-26 修回日期:2019-12-23 出版日期:2020-09-25
- 通讯作者:
- 陈震
- 作者简介:
- 张聪炫 男,1984年7月出生于河南省焦作市.分别于2007和2014年在南昌航空大学和南京航空航天大学获得学士和博士学位.现为南昌航空大学副教授,硕士生导师,中国科学院自动化研究所博士后.主要研究方向为图像检测与智能识别.E-mail:zcxdsg@163.com
  周仲凯 男,1994年3月出生于江苏省常州市.现为南昌航空大学测试与光电工程学院硕士研究生.主要研究方向为图像检测与智能识别.E-mail:jsczzzk123@163.com
- 基金资助:
- 国家自然科学基金 (No.61866026，No.61772255，No.61866025）; 江西省优势科技创新团队计划 (No.20165BCB19007）; 江西省科技创新杰出青年人才计划 (No.20192BCB23011）; 航空科学基金 (No.2018ZC56008）; 中国博士后科学基金 (No.2019M650894）; 江西省重点研发计划 (No.20171BBG70052）; 江西省研究生创新专项资金项目资助 (No.YC2018049）

Research Progress of Deep Learning Based Optical Flow Computation Technology

ZHANG Cong-xuan^1,2, ZHOU Zhong-kai¹, CHEN Zhen¹, GE Li-yue¹, LI Ming¹, JIANG Shao-feng¹, CHEN Hao¹

1. Key Laboratory of Nondestructive Testing, Ministry of Education, Nanchang Hangkong University, Nanchang, Jiangxi 330063, China;
2. Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China

Received:2019-08-26 Revised:2019-12-23 Online:2020-09-25 Published:2020-09-25
- Corresponding author:
- CHEN Zhen
- Supported by:
- National Natural Science Foundation of China (No.61866026, No.61772255, No.61866025); Superior Science and Technology Innovation Team Project of Jiangxi Province (No.20165BCB19007); Science and Technology Innovation Outstanding Young Talent Program of Jiangxi Province (No.20192BCB23011); Aeronautical Science Foundation of China, ASFC (No.2018ZC56008); China Postdoctoral Science Foundation (No.2019M650894); Key R&D Program of Jiangxi Province (No.20171BBG70052); Supported by Graduate Innovation Foundation of Jiangxi Province (No.YC2018049)

摘要/Abstract

摘要： 图像序列光流计算是图像处理与计算机视觉等领域的重要研究方向.随着深度学习技术的快速发展，以卷积神经网络为代表的深度学习理论与方法成为光流计算技术研究的热点.本文主要对深度学习光流计算技术研究进行综述，首先介绍了有监督学习、无监督学习和半监督学习的光流计算网络模型与训练策略，然后重点阐述并分析了不同网络模型优化方法.针对光流计算模型的评估问题，分别介绍了Middlebury、MPI-Sintel和KITTI等数据库及评价基准，并对不同类型深度学习和传统变分光流模型进行对比与分析.最后，总结了深度学习光流计算技术在模型复杂度与泛化性、光流估计鲁棒性、小样本训练准确性等方面的关键技术问题，并指出了可能的解决方案与研究思路.

关键词: 光流计算, 深度学习, 卷积神经网络, 训练策略, 优化方法, 评价基准

Abstract: Optical flow computation is an important research direction in image processing and computer vision. With the rapid development of the deep learning technology, the convolutional neural network based deep learning theories and methodologies have been the research focus of optical flow computation. This article mainly reviews the research progress of the deep learning based optical flow estimation technologies. First, the typical models and training strategies of the optical flow computing networks with supervised learning, unsupervised learning and semi-supervised learning are introduced. Second, the optimization methods of various network models are described and analyzed. Third, the evaluation benchmarks of Middlebury, MPI-Sintel and KITTI databases are summarized, and the experimental comparison results and analysis between the different deep-learning and variational optical flow methods are conducted. Finally, we discuss some issues of the deep learning based optical flow computation technology including the model complexity and generalization, the robustness of optical flow estimation and the accuracy of the small sample training. Afterwards, we point out several possible solutions and research ideas to address the above mentioned issues.

Key words: optical flow computation, deep learning, convolutional neural network, training strategy, optimization method, evaluation benchmark

中图分类号:

TP391

张聪炫, 周仲凯, 陈震, 等. 深度学习光流计算技术研究进展[J]. 电子学报, 2020, 48(9): 1841-1849.

ZHANG Cong-xuan, ZHOU Zhong-kai, CHEN Zhen, et al. Research Progress of Deep Learning Based Optical Flow Computation Technology[J]. Acta Electronica Sinica, 2020, 48(9): 1841-1849.

参考文献

[1] Horn B K P,Schunck B G.Determining optical flow[J].Artificial Intelligence,1980,17(1-3):185-203.
[2] 张聪炫,陈震,黎明.单目图像序列光流三维重建技术研究综述[J].电子学报,2016,44(12):3044-3052. ZHANG Cong-xuan,CHEN Zhen,LI Ming.Review of the 3D reconstruction technology based on optical flow of monocular image sequence[J].Acta Electronica Sinica,2016,44(12):3044-3052.(in Chinese).
[3] 陈震,张聪炫,晏文敬,吴燕平.基于图像局部结构的区域匹配变分光流算法[J].电子学报,2015,43(11):2200-2209. CHEN Zhen,ZHANG Cong-xuan,YAN Wen-jing,WU Yan-ping.Region matching variational optical flow algorithm based on image local structure[J].Acta Electronica Sinica,2015,43(11):2200-2209.(in Chinese).
[4] Bailer C,Taetz B,Stricker D.Flow fields:Dense correspondence fields for highly accurate large displacement optical flow estimation[A].International Conference on Computer Vision[C].Boston:IEEE,2015.4015-4023.
[5] Heeger D J.Optical flow using spatiotemporal filters[J].International Journal of Computer Vision,1988,1(4):279-302.
[6] Fleet D J,Jepson A D.Computation of component image velocity from local phase information[J].InternationalJournal of Computer Vision,1990,5(1):77-104.
[7] Zhang C X,Chen Z,Wang M R,Li M,Jiang S F.Robust non-local TV-L1 optical flow estimation with occlusion detection[J].IEEE Transactions on Image Processing,2017,26(8):4055-4067.
[8] 葛利跃,张聪炫,陈震,黎明,陈昊.相互结构引导滤波TV-L1变分光流估计[J].电子学报,2019,47(3):707-713. GE Li-yue,ZHANG Cong-xuan,CHEN Zhen,LI Ming,CHEN Hao.Mutual-structure guided filtering based TV-L1 optical flow estimation[J].Acta Electronica Sinica,2015,43(11):2200-2209.(in Chinese).
[9] 张聪炫,陈震,熊帆,黎明,葛利跃,陈昊.非刚性稠密匹配大位移运动光流估计[J].电子学报,2019,47(6):1316-1323. ZHANG Cong-xuan,CHEN Zhen,XIONG Fan,LI Ming,GE Li-yue,CHEN Hao.Large displacement motion optical flow estimation with non-rigid dense patch matching[J].Acta Electronica Sinica,2019,47(6):1316-1323.(in Chinese).
[10] Simonyan K,Zisserman A.Very deep convolutional networks for large-scale image recognition[A].International Conference on Learning Representations[C].San Diego:ICLR,2015.1-14.
[11] Szegedy C,Liu W,Jia Y,Sermanet P,Reed S,Anguelov D,Rabinovich A.Going deeper with convolutions[A].International Conference on Computer Vision and Pattern Recognition[C].Boston:IEEE,2015.1-9.
[12] He K,Zhang X,Ren S,Sun J.Deep residual learning for image recognition[A].International Conference on Computer Vision and Pattern Recognition[C].Las Vegas:IEEE,2016.770-778.
[13] Huang G,Liu Z,Van DerMaaten L,Weinberger K Q.Densely connected convolutional networks[A].International Conference on Computer Vision and Pattern Recognition[C].Hawaii:IEEE,2017.4700-4708.
[14] Jung S,Hwang S,Shin H,Shim D H.Perception,guidance,and navigation for indoor autonomous drone racing using deep learning[J].IEEE Robotics and Automation Letters,2018,3(3):2539-2544.
[15] Gomaa A,Abdelwahab M M,Abo-Zahhad M,Minematsu T,Taniguchi R I.Robust vehicle detection and counting algorithm employing a convolution neural network and optical flow[J].Sensors,2019,19(20):4588.
[16] Zhu Z,Wu W,Zou W,Yan J.End-to-end flow correlation tracking with spatial-temporal attention[A].International Conference on Computer Vision and Pattern Recognition[C].Salt Lake City:IEEE,2018.548-557.
[17] Djelouah A,Campos J,Schaub-Meyer S,Schroers C.Neural inter-frame compression for video coding[A].International Conference on Computer Vision[C].Long Beach:IEEE,2019.6421-6429.
[18] Dosovitskiy A,Fischer P,Ilg E,Hausser P,Hazirbas C,Golkov V,Brox T.Flownet:Learning optical flow with convolutional networks[A].International Conference on Computer Vision and Pattern Recognition[C].Boston:IEEE,2015.2758-2766.
[19] Ronneberger O,Fischer P,Brox T.U-net:Convolutional networks for biomedical image segmentation[A].International Conference on Medical Image Computing and Computer-Assisted Intervention[C].Munich:Springer,2015.234-241.
[20] Ilg E,Mayer N,Saikia T,Dosovitskiy A,Brox T.FlowNet 2.0:Evolution of optical flow estimation with deep networks[A].International Conference on Computer Vision and Pattern Recognition[C].Hawaii:IEEE,2017.1647-1655.
[21] Ranjan A,Black M J.Optical flow estimation using a spatial pyramid network[A].International Conference on Computer Vision and Pattern Recognition[C].Hawaii:IEEE,2017.4161-4170.
[22] Sun D,Yang X,Liu M Y,Kautz J.PWC-Net:CNNs for optical flow using pyramid,warping,and cost volume[A].International Conference on Computer Vision and Pattern Recognition[C].Salt Lake City:IEEE,2018.8934-8943.
[23] Ilg E,Saikia T,Keuper M,Brox T.Occlusions,motion and depth boundaries with a generic network for disparity,optical flow or scene flow estimation[A].European Conference on Computer Vision[C].Munich:Springer,2018.614-630.
[24] Hui T W,Tang X,Loy CC.LiteFlowNet:A lightweight convolutional neural network for optical flow estimation[A].International Conference on Computer Vision and Pattern Recognition[C].Salt Lake City:IEEE,2018.8981-8989.
[25] Sevilla-Lara L,Sun D,Jampani V,Black M J.Optical flow with semantic segmentation and localized layers[A].International Conference on Computer Vision and Pattern Recognition[C].Las Vegas:IEEE,2016.3889-3898.
[26] Cheng J,Tsai Y H,Wang S,Yang M H.Segflow:Joint learning for video object segmentation and optical flow[A].International Conference on Computer Vision[C].Venice:IEEE,2017.686-695.
[27] Kingma D P,Ba J.Adam:A Method for Stochastic Optimization[EB/OL].http://arxiv.org/abs/1412.6980,2014.
[28] Jason J Y,Harley A W,Derpanis K G.Back to basics:Unsupervised learning of optical flow via brightness constancy and motion smoothness[A].European Conference on Computer Vision[C].Amsterdam:Springer,2016.3-10.
[29] Meister,Simon,Junhwa Hur,Stefan Roth.UnFlow:Unsupervised learning of optical flow with a bidirectional census loss[A].Thirty-Second AAAI Conference on Artificial Intelligence[C].Hawaii:Springer,2018.682-691.
[30] Wang Y,Yang Y,Yang Z,Zhao L,Wang P,Xu W.Occlusion aware unsupervised learning of optical flow[A].International Conference on Computer Vision and Pattern Recognition[C].Salt Lake City:IEEE,2018.4884-4893.
[31] Janai J,Guney F,Ranjan A,Black M,Geiger A.Unsupervised learning of multi-frame optical flow with occlusions[A].European Conference on Computer Vision[C].Munich:Springer,2018.690-706.
[32] Zhu A Z,Yuan L,Chaney K,Daniilidis K.Unsupervised event-based learning of optical flow,depth,and egomotion[A].International Conference on Computer Vision and Pattern Recognition[C].Long Beach:IEEE,2019.989-997.
[33] Paredes-Vallés F,Scheper K Y W,De Croon G C H E.Unsupervised learning of a hierarchical spiking neural network for optical flow estimation:From events to global motion perception[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2020,2(8):2051-2064.
[34] Shedligeri P,Mitra K.Live Demonstration:Joint estimation of optical flow and intensity image from event sensors[A].International Conference on Computer Vision and Pattern Recognition[C].Long Beach:IEEE,2019.1-2.
[35] Rasmus A,Berglund M,Honkala M,Valpola H,Raiko T.Semi-supervised learning with ladder networks[A].Advances in Neural Information Processing Systems[C].Montréal:MIT,2015.3546-3554.
[36] Zhu Y,Lan Z,Newsam S,Hauptmann A G.Guided optical flow learning[A].International Conference on Computer Vision and Pattern Recognition[C].Hawaii:IEEE,2017.1-5.
[37] Yang G,Deng Z,Wang S,Li Z.Masked label learning for optical flow regression[A].International Conference on Pattern Recognition[C].Beijing:IEEE,2018.1139-1144.
[38] Lai W S,Huang J B,Yang M H.Semi-supervised learning for optical flow with generative adversarial networks[A].Advances in Neural Information Processing Systems[C].Long Beach:MIT,2017.354-364.
[39] Weinzaepfel P,Revaud J,Harchaoui Z,Schmid C.Deepflow:large displacement optical flow with deep matching[A].International Conference on Computer Vision and Pattern Recognition[C].Portland:IEEE,2013.1385-1392.
[40] Hosni A,Rhemann C,Bleyer M,Rother C,Gelautz M.Fast cost-volume filtering for visual correspondence and beyond[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2013,35(2):504-511.
[41] Lin T Y,Dollár P,Girshick R,He K,Hariharan B,Belongie S.Feature pyramid networks for object detection[A].International Conference on Computer Vision and Pattern Recognition[C].Hawaii:IEEE,2017.2117-2125.
[42] Sun D,Roth S,Black M J.A quantitative analysis of current practices in optical flow estimation and the principles behind them[J].International Journal of Computer Vision,2014,106(2):115-137.
[43] Xu L,Jia J,Matsushita Y.Motion detail preserving optical flow estimation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2011,34(9):1744-1757.
[44] Zhang C,Ge L,Chen Z,et al.Refined TV-L1optical flow estimation using joint filtering[J].IEEE Transactions on Multimedia,2019.
[45] Revaud J,Weinzaepfel P,Harchaoui Z,Schmid C.Epicflow:edge-preserving interpolation of correspondences for optical flow[A].International Conference on Computer Vision and Pattern Recognition[C].Boston:IEEE,2015.1164-1172.
[46] Yu F,Koltun V.Multi-scale context aggregation by dilated convolutions[EB/OL].http://arxiv.org/abs/1511.07122V2,2015.
[47] Lai H Y,Tsai Y H,Chiu W C.Bridging stereo matching and optical flow via spatiotemporal correspondence[A].International Conference on Computer Vision and Pattern Recognition[C].Long Beach:IEEE,2019.1890-1899.
[48] Ranjan A,Jampani V,Balles L,Kim K,Sun D,Wulff J,Black M J.Competitive collaboration:joint unsupervised learning of depth,camera motion,optical flow and motion segmentation[A].International Conference on Computer Vision and Pattern Recognition[C].Long Beach:IEEE,2019.12240-12249.
[49] Yin Z,Shi J.Geonet:Unsupervised learning of dense depth,optical flow and camera pose[A].International Conference on Computer Vision and Pattern Recognition[C].Salt Lake City:IEEE,2018.1983-1992.
[50] Baker S,Scharstein D,Lewis J P,Roth S,Black M J,Szeliski R.A database and evaluation methodology for optical flow[J].International Journal of Computer Vision,2011,92(1):1-31.
[51] Butler D J,Wulff J,Stanley G B,Black M J.A naturalistic open source movie for optical flow evaluation[A].European Conference on Computer Vision[C].Florence:Springer,2012.611-625.
[52] Geiger A,Lenz P,Stiller C,Urtasun R.Vision meets robotics:The KITTI dataset[J].International Journal of Robotics Research,2013,32(11):1231-1237.
[53] Aubry M,Maturana D,Efros A A,Russell B C,Sivic J.Seeing 3D chairs:Exemplar part-based 2d-3d alignment using a large dataset of cad models[A].International Conference on Computer Vision and Pattern Recognition[C].Columbus:IEEE,2014.3762-3769.
[54] Mayer N,Ilg E,Häusser P,Cremers D,Dosovitskiy A,Brox T.A large dataset to train convolutional networks for disparity,optical flow,and scene flow estimation[A].International Conference on Computer Vision and Pattern Recognition[C].Las Vegas:IEEE,2016.4040-4048.
[55] Fleet D J,Jepson A D.Computation of component image velocity from local phase information[J].International Journal of Computer Vision,1990,5(1):77-104.
[56] Ilg E,Cicek O,Galesso S,Klein A,Makansi O,Hutter F,Brox T.Uncertainty estimates and multi-hypotheses networks for optical flow[A].European Conference on Computer Vision[C].Munich:Springer,2018.652-667.

深度学习光流计算技术研究进展

Research Progress of Deep Learning Based Optical Flow Computation Technology

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	李钦, 刘伟, 牛朝阳, 宝音图, 惠周勃. 低信噪比下基于分裂EfficientNet网络的雷达信号调制方式识别[J]. 电子学报, 2023, 51(3): 675-686.
[2]	黄赟, 张帆, 郭威, 陈立, 羊光. 一种基于数据标准差的卷积神经网络量化方法[J]. 电子学报, 2023, 51(3): 639-647.
[3]	范兵兵, 何庭建, 张聪炫, 陈震, 黎明. 联合遮挡约束与残差补偿的特征金字塔光流计算方法[J]. 电子学报, 2023, 51(3): 648-657.
[4]	吕杭, 蒋明峰, 李杨, 张鞠成, 王志康. 基于混合时频域特征的卷积神经网络心律失常分类方法的研究[J]. 电子学报, 2023, 51(3): 701-711.
[5]	张聿远, 闫文君, 张立民. 基于多模态特征融合网络的空时分组码识别算法[J]. 电子学报, 2023, 51(2): 489-498.
[6]	许新征, 李杉. 基于特征膨胀卷积模块的轻量化技术研究[J]. 电子学报, 2023, 51(2): 355-364.
[7]	丁琪, 田萱, 孙国栋. 基于注意力增强的热点感知新闻推荐模型[J]. 电子学报, 2023, 51(1): 93-104.
[8]	李滔, 董秀成, 林宏伟. 基于深监督跨尺度注意力网络的深度图像超分辨率重建[J]. 电子学报, 2023, 51(1): 128-138.
[9]	郭晓轩, 冯其波, 冀振燕, 郑发家, 杨燕燕. 多线激光光条图像缺陷分割模型研究[J]. 电子学报, 2023, 51(1): 172-179.
[10]	贾童瑶, 卓力, 李嘉锋, 张菁. 基于深度学习的单幅图像去雾研究进展[J]. 电子学报, 2023, 51(1): 231-245.
[11]	何滢婕, 刘月峰, 边浩东, 郭威, 张小燕. 基于Informer的电池荷电状态估算及其稀疏优化方法[J]. 电子学报, 2023, 51(1): 50-56.
[12]	张永梅, 孙捷. 基于动静态特征双输入神经网络的咳嗽声诊断COVID-19算法[J]. 电子学报, 2023, 51(1): 202-212.
[13]	袁海英, 成君鹏, 曾智勇, 武延瑞. Mobile_BLNet：基于Big-Little Net的轻量级卷积神经网络优化设计[J]. 电子学报, 2023, 51(1): 180-191.
[14]	王神龙, 雍宇, 吴晨睿. 基于伪孪生神经网络的低纹理工业零件6D位姿估计[J]. 电子学报, 2023, 51(1): 192-201.
[15]	吴靖, 叶晓晶, 黄峰, 陈丽琼, 王志锋, 刘文犀. 基于深度学习的单帧图像超分辨率重建综述[J]. 电子学报, 2022, 50(9): 2265-2294.