Feature Pyramid Optical Flow Estimation Method Jointing Occlusion Constraint and Residual Compensation

FAN Bing-bing; HE Ting-jian; ZHANG Cong-xuan; CHEN Zhen; LI Ming

doi:10.12263/DZXB.20211052

您当前的位置：

首页 >

文章列表页 >

Feature Pyramid Optical Flow Estimation Method Jointing Occlusion Constraint and Residual Compensation

PAPERS | 更新时间：2025-12-08

- Feature Pyramid Optical Flow Estimation Method Jointing Occlusion Constraint and Residual Compensation
- ACTA ELECTRONICA SINICA Vol. 51, Issue 3, Pages: 648-657(2023)
- 作者机构：
  
  1.南昌航空大学无损检测技术教育部重点实验室, 江西南昌 330063
  2.中国科学院自动化研究所, 北京 100190
- 作者简介：
- 基金信息：
- DOI：10.12263/DZXB.20211052
  CLC： TP391
- Received：05 August 2021，
  
  Revised：2021-12-29，
  
  Published：25 March 2023
- 稿件说明：
移动端阅览
范兵兵,何庭建,张聪炫等.联合遮挡约束与残差补偿的特征金字塔光流计算方法[J].电子学报,2023,51(03):648-657.

FAN Bing-bing,HE Ting-jian,ZHANG Cong-xuan,et al.Feature Pyramid Optical Flow Estimation Method Jointing Occlusion Constraint and Residual Compensation[J].ACTA ELECTRONICA SINICA,2023,51(03):648-657.
范兵兵,何庭建,张聪炫等.联合遮挡约束与残差补偿的特征金字塔光流计算方法[J].电子学报,2023,51(03):648-657. DOI： 10.12263/DZXB.20211052.

FAN Bing-bing,HE Ting-jian,ZHANG Cong-xuan,et al.Feature Pyramid Optical Flow Estimation Method Jointing Occlusion Constraint and Residual Compensation[J].ACTA ELECTRONICA SINICA,2023,51(03):648-657. DOI： 10.12263/DZXB.20211052.

摘要

针对现有深度学习光流计算模型在运动遮挡和大位移等场景下光流计算的准确性与鲁棒性问题，本文提出一种联合遮挡约束与残差补偿的特征金字塔光流计算方法.首先，构造基于遮挡掩模的光流约束模块，通过预测遮挡掩模特征图抑制变形特征的边缘伪影，克服运动遮挡区域的图像边缘模糊问题.然后，采用特征图变形策略构建基于特征变形的光流残差补偿模块，利用该模块学习到的残差光流细化原始光流场，改善大位移运动区域的光流计算效果.最后，采用特征金字塔框架构建联合遮挡约束与残差补偿的光流计算网络模型，提升大位移和运动遮挡场景下的光流计算精度.分别采用MPI-Sintel （Max-Planck Institute and Sintel）和KITTI （Karlsruhe Institute of Technology and Toyota Technological Institute）数据集对本文方法和代表性传统光流计算方法、深度学习光流计算方法进行综合对比分析，实验结果表明本文方法相对于其他方法能够有效提升大位移和运动遮挡场景下的光流计算精度与鲁棒性.

Abstract

To improve the accuracy and robustness of the deep-learning based optical flow estimation under motion occlusions and large displacements

we propose a feature pyramid optical flow computation method by jointing the occlusion constraint with residual compensation. First

an optical flow constraint module is designed based on the learning occlusion mask. The proposed constraint module predicts the occlusion feature map to restrain the edge artifacts of the warping features

which is able to overcome the issue of edge blurring in the motion occlusion areas. Second

a residual compensation module is constructed by using the feature map warping strategy

and the residual optical flows learned from the presented module are employed to refine the original flow fields. Third

the proposed occlusion constraint model and residual compensation module are incorporated into a feature pyramid framework to construct an optical flow estimation network. Finally

the MPI-Sintel (Max-Planck Institute and Sintel) and KITTI (Karlsruhe Institute of Technology and Toyota Technological Institute) datasets are employed to conduct a comprehensive comparison between the proposed method and the representative traditional optical flow methods

deep-learning optical flow methods. The experimental results demonstrate that the presented method significantly improves the accuracy and robustness of optical flow estimation under large displacements and motionocclusions.

关键词

Keywords

references

吴晓军 , 鞠光亮 . 一种无标记点人脸表情捕捉与重现算法 [J]. 电子学报 , 2016 , 44 ( 9 ): 2141 - 2147 .

WU X J , JU G L . A markerless facial expression capture and reproduce algorithm [J]. Acta Electronica Sinica , 2016 , 44 ( 9 ): 2141 - 2147 . (in Chinese)

JI Y L , YANG Y , SHEN F M , et al . Arbitrary-view human action recognition: A varying-view RGB-D action dataset [J]. IEEE Transactions on Circuits and Systems for Video Technology , 2021 , 31 ( 1 ): 289 - 300 .

丁新尧 , 张鑫 . 基于显著性特征的选择性目标跟踪算法 [J]. 电子学报 , 2020 , 48 ( 1 ): 118 - 123 .

DING X Y , ZHANG X . Visual tracking with salient features and selective mechanism [J]. Acta Electronica Sinica , 2020 , 48 ( 1 ): 118 - 123 . (in Chinese)

张聪炫 , 陈震 , 黎明 . 单目图像序列光流三维重建技术研究综述 [J]. 电子学报 , 2016 , 44 ( 12 ): 3044 - 3052 .

ZHANG C X , CHEN Z , LI M . Review of the 3D reconstruction technology based on optical flow of monocular image sequence [J]. Acta Electronica Sinica , 2016 , 44 ( 12 ): 3044 - 3052 . (in Chinese)

HORN B K P , SCHUNCK B G . Determining optical flow [J]. Artificial Intelligence , 1981 , 17 ( 1/2/3 ): 185 - 203 .

MEI L , LAI J H , XIE X H , et al . Illumination-invariance optical flow estimation using weighted regularization transform [J]. IEEE Transactions on Circuits and Systems for Video Technology , 2020 , 30 ( 2 ): 495 - 508 .

ZHANG C X , GE L Y , CHEN Z , et al . Refined TV-L1 optical flow estimation using joint filtering [J]. IEEE Transactions on Multimedia , 2020 , 22 ( 2 ): 349 - 364 .

SUN D Q , ROTH S , BLACK M J . A quantitative analysis of current practices in optical flow estimation and the principles behind them [J]. International Journal of Computer Vision , 2014 , 106 ( 2 ): 115 - 137 .

ZHANG C X , CHEN Z , WANG M R , et al . Robust non-local TV- L1 optical flow estimation with occlusion detection [J]. IEEE Transactions on Image Processing , 2017 , 26 ( 8 ): 4055 - 4067 .

ZIMMER H , BRUHN A , WEICKERT J . Optic flow in harmony [J]. International Journal of Computer Vision , 2011 , 93 ( 3 ): 368 - 388 .

SEVILLA-LARA L , SUN D Q , JAMPANI V , et al . Optical flow with semantic segmentation and localized layers [C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2016 : 3889 - 3898 .

张聪炫 , 周仲凯 , 陈震 , 等 . 深度学习光流计算技术研究进展 [J]. 电子学报 , 2020 , 48 ( 9 ): 1841 - 1849 .

ZHANG C X , ZHOU Z K , CHEN Z , et al . Research progress of deep learning based optical flow computation technology [J]. Acta Electronica Sinica , 2020 , 48 ( 9 ): 1841 - 1849 . (in Chinese)

DOSOVITSKIY A , FISCHER P , ILG E , et al . FlowNet: learning optical flow with convolutional networks [C]// 2015 IEEE International Conference on Computer Vision . Boston : IEEE , 2015 : 2758 - 2766 .

ILG E , MAYER N , SAIKIA T , et al . FlowNet 2.0: evolution of optical flow estimation with deep networks [C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition . Hawaii : IEEE , 2017 : 1647 - 1655 .

RANJAN A , BLACK M J . Optical flow estimation using a spatial pyramid network [C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu : IEEE , 2017 : 2720 - 2729 .

SUN D Q , YANG X D , LIU M Y , et al . PWC-net: CNNs for optical flow using pyramid, warping, and cost volume [C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2018 : 8934 - 8943 .

HUI T W , TANG X O , LOY C C . LiteFlowNet: A lightweight convolutional neural network for optical flow estimation [C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2018 : 8981 - 8989 .

ZHAO S Y , SHENG Y L , DONG Y , et al . MaskFlownet: asymmetric feature matching with learnable occlusion mask [C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2020 : 6277 - 6286 .

LU Y , VALMADRE J , WANG H , et al . Devon: Deformable volume network for learning optical flow [C]// 2020 IEEE Winter Conference on Applications of Computer Vision . Snowmass : IEEE Press , 2020 . 2705 - 2713 .

YANG G , RAMANAN D . Volumetric correspondence networks for optical flow [C]// 33rd International Conference on Neural Information Processing Systems . Vancouver : Curran Associates Inc , 2019 : 794 - 805 .

YU J J , HARLEY A W , DERPANIS K G . Back to basics: Unsupervised learning of optical flow via brightness constancy and motion smoothness [C]// European Conference on Computer Vision . Cham : Springer , 2016 : 3 - 10 .

MEISTER S , HUR J , ROTH S . UnFlow: Unsupervised learning of optical flow with a bidirectional census loss [C]// AAAI Conference on Artificial Intelligence . New Orleans : AAAI Press , 2018 : 7251 - 7259 .

LIU P P , KING I , LYU M R , et al . DDFlow: learning optical flow with unlabeled data distillation [J]. Proceedings of the AAAI Conference on Artificial Intelligence. Hawaii: AAAI Press , 2019 , 33 : 8770 - 8777 .

LAI W S , HUANG J B , YANG M H . Semi-supervised learning for optical flow with Generative Adversarial Networks [C]// 31st International Conference on Neural Information Processing Systems . Long Beach : Curran Associates Inc , 2017 : 353 - 363 .

SONG X L , ZHAO Y Y , YANG J Y , et al . FPCR-net: Feature pyramidal correlation and residual reconstruction for optical flow estimation [EB/OL]. ( 2020-01-17 ). https://arxiv.org/abs/2001.06171 https://arxiv.org/abs/2001.06171 .

KINGMA D , BA J . Adam: A method for stochastic optimization [C]// International Conference for Learning Representations . San Diego : Elsevier Press , 2015 : 1 - 15 .

MAYER N , ILG E , HÄUSSER P , et al . A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation [C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2016 : 4040 - 4048 .

BUTLER D J , WULFF J , STANLEY G B , et al . A naturalistic open source movie for optical flow evaluation [C]// European Conference on Computer Vision . Berlin, Heidelberg : Springer , 2012 : 611 - 625 .

HU Y L , SONG R , LI Y S . Efficient coarse-to-fine patch match for large displacement optical flow [C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2016 : 5704 - 5712 .

REVAUD J , WEINZAEPFEL P , HARCHAOUI Z , et al . EpicFlow: Edge-preserving interpolation of correspondences for optical flow [C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2015 : 1164 - 1172 .

MENZE M , GEIGER A . Object scene flow for autonomous vehicles [C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2015 : 3061 - 3070 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

A Global Matching Optimization Approach for Optical Flow Estimation Using Joint Depth-Separable Residual Blocks and Multi-Scale Dual-Channel Attention

Detection of Moving Object Using a Fusion Method Based on Segmentation of Optical Flow Field and Edge Extracted by Canny's Operator

Review of the 3D Reconstruction Technology Based on Optical Flow of Monocular Image Sequence

Related Author

WANG Zi-ge

WANG Zi-xu

CHEN Zhen

ZHANG Cong-xuan

GE Li-yue

CHEN Hong-ye

XIE Wei-xin

PEI Ji-hong

Related Institution

School of Instrumentation and Optoelectronic Engineering, Beihang University

School of Information Engineering, Nanchang Hangkong University

School of Computer Science, Northwestern Polytechnical University

School of Instrument Science and Optoelectronic Engineering, Nanchang Hangkong University

School of Electronic Engineering, Xidian University

⁰