联合遮挡约束与残差补偿的特征金字塔光流计算方法

范兵兵; 何庭建; 张聪炫; 陈震; 黎明

doi:10.12263/DZXB.20211052

您当前的位置：

首页 >

文章列表页 >

联合遮挡约束与残差补偿的特征金字塔光流计算方法

学术论文 | 更新时间：2025-12-08

- 联合遮挡约束与残差补偿的特征金字塔光流计算方法
- Feature Pyramid Optical Flow Estimation Method Jointing Occlusion Constraint and Residual Compensation
- 电子学报 2023年51卷第3期页码：648-657
- 作者机构：
  
  1.南昌航空大学无损检测技术教育部重点实验室, 江西南昌 330063
  2.中国科学院自动化研究所, 北京 100190
- 作者简介：
  
  [ "范兵兵男，1997年3月出生于江西省瑞昌市. 现为南昌航空大学测试与光电工程学院硕士研究生. 主要研究方向为图像检测与智能识别. E-mail: 2652459616@qq.com" ]
  [ "何庭建男，1995年4月出生于江西省抚州市， 2021年6月南昌航空大学信息工程学院硕士研究生毕业，主要研究方向为图像检测与智能识别. E-mail: 511076738@qq.com" ]
  [ "张聪炫（通讯作者）男，1984年7月出生于河南省焦作市. 2014年在南京航空航天大学获博士学位. 现为南昌航空大学教授，硕士生导师. 主要研究方向为图像检测与智能识别. 中国电子学会会员编号：E190014621M." ]
  [ "陈震男，1969年11月出生于江西省九江市. 2003年在西北工业大学获得博士学位. 现为南昌航空大学教授，博士生导师. 主要研究方向为计算机视觉、图像处理与模式识别. E-mail: dr_chenzhen@163.com" ]
  [ "黎明男，1965年2月出生于江西省樟树市. 1997年获南京航空航天大学博士学位. 现为南昌航空大学教授，博士生导师. 主要研究方向为图像处理与模式识别、智能计算.E-mail: limingniat@hotmail.com" ]
- 基金信息：
  
  国家自然科学基金(61866026;61866025);江西省优势科技创新团队计划(20165BCB19007);江西省技术创新引导类计划项目(20212AEI91005);江西省自然科学基金重点项目(20202ACB214007);航空科学基金(2018ZC56008);中国博士后科学基金(2019M650894);江西省教育厅科学技术研究项目(GJJ210910);江西省图像处理与模式识别重点实验室开放基金资助(ET202104413)
- DOI：10.12263/DZXB.20211052
  中图分类号： TP391
- 收稿：2021-08-05，
  
  修回：2021-12-29，
  
  纸质出版：2023-03-25
- 稿件说明：
移动端阅览
范兵兵,何庭建,张聪炫等.联合遮挡约束与残差补偿的特征金字塔光流计算方法[J].电子学报,2023,51(03):648-657.

FAN Bing-bing,HE Ting-jian,ZHANG Cong-xuan,et al.Feature Pyramid Optical Flow Estimation Method Jointing Occlusion Constraint and Residual Compensation[J].ACTA ELECTRONICA SINICA,2023,51(03):648-657.
范兵兵,何庭建,张聪炫等.联合遮挡约束与残差补偿的特征金字塔光流计算方法[J].电子学报,2023,51(03):648-657. DOI： 10.12263/DZXB.20211052.

FAN Bing-bing,HE Ting-jian,ZHANG Cong-xuan,et al.Feature Pyramid Optical Flow Estimation Method Jointing Occlusion Constraint and Residual Compensation[J].ACTA ELECTRONICA SINICA,2023,51(03):648-657. DOI： 10.12263/DZXB.20211052.

摘要

针对现有深度学习光流计算模型在运动遮挡和大位移等场景下光流计算的准确性与鲁棒性问题，本文提出一种联合遮挡约束与残差补偿的特征金字塔光流计算方法.首先，构造基于遮挡掩模的光流约束模块，通过预测遮挡掩模特征图抑制变形特征的边缘伪影，克服运动遮挡区域的图像边缘模糊问题.然后，采用特征图变形策略构建基于特征变形的光流残差补偿模块，利用该模块学习到的残差光流细化原始光流场，改善大位移运动区域的光流计算效果.最后，采用特征金字塔框架构建联合遮挡约束与残差补偿的光流计算网络模型，提升大位移和运动遮挡场景下的光流计算精度.分别采用MPI-Sintel （Max-Planck Institute and Sintel）和KITTI （Karlsruhe Institute of Technology and Toyota Technological Institute）数据集对本文方法和代表性传统光流计算方法、深度学习光流计算方法进行综合对比分析，实验结果表明本文方法相对于其他方法能够有效提升大位移和运动遮挡场景下的光流计算精度与鲁棒性.

Abstract

To improve the accuracy and robustness of the deep-learning based optical flow estimation under motion occlusions and large displacements

we propose a feature pyramid optical flow computation method by jointing the occlusion constraint with residual compensation. First

an optical flow constraint module is designed based on the learning occlusion mask. The proposed constraint module predicts the occlusion feature map to restrain the edge artifacts of the warping features

which is able to overcome the issue of edge blurring in the motion occlusion areas. Second

a residual compensation module is constructed by using the feature map warping strategy

and the residual optical flows learned from the presented module are employed to refine the original flow fields. Third

the proposed occlusion constraint model and residual compensation module are incorporated into a feature pyramid framework to construct an optical flow estimation network. Finally

the MPI-Sintel (Max-Planck Institute and Sintel) and KITTI (Karlsruhe Institute of Technology and Toyota Technological Institute) datasets are employed to conduct a comprehensive comparison between the proposed method and the representative traditional optical flow methods

deep-learning optical flow methods. The experimental results demonstrate that the presented method significantly improves the accuracy and robustness of optical flow estimation under large displacements and motionocclusions.

关键词

Keywords

references

吴晓军 , 鞠光亮 . 一种无标记点人脸表情捕捉与重现算法 [J]. 电子学报 , 2016 , 44 ( 9 ): 2141 - 2147 .

WU X J , JU G L . A markerless facial expression capture and reproduce algorithm [J]. Acta Electronica Sinica , 2016 , 44 ( 9 ): 2141 - 2147 . (in Chinese)

JI Y L , YANG Y , SHEN F M , et al . Arbitrary-view human action recognition: A varying-view RGB-D action dataset [J]. IEEE Transactions on Circuits and Systems for Video Technology , 2021 , 31 ( 1 ): 289 - 300 .

丁新尧 , 张鑫 . 基于显著性特征的选择性目标跟踪算法 [J]. 电子学报 , 2020 , 48 ( 1 ): 118 - 123 .

DING X Y , ZHANG X . Visual tracking with salient features and selective mechanism [J]. Acta Electronica Sinica , 2020 , 48 ( 1 ): 118 - 123 . (in Chinese)

张聪炫 , 陈震 , 黎明 . 单目图像序列光流三维重建技术研究综述 [J]. 电子学报 , 2016 , 44 ( 12 ): 3044 - 3052 .

ZHANG C X , CHEN Z , LI M . Review of the 3D reconstruction technology based on optical flow of monocular image sequence [J]. Acta Electronica Sinica , 2016 , 44 ( 12 ): 3044 - 3052 . (in Chinese)

HORN B K P , SCHUNCK B G . Determining optical flow [J]. Artificial Intelligence , 1981 , 17 ( 1/2/3 ): 185 - 203 .

MEI L , LAI J H , XIE X H , et al . Illumination-invariance optical flow estimation using weighted regularization transform [J]. IEEE Transactions on Circuits and Systems for Video Technology , 2020 , 30 ( 2 ): 495 - 508 .

ZHANG C X , GE L Y , CHEN Z , et al . Refined TV-L1 optical flow estimation using joint filtering [J]. IEEE Transactions on Multimedia , 2020 , 22 ( 2 ): 349 - 364 .

SUN D Q , ROTH S , BLACK M J . A quantitative analysis of current practices in optical flow estimation and the principles behind them [J]. International Journal of Computer Vision , 2014 , 106 ( 2 ): 115 - 137 .

ZHANG C X , CHEN Z , WANG M R , et al . Robust non-local TV- L1 optical flow estimation with occlusion detection [J]. IEEE Transactions on Image Processing , 2017 , 26 ( 8 ): 4055 - 4067 .

ZIMMER H , BRUHN A , WEICKERT J . Optic flow in harmony [J]. International Journal of Computer Vision , 2011 , 93 ( 3 ): 368 - 388 .

SEVILLA-LARA L , SUN D Q , JAMPANI V , et al . Optical flow with semantic segmentation and localized layers [C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2016 : 3889 - 3898 .

张聪炫 , 周仲凯 , 陈震 , 等 . 深度学习光流计算技术研究进展 [J]. 电子学报 , 2020 , 48 ( 9 ): 1841 - 1849 .

ZHANG C X , ZHOU Z K , CHEN Z , et al . Research progress of deep learning based optical flow computation technology [J]. Acta Electronica Sinica , 2020 , 48 ( 9 ): 1841 - 1849 . (in Chinese)

DOSOVITSKIY A , FISCHER P , ILG E , et al . FlowNet: learning optical flow with convolutional networks [C]// 2015 IEEE International Conference on Computer Vision . Boston : IEEE , 2015 : 2758 - 2766 .

ILG E , MAYER N , SAIKIA T , et al . FlowNet 2.0: evolution of optical flow estimation with deep networks [C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition . Hawaii : IEEE , 2017 : 1647 - 1655 .

RANJAN A , BLACK M J . Optical flow estimation using a spatial pyramid network [C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu : IEEE , 2017 : 2720 - 2729 .

SUN D Q , YANG X D , LIU M Y , et al . PWC-net: CNNs for optical flow using pyramid, warping, and cost volume [C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2018 : 8934 - 8943 .

HUI T W , TANG X O , LOY C C . LiteFlowNet: A lightweight convolutional neural network for optical flow estimation [C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2018 : 8981 - 8989 .

ZHAO S Y , SHENG Y L , DONG Y , et al . MaskFlownet: asymmetric feature matching with learnable occlusion mask [C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2020 : 6277 - 6286 .

LU Y , VALMADRE J , WANG H , et al . Devon: Deformable volume network for learning optical flow [C]// 2020 IEEE Winter Conference on Applications of Computer Vision . Snowmass : IEEE Press , 2020 . 2705 - 2713 .

YANG G , RAMANAN D . Volumetric correspondence networks for optical flow [C]// 33rd International Conference on Neural Information Processing Systems . Vancouver : Curran Associates Inc , 2019 : 794 - 805 .

YU J J , HARLEY A W , DERPANIS K G . Back to basics: Unsupervised learning of optical flow via brightness constancy and motion smoothness [C]// European Conference on Computer Vision . Cham : Springer , 2016 : 3 - 10 .

MEISTER S , HUR J , ROTH S . UnFlow: Unsupervised learning of optical flow with a bidirectional census loss [C]// AAAI Conference on Artificial Intelligence . New Orleans : AAAI Press , 2018 : 7251 - 7259 .

LIU P P , KING I , LYU M R , et al . DDFlow: learning optical flow with unlabeled data distillation [J]. Proceedings of the AAAI Conference on Artificial Intelligence. Hawaii: AAAI Press , 2019 , 33 : 8770 - 8777 .

LAI W S , HUANG J B , YANG M H . Semi-supervised learning for optical flow with Generative Adversarial Networks [C]// 31st International Conference on Neural Information Processing Systems . Long Beach : Curran Associates Inc , 2017 : 353 - 363 .

SONG X L , ZHAO Y Y , YANG J Y , et al . FPCR-net: Feature pyramidal correlation and residual reconstruction for optical flow estimation [EB/OL]. ( 2020-01-17 ). https://arxiv.org/abs/2001.06171 https://arxiv.org/abs/2001.06171 .

KINGMA D , BA J . Adam: A method for stochastic optimization [C]// International Conference for Learning Representations . San Diego : Elsevier Press , 2015 : 1 - 15 .

MAYER N , ILG E , HÄUSSER P , et al . A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation [C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2016 : 4040 - 4048 .

BUTLER D J , WULFF J , STANLEY G B , et al . A naturalistic open source movie for optical flow evaluation [C]// European Conference on Computer Vision . Berlin, Heidelberg : Springer , 2012 : 611 - 625 .

HU Y L , SONG R , LI Y S . Efficient coarse-to-fine patch match for large displacement optical flow [C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2016 : 5704 - 5712 .

REVAUD J , WEINZAEPFEL P , HARCHAOUI Z , et al . EpicFlow: Edge-preserving interpolation of correspondences for optical flow [C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2015 : 1164 - 1172 .

MENZE M , GEIGER A . Object scene flow for autonomous vehicles [C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2015 : 3061 - 3070 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

联合深度可分离残差与多尺度双通道注意力的全局匹配优化光流估计方法

基于光流场分割和Canny边缘提取融合算法的运动目标检测

单目图像序列光流三维重建技术研究综述