基于空间结构化推理深度融合网络的RGB-D场景解析

doi:10.3969/j.issn.0372-2112.2018.05.035

电子学报 ›› 2018, Vol. 46 ›› Issue (5): 1253-1258.DOI: 10.3969/j.issn.0372-2112.2018.05.035

基于空间结构化推理深度融合网络的RGB-D场景解析

王泽宇¹, 吴艳霞¹, 张国印¹, 布树辉²

1. 哈尔滨工程大学计算机科学与技术学院, 黑龙江哈尔滨 150001;
2. 西北工业大学航空学院, 陕西西安 710072

收稿日期:2017-05-19 修回日期:2018-01-10 出版日期:2018-05-25
- 通讯作者:
- 吴艳霞
- 作者简介:
- 王泽宇 男,1989年3月出生于河南省郑州市,哈尔滨工程大学计算机科学与技术学院,博士研究生,主要研究方向为机器学习、计算机视觉.E-mail:wangzeyu2011@hrbeu.edu.cn;张国印 男,1962年9月出生于黑龙江省齐齐哈尔市,哈尔滨工程大学计算机科学与技术学院,教授,博士生导师,主要研究方向为机器学习、计算机视觉.E-mail:zhangguoyin@hrbeu.edu.cn;布树辉 男,1978年11月出生于河南省洛阳市,西北工业大学航空学院,教授,博士生导师,主要研究方向为机器学习、计算机视觉.E-mail:bushuhui@nwpu.edu.cn
- 基金资助:
- 国家重点研发计划 (No.2016YFB1000400）; 哈尔滨市杰出青年人才基金 (No.2017RAYXJ016）; 中央高校自由探索基金 (No.HEUCF170605）; 国家自然科学基金 (No.61573284）

RGB-D Scene Parsing Based on Spatial Structured Inference Deep Fusion Networks

WANG Ze-yu¹, WU Yan-xia¹, ZHANG Guo-yin¹, BU Shu-hui²

1. College of Computer Science and Technology, Harbin Engineering University, Harbin, Heilongjiang 150001, China;
2. School of Aeronautics, Northwestern Polytechnical University, Xi'an, Shaanxi 710072, China

Received:2017-05-19 Revised:2018-01-10 Online:2018-05-25 Published:2018-05-25
- Supported by:
- National Key Research and Development Program of China (No.2016YFB1000400); Harbin Outstanding Youth Talents Fund of Heilongjiang Province (No.2017RAYXJ016); Free Exploration Foundation for Central Universities (No.HEUCF170605); National Natural Science Foundation of China (No.61573284)

摘要/Abstract

摘要： 为了弥补RGB-D场景解析中卷积神经网络空间结构化学习能力的不足，本文基于深度学习提出空间结构化推理深度融合网络，内嵌的结构化推理层有机地结合条件随机场和空间结构化推理模型，该层能够较为全面而准确地学习物体所处三维空间的物体分布以及物体间的三维空间位置关系.在此基础上，网络的特征融合层巧妙地利用深度置信网络和改进的条件随机场，该层可以根据融合生成的物体综合语义信息和物体间语义相关性信息完成深度结构化学习.实验结果表明，在标准RGB-D数据集NYUDv2和SUNRGBD上，空间结构化推理深度融合网络分别实现最优的平均准确率53.8%和54.6%，从而有助于实现机器人任务规划、车辆自动驾驶等智能计算机视觉任务.

关键词: RGB-D场景解析, 深度学习, 卷积神经网络, 条件随机场, 空间结构化推理模型, 深度置信网络, 计算机视觉, 机器人任务规划, 车辆自动驾驶

Abstract: In order to make up the drawbacks that convolutional neural networks lack the ability of spatial structured learning in RGB-D scene parsing,we propose spatial structured inference deep fusion networks (SSIDFNs) on the basis of deep learning,the embedded structural inference layer organically combines conditional random fields (CRFs) and spatial structured inference model,which is able to learn the three-dimensional spatial distributions of objects and three-dimensional spatial relationships among objects in a more comprehensive and accurate way.Furthermore,the feature fusion layer takes both advantages of deep belief networks and improved CRFs,which is able to achieve deep structured learning according to the comprehensive semantic information of objects and semantic correlation in formation among objects.The experimental results demonstrate that the proposed SSIDFNs achieve the best mean accuracy 53.8% and 54.6% on the standard RGB-D datasets NYUDv2 and SUNRGBD respectively,which will be helpful to implement intelligent computer vision tasks,such as robot task planning and self-driving cars.

Key words: RGB-D scene parsing, deep learning, convolutional neural networks, conditional random fields, spatial structured inference model, deep belief networks, computer vision, robot task planning, self-driving cars

中图分类号:

王泽宇, 吴艳霞, 张国印, 等. 基于空间结构化推理深度融合网络的RGB-D场景解析[J]. 电子学报, 2018, 46(5): 1253-1258.

WANG Ze-yu, WU Yan-xia, ZHANG Guo-yin, et al. RGB-D Scene Parsing Based on Spatial Structured Inference Deep Fusion Networks[J]. Acta Electronica Sinica, 2018, 46(5): 1253-1258.

参考文献

[1] Long J,Shelhamer E,Darrell T.Fully convolutional networks for semantic segmentation[A].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition[C].Piscataway,NJ:IEEE Service Center,2015.3431-3440.
[2] Bojarski M,Del Testa D,Dworakowski D,et al.End to end learning for self-driving cars[J].Computer Science,2016,arXiv preprint arXiv:1604.07316.
[3] Sharma S,Kiros R,Salakhutdinov R.Action recognition using visual attention[J].Computer Science,2015,arXiv preprint arXiv:1511.04119.
[4] Kendall A,Badrinarayanan V,Cipolla R.Bayesian segnet:Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding[J].Computer Science,2015,arXiv preprint arXiv:1511.02680.
[5] Cadena C,Košecká J.Semantic segmentation with heterogeneous sensor coverages[A].Robotics and Automation[C].Piscataway,NJ:IEEE Service Center,2014.2639-2645.
[6] Bu S,Han P,Liu Z,et al.Scene parsing using inference embedded deep networks[J].Pattern Recognition,2016,59(11):188-198.
[7] Khan S H,Bennamoun M,Sohel F,et al.Integrating geometrical context for semantic labeling of indoor scenes using RGB-D images[J].International Journal of Computer Vision,2016,117(1):1-20.
[8] Wang Z,Wu Y,Zhang G,et al.Three-dimensional spatial structured encoding deep network for RGB-D scene parsing[J].Journal of Computer Applications,2017,37(12):3458-3466.
[9] Liang X,Shen X,Feng J,et al.Semantic object parsing with graph lstm[A].European Conference on Computer Vision[C].Berlin:Springer International Publishing,2016.125-143.
[10] Li Z,Gan Y,Liang X,et al.LSTM-CF:Unifying context modeling and fusion with LSTMs for RGB-D scene labeling[A].European Conference on Computer Vision[C].Berlin:Springer International Publishing,2016.541-557.
[11] Achanta R,Shaji A,Smith K,et al.SLIC superpixels compared to state-of-the-art superpixel methods[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2012,34(11):2274-2282.
[12] Hinton G E,Salakhutdinov R R.Reducing the dimensionality of data with neural networks[J].Science,2006,313(5786):504-507.
[13] Silberman N,Hoiem D,Kohli P,et al.Indoor segmentation and support inference from RGB-D images[A].European Conference on Computer Vision[C].Berlin:Springer International Publishing,2012.746-760.
[14] Song S,Lichtenberg S P,Xiao J.Sun RGB-D:A RGB-D scene understanding benchmark suite[A].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition[C].Piscataway,NJ:IEEE Service Center,2015.567-576.
[15] Schmidt M.UGM:A Matlab Toolbox for Probabilistic Undirected Graphical Models[EB/OL].http://www.cs.ubc.ca/~schmidtm/Software/UGM.html,2017.
[16] Perceptron M.DeepLearning 0.1 Documentation[EB/OL].http://deeplearning.net/tutorial/,2014-11-06.

基于空间结构化推理深度融合网络的RGB-D场景解析

RGB-D Scene Parsing Based on Spatial Structured Inference Deep Fusion Networks

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	李钦, 刘伟, 牛朝阳, 宝音图, 惠周勃. 低信噪比下基于分裂EfficientNet网络的雷达信号调制方式识别[J]. 电子学报, 2023, 51(3): 675-686.
[2]	黄赟, 张帆, 郭威, 陈立, 羊光. 一种基于数据标准差的卷积神经网络量化方法[J]. 电子学报, 2023, 51(3): 639-647.
[3]	范兵兵, 何庭建, 张聪炫, 陈震, 黎明. 联合遮挡约束与残差补偿的特征金字塔光流计算方法[J]. 电子学报, 2023, 51(3): 648-657.
[4]	吕杭, 蒋明峰, 李杨, 张鞠成, 王志康. 基于混合时频域特征的卷积神经网络心律失常分类方法的研究[J]. 电子学报, 2023, 51(3): 701-711.
[5]	张聿远, 闫文君, 张立民. 基于多模态特征融合网络的空时分组码识别算法[J]. 电子学报, 2023, 51(2): 489-498.
[6]	许新征, 李杉. 基于特征膨胀卷积模块的轻量化技术研究[J]. 电子学报, 2023, 51(2): 355-364.
[7]	丁琪, 田萱, 孙国栋. 基于注意力增强的热点感知新闻推荐模型[J]. 电子学报, 2023, 51(1): 93-104.
[8]	李滔, 董秀成, 林宏伟. 基于深监督跨尺度注意力网络的深度图像超分辨率重建[J]. 电子学报, 2023, 51(1): 128-138.
[9]	郭晓轩, 冯其波, 冀振燕, 郑发家, 杨燕燕. 多线激光光条图像缺陷分割模型研究[J]. 电子学报, 2023, 51(1): 172-179.
[10]	贾童瑶, 卓力, 李嘉锋, 张菁. 基于深度学习的单幅图像去雾研究进展[J]. 电子学报, 2023, 51(1): 231-245.
[11]	何滢婕, 刘月峰, 边浩东, 郭威, 张小燕. 基于Informer的电池荷电状态估算及其稀疏优化方法[J]. 电子学报, 2023, 51(1): 50-56.
[12]	张永梅, 孙捷. 基于动静态特征双输入神经网络的咳嗽声诊断COVID-19算法[J]. 电子学报, 2023, 51(1): 202-212.
[13]	袁海英, 成君鹏, 曾智勇, 武延瑞. Mobile_BLNet：基于Big-Little Net的轻量级卷积神经网络优化设计[J]. 电子学报, 2023, 51(1): 180-191.
[14]	王神龙, 雍宇, 吴晨睿. 基于伪孪生神经网络的低纹理工业零件6D位姿估计[J]. 电子学报, 2023, 51(1): 192-201.
[15]	吴靖, 叶晓晶, 黄峰, 陈丽琼, 王志锋, 刘文犀. 基于深度学习的单帧图像超分辨率重建综述[J]. 电子学报, 2022, 50(9): 2265-2294.