一种基于双流融合3D卷积神经网络的动态头势识别方法

doi:10.12263/DZXB.20201183

摘要/Abstract

摘要：

目前基于视觉的动态头势识别算法泛化能力弱、识别率低,头戴式传感器的方法经济性、便携性差.针对以上问题,提出了一种无需头戴设备的动态头势识别算法.这种基于双流融合3D卷积神经网络的方法用头部动作生成稠密光流,并将原始数据和光流数据并行输入构建的动作特征提取器,最后进行特征融合.结果表明所提算法比人工特征提取方法和C3D模型有更高的准确率、更好的泛化能力,在无需头戴传感器的情况下有近似头戴式传感器的识别率.

关键词: 深度学习, 机器视觉, 人机交互, 动作识别, 动态头势, 双流网络

Abstract:

Present vision based on dynamic head gesture recognition algorithms usually have disadvantages in generalization and recognition rate, and head?mounted sensors are expensive and inconvenient. In view of the above problems, a dynamic head gesture recognition algorithm without head?mounted sensors is proposed. Using this method based on two?stream 3DCNN(3D Convolutional Neural Network), the dense optical flow is generated by head movements, the original data and dense optical flow are put into the motion feature extractor in parallel, and finally, features are fused. Experimental results show that the proposed algorithm has higher recognition accuracy and better generalization than the artificial feature extraction and C3D(Convolutional 3D) methods, and its recognition rate is as good as those head mounted sensors.

Key words: deep learning, computer vision, human?computer interaction, action recognition, dynamic head gesture, two?stream network

中图分类号:

TP183

谢佳龙, 张波涛, 吕强. 一种基于双流融合3D卷积神经网络的动态头势识别方法[J]. 电子学报, 2021, 49(7): 1363-1369.

Jia‑long XIE, Bo‑tao ZHANG, Qiang LÜ. A Dynamic Head Gesture Recognition Method Based on 3D Convolutional Two‑Stream Network Fusion[J]. Acta Electronica Sinica, 2021, 49(7): 1363-1369.

图/表 9

参考文献 25

1	ZHAO J B， ALLISON R S. Real‑time head gesture recognition on head‑mounted displays using cascaded hidden Markov models［A］. Proceedings of the 2017 IEEE International Conference on Systems， Man and Cybernetics［C］. Banff， AB，Canada： IEEE， 2017. 2361 - 2366.
2	VADIRAJ S K， RAO A， GHOSH P K. Automatic identification of speakers from head gestures in a narration［A］. Proceedings of the 2020 IEEE International Conference on Acoustics， Speech and Signal Processing［C］. Barcelona， Spain： IEEE， 2020. 6314 - 6318.
3	SHARMA M， AHMETOVIC D， JENI L A， et al. Recognizing visual signatures of spontaneous head gestures［A］. Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision［C］. Nevada，USA： IEEE， 2018. 400 - 408.
4	ZHAO J B， ALLISON R S. Comparing head gesture hand gesture and gamepad interfaces for answering Yes/No questions in virtual environments［J］. Virtual Reality， 2020， 24： 515 - 524.
5	FUJIE S， EJIRI Y， NAKAJIMA K， et al. A conversation robot using head gesture recognition as para‑linguistic information［A］. Proceedings of the RO‑MAN 2004 13th IEEE International Workshop on Robot and Human Interactive Communication［C］. Kurashiki， Japan： IEEE， 2004. 159 - 164.
6	NG P C， SILVA L C D. Head gestures recognition［A］. Proceedings of the 2001 International Conference on Image Processing［C］. Thessaloniki， Greece： IEEE， 2001. 266 - 269.
7	SOLEA R， MARGARIT A， CERNEGA D， et al. Head movement control of powered wheelchair［A］. Proceedings of the 23rd International Conference on System Theory， Control and Computing ［C］. Sinaia， Romania： IEEE， 2019. 632 - 637.
8	JACKOWSKI A， GEBHARD M， THIETJE R. Head motion and head gesture‑based robot control： A usability study［J］. IEEE Transactions on Neural Systems and Rehabilitation Engineering， 2018， 26（1）： 161 - 170.
9	WU C W， YANG H Z， CHEN Y A， et al. Applying machine learning to head gesture recognition using wearables［A］. Proceedings of the 2017 IEEE 8th International Conference on Awareness Science and Technology［C］. Taichung， China： IEEE， 2017. 436 - 440.
10	JACKOWSKI A， GEBHARD M， GRÄSER A. A novel head gesture based interface for hands‑free control of a robot［A］. Proceedings of the 2016 IEEE International Symposium on Medical Measurements and Applications［C］. Benevento，Italy： IEEE， 2016. 1 - 6.
11	RUDIGKEIT N， GEBHARD M， GRÄSER A. An analytical approach for head gesture recognition with motion sensors［A］. Proceedings of the 2015 9th International Conference on Sensing Technology［C］. Auckland， New Zealand： IEEE， 2015. 1 - 6.
12	BANKAR R， SALANKAR S. Improvement of head gesture recognition using Camshift based face tracking with UKF［A］. Proceedings of the 2019 9th International Conference on Emerging Trends in Engineering and Tech‑ nology‑Signal and Information Processing［C］. Nagpur， India： IEEE， 2019. 1 - 5.
13	LU P， ZHANG M， ZHU X， et al. Head nod and shake recognition based on multi‑view model and hidden Markov model［A］. Proceedings of the International Conference on Computer Graphics， Imaging and Visualization［C］. Beijing， China： IEEE， 2005. 61 - 64.
14	HONG T， LI Y W， WANG Z Y. Real‑time head action recognition based on HOF and ELM［J］. IEICE Transactions on Information and Systems， 2019， 102（1）： 206 - 209.
15	罗会兰，童康，孔繁胜. 基于深度学习的视频中人体动作识别进展综述［J］. 电子学报， 2019， 47（5）： 1162 - 1173.
	LUO Hui‑lan， TONG Kang， KONG Fan‑sheng. The progress of human action recognition in videos based on deep learning： A review［J］. Acta Electronica Sinica， 2019， 47（5）： 1162 - 1173.（in Chinese）
16	FARNEBÄCK G. Two‑frame motion estimation based on polynomial expansion［A］. Proceedings of the 13th Scandinavian Conference on Image Analysis［C］. Halmstad， Sweden： Springer， 2003. 363 - 370.
17	LECUN Y， HUANG F J， BOTTOU L. Learning methods for generic object recognition with invariance to pose and lighting［A］. Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition［C］. Washington，USA： IEEE， 2004. 97 - 104.
18	SIMONYAN K， ZISSERMAN A. Two‑stream convolutional networks for action recognition in videos［J］. Advances in Neural Information Processing Systems， 2014， 27： 568 - 576.
19	JI S， XU W， YANG M， et al. 3D convolutional neural networks for human action recognition［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2012， 35（1）： 221 - 231.
20	TRAN D， BOUTDEV L， FERGUS R， et al. Learning spatiotemporal features with 3D convolutional networks［A］. Proceedings of the 2015 IEEE International Conference on Computer Vision［C］. Santiago，Chile： IEEE， 2015. 4489 - 4497.
21	FEICHTENHOFER C， PINZ A， ZISSERMAN A. Convolutional two‑stream network fusion for video action recognition［A］. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition［C］. Las Vegas， USA： IEEE， 2016. 1933 - 1941.
22	NAIR V， HINTON G E. Rectified linear units improve restricted Boltzmann machines［A］. Proceedings of the 27th International Conference on Machine Learning［C］. Haifa， Israel： ACM， 2010. 807 - 814.
23	SRIVASTAVA N， HINTON G， KRIZHEVSKY A， et al. Dropout： A simple way to prevent neural networks from overfitting［J］. The Journal of Machine Learning Research， 2014， 15（1）： 1929 - 1958.
24	KINGMA D P， Ba J. Adam： A method for stochastic optimization［A］. Proceedings of the 3rd International Conference on Learning Representations［C］. San Diego， USA： Conference Track Proceedings， 2015. 1884 - 2021.
25	SUNI S S， GOPAKUMAR K. A real time decision support system using head nod and shake［A］. Proceedings of the 2016 International Conference on Circuit， Power and Computing Technologies［C］. Nagercoil， India： IEEE， 2016. 1 - 5.

类别	训练集	验证集	测试集1	测试集2
点头	724	208	96	80
摇头	732	204	108	80
左偏	656	204	112	80
右偏	796	260	103	80

类别	训练集	验证集	测试集1	测试集2
点头	724	208	96	80
摇头	732	204	108	80
左偏	656	204	112	80
右偏	796	260	103	80

模型	参数量	验证集	测试集1识别率	测试集2识别率	识别耗时
C3D^［20］	78M	97.26%	93.32%	92.50%	94ms
本文方法	30M	99.09%	95.75%	97.50%	85ms

模型	参数量	验证集	测试集1识别率	测试集2识别率	识别耗时
C3D^［20］	78M	97.26%	93.32%	92.50%	94ms
本文方法	30M	99.09%	95.75%	97.50%	85ms

采取设备	算法	类别数	识别率
头戴式设备	FSM‑9^［10］	4类	81.80%
	GY‑87+HGR‑DT^［9］	6类	94.63%
	GY‑87+HGR‑RF^［9］	6类	99.17%
相机	Camshift with UKF^［12］	/	63.33%
	Eye Detection+SVM^［25］	3类	91.10%
	MS‑ConvLSTM^［3］	4类	73.25%
	肤色检测+HMM^［2］	3类	87.00%
	MVM+HMM^［13］	2类	88.10%
	HOF+ELM^［14］	7类	95.63%
	C3D^［20］	4类	92.50%
	双流融合3DCNN	4类	97.50%