基于时空图神经网络的手势识别

doi:10.12263/DZXB.20211069

电子学报 ›› 2022, Vol. 50 ›› Issue (4): 921-931.DOI: 10.12263/DZXB.20211069

所属专题：机器学习交叉融合创新

• 机器学习交叉融合创新 • 上一篇下一篇

基于时空图神经网络的手势识别

袁冠¹^,², 邴睿¹, 刘肖¹, 代伟³, 张艳梅¹, 蔡卓¹

^1.矿山数字化教育部工程研究中心，江苏徐州 221116
^2.中国矿业大学计算机科学与技术学院，江苏徐州 221116
^3.中国矿业大学信息与控制工程学院，江苏徐州 221116

收稿日期:2021-08-11 修回日期:2021-12-12 出版日期:2022-04-25
- 作者简介:
- 袁冠男，1982年生，江苏睢宁人.现为中国矿业大学计算机科学与技术学院教授.主要研究方向为时空大数据技术以及计算智能.E-mail： yuanguan@cumt.edu.cn
  邴睿男，1994年生，甘肃兰州人. 现为中国矿业大学博士生.主要研究方向为图数据挖掘. E-mail： bingrui@cumt.edu.cn
  刘肖男，1994年生，江苏徐州人. 主要研究方向为模式识别与感知计算. E-mail： liuxiaocumt2018@163.com
- 基金资助:
- 国家自然科学基金 (71774159); 中国博士后科学基金 (2021T140707)

Spatial-Temporal Graph Neural Network based Hand Gesture Recognition

YUAN Guan¹^,², BING Rui¹, LIU Xiao¹, DAI Wei³, ZHANG Yan-mei¹, CAI Zhuo¹

^1.Digitization of Mine, Engineering Research Center of Ministry of Education, Xuzhou, Jiangsu 221116, China
^2.School of Computer Science and Technology, China University of Mining and Technology, Xuzhou, Jiangsu 221116, China
^3.School of Information and Control Engineering, China University of Mining and Technology, Xuzhou, Jiangsu 221116, China

Received:2021-08-11 Revised:2021-12-12 Online:2022-04-25 Published:2022-04-25
- Supported by:
- National Natural Science Foundation of China (71774159); China Postdoctoral Science Foundation (2021T140707)

摘要/Abstract

摘要：

随着感知计算以及传感器集成技术的发展，使用各种传感设备实时捕捉的手势运动数据，为人机交互提供了新的驱动力，并被广泛地应用于智能家居、远程医疗、虚拟现实等领域.由于手势动作具有时序性与空间连接性，因此在手势识别中需要考虑手势空间连接关系和手势长距离依赖特性.然而现有的手势识别方法忽略了上述两种特性，导致识别精度不高.本文提出了基于时空图神经网络的手势识别算法，该方法从传感器空间分布角度出发，基于传感器的空间位置信息，借助图神经网络（Graph Neural Networks，GNN）对手势数据之间的空间关联性进行表征，并引入门控循环单元（Gated Recurrent Unit，GRU）解决手势的时序性和长距离依赖问题，增强手势识别性能.在多种数据集上的实验结果证明本文方法可行且有效.

关键词: 手势识别, 多传感器融合, 时空图神经网络, 循环神经网络

Abstract:

With the development of perceptual computing and sensor integration technology, hand gesture motion data collected by various sensor devices provides a new data-driven way for human-computer interaction, and widely used in smart home, telemedicine, virtual reality and other fields. Due to hand gestures have temporality and spatial connectivity, it is necessary to consider spatial connection and long-distance dependence of hand gesture in gesture recognition. However, existing hand gesture recognition models ignore the aforementioned two problems, resulting in low recognition accuracy. Therefore, we propose a spatial-temporal graph neural network based hand gesture recognition model(STGNN-HGR). From the perspective of spatial distribution of sensors, based on the spatial location information of sensors, the model represents spatial correlation of hand gesture data with the help of graph neural networks(GNN), and introduces gated recurrent unit(GRU) to solve temporality and long-distance dependence in dynamic hand gestures, so as to enhance the performance of gesture recognition. The experimental results on a variety of datasets show that our model is feasible and effective.

Key words: hand gesture recognition, multi-sensor fusion, spatial-temporal graph neural network, recurrent neural network

中图分类号:

TP391.4

袁冠, 邴睿, 刘肖, 等. 基于时空图神经网络的手势识别[J]. 电子学报, 2022, 50(4): 921-931.

Guan YUAN, Rui BING, Xiao LIU, et al. Spatial-Temporal Graph Neural Network based Hand Gesture Recognition[J]. Acta Electronica Sinica, 2022, 50(4): 921-931.

图/表 18

图1 手势建模

图2 STGNN-HGR框架

图3 手势时序特征

图4 GRU结构图

图5 CRE传感器数据图

表1 实验数据集

数据集	样本总数	手势种类	采集设备	说明
CRE	22 595	6	数据手套	单词级手势
CSL	95 040	72	数据手套	单词级手势

表2 对比算法

算法名称	特点	说明
K最近邻(KNN)	利用周围有限的邻近样本对类域进行判定	验证深度特征表示能力强于手动提取特征
支持向量机(SVM)	利用内积核函数解决非线性分类问题	验证深度特征表示能力强于手动提取特征
卷积神经网络 (CNN)	能够提取手势数据具有高语义的深度特征	验证时序性对手势识别效果的影响
门控循环单元 (GRU)	能够解决动态手势的时序性和长距离依赖问题	验证手势数据空间特征能够更好地表征手势数据
CNN-LSTM	CNN提取手势数据的深度特征,结合LSTM解决时序性和长距离依赖问题	验证STGNN能够更好地表征手势数据,提高手势识别准确率,增强手势识别性能
STGNN-HGR	使用GCN提取数据的空间特征,并结合GRU解决时序性和长距离依赖问题.	STGNN能够提取数据空间特征,解决动态手势数据的时序性及长距离依赖问题

图6 图卷积层数分析

表3 常用统计特征

统计特征	描述	计算公式
均值特征	窗口中信号的平均值	$m e a n = 1 n ∑ i = 1 n a i$
最值特征	窗口中信号的最大值和最小值	$m a x = m a x (a i), i = 1,2, ⋯, n$ $m i n = m i n (a i), i = 1,2, ⋯, n$
范围特征	窗口中信号最大值与最小值之差	$r a n g e = \| m a x - m i n \|$
方差特征	窗口中一组数据的分散程度度量	$s 2 = 1 n - 1 ∑ i = 1 n (x i - m e a n) 2$

表3 常用统计特征

统计特征	描述	计算公式
均值特征	窗口中信号的平均值	$m e a n = 1 n ∑ i = 1 n a i$
最值特征	窗口中信号的最大值和最小值	$m a x = m a x (a i), i = 1,2, ⋯, n$ $m i n = m i n (a i), i = 1,2, ⋯, n$
范围特征	窗口中信号最大值与最小值之差	$r a n g e = \| m a x - m i n \|$
方差特征	窗口中一组数据的分散程度度量	$s 2 = 1 n - 1 ∑ i = 1 n (x i - m e a n) 2$

图7 维度填充方式分析

图8 参数选择

图9 对比算法准确率分析

图10 混淆矩阵

表4 标签信息

标签	CSL	Acc(%)	CRE	Acc(%)
0	你好	99.45	Come	97.27
1	再见	100.00	Hurry up	95.92
2	擦	99.24	Rally point	93.64
3	吃	99.24	Vehicle	100.00
4	打	97.73	I don't understand	100.00
5	翻	100.00	Door	100.00
6	喝	98.10	Go here	99.32
7	很好	98.10	Ammunition	100.00
8	惊讶	84.09	Go Prone	97.04
9	谁	100.00	Point of Entry	100.00

图11 CSL数据集手势欧拉角

图12 CRE数据集

表5 消融实验结果

算法名称	CRE数据集	CSL数据集
STGNN-HGR-w/o GCN	93.79	94.18
STGNN-HGR-LSTM	96.18	96.13
STGNN-HGR	96.82	97.27

图13 GRU与LSTM性能对比

参考文献 29

1	李艳德. 基于穿戴传感感知的手势识别模型与应用研究[D]. 兰州: 兰州大学, 2019.
	LIY D. Research on Gesture Recognition Model and Its Application Based on Wear Sensing Perception[D]. Lanzhou: Lanzhou University, 2019.
2	王勇, 王沙沙, 田增山, 等. 基于FMCW雷达的双流融合神经网络手势识别方法[J]. 电子学报, 2019, 47(7): 1408-1415.
	WANGY, WANGS S, TIANZ S, et al. Two-stream fusion neural network approach for hand gesture recognition based on fmcw radar[J]. Acta Electronica Sinica, 2019, 47(7): 1408-1415. (in Chinese)
3	冯志全, 杨学文, 徐涛, 等. 结合手势二进制编码和类-Hausdorff距离的手势识别[J]. 电子学报, 2017, 45(9): 2281-2291.
	FENGZ Q, YANGX W, XUT, et al. Gesture recognition based on combining gesture binary descriptor and hausdorff-like distance[J]. Acta Electronica Sinica, 2017, 45(9): 2281-2291. (in Chinese)
4	KIMM, JEONC, KIMJ, et al. A study on immersion and presence of a portable hand haptic system for immersive virtual reality[J]. Sensors, 2017, 17(5): 1141-1158.
5	LIR Q, ZHENGD W, HANZ Y, et al. MHealth: A smartphone-controlled, wearable platform for tumour treatment[J]. Materials Today, 2020, 40(11): 91-100.
6	ISONM, VUJAKLIJAI, WHITSELLB, et al. High-density electromyography and motor skill learning for robust long-term control of a 7-DoF robot arm[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2015, 24(4): 424-433.
7	FANGB, SUNF, LIUH, et al. A novel data glove using inertial and magnetic sensors for motion capture and robotic arm-hand teleoperation[J]. Industrial Robot: An International Journal, 2017, 44(2): 155-165.
8	CHOUP H, HSUY L, LEEW L, et al. Development of a smart home system based on multi-sensor data fusion technology[C]//Proceedings of IEEE International Conference on Applied System Innovation. Sapporo: IEEE, 2017: 690-693.
9	MAB, CHENB, ZHANGZ, et al. Combat gesture classification using through-the-wall radar based on multi-domain features association[C]//Proceedings of IEEE Radar Conference. Florence: IEEE, 2020: 1-5.
10	李愚, 柴国钟, 卢纯福, 等. 基于增量自适应学习的在线肌电手势识别[J]. 计算机科学, 2019, 46(4): 274-279.
	LIY, CHAIG Z, LUC F, et al. On-line sEMG hand gesture recognition based on incremental adaptive learning[J]. Computer Science, 2019, 46(4): 274-279. (in Chinese)
11	陈国良, 葛凯凯, 李聪浩. 基于多特征HMM融合的复杂动态手势识别[J]. 华中科技大学学报(自然科学版), 2018, 46(12): 42-47.
	CHENG L, GEK K, LIC H. Complex dynamic gesture recognition based on multiple features and HMM fusion[J]. Journal of Huazhong University of Science and Technology(Natural Science Edition), 2018, 46(12): 42-47. (in Chinese)
12	CHENW, ZHANGZ. Hand gesture recognition using sEMG signals based on support vector machine[C]//Proceedings of the 8th Joint International Information Technology and Artificial Intelligence Conference. Chongqing: IEEE, 2019: 230-234.
13	KUMARP, GAUBAH, ROYP P, et al. Coupled HMM-based multi-sensor data fusion for sign language recognition[J]. Pattern Recognition Letters, 2017, 86(1): 1-8.
14	PURUSHOTHAMANA, PALANISWAMYS. Development of smart home using gesture recognition for elderly and disabled[J]. Journal of Computational and Theoretical Nanoscience, 2020, 17(1): 177-181.
15	CHENL, FUJ, WUY, et al. Hand gesture recognition using compact CNN via surface electromyography signals[J]. Sensors, 2020, 20(3): 672-680.
16	SHINS, KIMW Y. Skeleton-based dynamic hand gesture recognition using a part-based gru-rnn for gesture-based interface[J]. IEEE Access, 2020, 8(3): 50236-50243.
17	CHENY, MAG, YUANC, et al. Graph convolutional network with structure pooling and joint-wise channel attention for action recognition[J]. Pattern Recognition, 2020, 103(7): 107321-103731.
18	NUNEZJ C, CABIDOR, PANTRIGOJ J, et al. Convolutional neural networks and long short-term memory for skeleton-based human activity and hand gesture recognition[J]. Pattern Recognition, 2018, 76(4): 80-94.
19	CHENH, TONGR, CHENM, et al. A hybrid cnn-svm classifier for hand gesture recognition with surface emg signals[C]//Proceedings of the 17th International Conference on Machine Learning and Cybernetics. Chengdu: IEEE, 2018: 619-624.
20	刘肖, 袁冠, 张艳梅, 等. 基于自适应多分类器融合的手势识别[J]. 计算机科学, 2020, 47(7): 103-110.
	LIUX, YUANG, ZHANGY M, et al. Hand gesture recognition based on self-adaptive multi-classifiers fusion[J]. Computer Science, 2020, 47(7): 103-110. (in Chinese)
21	ZHUG, ZHANGL, YANGL, et al. Redundancy and attention in convolutional LSTM for gesture recognition[J]. IEEE Transactions on Neural Networks and Learning Systems, 2019, 31(4): 1323-1335.
22	YANS, XIONGY, LIND. Spatial temporal graph convolutional networks for skeleton-based action recognition[C]//Thirty-second AAAI conference on artificial intelligence. New Orleans: AAAI, 2018: 1-10.
23	白铂, 刘玉婷, 马驰骋, 等. 图神经网络[J]. 中国科学: 数学, 2020, 50(3): 367-384.
	BAIB, LIUY T, MAC C, et al. Graph neural networks[J]. Scientia Sinica: Mathematica, 2020, 50(3): 367-384. (in Chinese)
24	ZHANGX, CHENX, LIY. A framework for hand gesture recognition based on accelerometer and EMG sensors[J]. IEEE Transactions on Systems Man & Cybernetics Part A Systems & Humans, 2011, 41(6): 1064-1076.
25	TIDWELLR, AKUMALLAS, KARLAPUTIS, et al. Evaluating the feasibility of EMG and bend sensors for classifying hand gestures[C]//Proceedings of the International Conference on Multimedia and Human Computer Interaction. Toronto: ISSN, 2013: 1-8.
26	WAHIDM F, TAFRESHIR, AL-SOWAIDIM, et al. Subject-independent hand gesture recognition using normalization and machine learning algorithms[J]. Journal of Computational Science, 2018, 27(7): 69-76.
27	SHARMAS, MODIS, RANAP S, et al. Hand gesture recognition using Gaussian threshold and different SVM kernels[C]//Proceedings of International Conference on Advances in Computing and Data Sciences. Dehradun: Springer, 2018: 138-147.
28	LIG, TANGH, SUNY, et al. Hand gesture recognition based on convolution neural network[J]. Cluster Computing, 2019, 22(2): 2719-2729.
29	KHODABANDELOUG, JUNGP G, AMIRATY, et al. Attention-based gated recurrent unit for gesture recognition[J]. IEEE Transactions on Automation Science and Engineering, 2020, 18(2): 495-507.

[1]	杨利平, 侯振威, 辜小花, 郝峻永. 弱标签声音事件检测的空间-通道特征表征与自注意池化[J]. 电子学报, 2023, 51(2): 297-306.
[2]	徐兴荣, 刘聪, 李婷, 郭娜, 任崇广, 曾庆田. 基于双向准循环神经网络和注意力机制的业务流程剩余时间预测方法[J]. 电子学报, 2022, 50(8): 1975-1984.
[3]	冀振燕, 韩梦豪, 宋晓军, 冯其波. 面向激光光条图像修复的循环相似度映射网络[J]. 电子学报, 2022, 50(5): 1234-1242.
[4]	肖斌, 陈嘉博, 毕秀丽, 张俊辉, 李伟生, 王国胤, 马旭. 基于一维卷积神经网络与循环神经网络串联的心音分析方法[J]. 电子学报, 2022, 50(10): 2425-2432.
[5]	高俊涛, 王梅, 徐光会, 刘聪. 正则语言推断综述[J]. 电子学报, 2021, 49(12): 2479-2489.
[6]	桑海峰, 赵子裕, 何大阔. 基于循环区域关注和视频帧关注的视频行为识别网络设计[J]. 电子学报, 2020, 48(6): 1052-1061.
[7]	张丞, 何坚, 王伟东. 空间上下文与时序特征融合的交警指挥手势识别技术[J]. 电子学报, 2020, 48(5): 966-974.
[8]	尤洪峰, 田生伟, 禹龙, 吕亚龙. 基于Word Embedding的遥感影像检测分割[J]. 电子学报, 2020, 48(1): 75-83.
[9]	王勇, 王沙沙, 田增山, 周牧, 吴金君. 基于FMCW雷达的双流融合神经网络手势识别方法[J]. 电子学报, 2019, 47(7): 1408-1415.
[10]	查雄, 彭华, 秦鑫, 李天昀, 李广. 基于循环神经网络的卫星幅相信号调制识别与解调算法[J]. 电子学报, 2019, 47(11): 2443-2448.
[11]	吕品, 于文兵, 汪鑫, 计春雷, 周曦民. 异构分类器堆叠泛化及其在恶意评论检测中的应用[J]. 电子学报, 2019, 47(10): 2228-2234.
[12]	冯志全, 杨学文, 徐涛, 刘弘, 吕娜, 杨晓辉, 徐治鹏. 结合手势二进制编码和类-Hausdorff距离的手势识别[J]. 电子学报, 2017, 45(9): 2281-2291.
[13]	于昕妍, 沈艳霞, 陈杰, 纪志成. 考虑概率区间的微电网短期负荷多目标预测方法[J]. 电子学报, 2017, 45(4): 930-936.
[14]	徐树生, 林孝工. 船舶动力定位多传感器闭环分级融合算法[J]. 电子学报, 2014, 42(3): 512-516.
[15]	许言午;曹先彬;乔红. 行人检测系统研究新进展及关键技术展望[J]. 电子学报, 2008, 36(5): 962-968.

基于时空图神经网络的手势识别

Spatial-Temporal Graph Neural Network based Hand Gesture Recognition

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 18

参考文献 29

相关文章 15

编辑推荐

Metrics