Spatial-Temporal Graph Neural Network based Hand Gesture Recognition

YUAN Guan; BING Rui; LIU Xiao; DAI Wei; ZHANG Yan-mei; CAI Zhuo

doi:10.12263/DZXB.20211069

您当前的位置：

首页 >

文章列表页 >

Spatial-Temporal Graph Neural Network based Hand Gesture Recognition

CROSS-DISCIPLINARY INNOVATIONS OF MACHINE LEARNING | 更新时间：2025-12-08

- Spatial-Temporal Graph Neural Network based Hand Gesture Recognition
- ACTA ELECTRONICA SINICA Vol. 50, Issue 4, Pages: 921-931(2022)
- 作者机构：
  
  1.矿山数字化教育部工程研究中心，江苏徐州 221116
  2.中国矿业大学计算机科学与技术学院，江苏徐州 221116
  3.中国矿业大学信息与控制工程学院，江苏徐州 221116
- 作者简介：
- 基金信息：
- DOI：10.12263/DZXB.20211069
  CLC： TP391.4;
- Received：11 August 2021，
  
  Revised：2021-12-12，
  
  Published：25 April 2022
- 稿件说明：
移动端阅览
袁冠,邴睿,刘肖等.基于时空图神经网络的手势识别[J].电子学报,2022,50(04):921-931.

YUAN Guan,BING Rui,LIU Xiao,et al.Spatial-Temporal Graph Neural Network based Hand Gesture Recognition[J].ACTA ELECTRONICA SINICA,2022,50(04):921-931.
袁冠,邴睿,刘肖等.基于时空图神经网络的手势识别[J].电子学报,2022,50(04):921-931. DOI： 10.12263/DZXB.20211069.

YUAN Guan,BING Rui,LIU Xiao,et al.Spatial-Temporal Graph Neural Network based Hand Gesture Recognition[J].ACTA ELECTRONICA SINICA,2022,50(04):921-931. DOI： 10.12263/DZXB.20211069.

摘要

随着感知计算以及传感器集成技术的发展，使用各种传感设备实时捕捉的手势运动数据，为人机交互提供了新的驱动力，并被广泛地应用于智能家居、远程医疗、虚拟现实等领域.由于手势动作具有时序性与空间连接性，因此在手势识别中需要考虑手势空间连接关系和手势长距离依赖特性.然而现有的手势识别方法忽略了上述两种特性，导致识别精度不高.本文提出了基于时空图神经网络的手势识别算法，该方法从传感器空间分布角度出发，基于传感器的空间位置信息，借助图神经网络（Graph Neural Networks，GNN）对手势数据之间的空间关联性进行表征，并引入门控循环单元（Gated Recurrent Unit，GRU）解决手势的时序性和长距离依赖问题，增强手势识别性能.在多种数据集上的实验结果证明本文方法可行且有效.

Abstract

With the development of perceptual computing and sensor integration technology

hand gesture motion data collected by various sensor devices provides a new data-driven way for human-computer interaction

and widely used in smart home

telemedicine

virtual reality and other fields. Due to hand gestures have temporality and spatial connectivity

it is necessary to consider spatial connection and long-distance dependence of hand gesture in gesture recognition. However

existing hand gesture recognition models ignore the aforementioned two problems

resulting in low recognition accuracy. Therefore

we propose a spatial-temporal graph neural network based hand gesture recognition model(STGNN-HGR). From the perspective of spatial distribution of sensors

based on the spatial location information of sensors

the model represents spatial correlation of hand gesture data with the help of graph neural networks(GNN)

and introduces gated recurrent unit(GRU) to solve temporality and long-distance dependence in dynamic hand gestures

so as to enhance the performance of gesture recognition. The experimental results on a variety of datasets show that our model is feasible and effective.

关键词

Keywords

references

李艳德 . 基于穿戴传感感知的手势识别模型与应用研究 [D]. 兰州 : 兰州大学 , 2019 .

LI Y D . Research on Gesture Recognition Model and Its Application Based on Wear Sensing Perception [D]. Lanzhou : Lanzhou University , 2019 .

王勇 , 王沙沙 , 田增山 , 等 . 基于FMCW雷达的双流融合神经网络手势识别方法 [J]. 电子学报 , 2019 , 47 ( 7 ): 1408 - 1415 .

WANG Y , WANG S S , TIAN Z S , et al . Two-stream fusion neural network approach for hand gesture recognition based on fmcw radar [J]. Acta Electronica Sinica , 2019 , 47 ( 7 ): 1408 - 1415 . (in Chinese)

冯志全 , 杨学文 , 徐涛 , 等 . 结合手势二进制编码和类-Hausdorff距离的手势识别 [J]. 电子学报 , 2017 , 45 ( 9 ): 2281 - 2291 .

FENG Z Q , YANG X W , XU T , et al . Gesture recognition based on combining gesture binary descriptor and hausdorff-like distance [J]. Acta Electronica Sinica , 2017 , 45 ( 9 ): 2281 - 2291 . (in Chinese)

KIM M , JEON C , KIM J , et al . A study on immersion and presence of a portable hand haptic system for immersive virtual reality [J]. Sensors , 2017 , 17 ( 5 ): 1141 - 1158 .

LI R Q , ZHENG D W , HAN Z Y , et al . MHealth: A smartphone-controlled, wearable platform for tumour treatment [J]. Materials Today , 2020 , 40 ( 11 ): 91 - 100 .

ISON M , VUJAKLIJA I , WHITSELL B , et al . High-density electromyography and motor skill learning for robust long-term control of a 7-DoF robot arm [J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering , 2015 , 24 ( 4 ): 424 - 433 .

FANG B , SUN F , LIU H , et al . A novel data glove using inertial and magnetic sensors for motion capture and robotic arm-hand teleoperation [J]. Industrial Robot: An International Journal , 2017 , 44 ( 2 ): 155 - 165 .

CHOU P H , HSU Y L , LEE W L , et al . Development of a smart home system based on multi-sensor data fusion technology [C]// Proceedings of IEEE International Conference on Applied System Innovation . Sapporo : IEEE , 2017 : 690 - 693 .

MA B , CHEN B , ZHANG Z , et al . Combat gesture classification using through-the-wall radar based on multi-domain features association [C]// Proceedings of IEEE Radar Conference . Florence : IEEE , 2020 : 1 - 5 .

李愚 , 柴国钟 , 卢纯福 , 等 . 基于增量自适应学习的在线肌电手势识别 [J]. 计算机科学 , 2019 , 46 ( 4 ): 274 - 279 .

LI Y , CHAI G Z , LU C F , et al . On-line sEMG hand gesture recognition based on incremental adaptive learning [J]. Computer Science , 2019 , 46 ( 4 ): 274 - 279 . (in Chinese)

陈国良 , 葛凯凯 , 李聪浩 . 基于多特征HMM融合的复杂动态手势识别 [J]. 华中科技大学学报(自然科学版) , 2018 , 46 ( 12 ): 42 - 47 .

CHEN G L , GE K K , LI C H . Complex dynamic gesture recognition based on multiple features and HMM fusion [J]. Journal of Huazhong University of Science and Technology(Natural Science Edition) , 2018 , 46 ( 12 ): 42 - 47 . (in Chinese)

CHEN W , ZHANG Z . Hand gesture recognition using sEMG signals based on support vector machine [C]// Proceedings of the 8th Joint International Information Technology and Artificial Intelligence Conference . Chongqing : IEEE , 2019 : 230 - 234 .

KUMAR P , GAUBA H , ROY P P , et al . Coupled HMM-based multi-sensor data fusion for sign language recognition [J]. Pattern Recognition Letters , 2017 , 86 ( 1 ): 1 - 8 .

PURUSHOTHAMAN A , PALANISWAMY S . Development of smart home using gesture recognition for elderly and disabled [J]. Journal of Computational and Theoretical Nanoscience , 2020 , 17 ( 1 ): 177 - 181 .

CHEN L , FU J , WU Y , et al . Hand gesture recognition using compact CNN via surface electromyography signals [J]. Sensors , 2020 , 20 ( 3 ): 672 - 680 .

SHIN S , KIM W Y . Skeleton-based dynamic hand gesture recognition using a part-based gru-rnn for gesture-based interface [J]. IEEE Access , 2020 , 8 ( 3 ): 50236 - 50243 .

CHEN Y , MA G , YUAN C , et al . Graph convolutional network with structure pooling and joint-wise channel attention for action recognition [J]. Pattern Recognition , 2020 , 103 ( 7 ): 107321 - 103731 .

NUNEZ J C , CABIDO R , PANTRIGO J J , et al . Convolutional neural networks and long short-term memory for skeleton-based human activity and hand gesture recognition [J]. Pattern Recognition , 2018 , 76 ( 4 ): 80 - 94 .

CHEN H , TONG R , CHEN M , et al . A hybrid cnn-svm classifier for hand gesture recognition with surface emg signals [C]// Proceedings of the 17th International Conference on Machine Learning and Cybernetics . Chengdu : IEEE , 2018 : 619 - 624 .

刘肖 , 袁冠 , 张艳梅 , 等 . 基于自适应多分类器融合的手势识别 [J]. 计算机科学 , 2020 , 47 ( 7 ): 103 - 110 .

LIU X , YUAN G , ZHANG Y M , et al . Hand gesture recognition based on self-adaptive multi-classifiers fusion [J]. Computer Science , 2020 , 47 ( 7 ): 103 - 110 . (in Chinese)

ZHU G , ZHANG L , YANG L , et al . Redundancy and attention in convolutional LSTM for gesture recognition [J]. IEEE Transactions on Neural Networks and Learning Systems , 2019 , 31 ( 4 ): 1323 - 1335 .

YAN S , XIONG Y , LIN D . Spatial temporal graph convolutional networks for skeleton-based action recognition [C]// Thirty-second AAAI conference on artificial intelligence . New Orleans : AAAI , 2018 : 1 - 10 .

白铂 , 刘玉婷 , 马驰骋 , 等 . 图神经网络 [J]. 中国科学: 数学 , 2020 , 50 ( 3 ): 367 - 384 .

BAI B , LIU Y T , MA C C , et al . Graph neural networks [J]. Scientia Sinica: Mathematica , 2020 , 50 ( 3 ): 367 - 384 . (in Chinese)

ZHANG X , CHEN X , LI Y . A framework for hand gesture recognition based on accelerometer and EMG sensors [J]. IEEE Transactions on Systems Man & Cybernetics Part A Systems & Humans , 2011 , 41 ( 6 ): 1064 - 1076 .

TIDWELL R , AKUMALLA S , KARLAPUTI S , et al . Evaluating the feasibility of EMG and bend sensors for classifying hand gestures [C]// Proceedings of the International Conference on Multimedia and Human Computer Interaction . Toronto : ISSN , 2013 : 1 - 8 .

WAHID M F , TAFRESHI R , AL-SOWAIDI M , et al . Subject-independent hand gesture recognition using normalization and machine learning algorithms [J]. Journal of Computational Science , 2018 , 27 ( 7 ): 69 - 76 .

SHARMA S , MODI S , RANA P S , et al . Hand gesture recognition using Gaussian threshold and different SVM kernels [C]// Proceedings of International Conference on Advances in Computing and Data Sciences . Dehradun : Springer , 2018 : 138 - 147 .

LI G , TANG H , SUN Y , et al . Hand gesture recognition based on convolution neural network [J]. Cluster Computing , 2019 , 22 ( 2 ): 2719 - 2729 .

KHODABANDELOU G , JUNG P G , AMIRAT Y , et al . Attention-based gated recurrent unit for gesture recognition [J]. IEEE Transactions on Automation Science and Engineering , 2020 , 18 ( 2 ): 495 - 507 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

3D Object Detection Based on Multilayer Multimodal Fusion

An Fusion SLAM Method for LiDAR Visual and IMU Based on Factor Map Elimination Optimization

A Method of Heart Sound Analysis Based on One-Dimensional Convolutional Neural Network and Recurrent Neural Network

Recurrent Similarity Mapping Network Oriented to Laser Stripe Image Inpainting

Survey on the Latest Development of Pedestrian Detection System and Its Key Technologies Expectation

Related Author

MA Wen-hao

ZHOU Zhi-guo

YUAN Guo-shuai

QI Yong-sheng

LIU Li-qiang

SU Jian-qiang

ZHANG Li-jie

ZHANG Jun-hui

Related Institution

School of Integrated Circuits and Electronics， Beijing Institute of Technology

Engineering Research Center of Large Energy Storage Technology， Ministry of Education

Center for Intelligent Energy Technology and Equipment Engineering， Inner Mongolia University

School of Electric Power， Inner Mongolia University of Technology

The First Affiliated Hospital of Chongqing Medical University

⁰