基于时空图神经网络的手势识别

袁冠; 邴睿; 刘肖; 代伟; 张艳梅; 蔡卓

doi:10.12263/DZXB.20211069

您当前的位置：

首页 >

文章列表页 >

基于时空图神经网络的手势识别

机器学习交叉融合创新 | 更新时间：2025-12-08

- 基于时空图神经网络的手势识别
- Spatial-Temporal Graph Neural Network based Hand Gesture Recognition
- 电子学报 2022年50卷第4期页码：921-931
- 作者机构：
  
  1.矿山数字化教育部工程研究中心，江苏徐州 221116
  2.中国矿业大学计算机科学与技术学院，江苏徐州 221116
  3.中国矿业大学信息与控制工程学院，江苏徐州 221116
- 作者简介：
  
  [ "袁冠男，1982年生，江苏睢宁人.现为中国矿业大学计算机科学与技术学院教授.主要研究方向为时空大数据技术以及计算智能.E-mail： yuanguan@cumt.edu.cn" ]
  [ "邴睿男，1994年生，甘肃兰州人. 现为中国矿业大学博士生.主要研究方向为图数据挖掘. E-mail： bingrui@cumt.edu.cn" ]
  [ "刘肖男，1994年生，江苏徐州人. 主要研究方向为模式识别与感知计算. E-mail： liuxiaocumt2018@163.com" ]
- 基金信息：
  
  国家自然科学基金(71774159);中国博士后科学基金(2021T140707;2018M642358)
- DOI：10.12263/DZXB.20211069
  中图分类号： TP391.4;
- 收稿：2021-08-11，
  
  修回：2021-12-12，
  
  纸质出版：2022-04-25
- 稿件说明：
移动端阅览
袁冠,邴睿,刘肖等.基于时空图神经网络的手势识别[J].电子学报,2022,50(04):921-931.

YUAN Guan,BING Rui,LIU Xiao,et al.Spatial-Temporal Graph Neural Network based Hand Gesture Recognition[J].ACTA ELECTRONICA SINICA,2022,50(04):921-931.
袁冠,邴睿,刘肖等.基于时空图神经网络的手势识别[J].电子学报,2022,50(04):921-931. DOI： 10.12263/DZXB.20211069.

YUAN Guan,BING Rui,LIU Xiao,et al.Spatial-Temporal Graph Neural Network based Hand Gesture Recognition[J].ACTA ELECTRONICA SINICA,2022,50(04):921-931. DOI： 10.12263/DZXB.20211069.

摘要

随着感知计算以及传感器集成技术的发展，使用各种传感设备实时捕捉的手势运动数据，为人机交互提供了新的驱动力，并被广泛地应用于智能家居、远程医疗、虚拟现实等领域.由于手势动作具有时序性与空间连接性，因此在手势识别中需要考虑手势空间连接关系和手势长距离依赖特性.然而现有的手势识别方法忽略了上述两种特性，导致识别精度不高.本文提出了基于时空图神经网络的手势识别算法，该方法从传感器空间分布角度出发，基于传感器的空间位置信息，借助图神经网络（Graph Neural Networks，GNN）对手势数据之间的空间关联性进行表征，并引入门控循环单元（Gated Recurrent Unit，GRU）解决手势的时序性和长距离依赖问题，增强手势识别性能.在多种数据集上的实验结果证明本文方法可行且有效.

Abstract

With the development of perceptual computing and sensor integration technology

hand gesture motion data collected by various sensor devices provides a new data-driven way for human-computer interaction

and widely used in smart home

telemedicine

virtual reality and other fields. Due to hand gestures have temporality and spatial connectivity

it is necessary to consider spatial connection and long-distance dependence of hand gesture in gesture recognition. However

existing hand gesture recognition models ignore the aforementioned two problems

resulting in low recognition accuracy. Therefore

we propose a spatial-temporal graph neural network based hand gesture recognition model(STGNN-HGR). From the perspective of spatial distribution of sensors

based on the spatial location information of sensors

the model represents spatial correlation of hand gesture data with the help of graph neural networks(GNN)

and introduces gated recurrent unit(GRU) to solve temporality and long-distance dependence in dynamic hand gestures

so as to enhance the performance of gesture recognition. The experimental results on a variety of datasets show that our model is feasible and effective.

关键词

Keywords

references

李艳德 . 基于穿戴传感感知的手势识别模型与应用研究 [D]. 兰州 : 兰州大学 , 2019 .

LI Y D . Research on Gesture Recognition Model and Its Application Based on Wear Sensing Perception [D]. Lanzhou : Lanzhou University , 2019 .

王勇 , 王沙沙 , 田增山 , 等 . 基于FMCW雷达的双流融合神经网络手势识别方法 [J]. 电子学报 , 2019 , 47 ( 7 ): 1408 - 1415 .

WANG Y , WANG S S , TIAN Z S , et al . Two-stream fusion neural network approach for hand gesture recognition based on fmcw radar [J]. Acta Electronica Sinica , 2019 , 47 ( 7 ): 1408 - 1415 . (in Chinese)

冯志全 , 杨学文 , 徐涛 , 等 . 结合手势二进制编码和类-Hausdorff距离的手势识别 [J]. 电子学报 , 2017 , 45 ( 9 ): 2281 - 2291 .

FENG Z Q , YANG X W , XU T , et al . Gesture recognition based on combining gesture binary descriptor and hausdorff-like distance [J]. Acta Electronica Sinica , 2017 , 45 ( 9 ): 2281 - 2291 . (in Chinese)

KIM M , JEON C , KIM J , et al . A study on immersion and presence of a portable hand haptic system for immersive virtual reality [J]. Sensors , 2017 , 17 ( 5 ): 1141 - 1158 .

LI R Q , ZHENG D W , HAN Z Y , et al . MHealth: A smartphone-controlled, wearable platform for tumour treatment [J]. Materials Today , 2020 , 40 ( 11 ): 91 - 100 .

ISON M , VUJAKLIJA I , WHITSELL B , et al . High-density electromyography and motor skill learning for robust long-term control of a 7-DoF robot arm [J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering , 2015 , 24 ( 4 ): 424 - 433 .

FANG B , SUN F , LIU H , et al . A novel data glove using inertial and magnetic sensors for motion capture and robotic arm-hand teleoperation [J]. Industrial Robot: An International Journal , 2017 , 44 ( 2 ): 155 - 165 .

CHOU P H , HSU Y L , LEE W L , et al . Development of a smart home system based on multi-sensor data fusion technology [C]// Proceedings of IEEE International Conference on Applied System Innovation . Sapporo : IEEE , 2017 : 690 - 693 .

MA B , CHEN B , ZHANG Z , et al . Combat gesture classification using through-the-wall radar based on multi-domain features association [C]// Proceedings of IEEE Radar Conference . Florence : IEEE , 2020 : 1 - 5 .

李愚 , 柴国钟 , 卢纯福 , 等 . 基于增量自适应学习的在线肌电手势识别 [J]. 计算机科学 , 2019 , 46 ( 4 ): 274 - 279 .

LI Y , CHAI G Z , LU C F , et al . On-line sEMG hand gesture recognition based on incremental adaptive learning [J]. Computer Science , 2019 , 46 ( 4 ): 274 - 279 . (in Chinese)

陈国良 , 葛凯凯 , 李聪浩 . 基于多特征HMM融合的复杂动态手势识别 [J]. 华中科技大学学报(自然科学版) , 2018 , 46 ( 12 ): 42 - 47 .

CHEN G L , GE K K , LI C H . Complex dynamic gesture recognition based on multiple features and HMM fusion [J]. Journal of Huazhong University of Science and Technology(Natural Science Edition) , 2018 , 46 ( 12 ): 42 - 47 . (in Chinese)

CHEN W , ZHANG Z . Hand gesture recognition using sEMG signals based on support vector machine [C]// Proceedings of the 8th Joint International Information Technology and Artificial Intelligence Conference . Chongqing : IEEE , 2019 : 230 - 234 .

KUMAR P , GAUBA H , ROY P P , et al . Coupled HMM-based multi-sensor data fusion for sign language recognition [J]. Pattern Recognition Letters , 2017 , 86 ( 1 ): 1 - 8 .

PURUSHOTHAMAN A , PALANISWAMY S . Development of smart home using gesture recognition for elderly and disabled [J]. Journal of Computational and Theoretical Nanoscience , 2020 , 17 ( 1 ): 177 - 181 .

CHEN L , FU J , WU Y , et al . Hand gesture recognition using compact CNN via surface electromyography signals [J]. Sensors , 2020 , 20 ( 3 ): 672 - 680 .

SHIN S , KIM W Y . Skeleton-based dynamic hand gesture recognition using a part-based gru-rnn for gesture-based interface [J]. IEEE Access , 2020 , 8 ( 3 ): 50236 - 50243 .

CHEN Y , MA G , YUAN C , et al . Graph convolutional network with structure pooling and joint-wise channel attention for action recognition [J]. Pattern Recognition , 2020 , 103 ( 7 ): 107321 - 103731 .

NUNEZ J C , CABIDO R , PANTRIGO J J , et al . Convolutional neural networks and long short-term memory for skeleton-based human activity and hand gesture recognition [J]. Pattern Recognition , 2018 , 76 ( 4 ): 80 - 94 .

CHEN H , TONG R , CHEN M , et al . A hybrid cnn-svm classifier for hand gesture recognition with surface emg signals [C]// Proceedings of the 17th International Conference on Machine Learning and Cybernetics . Chengdu : IEEE , 2018 : 619 - 624 .

刘肖 , 袁冠 , 张艳梅 , 等 . 基于自适应多分类器融合的手势识别 [J]. 计算机科学 , 2020 , 47 ( 7 ): 103 - 110 .

LIU X , YUAN G , ZHANG Y M , et al . Hand gesture recognition based on self-adaptive multi-classifiers fusion [J]. Computer Science , 2020 , 47 ( 7 ): 103 - 110 . (in Chinese)

ZHU G , ZHANG L , YANG L , et al . Redundancy and attention in convolutional LSTM for gesture recognition [J]. IEEE Transactions on Neural Networks and Learning Systems , 2019 , 31 ( 4 ): 1323 - 1335 .

YAN S , XIONG Y , LIN D . Spatial temporal graph convolutional networks for skeleton-based action recognition [C]// Thirty-second AAAI conference on artificial intelligence . New Orleans : AAAI , 2018 : 1 - 10 .

白铂 , 刘玉婷 , 马驰骋 , 等 . 图神经网络 [J]. 中国科学: 数学 , 2020 , 50 ( 3 ): 367 - 384 .

BAI B , LIU Y T , MA C C , et al . Graph neural networks [J]. Scientia Sinica: Mathematica , 2020 , 50 ( 3 ): 367 - 384 . (in Chinese)

ZHANG X , CHEN X , LI Y . A framework for hand gesture recognition based on accelerometer and EMG sensors [J]. IEEE Transactions on Systems Man & Cybernetics Part A Systems & Humans , 2011 , 41 ( 6 ): 1064 - 1076 .

TIDWELL R , AKUMALLA S , KARLAPUTI S , et al . Evaluating the feasibility of EMG and bend sensors for classifying hand gestures [C]// Proceedings of the International Conference on Multimedia and Human Computer Interaction . Toronto : ISSN , 2013 : 1 - 8 .

WAHID M F , TAFRESHI R , AL-SOWAIDI M , et al . Subject-independent hand gesture recognition using normalization and machine learning algorithms [J]. Journal of Computational Science , 2018 , 27 ( 7 ): 69 - 76 .

SHARMA S , MODI S , RANA P S , et al . Hand gesture recognition using Gaussian threshold and different SVM kernels [C]// Proceedings of International Conference on Advances in Computing and Data Sciences . Dehradun : Springer , 2018 : 138 - 147 .

LI G , TANG H , SUN Y , et al . Hand gesture recognition based on convolution neural network [J]. Cluster Computing , 2019 , 22 ( 2 ): 2719 - 2729 .

KHODABANDELOU G , JUNG P G , AMIRAT Y , et al . Attention-based gated recurrent unit for gesture recognition [J]. IEEE Transactions on Automation Science and Engineering , 2020 , 18 ( 2 ): 495 - 507 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

一种多层多模态融合3D目标检测方法

一种基于因子图消元优化的激光雷达视觉惯性融合SLAM方法

基于一维卷积神经网络与循环神经网络串联的心音分析方法

面向激光光条图像修复的循环相似度映射网络

行人检测系统研究新进展及关键技术展望