基于多流空间注意力图卷积SRU网络的骨架动作识别

doi:10.12263/DZXB.20210416

电子学报 ›› 2022, Vol. 50 ›› Issue (7): 1579-1585.DOI: 10.12263/DZXB.20210416

基于多流空间注意力图卷积SRU网络的骨架动作识别

赵俊男, 佘青山, 孟明, 陈云

杭州电子科技大学自动化学院，浙江杭州 310018

收稿日期:2021-03-29 修回日期:2021-12-30 出版日期:2022-07-25
- 作者简介:
- 赵俊男男，1996年12月出生于浙江湖州.现为杭州电子科技大学自动化学院硕士研究生，研究方向为3D骨架动作识别、人体姿态估计.E-mail: 663261972@qq.com
  佘青山男，1980年2月出生于湖北松滋. 现为杭州电子科技大学教授，主要研究方向为机器学习与脑-机接口、康复机器人、图像/视频处理与分析.E-mail: qsshe@hdu.edu.cn
- 基金资助:
- 国家自然科学基金 (61871427); 浙江省自然科学基金重点项目 (LZ22F010003)

Skeleton Action Recognition Based on Multi-Stream Spatial Attention Graph Convolutional SRU Network

ZHAO Jun-nan, SHE Qing-shan, MENG Ming, CHEN Yun

College of Automation, Hangzhou Dianzi University, Hangzhou, Zhejiang 310018, China

Received:2021-03-29 Revised:2021-12-30 Online:2022-07-25 Published:2022-07-30
- Supported by:
- National Natural Science Foundation of China (61871427); Key Program of National Natural Science Foundation of Zhejiang Province, China (LZ22F010003)

摘要/Abstract

摘要：

基于骨架的动作识别越来越受到重视.针对现有算法推理速度慢、数据模式单一等问题，本文提出了一种轻量且高效的方法.该网络在简单循环单元（Simple Recurrent Unit，SRU）中嵌入图卷积算子构建图卷积SRU（GC-SRU）模型，来捕获数据的时空域信息.同时，为了加强节点间的区分，采用空间注意力网络和多流数据融合方式，将GC-SRU拓展成多流空间注意力图卷积SRU（MSAGC-SRU）.最后，在公开数据集上进行实验分析.结果表明，本文方法在Northwestern-UCLA上的分类准确率达到了93.1%，模型FLOPs为4.4G；NTU RGB+D上的分类准确率在CV、CS评估协议下分别达到92.7%和87.3%，模型FLOPs为21.3G，达到了计算效率和分类精度的良好平衡.

关键词: 动作识别, 图卷积, 注意力机制, 数据融合

Abstract:

Action recognition with skeleton data has attracted more attention. In order to solve the problems of low reasoning speed and single data mode of most algorithms, a lightweight and efficient method is proposed. The network embeds the graph convolution operator in the simple recurrent unit(SRU) to construct the graph convolutional SRU(GC-SRU), which can capture the spatial-temporal information of data. Meanwhile, to enhance the distinction between nodes, spatial attention network and multi-stream data fusion are used to expand GC-SRU into multi-stream spatial attention graph convolutional SRU(MSAGC-SRU). Finally, the proposed method is evaluated on two public datasets. Experimental results show that the classification accuracy of our method on Northwestern-UCLA reaches 93.1% and the FLOPs of the model is 4.4G. The accuracy on NTU RGB+D reaches 92.7% and 87.3% under the CV and CS evaluation protocols, respectively, and the FLOPs of the model is 21.3G. The proposed model has achieved good trade-off between computational efficiency and classification accuracy.

Key words: action recognition, graph convolution, attention mechanism, data fusion

中图分类号:

TP391.4

赵俊男, 佘青山, 孟明, 陈云. 基于多流空间注意力图卷积SRU网络的骨架动作识别[J]. 电子学报, 2022, 50(7): 1579-1585.

ZHAO Jun-nan, SHE Qing-shan, MENG Ming, CHEN Yun. Skeleton Action Recognition Based on Multi-Stream Spatial Attention Graph Convolutional SRU Network[J]. Acta Electronica Sinica, 2022, 50(7): 1579-1585.

图/表 9

参考文献 23

1	罗会兰, 童康, 孔繁胜. 基于深度学习的视频中人体动作识别进展综述[J].电子学报, 2019, 47(5): 1162-1173.
	LUOHui-lan, TONGKang, KONGFan-sheng.Review of human action recognition in videos based on deep learning[J]. Acta Electronica Sinica, 2019, 47(5): 1162-1173. (in Chinese)
2	YANS, XIONGY, LIND, et al. Spatial temporal graph convolutional networks for skeleton-based action recognition[C]//Proceedings of the AAAI Conference on Artificial Intelligence. New Orleans: AAAI, 2018: 7444-7452.
3	SIC, CHENW, WANGW, et al. An attention enhanced graph convolutional LSTM network for skeleton-based action recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE,2019: 1227-1236.
4	SHIL, ZHANGY, CHENGJ, et al. Skeleton-based action recognition with multi-stream adaptive graph convolutional networks[J]. IEEE Transactions on Image Processing, 2020, 29: 9532-9545.
5	LEIT, ZHANGY, WANGS I, et al. Simple recurrent units for highly parallelizable recurrence[EB/OL]. (2018)[2021]. .
6	SHEQ, MUG, GANH, et al. Spatio-temporal SRU with global context-aware attention for 3D human action recognition[J]. Multimedia Tools and Applications, 2020, 79(17-18): 12349-12371.
7	PARKC, LEEC, HONGL, et al. S2-Net: Machine reading comprehension with SRU based self-matching networks[J]. ETRI Journal, 2019, 41(3): 371-382.
8	ZHUW, LANC, XINGJ, et al. Co-occurrence feature learning for skeleton based action recognition using regularized deep LSTM networks[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Phoenix: AAAI, 2016: 3697-3703.
9	ZHANGL, ZHUG, MEIL, et al. Attention in convolutional LSTM for gesture recognition[C]//Proceedings of the Advances in Neural Information Processing Systems. Montréal: Curran Associates Inc., 2018: 1953-1962.
10	Di GangiM A, FedericoM. Deep neural machine translation with weakly-recurrent units[EB/OL]. (2018)[2021]. .
11	SONGS, LANC, XINGJ, et al. An end-to-end spatio-temporal attention model for human action recognition from skeleton data[C]//Proceedings of the AAAI Conference on Artificial Intelligence. San Francisco: AAAI, 2017:4263-4270.
12	朱红蕾, 朱昶胜, 徐志刚.人体行为识别数据集研究进展[J]. 自动化学报, 2018, 44(6): 978-1004.
	ZHUHong-lei, ZHUChang-sheng, XUZhi-gang. Research advances on human activity recognition datasets[J].Acta Automatica Sinica,2018,44(6):978-1004.(in Chinese)
13	XIEC, LIC, ZHANGB, et al. Memory attention networks for skeleton-based action recognition[EB/OL]. (2018)[2021]. .
14	WANGJ, NIEX, XIAY, et al. Cross-view action modeling, learning and recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Columbus: IEEE, 2014: 2649-2656.
15	CHENGK, ZHANGY, HEX, et al. Skeleton-based action recognition with shift graph convolutional Network[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 183-192.
16	穆高原. 基于深度学习的危险驾驶行为识别研究[D]. 杭州: 杭州电子科技大学, 2020.
	MUGao-yuan. Study on dangerous driving behavior recognition based on deep learning[D]. Hangzhou: Hangzhou Dianzi University, 2020. (in Chinese)
17	VemulapalliR, ArrateF, ChellappaR. Human action recognition by representing 3D skeletons as points in a lie group[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Columbus: IEEE, 2014: 588-595.
18	WANGJ, LIUZ, WUY, et al. Learning actionlet ensemble for 3D human action recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 36(5): 914-927.
19	DUY, WANGW, WANGL. Hierarchical recurrent neural network for skeleton based action recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 1110-1118.
20	LIUM, LIUH, CHENC. Enhanced skeleton visualization for view invariant human action recognition[J]. Pattern Recognition, 2017, 68: 346-362.
21	LEED Kim, KANGS, LEES. Ensemble deep learning for skeleton-based action recognition using temporal sliding LSTM networks[C]//Proceedings of the IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 1012-1020.
22	SHIL, ZHANGY, CHENGJ, et al. Two-stream adaptive graph convolutional networks for skeleton-based action recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 12026-12035.
23	SHAHROUDYA, LIUJ, NGT T, et al. Nturgb+d: A large scale dataset for 3D human activity analysis[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1010-1019.

Methods	Accuracy(%)	FLOPs(G)
Lie group^[17]	74.2	—
Actionletensemble^[18]	76.0	—
HBRNN-L^[19]	78.5	—
VisualizaionCNN^[20]	86.1	—
Ensemble TS-LSTM^[21]	89.2	—
AGC-LSTM^[3]	93.3	10.9
MSAGC-SRU	93.1	4.4

Methods	Accuracy(%)	FLOPs(G)
Lie group^[17]	74.2	—
Actionletensemble^[18]	76.0	—
HBRNN-L^[19]	78.5	—
VisualizaionCNN^[20]	86.1	—
Ensemble TS-LSTM^[21]	89.2	—
AGC-LSTM^[3]	93.3	10.9
MSAGC-SRU	93.1	4.4

Methods	Accuracy(%)	FLOPs(G)
SRU	81.3	—
GC-SRU	84.8	3.0
MSGC-SRU	90.3	3.5
SAGC-SRU	90.1	3.9
MSAGC-SRU	93.1	4.4

Methods	Accuracy(%)	FLOPs(G)
SRU	81.3	—
GC-SRU	84.8	3.0
MSGC-SRU	90.3	3.5
SAGC-SRU	90.1	3.9
MSAGC-SRU	93.1	4.4

Methods	Training time(s)	Testing time(s)
SRU	5.93	1.00
GC-SRU	6.08	1.86
MSGC-SRU	6.38	2.04
AGC-LSTM^[3]	14.30	4.74
SAGC-SRU	6.49	2.04
MSAGC-SRU	6.77	2.36

基于多流空间注意力图卷积SRU网络的骨架动作识别

Skeleton Action Recognition Based on Multi-Stream Spatial Attention Graph Convolutional SRU Network

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 9

参考文献 23

相关文章 15

编辑推荐

Metrics

Methods	CV(%)	CS(%)	FLOPs(G)
HBRNN-L^[19]	64.0	59.1	—
Deep-LSTM^[23]	67.3	60.7	—
ST-GCN^[2]	88.3	81.5	16.2
AGC-LSTM^[3]	95.0	89.2	54.4
2s-Adaptive GCN^[22]	95.1	88.5	35.8
MS-AAGCN^[4]	96.2	90.0	71.6
MSAGC-SRU	92.7	87.3	21.3

[1]	毛国君, 王者浩, 黄山, 王翔. 基于剪边策略的图残差卷积深层网络模型[J]. 电子学报, 2022, 50(9): 2205-2214.
[2]	刘群, 谭洪胜, 张优敏, 王国胤. 基于元路径的动态异质网络表示学习[J]. 电子学报, 2022, 50(8): 1830-1839.
[3]	张志昌, 于沛霖, 庞雅丽, 朱林, 曾扬扬. SMGN：用于对话状态跟踪的状态记忆图网络[J]. 电子学报, 2022, 50(8): 1851-1858.
[4]	孔玮, 刘云, 李辉, 崔雪红, 杨浩冉. 基于全局自适应有向图的行人轨迹预测[J]. 电子学报, 2022, 50(8): 1905-1916.
[5]	王延达, 陈炜通, 皮德常, 岳琳. 一种自适应记忆神经网络多跳读取与覆盖度机制结合的药物推荐模型[J]. 电子学报, 2022, 50(4): 943-953.
[6]	张凌明, 赵悦, 李鹏程, 刘洋, 高陈强. 基于局部注意力机制的三维牙齿模型分割网络[J]. 电子学报, 2022, 50(3): 681-690.
[7]	伍邦谷, 张苏林, 石红, 朱鹏飞, 王旗龙, 胡清华. 基于多分支结构的不确定性局部通道注意力机制[J]. 电子学报, 2022, 50(2): 374-382.
[8]	周登文, 李文斌, 李金新, 黄志勇. 一种轻量级的多尺度通道注意图像超分辨率重建网络[J]. 电子学报, 2022, 50(10): 2336-2346.
[9]	王波, 黄冕, 刘利军, 黄青松, 单文琦. 基于多层聚焦Inception-V3卷积网络的细粒度图像分类[J]. 电子学报, 2022, 50(1): 72-78.
[10]	赵琰, 赵凌君, 匡纲要. 基于注意力机制特征融合网络的SAR图像飞机目标快速检测[J]. 电子学报, 2021, 49(9): 1665-1674.
[11]	周东明, 张灿龙, 李志欣, 王智文. 基于多层级视觉融合的图像描述模型[J]. 电子学报, 2021, 49(7): 1286-1290.
[12]	常琦, 孟瑶, 赵恒. 基于概率成像的二维波束聚焦冲击定位方法研究[J]. 电子学报, 2021, 49(7): 1339-1345.
[13]	谢佳龙, 张波涛, 吕强. 一种基于双流融合3D卷积神经网络的动态头势识别方法[J]. 电子学报, 2021, 49(7): 1363-1369.
[14]	李志欣, 孙亚茹, 唐素勤, 张灿龙, 马慧芳. 双路注意力引导图卷积网络的关系抽取[J]. 电子学报, 2021, 49(2): 315-323.
[15]	江泽涛, 钱艺, 伍旭, 张少钦. 一种基于ARD‑GAN的低照度图像增强方法[J]. 电子学报, 2021, 49(11): 2160-2165.