LSCN:一种用于动作识别的长短时序关注网络

doi:10.3969/j.issn.0372-2112.2020.03.012

电子学报 ›› 2020, Vol. 48 ›› Issue (3): 503-509.DOI: 10.3969/j.issn.0372-2112.2020.03.012

LSCN:一种用于动作识别的长短时序关注网络

杨珂, 王敬宇, 戚琦, 孙海峰, 王晶, 廖建新

北京邮电大学网络与交换国家重点实验室, 北京 100876

收稿日期:2019-03-13 修回日期:2019-05-28 出版日期:2020-03-25
- 通讯作者:
- 王敬宇
- 作者简介:
- 杨珂男,1995年9月出生于辽宁省丹东市.北京邮电大学网络与交换国家重点实验室硕士研究生.研究方向为视频动作识别.E-mail:18811582632@163.com
- 基金资助:
- 国家自然科学基金 (No.61771068，No.61671079）; 北京市自然科学基金 (No.4182041）

LSCN: Concerning Long and Short Sequence Together for Action Recognition

YANG Ke, WANG Jing-yu, QI Qi, SUN Hai-feng, WANG Jing, LIAO Jian-xin

State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing 100876, China

Received:2019-03-13 Revised:2019-05-28 Online:2020-03-25 Published:2020-03-25
- Corresponding author:
- WANG Jing-yu
- Supported by:
- National Natural Science Foundation of China (No.61771068, No.61671079); National Natural Science Foundation of Beijing Municipality, China (No.4182041)

摘要/Abstract

摘要： 相较于图像分析，如何分析时序信息是动作识别中的一个主要问题.大多数先前的方法，如3D卷积网络、双流卷积网络，仅使用包含全局时域信息的特征作为视频的表征，忽略了局部时序特征的重要性.考虑到这样的问题，本文提出一种基于时序交互感知模块的长短时序关注网络——Long and Short Sequence Concerned Networks（LSCN），融合不同的时序信息，利用不同卷积层时序特征的交互加强对不同时序长度的动作实例的表示，兼顾长短动作实例对时序信息的需求.实验结果表明，基于3D ResNext101的LSCN在两个公共数据集（UCF101和HMDB51）上，相较于基础的网络分别有0.4%和2.9%的准确率提升.

关键词: 动作识别, 时序特征, 特征融合, 人机交互, 视频分析, 深度学习

Abstract: Compared with image analysis, how to analyze temporal information is a challenging problem in action recognition. Most of the previous methods, such as 3D CNNs (convolutional neural networks) and two-streams CNNs, only used features containing global temporal information as video representation, ignoring the importance of local temporal features. To solve this problem, we propose long and short sequence concerned networks (LSCN) based on temporal interaction perception module, which can combine different temporal information. LSCN makes use of the interactions of temporal features from different convolution layers to enhance the representation of videos and takes into account the needs of temporal information for long and short sequence actions. The results of experiments show that LSCN based on 3D ResNext101 can be generalized in two public datasets (UCF101 and HMDB51). Moreover, compared with the basic network, there are 0.4% and 2.9% accuracy improvements respectively.

Key words: action recognition, temporal feature, feature fusion, human-computer interaction, video analysis, deep learning

中图分类号:

TP391.4

杨珂, 王敬宇, 戚琦, 等. LSCN:一种用于动作识别的长短时序关注网络[J]. 电子学报, 2020, 48(3): 503-509.

YANG Ke, WANG Jing-yu, QI Qi, et al. LSCN: Concerning Long and Short Sequence Together for Action Recognition[J]. Acta Electronica Sinica, 2020, 48(3): 503-509.

参考文献

[1] WANG H,SCHMID C.Action recognition with improved trajectories[A].International Conference on Computer Vision[C].Australia:IEEE,2013.3551-3558.
[2] WANG L,Qiao Y,TANG X.MoFAP:A multi-level representation for action recognition[J].International Journal of Computer Vision,2016,119(3):254-271.
[3] SUN S,KUANG Z,SHENG L,et al.Optical flow guided feature:a fast and robust motion representation for video action recognition[A].Computer Vision and Pattern Recognition[C].USA:IEEE,2018.1390-1399.
[4] LEE M,LEE S,SON S,et al.Motion feature network:Fixed motion filter for action recognition[A].European Conference on Computer Vision(ECCV)[C].Germany:Springer,2018.387-403.
[5] LIN J,GAN C,HAN S.Temporal Shift Module for Efficient Video Understanding[DB/OL].arXiv:1811.08383,2018.
[6] ZOLFAGHARI M,SINGH K,BROX T.ECO:Efficient convolutional network for online video understanding[A].European Conference on Computer Vision[C].German:Springer,2018.713-730.
[7] DIBA A,SHARMA V,VAN GOOL L.Deep temporal linear encoding networks[A].Computer Vision and Pattern Recognition(CVPR)[C].USA:IEEE,2017.1541-1550.
[8] KARPATHY A,TODERRICI G,SHETTY S,et al.Large-scale video classification with convolutional neural networks[A].Computer Vision and Pattern Recognition[C].USA:IEEE,2014.1725-1732.
[9] DONAHUE J,ANNE HENDRICKS L,GUADARRAMA S,et al.Long-term recurrent convolutional networks for visual recognition and description[A].Computer Vision and Pattern Recognition[C].USA:IEEE,2015. 2625-2634.
[10] LEV G,SADEH G,KLEIN B,et al.Rnn fisher vectors for action recognition and image annotation[A].European Conference on Computer Vision[C].German:Springer,2016.833-850.
[11] ZHU J,ZHU Z,ZOU W.End-to-end video-level representation learning for action recognition[A].International Conference on Pattern Recognition (ICPR)[C].China:IEEE,2018:645-650.
[12] WANG L,XIONG Y,WANG Z,et al.Temporal segment networks:Towards good practices for deep action recognition[A].European Conference on Computer Vision[C].German:Springer,2016.20-36.
[13] SIMONYAN K,ZISSERMAN A.Two-stream convolutional networks for action recognition in videos[A].Neural Information Processing Systems[C].Canada:NIPS,2014.568-576.
[14] FEICHTENHOFER C,PINZ A,ZISSERMAN A.Convolutional two-stream network fusion for video action recognition[A].Computer Vision and Pattern Recognition[C].USA:IEEE,2016.1933-1941.
[15] TRAN D,BOURDEV L D,FERGUS R,et al.C3D:generic features for video analysis[J].Computer Research Repository,2014,2(7):1-8.
[16] HARA K,KATAOKA H,SATOH Y.Can spatiotemporal 3D CNNs retrace the history of 2D CNNs and ImageNet[A].Computer Vision and Pattern Recognition[C].USA:IEEE,2018.18-22.
[17] CARREIRA J,ZISSERMAN A.Quo vadis,action recognition? a new model and the kinetics dataset[A].Computer Vision and Pattern Recognition(CVPR)[C].USA:IEEE,2017.4724-4733.
[18] TRAN D,WANG H,TORRESANI L,et al.A closer look at spatiotemporal convolutions for action recognition[A].Computer Vision and Pattern Recognition[C].USA:IEEE,2018.6450-6459.
[19] QIU Z,YAO T,MEI T.Learning spatio-temporal representation with pseudo-3d residual networks[A].International Conference on Computer Vision(ICCV)[C].Italy:IEEE,2017.5534-5542.
[20] XIE S,SUN C,HUANG J,et al.Rethinking spatiotemporal feature learning for video understanding[J].Computer Research Repository,2018,27(7):1-10.
[21] DIBA A,FAYYAZ M,SHARMA V,et al.Temporal 3D convNets:New Architecture and Transfer Learning for Video Classification[DB/OL].arXiv:1711.08200,2017.
[22] WANG X,GIRSHICK R,GUPTA A,et al.Non-local neural networks[A].Computer Vision and Pattern Recognition[C].USA:IEEE,2018.7794-7803.
[23] DIBA A,FAYYAZ M,SHARMA V,et al.Spatio-temporal channel correlation networks for action classification[A].European Conference on Computer Vision[C].German:Springer,2018.284-299.
[24] SOOMRO K,ZAMIR A R,SHAH M.UCF101:A dataset of 101 human actions classes from videos in the wild[J].Computer Science,2012,3(12):2-9.
[25] KUEHNE H,JHUANG H,GARROTE E,et al.HMDB:a large video database for human motion recognition[A].International Conference on Computer Vision(ICCV)[C].Spain:IEEE,2011.2556-2563.
[26] FEICHTENHOFER C,PINZ A,WILDES R.Spatiotemporal residual networks for video action recognition[A].Neural Information Processing Systems[C].Spain:NIPS,2016.3468-3476.
[27] LI Z,GAVRILYUK K,GAVVES E,et al.VideoLSTM convolves,attends and flows for action recognition[J].Computer Vision and Image Understanding,2018,166(3):41-50.
[28] HU J F,ZHENG W S,PAN J,et al.Deep bilinear learning for rgb-d action recognition[A].European Conference on Computer Vision(ECCV)[C].German:Springer,2018.335-351.
[29] RENDLE S.Factorization machines[A].International Conference on Data Mining[C].USA:IEEE,2010.995-1000.

LSCN:一种用于动作识别的长短时序关注网络

LSCN: Concerning Long and Short Sequence Together for Action Recognition

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	李钦, 刘伟, 牛朝阳, 宝音图, 惠周勃. 低信噪比下基于分裂EfficientNet网络的雷达信号调制方式识别[J]. 电子学报, 2023, 51(3): 675-686.
[2]	范兵兵, 何庭建, 张聪炫, 陈震, 黎明. 联合遮挡约束与残差补偿的特征金字塔光流计算方法[J]. 电子学报, 2023, 51(3): 648-657.
[3]	张聿远, 闫文君, 张立民. 基于多模态特征融合网络的空时分组码识别算法[J]. 电子学报, 2023, 51(2): 489-498.
[4]	许新征, 李杉. 基于特征膨胀卷积模块的轻量化技术研究[J]. 电子学报, 2023, 51(2): 355-364.
[5]	张永梅, 孙捷. 基于动静态特征双输入神经网络的咳嗽声诊断COVID-19算法[J]. 电子学报, 2023, 51(1): 202-212.
[6]	袁海英, 成君鹏, 曾智勇, 武延瑞. Mobile_BLNet：基于Big-Little Net的轻量级卷积神经网络优化设计[J]. 电子学报, 2023, 51(1): 180-191.
[7]	王神龙, 雍宇, 吴晨睿. 基于伪孪生神经网络的低纹理工业零件6D位姿估计[J]. 电子学报, 2023, 51(1): 192-201.
[8]	李滔, 董秀成, 林宏伟. 基于深监督跨尺度注意力网络的深度图像超分辨率重建[J]. 电子学报, 2023, 51(1): 128-138.
[9]	郭晓轩, 冯其波, 冀振燕, 郑发家, 杨燕燕. 多线激光光条图像缺陷分割模型研究[J]. 电子学报, 2023, 51(1): 172-179.
[10]	贾童瑶, 卓力, 李嘉锋, 张菁. 基于深度学习的单幅图像去雾研究进展[J]. 电子学报, 2023, 51(1): 231-245.
[11]	何滢婕, 刘月峰, 边浩东, 郭威, 张小燕. 基于Informer的电池荷电状态估算及其稀疏优化方法[J]. 电子学报, 2023, 51(1): 50-56.
[12]	王硕, 王坚, 王亚男, 宋亚飞. 一种基于特征融合的恶意代码快速检测方法[J]. 电子学报, 2023, 51(1): 57-66.
[13]	吴靖, 叶晓晶, 黄峰, 陈丽琼, 王志锋, 刘文犀. 基于深度学习的单帧图像超分辨率重建综述[J]. 电子学报, 2022, 50(9): 2265-2294.
[14]	李雪莹, 王田路, 梁鹏, 王翀. 基于系统模型的用户评论中非功能需求的自动分类[J]. 电子学报, 2022, 50(9): 2079-2089.
[15]	琚长瑞, 秦晓燕, 袁广林, 李豪, 朱虹. 尺度敏感损失与特征融合的快速小目标检测方法[J]. 电子学报, 2022, 50(9): 2119-2126.