基于循环区域关注和视频帧关注的视频行为识别网络设计

doi:10.3969/j.issn.0372-2112.2020.06.002

电子学报 ›› 2020, Vol. 48 ›› Issue (6): 1052-1061.DOI: 10.3969/j.issn.0372-2112.2020.06.002

基于循环区域关注和视频帧关注的视频行为识别网络设计

桑海峰¹, 赵子裕¹, 何大阔²

1. 沈阳工业大学信息科学与工程学院, 辽宁沈阳 110870;
2. 东北大学信息科学与工程学院, 辽宁沈阳 110819

收稿日期:2019-06-03 修回日期:2019-11-22 出版日期:2020-06-25
- 通讯作者:
- 赵子裕
- 作者简介:
- 桑海峰 男,1978年生于辽宁沈阳,博士,沈阳工业大学教授,主要研究方向为视觉检测技术与图像处理. E-mail:sanghaif@163.com
- 基金资助:
- 国家自然科学基金 (No.61773105，No.61374147）; 辽宁省自然科学基金 (No.20170540675）; 辽宁省教育厅科研项目 (No.LQGD2017023）

Recurrent Region Attention and Video Frame Attention Based Video Action Recognition Network Design

SANG Hai-feng¹, ZHAO Zi-yu¹, HE Da-kuo²

1. School of Information Science&Engineering, Shenyang University of Technology, Shenyang, Liaoning 110870, China;
2. College of Information Science&Engineering, Northeastern University, Shenyang, Liaoning 110819, China

Received:2019-06-03 Revised:2019-11-22 Online:2020-06-25 Published:2020-06-25
- Corresponding author:
- ZHAO Zi-yu
- Supported by:
- National Natural Science Foundation of China (No.61773105, No.61374147); Natural Science Foundation of Liaoning Province (No.20170540675); Research Project of Education Department of Liaoning Province (No.LQGD2017023)

摘要/Abstract

摘要： 视频帧中复杂的环境背景、照明条件等与行为无关的视觉信息给行为空间特征带来了大量的冗余和噪声，一定程度上影响了行为识别的准确性.针对这一点，本文提出了一种循环区域关注单元以捕捉空间特征中与行为相关的区域视觉信息，并根据视频的时序特性又提出了循环区域关注模型.其次，本文又提出了一种能够突显整段行为视频序列中较为重要帧的视频帧关注模型，以减少异类行为视频序列间相似的前后关联给识别带来的干扰.最后，提出了一个能够端到端训练的网络模型：基于循环区域关注和视频帧关注的视频行为识别网络（Recurrent Region Attention and Video Frame Attention based video action recognition Network，RFANet）.在两个视频行为识别基准UCF101数据集和HMDB51数据集上的实验表明，本文提出的端到端网络RFANet能够可靠地识别出视频中行为的所属类别.受双流结构启发，本文构建了双模态RFANet网络.在相同的训练环境下，双模态RFANet网络在两个数据集上达到了最优的性能.

关键词: 行为识别, 循环区域关注, 视频帧关注, 循环神经网络

Abstract: In video frames, the complex environment background, lighting conditions and other visual information unrelated to action bring a lot of redundancy and noise to action spatial feature, which affects the accuracy of action recognition to some extent. In view of this, this paper proposes a recurrent region attention cell to capture the visual information of the region related to the action in spatial features. Based on the sequence nature of video, a recurrent region attention model (RRA) is proposed. Secondly, this paper proposes a video frame attention model (VFA) that can highlight the more important frames in the video sequence of the whole action, so as to reduce the interference brought by the similar before and after correlation between video sequences of different actions. Finally, this paper presents a network model which can perform end-to-end training: recurrent region attention and video frame attention based video action recognition network (RFANet). Experiments on two video action recognition benchmark UCF101 dataset and HMDB51 dataset show that the RFANet proposed in this paper can reliably identify the category of action in the video. Inspired by the two-stream structure, we construct a two-modalities RFANet network. In the same training conditions, the two-modalities RFANet network achieved optimal performance on both datasets.

Key words: action recognition, recurrent region attention, video frame attention, recurrent neural network

中图分类号:

TP391

桑海峰, 赵子裕, 何大阔. 基于循环区域关注和视频帧关注的视频行为识别网络设计[J]. 电子学报, 2020, 48(6): 1052-1061.

SANG Hai-feng, ZHAO Zi-yu, HE Da-kuo . Recurrent Region Attention and Video Frame Attention Based Video Action Recognition Network Design[J]. Acta Electronica Sinica, 2020, 48(6): 1052-1061.

参考文献

[1] Ioffe S,Szegedy C.Batch normalization:accelerating deep network training by reducing internal covariate shift[A].International Conference on Machine Learning[C].Lille,France:International Machine Learning Society,2015.448-456.
[2] Wang L,Xiong Y,Wang Z,et al.Temporal segment networks:towards good practices for deep action recognition[A].European Conference on Computer Vision[C].Amsterdam,Netherlands:Springer International Publishing,2016.20-36.
[3] Tran D,Bourdev L,Fergus R,et al.Learning spatiotemporal features with 3D convolutional networks[A].International Conference on Computer Vision[C].Santiago,Chile:IEEE,2015.4489-4497.
[4] Hochreiter S,Schmidhuber J.Longshort-term memory[J].Neural Computation,1997,9(8):1735-1780.
[5] Donahue J,Hendricks L A,Guadarrama S,et al.Long-termrecurrent convolutional networks for visual recognition and description[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017,39(4):677-691.
[6] Brox T,Bruhn A,Papenberg N,et al.High accuracy optical flow estimation based on a theory for warping[J].Computer Vision,2004,3024(10):25-36.
[7] Xu K,Ba J,Kiros R,et al.Show,attend and tell:neural image caption generation with visual attention[A].International Conference on Machine Learning[C].Lille,France:International Machine Learning Society,2015.2048-2057.
[8] Sharma S,Kiros R,Salakhutdinov R.Action Recognition Using Visual Attention[DB/OL].https://arxiv.org/abs/1511.04119,2015-11-12.
[9] Yan S,Smith J S,Lu W,et al.Hierarchical multi-scale attention networks for action recognition[J].Signal Processing:Image Communication,2018,61:73-84.
[10] T Yu,C Guo,L Wang,et al.Joint spatial-temporal attention for action recognition[J].Computer Science,2018,112(2018):226-233.
[11] Bahdanau D,Cho K,Bengio Y.Neural Machine Translation by Jointly Learning to Align and Translate[DB/OL].https://arxiv.org/abs/1409.0473,2014-09-01.
[12] Schuster M,Paliwal KK.Bidirectional recurrent neural networks[J].IEEE Transactions on Signal Processing, 1997,45(17):2673-2681.
[13] Soomro K,Zamir A R,Shah M.UCF101:A dataset of 101 human actions classes from videos in the wild[[DB/OL]].https://arxiv.org/abs/1212.0402,2012-12-03.
[14] Kuehne H,Jhuang H,Garrote E,et al.HMDB:a large video database for human motion recognition[A].International Conference on Computer Vision[C].Barcelona,Spain:IEEE,2011.2556-2563.
[15] Deng J,Dong W,Socher R,et al.ImageNet:A large-scale hierarchical image database[A].Computer Vision and Pattern Recognition[C].Miami,FL,USA:IEEE,2009.248-255.
[16] Xiang L,Chuang G,et al.Multimodal keyless attention fusion for video classification[A].32nd AAAI Conference on Artificial Intelligence[C].New Orleans,Louisiana,USA:AAAI,2018.7202-7209.
[17] Yuan Y,Wang D,Wang Q.Memory-Augmented Temporal Dynamic Learning for Action Recognition[DB/OL].https://arxiv.org/abs/1904.13080,2019-4-30.
[18] Fan L,Huang W,Gan C,et al.End-to-End Learning of Motion Representation for Video Understanding[A].Computer Vision and Pattern Recognition[C].Salt Lake City,UT,USA:IEEE,2018.6016-6025.
[19] Sengupta B,Qian Y.Pillar Networks++:Distributed Non-parametric Deep and Wide Networks[DB/OL].https://arxiv.org/abs/1708.06250,2017-08-18.
[20] Peng X,Wang L,Wang X,et al.Bag of visual words and fusion methods for action recognition:Comprehensive study and good practice[J].Computer Vision and Image Understanding,2016,150(2016):109-125.
[21] Li Z,Gavrilyuk K,Gavves E,et al.VideoLSTM convolves,attends and flows for action recognition[J].Computer Vision and Image Understanding,2018,166(2018):41-50.
[22] Wang L,Yu Q,Tang X.Action recognition with trajectory-pooled deep-convolutional descriptors[A].Computer Vision and Pattern Recognition[C].Boston,America:IEEE,2015.4305-4314.
[23] Carreira J,Zisserman A.Quo Vadis,Action Recognition? A New Model and the Kinetics Dataset[A].Computer Vision and Pattern Recognition[C].Hawaii,America:IEEE,2017.6299-6308.
[24] Feichtenhofer C,Pinz A,Wildes R P.Spatiotemporal residual networks for video action recognition[A].Advances in Neural Information Processing Systems[C].Barcelona:NIPS,2016.3468-3476.
[25] Cai Z,Wang L,Peng X,et al.Multi-view super vector for action recognition[A].Computer Vision and Pattern Recognition[C].Columbus,America:IEEE,2014.596-603.

基于循环区域关注和视频帧关注的视频行为识别网络设计

Recurrent Region Attention and Video Frame Attention Based Video Action Recognition Network Design

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	杨利平, 侯振威, 辜小花, 郝峻永. 弱标签声音事件检测的空间-通道特征表征与自注意池化[J]. 电子学报, 2023, 51(2): 297-306.
[2]	邓海刚, 王传旭, 李成伟, 林晓萌. 深度学习框架下群组行为识别算法综述[J]. 电子学报, 2022, 50(8): 2018-2036.
[3]	徐兴荣, 刘聪, 李婷, 郭娜, 任崇广, 曾庆田. 基于双向准循环神经网络和注意力机制的业务流程剩余时间预测方法[J]. 电子学报, 2022, 50(8): 1975-1984.
[4]	冀振燕, 韩梦豪, 宋晓军, 冯其波. 面向激光光条图像修复的循环相似度映射网络[J]. 电子学报, 2022, 50(5): 1234-1242.
[5]	袁冠, 邴睿, 刘肖, 代伟, 张艳梅, 蔡卓. 基于时空图神经网络的手势识别[J]. 电子学报, 2022, 50(4): 921-931.
[6]	肖斌, 陈嘉博, 毕秀丽, 张俊辉, 李伟生, 王国胤, 马旭. 基于一维卷积神经网络与循环神经网络串联的心音分析方法[J]. 电子学报, 2022, 50(10): 2425-2432.
[7]	高俊涛, 王梅, 徐光会, 刘聪. 正则语言推断综述[J]. 电子学报, 2021, 49(12): 2479-2489.
[8]	王传旭, 薛豪. 基于GFU和分层LSTM的组群行为识别研究方法[J]. 电子学报, 2020, 48(8): 1465-1471.
[9]	胡正平, 刁鹏成, 张瑞雪, 李淑芳, 赵梦瑶. 3D多支路聚合轻量网络视频行为识别算法研究[J]. 电子学报, 2020, 48(7): 1261-1268.
[10]	丰艳, 张甜甜, 王传旭. 基于伪3D残差网络与交互关系建模的群组行为识别方法[J]. 电子学报, 2020, 48(7): 1269-1275.
[11]	王传旭, 胡小悦, 孟唯佳, 闫春娟. 基于多流架构与长短时记忆网络的组群行为识别方法研究[J]. 电子学报, 2020, 48(4): 800-807.
[12]	尤洪峰, 田生伟, 禹龙, 吕亚龙. 基于Word Embedding的遥感影像检测分割[J]. 电子学报, 2020, 48(1): 75-83.
[13]	郑兴华, 孙喜庆, 吕嘉欣, 鲜征征, 李磊. 基于深度学习和智能规划的行为识别[J]. 电子学报, 2019, 47(8): 1661-1668.
[14]	查雄, 彭华, 秦鑫, 李天昀, 李广. 基于循环神经网络的卫星幅相信号调制识别与解调算法[J]. 电子学报, 2019, 47(11): 2443-2448.
[15]	吕品, 于文兵, 汪鑫, 计春雷, 周曦民. 异构分类器堆叠泛化及其在恶意评论检测中的应用[J]. 电子学报, 2019, 47(10): 2228-2234.