In video frames

the complex environment background

lighting conditions and other visual information unrelated to action bring a lot of redundancy and noise to action spatial feature

which affects the accuracy of action recognition to some extent. In view of this

this paper proposes a recurrent region attention cell to capture the visual information of the region related to the action in spatial features. Based on the sequence nature of video

a recurrent region attention model (RRA) is proposed. Secondly

this paper proposes a video frame attention model (VFA) that can highlight the more important frames in the video sequence of the whole action

so as to reduce the interference brought by the similar before and after correlation between video sequences of different actions. Finally

this paper presents a network model which can perform end-to-end training: recurrent region attention and video frame attention based video action recognition network (RFANet). Experiments on two video action recognition benchmark UCF101 dataset and HMDB51 dataset show that the RFANet proposed in this paper can reliably identify the category of action in the video. Inspired by the two-stream structure

we construct a two-modalities RFANet network. In the same training conditions

the two-modalities RFANet network achieved optimal performance on both datasets.