Attention-based model is a popular model in speech recognition

however it has a disadvantage that the attention-based model may produce abnormal scores. To solve this problem

this paper first proposes a forward attention model

which adopts normal attention score at the previous moment to smooth the abnormal score at the current moment. Then

the model is optimized to add constraint factors to the attention score at the previous moment to achieve the purpose of adaptive smoothing of the above abnormal scores. Then

a multi-scale forward attention model is proposed on the above model. This model introduces a multi-scale method to model the speech primitives of different levels

and then fuses the target vectors of different levels to solve the outliers of attention score. In the experiment

SwitchBoard is adopted as the training set and Hub5'00 as the test set. Compared with the baseline system

the Word Error Rate (WER) of the proposed system decreased by 14.28% relatively.