融合语言模型的端到端中文语音识别算法

doi:10.12263/DZXB.20201187

电子学报 ›› 2021, Vol. 49 ›› Issue (11): 2177-2185.DOI: 10.12263/DZXB.20201187

融合语言模型的端到端中文语音识别算法

吕坤儒¹, 吴春国¹^,²^,³, 梁艳春¹^,²^,³, 袁宇平¹, 任智敏¹, 周柚¹^,², 时小虎¹^,²^,³

^1.吉林大学计算机科学与技术学院，吉林长春 130012
^2.吉林大学符号计算与知识工程教育部重点实验室，吉林长春 130012
^3.珠海科技学院计算机学院，广东珠海 519041

收稿日期:2020-10-23 修回日期:2021-07-20 出版日期:2021-11-25 发布日期:2021-11-25
作者简介:吕坤儒　男，1993年生，河南安阳人.吉林大学软件学院硕士研究生.研究方向：机器学习.E-mail:892539843@qq.com
时小虎（通信作者）男，1974年生，河北玉田人.吉林大学计算机科学与技术学院教授.研究方向：机器学习.E-mail:shixh@jlu.edu.cn
基金资助:
国家自然科学基金(61972174);吉林省预算内基本建设资金(2021C044-1);广东省国际科技合作项目(2020A0505100018);吉林省自然科学基金(20200201163JC)

An End-to-End Chinese Speech Recognition Algorithm Integrating Language Model

Kun-ru LÜ¹, Chun-guo WU¹^,²^,³, Yan-chun LIANG¹^,²^,³, Yu-ping YUAN¹, Zhi-min REN¹, You ZHOU¹^,², Xiao-hu SHI¹^,²^,³

^1.College of Computer Science and Technology，Jilin University，Changchun，Jilin 130012，China
^2.Ministry of Education Key Laboratory of Symbol Computation and Knowledge Engineering，Jilin University，Changchun，Jilin 130012，China
^3.School of Computer Science，Zhuhai College of Science and Technology，Zhuhai，Guangdong 519041，China

Received:2020-10-23 Revised:2021-07-20 Online:2021-11-25 Published:2021-11-25

摘要/Abstract

摘要：

为了解决语音识别模型在识别中文语音时鲁棒性差，缺少语言建模能力而无法有效区分同音字或近音字的不足，本文提出了融合语言模型的端到端中文语音识别算法.算法建立了一个基于深度全序列卷积神经网络和联结时序分类的从语音到拼音的语音识别声学模型，并借鉴Transformer的编码模型，构建了从拼音到汉字的语言模型，之后通过设计语音帧分解模型将声学模型的输出和语言模型的输入相连接，克服了语言模型误差梯度无法传递给声学模型的难点，实现了声学模型和语言模型的联合训练.为验证本文方法，在实际数据集上进行了测试.实验结果表明，语言模型的引入将算法的字错误率降低了21%，端到端的联合训练算法起到了关键作用，其对算法的影响达到了43%.和已有5种主流算法进行比较的结果表明本文方法的误差明显低于其他5种对比模型，与结果最好的DeepSpeech2模型相比字错误率降低了28%.

关键词: 语音识别, 联结时序分类, 语言模型, 声学模型, 语音帧分解

Abstract:

To address the problems of poor robustness, lack of language modeling ability and inability to distinguish between homophones or near-tone characters effectively in the recognition of Chinese speech, an end-to-end Chinese speech recognition algorithm integrating language model is proposed. Firstly, an acoustic model from speech to Pinyin is established based on Deep Fully Convolutional Neural Network (DFCNN) and Connectionist Temporal Classification (CTC). Then the language model from Pinyin to Chinese character is constructed by using the encoder of Transformer. Finally, the speech frame decomposition model is designed to link the output of the acoustic model with the input of the language model, which overcomes the difficulty that the gradient of loss function cannot be passed from the language model to the acoustic model, and realizes the end-to-end training of the acoustic model and the language model. Real data sets are applied to verify the proposed method. Experimental results show that the introduction of language model reduces the word error rate (WER) of the algorithm by 21%, and the end-to-end integrating training algorithm plays a key role, which improves the performance by 43%. Compared with five up-to-date algorithms, our method achieves a 28% WER, lower than that of the best model among comparison methods—DeepSpeech2.

Key words: speech recognition, CTC, language model, acoustic model, speech frame decomposition

中图分类号:

TP18

吕坤儒, 吴春国, 梁艳春, 袁宇平, 任智敏, 周柚, 时小虎. 融合语言模型的端到端中文语音识别算法[J]. 电子学报, 2021, 49(11): 2177-2185.

Kun-ru LÜ, Chun-guo WU, Yan-chun LIANG, Yu-ping YUAN, Zhi-min REN, You ZHOU, Xiao-hu SHI. An End-to-End Chinese Speech Recognition Algorithm Integrating Language Model[J]. Acta Electronica Sinica, 2021, 49(11): 2177-2185.

图/表 7

参考文献 28

1	杨明浩,高廷丽,陶建华,等.对话意图及语音识别错误对交互体验的影响[J]. 软件学报, 2016, 27(S2): 69 - 75.
	Yang MH, Gao TL, Tao JH, et al. Error analysis of intention classification and speech recognition in human-computer dialog [J]. Journal of Software, 2016, 27(S2): 69 - 75. (in Chinese)
2	Rodriguez E, RuiZ B, Garcia-Crespo A, et al. Speech/speaker recognition using a HMM/GMM hybrid model [A]. International Conference on Audio-and Video-Based Biometric Person Authentication [C]. Berlin, Heidelberg: Springer, 1997. 227 - 234.
3	Mohamed AR, Sainath TN, Dahl G, et al. Deep belief networks using discriminative features for phone recognition [A]. 2011 IEEE International Conference on Acoustics, Speech and Signal Processing [C]. Prague, Czech Republic: IEEE, 2011. 5060 - 5063.
4	Yu D, Deng L. Deep learning and its applications to signal and information processing [J]. IEEE Signal Processing Magazine, 2011, 28(1): 145 - 154.
5	Graves A, Mohamed AR, Hinton G. Speech recognition with deep recurrent neural networks [A]. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) [C]. Vancouver, Canada: IEEE, 2013. 6645 - 6649.
6	Sak H, Senior A, Beaufays F. Long short-term memory based recurrent neural network architectures for large vocabulary speech recognition [A]. The 15th Annual Conference of the International Speech Communication Association [C]. Singapore: ISCA, 2014.338 - 342.
7	Abdel-Hamid O, Mohamed AR, Jiang H, et al. Convolutional neural networks for speech recognition [J]. IEEE/ACM Transactions on Audio Speech & Language Processing, 2014, 22(10): 1533 - 1545.
8	Graves A, Fernández S, Gomez F, et al. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks [A]. International Conference on Machine Learning, ICML 2006 [C]. Pittsburgh, PA: ACM, 2006. 369 - 376.
9	Zhang Y, Pezeshki M, Brakel P, et al. Towards end-to-end speech recognition with deep convolutional neural networks [A]. The 17th Annual Conference of the International Speech Communication Association [C]. San Francisco, CA: ISCA, 2016. 410 - 414.
10	Yang XD, Wang WZ, Yang HW, et al. Simple data augmented transformer end-to-end Tibetan speech recognition [A]. IEEE 3rd International Conference on Information Communication and Signal Processing [C]. NY: IEEE, 2020. 148 - 152.
11	Chang HJ, Liu AH, Lee HY, et al. End-to-end whispered speech recognition with frequency-weighted approaches and pseudo whisper pre-training [A]. IEEE Spoken Language Technology Workshop [C]. NY: IEEE, 2021. 186 - 193.
12	Fan CH, Yi JY, Tao JH, et al. Gated recurrent fusion with joint training framework for robust end-to-end speech recognition [J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021, 29: 198 - 209.
13	Graves A, Jürgen S. Framewise phoneme classification with bidirectional LSTM and other neural network architectures [J]. Neural Networks, 2005, 18(5-6): 602 - 610.
14	Sainath TN, Vinyals O, Senior A, et al. Convolutional, long short-term memory, fully connected deep neural networks [A]. 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) [C]. NY: IEEE, 2015. 4580 - 4584.
15	Amodei D, Ananthanarayanan S, Anubhai R, et al. Deep speech 2: end-to-end speech recognition in English and Mandarin [A]. International Conference on Machine Learning 2016 [C]. NY: ACM, 2016. 173 - 182.
16	王海坤,潘嘉,刘聪.语音识别技术的研究进展与展望[J]. 电信科学报, 2018, 2: 1 - 11.
	Wang HK, Pan J, Liu C. Research development and forecast of automatic speech recognition technologies [J]. Telecommunications Science, 2018, 2: 1 - 11. (in Chinese)
17	Kannan A, Wu YH, Nguyen P, et al. An analysis of incorporating an external language model into a sequence-to-sequence model [A]. 2017 IEEE International Conference on Acoustics, Speech and Signal Processing [C]. Calgary, Canada: IEEE, 2017. 5824-5828.
18	Gulcehre C, Firat O, Xu K, et al. On using monolingual corpora in neural machine translation [OL]. http://arxiv.org/abs/1503.03535, 2015.
19	Anuroop S, Heewoo J, Sanjeev S, et al. Cold fusion: Training seq2seq models together with language models [A]. The 19th Annual Conference of the International Speech Communication Association [C]. Hyderabad, India: ISCA, 2018. 387 - 391.
20	Toshniwal S, Kannan A, Chiu CC, et al. A comparison of techniques for language model integration in encoder-decoder speech recognition [A]. IEEE Workshop on Spoken Language Technology [C]. Athens, Greece: IEEE, 2018. 369 - 375.
21	Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need [A]. Advances in Neural Information Processing Systems[C]. Long Beach, CA: MIT Press, 2017. 5998-6008.
22	Bu H, Du J, Na X, et al. Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline [A]. The 20th Conference of the Oriental Chapter of the International Coordinating Committee on Speech Databases and Speech I/O Systems and Assessment (O-COCOSDA) [C]. Seoul, South Korea: IEEE, 2017. 58 - 62.
23	Wang CH, Zhang M, Ma SP, et al. Automatic online news issue construction in Web environment [A]. The 17th International World Wide Web Conference [C]. Beijing, China: ACM, 2008. 457 - 466.
24	Kingma D, Ba J. Adam: a method for stochastic optimization [A]. IEEE 17th International Conference on Computational Science and Engineering (CSE) [C]. Chengdu, China: IEEE, 2014. 563 - 568.
25	Sergey I, Christian S. Batch normalization: accelerating deep network training by reducing internal covariate shift [A]. International Conference on Machine Learning 2015 [C]. Lille France: ACM, 2015. 448 - 456.
26	Srivastava N, Hinton G, Krizhevsky A, et al. Dropout: a simple way to prevent neural networks from overfitting [J]. Journal of Machine Learning Research, 2014, 15(1): 1929 - 1958.
27	Graves A, Jaitly N. Towards end-to-end speech recognition with recurrent neural networks [A]. International Conference on Machine Learning[C]. Beijing, China: JMLR, 2014, 32(2): 1764 - 1772.
28	胡章芳,徐轩,付亚芹,等.基于ResNet-BLSTM的端到端语音识别[J].计算机工程与应用, 2020, 56(18): 124-130.
	Hu ZF, Xu X, Fu YQ, et al. End to end speech recognition based on ResNet-BLSTM[J]. Computer Engineering and Applications, 2020, 56(18): 124 - 130. (in Chinese)

模型	网络结构参数	声学建模单元
DFCNN	5×2D_CNN+2×FNN	汉字
DFCNN +Transformer Encoder	5×2D_CNN+2×FNN +Transformer Encoder	带音调音节
BLSTM-DNN-CTC^［27］	5×DNN+3×BLSTM+2×FNN	汉字
CNN-LSTM -DNN-CTC^［14］	3×CNN+3×LSTM+4×FNN	汉字
DCNN-DNN-CTC^［9］	10×CNN_maxout +3×FNN_maxout	汉字
ResNet-BLSTM^［28］	8×CNN（4×Res）+2×BLSTM	汉字
Deep Speech2^［15］	3×1D_CNN+3×GRU+FNN	汉字
ECSRILM	5×2D_CNN+2×FNN +Transformer Encoder	汉字

模型	网络结构参数	声学建模单元
DFCNN	5×2D_CNN+2×FNN	汉字
DFCNN +Transformer Encoder	5×2D_CNN+2×FNN +Transformer Encoder	带音调音节
BLSTM-DNN-CTC^［27］	5×DNN+3×BLSTM+2×FNN	汉字
CNN-LSTM -DNN-CTC^［14］	3×CNN+3×LSTM+4×FNN	汉字
DCNN-DNN-CTC^［9］	10×CNN_maxout +3×FNN_maxout	汉字
ResNet-BLSTM^［28］	8×CNN（4×Res）+2×BLSTM	汉字
Deep Speech2^［15］	3×1D_CNN+3×GRU+FNN	汉字
ECSRILM	5×2D_CNN+2×FNN +Transformer Encoder	汉字

模型结构	字错误率（%）
DFCNN	15.06
DFCNN+Transformer Encoder	20.72
ECSRILM	11.88

模型结构	字错误率（%）
DFCNN	15.06
DFCNN+Transformer Encoder	20.72
ECSRILM	11.88

模型	字错误率（%）
BLSTM-DNN-CTC^［23］	26.27
CNN-LSTM-DNN-CTC^［11］	24.55
DCNN-DNN-CTC^［9］	23.36
ResNet-BLSTM^［24］	20.84
Deep Speech2^［12］	16.60
ECSRILM	11.88

融合语言模型的端到端中文语音识别算法

An End-to-End Chinese Speech Recognition Algorithm Integrating Language Model

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 7

参考文献 28

相关文章 15

编辑推荐

Metrics

[1]	唐海桃, 薛嘉宾, 韩纪庆. 一种多尺度前向注意力模型的语音识别方法[J]. 电子学报, 2020, 48(7): 1255-1260.
[2]	张志昌, 曾扬扬, 庞雅丽. 融合语义角色和自注意力机制的中文文本蕴含识别[J]. 电子学报, 2020, 48(11): 2162-2169.
[3]	陈斌, 牛铜, 张连海, 屈丹, 李弼程. 不相关匹配追踪的分段区分性特征变换方法[J]. 电子学报, 2016, 44(12): 2924-2931.
[4]	郑李磊;谢磊;芦咪咪;王晓暄;杨玉莲;张艳宁. 全自动中文新闻字幕生成系统的设计与实现[J]. 电子学报, 2011, 39(3A): 69-74.
[5]	黄永文, 何中市, 王海燕. 基于时间序列分析的动态分布平滑方法[J]. 电子学报, 2008, 36(S1): 147-151.
[6]	温泉;王树勋;年桂君. DCT域音频水印:水印算法和不可感知性测度[J]. 电子学报, 2007, 35(9): 1702-1705.
[7]	全笑梅;张鸿宾. 基于小波包域听觉感知分析的统计音频水印算法[J]. 电子学报, 2007, 35(4): 673-678.
[8]	王守觉;曹文明;. 半导体神经计算机的硬件实现及其在连续语音识别中的应用[J]. 电子学报, 2006, 34(2): 267-271.
[9]	覃鸿;王守觉. 多权值神经元网络仿生模式识别方法在低训练样本数量非特定人语音识别中与HMM及DTW的比较研究[J]. 电子学报, 2005, 33(5): 957-960.
[10]	王守觉;潘晓霞;徐春燕;陈旭;安冬;曹文明. 一种基于高维空间覆盖动态搜索方法的非特定人连续数字语音识别的研究[J]. 电子学报, 2005, 33(10): 1790-1793.
[11]	罗宇;杜利民. 基于概率加权平均的Mel子带特征重建算法[J]. 电子学报, 2004, 32(10): 1738-1741.
[12]	罗宇;杜利民. 基于单高斯模型集的汉语美子带特征重建算法[J]. 电子学报, 2004, 32(10): 1654-1657.
[13]	王作英;肖熙. 基于段长分布的HMM语音识别模型[J]. 电子学报, 2004, 32(1): 46-49.
[14]	朱璇;陈一宁;刘加;刘润生. 语音识别片上系统中的多级搜索算法[J]. 电子学报, 2004, 32(1): 150-153.
[15]	欧智坚;王作英. 汉语连续语音识别中多项式拟合语音轨迹模型的研究[J]. 电子学报, 2003, 31(4): 608-611.