基于韵母发音事件匹配与位置时延分析的音唇一致性判决方法

doi:10.12263/DZXB.20190238

电子学报 ›› 2021, Vol. 49 ›› Issue (1): 140-148.DOI: 10.12263/DZXB.20190238

基于韵母发音事件匹配与位置时延分析的音唇一致性判决方法

朱铮宇^1,2, 廖丽平¹, 杨春玲², 王泳¹, 蔡君¹, 邱华愉¹

1. 广东技术师范大学网络空间安全学院, 广东广州 510665;
2. 华南理工大学电子与信息学院, 广东广州 510641

收稿日期:2019-03-03 修回日期:2020-05-10 出版日期:2021-01-25
- 作者简介:
- 朱铮宇 男,1984年出生,广东广州人,博士后,讲师.主要从事音视频多模态信号处理方面的研究工作.E-mail:zhuzhengyu0701@163.com;廖丽平 女,1981年出生,福建厦门人,教授,硕士生导师,广东省系统工程学会理事.主要从事软件定义、智能路由、大数据处理及应用等方面的研究工作.E-mail:liping1110@hotmail.com
- 基金资助:
- 国家自然科学基金 (No.61672173）; 广东省普通高校青年创新人才 (No.2018KQNCX140）

Lip Motion and Voice Consistency Recognition Based on Audio-Visual Matching of Vowel Pronunciation Events and Position Delay Analysis

ZHU Zheng-yu^1,2, LIAO Li-ping¹, YANG Chun-ling², WANG Yong¹, CAI Jun¹, QIU Hua-yu¹

1. School of Electronics and Information, Guangdong Polytechnic Normal University, Guangzhou, Guangdong 510665, China;
2. School of Electronic and Information Engineering, South China University of Technology, Guangzhou, Guangdong 510641, China

Received:2019-03-03 Revised:2020-05-10 Online:2021-01-25 Published:2021-01-25
- Supported by:
- National Natural Science Foundation of China (No.61672173); Youth Innovative Talents of Colleges and Universities in Guangdong Province (No.2018KQNCX140)

摘要/Abstract

摘要： 针对传统一致性判决方法主要对整句（段）话进行分析，并无对分析内容加以筛选，存在字典规模过大、计算复杂度高及结果易受静音等弱关联片段影响等不足，本文以唇型变化显著的韵母为代表性发音事件，结合音唇初始时延分布范围的统计结果，提出基于韵母发音事件匹配与位置时延分析的一致性判决方法.先利用提出的音视频结合韵母切分法对字典学习数据进行韵母段筛选，再通过学习所得的韵母字典分析韵母事件的音唇匹配度，并对各韵母出现位置的时延分布进行统计评分.最后由韵母发音事件音唇匹配度得分与位置时延分析评分相融合的评分机制判决一致性.实验结果表明，本文算法在识别性能上优于多种比较算法，且与传统字典法相比降低了一定的运算量.

关键词: 一致性分析, 声韵母切分, 字典学习

Abstract: For the mainstream lip motion and voice coherence judgment method, the whole sentence (segment) is analyzed without screening the content. This leads to large dictionary size and high computational complexity, and the result is vulnerable to weak related segments such as mute. Considering the vowel with significant lip shape changes as a representative pronunciation event and combining with the statistical results of the audio-visual initial delay distribution range, a consistent decision method based on audio-visual matching of vowel pronunciation events and position delay analysis is proposed. Firstly, the dictionary learning data is selected by the proposed audio-visual vowel segmentation method, and then the vowel dictionary is used to analyze the matching of the vowel event, and the time delay distribution of each vowel position is statistically scored. A consistency judgment is made by a scoring mechanism in which the vowel pronunciation event lip matching score and the position delay analysis score are combined. Experimental results show that the proposed method is superior to compared algorithms in recognition performance and reduces the amount of computation compared with the traditional dictionary method.

Key words: coherence analysis, initial/final segmentation, dictionary learning

中图分类号:

TP391

朱铮宇, 廖丽平, 杨春玲, 等. 基于韵母发音事件匹配与位置时延分析的音唇一致性判决方法[J]. 电子学报, 2021, 49(1): 140-148.

ZHU Zheng-yu, LIAO Li-ping, YANG Chun-ling, et al. Lip Motion and Voice Consistency Recognition Based on Audio-Visual Matching of Vowel Pronunciation Events and Position Delay Analysis[J]. Acta Electronica Sinica, 2021, 49(1): 140-148.

参考文献

[1] Han Y,Song S,Zhao W.Retrieval of TV talk-show speakers by associating audio transcript to visual clusters[J].IEEE Access,2017,5:20512-20523.
[2] Schonherr L,Zeiler S,et al.Spoofing detection via simultaneous verification of audio-visual synchronicity and transcription[A].2017 IEEE Automatic Speech Recognition and Understanding Workshop[C].Okinawa:IEEE,2017.591-598.
[3] Dov D,Talmon R,et al.Sequential audio-visual correspondence with alternating diffusion kernels[J].IEEE Transactions on Signal Processing,2018,66(12):3100-3111.
[4] Liu Y,Sato Y.Recovery of audio-to-video synchronization through analysis of cross-modality correlation[J].Pattern Recognition Letters,2010,31(8):696-701.
[5] Izadinia H,Saleemi I,et al.Multimodal analysis for identification and segmentation of moving-sounding objects[J].IEEE Transactions on Multimedia,2013,15(2):378-390.
[6] EA Rúa,H Bredin,et al.Audio-visual speech asynchrony detection using co-inertia analysis and coupled hidden markov models[J].Pattern Analysis and Applications,2009,12(3):271-284.
[7] Kumar K,Navratil J,et al.Audio-visual speech synchronization detection using a bimodal linear prediction model[A].IEEE Computer Society Conference on Computer Vision & Pattern Recognition Workshops[C].Florida:IEEE,2009.53-59.
[8] S Kumagai,K Doman,et al.Detection of inconsistency between subject and speaker based on the co-occurrence of lip motion and voice towards speech scene extraction from news videos[A].2011 IEEE International Symposium on Multimedia[C].California:IEEE,2011.311-318.
[9] 朱铮宇,贺前华,奉小慧,等.基于时空相关度融合的语音唇动一致性检测算法[J].电子学报,2014,42(4):779-785. ZHU Zheng-yu,HE Qian-hua,FENG Xiao-hui,et al.Lip motion and voice consistency algorithm based on fusing spatiotemporal correlation degree[J].Acta Electronica Sinica,2014,42(4):779-785.(in Chinese)
[10] Monaci G,Vandergheynst P,et al.Learning bimodal structure in audio-visual data[J].IEEE Transactions on Neural Networks,2009,20(12):1898-1910.
[11] Qingju Liu,Wenwu Wang,et al.Source separation of convolutive and noisy mixtures using audio-visual dictionary learning and probabilistic time-frequency masking[J].IEEE Transactions on Signal Processing,2013,61(22):5520-5535.
[12] 贺前华,朱铮宇,奉小慧.基于平移不变字典的语音唇动一致性判决方法[J].华中科技大学学报(自然科学版),2015,43(10):69-74. HE Qian-hua,ZHU Zheng-yu,FENG Xiao-hui.Lip motion and voice consistency analysis algorithm based on shift-invariant dictionary[J].Journal of Huazhong University of Science and Technology (Nature Science Edition),2015,43(10):69-74.(in Chinese)
[13] El-Sallam A A,Mian A S.Correlation based speech-video synchronization[J].Pattern Recognition Letters,2011,32(6):780-786.
[14] Eg R,Griwodz C,et al.Audiovisual robustness:exploring perceptual tolerance to asynchrony and quality distortion[J].Multimedia Tools & Applications,2015,74(2):345-365.
[15] Staelens N,Meulenaere J D,et al.Assessing the importance of audio/video synchronization for simultaneous translation of video sequences[J].Multimedia Systems,2012,18(6):445-457.
[16] 孙金城,倪宏,莫福源,等.普通话声母和韵母的统计特性[J].应用声学,1995,14(3):35-41. SUN Jin-cheng,NI Hong,MO Fu-yuan,et al.The statistical distribution of standard chinese initials and finals[J].Journal of Applied Acoustics,1995,14(3):35-41.(in Chinese)
[17] Song T,Lee K,et al.Visual voice activity detection via chaos based lip motion measure robust under illumination changes[J].IEEE Transactions on Consumer Electronics,2014,60(2):251-257.
[18] Siatras S,Nikolaidis N,et al.Visual lip activity detection and speaker detection using mouth region intensities[J].IEEE Transactions on Circuits and Systems for Video Technology,2009,19(1):133-137.
[19] Wang Q,Shi G,et al.Analysis and design of an optimum detector for weak sinusoidal signals[A].International Conference on Signal Processing[C].Beijing:IEEE,2002.1608-1611.
[20] Gustafsson F.Determining the initial states in forward-backward filtering[J].IEEE Transactions of Signal Processing,1996,44(4):988-992.
[21] 钱博,李燕萍,唐振民,等.基于频域能量分布分析的自适应元音帧提取算法[J].电子学报,2007,35(2):279-282. QIAN Bo,LI Yan-ping,TANG Zhen-min,et al.Self-adaptive vowel-frame detection algorithm based on energy distribution analysis in frequency domain[J].Acta Electronica Sinica,2007,35(2):279-282.(in Chinese)
[22] 李皓,唐朝京.采用损失函数和声学特征切分声韵母的方法[J].声学学报,2012,37(3):339-345. LI Hao,TANG Chao-jing.Initial/final segmentation using loss function and acoustic feature[J].Acta Acustica,2012,37(3):339-345.(in Chinese)
[23] 胡瑛,陈宁.基于小波变换的清浊音分类及基音周期检测算法[J].电子与信息学报,2008,30(2):353-356. HU Ying,CHEN Ning.Voiced/unvoiced classification and pitch period detection algorithm based on wavelet transform[J].Journal of Electronics and Information Technology,2008,30(2):353-356.(in Chinese)
[24] Yang B,Liu R,Chen X.Fault diagnosis for wind turbine generator bearing via sparse representation and shift-invariant K-SVD[J].IEEE Transactions on Industrial Informatics,2017,13(3):1321-1331.
[25] Ragnhild Eg,Dawn Behne,Carsten Griwodz.Audiovisual temporal integration in reverberant environments[J].Speech Communication,2015,66:91-106.
[26] Takahashi T,Kageyama Y,Ariuntsengel B,et al.Analysis of lip motion due to the influence of vocalization[A].SICE Annual Conference 2012[C].Akita:IEEE,2012.973-978.
[27] 邵健,赵庆卫,颜永红.基于鼻韵尾分离的汉语声韵母识别模型[J].声学学报,2010,35(5):587-592. SHAO Jian,ZHAO Qing-wei,YAN Yong-hong.Initial/final acoustic model based on separating nasal coda in chinese putonghua speech recognition[J].Acta Acustica,2010,35(5):587-592.(in Chinese)

基于韵母发音事件匹配与位置时延分析的音唇一致性判决方法

Lip Motion and Voice Consistency Recognition Based on Audio-Visual Matching of Vowel Pronunciation Events and Position Delay Analysis

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 11

编辑推荐

Metrics

[1]	曹晔. 一种基于局部排序的约束稀疏编码的图像分类方法[J]. 电子学报, 2019, 47(4): 832-836.
[2]	李亚峰. 一种基于多字典学习的图像分割模糊方法[J]. 电子学报, 2018, 46(7): 1700-1709.
[3]	余家林, 孙季丰, 李万益. 基于多核稀疏编码的三维人体姿态估计[J]. 电子学报, 2016, 44(8): 1899-1908.
[4]	詹曙, 方琪, 杨福猛, 常乐乐, 闫婷. 基于耦合特征空间下改进字典学习的图像超分辨率重建[J]. 电子学报, 2016, 44(5): 1189-1195.
[5]	薛模根, 朱虹, 袁广林. 在线鲁棒判别式字典学习视觉跟踪[J]. 电子学报, 2016, 44(4): 838-845.
[6]	黄慧娟, 禹晶, 孙卫东. 基于多字典稀疏表示的遥感图像亚像元映射[J]. 电子学报, 2015, 43(6): 1041-1049.
[7]	詹曙, 王俊, 杨福猛, 方琪. 基于Gabor特征和字典学习的高斯混合稀疏表示图像识别[J]. 电子学报, 2015, 43(3): 523-528.
[8]	潘宗序, 禹晶, 肖创柏, 孙卫东. 基于自适应多字典学习的单幅图像超分辨率算法[J]. 电子学报, 2015, 43(2): 209-216.
[9]	朱铮宇, 贺前华, 奉小慧, 叶婉玲, 李艳雄, 杨继臣. 基于时空相关度融合的语音唇动一致性检测算法[J]. 电子学报, 2014, 42(4): 779-785.
[10]	史郡, 王晓华. 基于改进K-SVD字典学习的超分辨率图像重构[J]. 电子学报, 2013, 41(5): 997-1000.
[11]	蔡泽民;赖剑煌;. 一种基于超完备字典学习的图像去噪方法[J]. 电子学报, 2009, 37(2): 347-350.