融合语义角色和自注意力机制的中文文本蕴含识别

doi:10.3969/j.issn.0372-2112.2020.11.010

电子学报 ›› 2020, Vol. 48 ›› Issue (11): 2162-2169.DOI: 10.3969/j.issn.0372-2112.2020.11.010

所属专题：自然语言处理技术

融合语义角色和自注意力机制的中文文本蕴含识别

张志昌, 曾扬扬, 庞雅丽

西北师范大学计算机科学与工程学院, 甘肃兰州 730000

收稿日期:2019-11-29 修回日期:2020-03-30 出版日期:2020-11-25
- 作者简介:
- 张志昌 男,1976年4月出生,甘肃天水人.教授、硕士生导师.1998年、2003年和2010年分别在西北师范大学、西北工业大学、哈尔滨工业大学获工学学士、工学硕士和工学博士学位.研究方向为自然语言处理,主要进行问答技术、医疗文本处理技术研究.E-mail:zzc@nwnu.edu.cn;曾扬扬 男,1996年4月出生,河南项城人.西北师范大学计算机科学与工程学院硕士研究生.研究方向为自然语言处理.E-mail:zeng2y@126.com
- 基金资助:
- 国家自然科学基金 (No.61762081，No.61662067，No.61662068）; 甘肃省重点研发计划 (No.17YF1GA016）

A Chinese Textual Entailment Recognition Method Incorporating Semantic Role and Self-Attention

ZHANG Zhi-chang, ZENG Yang-yang, PANG Ya-li

College of Computer Science and Engineering, Northwest Normal University, Lanzhou, Gansu 730000, China

Received:2019-11-29 Revised:2020-03-30 Online:2020-11-25 Published:2020-11-25
- Supported by:
- National Natural Science Foundation of China (No.61762081, No.61662067, No.61662068); Key Research and Development Project of Gansu Province (No.17YF1GA016)

摘要/Abstract

摘要： 文本蕴含识别旨在识别两个给定句子之间的逻辑关系.本文通过构造语义角色和自注意力机制融合模块，把句子的深层语义信息与Transformer模型的编码部分相结合，从而增强自注意力机制捕获句子语义的能力.针对中文文本蕴含识别在数据集上存在规模小和噪声大的问题，使用大规模预训练语言模型能够提升模型在小规模数据集上的识别性能.实验结果表明，提出的方法在第十七届中国计算语言学大会中文文本蕴含识别评测数据集CNLI上的准确率达到了80.28%.

关键词: 自然语言处理, 文本蕴含, 自注意力机制, 语义角色标注, 预训练语言模型

Abstract: Recognizing textual entailment is intended to infer the logical relationship between two given sentences. In this paper, we incorporate the deep semantic information of sentences and the encoder of Transformer by constructing the SRL-Attention fusion module, and it effectively improves the ability of self-attention mechanism to capture sentence semantics. Furthermore, concerning the small scale and high noise problems on the dataset, we use large-scale pre-trained language model improving the recognition performance of model on small-scale dataset. Experimental results show that the accuracy of our model on the dataset CNLI, it is released as Chinese textual entailment recognition evaluation corpus at the 17th China National Conference on Computational Linguistics, reaches 80.28%.

Key words: natural language processing, textual entailment, self-attention mechanism, semantic role labeling, pre-trained language model

中图分类号:

张志昌, 曾扬扬, 庞雅丽. 融合语义角色和自注意力机制的中文文本蕴含识别[J]. 电子学报, 2020, 48(11): 2162-2169.

ZHANG Zhi-chang, ZENG Yang-yang, PANG Ya-li . A Chinese Textual Entailment Recognition Method Incorporating Semantic Role and Self-Attention[J]. Acta Electronica Sinica, 2020, 48(11): 2162-2169.

参考文献

[1] 郭茂盛,张宇,刘挺.文本蕴含关系识别与知识获取研究进展及展望[J].计算机学报,2017,40(4):889-910. GUO Mao-sheng,ZHANG Yu,LIU Ting.Research advances and prospect of recognizing textual entailment and knowledge acquisition[J].Chinese Journal of Computers,2017,40(4):889-910.(in Chinese)
[2] ANKUR P P,OSCAR T,DIPANJAN D,et al.A decomposable attention model for natural language inference[A].Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing[C].Stroudsburg:Association for Computational Linguistics,2016.2249-2255.
[3] 袁里驰.利用配价信息的语义角色标注[J].电子学报,2017,45(10):2533-2539. YUAN Li-chi.Semantic role labeling utilizing valence information[J].Acta Electronica Sinica,2017,45(10):2533-2539.(in Chinese)
[4] SHI C,LIU S,REN S,et al.Knowledge-based semantic embedding for machine translation[A].Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers)[C].Berlin,Germany:Association for Computational Linguistics,2016.2245-2254.
[5] 杨震,范科峰,雷建军,等.基于语义的文本流形研究[J].电子学报,2009,37(3):557-561. YANG Zhen,FAN Ke-feng,LEI Jia-jun,et al.Text manifold based on semantic analysis[J].Acta Electronica Sinica,2009,37(3):557-561.(in Chinese)
[6] PETERS M E,NEUMANN M,IYYERM,et al.Deep contextualized word representations[A].The 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies[C].Stroudsburg:Association for Computational Linguistics,2018.2227-2237.
[7] RANFORD A,NARASIMHAN K,SALIMANS T,et al.Improving language understanding by generative pre-training[OL].https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf,2019-10-20.
[8] DEVLIN J,CHANG M W,LEE K,et al.BERT:pre-training of deep bidirectional transformers for language understanding[A].Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies[C].Stroudsburg:Association for Computational Linguistics,2019.4171-4186.
[9] CUI Y,CHE W,LIU T,et al.Pre-training with whole word masking for Chinese BERT[J].arXiv Preprint,2019,arXiv:1906.08101.
[10] VASWANI A,SHAZEER N,PARMAR N,et al.Attention is all you need[A].Procceeding of 31st Conference on Neural Information Processing Systems (NIPS 2017)[C].Long Beach,CA,USA:ACM,2017.6000-6010.
[11] BOWMAN S R,ANGELI G,POTTS C,et al.A large annotated corpus for learning natural language inference[A].Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing[C].Stroudsburg:Association for Computational Linguistics,2015.632-642.
[12] 翟延冬,王康平,张东娜,等.一种基于WordNet的短文本语义相似性算法[J].电子学报,2012,40(3):617-620. ZHAI Yan-dong,WANG Kang-ping,ZHANG Dong-na,et al.An algorithm for semantic similarity of short text based on WordNet[J].Acta Electronica Sinica,2012,40(3):617-620.(in Chinese)
[13] 袁里驰.基于配价结构和语义依存关系的句法分析统计模型[J].电子学报,2013,41(10):2029-2034. YUAN Li-chi.A statistical parsing model based on valence structure and semantic dependency[J].Acta Electronica Sinica,2013,41(10):2029-2034.(in Chinese)
[14] 刘茂福,李妍,姬东鸿.基于事件语义特征的中文文本蕴含识别[J].中文信息学报,2013,27(5):129-136. LIU Mao-fu,LI Yan,JI Dong-hong,et al.Event semantic feature based Chinese textual entailment recognition[J].Journal of Chinese Information Processing,2013,27(5):129-136.(in Chinese)
[15] 谭咏梅,刘姝雯,吕学强.基于CNN与双向LSTM的中文文本蕴含识别方法[J].中文信息学报,2018,32(7):11-19. TAN Yong-mei,LIU Shu-wen,LU Xue-qiang.CNN and BiLSTM based Chinese textual entailment recognition[J].Journal of Chinese Information Processing,2018,32(7):11-19.(in Chinese)
[16] ROCKTASCHEL T,GREFENSTETTE E,HERMANN K M,et al.Reasoning about entailment with neural attention[J].arXiv Preprint,2015,arXiv:1509.06664.
[17] WANG S,JIANG J.Learning natural language inference with LSTM[A].The 14th Annual Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies[C].Stroudsburg:Association for Computational Linguistics,2016.1442-1451.
[18] CHEN Q,ZHU X,LING Z,et al.Enhanced LSTM for natural language inference[A].Proceedings of the 55th Annual Meeting of the Association for Computational[C].Stroudsburg:Association for Computational Linguistics,2017.1657-1668.
[19] 刘广灿,曹宇,许家铭,等.基于对抗正则化的自然语言推理[J].自动化学报,2019,45(8):1455-1463. LIU Guang-can,CAO Yu,XU Jia-ming,et al.Natural language inference based on adversarial regularization[J].Acta Automatica Sinica,2019,45(8):1455-1463.(in Chinese)
[20] PAN B,YANG Y,ZHAO Z,et al.Discourse marker augmented network with reinforcement learning for natural language inference[A].Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers)[C].Melbourne:Association for Computational Linguistics,2018.989-999.
[21] ZHANG Z,WU Y,LI Z,et al.I know what you want:Semantic learning for text comprehension[J].arXiv Preprint,2018,arXiv:1809.02794.
[22] BA J L,KIROS J R,HINTON G E.Layer normalization[J].arXiv Preprint,2016,arXiv:1607.06450.
[23] Che W,Li Z,Liu T.Ltp:A chinese language technology platform[A].Proceedings of the 23rd International Conference on Computational Linguistics:Demonstrations[C].Association for Computational Linguistics,2010.13-16.
[24] 中国计算语言学大会(CCL 2018)技术评测任务发布[R/OL].http://www.cips-cl.org/static/CCL2018/call-evaluation.html,2019-9-23.

融合语义角色和自注意力机制的中文文本蕴含识别

A Chinese Textual Entailment Recognition Method Incorporating Semantic Role and Self-Attention

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	何滢婕, 刘月峰, 边浩东, 郭威, 张小燕. 基于Informer的电池荷电状态估算及其稀疏优化方法[J]. 电子学报, 2023, 51(1): 50-56.
[2]	张志昌, 于沛霖, 庞雅丽, 朱林, 曾扬扬. SMGN：用于对话状态跟踪的状态记忆图网络[J]. 电子学报, 2022, 50(8): 1851-1858.
[3]	杜晋华, 尹浩, 冯嵩. 中文电子病历命名实体识别的研究与进展[J]. 电子学报, 2022, 50(12): 3030-3053.
[4]	黄名选. 关联模式挖掘与词向量学习融合的伪相关反馈查询扩展[J]. 电子学报, 2021, 49(7): 1305-1313.
[5]	张昱, 刘开峰, 张全新, 王艳歌, 高凯龙. 基于组合-卷积神经网络的中文新闻文本分类[J]. 电子学报, 2021, 49(6): 1059-1067.
[6]	杨启萌, 禹龙, 田生伟, 艾山·吾买尔. 基于深度强化学习的维吾尔语人称代词指代消解[J]. 电子学报, 2020, 48(6): 1077-1083.
[7]	黄名选, 蒋曹清. 基于项权值排序挖掘的跨语言查询扩展[J]. 电子学报, 2020, 48(3): 568-576.
[8]	吴玉佳, 李晶, 宋成芳, 常军. 基于高效用神经网络的文本分类方法[J]. 电子学报, 2020, 48(2): 279-284.
[9]	马慧芳, 刘文, 李志欣, 蔺想红. 融合耦合距离区分度和强类别特征的短文本相似度计算方法[J]. 电子学报, 2019, 47(6): 1331-1336.
[10]	田生伟, 秦越, 禹龙, 吐尔根·依布拉音, 冯冠军. 基于Bi-LSTM的维吾尔语人称代词指代消解[J]. 电子学报, 2018, 46(7): 1691-1699.
[11]	马慧芳, 刘芳, 夏琴, 郝占军. 基于加权超图随机游走的文献关键词提取算法[J]. 电子学报, 2018, 46(6): 1410-1414.
[12]	黄名选, 蒋曹清. 基于完全加权正负关联模式挖掘的越-英跨语言查询译后扩展[J]. 电子学报, 2018, 46(12): 3029-3036.
[13]	袁里驰. 利用配价信息的语义角色标注[J]. 电子学报, 2017, 45(10): 2533-2539.
[14]	周炫余, 刘娟, 邵鹏, 卢笑, 罗飞. 基于测度优化Laplacian SVM的中文指代消解方法[J]. 电子学报, 2016, 44(12): 3064-3072.
[15]	利锦标;李童;刘璘. 基于类图语义框架的中文需求分析方法[J]. 电子学报, 2011, 39(3A): 94-98.