A Survey of Text Generation and Evaluation Based on Intrinsic Quality Constraints

LAN Yu-qian; RAO Yuan; LI Guan-cheng; SUN Ling; XIA Bing-can; XIN Ting-ting

doi:10.12263/DZXB.20230826

您当前的位置：

首页 >

文章列表页 >

A Survey of Text Generation and Evaluation Based on Intrinsic Quality Constraints

SURVEYS AND REVIEWS | 更新时间：2025-12-11

- A Survey of Text Generation and Evaluation Based on Intrinsic Quality Constraints
- ACTA ELECTRONICA SINICA Vol. 52, Issue 2, Pages: 633-659(2024)
- 作者机构：
  
  1.西安交通大学软件学院社会智能与复杂数据处理实验室，陕西西安 710049
  2.中国长峰机电技术研究设计院，北京 100854
- 作者简介：
- 基金信息：
  
  National Natural Science Foundation of China(U22B2036);National Key Research and Development Program of China(2019YFB2102300);Special Funds Program for Central Universities to Build World-class Universities (Disciplines) and Guide the Development of Special Features(PY3A022)
- DOI：10.12263/DZXB.20230826
  CLC： TP183;
- Received：01 September 2023，
  
  Revised：2024-01-02，
  
  Published：25 February 2024
- 稿件说明：
移动端阅览
兰玉乾,饶元,李冠呈,等.基于内在质量约束的文本生成和评价综述[J].电子学报,2024,52(02):633-659.

LAN Yu-qian, RAO Yuan, LI Guan-cheng, et al.A Survey of Text Generation and Evaluation Based on Intrinsic Quality Constraints[J].Acta Electronica Sinica, 2024, 52(02): 633-659.
兰玉乾,饶元,李冠呈,等.基于内在质量约束的文本生成和评价综述[J].电子学报,2024,52(02):633-659. DOI：10.12263/DZXB.20230826

LAN Yu-qian, RAO Yuan, LI Guan-cheng, et al.A Survey of Text Generation and Evaluation Based on Intrinsic Quality Constraints[J].Acta Electronica Sinica, 2024, 52(02): 633-659. DOI：10.12263/DZXB.20230826

摘要

近年来，以ChatGPT为代表的能够适应复杂场景、并能满足人类的各种应用需求为目标的文本生成算法模型成为学术界与产业界共同关注的焦点.然而，ChatGPT等大规模语言模型（Large Language Model，LLM）高度忠实于用户意图的优势隐含了部分的事实性错误，而且也需要依靠提示内容来控制细致的生成质量和领域适应性，因此，研究以内在质量约束为核心的文本生成方法仍具有重要意义.本文在近年来关键的内容生成模型和技术对比研究的基础上，定义了基于内在质量约束的文本生成的基本形式，以及基于“信、达、雅”的6种质量特征；针对这6种质量特征，分析并总结了生成器模型的设计和相关算法；同时，围绕不同的内在质量特征总结了多种自动评价和人工评价指标与方法.最后，本文对文本内在质量约束技术的未来研究方向进行了展望.

Abstract

Recently

the outstanding text generation language models represented by ChatGPT

which can adapt to complex scenes and meet various application demands of human beings

has become the focuses of both the academic and industrial circles. However

the advantage of large language models (LLM) such as ChatGPT that are highly faithful to user intent implies some factual errors

and it is also necessary to rely on prompt content to control the detailed generation quality and domain adaptability

so it is still of great significance to study text generation with intrinsic quality constraints as the core. Based on the comparative study of key content generation models and technologies in recent years

this paper defined the basic form of text generation with intrinsic quality constraints

and six quality features based on “credibility

expressiveness and elegance”. In view of these 6 quality features

we provided analysis and comparison of generator model design and related algorithms. Besides

various automatic and human evaluation methods for different intrinsic quality features are summarized. Finally

this paper looks forward to the future research directions of intrinsic quality constraint technology.

关键词

Keywords

references

OUYANG , WU J , JIANG X , et al . Training language models to follow instructions with human feedback [EB/OL]. ( 2022-03-04 )[ 2023-09-01 ]. https://arxiv.org/abs/2203.02155 https://arxiv.org/abs/2203.02155 .

SCHULMAN J , ZOPH B , KIM C , et al . ChatGPT: Optimizing language models for dialogue [R/OL]. ( 2022-11-30 )[ 2023-09-01 ]. https://openai.com/blog/chatgpt https://openai.com/blog/chatgpt .

BROWN T B , MANN B , RYDER N , et al . Language models are few-shot learners [C]// Proceedings of the 34th International Conference on Neural Information Processing Systems . New York : ACM , 2020 : 1877 - 1901 .

LEWIS M , LIU Y H , GOYAL N , et al . BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension [C]// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Stroudsburg : Association for Computational Linguistics , 2020 : 7871 - 80 .

TURING A M . Computing machinery and intelligence [M]//EPSTEIN R, ROBERTS G, BEBER G. Parsing the Turing Test . Dordrecht : Springer , 2009 : 23 - 65 .

ZHANG M . An inquiry into Yan fu’s translation theory of faithfulness, expressiveness, and elegance: The beginning of China’s modern translation theory [J]. Trans-Humanities Journal , 2013 , 6 ( 3 ): 179 - 196 .

RAFFEL C , SHAZEER N , ROBERTS A , et al . Exploring the limits of transfer learning with a unified text-to-text transformer [EB/OL]. ( 2019-10-23 )[ 2023-09-01 ]. https://arxiv.org/abs/1910.10683 https://arxiv.org/abs/1910.10683 .

BUBECK S , CHANDRASEKARAN V , ELDAN R , et al . Sparks of Artificial General Intelligence: Early experiments with GPT-4 [EB/OL]. ( 2023-03-22 )[ 2023-09-01 ]. http://arxiv.org/abs/2303.12712.pdf http://arxiv.org/abs/2303.12712.pdf .

VASWANI A , SHAZEER N , PARMAR N , et al . Attention is all you need [EB/OL]. ( 2017-06-12 )[ 2023-09-01 ]. http://arxiv.org/abs/1706.03762.pdf http://arxiv.org/abs/1706.03762.pdf .

DEVLIN J , CHANG M-W , LEE K , et al . Bert: Pre-training of deep bidirectional transformers for language understanding [C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) . Minneapolis : Association for Computational Linguistics , 2019 : 4171 - 4186 .

RADFORD A , NARASIMHAN K , SALIMANS T , et al . Improving language understanding by generative pre-training [EB/OL]. ( 2018-10-11 )[ 2023-09-01 ]. https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf .

RADFORD A , WU J , CHILD R , et al . Language models are unsupervised multitask learners [EB/OL]. ( 2019-02-14 )[ 2023-09-01 ]. https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf .

LIU Y H , OTT M , GOYAL N , et al . RoBERTa: A robustly optimized BERT pretraining approach [EB/OL]. ( 2019-07-26 )[ 2023-09-01 ]. https://arxiv.org/abs/1907.11692 https://arxiv.org/abs/1907.11692 .

JOSHI M , CHEN D Q , LIU Y H , et al . SpanBERT: Improving pre-training by representing and predicting spans [J]. Transactions of the Association for Computational Linguistics , 2020 , 8 : 64 - 77 .

REIMERS N , GUREVYCH I . Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks [C]// Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) . Stroudsburg : Association for Computational Linguistics , 2019 : 3980 - 3990 .

DAI Z H , YANG Z L , YANG Y M , et al . Transformer-XL: Attentive language models beyond a fixed-length context [C]// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . Stroudsburg : Association for Computational Linguistics , 2019 : 2978 - 2988 .

YANG Z L , DAI Z H , YANG Y M , et al . XLNet: Generalized autoregressive pretraining for language understanding [C]// NIPS’19: Proceedings of the 33rd International Conference on Neural Information Processing Systems . New York : ACM , 2019 : 5753 - 5763 .

TAN B W , YANG Z C , AI-SHEDIVAT M , et al . Progressive generation of long text with pretrained language models [EB/OL]. ( 2020-06-28 )[ 2023-09-01 ]. https://arxiv.org/abs/2006.15720 https://arxiv.org/abs/2006.15720 .

HUA X Y , WANG L . PAIR: Planning and iterative refinement in pre-trained transformers for long text generation [C]// Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) . Kerrville : Association for Computational Linguistics , 2020 : 781 - 793 .

JANG Y , LIM J , HUR Y , et al . Call for customized conversation: Customized conversation grounding persona and knowledge [J]. Proceedings of the AAAI Conference on Artificial Intelligence , 2022 , 36 ( 10 ): 10803 - 10812 .

CHEN Z , XU W D , WANG B T , et al . A blockchain-based preserving and sharing system for medical data privacy [J]. Future Generation Computer Systems , 2021 , 124 ( C ): 338 - 350 .

LE T , PARK N , LEE D . SHIELD: defending textual neural networks against multiple black-box adversarial attacks with stochastic multi-expert patcher [C]// Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Stroudsburg : Association for Computational Linguistics , 2022 : 6661 - 6674 .

YI X Y , LIU Z H , LI W H , et al . Text style transfer via learning style instance supported latent space [C]// Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence . California : International Joint Conferences on Artificial Intelligence Organization , 2020 : 3801 - 3807 .

ZHOU C L , CHEN L Y , LIU J C , et al . Exploring contextual word-level style relevance for unsupervised style transfer [C]// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Stroudsburg : Association for Computational Linguistics , 2020 : 7135 - 7144 .

MO L Z , WEI J L , HUANG Q B , et al . Incorporating sentimental trend into gated mechanism based transformer network for story ending generation [J]. Neurocomputing , 2021 , 453 ( C ): 453 - 464 .

WEN Z Y , CAO J N , YANG R S , et al . Automatically select emotion for response via personality-affected emotion transition [C]// Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 . Stroudsburg : Association for Computational Linguistics , 2021 : 5010 - 5020 .

YIN X J , WAN X J . How do Seq2Seq models perform on end-to-end data-to-text generation? [C]// Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Stroudsburg : Association for Computational Linguistics , 2022 : 7701 - 7710 .

SHU C , ZHANG Y S , DONG X Y , et al . Logic-consistency text generation from semantic parses [C]// Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 . Stroudsburg : Association for Computational Linguistics , 2021 : 4414 - 4426 .

DER LEE CHRIS V , ALBERT G , EMIEL V M , et al . Human evaluation of automatically generated text: Current trends and best practice guidelines [J]. Computer Speech & Language , 2021 , 67 : 101151 .

IQBAL T , S J J O K S U-C QURESHI , SCIENCES I . The survey: Text generation models in deep learning [J]. Journal of King Saud University - Computer and Information Sciences , 2020 , 34 ( 6 ): 2515 - 2528 .

LI J Y , TANG T Y , ZHAO W X , et al . Pretrained language model for text generation: A survey [C]// Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence . California : International Joint Conferences on Artificial Intelligence Organization , 2021 : 4492 - 4499 .

CELIKYILMAZ A , CLARK E , GAO J F . Evaluation of text generation: A survey [EB/OL]. ( 2020-06-26 )[ 2023-09-01 ]. http://arxiv.org/abs/2006.14799.pdf http://arxiv.org/abs/2006.14799.pdf .

JIN D , JIN Z J , HU Z T , et al . Deep learning for text style transfer: A survey [J]. Comput Linguistics , 2022 , 48 ( 1 ): 155 - 205 .

ZHOU J N , BHAT S . Paraphrase generation: A survey of the state of the art [C]// Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing . Stroudsburg : Association for Computational Linguistics , 2021 : 5075 - 5086 .

ZHAO H , PHUNG D , HUYNH V , et al . Topic modelling meets deep neural networks: A survey [C]// Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence . California : International Joint Conferences on Artificial Intelligence Organization , 2021 : 4713 - 4720 .

JI Z W , LEE N , FRIESKE R , et al . Survey of hallucination in natural language generation [J]. ACM Computing Surveys , 2022 , 55 ( 12 ): 248 .

TANG C , GUERIN F , LIN C H . Recent advances in neural text generation: A task-agnostic survey [EB/OL]. ( 2022-03-06 )[ 2023-09-01 ]. http://arxiv.org/abs/2203.03047.pdf http://arxiv.org/abs/2203.03047.pdf .

HALLIDAY M A K , MATTHIESSEN C M I M . Halliday’s Introduction to Functional Grammar [M]. London : Routledge , 2013 .

REBUFFEL C , ROBERTI M , SOULIER L , et al . Controlling hallucinations at word level in data-to-text generation [J]. Data Mining and Knowledge Discovery , 2022 , 36 ( 1 ): 318 - 354 .

RASHKIN H , REITTER D , TOMAR G S , et al . Increasing faithfulness in knowledge-grounded dialogue with controllable features [C]// Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) . Stroudsburg : Association for Computational Linguistics , 2021 : 704 - 718 .

LI H R , ZHU J N , ZHANG J J , et al . Ensure the correctness of the summary: Incorporate entailment knowledge into abstractive sentence summarization [C]// Proceedings of the 27th International Conference on Computational Linguistics . Stroudsburg : Association for Computational Linguistics , 2018 : 1430 - 1441 .

LIU T Y , ZHENG X , CHANG B B , et al . Towards faithfulness in open domain table-to-text generation from an entity-centric view [J]. Proceedings of the AAAI Conference on Artificial Intelligence , 2021 , 35 ( 15 ): 13415 - 13423 .

ZHANG J Q , ZHAO Y , SALEH M , et al . PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization [C]// Proceedings of the 37th International Conference on Machine Learning . New York : ACM , 2020 : 11328 - 11339 .

ARALIKATTE R , NARAYAN S , MAYNEZ J , et al . Focus attention: Promoting faithfulness and diversity in summarization [C]// Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) . Stroudsburg : Association for Computational Linguistics , 2021 : 6078 - 6095 .

CAO S Y , WANG L . CLIFF: contrastive learning for improving faithfulness and factuality in abstractive summarization [C]// Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing . Stroudsburg : Association for Computational Linguistics , 2021 : 6633 - 6649 .

LEE S , LEE D B , HWANG S J . Contrastive learning with adversarial perturbations for conditional text generation [EB/OL]. ( 2020-12-14 )[ 2023-09-01 ]. http://arxiv.org/abs/2012.07280.pdf http://arxiv.org/abs/2012.07280.pdf .

ZHU C G , XU Y C , REN X , et al . Knowledge-augmented methods for natural language processing [C]// Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts . Stroudsburg : Association for Computational Linguistics , 2022 : 12 - 20 .

ZHOU H , YOUNG T , HUANG M L , et al . Commonsense knowledge aware conversation generation with graph attention [C]// Proceedings of the 27th International Joint Conference on Artificial Intelligence . New York : ACM , 2018 : 4623 - 4629 .

TAI Y , YANG J , LIU X M , et al . MemNet: A persistent memory network for image restoration [C]// 2017 IEEE International Conference on Computer Vision (ICCV) . Piscataway : IEEE , 2017 : 4549 - 4557 .

GUAN J , HUANG F , ZHAO Z H , et al . A knowledge-enhanced pretraining model for commonsense story generation [J]. Transactions of the Association for Computational Linguistics , 2020 , 8 : 93 - 108 .

YU C L , ZHANG H M , SONG Y Q , et al . CoCoLM: complex commonsense enhanced language model with discourse relations [C]// Findings of the Association for Computational Linguistics: ACL 2022 . Stroudsburg : Association for Computational Linguistics , 2022 : 1175 - 1187 .

MADAAN N , PADHI I , PANWAR N , et al . Generate your counterfactuals: Towards controlled counterfactual generation for text [J]. Proceedings of the AAAI Conference on Artificial Intelligence , 2021 , 35 ( 15 ): 13516 - 13524 .

PARANJAPE B , LAMM M , TENNEY I . Retrieval-guided counterfactual generation for QA [C]// Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Stroudsburg : Association for Computational Linguistics , 2022 : 1670 - 1686 .

BARTOLO M , THRUSH T , JIA R , et al . Improving question answering model robustness with synthetic adversarial data generation [C]// Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing . Stroudsburg : Association for Computational Linguistics , 2021 : 8830 - 8848 .

CALDERON N , BEN-DAVID E , FEDER A , et al . DoCoGen: domain counterfactual generation for low resource domain adaptation [C]// Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Stroudsburg : Association for Computational Linguistics , 2022 : 7727 - 7746 .

ZHANG H N , LAN Y Y , PANG L , et al . Modeling topical relevance for multi-turn dialogue generation [C]// Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence . California : International Joint Conferences on Artificial Intelligence Organization , 2020 : 3737 - 3743 .

SERBAN I , SORDONI A , BENGIO Y , et al . Building end-to-end dialogue systems using generative hierarchical neural network models [J]. Proceedings of the AAAI Conference on Artificial Intelligence , 2016 , 30 ( 1 ): 3776 - 3784 .

HE G X , GAO Z , JIANG Z R , et al . Think beyond the word: Understanding the implied textual meaning by digesting context, local, and noise [C]// Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval . New York : ACM , 2020 : 2297 - 2306 .

QIN L H , GALLEY M , BROCKETT C , et al . Conversing by reading: Contentful neural conversation with on-demand machine reading [C]// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . Stroudsburg : Association for Computational Linguistics , 2019 : 5427 - 5436 .

LIU X D , SHEN Y L , DUH K , et al . Stochastic answer networks for machine reading comprehension [C]// Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Stroudsburg : Association for Computational Linguistics , 2018 : 1694 - 1704 .

TIAN Z L , BI W , LEE D , et al . Response-anticipated memory for on-demand knowledge integration in response generation [C]// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Stroudsburg : Association for Computational Linguistics , 2020 : 650 - 659 .

XU J , LEI Z Y , WANG H F , et al . Discovering dialog structure graph for coherent dialog generation [C]// Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) . Stroudsburg : Association for Computational Linguistics , 2021 : 1726 - 1739 .

PENG B L , LI C Y , LI J C , et al . SOLOIST: Few-shot task-oriented dialog with A single pre-trained auto-regressive model [EB/OL]. ( 2020-05-11 )[ 2023-09-01 ]. http://arxiv.org/abs/2005.05298.pdf http://arxiv.org/abs/2005.05298.pdf .

MORYOSSEF A , GOLDBERG Y , DAGAN I . Step-by-step: Separating planning from realization in neural data-to-text generation [EB/OL]. ( 2019-08-06 )[ 2023-09-01 ]. http://arxiv.org/abs/1904.03396.pdf http://arxiv.org/abs/1904.03396.pdf .

HUA X Y , HU Z , WANG L . Argument generation with retrieval, planning, and realization [C]// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . Stroudsburg : Association for Computational Linguistics , 2019 : 2661 - 1672 .

MARCHENKO O O , RADYVONENKO O S , IGNATOVA T S , et al . Improving text generation through introducing coherence metrics [J]. Cybernetics and Systems Analysis , 2020 , 56 ( 1 ): 13 - 21 .

WEI J , WANG X Z , SCHUURMANS D , et al . Chain-of-thought prompting elicits reasoning in large language models [EB/OL]. ( 2022-01-28 )[ 2023-09-01 ]. https://arxiv.org/abs/2201.11903 https://arxiv.org/abs/2201.11903 .

YAO S Y , YU D , ZHAO J , et al . Tree of thoughts: Deliberate problem solving with large language models [EB/OL]. ( 2023-05-17 )[ 2023-09-01 ]. http://arxiv.org/abs/2305.10601.pdf http://arxiv.org/abs/2305.10601.pdf .

CAO Y , BI W , FANG M , et al . A model-agnostic data manipulation method for persona-based dialogue generation [C]// Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Stroudsburg : Association for Computational Linguistics , 2022 : 7984 - 8002 .

KIM M , KWAK B W , KIM Y , et al . Dual task framework for improving persona-grounded dialogue dataset [J]. Proceedings of the AAAI Conference on Artificial Intelligence , 2022 , 36 ( 10 ): 10912 - 10920 .

FU T C , ZHAO X L , TAO C Y , et al . There are a thousand hamlets in a thousand People's eyes: Enhancing knowledge-grounded dialogue with personal memory [C]// Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Stroudsburg : Association for Computational Linguistics , 2022 : 3901 - 3913 .

YANG R Z , CHEN J X , NARASIMHAN K . Improving dialog systems for negotiation with personality modeling [C]// Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) . Stroudsburg : Association for Computational Linguistics , 2021 : 681 - 693 .

SONG H Y , WANG Y , ZHANG K Y , et al . BoB: BERT over BERT for training persona-based dialogue models from limited personalized data [C]// Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) . Stroudsburg : Association for Computational Linguistics , 2021 : 167 - 177 .

GU J C , TAO C Y , LING Z H , et al . MPC-BERT: A pre-trained language model for multi-party conversation understanding [C]// Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) . Stroudsburg : Association for Computational Linguistics , 2021 : 3682 - 3692 .

PARK J S , O'BRIEN J , CAI C J , et al . Generative agents: Interactive simulacra of human behavior [C]// Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology . New York : ACM , 2023 : 1 - 22 .

WANG Z L , CHIU Y Y , CHIU Y C . Humanoid agents: Platform for simulating human-like generative agents [C]// Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations . Stroudsburg : Association for Computational Linguistics , 2023 : 167 - 176 .

SCLAR M , KUMAR S , WEST P , et al . Minding language models' (lack of) theory of mind: A plug-and-play multi-character belief tracker [C]// Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Stroudsburg : Association for Computational Linguistics , 2023 : 13960 - 13980 .

LEE J . Stable style transformer: Delete and generate approach with encoder-decoder for text style transfer [C]// Proceedings of the 13th International Conference on Natural Language Generation . Stroudsburg : Association for Computational Linguistics , 2020 : 195 - 204 .

SHU L , PAPANGELIS A , WANG Y C , et al . Controllable text generation with focused variation [C]// Findings of the Association for Computational Linguistics: EMNLP 2020 . Stroudsburg : Association for Computational Linguistics , 2020 : 3805 - 3817 .

SOHN K , YAN X C , LEE H . Learning structured output representation using deep conditional generative models [J]. Advances in Neural Information Processing Systems , 2015 , 2015-January: 3483- 3491 .

DAI N , LIANG J Z , QIU X P , et al . Style transformer: Unpaired text style transfer without disentangled latent representation [C]// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . Stroudsburg : Association for Computational Linguistics , 2019 : 5997 - 6007 .

CHENG Y , GAN Z , ZHANG Y Z , et al . Contextual text style transfer [C]// Findings of the Association for Computational Linguistics: EMNLP 2020 . Stroudsburg : Association for Computational Linguistics , 2020 : 2915 - 2924 .

KHAYRALLAH H , SEDOC J . SMRT chatbots: Improving non-task-oriented dialog with simulated multiple reference training [C]// Findings of the Association for Computational Linguistics: EMNLP 2020 . Stroudsburg : Association for Computational Linguistics , 2020 : 4489 - 4505 .

SU H , SHEN X Y , ZHAO S Q , et al . Diversifying dialogue generation with non-conversational text [C]// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Stroudsburg : Association for Computational Linguistics , 2020 : 7087 - 7097 .

HU Z T , YANG Z C , LIANG X D , et al . Toward controlled generation of text [C]// Proceedings of the 34th International Conference on Machine Learning - Volume 70 . New York : ACM , 2017 : 1587 - 1596 .

KINGMA D P , WELLING M . Auto-encoding variational Bayes [EB/OL]. ( 2013-12-20 )[ 2023-09-01 ]. http://arxiv.org/abs/1312.6114.pdf http://arxiv.org/abs/1312.6114.pdf .

LACHAUX M A , JOULIN A , LAMPLE G . Target conditioning for one-to-many generation [C]// Findings of the Association for Computational Linguistics: EMNLP 2020 . Stroudsburg : Association for Computational Linguistics , 2020 : 2853 - 2862 .

CUI Z , LI Y R , ZHANG J Y , et al . Focus-constrained attention mechanism for CVAE-based response generation [C]// Findings of the Association for Computational Linguistics: EMNLP 2020 . Stroudsburg : Association for Computational Linguistics , 2020 : 2021 - 2030 .

WANG Z , RAO S W , ZHANG J , et al . Diversify question generation with continuous content selectors and question type modeling [C]// Findings of the Association for Computational Linguistics: EMNLP 2020 . Stroudsburg : Association for Computational Linguistics , 2020 : 2134 - 2143 .

SUN H , LIN Z R , ZHENG C J , et al . PsyQA: A Chinese dataset for generating long counseling text for mental health support [C]// Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 . Stroudsburg : Association for Computational Linguistics , 2021 : 1489 - 1503 .

XU J , SZLAM A , WESTON J . Beyond goldfish memory: Long-term open-domain conversation [C]// Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Stroudsburg : Association for Computational Linguistics , 2022 : 5180 - 5197 .

GU N L , ASH E , HAHNLOSER R . MemSum: extractive summarization of long documents using multi-step episodic Markov decision processes [C]// Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Stroudsburg : Association for Computational Linguistics , 2022 : 6507 - 6522 .

PAPINENI K , ROUKOS S , WARD T , et al . BLEU: A method for automatic evaluation of machine translation [C]// Proceedings of the 40th Annual Meeting on Association for Computational Linguistics . New York : ACM , 2002 : 311 - 318 .

FABBRI A , WU C S , LIU W H , et al . QAFactEval: improved QA-based factual consistency evaluation for summarization [C]// Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . Stroudsburg : Association for Computational Linguistics , 2022 : 2587 - 2601 .

DHINGRA B , FARUQUI M , PARIKH A , et al . Handling divergent reference texts when evaluating table-to-text generation [C]// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . Stroudsburg : Association for Computational Linguistics , 2019 : 4884 - 4895 .

DURMUS E , HE H , DIAB M . FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization [C]// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Stroudsburg : Association for Computational Linguistics , 2020 : 5055 - 5070 .

SCIALOM T , DRAY P A , LAMPRIER S , et al . QuestEval: summarization asks for fact-based evaluation [C]// Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing . Stroudsburg : Association for Computational Linguistics , 2021 : 6594 - 6604 .

FALKE T , RIBEIRO L F R , UTAMA P A , et al . Ranking generated summaries by correctness: An interesting but challenging application for natural language inference [C]// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . Stroudsburg : Association for Computational Linguistics , 2019 : 2214 - 2220 .

MAYNEZ J , NARAYAN S , BOHNET B , et al . On faithfulness and factuality in abstractive summarization [C]// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Stroudsburg : Association for Computational Linguistics , 2020 : 1906 - 1919 .

NIE Y X , WILLIAMSON M , BANSAL M , et al . I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling [C]// Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) . Stroudsburg : Association for Computational Linguistics , 2021 : 1699 - 1713 .

GOYAL T , DURRETT G . Evaluating factuality in generation with dependency-level entailment [C]// Findings of the Association for Computational Linguistics: EMNLP 2020 . Stroudsburg : Association for Computational Linguistics , 2020 : 3592 - 3603 .

KRYSCINSKI W , MCCANN B , XIONG C M , et al . Evaluating the factual consistency of abstractive text summarization [C]// Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) . Stroudsburg : Association for Computational Linguistics , 2020 : 9332 - 9346 .

DZIRI N , RASHKIN H , LINZEN T , et al . Evaluating groundedness in dialogue systems: The BEGIN benchmark [EB/OL]. ( 2021-04-30 )[ 2023-09-01 ]. http://arxiv.org/abs/2105.00071.pdf http://arxiv.org/abs/2105.00071.pdf .

WANG A , CHO K , LEWIS M . Asking and answering questions to evaluate the factual consistency of summaries [C]// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Stroudsburg : Association for Computational Linguistics , 2020 : 5008 - 5020 .

NAN F , NOGUEIRA DOS SANTOS C , ZHU H H , et al . Improving factual consistency of abstractive summarization via question answering [C]// Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) . Stroudsburg : Association for Computational Linguistics , 2021 : 6881 - 6894 .

QIN L B , XIE T B , HUANG S J , et al . Don't be contradicted with anything! CI-ToD: Towards benchmarking consistency for task-oriented dialogue system [C]// Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing . Stroudsburg : Association for Computational Linguistics , 2021 : 2357 - 2367 .

GUPTA P , WU C S , LIU W H , et al . DialFact: A benchmark for fact-checking in dialogue [C]// Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Stroudsburg : Association for Computational Linguistics , 2022 : 3785 - 3801 .

LEE N , PING W , XU P , et al . Factuality enhanced language models for open-ended text generation [EB/OL]. ( 2022-06-09 )[ 2023-09-01 ]. http://arxiv.org/abs/2206.04624.pdf http://arxiv.org/abs/2206.04624.pdf .

LIN C Y . ROUGE: A package for automatic evaluation of summaries [C]// Proceedings of the Text Summarization Branches Out . Stroudsburg : Association for Computational Linguistics , 2004 : 74 - 81 .

BANERJEE S , LAVIE A . METEOR: An automatic metric for MT evaluation with improved correlation with human judgments [C]// Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization . Stroudsburg : Association for Computational Linguistics , 2005 : 65 - 72 .

KARAKANTA A , GAIDO M , NEGRI M , et al . Between flexibility and consistency: Joint generation of captions and subtitles [C]// Proceedings of the 18th International Conference on Spoken Language Translation (IWSLT 2021) . Stroudsburg : Association for Computational Linguistics , 2021 : 215 - 225 .

XU P , SAGHIR H , KANG J S , et al . A cross-domain transferable neural coherence model [C]// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . Stroudsburg : Association for Computational Linguistics , 2019 : 678 - 687 .

ELSNER M , CHARNIAK E . Coreference-inspired coherence modeling [C]// Proceedings of the Annual Meeting of the Association for Computational Linguistics . Stroudsburg : Association for Computational Linguistics , 2008 : 41 - 44 .

ROEMMELE M , GORDON A , SWANSON R . Evaluating story generation systems using automated linguistic analyses [C]// Proceedings of the SIGKDD 2017 Workshop on Machine Learning for Creativity . New York : ACM , 2017 : 13 - 17 .

GONG Z X , ZHANG M , ZHOU G D . Document-level machine translation evaluation with gist consistency and text cohesion [C]// Proceedings of the Second Workshop on Discourse in Machine Translation . Stroudsburg : Association for Computational Linguistics , 2015 : 33 - 40 .

MADOTTO A , LIN Z J , WU C S , et al . Personalizing dialogue agents via meta-learning [C]// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . Stroudsburg : Association for Computational Linguistics , 2019 : 5454 - 5459 .

WANG Z Y , WANG X Y , AN B , et al . Towards faithful neural table-to-text generation with content-matching constraints [C]// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Stroudsburg : Association for Computational Linguistics , 2020 : 1072 - 1086 .

KANN K , ROTHE S , FILIPPOVA K . Sentence-level fluency evaluation: References help, but can be spared! [C]// Proceedings of the 22nd Conference on Computational Natural Language Learning . Stroudsburg : Association for Computational Linguistics , 2018 : 313 - 323 .

ZHANG T , KISHORE V , WU F , et al . Bertscore: Evaluating text generation with BERT [C]// Proceedings of the 8th International Conference on Learning Representations . Appleton : ICLR , 2020 : 1 - 43 .

LI W , WU W H , CHEN M Y , et al . Faithfulness in natural language generation: A systematic survey of analysis, evaluation and optimization methods [EB/OL]. ( 2022-03-10 )[ 2023-09-01 ]. http://arxiv.org/abs/2203.05227.pdf http://arxiv.org/abs/2203.05227.pdf .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Image Enhancement via Content Semantic-Aware Multimodal Fusion

A Two-Stage Framework for Complex Question Generation over Knowledge Graph

Chinese Long Text Summarization with Guided Attention

A Rumor Detection Approach Based on Multi-Feature Propagation Tree

Related Author

ZHU Han-cheng

LIU Xin-yu

YAO Rui

SHAO Zhi-wen

ZHOU Yong

LI Lei-da

ZHANG Kun

WANG Yuan-zhuo

Related Institution

School of Artificial Intelligence, Xidian University

Mine Digitization Engineering Research Center of the Ministry of Education

School of Computer Science and Technology / School of Artificial Intelligence, China University of Mining and Technology

School of Computer and Artificial Intelligence, Beijing Technology and Business University

CAS Key Laboratory of Network Data Science and Technology, Institute of Computing Technology, Chinese Academy of Sciences

⁰