电子学报 ›› 2021, Vol. 49 ›› Issue (10): 2048-2060.DOI: 10.12263/DZXB.20200669

• 综述评论 • 上一篇    下一篇

基于深度学习的图像描述综述

石义乐1, 杨文忠2, 杜慧祥1, 王丽花1, 王婷1, 理珊珊1   

  1. 1.新疆大学软件工程技术重点实验室,新疆 乌鲁木齐 830000
    2.新疆大学信息科学与工程学院,新疆 乌鲁木齐 830000
  • 收稿日期:2020-07-08 修回日期:2020-09-13 出版日期:2021-11-29
    • 作者简介:
    • 石义乐 男,1994年生,河南洛阳人.现为新疆大学软件学院研究生.主要研究方向为图像理解. E-mail:2229842870@qq.com
      杨文忠(通信作者) 男,1971年生,新疆乌鲁木齐人. 2011年于武汉大学获得博士学位.现为新疆大学信息科学与工程学院研究生导师,副教授.主要研究方向为网络空间安全、机器学习和算法设计与分析. E-mail:ywz_xy@163.com
    • 基金资助:
    • 国家自然科学基金 (U1603115); 新疆维吾尔自治区自然科学基金 (2017D01C042)

Overview of Image Captions Based on Deep Learning

SHI Yi-le1, YANG Wen-zhong2, DU Hui-xiang1, WANG Li-hua1, WANG Ting1, LI Shan-shan1   

  1. 1.Key Laboratory of Software Engineering Technology, Xinjiang University, Urumqi, Xinjiang 830000, China
    2.School of Information Science and Engineering, Xinjiang University, Urumqi, Xinjiang 830000, China
  • Received:2020-07-08 Revised:2020-09-13 Online:2021-11-29 Published:2021-10-25
    • Supported by:
    • National Natural Science Foundation of China (U1603115); Natural Science Foundation of Xinjiang Uygur Autonomous Region, China (2017D01C042)

摘要:

图像描述旨在通过提取图像的特征输入到语言生成模型中最后输出图像对应的描述,来解决人工智能中自然语言处理与计算机视觉的交叉领域问题——智能图像理解.现对2015—2020年间图像描述方向有代表性的论文进行汇总与分析,以不同核心技术作为分类标准将图像描述大致划分为基于Encoder-Decoder框架的图像描述、基于注意力机制的图像描述、基于强化学习的图像描述、基于生成对抗网络的图像描述和基于新融合数据集的图像描述五大类.使用NIC、Hard-Attention和Neural Talk三个模型在真实数据集MS-COCO数据集上进行实验,并从BLEU1、BLEU2、BLEU3、BLEU4四处平均评分对比分析,展示三个模型效果.本文点明了未来图像描述的发展趋势,并指出了图像描述将要面临的挑战和可深入挖掘的研究方向.

关键词: 智能图像理解, Encoder-Decoder框架, 注意力机制, 强化学习

Abstract:

Image caption aims to extract the features of the image and input the description of the final output image into the language generation model, which solves the intersection of natural language processing and computer vision in artificial intelligence-image understanding. Summarize and analyze representative thesis of image description orientation from 2015 to 2020,different core technologies as classification criteria,it can be roughly divided into: image caption based on Encoder-Decoder framework, image caption based on attention mechanism, image caption based on reinforcement learning, image caption based on Generative Adversarial Networks, and based on new fusion data set these five categories. Use three models of NIC, Hard-Attention and Neural Talk to conduct experiments on the real data set MS-COCO data set, and compare the average scores of BLEU1, BLEU2, BLEU3, and BLEU4 to show the effects of the three models. This article points out the development trend of image caption in the future, and the challenges that image caption will face and the research directions that can be digged in.

Key words: intelligence-image understanding, encoder-decoder framework, attention mechanism, reinforcement learning

中图分类号: