Image Captioning Model Based on Multi‑Level Visual Fusion

ZHOU Dong-ming; ZHANG Can-long; LI Zhi-xin; WANG Zhi-wen

doi:10.12263/DZXB.20191296

您当前的位置：

首页 >

文章列表页 >

Image Captioning Model Based on Multi‑Level Visual Fusion

PAPERS | 更新时间：2025-12-08

- Image Captioning Model Based on Multi‑Level Visual Fusion
- ACTA ELECTRONICA SINICA Vol. 49, Issue 7, Pages: 1286-1290(2021)
- 作者机构：
  
  1.广西师范大学广西多源信息挖掘与安全重点实验室，广西桂林 541004
  2.广西科技大学计算机科学与通信工程学院，广西柳州 545006
- 作者简介：
- 基金信息：
- DOI：10.12263/DZXB.20191296
  CLC： TP181
- Received：21 November 2019，
  
  Revised：2021-02-04，
  
  Published：25 July 2021
- 稿件说明：
移动端阅览
周东明,张灿龙,李志欣等.基于多层级视觉融合的图像描述模型[J].电子学报,2021,49(07):1286-1290.

ZHOU Dong-ming,ZHANG Can-long,LI Zhi-xin,et al.Image Captioning Model Based on Multi‑Level Visual Fusion[J].ACTA ELECTRONICA SINICA,2021,49(07):1286-1290.
周东明,张灿龙,李志欣等.基于多层级视觉融合的图像描述模型[J].电子学报,2021,49(07):1286-1290. DOI： 10.12263/DZXB.20191296.

ZHOU Dong-ming,ZHANG Can-long,LI Zhi-xin,et al.Image Captioning Model Based on Multi‑Level Visual Fusion[J].ACTA ELECTRONICA SINICA,2021,49(07):1286-1290. DOI： 10.12263/DZXB.20191296.

摘要

传统方法在视觉策略网络中只关注实体，不能够推理出实体和属性之间的联系，在语言策略网络存在暴露偏差和误差累计问题.为此，提出了一个基于强化学习的多层级视觉融合网络模型.在视觉策略网络中通过多层级神经网络模块将视觉特征转化为视觉知识的特征集.融合网络生成使描述语句更加流畅的虚词，用于视觉策略网络和语言策略网络的互动.在语言策略网络中使用基于强化学习的自批评策略梯度算法对视觉融合网络实现端到端的优化.实验结果表明，该模型在MS‑COCO数据集取得不错效果，将Karpathy分割测试中的CIDEr值从120.1提高到124.3.

Abstract

Traditional methods only focus on entities in the visual strategy network and cannot deduce the relationship between entities and attributes. There are problems of exposure bias and error accumulation in the language strategy network. Therefore

this paper proposes a multi‑level visual fusion network model based on reinforcement learning. In the visual strategy network

multi‑level sub‑neural network module is used to transform visual features into feature sets of visual knowledge. The fusion network generates the function words which make the description sentences more fluent and can be used for the interaction between the visual strategy network and the language strategy network. The gradient algorithm of self‑criticism strategy based on reinforcement learning is used to optimize the visual fusion network end‑to‑end. The experimental results show that the model can get good results in MS‑COCO data set and improve the CIDEr value of Karpathy segmentation test from 120.1 to 124.3.

关键词

Keywords

references

Chen S ， Jin Q ， Wang P . Say as you wish ： fine‑grained control of image caption generation with abstract scene graphs ［A］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition ［C］. USA ： IEEE ， 2020 . 9962 - 9971 .

Shi J ， Zhang H ， Li J . Explainable and explicit visual reasoning over scene graphs ［A］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition ［C］. California ： IEEE ， 2019 . 8376 - 8384 .

张志昌，曾扬扬，庞雅丽 . 融合语义角色和自注意力机制的中文文本蕴含识别［J］. 电子学报， 2020 ， 48 （ 11 ）： 2162 - 2169 .

ZHANG Zhi‑chang ， ZENG Yang‑yang ， PANG Ya‑li . A Chinese textual entailment recognition method incorporating semantic role and self‑attention ［J］. Acta Electronica Sinica ， 2020 ， 48 （ 11 ）： 2162 - 2169 . （in Chinese）

Rennie S J ， Marcheret E ， Mroueh Y . Self‑critical sequence training for image captioning ［A］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition ［C］. Hawaii ： IEEE ， 2017 . 7008 - 7024 .

Lu J ， Yang J ， Batra D . Neural baby talk ［A］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition ［C］. Salt Lake City ： IEEE ， 2018 . 7219 - 7228 .

Anderson P ， He X ， Buehler C . Bottom‑up and top‑down attention for image captioning and visual question answering ［A］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition ［C］. Salt Lake City ： IEEE ， 2018 . 6077 - 6086 .

汤鹏杰，王瀚漓，许恺晟 . LSTM 逐层多目标优化及多层概率融合的图像描述［J］. 自动化学报， 2018 ， 44 （ 7 ）： 1237 - 1249 .

TANG Peng‑jie ， WANG Han‑li ， XU Kai‑sheng . Multi‑ objective layer‑wise optimization and multi‑level probability fusion for image description generation using lstm ［J］. Acta Automatica Sinica ， 2018 ， 44 （ 7 ）： 1237 - 1249 . （in Chinese）

Deshpande A ， Aneja J ， Wang L. Fast ， diverse and accurate image captioning guided by part‑of‑speech ［A］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition ［C］. California ： IEEE ， 2019 . 10695 - 10704 .

Yang X ， Tang K ， Zhang H . Auto‑encoding scene graphs for image captioning ［A］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition ［C］. California ： IEEE ， 2019 . 10685 - 10694 .

Chen L ， Zhang H ， Xiao J . SCA‑CNN： spatial and channel‑ wise attention in convolutional networks for image captioning ［A］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition ［C］. Hawaii ： IEEE ， 2017 . 5659 - 5667 .

Feng Y ， Ma L ， Liu W . Unsupervised image captioning ［A］. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition ［C］. California ： IEEE ， 2019 . 4125 - 4134 .

Jiang W ， Ma L ， Jiang Y . Recurrent fusion network for image captioning ［A］. Proceedings of the European Conference on Computer Vision ［C］. Germany ： Springer ， 2018 . 499 - 515 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Overview of Image Captions Based on Deep Learning

Neighborhood and Hypergraph Collaboration for Session-Based Recommendation

Object Detection Based on EIMYOLO for High-Resolution Remote Sensing Images

A Detection Method for Android Repackaged Malware Based on Sensitive Component Function Call Graph

Single-Image High Dynamic Range Reconstruction Based on Multi-Attention and Perceptual Weighted Learning

Related Author

SHI Yi-le

YANG Wen-zhong

DU Hui-xiang

WANG Li-hua

WANG Ting

LI Shan-shan

CHEN Rong-yuan

WEN Jie-bin

Related Institution

Key Laboratory of Software Engineering Technology, Xinjiang University

School of Information Science and Engineering, Xinjiang University

College of Frontier Intersection, Hunan University of Technology and Business

Key Laboratory of Hunan Province for Statistical Learning and Intelligent Computation, Hunan University of Technology and Business

School of Computer Science, Hunan University of Technology and Business

⁰