基于双注意模型的图像描述生成方法研究

doi:10.12263/DZXB.20210696

电子学报 ›› 2022, Vol. 50 ›› Issue (5): 1123-1130.DOI: 10.12263/DZXB.20210696

基于双注意模型的图像描述生成方法研究

卓亚琦¹, 魏家辉², 李志欣²

^1.桂林理工大学理学院, 广西桂林 541004
^2.广西师范大学广西多源信息挖掘与安全重点实验室, 广西桂林 541004

收稿日期:2021-05-31 修回日期:2021-09-22 出版日期:2022-05-25 发布日期:2022-06-18
通讯作者: 李志欣
作者简介:卓亚琦　女,1976年6月出生,陕西咸阳人.桂林理工大学理学院讲师.研究方向为图像理解与机器学习.E-mail: zhuoyaqi@126.com
魏家辉　男,1993年11月出生,山东东营人.广西师范大学计算机科学与工程学院博士研究生.研究方向为图像理解与机器学习.E-mail: weijh@stu.gxnu.edu.cn
李志欣男,1971年10月出生,广西桂林人.现为广西师范大学计算机科学与工程学院教授、博士生导师.研究领域为图像理解、机器学习与跨媒体计算.E-mail: lizx@gxnu.edu.cn
基金资助:
国家自然科学基金(61966004);广西自然科学基金(2019GXNSFDA245018);广西研究生教育创新计划(XYCBZ2021002)

Research on Image Captioning Based on Double Attention Model

ZHUO Ya-qi¹, WEI Jia-hui², LI Zhi-xin²

^1.College of Science，Guilin University of Technology，Guilin，Guangxi 541004，China
^2.Guangxi Key Lab of Multi-source Information Mining and Security，Guangxi Normal University，Guilin，Guangxi 541004，China

Received:2021-05-31 Revised:2021-09-22 Online:2022-05-25 Published:2022-06-18
Contact: LI Zhi-xin

摘要/Abstract

摘要：

现有图像描述生成方法的注意模型通常采用单词级注意，从图像中提取局部特征作为生成当前单词的视觉信息输入，缺乏准确的图像全局信息指导.针对这个问题，提出基于语句级注意的图像描述生成方法，通过自注意机制从图像中提取语句级的注意信息，来表示生成语句所需的图像全局信息.在此基础上，结合语句级注意和单词级注意进一步提出了双注意模型，以此来生成更准确的图像描述.通过在模型的中间阶段实施监督和优化，以解决信息间的干扰问题.此外，将强化学习应用于两阶段的训练来优化模型的评估度量.通过在MSCOCO和Flickr30K两个基准数据集上的实验评估，结果表明本文提出的方法能够生成更加准确和丰富的描述语句，并且在各项评价指标上优于现有的多种基于注意机制的方法.

关键词: 图像描述生成, 编码器-解码器架构, 单词级注意, 语句级注意, 双注意模型, 强化学习

Abstract:

The attention model of existing image captioning approaches usually adopt word-level attention, which extracts local features from images. The local features are used as the visual information input to generate the current word, lacking accurate image global information guidance. To solve this problem, this paper proposed image captioning approach based on sentence-level attention. The approach employs the self-attention mechanism to extract the sentence-level attention information from the image, which is used to represent the global image information needed to generate sentences. On this basis, we further proposes a double attention model which combines sentence-level attention with word-level attention to generate more accurate description. We implement supervision and optimization in the intermediate stage of the model to solve the problem of information interference. In addition, reinforcement learning is applied in two-stage training to optimize the evaluation metric of the model. Finally, we evaluated our approach on two baseline datasets, i.e. MSCOCO and Flickr30K. Experimental results show that the proposed approach can generate more accurate and richer captions. Hence it outperforms many state-of-the-art image captioning approaches based on attention mechanism in various evaluation metrics.

Key words: image captioning, encoder-decoder architecture, word-level attention, sentence-level attention, double attention model, reinforcement learning

中图分类号:

TP391

卓亚琦, 魏家辉, 李志欣. 基于双注意模型的图像描述生成方法研究[J]. 电子学报, 2022, 50(5): 1123-1130.

ZHUO Ya-qi, WEI Jia-hui, LI Zhi-xin. Research on Image Captioning Based on Double Attention Model[J]. Acta Electronica Sinica, 2022, 50(5): 1123-1130.

图/表 7

参考文献 28

1	李志欣, 魏海洋, 张灿龙, 等. 图像描述生成研究进展[J]. 计算机研究与发展, 2021, 58(9): 1951‑1974.
	LI Zhi-xin, WEI Hai-yang, ZHANG Can-long, et al. Research progress on image captioning[J]. Journal of Computer Research and Development, 2021, 58(9): 1951‑1974. (in Chinese)
2	DAI J, LI Y, HE K, et al. R-FCN: Object detection via region-based fully convolutional networks[C]//Advances in Neural Information Processing Systems. Cambridge, USA: MIT Press, 2016: 379‑387.
3	LI Zhi-xin, LIN Lan, ZHANG Can-long, et al. A semi-supervised learning approach based on adaptive weighted fusion for automatic image annotation [J]. ACM Transactions on Multimedia Computing, Communications, and Applications, 2021, 17(1): article37.
4	VINYALS O, TOSHEV A, BENGIO S, et al. Show and tell: A neural image caption generator[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos, USA: IEEE Computer Society, 2015: 3156‑3164.
5	KARPATHY A, LI F F. Deep visual-semantic alignments for generating image descriptions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos, USA: IEEE Computer Society, 2015: 3128‑3137.
6	MAO J, XU W, YANG Y, et al. Deep captioning with multimodal recurrent neural networks(m-RNN)[EB/OL]. [2021-09-22]. .
7	XU K, BA J, KIROS R, et al. Show, attend and tell: Neural image caption generation with visual attention[C]//Proceedings of International Conference on Machine Learning. Cambridge, USA: MIT Press, 2015: 2048‑2057.
8	CHEN L, ZHANG H, XIAO J, et al. SCA-CNN: Spatial and channel-wise attention in convolutional networks for image captioning[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos, USA: IEEE Computer Society, 2017: 6298‑6306.
9	LU J, XIONG C, PARIKH D, et al. Knowing when to look: Adaptive attention via a visual sentinel for image captioning[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos, USA: IEEE Computer Society, 2017: 3242‑3250.
10	YOU Q, JIN H, WANG Z, et al. Image captioning with semantic attention[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos, USA: IEEE Computer Society, 2016: 4651‑ 4659.
11	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Advances in Neural Information Processing Systems. Cambridge, USA: MIT Press, 2017: 5998‑6008.
12	YU A W, DOHAN D, LUONG M T, et al. QANet: Combining local convolution with global self-attention for reading comprehension[EB/OL]. [2021-09-22]. .
13	RANZATO M A, CHOPRA S, AULI M, et al. Sequence level training with recurrent neural networks[EB/OL]. [2021-09-22]. .
14	RENNIE S J, MARCHERET E, MROUEH Y, et al. Self-critical sequence training for image captioning[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos, USA: IEEE Computer Society, 2017: 1179‑1195.
15	PAPINENI K, ROUKOS S, WARD T, et al. BLEU: a method for automatic evaluation of machine translation[C]//Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. Stroudsburg, USA: ACL, 2002: 311‑318.
16	BANERJEE S, LAVIE A. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments[C]//Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. Stroudsburg, USA: ACL, 2005: 65‑72.
17	LIN C Y. ROUGE: A package for automatic evaluation of summaries[C]//Proceedings of the ACL Workshop on Text Summarization Branches Out. Stroudsburg, USA: ACL, 2004: 74‑81.
18	VEDANTAM R, ZITNICK C L, PARIKH D. CIDEr: Consensus-based image description evaluation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos, USA: IEEE Computer Society, 2015: 4566‑4575.
19	WILLIAMS R J. Simple statistical gradient-following algorithms for connectionist reinforcement learning[J]. Machine learning, 1992, 8(3-4): 229‑256.
20	HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos, USA: IEEE Computer Society, 2016: 770‑778.
21	DIEDERIK K, JIMMY B. ADAM: A method for stochastic optimization[EB/OL]. [2021-09-22]. .
22	JIA X, GAVVES E, FERNANDO B, et al. Guiding the long-short term memory model for image caption generation[C]//Proceedings of the IEEE International Conference on Computer Vision. Piscataway, USA: IEEE, 2015: 2407‑2415.
23	WANG C, YANG H, BARTZ C, et al. Image captioning with deep bidirectional LSTMs[C]//Proceedings of the 24th ACM International Conference on Multimedia. New York, USA: ACM, 2016: 988‑997.
24	FU K, JIN J, CUI R, et al. Aligning where to see and what to tell: Image caption with region-based attention and scene-specific contexts[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(12): 2321‑2334.
25	CHEN X, MA L, JIANG W, et al. Regularizing RNNs for caption generation by reconstructing the past with the present[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos, USA: IEEE Computer Society, 2018: 7995‑8003.
26	LIU S, ZHU Z, YE N, et al. Improved image captioning via policy gradient optimization of SPIDEr[C]//Proceedings of the IEEE International Conference on Computer Vision. Piscataway, USA: IEEE, 2017: 873‑881.
27	ANDERSON P, HE X, BUEHLER C, et al. Bottom-up and top-down attention for image captioning and visual question answering[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos, USA: IEEE Computer Society, 2018: 6077‑ 6086.
28	HUANG Fei-cheng, LI Zhi-xin, WEI Hai-yang, et al. Boost image captioning with knowledge reasoning [J]. Machine Learning, 2020, 109(12): 2313‑2332.

APPROACH	BLEU‑1	BLEU‑2	BLEU‑3	BLEU‑4	METEOR	ROUGE‑L	CIDER
SAIC	73.6	56.8	42.6	32.7	25.9	54.1	100.4
SAIC（FT）	75.5	58.9	44.8	34.8	27.2	55.7	107.2
WAIC	73.3	56.9	42.8	33.0	26.3	54.3	101.7
WAIC（FT）	75.8	59.1	45.4	35.6	27.5	55.9	108.6
DAICs	74.2	58.0	44.1	34.2	26.5	54.6	105.4
DAICs（FT）	75.5	59.6	45.9	36.1	27.6	56.0	109.7
DAICp	74.8	58.1	44.4	34.7	26.5	54.9	105.7
DAICp（FT）	76.1	60.2	46.7	36.5	27.9	56.4	111.3

APPROACH	BLEU‑1	BLEU‑2	BLEU‑3	BLEU‑4	METEOR	ROUGE‑L	CIDER
SAIC	73.6	56.8	42.6	32.7	25.9	54.1	100.4
SAIC（FT）	75.5	58.9	44.8	34.8	27.2	55.7	107.2
WAIC	73.3	56.9	42.8	33.0	26.3	54.3	101.7
WAIC（FT）	75.8	59.1	45.4	35.6	27.5	55.9	108.6
DAICs	74.2	58.0	44.1	34.2	26.5	54.6	105.4
DAICs（FT）	75.5	59.6	45.9	36.1	27.6	56.0	109.7
DAICp	74.8	58.1	44.4	34.7	26.5	54.9	105.7
DAICp（FT）	76.1	60.2	46.7	36.5	27.9	56.4	111.3

APPROACH	BLEU‑1	BLEU‑2	BLEU‑3	BLEU‑4	METEOR	ROUGE‑L	CIDER
GOOGLE NIC^［4］	66.6	46.1	32.9	24.6	-	-	-
SOFT-ATTENTION^［7］	70.7	49.2	34.4	24.3	23.9	-	-
GLSTM^［22］	67.0	49.1	35.8	26.4	22.74	-	81.25
BI-LSTM^［23］	67.2	49.2	35.2	24.4	-	-	-
RA+SS^［24］	72.4	55.5	41.8	31.3	24.8	53.2	95.5
ATT^［10］	70.9	53.7	40.2	30.4	24.3	-	-
SCA-CNN^［8］	71.9	54.8	41.1	31.1	25.0	-	-
ADAPTIVE^［9］	74.2	58.0	43.9	33.2	26.6	-	108.5
ARNET^［25］	74.0	57.6	44.0	33.5	26.1	54.6	103.4
PG-BCMR^［26］	75.4	59.1	44.5	33.2	25.7	55.0	101.3
SCST：ATT2ALL^［14］	-	-	-	34.2	26.7	55.7	114.0
UP-DOWN^［27］	79.8	-	-	36.3	27.7	56.9	120.1
IIEK^［28］	79.3	63.8	49.0	37.3	27.3	57.4	121.2
SAIC（XE）	75.5	58.9	44.8	34.8	27.2	55.7	107.2
WAIC（XE）	75.8	59.1	45.4	35.6	27.5	55.9	108.6
DAIC（XE）	76.1	60.2	46.7	36.5	27.9	56.4	111.3
SAIC（RL）	78.9	62.8	47.6	36.1	27.7	57.0	117.7
WAIC（RL）	79.6	63.5	48.5	36.9	28.0	57.3	119.6
DAIC（RL）	80.4	64.6	49.3	37.8	28.2	57.9	121.8

APPROACH	BLEU‑1	BLEU‑2	BLEU‑3	BLEU‑4	METEOR	ROUGE‑L	CIDER
GOOGLE NIC^［4］	66.6	46.1	32.9	24.6	-	-	-
SOFT-ATTENTION^［7］	70.7	49.2	34.4	24.3	23.9	-	-
GLSTM^［22］	67.0	49.1	35.8	26.4	22.74	-	81.25
BI-LSTM^［23］	67.2	49.2	35.2	24.4	-	-	-
RA+SS^［24］	72.4	55.5	41.8	31.3	24.8	53.2	95.5
ATT^［10］	70.9	53.7	40.2	30.4	24.3	-	-
SCA-CNN^［8］	71.9	54.8	41.1	31.1	25.0	-	-
ADAPTIVE^［9］	74.2	58.0	43.9	33.2	26.6	-	108.5
ARNET^［25］	74.0	57.6	44.0	33.5	26.1	54.6	103.4
PG-BCMR^［26］	75.4	59.1	44.5	33.2	25.7	55.0	101.3
SCST：ATT2ALL^［14］	-	-	-	34.2	26.7	55.7	114.0
UP-DOWN^［27］	79.8	-	-	36.3	27.7	56.9	120.1
IIEK^［28］	79.3	63.8	49.0	37.3	27.3	57.4	121.2
SAIC（XE）	75.5	58.9	44.8	34.8	27.2	55.7	107.2
WAIC（XE）	75.8	59.1	45.4	35.6	27.5	55.9	108.6
DAIC（XE）	76.1	60.2	46.7	36.5	27.9	56.4	111.3
SAIC（RL）	78.9	62.8	47.6	36.1	27.7	57.0	117.7
WAIC（RL）	79.6	63.5	48.5	36.9	28.0	57.3	119.6
DAIC（RL）	80.4	64.6	49.3	37.8	28.2	57.9	121.8

[1]	易令, 李泽平. 基于深度强化学习的码率自适应算法研究[J]. 电子学报, 2022, 50(5): 1192-1200.
[2]	张涛, 张文涛, 代凌, 陈婧怡, 王丽, 魏倩茹. 基于序贯博弈多智能体强化学习的综合模块化航空电子系统重构方法[J]. 电子学报, 2022, 50(4): 954-966.
[3]	肖硕, 黄珍珍, 张国鹏, 杨树松, 江海峰, 李天旭. 基于SAC的多智能体深度强化学习算法[J]. 电子学报, 2021, 49(9): 1675-1681.
[4]	管永明, 王刚, 骆凯波, 吕梁, 吕晓雯, 史玉良. 基于极端不平衡学习的泛化低压异常箱表关系识别研究与应用[J]. 电子学报, 2021, 49(8): 1507-1514.
[5]	冀大雄, 方文巍, 朱华, 李硕, 唐元贵, 田宇, 姚倩. 基于相对测量的水下机器人主动定位方法研究[J]. 电子学报, 2021, 49(7): 1249-1256.
[6]	周东明, 张灿龙, 李志欣, 王智文. 基于多层级视觉融合的图像描述模型[J]. 电子学报, 2021, 49(7): 1286-1290.
[7]	韩冲, 王俊丽, 吴雨茜, 张超波. 基于神经进化的深度学习模型研究综述[J]. 电子学报, 2021, 49(2): 372-379.
[8]	刘润滋, 吴伟华, 张文柱, 周笛, 张琰. 基于图学习的密集空间网络传输资源调度方法[J]. 电子学报, 2021, 49(11): 2133-2137.
[9]	石义乐, 杨文忠, 杜慧祥, 王丽花, 王婷, 理珊珊. 基于深度学习的图像描述综述[J]. 电子学报, 2021, 49(10): 2048-2060.
[10]	陈思光, 陈佳民, 赵传信. 基于深度强化学习的云边协同计算迁移研究[J]. 电子学报, 2021, 49(1): 157-166.
[11]	黄志清, 曲志伟, 张吉, 张严心, 田锐. 基于深度强化学习的端到端无人驾驶决策[J]. 电子学报, 2020, 48(9): 1711-1719.
[12]	杨启萌, 禹龙, 田生伟, 艾山·吾买尔. 基于深度强化学习的维吾尔语人称代词指代消解[J]. 电子学报, 2020, 48(6): 1077-1083.
[13]	朱斐, 刘全, 傅启明, 陈冬火, 王辉, 伏玉琛. 一种不稳定环境下的策略搜索及迁移方法[J]. 电子学报, 2017, 45(2): 257-266.
[14]	刘全, 于俊, 王辉, 傅启明, 朱斐. 一种基于随机投影的贝叶斯时间差分算法[J]. 电子学报, 2016, 44(11): 2752-2757.
[15]	仵博, 郑红燕, 冯延蓬, 陈鑫. 一种基于模型的可分解贝叶斯在线强化学习[J]. 电子学报, 2014, 42(7): 1429-1434.

基于双注意模型的图像描述生成方法研究

Research on Image Captioning Based on Double Attention Model

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 7

参考文献 28

相关文章 15

编辑推荐

Metrics