融合两级相似度的跨媒体图像文本检索

doi:10.12263/DZXB.20191037

电子学报 ›› 2021, Vol. 49 ›› Issue (2): 268-274.DOI: 10.12263/DZXB.20191037

融合两级相似度的跨媒体图像文本检索

李志欣¹, 凌锋¹, 张灿龙¹, 马慧芳²

1. 广西师范大学广西多源信息挖掘与安全重点实验室, 广西桂林 541004;
2. 西北师范大学计算机科学与工程学院, 甘肃兰州 730070

收稿日期:2019-09-10 修回日期:2020-09-20 出版日期:2021-02-25
- 通讯作者:
- 李志欣
- 作者简介:
- 凌锋男,1993年11月出生,广西崇左人.广西师范大学计算机科学与信息工程学院硕士研究生.研究方向为机器学习与跨媒体检索.E-mail:lingfeng93@126.com;张灿龙 男,1975年10月出生,湖南娄底人.现为广西师范大学计算机科学与信息工程学院教授.研究领域为目标跟踪与模式识别;马慧芳 女,1981年7月出生,甘肃兰州人.现为西北师范大学计算机科学与工程学院教授.研究领域为数据挖掘与机器学习.
- 基金资助:
- 国家自然科学基金 (No.61663004，No.61966004，No.61866004，No.61762078）; 广西自然科学基金 (No.2019GXNSFDA245018，No.2018GXNSFDA281009，No.2017GXNSFAA198365）

Cross-Media Image-Text Retrieval with Two Level Similarity

LI Zhi-xin¹, LING Feng¹, ZHANG Can-long¹, MA Hui-fang²

1. Guangxi Key Laboratory of Multi-Source Information Mining and Security, Guangxi Normal University, Guilin, Guangxi 541004, China;
2. College of Computer Science and Engineering, Northwest Normal University, Lanzhou, Gansu 730070, China

Received:2019-09-10 Revised:2020-09-20 Online:2021-02-25 Published:2021-02-25
- Corresponding author:
- LI Zhi-xin
- Supported by:
- National Natural Science Foundation of China (No.61663004, No.61966004, No.61866004, No.61762078); Natural Science Foundation of Guangxi Zhuang Autonomous Region, China (No.2019GXNSFDA245018, No.2018GXNSFDA281009, No.2017GXNSFAA198365)

摘要/Abstract

摘要： 为了更好地揭示图像和文本之间潜在的语义关联，提出了一种融合两级相似度的跨媒体检索方法，构建两个子网分别处理全局特征和局部特征，以获取图像和文本之间更好的语义匹配.图像分为整幅图像和一些图像区域两种表示，文本也分为整个语句和一些单词两种表示.设计一个两级对齐方法分别匹配图像和文本的全局和局部表示，并融合两种相似度学习跨媒体的完整表示.在MSCOCO和Flickr30K数据集上的实验结果表明，本文方法能够使图像和文本的语义匹配更准确，优于许多当前先进的跨媒体检索方法.

关键词: 卷积神经网络, 自注意力网络, 两级相似度, 跨媒体检索

Abstract: To better reveal the latent semantic correlation between image and text, this paper proposes a cross media retrieval method by fusing two level similarity, which constructs two subnets to deal with global features and local features respectively so as to obtain better semantic matching between image and text. The image representation is divided into whole image and some image regions, and the text representation is also divided into whole sentence and some words. A two level alignment method is designed to match the global and local representation of image and text, and the two similarities are fused to learn the complete cross-media representation. The experimental results on MSCOCO and Flickr30K datasets show that the proposed method can make the semantic matching of image and text more accurate, and is superior to many state-of-the-art cross-media retrieval methods.

Key words: convolutional neural network, self-attention network, two level similarity, cross-media retrieval

中图分类号:

TP391

李志欣, 凌锋, 张灿龙, 等. 融合两级相似度的跨媒体图像文本检索[J]. 电子学报, 2021, 49(2): 268-274.

LI Zhi-xin, LING Feng, ZHANG Can-long, et al. Cross-Media Image-Text Retrieval with Two Level Similarity[J]. Acta Electronica Sinica, 2021, 49(2): 268-274.

参考文献

[1] Kiros R,Salakhutdinov R,Zemel R S.Unifying visual-semantic embeddings with multimodal neural language models[OL].http://arxiv.org/abs/1411.2539,2014-11-10.
[2] Feng F,Wang X,Li R.Cross-modal retrieval with correspondence autoencoder[A].Proceedings of the 22nd ACM International Conference on Multimedia[C].New York,USA:ACM,2014.7-16.
[3] Wang L,Li Y,Lazebnik S.Learning deep structure-preserving image-text embeddings[A].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition[C].Los Alamitos,USA:IEEE Computer Society,2016.5005-5013.
[4] Wang L,Li Y,Huang J,et al.Learning two-branch neural networks for image-text matching tasks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2019,41(2):394-407.
[5] Vendrov I,Kiros R,Fidler S,et al.Order-embeddings of images and language[OL].http://arxiv.org/abs/1511.06361,2015-11-19.
[6] Nam H,Ha J W,Kim J.Dual attention networks for multimodal reasoning and matching[A].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition[C].Los Alamitos,USA:IEEE Computer Society,2017.299-307.
[7] Faghri F,Fleet D J,Kiros J R,et al.VSE++:Improving visual-semantic embeddings with hard negatives[A].Proceedings of the 28th British Machine Vision Conference[C].Durham,UK:BMVA,2017.121-132.
[8] Huang Y,Wu Q,Song C,et al.Learning semantic concepts and order for image and sentence matching[A].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition[C].Los Alamitos,USA:IEEE Computer Society,2018.6163-6171.
[9] Lee K H,Chen X,Hua G,et al.Stacked cross attention for image-text matching[A].Proceedings of the European Conference on Computer Vision[C].Cham,Swit:Spriger,2018.201-216.
[10] He K,Zhang X,Ren S,et al.Deep residual learning for image recognition[A].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition[C].Los Alamitos,USA:IEEE Computer Society,2016.770-778.
[11] Zhang H,Goodfellow I,Metaxas D N,et al.Self-attention generative adversarial networks[A].Proceedings of International Conference on Machine Learning[C].New York,USA:ACM,2019.7354-7363.
[12] Zhang X,Zhao J,LeCun Y.Character-level convolutional networks for text classification[A].Advances in Neural Information Processing Systems[C].Cambridge,UK:MIT Press,2015.649-657.
[13] Hochreiter S,Schmidhuber J.Long short-term memory[J].Neural computation,1997,9(8):1735-1780.
[14] Ren S,He K,Girshick R,et al.Faster R-CNN:Towards real-time object detection with region proposal networks[A].Advances in Neural Information Processing Systems[C].Cambridge,UK:MIT Press,2015.91-99.
[15] Anderson P,He X,Buehler C,et al.Bottom-up and top-down attention for image captioning and visual question answering[A].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition[C].Los Alamitos,USA:IEEE Computer Society,2018.6077-6086.
[16] Huang Z,Xu W,Yu K.Bidirectional LSTM-CRF models for sequence tagging[OL].http://arxiv.org/abs/1508.01991,2015-08-09.
[17] Mnih V,Heess N,Graves A.Recurrent models of visual attention[A].Advances in Neural Information Processing Systems[C].Cambridge,UK:MIT Press,2014.2204-2212.
[18] Hoffer E,Ailon N.Deep metric learning using triplet network[A].Proceedings of International Workshop on Similarity-Based Pattern Recognition[C].Cham,Swit:Springer,2015.84-92.
[19] Karpathy A,Fei-Fei L.Deep visual-semantic alignments for generating image descriptions[A].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition[C].Los Alamitos,USA:IEEE Computer Society,2015.3128-3137.
[20] Yan F,Mikolajczyk K.Deep correlation for matching images and text[A].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition[C].Los Alamitos,USA:IEEE Computer Society,2015.3441-3450.
[21] Huang Y,Wang W,Wang L.Instance-aware image and sentence matching with selective multimodal LSTM[A].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition[C].Los Alamitos,USA:IEEE Computer Society,2017.2310-2318.
[22] Gu J,Cai J,Joty S,et al.Look,imagine and match:Improving textual-visual cross-modal retrieval with generative models[A].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition[C].Los Alamitos,USA:IEEE Computer Society,2018.7181-7189.
[23] Niu Z,Zhou M,Wang L,et al.Hierarchical multimodal LSTM for dense visual-semantic embedding[A].Proceedings of the IEEE International Conference on Computer Vision[C].Piscataway,USA:IEEE,2017.1881-1889.
[24] Ma L,Lu Z,Shang L,et al.Multimodal convolutional neural networks for matching image and sentence[A].Proceedings of the IEEE International Conference on Computer Vision[C].Piscataway,USA:IEEE,2015.2623-2631.

融合两级相似度的跨媒体图像文本检索

Cross-Media Image-Text Retrieval with Two Level Similarity

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	黄赟, 张帆, 郭威, 陈立, 羊光. 一种基于数据标准差的卷积神经网络量化方法[J]. 电子学报, 2023, 51(3): 639-647.
[2]	吕杭, 蒋明峰, 李杨, 张鞠成, 王志康. 基于混合时频域特征的卷积神经网络心律失常分类方法的研究[J]. 电子学报, 2023, 51(3): 701-711.
[3]	许新征, 李杉. 基于特征膨胀卷积模块的轻量化技术研究[J]. 电子学报, 2023, 51(2): 355-364.
[4]	丁琪, 田萱, 孙国栋. 基于注意力增强的热点感知新闻推荐模型[J]. 电子学报, 2023, 51(1): 93-104.
[5]	张永梅, 孙捷. 基于动静态特征双输入神经网络的咳嗽声诊断COVID-19算法[J]. 电子学报, 2023, 51(1): 202-212.
[6]	袁海英, 成君鹏, 曾智勇, 武延瑞. Mobile_BLNet：基于Big-Little Net的轻量级卷积神经网络优化设计[J]. 电子学报, 2023, 51(1): 180-191.
[7]	吴靖, 叶晓晶, 黄峰, 陈丽琼, 王志锋, 刘文犀. 基于深度学习的单帧图像超分辨率重建综述[J]. 电子学报, 2022, 50(9): 2265-2294.
[8]	毛国君, 王者浩, 黄山, 王翔. 基于剪边策略的图残差卷积深层网络模型[J]. 电子学报, 2022, 50(9): 2205-2214.
[9]	袁海英, 曾智勇, 成君鹏. 面向灵活并行度的稀疏卷积神经网络加速器[J]. 电子学报, 2022, 50(8): 1811-1818.
[10]	张志文, 刘天歌, 聂鹏举. 基于实景数据增强和双路径融合网络的实时街景语义分割算法[J]. 电子学报, 2022, 50(7): 1609-1620.
[11]	王相海, 赵晓阳, 王鑫莹, 赵克云, 宋传鸣. 非抽取小波边缘学习深度残差网络的单幅图像超分辨率重建[J]. 电子学报, 2022, 50(7): 1753-1765.
[12]	张波, 陆云杰, 秦东明, 邹国建. 一种卷积自编码深度学习的空气污染多站点联合预测模型[J]. 电子学报, 2022, 50(6): 1410-1427.
[13]	丁毅, 沈薇, 李海生, 钟琼慧, 田明宇, 李洁. 面向CNN的区块链可信隐私服务计算模型[J]. 电子学报, 2022, 50(6): 1399-1409.
[14]	张云, 化青龙, 姜义成, 徐丹. 基于混合型复数域卷积神经网络的三维转动舰船目标识别[J]. 电子学报, 2022, 50(5): 1042-1049.
[15]	廖勇, 李玉杰. 一种轻量化低复杂度的FDD大规模MIMO系统CSI反馈方法[J]. 电子学报, 2022, 50(5): 1211-1217.