基于松弛Hadamard矩阵的多模态融合哈希方法

doi:10.12263/DZXB.20210760

电子学报 ›› 2022, Vol. 50 ›› Issue (4): 909-920.DOI: 10.12263/DZXB.20210760

所属专题：机器学习交叉融合创新

• 机器学习交叉融合创新 • 上一篇下一篇

基于松弛Hadamard矩阵的多模态融合哈希方法

庾骏¹, 黄伟¹, 张晓波¹, 尹贺峰²()

^1.郑州轻工业大学计算机与通信工程学院，河南郑州 450000
^2.江南大学计算机与人工智能学院，江苏无锡 214000

收稿日期:2021-06-16 修回日期:2022-02-12 出版日期:2022-04-25
- 作者简介:
- 庾骏男，1990年生，湖南邵阳人.工学博士.现在郑州轻工业大学计算机与通信工程学院从事科研和教学工作.研究领域为多媒体分析和检索、深度学习、模式识别.E-mail: yujun@zzuli.edu.cn
  黄伟男，1982年生，河南郑州人.工学博士.现为郑州轻工业大学计算机与通信工程学院副教授.研究领域为模式识别、深度学习.E-mail: hnhw235@163.com
  尹贺峰男，1989年生，江苏无锡人.工学博士.现为江南大学计算机与人工智能学院博士后.研究领域为模式识别、机器学习.E-mail: yin_hefeng@jiangnan.edu.cn
- 基金资助:
- 河南省科技攻关计划项目 (222102210064); 郑州轻工业大学博士科研启动基金 (2021BSJJ025); 国家自然科学基金 (61902361)

Multimodal Fusion Hash Learning Method Based on Relaxed Hadamard Matrix

YU Jun¹, HUANG Wei¹, ZHANG Xiao-bo¹, YIN He-feng²()

^1.The College of Computer and Communication Engineering, Zhengzhou University of Light Industry, zhengzhou, Henan 450000, China
^2.The School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, Jiangsu 214000, China

Received:2021-06-16 Revised:2022-02-12 Online:2022-04-25 Published:2022-04-25
- Supported by:
- Science and Technology Research and Development Program of Henan Province (222102210064); Doctoral Research Fund for Zhengzhou University of Light Industry (2021BSJJ025); National Natural Science Foundation of China (61902361)

摘要/Abstract

摘要：

哈希作为一种有效的数据表征技术，已经在应对爆炸式增长的多媒体数据中扮演了重要的角色.它由于低存储和高效率的优势，在多媒体检索领域受到了越来越多的关注.目前多模态哈希学习方法在多媒体检索任务中得到了较好的研究和发展.然而，多数的方法通过编码特征的内积重构成对相似度来保持原始数据的结构信息，但是带来较复杂的优化问题.此外一些模型缺乏判别性使得检索性能的提升受到限制.为了克服上述问题，本文提出一种新型的多模态融合哈希方法，在类别信息的监督下利用Hadamard矩阵为数据生成目标编码，通过松弛严格的二值约束增大类间的间隔，同时采用图嵌入的方式促进类内的紧凑性.本文提出的方法既保证了模型具有很好的判别能力也简化了优化过程.在3个公开数据集上的实验结果表明，本文提出的方法在多媒体数据检索中是非常有效的，平均性能上相比最优的对比方法提高了8.47%.

关键词: 哈希学习, 多模态融合, Hadamard矩阵, 多媒体检索, 哈希中心

Abstract:

Hashing, as an effective data representation technology, has played an important role in dealing with the explosive growth of multimedia data. Due to the advantages of its low storage and high efficiency, it has received more and more attention in the field of multimedia retrieval. At present, multi-modal hashing methods have been well researched and developed in multimedia retrieval tasks. However, most of these methods usually use the inner product of hashing features to reconstruct larger pairwise similarity, aiming to preserve the structural information of the original data, which will bring more complex optimization problems. Besides, some models lack discriminant ability, which leads to limitations in the improvement of retrieval performance. In order to overcome the above-mentioned problems, this paper proposes a new multi-modal fusion hashing method. Under the supervision of category information, Hadamard matrix is ??used to generate target codes for data, and the margin between categories is increased by relaxing strict binary constraints. At the same time, the graph embedding approach is used to promote compactness within the class. The proposed method in this paper not only ensures the strong discriminative ability of the model, but also simplifies the optimization process. The experimental results on three public datasets show that the method proposed in this paper is very effective in multimedia data retrieval, and the average performance is 8.47% higher than that of the optimal comparison method.

Key words: hash learning, multimodal fusion, Hadamard matrix, multimedia retrieval, hash centers

中图分类号:

TP391

庾骏, 黄伟, 张晓波, 等. 基于松弛Hadamard矩阵的多模态融合哈希方法[J]. 电子学报, 2022, 50(4): 909-920.

Jun YU, Wei HUANG, Xiao-bo ZHANG, et al. Multimodal Fusion Hash Learning Method Based on Relaxed Hadamard Matrix[J]. Acta Electronica Sinica, 2022, 50(4): 909-920.

图/表 11

参考文献 36

1	WANGJ, SHENH T, SONGJ, et al. Hashing for similarity search: A survey[EB/OL]. (2014-08-13)[2022-02-12]..
2	李志欣, 凌锋, 张灿龙, 等. 融合两级相似度的跨媒体图像文本检索[J]. 电子学报, 2021, 49(2): 268-274.
	LIZ X, LINGF, ZHANGC L, et al. Cross-media image-text retrieval with two level similarity[J]. Acta Electronica Sinica, 2021, 49(2): 268-274. (in Chinese)
3	李武军, 周志华. 大数据哈希学习: 现状与趋势[J]. 科学通报, 2015, 60(5): 485-490.
	LIW J, ZHOUZ H. Learning to hash for big data: Current status and future trends[J]. Chinese Science Bulletin, 2015, 60(5): 485-490. (in Chinese)
4	高文. “存得下,查得快”拥抱多媒体大数据时代[J]. 创新科技, 2013, 25(6): 7.
	GAOW. "Save it, check it quickly" Embrace the era of multimedia big data[J]. Innovation Science and Technology, 2013, 25(6): 7. (in Chinese)
5	刘昊淼, 王瑞平, 山世光, 等. 基于离散优化的哈希编码学习方法[J]. 计算机学报, 2019, 42(5): 1149-1160.
	LIUH M, WANGR P, SHANS G, et al. Learning to hash with discrete optimization[J]. Chinese Journal of Computers, 2019, 42(5): 1149-1160. (in Chinese)
6	GONGY, LAZEBNIKS, GORDOA, et al. Iterative quantization: A procrustean approach to learning binary codes for large-scale image retrieval[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 35(12): 2916-2929.
7	KULISB, GRAUMANK. Kernelized locality-sensitive hashing for scalable image search[C]//Proceedings of 12th International Conference on Computer Vision. Kyoto: IEEE, 2009: 2130-2137.
8	JIR, LIUH, CAOL, et al. Toward optimal manifold hashing via discrete locally linear embedding[J]. IEEE Transactions on Image Processing, 2017, 26(11): 5411-5420.
9	KOUTAKIG, SHIRAIK, AMBAIM. Hadamard coding for supervised discrete hashing[J]. IEEE Transactions on Image Processing, 2018, 27(11): 5378-5392.
10	JINL, LIZ, PANY, et al. Weakly-supervised image hashing through masked visual-semantic graph-based reasoning[C]//Proceedings of the 28th ACM International Conference on Multimedia. Seattle: ACM, 2020: 916-924.
11	LIZ, TANGJ, ZHANGL, et al. Weakly-supervised semantic guided hashing for social image retrieval[J]. International Journal of Computer Vision, 2020, 128:2265-2278.
12	姚涛, 孔祥维, 付海燕, 等. 基于映射字典学习的跨模态哈希检索[J]. 自动化学报, 2018, 44(8): 1475-1485.
	YAOT, KONGX W, FUH Y, et al. Projective dictionary learning hashing for cross-modal retrieval[J]. Acta Automatica Sinica, 2018, 44(8): 1475-1485.
13	WANGD, GAOX, WANGX, et al. Label consistent matrix factorization hashing for large-scale cross-modal similarity search[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 41(10): 2466-2479.
14	WANGD, WANGQ, GAOX. Robust and flexible discrete hashing for cross-modal similarity search[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2017, 28(10): 2703-2715.
15	刘昊鑫, 吴小俊, 庾骏. 联合哈希特征和分类器学习的跨模态检索算法[J]. 模式识别与人工智能, 2020, 33(2): 160-165.
	LIUH X, WUX J, YUJ. Joint hashing feature and classifier learning for cross-modal retrieval[J]. Pattern Recognition and Artificial Intelligence, 2020, 33(2): 160-165. (in Chinese)
16	LIUH, JIR, WUY, et al. Supervised matrix factorization for cross-modality hashing[C]//Proceedings of International Joint Conference on Artificial Intelligence. New York: IJCAI, 2016: 1767-1773.
17	LINZ, DINGG, HUM, et al. Semantics-preserving hashing for cross-view retrieval[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 3864-3872.
18	王锦荟, 金露, 李泽超, 等. 基于知识蒸馏的跨模态哈希[J/OL]. (2021-11-04)[2022-02-12]. .
	WANGJ Y, JINL, LIZ C, et al. Cross-Modal Knowledge Distillation Hashing[J/OL]. (2021-11-04)[2022-02-12]. .1444.008.html. (in Chinese)
19	LIZ, TANGJ, MEIT. Deep collaborative embedding for social image understanding[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 41(9): 2070-2083.
20	SONGJ, YANGY, HUANGZ, et al. Effective multiple feature hashing for large scale near duplicate video retrieval[J]. IEEE Transaction on Multimedia, 2013, 15(8): 1997-2008.
21	LIUX, HEJ, LIUD, et al. Compact kernel hashing with multiple features[C]//Proceedings of the ACM International Conference on Multimedia. Seattle: ACM, 2012: 881-884.
22	SHENX, SHENF, SUNQ, et al. Multi-view latent hashing for efficient multimedia search[C]//Proceedings of the ACM International Conference on Multimedia, Seattle: ACM, 2015: 831-834.
23	SHENX, SHENF, LIUL, et al. Multiview discrete hashing for scalable multimedia search[J]. ACM Transactions on Intelligent Systems and Technology, 2018, 9(5): 53-73.
24	LUX, ZHUL, CHENGZ, et al. Online multi-modal hashing with dynamic query-adaption[C]//Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. Paris: ACM, 2019: 715-724.
25	SYLVESTERJ J. LX. Thoughts on inverse orthogonal matrices, simultaneous signsuccessions, and tessellated pavements in two or more colours, with applications to Newton's rule, ornamental tile-work, and the theory of numbers[J]. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 1867, 34(232): 461-475.
26	GIONISA, INDYKP, MOTWANIR. Similarity search in high dimensions via hashing[C]// Proceedings of the International Conference on Very Large Date Bases. Sydney: IEEE, 1999: 518-529.
27	LINM, JIR, LIUH, et al. Hadamard matrix guided online hashing[J]. International Journal of Computer Vision, 2020, 128(6): 2279-2306.
28	XIANGS, NIEF, MENGG, et al. Discriminative least squares regression for multiclass classification and feature selection[J]. IEEE Transactions on Neural Networks and Learning Systems, 2012, 23(11): 1738-1754.
29	RASIWASIAN, COSTAP J, COVIELLOE, et al. A new approach to cross-modal multimedia retrieval[C]//Proceedings of the 18th ACM International Conference on Multimedia, Seattle: ACM, 2010: 251-260.
30	BLEID M, NGA Y, JORDANM I, et al. Latent dirichlet allocation[J]. Journal of Machine Learning Research, 2003, 3(2): 993-1022.
31	RASHTCHIANC, YOUNGP, HODOSHM, et al. Collecting image annotations using amazon's mechanical turk[C]//Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk. Los Angeles: ACM, 2010: 139-147.
32	WEIY, ZHAOY, LUC, et al. Cross-modal retrieval with CNN visual features: A new baseline[J]. IEEE Transactions on Cybernetics, 2016, 47(2): 449-460.
33	KANGY, KIMS, CHOIS. Deep learning to hash with multiple representations[C]//Proceedings of the 12th IEEE Conference on Data Mining. Brussels: IEEE, 2012: 930-935.
34	WANGD, CUIP, OU M, et al. Deep multimodal hashing with orthogonal regularization[C]//Proceedings of the 24th International Joint Conference on Artificial Intelligence. Buenos, Aires: IEEE, 2015: 2291-2297.
35	ZHUL, LUX, CHENGZ, et al. Deep collaborative multi-view hashing for large-scale image search[J]. IEEE Transactions on Image Processing, 2020, 29: 4643-4655.
36	LUX, LIUL, NIEL, et al. Semantic-driven Interpretable Deep Multi-modal Hashing for Large-scale Multimedia Retrieval[J]. IEEE Transactions on Multimedia, 2021, 23: 4541-4554.

方法	方法类型	学习方式	图嵌入方法	引入Hadamard矩阵	两阶段学习
DLLE	单模态哈希	无监督	是	否	否
HCOH	单模态哈希	有监督	否	是	否
MFH	多模态哈希	无监督	是	否	否
MFKH	多模态哈希	有监督	是	否	否
MVLH	多模态哈希	无监督	否	否	否
MVDH	多模态哈希	无监督	是	否	否
OMH-DQ	多模态哈希	有监督	否	否	是
MFH-RH	多模态哈希	有监督	是	是	是

方法	方法类型	学习方式	图嵌入方法	引入Hadamard矩阵	两阶段学习
DLLE	单模态哈希	无监督	是	否	否
HCOH	单模态哈希	有监督	否	是	否
MFH	多模态哈希	无监督	是	否	否
MFKH	多模态哈希	有监督	是	否	否
MVLH	多模态哈希	无监督	否	否	否
MVDH	多模态哈希	无监督	是	否	否
OMH-DQ	多模态哈希	有监督	否	否	是
MFH-RH	多模态哈希	有监督	是	是	是

算法	WiKi				Pascal Sentence				MirFlickr
算法	16 bit	32 bit	64 bit	128 bit	16 bit	32 bit	64 bit	128 bit	16 bit	32 bit	64 bit	128 bit
ITQ	0.5122	0.5359	0.5490	0.5532	0.3602	0.3523	0.3675	0.3803	0.5821	0.5829	0.5854	0.5878
LSH	0.4306	0.4712	0.5085	0.5276	0.1011	0.1243	0.1572	0.2129	0.5515	0.5503	0.5494	0.5466
DLLE	0.5234	0.5330	0.5466	0.5506	0.3631	0.3720	0.3971	0.3959	0.5426	0.5432	0.5424	0.5426
HCOH	0.5450	0.5474	0.5494	0.5490	0.2135	0.4812	0.4846	0.4860	0.5921	0.6659	0.6704	0.6762
MFH	0.4630	0.5040	0.5455	0.5569	0.1834	0.2399	0.2729	0.2731	0.5818	0.5859	0.5848	0.5890
MVLH	0.3027	0.3166	0.3000	0.3045	0.1192	0.1347	0.1200	0.1202	0.5678	0.5701	0.5709	0.5682
OMH-DQ	0.4117	0.4393	0.4556	0.4319	0.4177	0.6719	0.7414	0.7662	0.6532	0.6767	0.6854	0.7030
MFH-RH	0.6890	0.6927	0.6980	0.7005	0.6867	0.7820	0.7820	0.7820	0.6251	0.6900	0.6950	0.7063

算法	WiKi				Pascal Sentence				MirFlickr
算法	16 bit	32 bit	64 bit	128 bit	16 bit	32 bit	64 bit	128 bit	16 bit	32 bit	64 bit	128 bit
ITQ	0.5122	0.5359	0.5490	0.5532	0.3602	0.3523	0.3675	0.3803	0.5821	0.5829	0.5854	0.5878
LSH	0.4306	0.4712	0.5085	0.5276	0.1011	0.1243	0.1572	0.2129	0.5515	0.5503	0.5494	0.5466
DLLE	0.5234	0.5330	0.5466	0.5506	0.3631	0.3720	0.3971	0.3959	0.5426	0.5432	0.5424	0.5426
HCOH	0.5450	0.5474	0.5494	0.5490	0.2135	0.4812	0.4846	0.4860	0.5921	0.6659	0.6704	0.6762
MFH	0.4630	0.5040	0.5455	0.5569	0.1834	0.2399	0.2729	0.2731	0.5818	0.5859	0.5848	0.5890
MVLH	0.3027	0.3166	0.3000	0.3045	0.1192	0.1347	0.1200	0.1202	0.5678	0.5701	0.5709	0.5682
OMH-DQ	0.4117	0.4393	0.4556	0.4319	0.4177	0.6719	0.7414	0.7662	0.6532	0.6767	0.6854	0.7030
MFH-RH	0.6890	0.6927	0.6980	0.7005	0.6867	0.7820	0.7820	0.7820	0.6251	0.6900	0.6950	0.7063

算法	Pascal Sentence
算法	16 bit	32 bit	64 bit	128 bit
DMVH^[33]	0.4832	0.5301	0.6010	0.6720
DMHOR^[34]	0.5701	0.5930	0.6693	0.6860
DCMVH^[35]	0.6793	0.7645	0.7734	0.7791
SIDMH^[36]	0.6681	0.7479	0.7596	0.7660
MFH-RH	0.6867	0.7820	0.7820	0.7820

基于松弛Hadamard矩阵的多模态融合哈希方法

Multimodal Fusion Hash Learning Method Based on Relaxed Hadamard Matrix

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 11

参考文献 36

相关文章 2

编辑推荐

Metrics

数据集	方法	哈希编码长度
数据集	方法	16 bit	32 bit	64 bit	128 bit
WiKi	MFH	0.6580	0.6674	0.6677	0.6752
WiKi	MFH-RH	0.6890	0.6927	0.6980	0.7005
Pascal Sentence	MFH	0.6519	0.7527	0.7527	0.7527
Pascal Sentence	MFH-RH	0.6867	0.7820	0.7820	0.7820
MirFlickr	MFH	0.6420	0.6867	0.6881	0.6868
MirFlickr	MFH-RH	0.6251	0.6900	0.6950	0.7063

[1]	杨冰, 莫文博, 姚金良. 三维掌纹局部方向特征二进制编码研究[J]. 电子学报, 2021, 49(8): 1533-1540.
[2]	尹安容, 谢湘, 匡镜明. Hadamard纠错码结合支持向量机在多分类问题中的应用[J]. 电子学报, 2008, 36(1): 122-126.