面向跨模态通信的信息恢复技术

doi:10.12263/DZXB.20210945

电子学报 ›› 2022, Vol. 50 ›› Issue (7): 1631-1642.DOI: 10.12263/DZXB.20210945

面向跨模态通信的信息恢复技术

徐建博¹^,², 魏昕¹^,², 周亮¹^,²

^1.南京邮电大学通信与信息工程学院，江苏南京 210003
^2.南京邮电大学宽带无线通信与传感网技术教育部重点实验室，江苏南京 210003

收稿日期:2021-07-19 修回日期:2021-09-28 出版日期:2022-07-25
- 作者简介:
- 徐建博男，1996年12月出生，江苏高邮人.南京邮电大学通信与信息工程学院硕士研究生.主要研究方向为多媒体通信. E-mail: xujianbo8881996@163.com
  魏昕男，1983年1月出生，江苏南京人.博士，南京邮电大学教授，硕士生导师.主要研究方向为多媒体通信. E-mail: xwei@njupt.edu.cn
  周亮（通讯作者）男，1981年11月出生，安徽芜湖人.博士，南京邮电大学教授，博士生导师.主要研究方向为多媒体通信. E-mail: liang.zhou@njupt.edu.cn
- 基金资助:
- 国家自然科学基金 (62071254); 江苏高校优势学科建设工程项目; 南京邮电大学宽带无线通信与传感网技术教育部重点实验室开放课题 (JZNY202111)

Information Recovery Technology for Cross-Modal Communications

XU Jian-bo¹^,², WEI Xin¹^,², ZHOU Liang¹^,²

^1.School of Telecommunications & Information Engineering, Nanjing University of Posts and Telecommunications, Nanjing, Jiangsu 210003, China
^2.Key Laboratory of Broadband Wireless Communication and Sensor Network Technology（Ministry of Education）, Nanjing University of Posts and Telecommunications, Nanjing, Jiangsu 210003, China

Received:2021-07-19 Revised:2021-09-28 Online:2022-07-25 Published:2022-07-30
- Supported by:
- National Natural Science Foundation of China (62071254); A Project Funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions; Open Project of Key Laboratory of Broadband Wireless Communication and Sensor Network Technology of Ministry of Education, Nanjing University of Posts and Telecommunications (JZNY202111)

摘要/Abstract

摘要：

针对多模态数据在传输过程中丢失、受到无线信道噪声污染而严重影响跨模态通信质量的问题，提出了一种面向跨模态通信的信息恢复技术，通过充分利用接收端已有数据，采用同模态一对一检索、跨模态一对一检索、跨模态一对多检索等方式，在接收端进行信息恢复.所提方法在公共数据集以及实际跨模态通信平台上进行验证，实验表明，该方法可以实现精准的信息恢复，有效提升了跨模态通信质量.

关键词: 跨模态通信, 信息恢复, 语义融合, 多模态业务

Abstract:

Aiming at the issues of multi-modal data loss and data pollution by noise of wireless channel during the transmission, which seriously affect the cross-modal communication quality, an information recovery technology for cross-modal communications is proposed. In this scheme, by making full use of the existing data at the receiving end, the information is recovered at the receiving end by means of one-to-one intra-modal retrieval, one-to-one cross-modal retrieval, one-to-many cross-modal retrieval, etc. Moreover, the proposed scheme is validated on an open data set and the practical cross-modal communication platform. Experimental results show that the scheme can achieve accurate multi-modal information recovery and effectively improve the quality of cross-modal communications.

Key words: cross-modal communications, information recovery, semantic fusion, multi-modal services

中图分类号:

TP302

徐建博, 魏昕, 周亮. 面向跨模态通信的信息恢复技术[J]. 电子学报, 2022, 50(7): 1631-1642.

XU Jian-bo, WEI Xin, ZHOU Liang. Information Recovery Technology for Cross-Modal Communications[J]. Acta Electronica Sinica, 2022, 50(7): 1631-1642.

图/表 14

参考文献 31

1	CHENS Z, LIANGY C, SUNS H, et al. Vision, requirements, and technology trend of 6G: How to tackle the challenges of system coverage, capacity, user data-rate and movement speed[J]. IEEE Wireless Communications, 2020, 27(2): 218-228.
2	张宏科, 冯博昊, 权伟. 智融标识网络基础研究[J]. 电子学报, 2019, 47(5): 977-982.
	ZHANGH K, FENGB H, QUANW. Fundamental research on smart integration identifier networking[J]. Acta Electronica Sinica, 2019, 47(5): 977-982. (in Chinese)
3	CARLTONB. Nissan partners with HaptX to bring realistic touch to vehicle design[EB/OL]. (2019-03-08)[2022-04-26]. .
4	ZHOUL, WUD, CHENJ X, et al. Cross-modal collaborative communications[J]. IEEE Wireless Communications, 2020, 27(2): 112-117.
5	MOSKVITCHK. Tactile Internet: 5G and the cloud on steroids[J]. Engineering & Technology, 2015, 10(4): 48-53.
6	YUANZ, WEIX, CHENJ X, et al. Ultra-reliability connectivity with redundant D2D transmission scheme for tactile Internet[C]//2019 IEEE Globecom Workshops. Waikoloa, HI: IEEE, 2019: 1-6.
7	ZHOUL. On data-driven delay estimation for media cloud[J]. IEEE Transactions on Multimedia, 2016, 18(5): 905-915.
8	JANKOWSKIM, GÜNDÜZD, MIKOLAJCZYKK, et al. Wireless image retrieval at the edge[J]. IEEE Journal on Selected Areas in Communications, 2021, 39(1): 89-100.
9	JANKOWSKIM, GÜNDÜZD, MIKOLAJCZYKK. Deep joint source-channel coding for wireless image retrieval[C]//2020 IEEE International Conference on Acoustics, Speech and Signal Processing. Barcelona: IEEE, 2020: 5070-5074.
10	ZHOUL, WUD, WEIX, et al. Cross-modal stream scheduling for eHealth[J]. IEEE Journal on Selected Areas in Communications, 2021, 39(2): 426-437.
11	GAOY, WEIX, KANGB, et al. Edge intelligence empowered cross-modal streaming transmission[J]. IEEE Network, 2021, 35(2): 236-243.
12	LIUC F, HUANGW B, SUNF C, et al. LDS-FCM: A linear dynamical system based fuzzy C-means method for tactile recognition[J]. IEEE Transactions on Fuzzy Systems, 2019, 27(1): 72-83.
13	LUOS, MOUW X, ALTHOEFERK, et al. Novel tactile-SIFT descriptor for object shape recognition[J]. IEEE Sensors Journal, 2015, 15(9): 5001-5009.
14	CHUV, MCMAHONI, RIANOL, et al. Robotic learning of haptic adjectives through physical interaction[J]. Robotics and Autonomous Systems, 2015, 63(3): 279-292.
15	WARD-CHERRIERB, PESTELLN, LEPORAN F. NeuroTac: A neuromorphic optical tactile sensor applied to texture recognition[C]//2020 IEEE International Conference on Robotics and Automation. Paris: IEEE, 2020: 2654-2660.
16	李志欣, 凌锋, 张灿龙, 等. 融合两级相似度的跨媒体图像文本检索[J]. 电子学报, 2021, 49(2): 268-274.
	LIZ X, LINGF, ZHANGC L, et al. Cross-media image-text retrieval with two level similarity[J]. Acta Electronica Sinica, 2021, 49(2): 268-274. (in Chinese)
17	HARDOOND R, SZEDMAKS, SHAWE-TAYLORJ. Canonical correlation analysis: An overview with application to learning methods[J]. Neural Computation, 2004, 16(12): 2639-2664.
18	AKAHOS. A kernel method for canonical correlation analysis[J]. Tsukuba, Japan, 2006: 263-269.
19	周沛, 陈后金, 于泽宽, 等. 跨模态医学图像预测综述[J]. 电子学报, 2019, 47(1):220-226.
	ZHOUP, CHENH J, YUZ K, et al. Review of cross-modality medical image prediction[J]. Acta Electronica Sinica, 2019, 47(1): 220-226. (in Chinese)
20	SHANGX D, ZHANGH W, CHUAT S. Deep learning generic features for cross-media retrieval[C]//MMM 2016: Proceedings, Part I, of the 22nd International Conference on MultiMedia Modeling. Miami, FL: Springer, 2016: 264-275.
21	WANGC, YANGH J, MEINELC. Deep semantic mapping for cross-modal retrieval[C]//2015 IEEE 27th International Conference on Tools with Artificial Intelligence. Vietri sul Mare: IEEE, 2015: 234-241.
22	FAYEKH. Speech processing for machine learning: Filter banks, mel-frequency cepstral coefficients(MFCCs) and what's in-between[Z/OL]. (2016-04-21)[2022-04-26]. .
23	SALVADORA, HYNESN, AYTARY, et al. Learning cross-modal embeddings for cooking recipes and food images[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI: IEEE, 2017: 3068-3076.
24	HORIGUCHIS, KANDAN, NAGAMATSUK. Face-voice matching using cross-modal embeddings[C]//Proceedings of the 26th ACM international conference on Multimedia. Seoul: ACM, 2018: 1011-1019.
25	STRESEM, SCHUWERKC, IEPUREA, et al. Multimodal feature-based surface material classification[J]. IEEE Transactions on Haptics, 2017, 10(2): 226-239.
26	张峰, 钟宝江. 基于兴趣目标的图像检索[J]. 电子学报, 2018, 46(8):1915-1923.
	ZHANGF, ZHONGB J. Image retrieval based on interested objects[J]. Acta Electronica Sinica, 2018, 46(8): 1915-1923. (in Chinese)
27	LUC Y, FENGJ S, CHENY D, et al. Tensor robust principal component analysis with a new tensor nuclear norm[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(4): 925-938.
28	ECKERTM A, KAMDARN V, CHANGC E, et al. A cross-modal system linking primary auditory and visual cortices: Evidence from intrinsic fMRI connectivity analysis[J]. Human Brain Mapping, 2008, 29(7): 848-857.
29	VINCENTP, LAROCHELLEH, LAJOIEI, et al. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion[J]. Journal of Machine Learning Research, 2010, 11(12): 3371-3408.
30	WUY L, WANGS H, HUANGQ M. Multi-modal semantic autoencoder for cross-modal retrieval[J]. Neurocomputing, 2019, 331: 165-175.
31	秦姣华, 黄家华, 向旭宇, 等. 基于卷积神经网络和注意力机制的图像检索[J]. 电讯技术, 2021, 61(3): 304-310.
	QINJ H, HUANGJ H, XIANGX Y, et al. Image retrieval based on convolutional neural network and attention mechanism[J]. Telecommunication Engineering, 2021, 61(3): 304-310. (in Chinese)

特性	音频	视频	触觉
时延	≤150 ms	≤350 ms	[1～60] ms
抖动	≤30 ms	≤30 ms	[1～10] ms
数据丢失率	≈30%	≈20%	≈10^-5%
数据传输速率	[22~200] kb/s	[2.5~40] Mb/s	≈128 kb/s

特性	音频	视频	触觉
时延	≤150 ms	≤350 ms	[1～60] ms
抖动	≤30 ms	≤30 ms	[1～10] ms
数据丢失率	≈30%	≈20%	≈10^-5%
数据传输速率	[22~200] kb/s	[2.5~40] Mb/s	≈128 kb/s

材质	训练集	验证集	测试集	总计
小类材质	12	4	4	20
所有材质	1 296	432	432	2 160

材质	训练集	验证集	测试集	总计
小类材质	12	4	4	20
所有材质	1 296	432	432	2 160

全连接1长度	MAP值
64	0.571
128	0.520
256	0.613
512	0.602
1 024	0.606

面向跨模态通信的信息恢复技术

Information Recovery Technology for Cross-Modal Communications

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 14

参考文献 31

相关文章 2

编辑推荐

Metrics

方法	V→V	V→A	V→H	A→V	A→A	A→H	H→V	H→A	H→H	V→All	A→All	H→All
CCA	0.179	0.140	0.138	0.143	0.168	0.139	0.139	0.138	0.185	0.141	0.140	0.146
KCCA	0.199	0.198	0.199	0.201	0.217	0.205	0.200	0.216	0.216	0.201	0.205	0.205
PCA	0.302	0.144	0.152	0.161	0.250	0.159	0.172	0.151	0.279	0.183	0.165	0.175
ICA	0.297	0.147	0.144	0.152	0.247	0.178	0.140	0.178	0.268	0.188	0.175	0.181
AE	0.303	0.154	0.165	0.154	0.242	0.151	0.157	0.145	0.290	0.188	0.162	0.178
VAE	0.296	0.149	0.155	0.151	0.248	0.161	0.153	0.154	0.295	0.187	0.170	0.186
Attention	0.673	0.453	0.367	0.485	0.425	0.338	0.420	0.353	0.338	0.506	0.403	0.355
本文	0.921	0.702	0.588	0.712	0.560	0.474	0.636	0.479	0.451	0.762	0.580	0.510

[1]	胡根生, 查慧敏, 梁栋, 鲍文霞. 结合分类与迁移学习的薄云覆盖遥感图像地物信息恢复[J]. 电子学报, 2017, 45(12): 2855-2862.
[2]	许晓晶;饶妮妮. 一类改进遗传算法的图像信息恢复研究[J]. 电子学报, 2004, 32(7): 1120-1123.