电子学报 ›› 2022, Vol. 50 ›› Issue (7): 1631-1642.DOI: 10.12263/DZXB.20210945

• 学术论文 • 上一篇    下一篇

面向跨模态通信的信息恢复技术

徐建博1,2, 魏昕1,2, 周亮1,2   

  1. 1.南京邮电大学通信与信息工程学院,江苏 南京 210003
    2.南京邮电大学宽带无线通信与传感网技术教育部重点实验室,江苏 南京 210003
  • 收稿日期:2021-07-19 修回日期:2021-09-28 出版日期:2022-07-25
    • 作者简介:
    • 徐建博 男,1996年12月出生,江苏高邮人.南京邮电大学通信与信息工程学院硕士研究生.主要研究方向为多媒体通信. E-mail: xujianbo8881996@163.com
      魏 昕 男,1983年1月出生,江苏南京人.博士,南京邮电大学教授,硕士生导师.主要研究方向为多媒体通信. E-mail: xwei@njupt.edu.cn
      周 亮(通讯作者) 男,1981年11月出生,安徽芜湖人.博士,南京邮电大学教授,博士生导师.主要研究方向为多媒体通信. E-mail: liang.zhou@njupt.edu.cn
    • 基金资助:
    • 国家自然科学基金 (62071254); 江苏高校优势学科建设工程项目; 南京邮电大学宽带无线通信与传感网技术教育部重点实验室开放课题 (JZNY202111)

Information Recovery Technology for Cross-Modal Communications

XU Jian-bo1,2, WEI Xin1,2, ZHOU Liang1,2   

  1. 1.School of Telecommunications & Information Engineering, Nanjing University of Posts and Telecommunications, Nanjing, Jiangsu 210003, China
    2.Key Laboratory of Broadband Wireless Communication and Sensor Network Technology(Ministry of Education), Nanjing University of Posts and Telecommunications, Nanjing, Jiangsu 210003, China
  • Received:2021-07-19 Revised:2021-09-28 Online:2022-07-25 Published:2022-07-30
    • Supported by:
    • National Natural Science Foundation of China (62071254); A Project Funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions; Open Project of Key Laboratory of Broadband Wireless Communication and Sensor Network Technology of Ministry of Education, Nanjing University of Posts and Telecommunications (JZNY202111)

摘要:

针对多模态数据在传输过程中丢失、受到无线信道噪声污染而严重影响跨模态通信质量的问题,提出了一种面向跨模态通信的信息恢复技术,通过充分利用接收端已有数据,采用同模态一对一检索、跨模态一对一检索、跨模态一对多检索等方式,在接收端进行信息恢复.所提方法在公共数据集以及实际跨模态通信平台上进行验证,实验表明,该方法可以实现精准的信息恢复,有效提升了跨模态通信质量.

关键词: 跨模态通信, 信息恢复, 语义融合, 多模态业务

Abstract:

Aiming at the issues of multi-modal data loss and data pollution by noise of wireless channel during the transmission, which seriously affect the cross-modal communication quality, an information recovery technology for cross-modal communications is proposed. In this scheme, by making full use of the existing data at the receiving end, the information is recovered at the receiving end by means of one-to-one intra-modal retrieval, one-to-one cross-modal retrieval, one-to-many cross-modal retrieval, etc. Moreover, the proposed scheme is validated on an open data set and the practical cross-modal communication platform. Experimental results show that the scheme can achieve accurate multi-modal information recovery and effectively improve the quality of cross-modal communications.

Key words: cross-modal communications, information recovery, semantic fusion, multi-modal services

中图分类号: