多模态意图感知Token通信方法

于静; 茹志鹏; 王路娟; 徐浩; 张翰林

doi:10.12263/DZXB.20260703

您当前的位置：

首页 >

文章列表页 >

多模态意图感知Token通信方法

更新时间：2026-10-13

- 多模态意图感知Token通信方法
- Multimodal Intent-Aware Token Communication
- 电子学报 2026年页码：1-19
- 作者机构：
  
  1.中央民族大学人工智能研究院，北京 100081
  2.中央民族大学信息工程学院，北京 100081
  3.民族语言智能分析与安全治理教育部重点实验室，北京 100081
- 作者简介：
  
  于静女，1989年3月出生于河北省承德市。现为中央民族大学人工智能研究院、信息工程学院教授，博士生导师。长期从事内容安全治理、人工智能安全等方向的研究。发表学术论文100余篇，获得国际会议最佳论文奖5项。E-mail: jing.yu@muc.edu.cn
  茹志鹏男，2003年6月出生于浙江省绍兴市。现为中央民族大学信息工程学院硕士研究生。主要研究方向为组合图像检索、多模态大模型等。E-mail: ruzhipeng@muc.edu.cn
  王路娟女，2000年6月出生于河北省邢台市。现为中央民族大学信息工程学院硕士研究生。主要研究方向为大语言模型水印、AIGC内容安全等。E-mail: lujuan_wang@muc.edu.cn
  徐浩男，2001年1月出生于云南省曲靖市。现为中央民族大学信息工程学院硕士研究生。主要研究方向为多模态大模型微调、知识编辑等。E-mail: hao_xu@muc.edu.cn
  张翰林男，2002年11月出生于北京市。现为中央民族大学信息工程学院硕士研究生。主要研究方向为大语言模型后训练、指令数据优化、大模型评测与安全对齐。E-mail: hanlin_zhang@muc.edu.cn
- 基金信息：
  
  新一代人工智能国家科技重大专项(2025ZD0123303);北京市高层次创新创业人才支持计划科技新星计划资助(20250484921)
- DOI：10.12263/DZXB.20260703
  中图分类号： TN911.2;TP181
- 收稿：2026-08-05，
  
  录用：2026-08-26，
  
  网络首发：2026-10-13，
- 稿件说明：
移动端阅览
于静, 茹志鹏, 王路娟, 等. 多模态意图感知Token通信方法[J/OL]. 电子学报, 2026,1-19.

YU Jing, RU Zhipeng, WANG Lujuan, et al. Multimodal Intent-Aware Token Communication[J/OL]. ACTA ELECTRONICA SINICA, 2026, 1-19.
于静, 茹志鹏, 王路娟, 等. 多模态意图感知Token通信方法[J/OL]. 电子学报, 2026,1-19. DOI： 10.12263/DZXB.20260703.

YU Jing, RU Zhipeng, WANG Lujuan, et al. Multimodal Intent-Aware Token Communication[J/OL]. ACTA ELECTRONICA SINICA, 2026, 1-19. DOI： 10.12263/DZXB.20260703.

摘要

面向第六代移动通信（sixth-generation mobile communication，6G）人工智能原生网络中可理解、可推理和可协同的多模态传输需求，传统语义通信方法多以文本、图像或多模态特征为传输对象，通常关注源内容压缩和恢复，难以在通信前显式理解并压缩用户由多源信息表达的任务意图。为验证多模态意图在Token通信中的传输价值，本文采用组合图像检索作为下游任务，并提出一种多模态意图感知Token通信方法（Multimodal Intent-Aware Token Communication，MIA-TC）。该方法面向多模态输入中蕴含的任务意图如何在通信前被理解、压缩并传输这一问题，将发送端多模态意图抽取、目标描述生成、文本Token化传输和接收端任务检索组织为统一流程。在具体实现中，发送端不直接传输原始图像、操作文本或高维多模态特征，而是利用多模态大语言模型（Multimodal Large Language Models，MLLM）从参考图像与操作文本中理解用户检索意图，生成面向任务的目标意图描述，并将该描述离散化为少量文本Token后进入基础Token通信链路。接收端恢复目标描述后，利用视觉语言模型完成文本到图像检索。针对发送端意图压缩，本文设计了包含视觉定位、意图推断、反思过滤和压缩输出的提示词，使模型在传输前完成任务相关信息选择，并把中间推理信息压缩为可通信的目标描述。本文进一步提出Token级通信感知检索得分（Token-level Communication-Aware Retrieval Score，Token-CARS）与信道使用级通信感知检索得分（Channel-use-level Communication-Aware Retrieval Score，Channel-CARS），将下游任务性能分别与Token数量和信道资源消耗进行联合评价；通信链路采用基础Token分组、校验、信道编码、调制以及单次传输和可靠重传流程。组合图像检索（Composed Image Retrieval on Common Objects in context，CIRCO）验证集上的实验结果表明，MIA-TC与原始图文Token传输后推理方案在下游任务指标上保持相近水平，并在中高信噪比（Signal-to-Noise Ratio，SNR）的可靠传输设置下取得更好结果；在SNR = 5 dB的可靠重传模式下，MIA-TC在下游任务性能指标前5位召回率（Recall at 5，Recall@5）取得了16.82，Token-CARS@5达到0.506，Channel-CARS@5达到7.124，均高于传输后MLLM推理检索和两阶段语义压缩检索；相比原始图文Token方案，平均Token数量减少93.58%，信道使用次数减少87.34%，平均端到端推理时延减少11.89 ms。相比两阶段语义压缩方法，MIA-TC在下游任务检索性能以及Token-CARS和Channel-CARS综合指标上均表现更好。尤其在Token-CARS和Channel-CARS指标上，MIA-TC均高于原始图文Token方案和两阶段语义压缩方法，说明发送端先进行多模态意图理解与语义压缩，能够在保持任务有效性的同时降低通信负载，为6G AI原生语义通信中的多模态意图传输提供了一种可复现的研究范式。

Abstract

To meet the demand for multimodal transmission with understanding

reasoning

and collaboration capabilities in sixth-generation artificial intelligence (AI)-native networks

conventional semantic communication methods generally take text

images

or multimodal features as transmission objects and mainly focus on source-content compression and reconstruction. Consequently

they have difficulty explicitly understanding and compressing task intent expressed by users through multiple information sources before transmission. To investigate the value of transmitting multimodal intent in Token communication

this paper adopts composed image retrieval as the downstream task and proposes a multimodal intent-aware token communication (MIA-TC) method. MIA-TC addresses how task intent embedded in multimodal inputs can be understood

compressed

and transmitted before communication

and organizes transmitter-side multimodal intent extraction

target-description generation

text Token transmission

and receiver-side task retrieval into a unified pipeline. Specifically

rather than directly transmitting raw images

manipulation text

or high-dimensional multimodal features

the transmitter employs a multimodal large language model (MLLM) to understand the user’s retrieval intent jointly expressed by a reference image and manipulation text

generate a task-oriented target intent description

and discretize the description into a small number of text Tokens for transmission over a basic Token communication link. After recovering the target description

the receiver performs text-to-image retrieval using a vision-language model. For transmitter-side intent compression

we design a prompt consisting of visual grounding

intent inference

reflective filtering

and compressed output

enabling the model to select task-relevant information before transmission and compress intermediate reasoning information into a communicable target description. We further propose the Token-level communication-aware retrieval score (Token-CARS) and the channel-use-level communication-aware retrieval score (Channel-CARS)

which jointly evaluate downstream task performance with the number of Tokens and channel resource consumption

respectively. The communication link employs basic Token grouping

error checking

channel coding

modulation

single-shot transmission

and reliable retransmission. Preliminary experiments on the composed Image retrieval on common objects in context (CIRCO) validation set show that MIA-TC achieves downstream task performance comparable to that of the raw image-text Token scheme with post-transmission reasoning and obtains better results under reliable transmission settings at medium-to-high signal-to-noise ratios (SNRs). In reliable retransmission mode at an SNR of 5 dB

MIA-TC achieves a Recall at 5 (Recall@5) value of 16.82

a Token-CARS@5 of 0.506

and a Channel-CARS@5 value of 7.124

all of which outperform post-transmission MLLM reasoning retrieval and two-stage semantic compression retrieval. Compared with the raw image-text Token scheme

MIA-TC reduces the average number of Tokens by 93.58%

the number of channel uses by 87.34%

and the average end-to-end inference latency by 11.89 ms. MIA-TC also outperforms the two-stage semantic compression method in downstream retrieval performance as well as the Token-CARS and Channel-CARS metrics. In particular

MIA-TC achieves higher Token-CARS and Channel-CARS than both the raw image-text Token scheme and the two-stage semantic compression method. These results demonstrate that performing multimodal intent understanding and semantic compression at the transmitter can reduce communication overhead while preserving task effectiveness

providing a reproducible research paradigm for multimodal intent transmission in sixth-generation AI-native semantic communication systems.

关键词

Keywords

references

Shannon C E . A mathematical theory of communication [J ] . Bell System Technical Journal , 1948 , 27 ( 4 ): 623 - 656 . DOI: 10.1002/j.1538-7305.1948.tb00917.x http://dx.doi.org/10.1002/j.1538-7305.1948.tb00917.x

Bao Jie , Basu P , Dean M , et al . Towards a theory of semantic communication [C ] // 2011 IEEE Network Science Workshop . Piscataway : IEEE , 2011 : 110 - 117 . DOI: 10.1109/nsw.2011.6004632 http://dx.doi.org/10.1109/nsw.2011.6004632

Gündüz D , Qin Zhijin , Aguerri I E , et al . Beyond transmitting bits: Context, semantics, and task-oriented communications [J ] . IEEE Journal on Selected Areas in Communications , 2023 , 41 ( 1 ): 5 - 41 . DOI: 10.1109/jsac.2022.3221853 http://dx.doi.org/10.1109/jsac.2022.3221853

Yang Wanting , Du Hongyang , Zi Qin Liew , et al . Semantic communications for future Internet: Fundamentals, applications, and challenges [J ] . IEEE Communications Surveys & Tutorials , 2023 , 25 ( 1 ): 213 - 250 . DOI: 10.1109/comst.2022.3223224 http://dx.doi.org/10.1109/comst.2022.3223224

Xie Huiqiang , Qin Zhijin , Li Geoffrey Ye , et al . Deep learning enabled semantic communication systems [J ] . IEEE Transactions on Signal Processing , 2021 , 69 : 2663 - 2675 . DOI: 10.1109/tsp.2021.3071210 http://dx.doi.org/10.1109/tsp.2021.3071210

Bourtsoulatze E , Burth Kurka D , Gündüz D . Deep joint source-channel coding for wireless image transmission [J ] . IEEE Transactions on Cognitive Communications and Networking , 2019 , 5 ( 3 ): 567 - 579 . DOI: 10.1109/tccn.2019.2919300 http://dx.doi.org/10.1109/tccn.2019.2919300

Qiao Li , Mashhadi M B , Gao Zhen , et al . Token communications: A large model-driven framework for cross-modal context-aware semantic communications [J ] . IEEE Wireless Communications , 2025 , 32 ( 5 ): 80 - 88 . DOI: 10.1109/mwc.001.2500084 http://dx.doi.org/10.1109/mwc.001.2500084

Liang Chengsi , Du Hongyang , Sun Yao , et al . Generative AI-driven semantic communication networks: Architecture, technologies, and applications [J ] . IEEE Transactions on Cognitive Communications and Networking , 2025 , 11 ( 1 ): 27 - 47 . DOI: 10.1109/tccn.2024.3435524 http://dx.doi.org/10.1109/tccn.2024.3435524

Qiao Li , Mashhadi M B , Gao Zhen , et al . Latency-aware generative semantic communications with pre-trained diffusion models [J ] . IEEE Wireless Communications Letters , 2024 , 13 ( 10 ): 2652 - 2656 . DOI: 10.1109/lwc.2024.3429295 http://dx.doi.org/10.1109/lwc.2024.3429295

Delétang G , Ruoss A , Duquenne P A , et al . Language modeling is compression [C ] // Proceedings of the 12th International Conference on Learning Representations . 2024 . https://proceedings.iclr.cc/paper_files/paper/2024/hash/3cbf627fa24fb6cb576e04e689b9428b-Abstract-Conference.html https://proceedings.iclr.cc/paper_files/paper/2024/hash/3cbf627fa24fb6cb576e04e689b9428b-Abstract-Conference.html .

Vaswani A , Shazeer N , Parmar N , et al . Attention is all you need [C ] // Advances in Neural Information Processing Systems 30 . Curran Associates , Inc. , 2017 : 5998 - 6008 .

Devlin J , Chang Ming-Wei , Lee K , et al . BERT: Pre-training of deep bidirectional transformers for language understanding [C ] // Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , Volume 1 . Stroudsburg : ACL , 2019: 4171 - 4186 . DOI: 10.18653/v1/n19-1423 http://dx.doi.org/10.18653/v1/n19-1423

Brown T B , Mann B , Ryder N , et al . Language models are few-shot learners [C ] // Advances in Neural Information Processing Systems 33 . Curran Associates , Inc. , 2020 : 1877 - 1901 . https://proceedings.mlr.press/v139/radford21a https://proceedings.mlr.press/v139/radford21a . DOI: 10.18653/v1/2021.emnlp-main.734 http://dx.doi.org/10.18653/v1/2021.emnlp-main.734

Radford A , Kim J W , Hallacy C , et al . Learning transferable visual models from natural language supervision [C ] // Proceedings of the 38th International Conference on Machine Learning . PMLR , 2021 , 139 : 8748 - 8763 . https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html .

Dosovitskiy A , Beyer L , Kolesnikov A , et al . An image is worth 16×16 words: Transformers for image recognition at scale [C ] // Proceedings of the 9th International Conference on Learning Representations . 2021 . https://openreview.net/forum id=YicbFdNTTy https://openreview.net/forumid=YicbFdNTTy .

Liu Haotian , Li Chunyuan , Wu Qingyang , et al . Visual instruction tuning [C ] // Advances in Neural Information Processing Systems 36 . Neural Information Processing Systems Foundation, Inc. (NeurIPS) , 2023 : 34892 - 34916 . DOI: 10.52202/075280-1516 http://dx.doi.org/10.52202/075280-1516

OpenAI , Achiam J , Adler S , et al . GPT-4 technical report [PP/OL ] . V6. arXiv ( 2024-03-04 )[ 2026-07-13 ] . https://doi.org/10.48550/arXiv.2303.08774 https://doi.org/10.48550/arXiv.2303.08774 .

Bai Shuai , Cai Yuxuan , Chen Ruizhe , et al . Qwen3-VL technical report [PP/OL ] . V2. arXiv ( 2025-11-27 )[ 2026-07-13 ] . https://doi.org/10.48550/arXiv.2511.21631 https://doi.org/10.48550/arXiv.2511.21631 .

Wei Jason , Wang Xuezhi , Schuurmans D , et al . Chain-of-thought prompting elicits reasoning in large language models [C ] // Advances in Neural Information Processing Systems 35 . Neural Information Processing Systems Foundation, Inc. (NeurIPS) , 2022 : 24824 - 24837 . DOI: 10.52202/068431-1800 http://dx.doi.org/10.52202/068431-1800

Kojima T , Gu Shixiang , Reid M , et al . Large language models are zero-shot reasoners [C ] // Advances in Neural Information Processing Systems 35 . Neural Information Processing Systems Foundation, Inc. (NeurIPS) , 2022 : 22199 - 22213 . DOI: 10.52202/068431-1613 http://dx.doi.org/10.52202/068431-1613

Shinn N , Cassano F , Gopinath A , et al . Reflexion: Language agents with verbal reinforcement learning [C ] // Advances in Neural Information Processing Systems 36 . Neural Information Processing Systems Foundation, Inc. (NeurIPS) , 2023 : 8634 - 8652 . DOI: 10.52202/075280-0377 http://dx.doi.org/10.52202/075280-0377

Vo N , Jiang Lu , Sun Chen , et al . Composing text and image for image retrieval - an empirical odyssey [C ] // 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2019 : 6432 - 6441 . DOI: 10.1109/cvpr.2019.00660 http://dx.doi.org/10.1109/cvpr.2019.00660

Tang Yuanmin , Zhang Jue , Qin Xiaoting , et al . Reason-before-retrieve: One-stage reflective chain-of-thoughts for training-free zero-shot composed image retrieval [C ] // 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2025 : 14400 - 14410 . DOI: 10.1109/cvpr52734.2025.01343 http://dx.doi.org/10.1109/cvpr52734.2025.01343

Wu Hui , Gao Yupeng , Guo Xiaoxiao , et al . Fashion IQ: A new dataset towards retrieving images by natural language feedback [C ] // 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2021 : 11302 - 11312 . DOI: 10.1109/cvpr46437.2021.01115 http://dx.doi.org/10.1109/cvpr46437.2021.01115

Liu Zheyuan , Rodriguez-Opazo C , Teney D , et al . Image retrieval on real-life images with pre-trained vision-and-language models [C ] // 2021 IEEE/CVF International Conference on Computer Vision . Piscataway : IEEE , 2021 : 2105 - 2114 . DOI: 10.1109/iccv48922.2021.00213 http://dx.doi.org/10.1109/iccv48922.2021.00213

Baldrati A , Bertini M , Uricchio T , et al . Effective conditioned and composed image retrieval combining CLIP-based features [C ] // 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2022 : 21434 - 21442 . DOI: 10.1109/cvpr52688.2022.02080 http://dx.doi.org/10.1109/cvpr52688.2022.02080

Saito K , Sohn K , Zhang Xiang , et al . Pic2Word: Mapping pictures to words for zero-shot composed image retrieval [C ] // 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2023 : 19305 - 19314 . DOI: 10.1109/cvpr52729.2023.01850 http://dx.doi.org/10.1109/cvpr52729.2023.01850

Baldrati A , Agnolucci L , Bertini M , et al . Zero-shot composed image retrieval with textual inversion [C ] // 2023 IEEE/CVF International Conference on Computer Vision . Piscataway : IEEE , 2023 : 15292 - 15301 . DOI: 10.1109/iccv51070.2023.01407 http://dx.doi.org/10.1109/iccv51070.2023.01407

Karthik S , Roth K , Mancini M , et al . Vision-by-language for training-free compositional image retrieval [C ] // Proceedings of the 12th International Conference on Learning Representations . 2024 .

Sennrich R , Haddow B , Birch A . Neural machine translation of rare words with subword units [C ] // Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics . Stroudsburg : ACL , 2016 : 1715 - 1725 . DOI: 10.18653/v1/p16-1162 http://dx.doi.org/10.18653/v1/p16-1162

Schuster M , Nakajima K . Japanese and Korean voice search [C ] // 2012 IEEE International Conference on Acoustics, Speech and Signal Processing . Piscataway : IEEE , 2012 : 5149 - 5152 . DOI: 10.1109/icassp.2012.6289079 http://dx.doi.org/10.1109/icassp.2012.6289079

Kudo T . Subword regularization: Improving neural network translation models with multiple subword candidates [C ] // Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics . Stroudsburg : ACL , 2018 : 66 - 75 . DOI: 10.18653/v1/p18-1007 http://dx.doi.org/10.18653/v1/p18-1007

Van den Oord A , Vinyals O , Kavukcuoglu K . Neural discrete representation learning [C ] // Proceedings of the 31st International Conference on Neural Information Processing Systems . New York : ACM , 2017 : 6309 - 6318 . DOI: 10.48550/arXiv.1711.00937 http://dx.doi.org/10.48550/arXiv.1711.00937

Esser P , Rombach R , Ommer B . Taming transformers for high-resolution image synthesis [C ] // 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2021 : 12868 - 12878 . DOI: 10.1109/cvpr46437.2021.01268 http://dx.doi.org/10.1109/cvpr46437.2021.01268

Lin Tsung-Yi , Maire M , Belongie S , et al . Microsoft COCO: Common objects in context [M ] // Computer Vision - ECCV 2014 . ChamSpringer International Publishing2014: 740 - 755 . DOI: 10.1007/978-3-319-10602-1_48 http://dx.doi.org/10.1007/978-3-319-10602-1_48

Proakis J G , Salehi M . Digital communications [M ] . 5th ed . Boston : McGraw-Hill , 2008 .

Forney G D . The viterbi algorithm [J ] . Proceedings of the IEEE , 1973 , 61 ( 3 ): 268 - 278 . DOI: 10.1109/proc.1973.9030 http://dx.doi.org/10.1109/proc.1973.9030

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于稀疏贝叶斯学习的OTFS系统脉冲噪声抑制方法

MDW：最小双权差分特征搜索模型

基于RSMA的通信导航一体化信号波形设计

RIS辅助宽带太赫兹ISAC系统鲁棒保密传输方法

一种加载介质钉层的宽角低扫描损耗毫米波双极化相控阵