基于多模态一致性协同提示微调的判断标记学习方法

韩日东; 许靖昊; 李仲年; 孙统风; 许新征

doi:10.12263/DZXB.20250727

您当前的位置：

首页 >

文章列表页 >

基于多模态一致性协同提示微调的判断标记学习方法

学术论文 | 更新时间：2026-10-16

- 基于多模态一致性协同提示微调的判断标记学习方法
- Learning From True-False Labels Via Multi-Modal Consistency Collaborative Prompt Tuning
- 电子学报 2026年54卷第6期页码：2846-2864
- 作者机构：
  
  1.中国矿业大学计算机科学与技术学院/人工智能学院，江苏徐州 221116
  2.浙江大学计算机辅助设计与图形系统全国重点实验室，浙江杭州 310058
  3.中国矿业大学矿山数字化教育部工程研究中心，江苏徐州 221116
- 作者简介：
  
  韩日东男，1995年12月出生于黑龙江省绥化市。2024年博士毕业于吉林大学计算机科学与技术学院。现为中国矿业大学计算机科学与技术学院/人工智能学院讲师。主要研究方向为机器学习、自然语言处理。E-mail: hanrd@cumt.edu.cn
  许靖昊男，2000年6月出生于江苏省徐州市。2025年硕士毕业于中国矿业大学计算机科学与技术学院/人工智能学院。主要研究方向为弱监督学习、预训练模型的提示学习。E-mail: jinghaoxu@cumt.edu.cn
  李仲年男，1990年5月出生于江苏省徐州市。现为中国矿业大学计算机科学与技术学院/人工智能学院预聘副教授、硕士生导师。主要研究方向为机器学习、医学影像处理。E-mail: zhongnianli@cumt.edu.cn
  孙统风男，1977年4月出生于江苏省徐州市。现为中国矿业大学计算机科学与技术学院/人工智能学院副教授、硕士生导师。主要研究方向为深度学习、知识重检索。E-mail: suntf@cumt.edu.cn
  许新征男，1980年8月出生于安徽省宿州市。现为中国矿业大学计算机科学与技术学院/人工智能学院教授、博士生导师。主要研究方向为机器学习与数据挖掘、人工智能与模式识别、医学图像处理等。E-mail: xxzheng@cumt.edu.cn
- 基金信息：
  
  国家自然科学基金(61976217;62306320);江苏省自然科学基金(BK20231063);浙江大学计算机辅助设计与图形系统全国重点实验室开放课题(A2424)
- DOI：10.12263/DZXB.20250727
  中图分类号： TP181;TP391
- 收稿：2025-08-20，
  
  录用：2026-04-30，
  
  纸质出版：2026-06-25
- 稿件说明：
移动端阅览
韩日东, 许靖昊, 李仲年, 等. 基于多模态一致性协同提示微调的判断标记学习方法[J]. 电子学报, 2026, 54(06): 2846-2864.

HAN Ridong, XU Jinghao, LI Zhongnian, et al. Learning From True-False Labels Via Multi-Modal Consistency Collaborative Prompt Tuning[J]. Acta Electronica Sinica, 2026, 54(06): 2846-2864.
韩日东, 许靖昊, 李仲年, 等. 基于多模态一致性协同提示微调的判断标记学习方法[J]. 电子学报, 2026, 54(06): 2846-2864. DOI：10.12263/DZXB.20250727

HAN Ridong, XU Jinghao, LI Zhongnian, et al. Learning From True-False Labels Via Multi-Modal Consistency Collaborative Prompt Tuning[J]. Acta Electronica Sinica, 2026, 54(06): 2846-2864. DOI：10.12263/DZXB.20250727

摘要

近期利用大规模未标注数据进行提示学习受到广泛关注。其中，判断标记学习范式通过从候选集中随机采样类别，并借助预训练视觉-语言模型进行二值化（True/False）判断，从而生成高精度的弱监督标记，大幅提升了标注效率，推动了提示学习领域的快速发展。然而，其随机均匀的采样机制不可避免地导致部分类别监督样本缺失，致使下游任务模型难以构建精确的类别语义空间，并且现有方法切断了文本与视觉特征之间的跨模态联合优化路径，破坏了预训练模型原有的跨模态泛化能力。针对上述局限性，本文提出一种基于多模态一致性协同提示微调的判断标记学习方法（true-false labels learning method based on Multi-modal Consistency Collaborative Prompt tuning，MCCP）。该方法在文本和视觉双模态上分别引入一致性约束，并设计了跨模态协同优化机制。具体而言，在文本模态，设计了大语言模型驱动的知识协同机制，利用大语言模型为各个类别生成细粒度的文本描述，通过约束原始文本提示与类别文本描述之间的语义一致性，补齐监督样本缺失类别的语义信息盲区，并且隐式缓解监督信息不足的不利影响；在视觉模态，提出了基于自监督学习的一致性提示学习机制，通过构建多视图，约束原始图像、弱增强视图与强增强视图两两之间的特征分布一致性，迫使视觉提示网络提取对数据增强操作不敏感的深层语义特征，显著提升了视觉语义表征的鲁棒性与泛化能力；在跨模态联合优化方面，引入了基于训练轮次的余弦动态加权策略，以实现预训练先验知识与领域适应性学习表征之间的平滑过渡与渐进式融合，同时采用了分阶段梯度更新策略，分别优化文本、视觉特定模态的一致性损失与跨模态图像分类的经验风险一致性交叉熵损失，有效避免了多模态学习中的梯度冲突问题，促进了跨模态语义空间的深度协同对齐。在CIFAR-100、Tiny-ImageNet、Caltech-101、Food-101、Stanford Cars、DTD（Describable Textures Dataset）、FGVC-Aircraft（Fine-Grained Visual Classification of Aircraft）、ImageNet-A和ImageNet-R九个多类别图像分类数据集上的实验结果表明，该方法在利用对比语言-图像预训练模型（Contrastive Language-Image Pre-training，CLIP）标注和人工标注的两种判断标记数据集上，分别取得了73.58%和74.20%的最优平均分类准确率，同时在跨数据集迁移设置下的绝大多数目标数据集上均展现出最优的迁移泛化性能，验证了该方法的有效性。本文有效解决了判断标记学习范式的监督信号缺失挑战，并为实现高效、鲁棒的多模态跨域知识迁移提供了方法支持。

Abstract

Prompt learning utilizing large-scale unlabeled data has recently received widespread attention. Among these

the true-false label learning paradigm randomly samples a category from the candidate set and leverages pre-trained vision-language models to make binary judgments

i.e.

True or False

thereby generating high-precision weak supervision signals

substantially improving annotation efficiency

and driving the rapid development of the prompt learning field. However

its random uniform sampling mechanism inevitably leads to missing supervision samples for certain categories

making it difficult for downstream task models to construct an accurate category semantic space

and existing methods sever the cross-modal joint optimization path between textual and visual features

destroying the original cross-modal generalization capability of pre-trained models. To address the aforementioned limitations

this paper proposes a true-false labels learning method based on multi-modal consistency collaborative prompt tuning (MCCP). This method introduces consistency constraints in both textual and visual modalities

and designs a cross-modal collaborative optimization strategy. Specifically

in the textual modality

a large language model-driven knowledge collaboration mechanism is designed to generate fine-grained textual descriptions for each category. By constraining the semantic consistency between original textual prompts and category textual descriptions

it fills the semantic information blind spots of categories with missing supervision samples

and implicitly mitigates the effects of insufficient supervision information. In the visual modality

a consistency prompt learning mechanism based on self-supervised learning is proposed. By constructing multiple views

it constrains the pairwise feature distribution consistency among original images

weakly augmented views

and strongly augmented views

thereby forcing the visual prompt network to extract deep semantic features insensitive to data augmentation operations

significantly enhancing the robustness and generalization of visual-semantic representations. In terms of cross-modal joint optimization

an epoch-based cosine dynamic weighting strategy is introduced to achieve a smooth transition and gradual fusion between pre-trained prior knowledge and domain-adaptive feature representations

while a staged gradient update scheme is adopted to separately optimize the consistency losses of text- and vision-specific modalities and the empirical risk-consistency cross-entropy loss of cross-modal image classification

effectively avoiding the gradient conflict problem in multi-modal learning and promoting the deep collaborative alignment of cross-modal semantic spaces. Experimental results on nine multi-class image classification datasets

including CIFAR-100

Tiny-ImageNet

Caltech-101

Food-101

Stanford Cars

describable textures dataset (DTD)

fine-grained visual classification of aircraft (FGVC-Aircraft)

ImageNet-A and ImageNet-R

demonstrate that this method achieves state-of-the-art average classification accuracies of 73.58% and 74.20% on the True-False labels datasets annotated by the contrastive language-image pre-training model (CLIP) and human annotators

respectively

and simultaneously exhibits optimal transfer generalization performance on the vast majority of target datasets under the cross-dataset transfer setting

verifying the effectiveness of MCCP. This paper effectively addresses the missing supervision signal challenge of true-false labels learning paradigm

providing methodological support for achieving efficient and robust multi-modal cross-domain knowledge transfer.

关键词

Keywords

references

Radford A , Kim J W , Hallacy C , et al . Learning transferable visual models from natural language supervision [C ] // Proceedings of the International Conference on Machine Learning . PMLR , 2021 : 8748 - 8763 . DOI: 10.48550/arXiv.2103.00020 http://dx.doi.org/10.48550/arXiv.2103.00020

Li Junnan , Li Dongxu , Xiong Caiming , et al . BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation [C ] // Proceedings of the International Conference on Machine Learning . PMLR , 2022 : 12888 - 12900 .

Jia Chao , Yang Yinfei , Xia Ye , et al . Scaling up visual and vision-language representation learning with noisy text supervision [C ] // Proceedings of the International Conference on Machine Learning . PMLR , 2021 : 4904 - 4916 . DOI: 10.1145/3474085.3475622 http://dx.doi.org/10.1145/3474085.3475622

Liu Haotian , Li Chunyuan , Wu Qingyang , et al . Visual instruction tuning [C ] // Advances in Neural Information Processing Systems 36 . Neural Information Processing Systems Foundation, Inc. (NeurIPS) , 2023 : 34892 - 34916 . DOI: 10.52202/075280-1516 http://dx.doi.org/10.52202/075280-1516

李鑫尧 , 李晶晶 , 朱磊 , 等 . 资源受限的大模型高效迁移学习算法研究综述 [J ] . 计算机学报 , 2024 , 47 ( 11 ): 2491 - 2521 .

Li Xinyao , Li Jingjing , Zhu Lei , et al . Efficient transfer learning of large models with limited resources: A survey [J ] . Chinese Journal of Computers , 2024 , 47 ( 11 ): 2491 - 2521 . (in Chinese)

Zhou Kaiyang , Yang Jingkang , Loy C C , et al . Learning to prompt for vision-language models [J ] . International Journal of Computer Vision , 2022 , 130 ( 9 ): 2337 - 2348 . DOI: 10.1007/s11263-022-01653-1 http://dx.doi.org/10.1007/s11263-022-01653-1

Zhou Kaiyang , Yang Jingkang , Loy C C , et al . Conditional prompt learning for vision-language models [C ] // 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2022 : 16795 - 16804 . DOI: 10.1109/cvpr52688.2022.01631 http://dx.doi.org/10.1109/cvpr52688.2022.01631

Khattak M U , Rasheed H , Maaz M , et al . Maple: Multi-modal prompt learning [C ] // 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2023 : 19113 - 19122 . DOI: 10.1109/cvpr52729.2023.01832 http://dx.doi.org/10.1109/cvpr52729.2023.01832

Jia Menglin , Tang Luming , Chen B C , et al . Visual prompt tuning [C ] // Proceedings of the European Conference on Computer Vision . Cham : Springer Nature Switzerland , 2022 : 709 - 727 . DOI: 10.1007/978-3-031-19827-4_41 http://dx.doi.org/10.1007/978-3-031-19827-4_41

Gao Peng , Geng Shijie , Zhang Renrui , et al . CLIP-adapter: Better vision-language models with feature adapters [J ] . International Journal of Computer Vision , 2024 , 132 ( 2 ): 581 - 595 . DOI: 10.1007/s11263-023-01891-x http://dx.doi.org/10.1007/s11263-023-01891-x

Menghini C , Delworth A , Bach S . Enhancing CLIP with CLIP: Exploring pseudolabeling for limited-label prompt tuning [C ] // Advances in Neural Information Processing Systems 36 . Neural Information Processing Systems Foundation, Inc. (NeurIPS) , 2023 : 60984 - 61007 . DOI: 10.52202/075280-2665 http://dx.doi.org/10.52202/075280-2665

Zhang Jiahan , Wei Qi , Liu Feng , et al . Candidate pseudolabel learning: Enhancing vision-language models by prompt tuning with unlabeled data [C ] // Proceedings of the 41st International Conference on Machine Learning . PMLR , 2024 : 60004 - 60020 .

Li Zhongnian , Xu Jinghao , Ying Peng , et al . Learning from true-false labels via multi-modal prompt retrieving [C ] // Proceedings of the 42nd International Conference on Machine Learning . New York : ACM , 2025 : 36652 - 36666 .

Wang Xudong , Wu Zhirong , Lian Long , et al . Debiased learning from naturally imbalanced pseudo-labels [C ] // 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2022 : 14627 - 14637 . DOI: 10.1109/cvpr52688.2022.01424 http://dx.doi.org/10.1109/cvpr52688.2022.01424

李仲年 , 皇甫志宇 , 杨凯杰 , 等 . 基于图提示的半监督开放词汇多标记学习 [J ] . 计算机研究与发展 , 2025 , 62 ( 2 ): 432 - 442 .

Li Zhongnian , Huangfu Zhiyu , Yang Kaijie , et al . Semi-supervised open vocabulary multi-label learning based on graph prompting [J ] . Journal of Computer Research and Development , 2025 , 62 ( 2 ): 432 - 442 . (in Chinese)

孙仁科 , 许靖昊 , 皇甫志宇 , 等 . 基于视觉-语言预训练模型的零样本迁移学习方法综述 [J ] . 计算机工程 , 2024 , 50 ( 10 ): 1 - 15 .

Sun Renke , Xu Jinghao , Huangfu Zhiyu , et al . Survey of zero-shot transfer learning methods based on vision-language pre-trained models [J ] . Computer Engineering , 2024 , 50 ( 10 ): 1 - 15 . (in Chinese)

Menon S , Vondrick C . Visual classification via description from large language models [C ] // Proceedings of the 11th International Conference on Learning Representations . PMLR , 2023 : 1 - 17 .

Pratt S , Covert I , Liu R , et al . What does a platypus look like? Generating customized prompts for zero-shot image classification [C ] // 2023 IEEE/CVF International Conference on Computer Vision . Piscataway : IEEE , 2023 : 15645 - 15655 . DOI: 10.1109/iccv51070.2023.01438 http://dx.doi.org/10.1109/iccv51070.2023.01438

Yao Hantao , Zhang Rui , Xu Changsheng . TCP: Textual-based class-aware prompt tuning for visual-language model [C ] // 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2024 : 23438 - 23448 . DOI: 10.1109/cvpr52733.2024.02212 http://dx.doi.org/10.1109/cvpr52733.2024.02212

Xie Jingjing , Zhang Yuxin , Peng Jun , et al . TextRefiner: Internal visual feature as efficient refiner for vision-language models prompt tuning [J ] . Proceedings of the AAAI Conference on Artificial Intelligence , 2025 , 39 ( 8 ): 8718 - 8726 . DOI: 10.1609/aaai.v39i8.32942 http://dx.doi.org/10.1609/aaai.v39i8.32942

Dosovitskiy A , Beyer L , Kolesnikov A , et al . An image is worth 16×16 words: Transformers for image recognition at scale [C ] // Proceedings of the International Conference on Learning Representations . LCLR , 2021 : 1 - 21 .

张震宇 , 姜远 . 面向特征演变环境的标记噪声鲁棒学习算法 [J ] . 计算机研究与发展 , 2023 , 60 ( 8 ): 1740 - 1753 .

Zhang Zhenyu , Jiang Yuan . Label noise robust learning algorithm in environments evolving features [J ] . Journal of Computer Research and Development , 2023 , 60 ( 8 ): 1740 - 1753 . (in Chinese)

Cour T , Sapp B , Taskar B . Learning from partial labels [J ] . Journal of Machine Learning Research , 2011 , 12 : 1501 - 1536 .

Tang Caizhi , Zhang Minling . Confidence-rated discriminative partial label learning [C ] // Proceedings of the Thirty-First AAAl Conference on Artificial Intelligence . New York : ACM , 2017 : 2611 - 2617 . DOI: 10.1609/aaai.v31i1.10775 http://dx.doi.org/10.1609/aaai.v31i1.10775

Zhang Minling , Zhou Binbin , Liu Xuying . Partial label learning via feature-aware disambiguation [C ] // Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . New York : ACM , 2016 : 1335 - 1344 . DOI: 10.1145/2939672.2939788 http://dx.doi.org/10.1145/2939672.2939788

Xu Ning , Lv Jiaqi , Geng Xin . Partial label learning via label enhancement [J ] . Proceedings of the AAAI Conference on Artificial Intelligence , 2019 , 33 ( 1 ): 5557 - 5564 . DOI: 10.1609/aaai.v33i01.33015557 http://dx.doi.org/10.1609/aaai.v33i01.33015557

Hüllermeier E , Beringer J . Learning from ambiguously labeled examples [J ] . Intelligent Data Analysis , 2006 , 10 ( 5 ): 419 - 439 . DOI: 10.3233/ida-2006-10503 http://dx.doi.org/10.3233/ida-2006-10503

Ishida T , Niu Gang , Hu Weihua , et al . Learning from complementary labels [C ] // Advances in Neural Information Processing Systems . Curran Associates , Inc. , 2017 : 5644 - 5654 .

Ishida T , Niu Gang , Menon A , et al . Complementary-label learning for arbitrary losses and models [C ] // Proceedings of the 36th International Conference on Machine Learning . PMLR , 2019 : 2971 - 2980 .

Gao Yi , Zhang Minling . Discriminative complementary-label learning with weighted loss [C ] // Proceedings of the 38th International Conference on Machine Learning . PMLR , 2021 : 3587 - 3597 .

Yu Xiyu , Liu Tongliang , Gong Mingming , et al . Learning with biased complementary labels [C ] // Proceedings of the European Conference on Computer Vision . Cham : Springer International Publishing , 2018 : 69 - 85 .

李自强 , 杨薇 , 杨先凤 , 等 . 基于弱标签争议的半自动分类数据标注方法 [J ] . 电子学报 , 2024 , 52 ( 8 ): 2891 - 2899 .

Li Ziqiang , Yang Wei , Yang Xianfeng , et al . The semi-automatic classification data labeling method based on dispute about weak label [J ] . Acta Electronica Sinica , 2024 , 52 ( 8 ): 2891 - 2899 . (in Chinese)

DeepSeek-AI , Zhu Qihao , Guo Daya , et al . DeepSeek-coder-V2: Breaking the barrier of closed-source models in code intelligence [PP/OL ] . V1.arXiv ( 2024-06-17 )[ 2025-08-20 ] . https://doi.org/10.48550/arXiv.2406.11931 https://doi.org/10.48550/arXiv.2406.11931 .

Krizhevsky A . Learning multiple layers of features from tiny images [D ] . Toronto : University of Toronto , 2009 .

Le Ya , Yang Xuan . Tiny ImageNet visual recognition challenge [R ] . Stanford : Stanford University , 2015 .

Li Feifei , Fergus R , Perona P . Learning generative visual models from few training examples: An incremental Bayesian approach tested on 101 object categories [J ] . Computer Vision and Image Understanding , 2007 , 106 ( 1 ): 59 - 70 .

Bossard L , Guillaumin M , Van Gool L . Food-101-mining discriminative components with random forests [C ] // Proceedings of the European Conference on Computer Vision . Cham : Springer International Publishing , 2014 : 446 - 461 . DOI: 10.1007/978-3-319-10599-4_29 http://dx.doi.org/10.1007/978-3-319-10599-4_29

Krause J , Stark M , Jia Deng , et al . 3D object representations for fine-grained categorization [C ] // 2013 IEEE International Conference on Computer Vision Workshops . Piscataway : IEEE , 2013 : 554 - 561 . DOI: 10.1109/iccvw.2013.77 http://dx.doi.org/10.1109/iccvw.2013.77

Cimpoi M , Maji S , Kokkinos I , et al . Describing textures in the wild [C ] // 2014 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2014 : 3606 - 3613 . DOI: 10.1109/cvpr.2014.461 http://dx.doi.org/10.1109/cvpr.2014.461

Maji S , Rahtu E , Kannala J , et al . Fine-grained visual classification of aircraft [PP/OL ] . V1.arXiv ( 2013-06-21 )[ 2025-08-20 ] . https://doi.org/10.48550/arXiv.1306.5151 https://doi.org/10.48550/arXiv.1306.5151 .

Hendrycks D , Zhao K , Basart S , et al . Natural adversarial examples [C ] // 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2021 : 15257 - 15266 . DOI: 10.1109/cvpr46437.2021.01501 http://dx.doi.org/10.1109/cvpr46437.2021.01501

Hendrycks D , Basart S , Mu N , et al . The many faces of robustness: A critical analysis of out-of-distribution generalization [C ] // 2021 IEEE/CVF International Conference on Computer Vision . Piscataway : IEEE , 2021 : 8320 - 8329 . DOI: 10.1109/iccv48922.2021.00823 http://dx.doi.org/10.1109/iccv48922.2021.00823

Cao Kaidi , Brbic M , Leskovec J . Open-world semi-supervised learning [C ] // Proceedings of the 10th International Conference on Learning Representations . ICLR , 2022 : 1 - 19 .

Guo Lanzhe , Zhang Yige , Wu Zhifan , et al . Robust semi-supervised learning when not all classes have labels [C ] // Proceedings of the 36th International Conference on Neural Information Processing Systems . New York : ACM , 2022 : 3305 - 3317 . DOI: 10.52202/068431-0239 http://dx.doi.org/10.52202/068431-0239

Xia Shiyu , Lv Jiaqi , Xu Ning , et al . Towards effective visual representations for partial-label learning [C ] // 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2023 : 15589 - 15598 . DOI: 10.1109/cvpr52729.2023.01496 http://dx.doi.org/10.1109/cvpr52729.2023.01496

Wang Wei , Ishida T , Zhang Yujie , et al . Learning with complementary labels revisited: The selected-completely-at-random setting is more practical [C ] // Proceedings of the 41st International Conference on Machine Learning . PMLR , 2024 : 50683 - 50710 .

Gan Kai , Wei Tong . Erasing the bias: Fine-tuning foundation models for semi-supervised learning [C ] // Proceedings of the 41st International Conference on Machine Learning . PMLR , 2024 : 14453 - 14470 .

Zhang Renrui , Zhang Wei , Fang Rongyao , et al . Tip-adapter: Training-free adaption of CLIP for few-shot classification [C ] // Proceedings of the 17th European Conference on Computer Vision . Cham : Springer Nature Switzerland , 2022 : 493 - 510 . DOI: 10.1007/978-3-031-19833-5_29 http://dx.doi.org/10.1007/978-3-031-19833-5_29

Loshchilov I , Hutter F . Decoupled weight decay regularization [C ] // Proceedings of the 7th International Conference on Learning Representations . OpenReview . net , 2019 : 1 - 18 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

动态时序引导的病理图像分类扩散模型

基于多视角置信度融合的对抗样本迁移性提升方法

衰减解耦机制的图像分类网络

基于文本语义引导的红外与可见光图像融合方法

基于DD域LFM脉冲信号嵌入的OTFS通感一体化波形设计