基于图像内容理解的判别性类别提示学习

王楠井; 刘阿建; 梁凤梅; 张小梅; 万军; 谢珺; 雷震

doi:10.12263/DZXB.20240522

您当前的位置：

首页 >

文章列表页 >

基于图像内容理解的判别性类别提示学习

学术论文 | 更新时间：2025-12-08

- 基于图像内容理解的判别性类别提示学习
- Discriminative Category Prompt Learning Based on Image Content Understanding
- 电子学报 2025年53卷第2期页码：493-502
- 作者机构：
  
  1.太原理工大学电子信息工程学院，山西晋中 030600
  2.中国科学院自动化研究所，北京 100190
- 作者简介：
  
  [ "王楠井男，2000年6月出生于山西省吕梁市.现为太原理工大学电子信息工程学院硕士研究生.主要研究方向为计算机视觉、图像识别与处理.E-mail: 2632835656@qq.com" ]
  [ "刘阿建男，1992年6月出生于山西省运城市.中国科学院自动化研究所助理研究员，“澳门青年学者计划”“基金委青年科学基金”“北京市青年人才托举工程”“博士后面上资助项目”获得者.主要研究方向为计算机视觉、图像识别与处理.中国电子学会会员编号：E190159314M.E-mail: ajianliu92@gmail.com" ]
  [ "梁凤梅女，1969年5月出生于山西省太原市.现为太原理工大学副教授、硕士生导师.主要研究方向为计算机视觉、智能信息处理.获得省科技进步二等奖1项、省科技进步三等奖2项.E-mail: fm_liang@163.com" ]
  [ "张小梅女，1995年2月出生于山东省菏泽市.中国科学院自动化研究所多模态人工智能系统全国重点实验室副研究员.主要研究方向为模式识别和计算机视觉.E-mail: xiaomei.zhang@nlpria.ac.cn" ]
  [ "万军男，1984年10月出生于江西省高安市.中国科学院自动化研究所多模态人工智能系统全国重点实验室研究员.主要研究方向为智能视频分析与交互技术.中国电子学会会员编号：E190014368M.E-mail: joewan10@gmail.com" ]
  [ "谢珺女，1979年6月出生于山西省太原市.现为太原理工大学副教授、硕士生导师.主要研究方向为智能信号处理.中国电子学会会员编号：E190091975M.E-mail: xiejun@tyut.edu.cn" ]
  [ "雷震男，1983年6月出生于浙江省嵊州市.IEEE Fellow，中国科学院自动化研究所多模态人工智能系统全国重点实验室研究员，中国科学院大学岗位教授，中国科学院香港创新院人工智能与机器人创新中心教授，博士生导师.主要研究方向为计算机视觉、模式识别、人脸识别、目标检测与识别、智能视频分析、三维人脸/人体重建等.E-mail: zhen.lei@ia.ac.cn" ]
- 基金信息：
  
  虚拟现实技术与系统全国重点实验室开放课题(VRLAB2023A06);山西省科技合作交流专项(202104041101030)
- DOI：10.12263/DZXB.20240522
  中图分类号： TP391.4;
- 收稿：2024-06-05，
  
  修回：2024-10-30，
  
  纸质出版：2025-02-25
- 稿件说明：
移动端阅览
王楠井, 刘阿建, 梁凤梅, 等. 基于图像内容理解的判别性类别提示学习[J]. 电子学报, 2025, 53(02): 493-502.

WANG Nan-jing, LIU A-jian, LIANG Feng-mei, et al. Discriminative Category Prompt Learning Based on Image Content Understanding[J]. Acta Electronica Sinica, 2025, 53(02): 493-502.
王楠井, 刘阿建, 梁凤梅, 等. 基于图像内容理解的判别性类别提示学习[J]. 电子学报, 2025, 53(02): 493-502. DOI：10.12263/DZXB.20240522

WANG Nan-jing, LIU A-jian, LIANG Feng-mei, et al. Discriminative Category Prompt Learning Based on Image Content Understanding[J]. Acta Electronica Sinica, 2025, 53(02): 493-502. DOI：10.12263/DZXB.20240522

摘要

近年来，通过图像与文本的联合表示，基于对比语言-图像预训练（Contrastive Language-Image Pre-training，CLIP）的方法将文本信息作为分类器的权值，在通用图像识别任务中展现出卓越性能.但是现有方法仅单独构建类别文本提示，比如上下文优化（Context Optimization，CoOp）和条件上下文优化（Conditional Context Optimization，CoCoOp）等，没有考虑图像的内容语义信息与类别的重要性，限制了模型对图像类别的理解与判别.为了解决上述问题，本文在CLIP的基础上提出了一种新方法：基于图像内容理解的判别性类别提示学习（Discriminative Category Prompt Learning based on image content understanding，DCPL），借助图像中丰富的内容特征来学习文本提示，提高文本提示对类别的判别性.具体来说，DCPL包含提示生成（Prompt Generation，PG）模块和文本监督（Text Supervision，TS）模块. PG模块将图像特征和初始化的查询向量作为输入，通过自注意力机制和交叉注意力机制使输出的文本提示中包含充分的图像语义信息；TS模块将固定的类别提示模板作为监督，为可学习文本提示在类别层面和logits层面注入类别信息，增强了类别的重要性.最后，DCPL在ImageNet、Caltech101和Oxford-Pets等11个公开分类数据集上的16-shots平均准确率达到了81.84%，较以往最优方法Cross-Modal的平均准确率提升了0.98个百分点.

Abstract

In recent years

the contrastive language-image pre-training (CLIP)-based method takes the text information as the weight of the classifier through the joint representation of image and text and shows excellent performance in the general image recognition task. However

the existing methods only construct text prompts of categories

such as context optimization (CoOp) and conditional context optimization (CoCoOp)

without considering the importance of image content semantic information and categories

which limits the model’s understanding and discrimination of image categories. To solve the above problems

this article proposes a new method based on CLIP: discriminative category prompt learning based on image content understanding (DCPL)

which uses rich content features in images to learn text prompts and introduces manual templates to improve the discrimination of text prompts on categories. Specifically

DCPL includes a prompt generation module and a text supervision module: The prompt generation module takes image features and initialized query vectors as inputs

and makes the output text prompt contain sufficient image semantic information through the self/cross-attention mechanism; The text supervision module uses the fixed category prompt template as the supervision to inject category information into the category level and logits level for the learnable text prompt

increasing the importance of categories. Finally

the average accuracy of 16 shots of DCPL on 11 public classified datasets

such as ImageNet

Caltech101

Oxford pets

etc.

is 81.84%

The average accuracy has increased by 0.98 percentage points compared with that of the previous optimal method

Cross-Modal.

关键词

Keywords

references

KRIZHEVSKY A , SUTSKEVER I , HINTON G E . Imagenet classification with deep convolutional neural networks [J ] . Communications of the ACM , 2017 , 60 ( 6 ): 84 - 90 .

SIMONYAN K , ZISSERMAN A . Very deep convolutional networks for large-scale image recognition [EB/OL ] . ( 2014-09-04 )[ 2024-06-05 ] . https://arxiv.org/abs/1409.1556v6 https://arxiv.org/abs/1409.1556v6 .

SZEGEDY C , LIU W , JIA Y Q , et al . Going deeper with convolutions [C ] // 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2015 : 1 - 9 .

HE K M , ZHANG X Y , REN S Q , et al . Deep residual learning for image recognition [C ] // 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2016 : 770 - 778 .

殷炯 , 张哲东 , 高宇涵 , 等 . 视觉语言预训练综述 [J ] . 软件学报 , 2023 , 34 ( 5 ): 2000 - 2023 .

YIN J , ZHANG Z D , GAO Y H , et al . Survey on vision-language pre-training [J ] . Journal of Software , 2023 , 34 ( 5 ): 2000 - 2023 . (in Chinese)

RADFORD A , KIM J W , HALLACY C , et al . Learning transferable visual models from natural language supervision [C ] // International Conference on Machine Learning . New York : PMLR , 2021 , 139 : 8748 - 8763 .

GU X Y , LIN T Y , KUO W C , et al . Open-vocabulary object detection via vision and language knowledge distillation [EB/OL ] . ( 2021-04-28 )[ 2024-06-05 ] . https://arxiv.org/abs/2104.13921v3 https://arxiv.org/abs/2104.13921v3 .

SAHARIA C , CHAN W , SAXENA S , et al . Photorealistic text-to-image diffusion models with deep language understanding [C ] // Proceedings of the 36th International Conference on Neural Information Processing Systems . New York : Curran Associates Inc. , 2022 , 35 : 36479 - 36494 .

ALAYRAC J B , DONAHUE J , LUC P , et al . Flamingo: A visual language model for few-shot learning [J ] . Advances in Neural Information Processing Systems , 2022 , 35 : 23716 - 23736 .

ZHOU K Y , YANG J K , LOY C C , et al . Learning to prompt for vision-language models [J ] . International Journal of Computer Vision , 2022 , 130 ( 9 ): 2337 - 2348 .

ZHOU K Y , YANG J K , LOY C C , et al . Conditional prompt learning for vision-language models [C ] // 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2022 : 16795 - 16804 .

KHATTAK M U , RASHEED H , MAAZ M , et al . MaPLe: Multi-modal prompt learning [C ] // 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2023 : 19113 - 19122 .

VASWANI A , SHAZEER N , PARMAR N , et al . Attention is all you need [C ] // Proceedings of the 31st International Conference on Neural Information Processing Systems , New York : Curran Associates Inc. , 2017 : 6000 - 6010 .

CHEN C F R , FAN Q F , PANDA R . CrossViT: Cross-attention multi-scale vision transformer for image classification [C ] // 2021 IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway : IEEE , 2021 : 347 - 356 .

LI J N , LI D X , SAVARESE S , et al . BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models [C ] // International Conference on Machine Learning . New York : PMLR , 2023 : 19730 - 19742 .

RADFORD A , WU J , CHILD R , et al . Language models are unsupervised multitask learners [J ] . OpenAI blog , 2019 , 1 ( 8 ): 9 .

廖宁 , 曹敏 , 严骏驰 . 视觉提示学习综述 [J ] . 计算机学报 , 2024 , 47 ( 4 ): 790 - 820 .

LIAO N , CAO M , YAN J C . Visual prompt learning: A survey [J ] . Chinese Journal of Computers , 2024 , 47 ( 4 ): 790 - 820 . (in Chinese)

YAO H T , ZHANG R , XU C S . Visual-language prompt tuning with knowledge-guided context optimization [C ] // 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2023 : 6757 - 6767 .

GAO T Y , FISCH A , CHEN D Q . Making pre-trained language models better few-shot learners [EB/OL ] . ( 2020-12-31 )[ 2024-06-05 ] . https://arxiv.org/abs/2012.15723v2 https://arxiv.org/abs/2012.15723v2 .

CUI G Q , HU S D , DING N , et al . Prototypical verbalizer for prompt-based few-shot tuning [EB/OL ] . ( 2022-3-18 )[ 2024-06-05 ] . https://arxiv.org/abs/2203.09770v1 https://arxiv.org/abs/2203.09770v1 .

DENG J , DONG W , SOCHER R , et al . ImageNet: A large-scale hierarchical image database [C ] // 2009 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2009 : 248 - 255 .

LI F F , FERGUS R , PERONA P . Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories [C ] // 2004 Conference on Computer Vision and Pattern Recognition Workshop . Piscataway : IEEE , 2005 : 178 .

XIAO J X , HAYS J , EHINGER K A , et al . SUN database: Large-scale scene recognition from abbey to zoo [C ] // 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2010 : 3485 - 3492 .

HELBER P , BISCHKE B , DENGEL A , et al . EuroSAT: A novel dataset and deep learning benchmark for land use and land cover classification [J ] . IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 2019 , 12 ( 7 ): 2217 - 2226 .

SOOMRO K , ZAMIR A R , SHAH M . UCF101: A dataset of 101 human actions classes from videos in the wild [EB/OL ] . ( 2012-12-03 )[ 2024-06-05 ] . https://arxiv.org/abs/1212.0402v1 https://arxiv.org/abs/1212.0402v1 .

CIMPOI M , MAJI S , KOKKINOS I , et al . Describing textures in the wild [C ] // 2014 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2014 : 3606 - 3613 .

BOSSARD L , GUILLAUMIN M , VAN GOOL L . Food-101 - mining discriminative components with random forests [M ] // Lecture Notes in Computer Science . Cham : Springer International Publishing , 2014 : 446 - 461 .

KRAUSE J , STARK M , JIA D , et al . 3D object representations for fine-grained categorization [C ] // 2013 IEEE International Conference on Computer Vision Workshops . Piscataway : IEEE , 2013 : 554 - 561 .

MAJI S , RAHTU E , KANNALA J , et al . Fine-grained visual classification of aircraft [EB/OL ] . ( 2013-06-21 )[ 2024-06-05 ] . https://arxiv.org/abs/1306.5151v1 https://arxiv.org/abs/1306.5151v1 .

PARKHI O M , VEDALDI A , ZISSERMAN A , et al . Cats and dogs [C ] // 2012 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2012 : 3498 - 3505 .

NILSBACK M E , ZISSERMAN A . Automated flower classification over a large number of classes [C ] // 2008 Sixth Indian Conference on Computer Vision , Graphics & Image Processing . Piscataway : IEEE , 2008 : 722 - 729 .

LOSHCHILOV I , HUTTER F . Decoupled weight decay regularization [EB/OL ] . ( 2017-11-14 )[ 2024-06-05 ] . https://arxiv.org/abs/1711.05101v3 https://arxiv.org/abs/1711.05101v3 .

ZHANG R R , FANG R Y , ZHANG W , et al . Tip-adapter: Training-free CLIP-adapter for better vision-language modeling [EB/OL ] . ( 2021-11-06 )[ 2024-06-05 ] . https://arxiv.org/abs/2111.03930v2 https://arxiv.org/abs/2111.03930v2 .

JIE S B , DENG Z H . Convolutional bypasses are better vision transformer adapters [EB/OL ] . ( 2022-07-14 )[ 2024-06-05 ] . https://arxiv.org/abs/2207.07039v3 https://arxiv.org/abs/2207.07039v3 .

LIN Z Q , YU S , KUANG Z Y , et al . Multimodality helps unimodality: Cross-modal few-shot learning with multimodal models [C ] // 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2023 : 19325 - 19337 .

RECHT B , ROELOFS R , SCHMIDT L , et al . Do imagenet classifiers generalize to imagenet? [C ] // International Conference on Machine Learning . New York : PMLR , 2019 : 5389 - 5400 .

WANG H H , GE S W , LIPTON Z , et al . Learning robust global representations by penalizing local predictive power [C ] // 33rd Conference on Neural Information Processing Systems . Vancouver : NeurIPS , 2019 : 10506 - 10518 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于改进Res2Net与自适应多尺度窗口池化的调制识别方法

面向在线连续学习的特征融合引导的梯度重加权算法

基于序列与跨模态对齐的蛋白质功能预测模型

NAHGNN：邻域感知异构图神经网络