基于自适应噪声和方面图关联学习增强多模态方面级情感分析

黄辰; 刘会杰; 张龑; 杨超; 宋建华

doi:10.12263/DZXB.20250533

您当前的位置：

首页 >

文章列表页 >

基于自适应噪声和方面图关联学习增强多模态方面级情感分析

学术论文 | 更新时间：2025-12-27

- 基于自适应噪声和方面图关联学习增强多模态方面级情感分析
- Enhancing Multimodal Aspect-Based Sentiment Analysis with Adaptive Noise and Aspect Graph Association Learning
- 电子学报 2025年53卷第9期页码：3397-3409
- 作者机构：
  
  1.湖北大学计算机学院，湖北武汉 430062
  2.智能感知系统与安全教育部重点实验室，湖北武汉 430062
  3.大数据智能分析与行业应用湖北省重点实验室，湖北武汉 430062
  4.湖北省高校人文社科重点研究基地-绩效评价信息管理研究中心，湖北武汉 430062
  5.湖北大学网络空间安全学院，湖北武汉 430062
- 作者简介：
  
  [ "黄辰男，1983年生，福建龙岩人.湖北大学计算机学院教授.主要研究方向为人工智能、脑机接口.E-mail: huang@hubu.edu.cn" ]
  [ "刘会杰男，2000年生，湖北黄石人.湖北大学计算机学院硕士研究生.主要研究方向为人工智能、脑科学、情感分析. E-mail: liuhj@stu.hubu.edu.cn" ]
  [ "张龑男，1974年生，湖北宜昌人.湖北大学计算机学院教授.主要研究领域为信息安全、大数据分析.中国电子学会会员编号：E190197582M. E-mail: zhangyan@hubu.edu.cn" ]
  [ "杨超男，1982年生，湖北武汉人.湖北大学计算机学院教授.主要研究方向为智能计算、信息安全等.E-mail: stevenyc@hubu.edu.cn" ]
  [ "宋建华女，1973年生，湖北襄阳人.湖北大学网络空间安全学院教授.主要研究方向为网络与信息安全.E-mail: sjhhubu@126.com" ]
- 基金信息：
  
  湖北省重大攻关项目（JD）(2023BAA018);湖北省科技计划重大科技专项(2024BAA008);武汉市知识创新专项项目(202311901251001)
- DOI：10.12263/DZXB.20250533
  中图分类号： TP391;TP399
- 收稿：2025-06-22，
  
  录用：2025-08-27，
  
  纸质出版：2025-09-25
- 稿件说明：
移动端阅览
黄辰, 刘会杰, 张龑, 等. 基于自适应噪声和方面图关联学习增强多模态方面级情感分析[J]. 电子学报, 2025, 53(09): 3397-3409.

HUANG Chen, LIU Hui-jie, ZHANG Yan, et al. Enhancing Multimodal Aspect-Based Sentiment Analysis with Adaptive Noise and Aspect Graph Association Learning[J]. Acta Electronica Sinica, 2025, 53(09): 3397-3409.
黄辰, 刘会杰, 张龑, 等. 基于自适应噪声和方面图关联学习增强多模态方面级情感分析[J]. 电子学报, 2025, 53(09): 3397-3409. DOI：10.12263/DZXB.20250533

HUANG Chen, LIU Hui-jie, ZHANG Yan, et al. Enhancing Multimodal Aspect-Based Sentiment Analysis with Adaptive Noise and Aspect Graph Association Learning[J]. Acta Electronica Sinica, 2025, 53(09): 3397-3409. DOI：10.12263/DZXB.20250533

摘要

多模态方面级情感分析（Multimodal Aspect-Based Sentiment Analysis，MABSA）旨在从多模态输入数据中准确识别方面术语并判定其情感极性.现有研究致力于融合多模态信息以提升情感分析性能.然而，在面临多方面和多情感场景时，它们仍然面临两个关键挑战：（1）缺乏对多模态输入数据中方面术语的全面感知；（2）存在情感语义偏差，现有模型倾向于关注与特定方面术语关联性强的情感语义，而忽略了关联性较低但同样重要的情感语义.为了克服这些问题，本文提出了一种结合自适应噪声和方面图关联学习的新型多模态方面级情感分析方法（Adaptive Noise and Aspect Graph Association Learning，ANAGAL），旨在增强多方面和多情感场景下的分析性能.具体而言，通过专门设计的自适应噪声增强模块以补充方面信息，从而增强模型对方面术语的感知能力，并提高模型鲁棒性.此外，设计方面图关联学习模块来关联所有方面术语，并学习与之相关的情感语义.同时，引入额外的参数进行情感校准，使模型能够学习更多常见的情感语义偏差，从而更准确地捕捉方面术语及其对应的情感极性.在公共数据集上的大量实验评估表明，ANAGAL在克服这些挑战方面表现优异.与现有基线模型相比，ANAGAL在Twitter-2015和Twitter-2017数据集上将精确率分别提升了1.46个百分点和1.56个百分点，在MASAD（Multimodal Aspect Sentiment Analysis Dataset）和EmoMeta数据集上将精确率提升了2.48个百分点和1.55个百分点.

Abstract

Multimodal aspect-based sentiment analysis (MABSA) aims to accurately identify aspect terms and determine their sentiment polarity from multimodal input data. Existing studies focus on integrating multimodal information to improve sentiment analysis performance. However

they still face two critical challenges in multi-aspect and multi-sentiment scenarios: (1) a lack of comprehensive perception of aspect terms in multimodal input data; and (2) sentiment semantic bias

where current models tend to focus on sentiment semantics strongly correlated with specific aspect terms

while ignoring weakly associated yet equally important sentiment cues. To address these issues

we propose a novel multimodal aspect-based sentiment analysis method

ANAGAL (Adaptive Noise and Aspect Graph Association Learning)

which integrates adaptive noise handling and aspect-graph association learning to enhance analytical performance in scenarios involving multiple aspects and multiple sentiments. Specifically

an adaptive noise enhancement module is designed to supplement aspect information

thereby improving the model’s aspect perception and robustness. In addition

an aspect graph correlation learning module is introduced to associate all aspect terms and learn related sentiment semantics. Extra parameters are further incorporated to calibrate sentiment representations

enabling the model to capture more generalized sentiment biases and better identify sentiment polarity associated with each aspect term. Extensive experimental evaluations on public datasets demonstrate that ANAGAL performs exceptionally well in addressing these challenges. Compared to existing state-of-the-art MABSA models

ANAGAL improves precision by 1.46 percentage points and 1.56 percentage points on the Twitter-2015 and Twitter-2017 datasets

and by 2.48 percentage points and 1.55 percentage points on the MASAD (Multimodal Aspect Sentiment Analysis Dataset) and EmoMeta datasets.

关键词

Keywords

references

WANG J Y , MOU L T , MA L , et al . AMSA: Adaptive multimodal learning for sentiment analysis [J ] . ACM Transactions on Multimedia Computing, Communications, and Applications , 2023 , 19 ( 3 s): 1 - 21 .

张换香 , 彭俊杰 . 基于方面级情感分析的深度语义挖掘模型 [J ] . 电子学报 , 2024 , 52 ( 7 ): 2307 - 2319 .

ZHANG H X , PENG J J . A deep semantic mining model based on aspect-level sentiment analysis [J ] . Acta Electronica Sinica , 2024 , 52 ( 7 ): 2307 - 2319 . (in Chinese)

YIN S , ZHONG G Q . TextGT: A double-view graph transformer on text for aspect-based sentiment analysis [J ] . Proceedings of the AAAI Conference on Artificial Intelligence , 2024 , 38 ( 17 ): 19404 - 19412 .

YANG X C , FENG S , WANG D L , et al . Few-shot joint multimodal aspect-sentiment analysis based on generative multimodal prompt [EB/OL ] . ( 2022-05-18 )[ 2025-03-24 ] . https://arXiv.org/abs/2305.10169 https://arXiv.org/abs/2305.10169 .

LING Y , YU J F , XIA R . Vision-language pre-training for multimodal aspect-based sentiment analysis [EB/OL ] . ( 2022-04-21 )[ 2025-03-24 ] . https://arXiv.org/abs/2204.07955 https://arXiv.org/abs/2204.07955 .

HAN Z X , HU M T , BAI Y H , et al . DEQA: Descriptions enhanced question-answering framework for multimodal aspect-based sentiment analysis [J ] . Proceedings of the AAAI Conference on Artificial Intelligence , 2025 , 39 ( 22 ): 23987 - 23995 .

WANG D , HE Y , LIANG X , et al . TMFN: A target-oriented multi-grained fusion network for end-to-end aspect-based multimodal sentiment analysis [C ] // Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) . Paris : ELRA , 2024 : 16187 - 16197 .

XIAO L W , MAO R , ZHAO S , et al . Exploring cognitive and aesthetic causality for multimodal aspect-based sentiment analysis [J ] . IEEE Transactions on Affective Computing , 2025 . DOI: 10.1109/TAFFC.2025.3565506 http://dx.doi.org/10.1109/TAFFC.2025.3565506 .

JU X C , ZHANG D , XIAO R , et al . Joint multi-modal aspect-sentiment analysis with auxiliary cross-modal relation detection [C ] // Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing . Stroudsburg : ACL , 2021 : 4395 - 4405 .

YANG L , NA J C , YU J F . Cross-modal multitask transformer for end-to-end multimodal aspect-based sentiment analysis [J ] . Information Processing & Management , 2022 , 59 ( 5 ): 103038 .

ZHOU R , GUO W Y , LIU X M , et al . AoM: Detecting aspect-oriented information for multimodal aspect-based sentiment analysis [EB/OL ] . ( 2023-05-31 )[ 2025-03-24 ] . https://arXiv.org/abs/2306.01004 https://arXiv.org/abs/2306.01004 .

ZHU L L , SUN H L , GAO Q S , et al . Aspect enhancement and text simplification in multimodal aspect-based sentiment analysis for multi-aspect and multi-sentiment scenarios [J ] . Proceedings of the AAAI Conference on Artificial Intelligence , 2025 , 39 ( 2 ): 1683 - 1691 .

LU X Y , LIU Y X , ZHANG D Y , et al . EmoMeta: A multimodal dataset for fine-grained emotion classification in Chinese metaphors [C ] // Companion Proceedings of the ACM on Web Conference 2025 . New York : ACM , 2025 : 3080 - 3083 .

ZHAO T Y , MENG L G , SONG D W . Multimodal aspect-based sentiment analysis: A survey of tasks, methods, challenges and future directions [J ] . Information Fusion , 2024 , 112 : 102552 .

ZHENG C M , FENG J H , CAI Y , et al . Rethinking multimodal entity and relation extraction from a translation point of view [C ] // Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics . Stroudsburg : ACL , 2023 : 6810 - 6824 .

王友卫 , 刘瑞 , 凤丽洲 . 基于用户性格和语义-结构特征的文本评论情感分类方法 [J ] . 电子学报 , 2024 , 52 ( 5 ): 1657 - 1669 .

WANG Y W , LIU R , FENG L Z . A sentiment classification method for text comments based on user personality and semantic-structural features [J ] . Acta Electronica Sinica , 2024 , 52 ( 5 ): 1657 - 1669 . (in Chinese)

YU J F , CHEN K , XIA R . Hierarchical interactive multimodal transformer for aspect-based multimodal sentiment analysis [J ] . IEEE Transactions on Affective Computing , 2023 , 14 ( 3 ): 1966 - 1978 .

WENG Y , CHEN L , WANG S , et al . MIECF: Multi-faceted information extraction and cross-mixture fusion for multimodal aspect-based sentiment analysis [J ] . Heliyon , 2024 , 10 ( 12 ): e32967 .

GUO A B , ZHAO X , TAN Z , et al . MGICL: Multi-grained interaction contrastive learning for multimodal named entity recognition [C ] // Proceedings of the 32nd ACM International Conference on Information and Knowledge Management . New York : ACM , 2023 : 639 - 648 .

TRUONG Q T , LAUW H W . VistaNet: Visual aspect attention network for multimodal sentiment analysis [J ] . Proceedings of the AAAI Conference on Artificial Intelligence , 2019 , 33 ( 1 ): 305 - 312 .

YE J J , ZHOU J , TIAN J F , et al . Sentiment-aware multimodal pre-training for multimodal sentiment analysis [J ] . Knowledge-Based Systems , 2022 , 258 : 110021 .

YU J F , JIANG J . Adapting BERT for target-oriented multimodal sentiment classification [C ] // International Joint Conference on Artificial Intelligence (IJCAI 2020) . California : IJCAI , 2020 : 5407 - 5414 .

FAN R , HE T T , CHEN M H , et al . Dual causes generation assisted model for multimodal aspect-based sentiment classification [J ] . IEEE Transactions on Neural Networks and Learning Systems , 2025 , 36 ( 5 ): 9298 - 9312 .

KHAN Z , FU Y . Exploiting BERT for multimodal target sentiment classification through input space translation [C ] // Proceedings of the 29th ACM International Conference on Multimedia . New York : ACM , 2021 : 3034 - 3042 .

JIA L , MA T H , RONG H , et al . Affective region recognition and fusion network for target-level multimodal sentiment classification [J ] . IEEE Transactions on Emerging Topics in Computing , 2024 , 12 ( 3 ): 688 - 699 .

YU J F , WANG J M , XIA R , et al . Targeted multimodal sentiment classification based on coarse-to-fine grained image-target matching [C ] // Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence . New York : IJCAI Press , 2022 : 4482 - 4488 .

YANG J , XIAO Y L , DU X . Multi-grained fusion network with self-distillation for aspect-based multimodal sentiment analysis [J ] . Knowledge-Based Systems , 2024 , 293 : 111724 .

WU C , XIONG Q Y , YI H L , et al . Multiple-element joint detection for aspect-based sentiment analysis [J ] . Knowledge-Based Systems , 2021 , 223 : 107073 .

LIU Y H , OTT M , GOYAL N , et al . RoBERTa: A robustly optimized BERT pretraining approach [EB/OL ] . ( 2019-07-26 )[ 2025-03-24 ] . https://arXiv.org/abs/1907.11692 https://arXiv.org/abs/1907.11692 .

DOSOVITSKIY A , BEYER L , KOLESNIKOV A , et al . An image is worth 16 x 16 words: Transformers for image recognition at scale[EB/OL ] . ( 2021-06-03 )[ 2025-03-24 ] . https://arXiv.org/abs/2010.11929 https://arXiv.org/abs/2010.11929 .

XUE X J , ZHANG C X , NIU Z D , et al . Multi-level attention map network for multimodal sentiment analysis [J ] . IEEE Transactions on Knowledge and Data Engineering , 2023 , 35 ( 5 ): 5105 - 5118 .

WANG S J , CAI G Y , LV G R . Aspect-level multimodal sentiment analysis based on co-attention fusion [J ] . International Journal of Data Science and Analytics , 2025 , 20 ( 2 ): 903 - 916 .

ZHOU J , ZHAO J B , HUANG J X , et al . MASAD: A large-scale dataset for multimodal aspect-based sentiment analysis [J ] . Neurocomputing , 2021 , 455 : 47 - 58 .

HU M H , PENG Y X , HUANG Z , et al . Open-domain targeted sentiment analysis via span-based extraction and classification [EB/OL ] . ( 2019-06-10 )[ 2025-03-24 ] . https://doi.org/10.48550/arXiv.1906.03820 https://doi.org/10.48550/arXiv.1906.03820 .

CHEN G M , TIAN Y H , SONG Y . Joint aspect extraction and sentiment analysis with directional graph convolutional networks [C ] // Proceedings of the 28th International Conference on Computational Linguistics . New York : IJCAI Press , 2020 : 272 - 279 .

YAN H , DAI J Q , JI T , et al . A unified generative framework for aspect-based sentiment analysis [EB/OL ] . ( 2021-06-08 )[ 2025-03-24 ] . https://arXiv.org/abs/2106.04300 https://arXiv.org/abs/2106.04300 .

WU Z W , ZHENG C M , CAI Y , et al . Multimodal representation with embedded visual guiding objects for named entity recognition in social media posts [C ] // Proceedings of the 28th ACM International Conference on Multimedia . New York : ACM , 2020 : 1038 - 1046 .

SUN L , WANG J Q , ZHANG K , et al . RpBERT: A text-image relation propagation-based BERT model for multimodal NER [J ] . Proceedings of the AAAI Conference on Artificial Intelligence , 2021 , 35 ( 15 ): 13860 - 13868 .

YU J F , JIANG J , YANG L , et al . Improving multimodal named entity recognition via entity span detection with unified multimodal transformer [C ] // Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Stroudsburg : ACL , 2020 : 3342 - 3352 .

XIAO L W , WU X J , XU J J , et al . Atlantis: Aesthetic-oriented multiple granularities fusion network for joint multimodal aspect-based sentiment analysis [J ] . Information Fusion , 2024 , 106 : 102304 .

PENG T S , LI Z C , WANG P , et al . A novel energy based model mechanism for multi-modal aspect-based sentiment analysis [J ] . Proceedings of the AAAI Conference on Artificial Intelligence , 2024 , 38 ( 17 ): 18869 - 18878 .

WU H Q , CHENG S L , WANG J J , et al . Multimodal aspect extraction with region-aware alignment network [M ] // Natural Language Processing and Chinese Computing . Cham : Springer International Publishing , 2020 : 145 - 156 .

YU J F , JIANG J , XIA R . Entity-sensitive attention and fusion network for entity-level multimodal sentiment classification [J ] . IEEE/ACM Transactions on Audio, Speech, and Language Processing , 2020 , 28 : 429 - 439 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

SCG-Detector：基于图注意力网络的智能合约漏洞检测方法

GAT-IL：一种基于图注意力网络与模仿学习的服务功能链部署方法

基于图注意力和改进Transformer的节点分类方法

基于异质图注意力网络的miRNA与疾病关联预测算法