Fine-Grained Facial Expression Recognition Algorithm Based on Relationship-Awareness and Label Disambiguation

LIU Ya-zhi; XU Zhe-ming; LANG Cong-yan; WANG Tao; LI Yi-dong

doi:10.12263/DZXB.20240364

您当前的位置：

首页 >

文章列表页 >

Fine-Grained Facial Expression Recognition Algorithm Based on Relationship-Awareness and Label Disambiguation

PAPERS | 更新时间：2026-05-07

- Fine-Grained Facial Expression Recognition Algorithm Based on Relationship-Awareness and Label Disambiguation
- ACTA ELECTRONICA SINICA Vol. 52, Issue 10, Pages: 3336-3346(2024)
- 作者机构：
  
  1.北京交通大学计算机科学与技术学院，北京 100044
  2.北京交通大学交通大数据与人工智能教育部重点实验室，北京 100044
- 作者简介：
- 基金信息：
  
  National Natural Science Foundation of China(62072027;62376020)
- DOI：10.12263/DZXB.20240364
  CLC： TP391
- Received：22 April 2024，
  
  Revised：2024-08-11，
  
  Published：25 October 2024
- 稿件说明：
移动端阅览
刘雅芝, 许喆铭, 郎丛妍, 等. 基于关系感知和标签消歧的细粒度面部表情识别算法[J]. 电子学报, 2024, 52(10): 3336-3346.

LIU Ya-zhi, XU Zhe-ming, LANG Cong-yan, et al. Fine-Grained Facial Expression Recognition Algorithm Based on Relationship-Awareness and Label Disambiguation[J]. Acta Electronica Sinica, 2024, 52(10): 3336-3346.
刘雅芝, 许喆铭, 郎丛妍, 等. 基于关系感知和标签消歧的细粒度面部表情识别算法[J]. 电子学报, 2024, 52(10): 3336-3346. DOI：10.12263/DZXB.20240364

LIU Ya-zhi, XU Zhe-ming, LANG Cong-yan, et al. Fine-Grained Facial Expression Recognition Algorithm Based on Relationship-Awareness and Label Disambiguation[J]. Acta Electronica Sinica, 2024, 52(10): 3336-3346. DOI：10.12263/DZXB.20240364

摘要

细粒度表情识别任务因其包含更丰富真实的人类情感而备受关注.现有面部表情识别算法通过提取局部关键区域等方式学习更优的图像表征.然而，这些方法忽略了图像数据集内在的结构关系，且没有充分利用标签间的语义关联度以及图像和标签间的相关性，导致所学特征带来的性能提升有限.其次，现有细粒度表情识别方法并未有效利用和挖掘粗细粒度的层级关系，因而限制了模型的识别性能.此外，现有细粒度表情识别算法忽略了由于标注主观性和情感复杂性导致的标签歧义性问题，极大影响了模型的识别性能.针对上述问题，本文提出一种基于关系感知和标签消歧的细粒度面部表情识别算法（fine-grained facial expression recognition algorithm based on Relationship-Awareness and Label Disambiguation，RALD）.该算法通过构建层级感知的图像特征增强网络，充分挖掘图像之间、层级标签之间以及图像和标签之间的依赖关系，以获得更具辨别性的图像特征.针对标签歧义性问题，算法设计了基于近邻样本的标签分布学习模块，通过整合邻域信息进行标签消歧，进一步提升模型识别性能.在细粒度表情识别数据集FG-Emotions上算法的准确度达到97.34%，在粗粒度表情识别数据集RAF-DB上比现有主流表情分类方法提高了0.80%～4.55%.

Abstract

There has been a growing interest in fine-grained facial expression recognition due to its ability to capture more subtle and realistic human emotions. Existing facial expression recognition algorithms enhance image representations by extracting local key regions and other relevant features. However

these methods disregard the inherent structural relationships within the image dataset and fail to fully exploit the semantic correlation between labels and the relationship between images and labels

which restricts the enhancement of feature learning. Besides

current fine-grained expression recognition methods do not effectively explore and utilize the hierarchical relationship between coarse and fine-grained levels

which limits the recognition performance of the model. In addition

existing fine-grained expression recognition algorithms ignore the label ambiguity problem caused by labeling subjectivity and emotional complexity

which greatly affects the recognition performance of the model. To address these issues

we propose a fine-grained facial expression recognition algorithm based on relationship-awareness and label disambiguation (RALD). This algorithm enhances image features by constructing a hierarchy-aware image feature enhancement network

thoroughly exploring the dependencies among images

hierarchical labels

and between images and labels to obtain more discriminative image features. As for the issue of label ambiguity

this algorithm designs a nearest neighbors-based label distribution learning module

which further improves recognition performance by integrating neighborhood information for label disambiguation. Our algorithm achieves 97.34% in terms of accuracy on the FG-Emotions dataset for fine-grained expression recognition. Additionally

it outperforms existing mainstream facial expression recognition algorithms by 0.80% to 4.55% on the RAF-DB dataset for coarse-grained expression recognition.

关键词

Keywords

references

KARPATHY A , LI F F . Deep visual-semantic alignments for generating image descriptions [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2017 , 39 ( 4 ): 664 - 676 .

ZHANG H , KOH J Y , BALDRIDGE J , et al . Cross-modal contrastive learning for text-to-image generation [C ] // 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2021 : 833 - 842 .

LIU T T , WANG J X , YANG B , et al . Facial expression recognition method with multi-label distribution learning for non-verbal behavior understanding in the classroom [J ] . Infrared Physics and Technology , 2021 , 112 : 103594 .

AGRAWAL A , LU J S , ANTOL S , et al . VQA: Visual question answering [J ] . International Journal of Computer Vision , 2017 , 123 ( 1 ): 4 - 31 .

EKMAN P , FRIESEN W V . Constants across cultures in the face and emotion [J ] . Journal of Personality and Social Psychology , 1971 , 17 ( 2 ): 124 - 129 .

LEE J , KIM S , KIM S , et al . Context-aware emotion recognition networks [C ] // 2019 IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway : IEEE , 2019 : 10143 - 10152 .

姚乃明 , 郭清沛 , 乔逢春 , 等 . 基于生成式对抗网络的鲁棒人脸表情识别 [J ] . 自动化学报 , 2018 , 44 ( 5 ): 865 - 877 .

YAO N M , GUO Q P , QIAO F C , et al . Robust facial expression recognition with generative adversarial networks [J ] . Acta Automatica Sinica , 2018 , 44 ( 5 ): 865 - 877 . (in Chinese)

LIU Z Y , YUAN X Y , LI Y T , et al . PRA-Net: Part-and-relation attention network for depression recognition from facial expression [J ] . Computers in Biology and Medicine , 2023 , 157 : 106589 .

孙晓 , 潘汀 . 基于兴趣区域深度神经网络的静态面部表情识别 [J ] . 电子学报 , 2017 , 45 ( 5 ): 1189 - 1197 .

SUN X , PAN T . Static facial expression recognition system using ROI deep neural networks [J ] . Acta Electronica Sinica , 2017 , 45 ( 5 ): 1189 - 1197 . (in Chinese)

张瑞 , 蒋晨之 , 苏剑波 . 基于稀疏特征挑选和概率线性判别分析的表情识别研究 [J ] . 电子学报 , 2018 , 46 ( 7 ): 1710 - 1718 .

ZHANG R , JIANG C Z , SU J B . Expression recognition based on sparse selection and PLDA [J ] . Acta Electronica Sinica , 2018 , 46 ( 7 ): 1710 - 1718 . (in Chinese)

ZHANG W , JI X P , CHEN K Y , et al . Learning a facial expression embedding disentangled from identity [C ] // 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2021 : 6759 - 6768 .

章思远 , 肖世明 , 张蓬 , 等 . 图像生成和深度度量学习的身份感知面部表情识别方法 [J ] . 计算机辅助设计与图形学学报 , 2021 , 33 ( 5 ): 724 - 732 .

ZHANG S Y , XIAO S M , ZHANG P , et al . Identity-aware facial expression recognition method based on synthesized images and deep metric learning [J ] . Journal of Computer-Aided Design & Computer Graphics , 2021 , 33 ( 5 ): 724 - 732 . (in Chinese)

ZENG J B , SHAN S G , CHEN X L . Facial expression recognition with inconsistently annotated datasets [M ] // Lecture Notes in Computer Science . Cham : Springer International Publishing , 2018 : 227 - 243 .

ZHU Y C , WEI L L , LANG C Y , et al . Fine-grained facial expression recognition via relational reasoning and hierarchical relation optimization [J ] . Pattern Recognition Letters , 2022 , 164 : 67 - 73 .

CAI J , MENG Z B , KHAN A S , et al . Probabilistic attribute tree structured convolutional neural networks for facial expression recognition in the wild [J ] . IEEE Transactions on Affective Computing , 2023 , 14 ( 3 ): 1927 - 1941 .

ZHENG C , MENDIETA M , CHEN C . POSTER: A pyramid cross-fusion Transformer network for facial expression recognition [C ] // 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) . Piscataway : IEEE , 2023 : 3146 - 3155 .

QI Y F , ZHOU C Y , CHEN Y X . NA-Resnet: Neighbor block and optimized attention module for global-local feature extraction in facial expression recognition [J ] . Multimedia Tools and Applications , 2023 , 82 ( 11 ): 16375 - 16393 .

CHEN D L , WEN G H , LI H H , et al . Multi-relations aware network for in-the-wild facial expression recognition [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2023 , 33 ( 8 ): 3848 - 3859 .

ZHANG Y H , WANG C R , LING X , et al . Learn from all: erasing attention consistency for noisy label facial expression recognition [M ] // Lecture Notes in Computer Science . Cham : Springer Nature Switzerland , 2022 : 418 - 434 .

XUE F L , WANG Q C , GUO G D . TransFER: Learning relation-aware facial expression representations with Transformers [C ] // 2021 IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway : IEEE , 2021 : 3601 - 3610 .

胡敏 , 江河 , 王晓华 , 等 . 基于几何和纹理特征的表情层级分类方法 [J ] . 电子学报 , 2017 , 45 ( 1 ): 164 - 172 .

HU M , JIANG H , WANG X H , et al . A hierarchical classification method of expressions based on geometric and texture features [J ] . Acta Electronica Sinica , 2017 , 45 ( 1 ): 164 - 172 . (in Chinese)

廖海斌 , 徐斌 . 基于性别和年龄因子分析的鲁棒性人脸表情识别 [J ] . 计算机研究与发展 , 2021 , 58 ( 3 ): 528 - 538 .

LIAO H B , XU B . Robust face expression recognition based on gender and age factor analysis [J ] . Journal of Computer Research and Development , 2021 , 58 ( 3 ): 528 - 538 . (in Chinese)

LIU X F , KUMAR B V K V , YOU J , et al . Adaptive deep metric learning for identity-aware facial expression recognition [C ] // 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) . Piscataway : IEEE , 2017 : 20 - 29 .

FARZANEH A H , QI X J . Facial expression recognition in the wild via deep attentive center loss [C ] // 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) . Piscataway : IEEE , 2021 : 2402 - 2411 .

LIANG L Q , LANG C Y , LI Y D , et al . Fine-grained facial expression recognition in the wild [J ] . IEEE Transactions on Information Forensics and Security , 2021 , 16 : 482 - 494 .

DOSOVITSKIY A , BEYER L , KOLESNIKOV A , et al . An image is worth 16 x 16 words: Transformers for image recognition at scale[EB/OL ] ( 2021-06-03 )[ 2024-04-22 ] . https://arxiv.org/abs/2010.11929 https://arxiv.org/abs/2010.11929 .

CARION N , MASSA F , SYNNAEVE G , et al . End-to-end object detection with transformers [M ] // Lecture Notes in Computer Science . Cham : Springer International Publishing , 2020 : 213 - 229 .

ZHENG S X , LU J C , ZHAO H S , et al . Rethinking semantic segmentation from a sequence-to-sequence perspective with Transformers [C ] // 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2021 : 6881 - 6890 .

XUE F L , WANG Q C , TAN Z C , et al . Vision Transformer with attentive pooling for robust facial expression recognition [J ] . IEEE Transactions on Affective Computing , 2023 , 14 ( 4 ): 3244 - 3256 .

GENG X . Label distribution learning [J ] . IEEE Transactions on Knowledge and Data Engineering , 2016 , 28 ( 7 ): 1734 - 1748 .

GAO B B , XING C , XIE C W , et al . Deep label distribution learning with label ambiguity [J ] . IEEE Transactions on Image Processing , 2017 , 26 ( 6 ): 2825 - 2838 .

LE N , NGUYEN K , TRAN Q , et al . Uncertainty-aware label distribution learning for facial expression recognition [C ] // 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) . Piscataway : IEEE , 2023 : 6088 - 6097 .

CHEN S K , WANG J F , CHEN Y D , et al . Label distribution learning on auxiliary label space graphs for facial expression recognition [C ] // 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2020 : 13984 - 13993 .

ZHAO Z Q , LIU Q S , ZHOU F . Robust lightweight facial expression recognition network with label distribution training [J ] . Proceedings of the AAAI Conference on Artificial Intelligence , 2021 , 35 ( 4 ): 3510 - 3519 .

MIKOLOV T , CHEN K , CORRADO G , et al . Efficient estimation of word representations in vector space [J ] . 1st International Conference on Learning Representations , ICLR 2013 - Workshop Track Proceedings, 2013 .

YING C , CAI T , LUO S J , et al . Do Transformers really perform badly for graph representation? [C ] // Neural Information Processing Systems , 2021 , 34 : 28877 - 28888 .

MOLLAHOSSEINI A , HASANI B , MAHOOR M H . AffectNet: A database for facial expression, valence, and arousal computing in the wild [J ] . IEEE Transactions on Affective Computing , 2017 , 10 : 18 - 31 .

LI S , DENG W H , DU J P . Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild [C ] // 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2017 : 2852 - 2861 .

HE K M , ZHANG X Y , REN S Q , et al . Deep residual learning for image recognition [C ] // 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2016 : 770 - 778 .

WANG K , PENG X J , YANG J F , et al . Suppressing uncertainties for large-scale facial expression recognition [C ] // 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2020 : 6897 - 6906 .

WEN Z Y , LIN W Z , WANG T , et al . Distract your attention: Multi-head cross attention network for facial expression recognition [J ] . Biomimetics , 2023 , 8 ( 2 ): 199 .

CAI J , MENG Z B , KHAN A S , et al . Island loss for learning discriminative features in facial expression recognition [C ] // 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) . Piscataway : IEEE , 2018 : 302 - 309 .

LI H Y , WANG N N , YANG X , et al . Towards semi-supervised deep facial expression recognition with an adaptive confidence margin [C ] // 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2022 : 4166 - 4175 .

陈公冠 , 张帆 , 王桦 , 等 . 区域增强型注意力网络下的人脸表情识别 [J ] . 计算机辅助设计与图形学学报 , 2024 , 36 ( 1 ): 152 - 160 .

CHEN G G , ZHANG F , WANG H , et al . Facial expression recognition based on region enhanced attention network [J ] . Journal of Computer-Aided Design & Computer Graphics , 2024 , 36 ( 1 ): 152 - 160 . (in Chinese)

VAN DER MAATEN L , HINTON G . Visualizing data using t-SNE [J ] . Journal of Machine Learning Research , 2008 , 9 : 2579 - 2625 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Modulation Recognition Method Based on Improved Res2Net and Adaptive Multi-Scale Window Pooling

Global Dependency Guided Feature Reconstruction for Image Classification

Sequence-Based and Cross-Modal Alignment Model for Protein Function Prediction

NAHGNN: Neighborhood Aware Heterogeneous Graph Neural Network

Related Author

LIU Ya-zhi

XU Zhe-ming

LANG Cong-yan

WANG Tao

LI Yi-dong

WANG Dan

LI Wanjie

JIANG Fengyang

Related Institution

School of Computer Science and Technology, Beijing Jiaotong University

Key Laboratory of Big Data and Artificial Intelligence in Transportation(Beijing Jiaotong University), Ministry of Education, Beijing Jiaotong University

School of Communications and Information Engineering, Chongqing University of Posts and Telecommunications

School of Software, Liaoning Technical University

Key Laboratory of Optoelectronic Information Control and Security Technology

⁰