Survey of Expression Action Unit Recognition Based on Deep Learning

doi:10.12263/DZXB.20210639

Acta Electronica Sinica ›› 2022, Vol. 50 ›› Issue (8): 2003-2017.DOI: 10.12263/DZXB.20210639

• SURVEYS AND REVIEWS • Previous Articles Next Articles

Survey of Expression Action Unit Recognition Based on Deep Learning

SHAO Zhi-wen¹^,², ZHOU Yong¹^,², TAN Xin³, MA Li-zhuang³^,⁴, LIU Bing¹^,², YAO Rui¹^,²

^1.School of Computer Science and Technology, China University of Mining and Technology, Xuzhou, Jiangsu 221116, China
^2.Engineering Research Center of Mine Digitization, Ministry of Education of the People’s Republic of China, Xuzhou, Jiangsu 221116, China
^3.Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
^4.School of Computer Science and Technology, East China Normal University, Shanghai 200062, China

Received:2021-05-18 Revised:2021-09-16 Published:2022-08-25
- Corresponding author:
- ZHOU Yong
- Supported by:
- National Natural Science Foundation of China(62106268);Natural Science Foundation of Jiangsu Province(BK20201346);Six Talents Peaks in Jiangsu Province(2015-DZXX-010);Fundamental Research Funds for the Central Universities(2021QN1072)

基于深度学习的表情动作单元识别综述

邵志文¹^,², 周勇¹^,², 谭鑫³, 马利庄³^,⁴, 刘兵¹^,², 姚睿¹^,²

^1.中国矿业大学计算机科学与技术学院，江苏徐州 221116
^2.矿山数字化教育部工程研究中心，江苏徐州 221116
^3.上海交通大学计算机科学与工程系，上海 200240
^4.华东师范大学计算机科学与技术学院，上海 200062

- 通讯作者:
- 周勇
- 作者简介:
- 邵志文男，1994年生，安徽马鞍山人. 2020年获得上海交通大学博士学位.现为中国矿业大学计算机科学与技术学院准聘副教授.曾主持国家自然科学基金青年项目、江苏省“双创博士”项目、中央高校基本科研业务费青年项目等.研究方向为计算机视觉和深度学习，涵盖人脸表情识别、人脸表情合成、人脸配准等领域.E-mail: zhiwen_shao@cumt.edu.cn
  周勇男，1974年生，江苏徐州人.现为中国矿业大学计算机科学与技术学院教授、博士生导师.曾主持国家自然科学基金面上项目、国家863计划子课题、江苏省“333人才工程”和“六大人才高峰”项目等.研究方向为数据挖掘、机器学习和人工智能.E-mail: yzhou@cumt.edu.cn
- 基金资助:
- 国家自然科学基金(62106268);江苏省自然科学基金(BK20201346);江苏省“六大人才高峰”项目(2015-DZXX-010);中央高校基本科研基金(2021QN1072)

Abstract

Abstract:

Expression action unit(AU) recognition based on deep learning is a hot topic in the fields of computer vision and affective computing. Each AU describes a facial local expression action, and the combinations of AUs can quantitatively represent any expression. Current AU recognition mainly faces three challenging factors, scarcity of labels, difficulty of feature capture, and imbalance of labels. On this basis, this paper categorizes the existing researches into transfer learning based, region learning based, and relation learning based methods, and comments and summarizes each category of representative methods. Finally, this paper compares and analyzes different methods, and further discusses the future research directions of AU recognition.

Key words: expression action unit recognition, scarcity of labels, difficulty of feature capture, imbalance of labels, transfer learning, region learning, relation learning

摘要：

基于深度学习的表情动作单元识别是计算机视觉与情感计算领域的热点课题.每个动作单元描述了一种人脸局部表情动作，其组合可定量地表示任意表情.当前动作单元识别主要面临标签稀缺、特征难捕捉和标签不均衡3个挑战因素.基于此，本文将已有的研究分为基于迁移学习、基于区域学习和基于关联学习的方法，对各类代表性方法进行评述和总结.最后，本文对不同方法进行了比较和分析，并在此基础上探讨了未来动作单元识别的研究方向.

关键词: 表情动作单元识别, 标签稀缺性, 特征难捕捉性, 标签不均衡性, 迁移学习, 区域学习, 关联学习

CLC Number:

TP391.4

SHAO Zhi-wen, ZHOU Yong, TAN Xin, MA Li-zhuang, LIU Bing, YAO Rui. Survey of Expression Action Unit Recognition Based on Deep Learning[J]. Acta Electronica Sinica, 2022, 50(8): 2003-2017.

邵志文, 周勇, 谭鑫, 马利庄, 刘兵, 姚睿. 基于深度学习的表情动作单元识别综述[J]. 电子学报, 2022, 50(8): 2003-2017.

Figures/Tables 10

References 96

1	邱玉, 赵杰煜, 汪燕芳. 结合运动时序性的人脸表情识别方法[J]. 电子学报, 2016, 44(6): 1307-1313.
	QIUY, ZHAOJ Y, WANGY F. Facial expression recognition using temporal relations among facial movements[J]. Acta Electronica Sinica, 2016, 44(6): 1307-1313. (in Chinese)
2	孙晓, 潘汀. 基于兴趣区域深度神经网络的静态面部表情识别[J]. 电子学报, 2017, 45(5): 1189-1197.
	SUNX, PANT. Static facial expression recognition system using roi deep neural networks[J]. Acta Electronica Sinica, 2017, 45(5): 1189-1197. (in Chinese)
3	张瑞, 蒋晨之, 苏剑波. 基于稀疏特征挑选和概率线性判别分析的表情识别研究[J]. 电子学报, 2018, 46(7): 1710-1718.
	ZHANGR, JIANGC Z, SUJ B. Expression recognition based on sparse selection and plda[J]. Acta Electronica Sinica, 2018, 46(7): 1710-1718. (in Chinese)
4	孔德壮, 朱梦宇, 于江坤. 人脸表情识别在辅助医疗中的应用及方法研究[J]. 生命科学仪器, 2019, 18(2): 43-48.
	KONGD Z, ZHUM Y, YUJ K. Research on the application and method of facial expression recognition in assistive medical care[J]. Life Science Instruments, 2019, 18(2): 43-48. (in Chinese)
5	FRANKM G, EKMANP. The ability to detect deceit generalizes across different types of high-stake lies[J]. Journal of Personality and Social Psychology, 1997, 72(6): 1429-1439.
6	EKMANP, FRIESENW V. Facial Action Coding System: A Technique for the Measurement of Facial Movement[M]. Palo Alto: Consulting Psychologists Press, 1978.
7	EKMANP, FRIESENW V, HAGERJ C. Facial Action Coding System[M]. Salt Lake City: Research Nexus, 2002.
8	SCHERERK R, EKMANP. Handbook of Methods in Nonverbal Behavior Research[M]. Cambridge: Cambridge University Press, 1982.
9	FASELB, LUETTINJ. Automatic facial expression analysis: Survey[J]. Pattern Recognition, 2003, 36(1): 259-275.
10	刘晓旻, 谭华春, 章毓晋. 人脸表情识别研究的新进展[J]. 中国图象图形学报, 2006, 11(10): 1359-1368.
	LIUX M, TANH C, ZHANGY J. New research advances in facial expression recognition[J]. Journal of Image and Graphics, 2006, 11(10): 1359-1368. (in Chinese)
11	KUMARIJ, RAJESHR, POOJAK M. Facial expression recognition: A survey[J]. Procedia Computer Science, 2015, 58(1): 486-491.
12	CORNEANUC A, SIMÓNM O, COHNJ F, et al. Survey on rgb, 3d, thermal, and multimodal approaches for facial expression recognition: History, trends, and affect-related applications[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(8): 1548-1568.
13	LIS, DENGW. Deep facial expression recognition: A survey[EB/OL]. (2020-03-17)[2022-05-09]. .
14	贲晛烨, 杨明强, 张鹏, 等. 微表情自动识别综述[J]. 计算机辅助设计与图形学学报, 2014, 26(9): 1385-1395.
	BENX Y, YANGM Q, ZHANGP, et al. Survey on automatic micro expression recognition methods[J]. Journal of Computer-Aided Design and Computer Graphics, 2014, 26(9): 1385-1395. (in Chinese)
15	徐峰, 张军平. 人脸微表情识别综述[J]. 自动化学报, 2017, 43(3): 333-348.
	XUF, ZHANGJ P. Facial microexpression recognition: A survey[J]. Acta Automatica Sinica, 2017, 43(3): 333-348. (in Chinese)
16	MARTINEZB, VALSTARM F, JIANGB, et al. Automatic analysis of facial actions: A survey[J]. IEEE Transactions on Affective Computing, 2019, 10(3): 325-347.
17	ZHIR, LIUM, ZHANGD. A comprehensive survey on automatic facial action unit analysis[J]. The Visual Computer, 2020, 36(5): 1067-1093.
18	PANTICM, VALSTARM, RADEMAKERR, et al. Web-based database for facial expression analysis[C]//Proceedings of the IEEE International Conference on Multimedia and Expo. Amsterdam: IEEE, 2005: 1-5.
19	MATSUMOTOD, YOOS H, S.CultureNAKAGAWA, regulationemotion, and adjustment [J]. Journal of Personality and Social Psychology, 2008, 94(6): 925-937.
20	YANW J, WUQ, LIANGJ, et al. How fast are the leaked facial expressions: The duration of micro-expressions[J]. Journal of Nonverbal Behavior, 2013, 37(4): 217-230.
21	SHENX, WUQ, FUX. Effects of the duration of expressions on the recognition of microexpressions[J]. Journal of Zhejiang University-Science B, 2012, 13(3): 221-230.
22	DAVISONA K, MERGHANIW, YAPM H. Objective classes for micro-facial expression recognition[J]. Journal of Imaging, 2018, 4(10): 119.
23	GUDIA, TASLIH E, DENUYLT M, et al. Deep learning based facs action unit occurrence and intensity estimation[C]//Proceedings of the IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. Ljubljana: IEEE, 2015: 1-5.
24	SHAOZ, LIUZ, CAIJ, et al. Facial action unit detection using attention and relation learning[EB/OL]. (2019-10-23)[2022-05-09]. .
25	KANADET, COHNJ F, TIANY. Comprehensive database for facial expression analysis[C]//Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition. Grenoble: IEEE, 2000: 46-53.
26	ZHANGX, YINL, COHNJ F, et al. Bp4d-spontaneous: A high-resolution spontaneous 3d dynamic facial expression database[J]. Image and Vision Computing, 2014, 32(10): 692-706.
27	MAVADATIS M, MAHOORM H, BARTLETTK, et al. Disfa: A spontaneous facial action intensity database[J]. IEEE Transactions on Affective Computing, 2013, 4(2): 151-160.
28	BENITEZ-QUIROZC F, SRINIVASANR, MARTINEZA M. Emotionet: An accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 5562-5570.
29	KOLLIASD, ZAFEIRIOUS., affectExpression, action unit recognition: Aff-wild2, multi-task learning and arcface[C]//Proceedings of the British Machine Vision Conference. Cardiff: BMVA Press, 2019: 297.
30	LUCEYP, COHNJ F, KANADET, et al. The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. San Francisco: IEEE, 2010: 94-101.
31	VALSTARM, PANTICM. Induced disgust, happiness and surprise: An addition to the mmi facial expression database[C]//Proceedings of the International Conference on Language Resources and Evaluation Workshops. Valletta: ELRA2010: 65-70.
32	SAVRANA, ALYÜZN, DIBEKLIOĞLUH, et al. Bosphorus database for 3D face analysis[C]//Proceedings of the European Workshop on Biometrics and Identity Management. Roskilde: Springer, 2008: 47-56.
33	STRATOUG, GHOSHA, DEBEVECP, et al. Effect of illumination on automatic expression recognition: A novel 3D relightable facial database[C]//Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition. Santa Barbara: IEEE, 2011: 611-618.
34	COSKERD, KRUMHUBERE, HILTONA. A facs valid 3D dynamic action unit database with applications to 3D dynamic morphable facial modeling[C]//Proceedings of the IEEE International Conference on Computer Vision. Barcelona: IEEE, 2011: 2296-2303.
35	ZHANGZ, GIRARDJ M, WUY, et al. Multimodal spontaneous emotion corpus for human behavior analysis[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 3438-3446.
36	GIRARDJ M, CHUW S, JENIL A,et al. Sayette group formation task (gft) spontaneous facial expression database[C]//Proceedings of the IEEE International Conference on Automatic Face & Gesture Recognition. Washington: IEEE, 2017: 581-588.
37	YANW J, LIX, WANGS J, ZHAOG, et al. Casme ii: An improved spontaneous micro-expression database and the baseline evaluation[J]. PloS One, 2014, 9(1): e86041.
38	DAVISONA K, LANSLEYC, COSTENN, et al. Samm: A spontaneous micro-facial movement dataset[J]. IEEE Transactions on Affective Computing, 2018, 9(1): 116-129.
39	BEN X, RENY, ZHANGJ, et al. Video-based facial micro-expression analysis: A survey of datasets, features and algorithms[EB/OL]. (2021-03-19)[2022-05-09]. .
40	ZHOUY, PIJ, SHIB E. Pose-independent facial action unit intensity regression based on multi-task deep transfer learning[C]/Proceedings of the IEEE International Conference on Automatic Face & Gesture Recognition. Washington: IEEE, 2017: 872-877.
41	RUSSAKOVSKYO, DENGJ, SUH, et al. Imagenet large scale visual recognition challenge[J]. International Journal of Computer Vision, 2015, 115(3): 211-252.
42	SIMONYANK, ZISSERMANA. Very deep convolutional networks for large-scale image recognition[C]//Proceedings of the International Conference on Learning Representations. San Diego: OpenReview, 2015: 1-14.
43	JIS, WANGK, PENGX, et al. Multiple transfer learning and multi-label balanced training strategies for facial AU detection in the wild[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. Seattle: IEEE, 2020: 1657-1661.
44	HEK, ZHANGX, RENS, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778.
45	WERNERP, SAXENF, AL-HAMADIA. Facial action unit recognition in the wild with multi-task cnn self-training for the emotionet challenge[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. Seattle: IEEE, 2020: 410-411.
46	PENGG, WANGS. Weakly supervised facial action unit recognition through adversarial training[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 2188-2196.
47	PENGG, WANGS. Dual semi-supervised learning for facial action unit recognition[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Honolulu: AAAI Press, 2019: 8827-8834.
48	GOODFELLOWI, POUGET-ABADIEJ, MIRZAM, et al. Generative adversarial nets[C]//Proceedings of the Advances in Neural Information Processing Systems. Montreal: MIT Press, 2014: 2672-2680.
49	ZHANGY, DONGW, HUB G, et al. Classifier learning with prior probabilities for facial action unit recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 5108-5116.
50	WUS, WANGS, PANB, et al. Deep facial action unit recognition from partially labeled data[C]//Proceedings of the IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 3951-3959.
51	ZHANGY, DONGW, HUB G, et al. Weakly-supervised deep convolutional neural network learning for facial action unit intensity estimation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 2314-2323.
52	SHAOZ, LIUZ, CAIJ, et al. Jâa-net: Joint facial action unit detection and face alignment via adaptive attention[J]. International Journal of Computer Vision, 2021, 129(2): 321-340.
53	JYOTIS, SHARMAG, DHALLA. Expression empowered residen network for facial action unit detection[C]//Proceedings of the IEEE International Conference on Automatic Face & Gesture Recognition. Lille: IEEE, 2019: 1-8.
54	TUC H, YANGC Y, HSUJ Y. Idennet: Identity-aware facial action unit detection[C]//Proceedings of the IEEE International Conference on Automatic Face & Gesture Recognition. Lille: IEEE, 2019: 1-8.
55	LIUZ, SONGG, CAIJ, et al. Conditional adversarial synthesis of 3D facial action units[J]. Neurocomputing, 2019, 355: 200-208.
56	MIRZAM, OSINDEROS. Conditional generative adversarial nets[EB/OL]. (2014-11-06)[2021-05-18]. .
57	BLANZV, VETTERT. A morphable model for the synthesis of 3D faces[C]//Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques of SIGGRAPH. Los Angeles: ACM, 1999: 187-194.
58	WANGC, WANGS. Personalized multiple facial action unit recognition through generative adversarial recognition network[C]//Proceedings of the ACM International Conference on Multimedia. Seoul: ACM, 2018: 302-310.
59	WILESO, KOEPKEA S, ZISSERMANA. Self-supervised learning of a facial attribute embedding from video[C]//Proceedings of the British Machine Vision Conference. Newcastle: BMVA Press, 2018: 302.
60	LIY, ZENGJ, SHANS. Learning representations for facial actions from unlabeled videos[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(1): 302-317.
61	LIW, ABTAHIF, ZHUZ, et al. Eac-net: Deep nets with enhancing and cropping for facial action unit detection[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(11): 2583-2596.
62	JAISWALS, VALSTARM. Deep learning the dynamic appearance and shape of facial action units[C]//Proceedings of the IEEE Winter Conference on Applications of Computer Vision. Lake Placid: IEEE, 2016: 1-8.
63	ALIA M, ALKABBANYI, FARAGA, et al. Facial action units detection under pose variations using deep regions learning[C]//Proceedings of the International Conference on Affective Computing and Intelligent Interaction. San Antonio: IEEE, 2017: 395-400.
64	MAC, CHENL, YONGJ. AU R-CNN: Encoding expert prior knowledge into R-CNN for action unit detection[J]. Neurocomputing, 2019, 355: 35-47.
65	LIW, ABTAHIF, ZHUZ. Action unit detection with region adaptation, multi-labeling learning and optimal temporal fusing[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 6766-6775.
66	SANCHEZE, TZIMIROPOULOSG, VALSTARM. Joint action unit localisation and intensity estimation through heatmap regression[C]//Proceedings of the British Machine Vision Conference. Newcastle: BMVA Press, 2018: 233.
67	LIUM, LIS, SHANS, et al. AU-aware deep Networks for facial expression recognition[C]//Proceedings of the IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. Shanghai: IEEE, 2013: 1-6.
68	ZHAOK, CHUW S, ZHANGH. Deep region and multi-label learning for facial action unit detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 3391-3399.
69	HANS, MENGZ, LIZ, et al. Optimizing filter size in convolutional neural networks for facial action unit recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 5070-5078.
70	ERTUGRULI O, JENIL A, COHNJ F. Pattnet: Patch-attentive deep network for action unit detection[C]//Proceedings of the British Machine Vision Conference. Cardiff: BMVA Press, 2019: 114.1-114.13.
71	ERTUGRULI O, YANGL, JENIL A, et al. D-pattnet: Dynamic patch-attentive deep network for action unit detection[J]. Frontiers in Computer Science, 2019, 1(11): 1-13.
72	NIUX, HANH, YANGS, et al. Local relationship learning with person-specific shape regularization for facial action unit detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 11917-11926.
73	FANY, LINZ. G2rl: Geometry-guided representation learning for facial action unit intensity estimation[C]//Proceedings of the International Joint Conference on Artificial Intelligence. Virtual Conference: IJCAI, 2020: 731-737.
74	TRAND L, WALECKIR, RUDOVICO, et al. Deepcoder: Semi-parametric variational autoencoders for automatic facial action coding[C]//Proceedings of the IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 3190-3199.
75	BENITEZ-QUIROZC F, WANGY, MARTINEZA M. Recognition of action units in the wild with deep nets and a new global-local Loss[C]//Proceedings of the IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 3970-3979.
76	WALECKIR, RUDOVICO, PAVLOVICV, et al. Deep structured learning for facial action unit intensity estimation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 3405-3414.
77	CORNEANUC A, MADADIM, ESCALERAS. Deep structure inference network for facial action unit recognition[C]//Proceedings of the European Conference on Computer Vision. Munich: Springer, 2018: 309-324.
78	JACOBG M, STENGERB. Facial action unit detection with transformers[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Virtual Conference: IEEE, 2021: 7680-7689.
79	LIG, ZHUX, ZENGY, et al. Semantic relationships guided representation learning for facial action unit recognition[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Honolulu: AAAI, 2019: 8594-8601.
80	LIY, TARLOWD, BROCKSCHMIDTM, et al. Gated graph sequence neural networks[C]//Proceedings of the International Conference on Learning Representations. San Juan: OpenReview, 2016: 1-16.
81	LIUZ, DONGJ, ZHANGC, et al. Relation modeling with graph convolutional networks for facial action unit detection[C]//Proceedings of the International Conference on Multimedia Modeling. Daejeon: Springer, 2020: 489-501.
82	NIUX, HANH, SHANS, et al. Multi-label co-regularization for semi-supervised facial action unit recognition[C]//Proceedings of the Advances in Neural Information Processing Systems. Vancouver: Curran Associates2019: 909-919.
83	FANY, LAMJ C K, LIV O K. Facial action unit intensity estimation via semantic correspondence learning with dynamic graph convolution[C]//Proceedings of the AAAI Conference on Artificial Intelligence. New York: AAAI, 2020: 12701-12708.
84	SONGT, CHENL, ZHENGW, et al. Uncertain graph neural networks for facial action unit detection[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Virtual Conference: AAAI, 2021: 5993-6001.
85	SONGT, CUIZ, ZHENGW, et al. Hybrid message passing with performance-driven structures for facial action unit detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Virtual Conference: IEEE, 2021: 6267-6276.
86	CHUW S, DE LA TORREF, COHNJ F. Learning spatial and temporal cues for multi-label facial action unit detection[C]//Proceedings of the IEEE International Conference on Automatic Face & Gesture Recognition. Washington: IEEE, 2017: 25-32.
87	BISHAYM, PATRASI. Fusing multilabel deep networks for facial action unit detection[C]//Proceedings of the IEEE International Conference on Automatic Face & Gesture Recognition. Washington: IEEE, 2017: 681-688.
88	HEJ, LID, YANGB, et al. Multi view facial action unit detection based on CNN and BLSTM-RNN[C]//Proceedings of the IEEE International Conference on Automatic Face & Gesture Recognition. Washington: IEEE, 2017: 848-853.
89	SONGT, CUIZ, WANGY, et al. Dynamic probabilistic graph convolution for facial action unit intensity estimation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Virtual Conference: IEEE, 2021: 4845-4854.
90	YANGL, ERTUGRULI O, COHNJ F, et al. FACS3D-NET: 3D convolution based spatiotemporal representation for action unit detection[C]//Proceedings of the International Conference on Affective Computing and Intelligent Interaction. Cambridge: IEEE, 2019: 538-544.
91	YANGH, YINL. Learning temporal information from a single image for au detection[C]//Proceedings of the IEEE International Conference on Automatic Face & Gesture Recognition. Lille: IEEE, 2019: 1-8.
92	ZHANGY, JIANGH, WUB, et al. Context-aware feature and label fusion for facial action unit intensity estimation with partially labeled data[C]//Proceedings of the IEEE International Conference on Computer Vision. Seoul: IEEE, 2019: 733-742.
93	SHROUTP E, FLEISSJ L. Intraclass correlations: Uses in assessing rater reliability[J]. Psychological Bulletin, 1979, 86(2): 420-428.
94	LINL, WANGK, MENGD, et al. Active self-paced learning for cost-effective and progressive face identification[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 40(1): 7-19.
95	胡小娟, 刘磊, 邱宁佳. 基于主动学习和否定选择的垃圾邮件分类算法[J]. 电子学报, 2018, 46(1): 203-209.
	HUX J, LIUL, QIUN J. A novel spam categorization algorithm based on active learning method and negative selection algorithm[J]. Acta Electronica Sinica, 2018, 46(1): 203-209. (in Chinese)
96	姚拓中, 安鹏, 宋加涛. 基于历史分类加权和分级竞争采样的多视角主动学习[J]. 电子学报, 2017, 45(1): 46-53.
	YAOT Z, ANP, SONGJ T. Multi-view active learning based on weighted hypothesis boosting and hierarchical competition sampling[J]. Acta Electronica Sinica, 2017, 45(1): 46-53. (in Chinese)

整体表情	关联的AU
快乐	6, 12, 25
惊讶	1, 2, 5, 26, 27
愤怒	4, 5, 7, 10, 17, 22, 23, 24, 25, 26
厌恶	9, 10, 16, 17, 25, 26
悲伤	1, 4, 6, 11, 15, 17
恐惧	1, 2, 4, 5, 20, 25, 26, 27

整体表情	关联的AU
快乐	6, 12, 25
惊讶	1, 2, 5, 26, 27
愤怒	4, 5, 7, 10, 17, 22, 23, 24, 25, 26
厌恶	9, 10, 16, 17, 25, 26
悲伤	1, 4, 6, 11, 15, 17
恐惧	1, 2, 4, 5, 20, 25, 26, 27

微表情	AU组合
I	6, 12, 6+12, 6+7+12, 7+12
II	1+2, 5, 25, 1+2+25, 25+26, 5+24
III	23, 4, 4+7, 4+5, 4+5+7, 17+24, 4+6+7, 4+38
IV	10, 9, 4+9, 4+40, 4+5+40, 4+7+9, 4 +9+17, 4+7+10, 4+5+7+9, 7+10
V	1, 15, 1+4, 6+15, 15+17
VI	1+2+4, 20
VII	其他AU组合

微表情	AU组合
I	6, 12, 6+12, 6+7+12, 7+12
II	1+2, 5, 25, 1+2+25, 25+26, 5+24
III	23, 4, 4+7, 4+5, 4+5+7, 17+24, 4+6+7, 4+38
IV	10, 9, 4+9, 4+40, 4+5+40, 4+7+9, 4 +9+17, 4+7+10, 4+5+7+9, 7+10
V	1, 15, 1+4, 6+15, 15+17
VI	1+2+4, 20
VII	其他AU组合

数据集	采集环境	表情激发方式	人脸身份数	图片或视频数	数据形式	发布年份
CK^[25]	受控	人为	97	486个视频	2D	2000
BP4D^[26]	受控	自发	41	328个视频	2D, 3D	2014
DISFA^[27]	受控	自发	27	27个视频	2D	2013
EmotioNet^[28]	非受控	自发	—	约975 000张图片	2D	2016
Aff-Wild2 (AU Set)^[29]	非受控	自发	63	63个视频	2D	2019
CK+^[30]	受控	自发	26	107个视频	2D	2010
MMI^[18,31]	受控	人为、自发	67	2 390个视频和493张图片	2D	2005, 2010
Bosphorous^[32]	受控	人为	105	4 652张图片	3D	2008
ICT-3DRFE^[33]	受控	人为	23	345张图片	2D, 3D	2011
D3DFACS^[34]	受控	人为	10	519个视频	2D, 3D	2011
BP4D+^[35]	受控	自发	140	1 400个视频	2D, 3D	2016
GFT^[36]	受控	自发	96	96个视频	2D	2017
CASME II^[37]	受控	自发	35	247个视频	2D	2014
SAMM^[38]	受控	自发	32	159个视频	2D	2018
MMEW^[39]	受控	自发	30	300个视频和900张图片	2D	2021

Survey of Expression Action Unit Recognition Based on Deep Learning

基于深度学习的表情动作单元识别综述

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 10

References 96

Related Articles 15

Recommended Articles

Metrics

方法	基于已有模型的迁移学习	基于已有标签的迁移学习	基于域映射的迁移学习	特征点辅助的区域学习	自适应区域学习	像素级关联学习	AU级关联学习	时域关联学习	BP4D^[26]/DISFA^[27]
DRML^[68]					√				0.483/0.267
EAC-Net^[61]	√			√			√		0.559/0.485
R-T1^[65]	√			√			√	√	0.661/0.513
DSIN^[77]				√			√		0.589/0.536
LP-Net^[72]	√			√		√			0.610/0.569
MLCR^[82]		√					√		0.598/—
SRERL^[79]	√			√			√		0.629/0.559
ARL^[24]					√	√	√		0.611/0.587
PAttNet^[70]				√					0.626/—
D-PAttNet^[71]				√				√	0.641/—
TAE^[60]			√		√				0.603/0.515
OF-Net^[91]		√						√	0.597/0.537
AU R-CNN^[64]		√		√					0.630/0.513
AU-GCN^[81]				√			√		0.628/0.550
JÂA-Net^[52]		√		√			√		0.624/0.635
UGN-B^[84]	√						√		0.633/0.600
Transformer^[78]	√			√			√		0.642/0.615
HMP-PS^[85]	√						√		0.634/0.610

方法	基于已有模型的迁移学习	基于已有标签的迁移学习	特征点辅助的区域学习	自适应区域学习	像素级关联学习	AU级关联学习	时域关联学习	BP4D^[26]/DISFA^[27]
DRML^[68,76]				√				0.52/0.29
CCNN-IT^[76]						√		0.63/0.45
2DC^[74]						√		0.66/0.50
KBSS^[51]		√					√	0.67/0.36
ARL^[24]				√	√			0.66/0.48
SCC^[83]			√			√		0.72/0.47
G2RL^[73]	√		√		√			0.69/0.52
DPG^[89]						√	√	0.72/0.56

[1]	WU Jun-jun, WANG Tao, WANG Ying-kai, WANG Xing-hui. Transfer-Learning-Based Virtual Process Optimization for LiPON [J]. Acta Electronica Sinica, 2023, 51(3): 687-693.
[2]	YANG Wei, LI Wei-jie, LIU Yong-xiang, LI Xiang. Radar Target Recognition Method for HRRP Based on Geodesic Flow Kernel [J]. Acta Electronica Sinica, 2023, 51(3): 527-536.
[3]	LI Wei-gang, GAN Ping, XIE Lu, LI Song-tao. A Few-Shot Image Classification Method by Pairwise-Based Meta Learning [J]. Acta Electronica Sinica, 2022, 50(2): 295-304.
[4]	CHEN Wen-jun, YANG Chun-ling. Feature-Space Optimization-Inspired and Self-Attention Enhanced Neural Network Reconstruction Algorithm for Image Compressive Sensing [J]. Acta Electronica Sinica, 2022, 50(11): 2629-2637.
[5]	ZHANG Xiao-heng, ZHANG Xin-yue, LI Yong-ming, WANG Pin, LIU Yu-chuan. An Unsupervised Two-Step Convolution Sparse Transfer Learning Algorithm for Parkinson’s Disease Speech Diagnosis [J]. Acta Electronica Sinica, 2022, 50(1): 177-184.
[6]	LI Yong-bin, WANG Bin, SHAO Gao-ping, SHAO Shuai. A Method of Noise Reduction for Underwater Acoustic Communication Signal Based on RCGAN [J]. Acta Electronica Sinica, 2022, 50(1): 54-62.
[7]	TANG Huan-ling, ZHENG Han, LIU Yan-hong, MA Si-yuan, DOU Quan-sheng, LU Ming-yu. Tr-SLDA: A Transfer Topic Model for Cross-Domains [J]. Acta Electronica Sinica, 2021, 49(3): 605-613.
[8]	ZHANG Jin, LI Yang, REN Chuan-lun, HUANG Lian, WANG Shuai-hui, DUAN Ye-xin, PAN Zhi-song, XIE Jun. Cross-Scene Foreground Segmentation Algorithm Based on High-Level Feature Differencing Between Frames [J]. Acta Electronica Sinica, 2021, 49(10): 2032-2040.
[9]	WANG Ge-ge, GUO Tao, YU You, SU Han. Unsupervised Domain Adaptation Classification Model Based on Generative Adversarial Network [J]. Acta Electronica Sinica, 2020, 48(6): 1190-1197.
[10]	ZHANG Hong-bin, JIANG Zi-liang, XIONG Qi-peng, WU Jin-peng, WU Ren-zhong, YUAN Tian, JI Dong-hong. Image Attribute Annotation via a Modified Effective Range Based Gene Selection and Cross-Modal Semantics Mining [J]. Acta Electronica Sinica, 2020, 48(4): 790-799.
[11]	ZHAO Peng, WANG Mei-yu, JI Xia, LIU Hui-ting. A Novel Feature Representation Based on Tensor and Domain Adaption for Transfer Learning [J]. Acta Electronica Sinica, 2020, 48(2): 359-368.
[12]	QUAN Yu, LI Zhi-xin, ZHANG Can-long, MA Hui-fang. Fusing Deep Dilated Convolutions Network and Light-Weight Network for Object Detection [J]. Acta Electronica Sinica, 2020, 48(2): 390-397.
[13]	LIU Zhen, TIAN Jing-yu, YUAN Bao-xin, SUN Yong-qi. Cross-Domain Recommendation Algorithm Based on Knowledge Aggregation and Transfer [J]. Acta Electronica Sinica, 2020, 48(10): 1928-1932.
[14]	JI Ding-cheng, JIANG Yi-zhang, WANG Shi-tong. Multi-source Transfer Learning Method by Balancing both the Domains and Instances [J]. Acta Electronica Sinica, 2019, 47(3): 692-699.
[15]	YU You, FENG Lin, WANG Ge-ge, XU Qi-feng. A Few-Shot Learning Model Based on Semi-Supervised with Pseudo Label [J]. Acta Electronica Sinica, 2019, 47(11): 2284-2291.