Multi-task Learning and Identity-constrained Generative Adversarial Network for Face Frontalization and Recognition

HUANG Xin-yan; LIU Fang; BAO Qian-yue; LI Ren-peng; LIU Xu; LI Ling-ling; CHEN Pu-hua; LIU Yang

doi:10.12263/DZXB.20211352

您当前的位置：

首页 >

文章列表页 >

Multi-task Learning and Identity-constrained Generative Adversarial Network for Face Frontalization and Recognition

PAPERS | 更新时间：2025-12-08

- Multi-task Learning and Identity-constrained Generative Adversarial Network for Face Frontalization and Recognition
- ACTA ELECTRONICA SINICA Vol. 51, Issue 10, Pages: 2936-2949(2023)
- 作者机构：
  
  1.西安电子科技大学人工智能学院,陕西西安 710071
  2.智能感知与图像理解教育部重点实验室,陕西西安 710071
  3.智能感知与计算国际联合研究中心,陕西西安 710071
  4.智能感知与计算国际合作联合实验室,陕西西安 710071
- 作者简介：
- 基金信息：
  
  National Natural Science Foundation of China(62076192);State Key Program of National Natural Science of China(61836009);Program for Cheung Kong Scholars and Innovative Research Team in University(IRT_15R53);Fund for Foreign Scholars in University Research and Teaching Programs(B07048);Key Scientific Technological Innovation Research Project by Ministry of Education;National Key Research and Development Program of China;CAAI Huawei MindSpore Open Fund
- DOI：10.12263/DZXB.20211352
  CLC： TP391.41;
- Received：08 October 2021，
  
  Revised：2023-09-01，
  
  Published：25 October 2023
- 稿件说明：
移动端阅览
黄欣研,刘芳,鲍骞月等.基于多任务学习和身份约束的生成对抗网络人脸校正识别方法[J].电子学报,2023,51(10):2936-2949.

HUANG Xin-yan,LIU Fang,BAO Qian-yue,et al.Multi-task Learning and Identity-constrained Generative Adversarial Network for Face Frontalization and Recognition[J].ACTA ELECTRONICA SINICA,2023,51(10):2936-2949.
黄欣研,刘芳,鲍骞月等.基于多任务学习和身份约束的生成对抗网络人脸校正识别方法[J].电子学报,2023,51(10):2936-2949. DOI： 10.12263/DZXB.20211352.

HUANG Xin-yan,LIU Fang,BAO Qian-yue,et al.Multi-task Learning and Identity-constrained Generative Adversarial Network for Face Frontalization and Recognition[J].ACTA ELECTRONICA SINICA,2023,51(10):2936-2949. DOI： 10.12263/DZXB.20211352.

摘要

针对DR-GAN（Disentangled Representation learning-Generative Adversarial Network）方法在将大偏转角度侧脸图像生成其正脸图像的整个生成过程中，没有考虑身份类别信息，从而导致在身份和姿态的解耦中存在真实的侧脸图像与其生成的正脸图像身份一致性较弱的问题，本文提出了一种基于多任务学习和身份约束的生成对抗网络人脸校正识别方法.该方法通过借鉴多任务学习机制，在生成网络的编码器与解码器之间构建了角度姿态分类模块和身份约束识别模块.这两个模块不但在生成过程中实现了人脸身份和姿态的解耦，更重要的是在由侧脸生成正脸的过程中加入了人脸身份监督信息.在训练过程中，该方法将身份和姿态类别直接作为身份编码特征和姿态编码特征的监督信息，并通过设计身份特征损失函数来约束侧脸的身份编码特征逼近其正脸的身份编码特征，实现了侧脸编码特征中身份信息和姿态信息的有效解耦，使解码器能更准确地生成与原侧脸图像保持身份一致的正脸图像.在M

FPA数据集上，对不同角度的侧脸图像使用所提方法生成的正脸图像进行识别，达到了更高的人脸识别准确率.实验结果表明，即使在偏转角度较大时，所提方法仍然能够较好地生成保持身份一致的正脸图像，显著提升了较大偏转角下人脸识别准确率.

Abstract

For the DR-GAN (Disentangled Representation learning-Generative Adversarial Network)

the identity information is not considered in the whole process of generating frontal faces from non-frontal faces with large pose variations. It results in the weak identity consistency between non-frontal faces and the generated frontal faces for disentangling pose from identity. This paper proposes a multi-task learning and identity-constrained generative adversarial network for face frontalization and recognition. Based on the multi-task learning mechanism

a pose classification module and an identity constraint recognition module are constructed between the encoder and decoder of the generative network. These two modules consider the disentangling of face identity and pose in the generating process. More importantly

face identity supervision information is added in the process of generating faces from non-frontal faces. In the process of training

identity and pose categories are directly used as the supervision information for learning identity coding features and pose coding features. The identity feature loss function is designed to constrain the identity coding features of the non-frontal faces to approximate the identity coding features of the frontal faces. The effective disentangling of identity and pose information in the non-frontal coding feature is realized. The decoder can more accurately generate a frontal face consistent with the non-frontal face. On the M

FPA dataset

the frontal faces generated from the non-frontal faces with different poses by the proposed method are used to recognize

achieving a higher face recognition accuracy. The experimental results show that even when the pose variations are large

the proposed method can still generate a frontal face with a consistent identity

significantly improving face recognition accuracy under large pose variations.

关键词

Keywords

references

TRAN L , YIN X , LIU X . Disentangled representation learning gan for pose-invariant face recognition [C]// Computer Vision and Pattern Recognition . Honolulu : IEEE , 2017 : 1415 - 1424 .

李倩玉 , 蒋建国 , 齐美彬 . 基于改进深层网络的人脸识别基于改进深层网络的人脸识别算法 [J]. 电子学报 , 2017 , 45 ( 3 ): 619 - 625 .

LI Q Y , JIANG J G , QI M B . Face recognition algorithm based on improved deep networks [J]. Acta Electronica Sinica , 2017 , 45 ( 3 ): 619 - 625 . (in Chinese)

徐先峰 , 张丽 , 郎彬 , 等 . 引入感知模型的改进孪生卷积神经网络实现人脸识别算法研究 [J]. 电子学报 , 2020 , 48 ( 4 ): 643 - 647 .

XU X F , ZHANG L , LANG B , et al . Research on inception module incorporated siamese convolutional neural networks to realize face recognition [J]. Acta Electronica Sinica , 2020 , 48 ( 4 ): 643 - 647 . (in Chinese)

SENGUPTA S , CHEN J C , CASTILLO C , et al . Frontal to profile face verification in the wild [C]// 2016 IEEE Winter Conference on Applications of Computer Vision . Lake Placid : IEEE , 2016 : 1 - 9 .

KAN M , SHAN S , CHEN X . Multi-view deep network for cross-view classification [C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas : IEEE , 2016 : 4847 - 4855 .

JAMPOUR M , MAUTHNER T , BISCHOF H . Pairwise linear regression: an efficient and fast multi-view facial expression recognition [C]// 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition . Ljubljana : IEEE , 2015 : 1 - 8 .

JIAO L , SHANG R , LIU F , et al . Brain and Nature-Inspired Learning, Computation and Recognition [M]. Amsterdam : Elsevier Press , 2020 .

焦李成 . 神经网络系统理论 [M]. 西安 : 西安电子科技大学出版社 , 1990 .

JIAO L C . Neural Network System Theory [M]. Xi'an : Xidian University Press , 1990 . (in Chinese)

焦李成 . 神经网络计算 [M]. 西安 : 西安电子科技大学出版社 , 1993 .

JIAO L C . Neural Network Computing [M]. Xi'an : Xidian University Press , 1993 . (in Chinese)

焦李成 . 神经网络的应用与实现 [M]. 西安 : 西安电子科技大学出版社 , 1993 .

JIAO L C . Application and Realization of Neural Network [M]. Xi'an : Xidian University Press , 1993 . (in Chinese)

焦李成 , 赵进 , 杨淑媛 , 等 . 深度神经网络学习、优化与识别 [M]. 北京 : 清华大学出版社 , 2017 .

JIAO L C , ZHAO J , YANG S Y . Deep Learning, Optimization and Recognition [M]. Beijing : Tsinghua University Press , 2017 . (in Chinese)

SCHROFF F , KALENICHENKO D , PHILBIN J . FaceNet: A unified embedding for face recognition and clustering [C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition . Boston : IEEE , 2015 : 815 - 823 .

WEI W , TIAN C , ZHANG Y . Robust face pose classification method based on geometry-preserving visual phrase [C]// 2014 IEEE International Conference on Image Processing . Paris : IEEE , 2015 : 815 - 823 .

ZHANG H , ZHANG Y , HUANG T S . Pose-robust face recognition via sparse representation [J]. Pattern Recognition , 2013 , 46 ( 5 ): 1511 - 1521 .

LIN L , WANG K , MENG D , et al . Active self-paced learning for cost-effective and progressive face identification [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2018 , 40 ( 1 ): 7 - 19 .

CHEN X , ZHU Y , ZHANG Y , Adversarial learning with collaborative attention for facial makeup removal [J]. Neurocomputing , 2021 , 434 : 249 - 260 .

SHAO M , Li L , WANG H , et al . Selective generative adversarial network for raindrop removal from a single image [J]. Neurocomputing , 2021 , 426 : 265 - 273 .

DUAN Y , HAN C , TAO X , et al . Panoramic image generation: from 2-d sketch to spherical image [J]. IEEE Journal of Selected Topics in Signal Processing , 2020 , 14 ( 1 ): 194 - 208 .

HAN C , DUAN Y , TAO X , et al . Toward variable-rate generative compression by reducing the channel redundancy [J]. IEEE Transactions on Circuits and Systems for Video Technology , 2020 , 30 ( 7 ): 1789 - 1802 .

方晨 , 郭渊博 , 王娜 , 等 . 基于生成对抗网络的差分隐私数据发布方法 [J]. 电子学报 , 2020 , 48 ( 10 ): 1983 - 1992 .

FANG C , GUO Y B , WANG N , et al . Differential private data publishing method based on generative adversarial network [J]. Acta Electronica Sinica , 2020 , 48 ( 10 ): 1983 - 1992 . (in Chinese)

江泽涛 , 覃露露 . 一种基于U-Net生成对抗网络的低照度图像增强方法 [J]. 电子学报 , 2020 , 48 ( 2 ): 258 - 264 .

JIANG Z T , QIN L L . Low-light image enhancement method based on u-net generative adversarial network [J]. Acta Electronica Sinica , 2020 , 48 ( 2 ): 258 - 264 . (in Chinese)

王格格 , 郭涛 , 余游 , 等 . 基于生成对抗网络的无监督域适应分类模型 [J]. 电子学报 , 2020 , 48 ( 6 ): 1190 - 1197 .

WANG G G , GUO T , YU Y , et al . D Unsupervised domain adaptation classification model based on generative adversarial network [J]. Acta Electronica Sinica , 2020 , 48 ( 6 ): 1190 - 1197 . (in Chinese)

PENG X , YU X , SOHN K , et al . Reconstruction-based disentanglement for pose-invariant face recognition [C]// 2017 IEEE International Conference on Computer Vision . Venice : IEEE , 2017 : 1632 - 1641 .

GAO H , EKENEL H K , STIEFELHAGEN R . Pose normalization for local appearance-based face recognition [C]// Advances in Biometrics , Third International Conference. Alghero : Springer , 2009 : 32 - 41 .

BOOKSTEIN F L . Principal warps: thin-plate splines and the decomposition of deformations [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence , 1989 , 11 ( 6 ): 567 - 585 .

TAIGMAN Y , YANG M , RANZATO M A , et al . Deepface: closing the gap to human-level performance in face verification [C]// 2014 IEEE Conference on Computer Vision and Pattern Recognition . Columbus : IEEE , 2014 : 1701 - 1708 .

BERG T , BELHUMEUR P N . Tom-vs-Pete classifiers and identity-preserving alignment for face verification [C]// British Machine Vision Conference . Surrey : BMVA Press , 2012 : 1 - 11 .

ASHRAF A B , LUCEY S , CHEN T . Learning patch correspondences for improved viewpoint invariant face recognition [C]// 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition . Salt Lake City : IEEE , 2008 : 1 - 8 .

BEYMER D , POGGIO T . Face recognition from one example view [C]// Procedings of the Fifth International Conference on Computer Vision . Cambridge : IEEE , 1995 : 500 - 507 .

HASSNER T , HAREL S , PAZ E , et al . Effective face frontalization in unconstrained images [C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition . Boston : IEEE , 2015 : 4295 - 4304 .

BLANZ V , VETTER T . A morphable model for the synthesis of 3D faces [C]// Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques . Los Angeles : ACM , 1999 : 187 - 194 .

ROMDHANI S , VETTER T . Estimating 3D shape and texture using pixel intensity, edges, specular highlights, texture constraints and a prior [C]// 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition . San Diego : IEEE , 2005 : 986 - 993 .

YIM J , JUNG H , YOO B I , et al . Rotating your face using multi-task deep neural network [C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition . Boston : IEEE , 2015 : 676 - 684 .

ZHANG Y , SHAO M , WONG E K , et al . Random faces guided sparse many-to-one encoder for pose-invariant face recognition [C]// 2013 IEEE International Conference on Computer Vision . Sydney : IEEE , 2013 : 2416 - 2423 .

KAN M , SHAN S , CHANG H , et al . Stacked progressive auto-encoders (SPAE) for face recognition across poses [C]// 2014 IEEE Conference on Computer Vision and Pattern Recognition . Columbus : IEEE , 2014 : 1883 - 1890 .

HUANG R , ZHANG S , LI T , et al . Beyond face rotation: global and local perception GAN for photorealistic and identity preserving frontal view synthesis [C]// 2017 IEEE International Conference on Computer Vision . Venice : IEEE , 2017 : 2458 - 2467 .

ZHAO J , CHENG Y , XU Y , et al . Towards Pose Invariant Face Recognition in the Wild [C]// 2018 IEEE Conference on Computer Vision and Pattern Recognition . Salt Lake City : IEEE , 2018 : 2207 - 2216 .

QIAN Y , DENG W , HU J . Unsupervised face normalization with extreme pose and expression in the wild [C]// 2019 IEEE Conference on Computer Vision and Pattern Recognition . Long Beach : IEEE , 2019 : 9851 - 9858 .

YIN Y , JIANG S , ROBINSON J P , et al . Dual-attention GAN for large-pose face frontalization [C]// 15th IEEE International Conference on Automatic Face and Gesture Recognition . Buenos Aires : IEEE , 2020 : 249 - 256 .

HU Y , WU X , YU B , et al . Pose-guided photorealistic face rotation [C]// 2018 IEEE Conference on Computer Vision and Pattern Recognition . Salt Lake City : IEEE , 2018 : 8398 - 8406 .

CAO K , RONG Y , LI C , et al . Pose-robust face recognition via deep residual equivariant mapping [C]// 2018 IEEE Conference on Computer Vision and Pattern Recognition . Salt Lake City : IEEE , 2018 : 5187 - 5196 .

DENG J , GUO J , ZHOU Y , et al . Retinaface: Single-stage dense face localisation in the wild [EB/OL]. ( 2019 )[2021]. https://arxiv.org/abs/1905.00641 https://arxiv.org/abs/1905.00641 .

RADFORD A , METZ L , CHINTALA S . Unsupervised representation learning with deep convolutional generative adversarial networks [C]// 4th International Conference on Learning Representations . Piscataway : IEEE , 2016 : 31 - 38 .

JOHNSON J , ALAHI A , LI F F . Perceptual losses for real-time style transfer and super-resolution [C]// European Conference on Computer Vision . Amsterdam : Springer , 2016 : 694 - 711 .

CHEN S , LIU Y , GAO X , et al . Mobilefacenets: efficient cnns for accurate real-time face verification on mobile devices [C]// Chinese Conference on Biometric Recognition . Urumqi : Springer , 2018 : 428 - 438 .

LI P , WU X , HU Y , et al . M 2 fpa: a multi-yaw multi-pitch high-quality dataset and benchmark for facial pose analysis [C]// 2019 IEEE/CVF International Conference on Computer Vision . Seoul : IEEE , 2019 : 10042 - 10050 .

GUOY , ZHANG L , HU Y , et al . Ms-celeb-1m: a dataset and benchmark for large-scale face recognition [C]// European Conference on Computer Vision . Amsterdam : Springer , 2016 : 87 - 102 .

KINGMA D P , BA J . Adam: a method for stochastic optimization [C]// 3rd International Conference on Learning Representations . San Diego : IEEE , 2015 : 1 - 8 .

WANG G , MA J , ZHANG Q , et al . Pseudo facial generation with extreme poses for face recognition [C]// 2021 IEEE Conference on Computer Vision and Pattern Recognition . Virtual : IEEE , 2021 : 1994 - 2003 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Attention Mechanism Optimized Generative Adversarial Networks and Their Application in Sea Clutter Simulation

Multimodal Intent Recognition Based on Hierarchical Semantic-Consistency Learning

Land-Sea Clutter Image Enhancement and Detector Design for Sky-Wave Over-the-Horizon Radar

DRE-3DC: Document-Level Relation Extraction with Three-Dimensional Representation Combination Modeling

Related Author

LIU Fang

ZHANG Su-kai

CHEN Peng

DONG Zi-ying

WANG Wei

PENG Jun-jie

LI Zheng-yi

ZHANG Huan-xiang

Related Institution

School of Information Engineering, Chang’an University

School of Computer Engineering and Science, Shanghai University

School of Innovation and Entrepreneurship Education, Inner Mongolia University of Science and Technology

School of Communication and Information Engineering, Chongqing University of Posts and Telecommunications

Nanjing Research Institute of Electronics Technology

⁰