基于多任务学习和身份约束的生成对抗网络人脸校正识别方法

黄欣研; 刘芳; 鲍骞月; 李任鹏; 刘旭; 李玲玲; 陈璞花; 刘洋

doi:10.12263/DZXB.20211352

您当前的位置：

首页 >

文章列表页 >

基于多任务学习和身份约束的生成对抗网络人脸校正识别方法

学术论文 | 更新时间：2025-12-08

- 基于多任务学习和身份约束的生成对抗网络人脸校正识别方法
- Multi-task Learning and Identity-constrained Generative Adversarial Network for Face Frontalization and Recognition
- 电子学报 2023年51卷第10期页码：2936-2949
- 作者机构：
  
  1.西安电子科技大学人工智能学院,陕西西安 710071
  2.智能感知与图像理解教育部重点实验室,陕西西安 710071
  3.智能感知与计算国际联合研究中心,陕西西安 710071
  4.智能感知与计算国际合作联合实验室,陕西西安 710071
- 作者简介：
  
  [ "黄欣研女，1992年7月出生，陕西宝鸡人.现为西安电子科技大学人工智能学院博士研究生.主要研究方向为深度学习、图像处理和计算机视觉. E-mail: xinyanh@stu.xidian.edu.cn" ]
  [ "刘芳（通讯作者）女，1963年2月出生，湖南华容人.1995年毕业于西安电子科技大学.现为西安电子科技大学人工智能学院教授，博士生导师.主要研究方向为人工智能和模式识别、机器学习、图像感知和场景理解、进化计算和数据挖掘." ]
  [ "鲍骞月男，1998年8月出生，山西朔州人.现为西安电子科技大学人工智能学院博士研究生.主要研究方向为机器学习、图像处理和模式识别. E-mail: baoqianyue@163.com" ]
  [ "李任鹏男，1997年9月出生，湖南娄底人.现为西安电子科技大学人工智能学院硕士研究生.主要研究方向为深度学习和图像处理. E-mail: lrpp121314@163.com" ]
  [ "刘旭男，1989年5月出生，陕西咸阳人.2019年毕业于西安电子科技大学.现为西安电子科技大学人工智能学院副教授.主要研究方向为机器学习/深度学习理论、图像/视频处理方法.中国电子学会会员编号：E190027251M." ]
  [ "李玲玲女，1988年4月出生，陕西白水人.2017年毕业于西安电子科技大学.现为西安电子科技大学人工智能学院副教授.主要研究方向为量子进化优化学习、深度学习方法与应用、复杂遥感影像理解与解译. E-mail: llli@xidian.edu.cn" ]
  [ "陈璞花女，1986年7月出生，陕西汉中人.2016年毕业于西安电子科技大学.现为西安电子科技大学人工智能学院副教授.主要研究方向为机器学习、模式识别和遥感图像解译. E-mail: phchen@xidian.edu.cn" ]
  [ "刘洋女，1998 年 6 月出生，山西朔州人.现为西安电子科技大学人工智能学院博士研究生.主要研究方向为机器学习、图像处理和模式识别. E-mail: yyyliu98@163.com" ]
- 基金信息：
  
  国家自然科学基金(62076192);国家自然科学基金重点项目(61836009);长江学者及大学创新研究团队计划(IRT_15R53);高等学校学科创新引智计划(B07048);教育部重点科技创新研究项目;国家重点研发计划;CAAI华为MindSpore开放基金
- DOI：10.12263/DZXB.20211352
  中图分类号： TP391.41;
- 收稿：2021-10-08，
  
  修回：2023-09-01，
  
  纸质出版：2023-10-25
- 稿件说明：
移动端阅览
黄欣研,刘芳,鲍骞月等.基于多任务学习和身份约束的生成对抗网络人脸校正识别方法[J].电子学报,2023,51(10):2936-2949.

HUANG Xin-yan,LIU Fang,BAO Qian-yue,et al.Multi-task Learning and Identity-constrained Generative Adversarial Network for Face Frontalization and Recognition[J].ACTA ELECTRONICA SINICA,2023,51(10):2936-2949.
黄欣研,刘芳,鲍骞月等.基于多任务学习和身份约束的生成对抗网络人脸校正识别方法[J].电子学报,2023,51(10):2936-2949. DOI： 10.12263/DZXB.20211352.

HUANG Xin-yan,LIU Fang,BAO Qian-yue,et al.Multi-task Learning and Identity-constrained Generative Adversarial Network for Face Frontalization and Recognition[J].ACTA ELECTRONICA SINICA,2023,51(10):2936-2949. DOI： 10.12263/DZXB.20211352.

摘要

针对DR-GAN（Disentangled Representation learning-Generative Adversarial Network）方法在将大偏转角度侧脸图像生成其正脸图像的整个生成过程中，没有考虑身份类别信息，从而导致在身份和姿态的解耦中存在真实的侧脸图像与其生成的正脸图像身份一致性较弱的问题，本文提出了一种基于多任务学习和身份约束的生成对抗网络人脸校正识别方法.该方法通过借鉴多任务学习机制，在生成网络的编码器与解码器之间构建了角度姿态分类模块和身份约束识别模块.这两个模块不但在生成过程中实现了人脸身份和姿态的解耦，更重要的是在由侧脸生成正脸的过程中加入了人脸身份监督信息.在训练过程中，该方法将身份和姿态类别直接作为身份编码特征和姿态编码特征的监督信息，并通过设计身份特征损失函数来约束侧脸的身份编码特征逼近其正脸的身份编码特征，实现了侧脸编码特征中身份信息和姿态信息的有效解耦，使解码器能更准确地生成与原侧脸图像保持身份一致的正脸图像.在M

FPA数据集上，对不同角度的侧脸图像使用所提方法生成的正脸图像进行识别，达到了更高的人脸识别准确率.实验结果表明，即使在偏转角度较大时，所提方法仍然能够较好地生成保持身份一致的正脸图像，显著提升了较大偏转角下人脸识别准确率.

Abstract

For the DR-GAN (Disentangled Representation learning-Generative Adversarial Network)

the identity information is not considered in the whole process of generating frontal faces from non-frontal faces with large pose variations. It results in the weak identity consistency between non-frontal faces and the generated frontal faces for disentangling pose from identity. This paper proposes a multi-task learning and identity-constrained generative adversarial network for face frontalization and recognition. Based on the multi-task learning mechanism

a pose classification module and an identity constraint recognition module are constructed between the encoder and decoder of the generative network. These two modules consider the disentangling of face identity and pose in the generating process. More importantly

face identity supervision information is added in the process of generating faces from non-frontal faces. In the process of training

identity and pose categories are directly used as the supervision information for learning identity coding features and pose coding features. The identity feature loss function is designed to constrain the identity coding features of the non-frontal faces to approximate the identity coding features of the frontal faces. The effective disentangling of identity and pose information in the non-frontal coding feature is realized. The decoder can more accurately generate a frontal face consistent with the non-frontal face. On the M

FPA dataset

the frontal faces generated from the non-frontal faces with different poses by the proposed method are used to recognize

achieving a higher face recognition accuracy. The experimental results show that even when the pose variations are large

the proposed method can still generate a frontal face with a consistent identity

significantly improving face recognition accuracy under large pose variations.

关键词

Keywords

references

TRAN L , YIN X , LIU X . Disentangled representation learning gan for pose-invariant face recognition [C]// Computer Vision and Pattern Recognition . Honolulu : IEEE , 2017 : 1415 - 1424 .

李倩玉 , 蒋建国 , 齐美彬 . 基于改进深层网络的人脸识别基于改进深层网络的人脸识别算法 [J]. 电子学报 , 2017 , 45 ( 3 ): 619 - 625 .

LI Q Y , JIANG J G , QI M B . Face recognition algorithm based on improved deep networks [J]. Acta Electronica Sinica , 2017 , 45 ( 3 ): 619 - 625 . (in Chinese)

徐先峰 , 张丽 , 郎彬 , 等 . 引入感知模型的改进孪生卷积神经网络实现人脸识别算法研究 [J]. 电子学报 , 2020 , 48 ( 4 ): 643 - 647 .

XU X F , ZHANG L , LANG B , et al . Research on inception module incorporated siamese convolutional neural networks to realize face recognition [J]. Acta Electronica Sinica , 2020 , 48 ( 4 ): 643 - 647 . (in Chinese)

SENGUPTA S , CHEN J C , CASTILLO C , et al . Frontal to profile face verification in the wild [C]// 2016 IEEE Winter Conference on Applications of Computer Vision . Lake Placid : IEEE , 2016 : 1 - 9 .

KAN M , SHAN S , CHEN X . Multi-view deep network for cross-view classification [C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas : IEEE , 2016 : 4847 - 4855 .

JAMPOUR M , MAUTHNER T , BISCHOF H . Pairwise linear regression: an efficient and fast multi-view facial expression recognition [C]// 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition . Ljubljana : IEEE , 2015 : 1 - 8 .

JIAO L , SHANG R , LIU F , et al . Brain and Nature-Inspired Learning, Computation and Recognition [M]. Amsterdam : Elsevier Press , 2020 .

焦李成 . 神经网络系统理论 [M]. 西安 : 西安电子科技大学出版社 , 1990 .

JIAO L C . Neural Network System Theory [M]. Xi'an : Xidian University Press , 1990 . (in Chinese)

焦李成 . 神经网络计算 [M]. 西安 : 西安电子科技大学出版社 , 1993 .

JIAO L C . Neural Network Computing [M]. Xi'an : Xidian University Press , 1993 . (in Chinese)

焦李成 . 神经网络的应用与实现 [M]. 西安 : 西安电子科技大学出版社 , 1993 .

JIAO L C . Application and Realization of Neural Network [M]. Xi'an : Xidian University Press , 1993 . (in Chinese)

焦李成 , 赵进 , 杨淑媛 , 等 . 深度神经网络学习、优化与识别 [M]. 北京 : 清华大学出版社 , 2017 .

JIAO L C , ZHAO J , YANG S Y . Deep Learning, Optimization and Recognition [M]. Beijing : Tsinghua University Press , 2017 . (in Chinese)

SCHROFF F , KALENICHENKO D , PHILBIN J . FaceNet: A unified embedding for face recognition and clustering [C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition . Boston : IEEE , 2015 : 815 - 823 .

WEI W , TIAN C , ZHANG Y . Robust face pose classification method based on geometry-preserving visual phrase [C]// 2014 IEEE International Conference on Image Processing . Paris : IEEE , 2015 : 815 - 823 .

ZHANG H , ZHANG Y , HUANG T S . Pose-robust face recognition via sparse representation [J]. Pattern Recognition , 2013 , 46 ( 5 ): 1511 - 1521 .

LIN L , WANG K , MENG D , et al . Active self-paced learning for cost-effective and progressive face identification [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2018 , 40 ( 1 ): 7 - 19 .

CHEN X , ZHU Y , ZHANG Y , Adversarial learning with collaborative attention for facial makeup removal [J]. Neurocomputing , 2021 , 434 : 249 - 260 .

SHAO M , Li L , WANG H , et al . Selective generative adversarial network for raindrop removal from a single image [J]. Neurocomputing , 2021 , 426 : 265 - 273 .

DUAN Y , HAN C , TAO X , et al . Panoramic image generation: from 2-d sketch to spherical image [J]. IEEE Journal of Selected Topics in Signal Processing , 2020 , 14 ( 1 ): 194 - 208 .

HAN C , DUAN Y , TAO X , et al . Toward variable-rate generative compression by reducing the channel redundancy [J]. IEEE Transactions on Circuits and Systems for Video Technology , 2020 , 30 ( 7 ): 1789 - 1802 .

方晨 , 郭渊博 , 王娜 , 等 . 基于生成对抗网络的差分隐私数据发布方法 [J]. 电子学报 , 2020 , 48 ( 10 ): 1983 - 1992 .

FANG C , GUO Y B , WANG N , et al . Differential private data publishing method based on generative adversarial network [J]. Acta Electronica Sinica , 2020 , 48 ( 10 ): 1983 - 1992 . (in Chinese)

江泽涛 , 覃露露 . 一种基于U-Net生成对抗网络的低照度图像增强方法 [J]. 电子学报 , 2020 , 48 ( 2 ): 258 - 264 .

JIANG Z T , QIN L L . Low-light image enhancement method based on u-net generative adversarial network [J]. Acta Electronica Sinica , 2020 , 48 ( 2 ): 258 - 264 . (in Chinese)

王格格 , 郭涛 , 余游 , 等 . 基于生成对抗网络的无监督域适应分类模型 [J]. 电子学报 , 2020 , 48 ( 6 ): 1190 - 1197 .

WANG G G , GUO T , YU Y , et al . D Unsupervised domain adaptation classification model based on generative adversarial network [J]. Acta Electronica Sinica , 2020 , 48 ( 6 ): 1190 - 1197 . (in Chinese)

PENG X , YU X , SOHN K , et al . Reconstruction-based disentanglement for pose-invariant face recognition [C]// 2017 IEEE International Conference on Computer Vision . Venice : IEEE , 2017 : 1632 - 1641 .

GAO H , EKENEL H K , STIEFELHAGEN R . Pose normalization for local appearance-based face recognition [C]// Advances in Biometrics , Third International Conference. Alghero : Springer , 2009 : 32 - 41 .

BOOKSTEIN F L . Principal warps: thin-plate splines and the decomposition of deformations [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence , 1989 , 11 ( 6 ): 567 - 585 .

TAIGMAN Y , YANG M , RANZATO M A , et al . Deepface: closing the gap to human-level performance in face verification [C]// 2014 IEEE Conference on Computer Vision and Pattern Recognition . Columbus : IEEE , 2014 : 1701 - 1708 .

BERG T , BELHUMEUR P N . Tom-vs-Pete classifiers and identity-preserving alignment for face verification [C]// British Machine Vision Conference . Surrey : BMVA Press , 2012 : 1 - 11 .

ASHRAF A B , LUCEY S , CHEN T . Learning patch correspondences for improved viewpoint invariant face recognition [C]// 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition . Salt Lake City : IEEE , 2008 : 1 - 8 .

BEYMER D , POGGIO T . Face recognition from one example view [C]// Procedings of the Fifth International Conference on Computer Vision . Cambridge : IEEE , 1995 : 500 - 507 .

HASSNER T , HAREL S , PAZ E , et al . Effective face frontalization in unconstrained images [C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition . Boston : IEEE , 2015 : 4295 - 4304 .

BLANZ V , VETTER T . A morphable model for the synthesis of 3D faces [C]// Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques . Los Angeles : ACM , 1999 : 187 - 194 .

ROMDHANI S , VETTER T . Estimating 3D shape and texture using pixel intensity, edges, specular highlights, texture constraints and a prior [C]// 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition . San Diego : IEEE , 2005 : 986 - 993 .

YIM J , JUNG H , YOO B I , et al . Rotating your face using multi-task deep neural network [C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition . Boston : IEEE , 2015 : 676 - 684 .

ZHANG Y , SHAO M , WONG E K , et al . Random faces guided sparse many-to-one encoder for pose-invariant face recognition [C]// 2013 IEEE International Conference on Computer Vision . Sydney : IEEE , 2013 : 2416 - 2423 .

KAN M , SHAN S , CHANG H , et al . Stacked progressive auto-encoders (SPAE) for face recognition across poses [C]// 2014 IEEE Conference on Computer Vision and Pattern Recognition . Columbus : IEEE , 2014 : 1883 - 1890 .

HUANG R , ZHANG S , LI T , et al . Beyond face rotation: global and local perception GAN for photorealistic and identity preserving frontal view synthesis [C]// 2017 IEEE International Conference on Computer Vision . Venice : IEEE , 2017 : 2458 - 2467 .

ZHAO J , CHENG Y , XU Y , et al . Towards Pose Invariant Face Recognition in the Wild [C]// 2018 IEEE Conference on Computer Vision and Pattern Recognition . Salt Lake City : IEEE , 2018 : 2207 - 2216 .

QIAN Y , DENG W , HU J . Unsupervised face normalization with extreme pose and expression in the wild [C]// 2019 IEEE Conference on Computer Vision and Pattern Recognition . Long Beach : IEEE , 2019 : 9851 - 9858 .

YIN Y , JIANG S , ROBINSON J P , et al . Dual-attention GAN for large-pose face frontalization [C]// 15th IEEE International Conference on Automatic Face and Gesture Recognition . Buenos Aires : IEEE , 2020 : 249 - 256 .

HU Y , WU X , YU B , et al . Pose-guided photorealistic face rotation [C]// 2018 IEEE Conference on Computer Vision and Pattern Recognition . Salt Lake City : IEEE , 2018 : 8398 - 8406 .

CAO K , RONG Y , LI C , et al . Pose-robust face recognition via deep residual equivariant mapping [C]// 2018 IEEE Conference on Computer Vision and Pattern Recognition . Salt Lake City : IEEE , 2018 : 5187 - 5196 .

DENG J , GUO J , ZHOU Y , et al . Retinaface: Single-stage dense face localisation in the wild [EB/OL]. ( 2019 )[2021]. https://arxiv.org/abs/1905.00641 https://arxiv.org/abs/1905.00641 .

RADFORD A , METZ L , CHINTALA S . Unsupervised representation learning with deep convolutional generative adversarial networks [C]// 4th International Conference on Learning Representations . Piscataway : IEEE , 2016 : 31 - 38 .

JOHNSON J , ALAHI A , LI F F . Perceptual losses for real-time style transfer and super-resolution [C]// European Conference on Computer Vision . Amsterdam : Springer , 2016 : 694 - 711 .

CHEN S , LIU Y , GAO X , et al . Mobilefacenets: efficient cnns for accurate real-time face verification on mobile devices [C]// Chinese Conference on Biometric Recognition . Urumqi : Springer , 2018 : 428 - 438 .

LI P , WU X , HU Y , et al . M 2 fpa: a multi-yaw multi-pitch high-quality dataset and benchmark for facial pose analysis [C]// 2019 IEEE/CVF International Conference on Computer Vision . Seoul : IEEE , 2019 : 10042 - 10050 .

GUOY , ZHANG L , HU Y , et al . Ms-celeb-1m: a dataset and benchmark for large-scale face recognition [C]// European Conference on Computer Vision . Amsterdam : Springer , 2016 : 87 - 102 .

KINGMA D P , BA J . Adam: a method for stochastic optimization [C]// 3rd International Conference on Learning Representations . San Diego : IEEE , 2015 : 1 - 8 .

WANG G , MA J , ZHANG Q , et al . Pseudo facial generation with extreme poses for face recognition [C]// 2021 IEEE Conference on Computer Vision and Pattern Recognition . Virtual : IEEE , 2021 : 1994 - 2003 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于注意力机制优化的生成对抗网络及其在海杂波模拟中的应用

基于层次化一致性语义学习的多模态意图识别

天波超视距雷达地海杂波图像增强与检测器设计

DRE-3DC: 基于三维表征建模的篇章级关系抽取模型