Cosmetic-Aware Makeup Cleanser

Yi Li, Huaibo Huang, Junchi Yu, Ran He, Tieniu TanCRIPAC, National Laboratory of Pattern Recognition, CASIA

Center for Excellence in Brain Science and Intelligence Technology, CASUniversity of Chinese Academy of Sciences

{yi.li,huaibo.huang}@cripac.ia.ac.cn, rhe@nlpr.ia.ac.cn

Abstract

Face verification aims at determining whether a pair offace images belongs to the same identity. Recent studieshave revealed the negative impact of facial makeup on theverification performance. With the rapid development ofdeep generative models, this paper proposes a semantic-aware makeup cleanser (SAMC) to remove facial makeupunder different poses and expressions and achieve verifi-cation via generation. The intuition lies in the fact thatmakeup is a combined effect of multiple cosmetics and tai-lored treatments should be imposed on different cosmeticregions. To this end, we present both unsupervised and su-pervised semantic-aware learning strategies in SAMC. Atimage level, an unsupervised attention module is jointlylearned with the generator to locate cosmetic regions andestimate the degree. At feature level, we resort to the effortof face parsing merely in training phase and design a local-ized texture loss to serve complements and pursue superiorsynthetic quality. The experimental results on four makeup-related datasets verify that SAMC not only produces ap-pealing de-makeup outputs at a resolution of 256×256, butalso facilitates makeup-invariant face verification throughimage generation.

1. IntroductionAs one of the most representative regions of a human be-

ing, face acts as a vital role in biometrics. The appearanceof a face is often deemed as a crucial feature for identify-ing individuals. Most face verification methods determinewhether a pair of face images refers to the same person bycomparing certain facial features from the given images.Following the remarkable progress of face verification re-search [26, 27, 24, 13, 30, 10, 29], the related technologybrings great convenience to our lives ranging from socialmedia to security services. Nevertheless, it is worth notic-

978-1-7281-1522-1/19/$31.00 c©2019 IEEE

Figure 1. Conceptual illustration of verification via generation.The makeup problem is quite challenging since it often jointlyoccurrs with pose and expression variations. A semantic-awaremakeup cleanser is proposed to remove makeup while keep otherfactors unchanged .

ing that most of these application scenarios are concernedin information security and the importance of algorithm re-liability is self-evident. Facing various special situations,there is still a long way to solve the problem thoroughly.

Although face is an inherent surface area of a person,its appearance on images may alter due to many factors,e.g., view angles, expressions and makeup changes. Amongthese factors, the makeup change has caught more attentionthan ever because of the striking development of facial cos-metics and virtual face beautification. Makeup is usuallyused to conceal flaws on the face, enhance attractivenessor alter appearance. Not only females, there are more andmore males that agree makeup being a daily necessity andeven a courtesy on certain occasions. Comparing to viewangle and expression changes that are mostly out of unco-operative behaviors, makeup is a special case that may beinevitable during photographing. However, the appearancechanges caused by makeup will also decrease the verifica-tion performance, just like other factors. This paper studiesto remove facial makeup even under cases with pose andexpression variations.

A typical facial makeup style can be divided into three

arX

iv:2

004.

0914

7v1

[cs

.CV

] 2

0 A

pr 2

020

main parts according to their locations: eye shadows, lip-stick and foundation [3, 16]. The comprehensive effectsof various cosmetics can lead to significant changes on fa-cial appearance. Since there are theoretically innumerablekinds of makeup styles, the nondeterminacy and variabil-ity make it quite difficult to match the images before andafter makeup. Besides, makeup is different from plasticsurgery for it concerns non-permanent changes and is easyto be cleansed. In contrast to skin care products, cosmeticsare regarded as a harm to skin health and thus most peopleonly wear makeup when meeting others. The convenient fa-cial changes induced by makeup are reported in [5] to haveposed a severe impact on face verification systems (whichoften depend much on capturing various cues from facialappearance). In this paper, we study the effect of makeupand contrive to alleviate its impacts on face verification per-formance in a generative way.

To address makeup-invariant face verification, much ef-forts have been made in the past decade and we sum them upin two main streams. Early methods concentrate on exact-ing discriminative features directly from input images. Bymaximizing correlation between images of the same sub-jects, these methods map images into a latent space for bet-ter classification performance. The mapping function canbe designed manually [11, 7, 4] or learned by a deep net-work [25]. Another stream resorts to the success of deepgenerative models and uses the makeup removal outputs forverification, represented by [17, 18]. Given a face imagewith makeup, these methods first generate a non-makeupand identity-preserving image based on the input. Andthen the generated images along with real non-makeup im-ages are utilized for verification. Since this stream makeschanges at image level, it achieves the advantage of adapt-ing existing verification methods for makeup problems withno requirement of retraining.

However, [17, 18] formulate makeup removal as a one-to-one image translation problem, which in fact lacks ratio-nality. During training, there is a pair of makeup and non-makeup images of a subject, the networks in [17, 18] forcethe output to be like the non-makeup ground truth insteadof just removing facial makeup. We show a sample pair inFigure 1 and obvious data misalignment can be observed.Apart from makeup, many other factors are also differentin two images, including the hair style and the background.In this paper, we argue that makeup removal is naturally atask with unpaired data for it is illogical to obtain a pair ofimages with and without makeup simultaneously. When re-moving the makeup, other factors of an output are expectedto retain the same with its input, based on the premise ofrealistic visual quality.

To this end, we propose a Semantic-Aware MakeupCleanser (SAMC) to facilitate makeup-invariant face verifi-cation via generation. As mentioned above, facial makeup

is the results of applying multiple cosmetics and differentregions can appear different effects. Thus we raise the ideaof removing the cosmetics with tailored schemes. Con-cretely, we adopt two semantic-aware learning strategies inSAMC, in both unsupervised and supervised manners. Foran input image with makeup, we first utilize an attentionmodule to locate the cosmetics and estimate their degrees.The attention module is jointly learned with the generator inan unsupervised way, since there is no access to applicableattention maps serving as supervision. The aim of this pro-cess is to get explicit knowledge of makeup automatically aswell as adaptively. Although the attention module suggestswhere to stress, its confidence and accuracy are not as satis-fying as desired, especially at the beginning of the trainingphase. To address this issue, we propose a semantic-awaretexture loss to set a constraint on the synthesized texture ofdifferent cosmetic regions. In addition, the misalignmentproblem is overcome by a feat of face image warping. Bywarping the non-makeup ground truth according to its facekeypoints, we obtain pixel-wise alignment data, benefitingprecise supervision and favorable generation.

2. Approach

2.1. Motivation and Overview

In reality, makeup images are generally acquired in unre-stricted conditions, i.e., in the wild. Besides makeup, thereexist complex variations including pose, expression, illumi-nation. Although these factor may also impact verificationperformance and should be adjusted, the proposed SAMCfocuses on makeup removal but leaves out other factors forbetter user experience. When a customer is using a makeupremoval system, the changes are expected to be limited tomakeup. Therefore, given an image with makeup as the in-put, our method aims at removing the makeup in a genera-tive manner while retaining other information. The output isexpected to achieve appealing visualization as well as suf-fice face verification. Different from image style transfer,the effect of cosmetics merely involves certain face areasinstead of the global image. Moreover, despite the inacces-sibility of exactly aligned data, we can acquire image pairsof a subject, one with makeup and the other without. Withproper processes and strategies, these image pairs can pro-vide superior supervision.

To fully utilize the image pairs without exact alignment,we propose a semantic-aware makeup cleanser to removefacial cosmetics with tailored strategies. The simple yet ef-fective method is implemented by a network that mainlycontains a generator G and an attention module A. We em-ploy image pairs {X1, Y1}, {X2, Y2} · · · to train the net-work and assume that X ∈ X and Y ∈ Y represent imageswith and without makeup. It is noting that {Xi, Yi} refer tothe same identity but may differ in expression, pose, back-

Figure 2. The network diagram of the Semantic-Aware Makeup Cleanser (SAMC). X and Y are a pair of images with and without makeupbelonging to the same subject. Obvious misalignment can be observed. Instead of directly using Y as the ground truth, we first warp Yaccording to X and obtain W for better supervision. Then an unsupervised attention module is learned to locate the makeup and estimatethe degree. The attention map A indicates where and how much should be changed. Finally, we propose a SAT loss to constrain the localtexture of the generated result Z. The generator receives four types of losses in the full structure, while the attention module is merelylearned by the reconstruction loss.

ground, etc. To obtain pixel-wise supervision, we warp thenon-makeup ground truth Yi according to Xi, yielding thewarped non-makeup ground truth Wi. The warping processconsists two steps: 1) detecting 68 facial keypoints by [1],and 2) non-linear transformation in [22]. In the following,we will elaborate the network and loss functions in details.

2.2. Basic Model

We first describe the basic model of SAMC which is typ-ically a conditional generative adversarial network [6, 12]with an identity constraint, similar as [17]. The diagram ofthe basic model lies in the top right corner of Figure 2. Tak-ingX ∈ Rw×h×3 as the input, the basic model produces themakeup removal result Z ∈ Rw×h×3 though the generatorG. Different from [17] that forces Z to resemble Y , we ex-pect the changes to be limited in cosmetic areas while otherregions are kept the same as the input. Therefore, the U-netstructure [21, 12] is adopted in the generator G for its skipconnections help to maintain abundant context informationduring the forward pass. Instead of mapping to the outputZ directly, the network learns a residual result as a bridge,inspired by the success of ResNet [9]. The final output Z isobtained by adding the residual result to the input X .

The generator G receives two types of losses to updateparameters in the basic model, i.e., an adversarial loss andan identity loss. The vanilla GAN [6] uses the idea of atwo-player game tactfully to achieve the most promisingsynthetic quality. The two players are a generator and adiscriminator that compete with each other. The generator

aims at producing samples to full the discriminator, whilethe discriminator endevours to tell real and fake data apart.To refrain from artifacts and blurs in the output, we traina discriminator to serve the adversarial loss which can beformulated as

Ladv = EY [logD(Y )] + EX,Y [log(1−D(Z))]. (1)

In addition to removing makeup, we also expect that thegenerated images can maintain the identity of the originalimage, contributing to improve the verification performanceacross makeup status. Different from visual quality, verifi-cation performance is calculated by comparing image fea-tures extracted by tailored schemes. For face verification,the key issue is to generate images with qualified featuresthat indicates identity. Thus we also use the identity loss tokeep the identity information consistent. The identity lossis used as a classical constraint in a wide range of applica-tions, e.g., super-resolution [14], face frontalization [2], ageestimation [15], and makeup removal [17]. Similar to [17],we employ one of the leading face verification networks,i.e. Light CNN [29], to obtain the identity information ofan face image. The ID loss is calculated by

LID = EY,Z‖F (Z)− F (Y )‖2 (2)

where F (·) represents the pre-trained feature extractor.Noting that the parameters in F (·) stay constant duringtraining.

2.3. Attention Module

By comparing images before and after makeup of a cer-tain subject, we observe that the facial changes caused bymakeup only concern several typical regions, like eye shad-ows and lips. Existing makeup removal methods like [17]treat the problem as a one-to-one image translation task andforce the output to resemble the non-makeup image in thedataset. This behavior violates the observation above andthus is not capable of generating satisfying results. Instead,when removing makeup, we should concentrate on thesecosmetic regions and leave other unrelated image areas out.In this paper, an attention module is developed to enhancethe basic model by distinguishing where and how much arethe cosmetics in a pixel-wise way.

On the other hand, the attention map can also contributeto ignore the severe distortion in the warped non-makeupimage W as shown in Figure 2. The aim of the warpingprocess is to alleviate data misalignment and provide bet-ter supervision. However, the warping is based on match-ing the facial keypoints on two images and the distortionis somewhat inevitable after the non-linear transformation.If provided with distortion-polluted supervision, the gen-erator may be misguided and produce results with falla-cious artifacts and distortion. To mitigate this risk, weadopt the attention map to adaptively decide where to be-lieve in the warped image. We present a learned attentionmap A ∈ Rw×h in Figure 2. The element in A is normal-ized between 0 and 1. In Figure 2, we adjust the colourof A for better visualization. It can be concluded that themakeup focuses on eye shadows and lips, in accord withcommon sense. In general, there are dark and light pixels inA. The dark ones indicate that the makeup removal resultat this pixel should be like W , and the light pixel should belike X . In this way, the distortion pixels are successfullyneglected owing to their weights close to 0. We formulatethe intuition as the reconstruction loss, which is calculatedby

Lrec = |A⊗W−A⊗Z|+|(1−A)⊗X−(1−A)⊗Z| (3)

where ⊗ represents element-wise multiplication betweenmatrices.

Nevertheless, there is another problem that the attentionmap is easy to converge to all 0 or all 1. When the attentionmap is all 0, it means that Z is driven to be the reconstruc-tion of A. If the attention map is all 1, Z is induced toimitate W . Neither case is expected to occur. To avert thesecases, [19] introduces a threshold to control the learning ef-fect of the attention map. However, it is difficult to choosean appropriate value for the threshold. Different from it,we employ a simple regularization to address the problem,which is implemented by the additional L1 norm constraintbetween the two terms in Equation 3. This regularizationprevents the two terms from being too large or too small,

Figure 3. A face parsing sample. Taking (a) as the input, [28] pro-duces the face parsing result in (b). We make further modificationand obtain (c), for calculating SAT loss.

and thus restricts the value in the attention map consequen-tially. A balanced weight is introduced to control the con-tribution of the regularization and we empirically set it as0.2.

2.4. Semantic-Aware Texture Loss

The attention map is learned along with the generatorto indicate makeup at image level. Considering that it islearned in an unsupervised manner, the confidence and ac-curacy cannot be ensured, especially at the beginning of thetraining process. Hence, we explore other supervision topursue better generation quality and stable network train-ing. To this end, a Semantic-Aware Texture (SAT) loss isproposed to make the synthesized texture of different cos-metic regions realistic. As has been analyzed, makeup issubstantially a combination of cosmetics applied to multi-ple facial regions. A typical makeup effect can be dividedinto foundation, eye shadows and lipsticks. Based on these,we resort to the progress of face parsing [28] and furtheradapt the parsing results to obtain different cosmetic re-gions. Figure 3 presents a set of parsing results. There arethree colours in Figure 3(c), each standing for a cosmeticregion.

The aim of the SAT loss is to resemble the local textureof Z to that of Y . After acquiring the cosmetic region labelmap of X and Y , the mean µ and standard deviation σ ofeach feature region is calculated accordingly. Noting that 1)we assume that Z shares the label map with X for their ap-pearance merely differs in makeup effects, and 2) the labelmap has been resized according to the corresponding fea-ture map. As for the texture extractor, we continue to adoptF (·) but only use the output of the second convolutionallayer. Finally, the SAT loss is defined as

Lsat =∑i

‖µZi − µY

i ‖2 + ‖σZi − σY

i ‖2 (4)

where µ∗i and σ∗

i represent the mean and standard deviationof ∗with i = {1, 2, 3} indicating the three cosmetic regions.

In total, the generator updates its parameters based onthe elaborated four losses and the full objective is

LG , Ladv + LID + Lrec + Lsat. (5)

Figure 4. Visualization of makeup removal and the attention mapobtained by SAMC. The best results are generated by our methodon the right. SAMC sucessfully cleanses the facial cosmetics, es-pecially in regions like eye shadows and lips. For better visualeffects, the attention map is processed by reducing brightness andthe black pixels locate the cosmetics successfully.

3. Experiments

3.1. Datasets and Training Details

We use three public datasets to build our training andtest sets. Dataset1 is collected in [7] and contains 501identities. For each identity, a makeup image and a non-makeup image are included. Dataset2 is assembled in [25]and contains 406 images. Similarly, each identity includes amakeup image and a non-makeup image. Dataset3 (FAM)[11] consists of 519 pairs of images. For fair comparisonwith other methods, we resize all the images to the resolu-tion of 128 × 128 in our experiments. We also employ thehigh-quality makeup data collected by our lab to train andtest the network. We refer to it as the Cross-Makeup Face(CMF) dataset. There are 2600+ image pairs at high res-olution (at least 256 × 256) of 1400+ identities, involvingboth makeup variations and identity information.

The experiments involve images with two resolutions

Table 1. The verification results(%) on the CMF dataset.

Method Rank-1TPR@FPR

=0.1%TPR@FPR

=1%baseline 91.92 63.29 86.16

Pix2pix [12] 76.54 36.24 69.34CycleGAN [31] 88.65 57.00 82.78

BLAN [17] 91.03 55.62 85.29ours 94.04 77.36 93.55

ours \ ID 90.58 63.44 86.40ours \ SAT 91.54 65.96 87.50ours \ adv 91.35 65.64 87.42ours-att 0 91.15 66.27 87.42ours-att 1 91.73 64.86 86.95

(128 × 128 and 256 × 256). Our network is implementedbased on pytorch. We train SAMC on the CMF dataset andtest it on all the four datasets mentioned above. Therefore,the experiments on Dataset 1–3 are conducted in a cross-dataset setting. There are two main subnetworks in SAMC,i.e., the generator G and the attention module A. In theimplementation, G and A share the same architecture butupdate parameters separately. As for the feature extractorF (·) in Eq. 2 and 4, we utilize the released model of LightCNN which is trained on MS-Celeb-1M [8]. The batch sizeis set to 8. The model is trained using the Adam algorithmwith a learning rate of 0.0001. We set the balanced weightsof all the losses as 1 without loss of generality.

3.2. Visualization of Makeup Removal

Figure 4 presents some sample results of makeup re-moval, along with the learned attention of SAMC. Notingthat we modify the display effect of attention maps for bet-ter visualization. We compare our results with other meth-ods, including Pix2pix [12], CycleGAN [31] and BLAN[17]. Pix2pix and CycleGAN are widely considered as rep-resentative methods in supervised and unsupervised image-to-image translation, respectively. To the best of our knowl-edge, BLAN firstly propose to generate non-makeup im-ages from makeup ones for makeup-invariant face verifica-tion. We train these networks from scratch on CMF trainingdata and the configurations are kept the same as describedin their papers.

In Figure 4, it can be observed that Pix2pix and BLANgenerates images with severe artifacts and distortion. Thereason lies in that these methods assume the existence ofwell aligned data pairs and formulate the image translationproblem at pixel level. As has been analyzed, makeup im-ages inherently lack paired data, making the problem moredifficult to tackle with. Although CycleGAN is learnedin an unsupervised manner and produces images of higherquality than its neighbours, there exist apparent cosmetic

Table 2. Rank-1 accuracy (%) on three released makeup datasets.Method [7] [25] [20] [11] VGG [23] LightCNN [29] BLAN [17] ours

Dataset 1 80.5 82.4 - - 89.4 92.4 94.8 96.1Dataset 2 - 68.0 - - 86.0 91.5 92.3 96.7

FAM - - 59.6 62.4 81.6 86.3 88.1 94.8

residues due to the lack of proper supervision. As for theattention map, it demonstrates that our attention modulecan locate the cosmetics. And we will discuss the contri-bution of the attention module in the ablation studies. Com-paring with other methods, our network achieves the mostpromising results. Not only are the cosmetics successfullyremoved, but also other image details are well preserved.

3.3. Makeup-Invariant Face Verification

In this paper, we propose a makeup removal method inthe aim of facilitating makeup-invariant face verification viageneration. To quantitatively evaluate the performance ofour makeup cleanser, we show the verification results ofSAMC and related algorithms in Table 1. As introducedabove, we adopt the released Light CNN model as the fea-ture extractor. Concretely, each image goes through theLight CNN and becomes a 256-d feature vector. The simi-larity metric used in all experiments is cosine distance. Forthe baseline, we use the original images with and withoutmakeup as inputs. For other methods, we instead use themakeup removal results and the original non-makeup im-ages as inputs for verification.

For the CMF dataset, we observe that our approachbrings significant improvements on all the three criteria.Instead of forcing the output to resemble the ground truthnon-makeup image in the dataset like BLAN, we learn tolocate and remove the cosmetics while maintaining other in-formation including pose and expression. The accuracy im-provements demonstrate that our network alleviates the sideeffects of makeup on face verification by generating high-quality images. On the other hand, CycleGAN fails to pre-serve identity information during generation, even thoughit produces outputs with moderate quality. The reason isthat Pix2pix and CycleGAN are designed for general imagetranslation and take no discrimination into account. For faircomparison, we further conduct experiments on three pub-lic makeup datasets and the results are exhibited in Table 2.It is worth noticing that BLAN is trained on these datasetswhile our SAMC is trained on CMF and tested on thesewithout adaptation. Thanks to the stability of our network,SAMC can still outperform other methods in cross-datasetsettings.

Besides, ablation studies are conducted to evaluate thecontribution of each component in SAMC. We remove ormodify one of the components and concern the changes inthe corresponding metrics to verify their importance. To

study the used loss functions to train the generator, we buildthree variants: 1) training without the ID loss, 2) trainingwithout the SAT loss, and 3) training without the adversar-ial loss. Since the reconstruction loss involves the learnedattention map, we utilize different attention schemes to an-alyze the effect of the attention module. In particular, theattention map is set to all 0 and all 1, respectively. The quan-titative verification results are reported in Table 1 for com-prehensive and fair comparison. The “\” and the “w\o”represent “without”. As expected, the performance of re-moving either component will experience a drop. By ob-serving the accuracies in Table 1, we can find that there isan apparent decline when removing the ID loss. It indicatesthe effectiveness and importance of the ID loss in preserv-ing discriminative information at feature level.

4. ConclusionIn this paper, we focus on the negative impact of fa-

cial makeup on verification and propose a semantic-awaremakeup cleanser (SAMC) to remove cosmetics. Insteadof considering makeup as an overall effect, we argue thatmakeup is the combination of various cosmetics applied todifferent facial regions. Therefore, a makeup cleanser net-work is designed with integration of two elaborate schemes.At image level, an attention module is learned along withthe generator to locate the cosmetics in an unsupervisedmanner. Specifically, the elements in the attention maprange from 0 to 1 with different values indicating themakeup degree. At feature level, a semantic-aware textureloss is designed to serve complements and provide supervi-sion. Experiments are conducted on four makeup datasets.Both appealing makeup removal images and promisingmakeup-invariant face verification accuracies are achieved,verifying the effectiveness of SAMC.

5. AcknowledgmentsThis work is partially funded by National Natural Sci-

ence Foundation of China (Grant No. 61622310), BeijingNatural Science Foundation (Grant No. JQ18017), andYouth Innovation Promotion Association CAS (Grant No.2015109).

References[1] A. Bulat and G. Tzimiropoulos. How far are we from solv-

ing the 2d & 3d face alignment problem?(and a dataset of

230,000 3d facial landmarks). In Proceedings of the IEEEInternational Conference on Computer Vision, pages 1021–1030, 2017. 3

[2] J. Cao, Y. Hu, H. Zhang, R. He, and Z. Sun. Learning a highfidelity pose invariant model for high-resolution face frontal-ization. In Advances in Neural Information Processing Sys-tems, pages 2872–2882, 2018. 3

[3] H. Chang, J. Lu, F. Yu, and A. Finkelstein. Pairedcycle-gan: Asymmetric style transfer for applying and removingmakeup. In IEEE Conference on Computer Vision and Pat-tern Recognition, 2018. 2

[4] C. Chen, A. Dantcheva, and A. Ross. An ensemble of patch-based subspaces for makeup-robust face recognition. Infor-mation Fusion, 32:80–92, 2016. 2

[5] A. Dantcheva, C. Chen, and A. Ross. Can facial cosmeticsaffect the matching accuracy of face recognition systems?In the Fifth International Conference on Biometrics: Theory,Applications and Systems, pages 391–398. IEEE, 2012. 2

[6] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu,D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Gen-erative adversarial nets. In Advances in neural informationprocessing systems, pages 2672–2680, 2014. 3

[7] G. Guo, L. Wen, and S. Yan. Face authentication withmakeup changes. IEEE Transactions on Circuits and Sys-tems for Video Technology, 24(5):814–825, 2014. 2, 5, 6

[8] Y. Guo, L. Zhang, Y. Hu, X. He, and J. Gao. Ms-celeb-1m:A dataset and benchmark for large-scale face recognition. InEuropean Conference on Computer Vision, pages 87–102.Springer, 2016. 5

[9] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learningfor image recognition. In the IEEE conference on computervision and pattern recognition, pages 770–778, 2016. 3

[10] R. He, X. Wu, Z. Sun, and T. Tan. Learning invari-ant deep representation for nir-vis face recognition. InThe Thirty-First AAAI Conference on Artificial Intelligence,pages 2000–2006. AAAI Press, 2017. 1

[11] J. Hu, Y. Ge, J. Lu, and X. Feng. Makeup-robust face ver-ification. In International Conference on Acoustics, Speechand Signal Processing, pages 2342–2346, 2013. 2, 5, 6

[12] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. InIEEE Conference on Computer Vision and Pattern Recogni-tion, pages 5967–5976. IEEE, 2017. 3, 5

[13] X.-Y. Jing, F. Wu, X. Zhu, X. Dong, F. Ma, and Z. Li.Multi-spectral low-rank structured dictionary learning forface recognition. Pattern Recognition, 59:14–25, 2016. 1

[14] J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses forreal-time style transfer and super-resolution. In EuropeanConference on Computer Vision, pages 694–711. Springer,2016. 3

[15] P. Li, Y. Hu, Q. Li, R. He, and Z. Sun. Global and local con-sistent age generative adversarial networks. arXiv preprintarXiv:1801.08390, 2018. 3

[16] T. Li, R. Qian, C. Dong, S. Liu, Q. Yan, W. Zhu, and L. Lin.Beautygan: Instance-level facial makeup transfer with deepgenerative adversarial network. In 2018 ACM Multime-dia Conference on Multimedia Conference, pages 645–653.ACM, 2018. 2

[17] Y. Li, L. Song, X. Wu, R. He, and T. Tan. Anti-makeup:Learning a bi-level adversarial network for makeup-invariantface verification. In The Thirty-Second AAAI Conference onArtificial Intelligence, 2018. 2, 3, 4, 5, 6

[18] Y. Li, L. Song, X. Wu, R. He, and T. Tan. Learning a bi-level adversarial network with global and local perceptionfor makeup-invariant face verification. Pattern Recognition,2019. 2

[19] Y. A. Mejjati, C. Richardt, J. Tompkin, D. Cosker, and K. I.Kim. Unsupervised attention-guided image to image trans-lation. In Thirty-second Conference on Neural InformationProcessing Systems, 2018. 4

[20] H. V. Nguyen and L. Bai. Cosine similarity metric learn-ing for face verification. In Asian Conference on ComputerVision, pages 709–720. Springer, 2010. 6

[21] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolu-tional networks for biomedical image segmentation. In In-ternational Conference on Medical Image Computing andComputer-Assisted Intervention, pages 234–241. Springer,2015. 3

[22] D. Ruprecht and H. Muller. Image warping with scattereddata interpolation. IEEE Computer Graphics and Applica-tions, 15(2):37–43, 1995. 3

[23] K. Simonyan and A. Zisserman. Very deep convolutionalnetworks for large-scale image recognition. In Proceedingsof the 3rd International Conference on Learning Represen-tations, 2015. 6

[24] Y. Sun, Y. Chen, X. Wang, and X. Tang. Deep learningface representation by joint identification-verification. InAdvances in neural information processing systems, pages1988–1996, 2014. 1

[25] Y. Sun, L. Ren, Z. Wei, B. Liu, Y. Zhai, and S. Liu. Aweakly supervised method for makeup-invariant face veri-fication. Pattern Recognition, 66:153–159, 2017. 2, 5, 6

[26] Y. Sun, X. Wang, and X. Tang. Hybrid deep learning forface verification. In the IEEE International Conference onComputer Vision, pages 1489–1496, 2013. 1

[27] Y. Taigman, M. Yang, M. Ranzato, and L. Wolf. Deepface:Closing the gap to human-level performance in face verifica-tion. In the IEEE Conference on Computer Vision and Pat-tern Recognition, pages 1701–1708, 2014. 1

[28] Z. Wei, Y. Sun, J. Wang, H. Lai, and S. Liu. Learning adap-tive receptive fields for deep image parsing network. In Pro-ceedings of the IEEE Conference on Computer Vision andPattern Recognition, pages 2434–2442, 2017. 4

[29] X. Wu, R. He, Z. Sun, and T. Tan. A light cnn for deep facerepresentation with noisy labels. IEEE Transactions on In-formation Forensics and Security, 13(11):2884–2896, 2018.1, 3, 6

[30] S. Zhang, R. He, Z. Sun, and T. Tan. Multi-task convnet forblind face inpainting with application to face verification. InInternational Conference on Biometrics, pages 1–8, 2016. 1

[31] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial net-works. In IEEE International Conference on Computer Vi-sion, 2017. 5