Model and Dictionary guided Face In-painting in the Wild Reuben Farrugia (University of Malta) and Christine Guillemot (INRIA Rennes Bretagne) Proposed Face In - painting Algorithm Facial occlusion is a critical issue in image editing and forensic scenarios – and can be used to improve face recognition of occluded and non-frontal faces. This work presents a method that can be used to in-paint occluded facial regions with unconstrained pose and orientation. This approach first warps the facial region onto a reference model to synthesize a frontal view (a.k.a. Frontalization) A modified Robust Principal Component Analysis (RPCA) approach is then used to suppress warping errors. It then uses a novel local patch-based in-painting algorithm which hallucinates missing pixels using a dictionary of face images which are pre-aligned to the same reference model. The hallucinated region is then warped back onto the original image to restore occluded pixels NTIRE 2016 in conjunction with ACCV 2016 Face Frontalization The face image is first warped using affine transformation. The warped image is then stacked with a set of undistorted frontally aligned training face images to form the augmented matrix M a . This matrix is then decomposed using the following low-rank optimization problem Figure: Schematic diagram of the proposed face In-painting in the Wild method minimize ∗ + 1 subject to + . = The first column of A corresponds to the de-noised frontal face image which removes the warping errors Figure: The first column from each ell snows the cropped image from the LFW dataset. The second column represents the piecewise affine warped face image and the third column were restored using our modified RPCA method. Face In - Painting The proposed face in-painting method is based on the classical approach of Criminisi. However, instead of using a dictionary of neighboring patches we use collocated patches from a dictionary of aligned face images. The weights w* to reconstruct the unknown part of a patch are computed by solving the following optimization problem. ∗ = argmin ∗ − ∗ σ = ∗ = The unknown part of each patch is then reconstructed using = ∗ Figure: Schematic diagram of the local face in-painting of frontalized face images. Experimental Results In this experiment we used 84 images from the AR dataset witch were occluded using the following masks to mimic real-world scenarios. Experimental results demonstrate a significant gain in performance over existing face in-painting algorithms.11 The proposed method was evaluated on real-world images at different orientations, resolutions and quality including CCTV quality images taken during riots in UK. Figure: Original image (left), the original image with the region to ebe concealed marked in green (center-left), the inpainted image of the frontal region (center-right), and the final inpainted face using off the shelf inpainting method..

Reuben Farrugia (University of Malta) and Christine Guillemot …staff.um.edu.mt/__data/assets/file/0005/293207/accv2016_poster_A… · Reuben Farrugia (University of Malta) and Christine

Download PDF Report

Upload
others
View
2
Download
0

Embed Size (px)

Citation preview

Page 1: Reuben Farrugia (University of Malta) and Christine Guillemot …staff.um.edu.mt/__data/assets/file/0005/293207/accv2016_poster_A… · Reuben Farrugia (University of Malta) and Christine

Model and Dictionary guided Face In-painting in the WildReuben Farrugia (University of Malta) and Christine Guillemot (INRIA Rennes Bretagne)

Proposed Face In-painting Algorithm

Facial occlusion is a critical issue in image editing and forensicscenarios – and can be used to improve face recognition ofoccluded and non-frontal faces.

This work presents a method that can be used to in-paint occludedfacial regions with unconstrained pose and orientation.

This approach first warps the facial region onto a reference modelto synthesize a frontal view (a.k.a. Frontalization)

A modified Robust Principal Component Analysis (RPCA)approach is then used to suppress warping errors.

It then uses a novel local patch-based in-painting algorithm whichhallucinates missing pixels using a dictionary of face images whichare pre-aligned to the same reference model.

The hallucinated region is then warped back onto the originalimage to restore occluded pixels

NTIRE 2016in conjunction with ACCV 2016

Face Frontalization

The face image is first warped using affine transformation. Thewarped image is then stacked with a set of undistorted frontallyaligned training face images to form the augmented matrix Ma.This matrix is then decomposed using the following low-rankoptimization problem

Figure: Schematic diagram of the proposed face In-painting in theWild method

minimize 𝑨 ∗ + 𝜇 𝑬 1 subject to 𝑨 + 𝐄.= 𝑀𝒂

The first column of A corresponds to the de-noised frontal faceimage which removes the warping errors

Figure: The first column from each ell snows the cropped imagefrom the LFW dataset. The second column represents thepiecewise affine warped face image and the third column wererestored using our modified RPCA method.

Face In-Painting

The proposed face in-painting method is based on the classicalapproach of Criminisi. However, instead of using a dictionary ofneighboring patches we use collocated patches from a dictionary ofaligned face images.

The weights w* to reconstruct the unknown part of a patch arecomputed by solving the following optimization problem.

𝒘∗ = argmin𝒘∗ 𝝋𝒑𝒌 − 𝑫𝒑

𝒌 𝒔 𝒘∗𝟐

𝟐𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 σ𝒊=𝟏

𝑲 𝒘∗ = 𝟏

The unknown part of each patch is then reconstructed using

𝝋𝒑𝒖 = 𝑫𝒑

𝒖 𝒔 𝒘∗

Figure: Schematic diagram of the local face in-painting offrontalized face images.

Experimental Results

In this experiment we used 84 images from the AR dataset witchwere occluded using the following masks to mimic real-worldscenarios.

Experimental results demonstrate a significant gain in performanceover existing face in-painting algorithms.11

The proposed method was evaluated on real-world images atdifferent orientations, resolutions and quality including CCTVquality images taken during riots in UK.

Figure: Original image (left), the original image with the region toebe concealed marked in green (center-left), the inpainted imageof the frontal region (center-right), and the final inpainted faceusing off the shelf inpainting method..