S7504: Improving Consumer Compliance

Through Better Product Recommendation

New Skin Advisor Tool Powered by AI

Jun Xu PhD, Faiz Sherman PhD, Matthew Barker PhD*

Frauke Neuser PhD, Shannon Weitz

The Procter & Gamble Company

THE SCIENCE BEHINDOLAY SKIN ADVISORskinadvisor.olay.com

5 of the top 10

Non-Food Product Launches

P&G scientists around the world work together to develop products that improve more

lives in more meaningful ways – now and for generations to come.

BRAND INNOVATION

BUSINESS INNOVATION

SOCIAL INNOVATION

Years of

innovation

history

50 100 150

179

40,000Active patents

worldwide

More than

1,000

Ph.Ds

In 120 scientific disciplines

$2 billion+Annual R&D investment

7,500R&D employees

Procter & Gamble (P&G) NYSE: PG

Global Research & Development

2016 IRI New Product Pacesetters

P&G 25+ Years of Industry Leading Skin Imaging

Background

• Consumers struggle to find the right cosmetic skin care products suited to their

personal needs and preferences. The ability to make an informed product selection

decision can drive product compliance and delight.

• A new skin advisor tool has been developed to deliver a personalized beauty

consultation tailored for consumers’ unique skin needs right at her fingertips.

• This tool combines deep learning with consumer preferences related to visible skin

concerns, cosmetic product use, and skin feel for the optimal product

recommendation.

Development Overview

Deep Learning AlgorithmVisible skin

age prediction with aging

area identification.

Visible Skin Age Validation

Predictions compared to expert.

Aging Area InsightsFacial Mapping Study

informs how appearance of aging areas change with chronological age.

Compliance VerificationProving skin advisor with

deep learning algorithm,

visible aging insights and consumer

preferences drives

compliance.

Facial Features & Aging

nasolabial folds

glabella

marionette lines

Deep Neural Network application• The skin advisor uses convolutional neural networks trained using NVIDIA

graphics processors to perform trillions of calculations per second. The

model was trained on 50,000 images with chronological age data tags.

• When an image of a user is received, the model is used to determine the

visible skin age based on the pixels in the image, further a two-

dimensional heat map is generated that identifies a region of the image

that contributes to the visible skin age.

Predicted Age

Raw Image Pixels

Data Setup

• Face detection & alignment performed using dlib: rotated, scaled &

cropped to a standard size.

• Spatial augmentation was applied: random horizontal flipping, rotation,

scaling, zoom cropping causing slight translation.

• HSV Color augmentation: random changes to saturation & exposure.

• Oval Mask, global contrast normalization GCN, reapply Oval Mask.

CNN using Torch

• Regime of learning rate as epochs increase

• Small batch size was utilized, using less memory, explorer more places to

find parameters estimates corresponding minimum

• 20+ layers: convolution, max pooling, leaky ReLU, decreasing spatial

dimension while increasing depth dimension. Dropout was also used in

later layers … consistent with Alexnet architecture with adaption

• Multi-threading to aid speed of decompressing JPG and send data to the

GPU, otherwise the GPU is starved

• RMSProp was used to optimize gradient descent

• Model training took roughly 8 hours on NVIDIA Titan X GPU

Gradient Heat Map for Visualization

• After training, with fixed model parameters. A gradient heat map was

created in order to localize pixel differences of a subject’s image relative

to younger than their predicted age.

• An input image was forward propagated through the model to obtain a

predicted age. Then a target of predicted age minus 10 years was set

and the gradients were propagated back through the network to the input

image. A heat map was created by summing absolute values of the RGB

gradients for each pixel and rescaling from 0 to 1 for display purposes.

• The gradient heat map was then blended with the original image to

visualize areas that were different from their younger predicted age.

Visible Skin Age Validation

Evaluate robustness of the visible skin age algorithm by comparing output to

a gold standard dermatologist assessment.

1. A validation set of 630 selfie images representing the general US female

population were obtained.

2. These images were presented to 615 dermatologists, who represent the

gold standard in visible skin evaluation, in a randomized order in sets of 8

images. Each dermatologist evaluated images.

3. The dermatologists were asked to input the perceived age of each image.

Validation ResultsThe mean difference of the predicted visible skin age versus the chronological age

using the skin advisor deep learning algorithm was comparable to the mean difference

of the perceived age versus the chronological age by dermatologists.

Me

an

Age

Diffe

rence

Deep

Learning

Algorithm

Dermatologists

• To build a fundamental understanding of the underlying mechanisms of facial aging

across different facial sites, a clinical Facial Mapping Study enrolling over 150

subjects

• Study assessed facial skin genomics, image analysis parameters, lifestyle factors,

and skin measurements in two groups of female subjects: a younger ages (20-29

years) and an older ages (55-75 years). Study did not assess applying cosmetics.

• Facial locations analyzed included the forehead, crow’s feet area, under eye,

nasolabial fold, cheek, glabella, marionette lines, above mouth, and nose regions.

Lifestyle Molecular Physical Optical Visual

Facial Area Insights – Mapping Study

Facial Mapping Study - Results

GB = GlabellaFH = ForeheadUE = Under EyeCF = Crow’s Feet

CK = CheekNL = Nasolabial FoldMN = MarionetteLP = Above Lip NS = Nose

• The Skin Advisor Tool shares the best aging area and the area that needs

improvement based on the deep learning algorithm. Key educational

information about how those areas age is also given.

• Insights from the facial mapping study were used to inform how visible aging

areas change with chronological age.

• Quantitative assessment of wrinkles revealed distinct visible topography feature

presentation across facial zones and with aging.

Ages 55-75 Ages 20-29

Compliance Verification

• 100 US women, age 25-65, facial moisturizer users, were

enrolled in a 4-week online consumer test.

• Group 1 (n=50) received a product regimen based on the skin

advisor deep learning algorithm and preferences and

Group 2 (n=50) self-selected a product regimen.

• Self-assessment questions were completed pre-use and post-

4 weeks product use.

Compliance Results

Figure 5

Post 4 weeks product use indicates satisfaction

with the skin advisor product recommendation

and improved consumer compliance with longer

product use.

Figure 4

Pre-product use indicates satisfaction

with the skin advisor product

recommendation.

Demo

Olay Skin Advisor – Website Results

• Over 1.4 million visits to the site

• High engagement rates

• Half the bounce ratio of a typical beautybrand.com website

• Twice the time spent vs. a typical beautybrand.com website

• Huge opportunity for real time consumer learnings

• 2.0 upgrade launched 6 weeks ago

Conclusion

Creating a tool that leverages a deep learning algorithm to predict

visible skin age and aging areas creates motivation to comply to a

cosmetic skin care regimen. Visible skin age and aging area

analysis is further backed by dermatologist validation and clinical

data to support a robust product recommendation. The new skin

advisor tool combines this technical information with consumer

preferences to recommended cosmetic products that provide

delight required for skin care regimen compliance.

Thank you! Questions?Barker.ML@pg.com