CUSTOMER EXPERIENCE IMPROVEMENT MADE POSSIBLE …2013.sentimentsymposium.com/presentations/SAS13-HanLai.pdf1. More than 80,000 English customer feedback left on PayPal.com pages (desktop

CUSTOMER EXPERIENCE IMPROVEMENT MADE POSSIBLE BY BASIC SENTIMENT ANALYSIS

Han Lai Director, Operational Excellence & Customer Advocacy MAY 8, 2013

Confidential and Proprietary

MY STORY TODAY

2

Confidential and Proprietary 3

OUTLINE

1. Context: What’s PayPal? … and a few numbers

2. Customer verbatim/text analytics approach at PayPal

3. Customer Experience Improvement

§  PayPal iPhone App crashes

§  Trend: Merchant Transaction Payment Hold

§  Sentiment on PayPal Fees

§  Password Recovery

4. Questions / General Discussion


PAYPAL FACTS… WITH NUMBERS

4



Basic Facts about PayPal .. §  128MM active accounts §  Supports payments in 25 currencies incl.: U.S. Dollar,

Australian Dollar, Canadian Dollar, Hong Kong Dollar, Singapore Dollar, Taiwan New Dollar, New Zealand Dollar, Euro, Swiss Franc, Czech Koruna, Swedish Krona, Danish Krone, Norwegian Krone, Hungarian Forint, Mexican Peso, Philippine Peso, Malaysian Ringgit, Chinese RMB, Israeli New Shekel, Pounds Sterling, Brazilian Real, Polish Zloty, Thai Baht, Turkish Lira and Japanese Yen.

§ HQ in San Jose, Calif., European HQ in Luxembourg, international HQ in Singapore.

§ Operation centers in Berlin; Chandler, Ariz; Dublin and Dundalk, Ireland; near Omaha, Neb.; Petaling Jaya, Malaysia; Sao Paulo, Brazil; Shanghai, China


PayPal expects to process $20 billion in mobile payment volume in 2013

$145B NET TOTAL PAYMENT VOLUME (TPV) in 2012




CUSTOMER FEEDBACK AT PAYPAL (2012)

1. More than 80,000 English customer feedback left on PayPal.com pages (desktop application)

2. More than 1.2 Million customer feedback (English over 2012 collected from post customer support contact surveys

3. Customer feedback left on apple.com on PayPal mobile app

4. More than 6.8 Million tweets mentioning eBay or PayPal


PAYPAL MOBILE APP CRASH

11

12 Confidential and Proprietary

WE EXTRACTED CUSTOMER FEEDBACK ON APP CRASHES FROM APPLE.COM SITE AND PROVIDED THE MOBILE TEAM WITH ACTIONABLE INSIGHTS THAT LED TO FIX (2011)

17

7

4

3.4 3.4.1 3.5

Num

ber o

f cus

tom

er v

erba

tim

men

tioni

ng P

ayP

al a

pp c

rash

es

Before Fix (07/2011)

PayPal iPhone app version number

After Fix (08/2011)

1: Tool: Tovek

Customer Feedback spidered from Apple.com site


MONITORED THE INDIVIDUAL CUSTOMER LEVEL EXPERIENCE BEFORE & AFTER FIX

1: Tool: Tovek


BEFORE FIX AFTER FIX

AFTER FIX Fix confirmation and

positive sentiment from the same customer


RECEIVED PAYMENT/MONEY HOLD


CUSTOMER FEEDBACK TREND ON PAYPAL MONEY HOLD POLICY

1: based on existing text mining approach & tool: Tovek

Proportion of customer complaint on csNPS went from 0.54% to 1.37%, a 2.5X increase over a year

Overall upward trend over 12 months (from three VOC sources )


SENTIMENT ON PAYPAL FEES

16


PRICING SENTIMENT

-20%

-10%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Jan-12 Feb-12 Mar-12 Apr-12 May-12 Jun-12 Jul-12 Aug-12 Sep-12 Oct-12 Nov-12 Dec-12

NPS

Sco

re

pNPS in Comparison to csNPS and rNPS

pNPS csNPS rNPS

Pricing sentiment

Overall Sentiment on PayPal


PRICING SENTIMENT

+/-‐ Global Query for Pricing (source: OthercsNPS only) No. Verbatim %

Detractor

(( (fee,charge,price) <near/3> (high,too much,expensive))) <in> (h1, h2, acronym, au) and ((country <contains> us) or (country <contains> ca) or (country <contains> gb) or (country <contains> au) or (country <contains> hk) or (country <contains> sg)) and NpsDetrPasiPromTxt <near/2> detractor and (date >= 2010-‐10-‐01 and date < 2012-‐11-‐01)

7121 28%

Promoter

(( (fee,charge,price) <near/3> (low,reasonable,ok,good,low,simple,no)) <in> (base, cite)) and ((country <contains> us) or (country <contains> ca) or (country <contains> gb) or (country <contains> au) or (country <contains> hk) or (country <contains> sg)) and NpsDetrPasiPromTxt <near/2> promoter and (date >= 2010-‐10-‐01 and date < 2012-‐11-‐01)

5032 20%


PASSWORD RECOVERY

19


CUSTOMERS DO FORGET THEIR PASSWORD


~6 Million “Forgot Password” per month globally

PASSWORD RECOVERY UP TO AROUND MARCH 2012



PASSWORD RECOVERY RATE WAS ~50% IN 2011 NOVEMBER (BEFORE IMPROVEMENT)

358 M

6.4M

3.4 M (53%) 3.1 M (48%)

Visits* to PayPal website

# of clicks on “Forgot Password”

(on various PP.com pages)

Password Reset landing page

# successful password reset

A B

~3 Million per month (47%)

# of fallouts (include user exits, identification flows,

password reset emails etc.)

Based on Nov 2011 # of visits to PP site globally (excl. PP mobile app visits) defined as "sequence of consecutive page visits of a PP page with no a 30 mins break

C


WHY IS THERE SO MUCH FALLOUT?


THE “CIRCULAR LOOP” – THE BIGGEST PASSWORD DEFECT

“ you keep sending a password reset email but the link does not take you to the reset page!!!! The only way I can get around this is to call you!!! which is long distance!!!! at least have a toll free number... Really p**sing me off now... Totally thinking of dropping your product due to this 3 hours unresolved frustration!!!! Oh... And I'll tell others!!!!!! “ Sep 3, 2011 “ why when I need to reset my password does the password reset Email send me to the log on page where I need to use the password that I don't know to log on and change it? “ Oct 3, 2011

~600,000 OCCURRENCES PER MONTH (SOURCE: SITE CATALYST)

1) Using existing text mining approach & tools: Tovek, Rapid-Miner


THE ‘%40’ BUG - WE PROVED THAT A PREVIOUS TECHNICAL FIX (1/5/2012 ) DID NOT SOLVE THE LOGIN-TO-PAY PROBLEM (CONTROL) – WE RAISED IT AS P1 (DEFECT 2)

I cannot log in! I know my password and username are correct. Unfortunately when I enter my password the browser or the site changes the @ symbol in the email address to %40 which means it doesn't work.I've just spent money elsewhere. (US, OpinionLab, 1/27,2012)

When I tried to use paypal with ebay, I typed in my email address and it would come up %40 instead of the @ sign. I tried to get it resolved through ebay customer service but they did not resolve it and hung up on me twice and did not attempt to call me back. I used paypal with Amazon and I did not have any problems. I only have problems with ebays paypal. I will never use ebay again because their paypal will not put my email address in correctly no matter how slow I typepeck it in. This is what it gives me xxxxxxx%40gmail.com I am finished with ebay. Thank you (US, csNPS, Newly Churned consumer, 12/22/11)

It won't let me log in to my account to process an urgent order. When I enter my e-mail address it rejects it and shows %40 in place of the @ symbol which is NOT what I entered of course!This has seriously inconvenienced me and I will avoid your service in the future.. (AU, OpinionLab, 1/27,2012)

1: Using existing text mining approach & tools: Tovek, Rapid-Miner 2: Site Catalyst based

~400,0002 occurrences per month


WE ANALYZED TEXT MINED CUSTOMER VERBATIM…

Confidential and Proprietary Confidential and Proprietary 28

~80% of Sitefeedback complaints mentioning “ the 40%” password issue were left on checkout pages (sample verbatim n=200, Nov 2011)

DOES PASSWORD RECOVERY MATTER?


TRACKED CUSTOMER EXPERIENCE IMPROVEMENT


TRACKED CUSTOMER EXPERIENCE IMPROVEMENT (CUSTOMER VERBATIM)


TRACKING IMPROVEMENT

Confidential and Proprietary Confidential and Proprietary 32

6 MM “Forgot Password” / mo.

5 MM / mo.

REDUCED ‘FORGOT PASSWORD’ (MONTHLY NUMBERS)


Approx. one out of five login successes resulted in a successful checkout transaction $48 average payment value (APV) per transaction Estimated monthly financial improvement from reducing login issues = 1,000,000 x 0.2 x $48 = ~$9.6 million TPV per month. Customer Contact Savings: 560,000 less customer calls per year => $2.8 Million/year

ESTIMATED FINANCIAL IMPACT


QUESTIONS?

34

THANK YOU

Han Lai [email protected] Linkedin profile: ../Han-Sheong Lai


WE ANALZED PASSWORD-TEXT MINED CUSTOMER VERBATIM AND GENERATED A PASSWORD PAIN POINT PARETO

0 5 10 15 20 25 30

Circular password reset loop (from password-reset email)

Misc log-in issues (more investigation needed to determine issues)

Unclear password issue articulation by users (hard to categorize)

Unable to login after resetting the password

Unable to login even using correct email and password

Unintended email address notation conversion bug (PPSCR01460894)

Invalid comment

Password requirement too stringent

Unable to reset the password

Never received password reset email

Unable to make payment and shows error that password is wrong

Password reset takes too much time

Password recovery link expired

Circular Loop

600,000 occurrences per month, $2.8MM per year revenue

impact

“%40” bug (Defect 2)

~450,000 occurrences per

month, ~$2M revenue per year impact

[email protected] name%40hotmail.com

1: Using existing text mining approach & tools: Tovek, Rapid-Miner

DEFECT 1

DEFECT 2


BEFORE IMPROVEMENT, OUR PASSWORD RECOVERY DEFECT RATE WAS ~50%

358 M

6.4M

3.4 M (53%) 3.1 M (48%)

Visits* to PayPal website

# of clicks on “Problem with Login”

(on various PP.com pages)

Password Reset landing page

# successful password reset

A B

~3 Million per month (47%)

# of fallouts (include user exits, identification flows,

password reset emails etc.)

Based on Nov 2011 # of visits to PP site globally (excl. PP mobile app visits) defined as "sequence of consecutive page visits of a PP page with no a 30 mins break 1: 3M / mo. x $219 per year Send TPV (assume Casual consumer) x 0.5 reduction/error factor = $27 MM/mo TPV (source: Consumer Engagement Segment Profile Rpt) 2: 3 M / mo. x $5.30/12 mo. Revenue per year x 0.5 = $662,000/mo. ((source: Consumer Engagement Segment Profile Rpt) 4: 3 M / mo. x 2% contact rate x $4 per contact = $240,000 /mo. (source: Global Ops)

C

Our 50% account login / password recovery rate* is not acceptable, resulting in: ~ $3301 Million per year send TPV loss ~ $82 Million per year revenue loss ~ $2.8 Million per year CS contact costs

~$11 MM/year revenue loss


QUANTIFYING THE ‘CIRCULAR LOOP’ PASSWORD RESET CUSTOMER ISSUE (DEFECT 1)

1: Using existing text mining approach & tools: Tovek, Rapid-Miner

How many went to INCORRECT path ?? 665,328

PayPal Home Login page

Express Checkout (main:ec::ux:start)

srd: main home

main:wps:ux:button:start

srd: individual home

6.4M visits indicated inability to login from different pages


WE SIZED THE “%40” P1 BUG FIX: ERROR MESSAGES ON BADLY-FORMED EMAIL ADDRESSES

Source: Sitecatalyst ( total = 251,760 + 69,601 + 64,005) = 385,366, apportion to full month of January = 497,700 error messages

39

January 1-‐ 24 ,2012

Page/Form Error (prop14) Page Views

1. Please check your email address and password and try again 35,202,207

2. You must enter both your email address and password. Please try again. 593,126

3. We couldn't find your email address. Please try again or select <b>I don't know what email ad 571,378

4. Email: Please enter a valid email address. UTF-‐8 email addresses are not currently supported. Emai 251,760

5. To make this email address your primary one, you first need to confirm it. 227,092

6. You have not entered an email address. Please try again. 172,297

7. You cannot remove your primary email address. 163,106

8. This email address has already been confirmed. 153,586

9. You have already confirmed that email address. 113,965

10. Vérifiez votre adresse email et votre mot de passe, puis réessayez. 88,235

11. We couldn't find your email addresses. Please try again. Or, we can try a <a href='hWps://www.p 83,105

12. In order to Request Money, you must first confirm your email address. 73,929

13. Your email address needs to be in the format [email protected]. Please check and try again. 69,601

14. We couldn't find your email address. Please try again. Or, we can try a <a href='hWps://www.pay 68,317

15. Email Address: Please enter a valid address in the format [email protected]. 64,005

1

2

3

Global 14-day Consumer Adoption

AU 14-day Consumer Adoption

AU, CEMEA, NA, UK 14-day Consumer Adoption Comparison (Statistical Significance)


RECENT PASSWORD ISSUES

Documents

CUSTOMER EXPERIENCE IMPROVEMENT MADE POSSIBLE …2013.sentimentsymposium.com/presentations/SAS13-HanLai.pdf1. More than 80,000 English customer feedback left on PayPal.com pages (desktop