Upload
dan-olsen
View
28.437
Download
0
Embed Size (px)
DESCRIPTION
Best practices in using metrics to optimize your web product. I gave this webinar on Dec 17, 2008, as part of FeaturePlan's series "The Product Management View".
Citation preview
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC11
Dan OlsenDan OlsenCEO & Founder, YourVersionCEO & Founder, YourVersionDec 17, 2008Dec 17, 2008
Product Management by Numbers:Product Management by Numbers:Using Metrics to Optimize your ProductUsing Metrics to Optimize your Product
CopyrightCopyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
Using Metrics to Optimize your ProductUsing Metrics to Optimize your Product
Product ManagementProduct Management’’s Roles Role
Integrating metrics & optimizationIntegrating metrics & optimization
Using metrics to optimize value creationUsing metrics to optimize value creationBusiness ValueBusiness Value
Customer ValueCustomer Value
UI DesignUI Design
Illustrate with real world examplesIllustrate with real world examples
My BackgroundEducation
BS, Electrical Engineering, NorthwesternMS, Industrial Engineering, Virginia TechMBA, StanfordPHP, MySQL, JavaScript, XHTML, CSS, UI design
17 years of Product Management ExperienceManaged submarine design for 5 years5 years at Intuit, led Quicken Product ManagementLed Product Management at FriendsterOlsen Solutions LLC, PM consultant for start‐upsCEO & Founder of YourVersion, Social Discovery start‐up
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
Product ManagementProduct Management’’s s RoleRole
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
A Product Manager by any Other Name Would Smell as Sweet
Product managers are sometimes calledProduct marketing manager
Program manager
Project manager
Label and definition of role can varyBased on industry or company
Based on B2C (consumer) vs. B2B (enterprise)
Based on stage of company
Can be area of responsibility vs. actual position
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
Product Management isCritical Link in Value Creation
Market• Current customers
• Prospectivecustomers
• Competitors
Product Management
Development Team
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
A Process View of Product Management
BusinessStrategy
ProductStrategy
BusinessObjectives
ProductObjectives
ProductDevelopment
Service/Support
Market/Sell
LongTerm
ShortTerm
“Inbound”Product
Management“Outbound”
Product Management
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
PM’s Job: A Successful ProductKnow market and customer better than anyone elseTranslate business objectives and customer needs into product requirementsBe the clearinghouse for all product ideas Identify, plan & prioritize product ideas to maximize ROI on engineering resourcesWork with team to design and build a great productEvangelize product within and outside the companyKeep the product team motivated
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
Integrating Metrics and Integrating Metrics and Optimization into your Optimization into your
Product ProcessProduct Process
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
Adding Metrics and Optimization to your Product Process
PlanPlan
DesignDesign
DevelopDevelop
BusinessObjectives
ProductObjectives
Prioritized Feature List
Scoping
Requirements & Design
Code Test Launch
Site Level
Feature Level
OptimizeOptimize Metrics & User Feedback
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
Optimization through Iteration:Optimization through Iteration:Continuous ImprovementContinuous Improvement
Measurethe metric
Analyzethe metric
Identify top opportunitiesto improve
Design & develop the enhancement
Launch theenhancement
Learning
Gaining knowledge:
• Market
• Customer
• Domain
• Usability
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
OptimizingOptimizingBusiness Value Creation Business Value Creation
with Metricswith Metrics
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
Approaching Business as an Approaching Business as an Optimization ExerciseOptimization Exercise
Given reality as it exists today,Given reality as it exists today,
optimize our business resultsoptimize our business results
subject to our resource constraints.subject to our resource constraints.
Profit = Profit = RevenueRevenue ‐‐ CostCost
Unique VisitorsUnique Visitors x x Ad Revenue per VisitorAd Revenue per Visitor
Impressions/VisitorImpressions/Visitor x Effective CPM / 1000x Effective CPM / 1000
Visits/Visitor x Visits/Visitor x PageviewsPageviews/Visit x Impressions/PV/Visit x Impressions/PV
New VisitorsNew Visitors + Returning Visitors+ Returning Visitors
Invited VisitorsInvited Visitors + Uninvited Visitors+ Uninvited Visitors
# of Users Sending Invites x Invites Sent/User x Invite Clic# of Users Sending Invites x Invites Sent/User x Invite Clickk‐‐through Ratethrough Rate
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
Define the Equation of your BusinessDefine the Equation of your Business““Peeling the OnionPeeling the Onion””
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
How to Track Your MetricsHow to Track Your Metrics
Track each metric as daily time seriesTrack each metric as daily time series
Create ratios from primary metrics: X / YCreate ratios from primary metrics: X / YExample: How good is your registration page?Example: How good is your registration page?Okay:Okay: # of registered users per day# of registered users per dayBetter:Better: registration conversion rate =registration conversion rate =
# registered users / # # registered users / # uniquesuniques to to regreg pagepage
DateDateUnique Unique VisitorsVisitors
Page Page viewsviews
Ad Ad RevenueRevenue
New User New User SignSign‐‐upsups ……
4/24/084/24/08 10,10010,100 29,60029,600 2525 490490
4/25/084/25/08 10,50010,500 27,10027,100 2424 480480
……
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
Sample Signup Page Yield DataSample Signup Page Yield Data
Daily Signup Page Yield vs. TimeNew Registered Users divided by Unique Visitors to Signup Page
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/10
Dai
ly S
ignu
p Pa
ge Y
ield
Changedmessaging
Added questionsto signup page
Started requiringregistration
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
Using Metrics to Optimize the Using Metrics to Optimize the Equation of your BusinessEquation of your Business
What are the metrics for your business?What are the metrics for your business?Where is current value for each metric? Where is current value for each metric? How many resources to How many resources to ““movemove”” each metric?each metric?
DeveloperDeveloper‐‐hours, time, moneyhours, time, moneyWhich metrics have highest ROI opportunities?Which metrics have highest ROI opportunities?
Return
Return
InvestmentInvestment
Return
Return
InvestmentInvestmentRe
turn
Return
InvestmentInvestment
Metric AMetric AGood ROIGood ROI
Metric BMetric BBad ROIBad ROI
Metric CMetric CGreat ROIGreat ROI
• Which metric has highest ROI opportunity?
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
Case Study:Case Study:Optimizing Viral Loop MetricsOptimizing Viral Loop Metrics
ActiveUsers
Prospective Users
Invite Click
Succeed
Inviteclick-through rate
Conversion rate
Don’t Click
Fail
Invites per sender
% of users sending invites
• Multiplied together, these metrics determine your viral ratio
Users
% of users who are active
= 15% = 2.3
= 85%
Registration Process
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
Maximum Upside Potential of a MetricMaximum Upside Potential of a Metric
0
100%
Registration conversion rate
0
100%
% of users sending invitations
0
?
Avg # of invites sent per sender
2.3
85%
15%
Max possible improvement
0.15 / 0.85 = 18% 0.85 / 0.15 = 570% ? / 2.3 = ?%
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
Doubling Number of Invitations Sent per Sender by Doubling Number of Invitations Sent per Sender by Adding Address Book ImporterAdding Address Book Importer
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
OptimizingOptimizingCustomer Value Creation Customer Value Creation
with Metricswith Metrics
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
The Fuzzy Math of Customer ValueThe Fuzzy Math of Customer Value
You create customer value with a product thatYou create customer value with a product thatSatisfies customersSatisfies customers’’ needsneedsIs easy to useIs easy to useHas a good priceHas a good priceIs better than other alternativesIs better than other alternatives
Not fuzzy: # of users and frequency of useNot fuzzy: # of users and frequency of useApplying metrics approach to customer valueApplying metrics approach to customer value
Need framework of user needs & satisfactionNeed framework of user needs & satisfactionQuantitative: site analytics, usage metrics, surveysQuantitative: site analytics, usage metrics, surveysQualitative: 1Qualitative: 1‐‐onon‐‐1 user feedback sessions are best1 user feedback sessions are best
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
Is the site up when I want to use it?
Is the site fast enough?
Does the functionality work?
Does the functionality meet my needs?
Olsen’s Hierarchy of Web User Needs(adapted from Maslow)
Customer’s Perspective What does it mean to us?
Uptime
Page Load Time
Absence of Bugs
Feature Set
Usability & Design
Decreasing
Dissatisfaction
Increasing S
atisfaction
How easy to use is it?
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
Kano Model: User Needs & Satisfaction
User SatisfactionUser Satisfaction
User DissatisfactionUser Dissatisfaction
Performance Performance (more is better)(more is better)
Delighter (wow)Delighter (wow)
NeedNeednot metnot met
NeedNeedfully metfully met
Must HaveMust Have
Needs & features Needs & features migrate over timemigrate over time
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
Importance vs. SatisfactionAsk Users to Rate for Each Feature
98
8784
8679 847055 80
7280
75
4150
55
60
65
70
75
80
85
90
95
100
40 50 60 70 80 90 100
Satisfaction
Impo
rtan
ce
Recommended reading: “What Customers Want” by Anthony Ulwick
BadBad
GreatGreat
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
What is Your Value Proposition?What is Your Value Proposition?
Which user benefits are most important?Which user benefits are most important?How well does your product deliver?How well does your product deliver?
Benefit 1
ImportanceImportanceto Userto User
Current UserCurrent UserSatisfactionSatisfaction
Benefit 2
User BenefitUser BenefitUpsideUpside
PotentialPotential
Benefit 4
Benefit 5
Benefit 6
High
High
Med
Low
Low
Low High – +
Doesn’t matter
Benefit 3 Lowopportunity
opportunity
?
?
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
Analyzing Product Ideas by ROIAnalyzing Product Ideas by ROI
Investment (developer-weeks)
Ret
urn
(Val
ue C
reat
ed)
Idea C
Idea B
Idea D
Idea A
Idea F
1
1
2 3 4
2
3
4?
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
Prioritizing Product Ideas by ROIPrioritizing Product Ideas by ROI
Investment (developer-weeks)
Ret
urn
(Val
ue C
reat
ed)
Idea B
Idea A
Idea C
1
1
2 3 4
2345678
5
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
The UI Design IcebergThe UI Design Iceberg
VisualDesign
InteractionDesign
InformationArchitecture
ConceptualDesign
Recommended reading: Jesse James Garrett’s“Elements of User Experience” chart, free at www.jjg.net
What most people seeand react to
What good PMs and Designers think about
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
UI questions are never yes/no! (not binary)UI questions are never yes/no! (not binary)Should ask: Should ask: ““What percentage of users What percentage of users ……??””UI changes impact your metricsUI changes impact your metrics
Impact can be positive, negative, small, largeImpact can be positive, negative, small, largeSeek highSeek high‐‐ROI UI changesROI UI changes
Approaching UI Design AnalyticallyApproaching UI Design Analytically
Typical UI design question:Typical UI design question:““When using web pages, do users scroll down?When using web pages, do users scroll down?””
‐‐ YesYes‐‐ NoNo
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
Put Key Conversion Actions Above The FoldPut Key Conversion Actions Above The Fold
Landing Page ALanding Page A Landing Page BLanding Page B
Key conversion action is above the fold
Key conversion action is below the fold
The Fold
The Fold: “Chrome” Takes Away Pixels
768pixels
1024 pixels
Only 600
pixels
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
The Fold IsnThe Fold Isn’’t Binary Eithert Binary Either
600 px
768 px
1024 px
Data courtesy of Data courtesy of ClickTaleClickTaleFree trial at Free trial at www.clicktale.comwww.clicktale.com
% of U
sers The chrome steals
about 170 pixels
Analyze User Screen Height Distributionto Select Design Height for Key Pages
1030, 5%
866, 25%
600, 100%
1024, 20%
1000, 20%
864, 29%
807, 29%
768, 92%
772, 38%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
600 650 700 750 800 850 900 950 1000 1050 1100Monitor Screen Height (pixels)
% o
f U
sers
with
Giv
en (o
r Hig
her)
Scr
een
Hei
ght
Going above768 px = big drop
Page Design Height (pixels)
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
Case Study: Account Signup Process RedesignCase Study: Account Signup Process Redesign
100%
62.3%58.8%
50.9%
34.4% 32.7%
0%
20%
40%
60%
80%
100%
% of U
sers
% of U
sers
Sign in /Sign in /RegistrationRegistration
Account Account TypeType
Cash vs.Cash vs.MarginMargin
5 Partner5 PartnerPagesPages
3 Partner3 PartnerPagesPages
Biggest drop
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
OpenAccount
Sign in
Account Selection
Register
56%
44%
Forget Password
Registration Process
45% drop off(20% of total)
36% overall drop off for this step
70%(32% of Total)
17% drop off (10% of total)
20% drop off(6% of total)
30%(14% of Total)
80%(26% of Total)
55%(24% of Total)
64%of Total
Analysis of Sign In/Registration FlowAnalysis of Sign In/Registration Flow
Change Password
83%(46% of Total)
Abandonment Rate (7 Day Moving Average)
0%
10%
20%
30%
40%
50%
60%
70%
80%
10/7
/02
10/1
4/02
10/2
1/02
10/2
8/02
11/4
/02
11/1
1/02
11/1
8/02
11/2
5/02
12/2
/02
12/9
/02
12/1
6/02
12/2
3/02
12/3
0/02
1/6/
03
1/13
/03
1/20
/03
Aba
ndon
men
t Rat
e (7
Day
Mov
ing
Ave
rage
)
Steps 1-2
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
Redesigned User Flow Improved Redesigned User Flow Improved Registration Conversion RateRegistration Conversion Rate
37% improvement in conversion rate
ReleasedNew Design
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
Product Management by NumbersProduct Management by NumbersCheat SheetCheat Sheet
Define what success meansDefine what success meansCustomer value propositionCustomer value proposition
Equation of your businessEquation of your business
Instrument your site and track key metricsInstrument your site and track key metrics
Identify opportunities and prioritize by ROIIdentify opportunities and prioritize by ROI
Launch, learn, and iterateLaunch, learn, and iterate
Discover ShareBookmark
YourVersion proactively discovers relevant web content for you based on your interests.
Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC4040 CopyrightCopyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC
[email protected]@yourversion.comwww.yourversion.comwww.yourversion.com