© 2014 ConceptSpring

Elaine ChenSeptember 2014

Primary Market Research

Methodologies Overview

Why do research?

You are not your customer

A few tenets

Not a sales pitch

Keep your eyes and ears open

Be open minded

Sausage Making

2 classes of research

• Problem Research– To understand who the buyers and users are– To understand the problem statement– To understand the context in which the product will

be used– To understand use cases

• Solution Research– To understand the usability and utility of the product– To help prioritize the feature set– To understand pricing elasticity– To understand customer satisfaction

The anatomy of a research program

• Clearly articulate a hypothesis to be tested (aka research goals and objectives)

• Define the market segment to do research in• Develop subject recruitment criteria according to the

definition of the chosen segment• Pick a methodology• Run the research program• Crunch the data• Take steps based on findings

Example recruitment questionnaire (Segment: people with sleep issues)

• What is your age, gender and profession? (terminate once age/gender quota has been filled for the matching segment)

• What is your household income? (terminate if <$xxk)• Are you interested in learning more about your sleep? (terminate if not)• Are you currently encountering sleep problems? (terminate if not)• Are you currently under the care of a doctor for your sleep issues?

(terminate if true)• How many nights in an average week are you encountering sleep issues?

(select between 2-5)• Please describe the sleep issues you are currently encountering (check all

that apply). (cannot fall asleep; multiple awakenings; cannot go back to sleep after awakening; snoring spouse; etc)

• Do you currently share your bed? (partner, child, pet) If so: how many nights in an average week is your sleep affected by your bed partner(s)?

Qualitative versus quantitative

• Open ended: you are having a conversation

• Anecdotal: you derive meaning from each conversation

• Requires excellent listening and observation skills

• Best done face to face (or at least by video Skype or by voice call)

• Best used for exploratory research where you are trying to get to know a problem or a persona

• Qualitative (<50 samples) • Quantitative (>1000 samples)• Close ended: you are

administering a questionnaire• Statistical: you derive meaning

from looking at aggregate results

• Requires systematic analysis skills

• Best done via on-line survey (or, in the olden days, via phone survey)

• Best used for confirmation research where you are trying to verify / quantify what you think is important

Qualitative, then quantitative

The qualitative pass helps you develop your quantitative pass and interpret the result with context and meaning

Common methodologies

• Qualitative (<30 samples)

• Contextural interview• Observation / shadowing• Immersion• Longitudinal diary study• Photo essay• Usability benchmark• Focus groups• … etc

• Quantitative (>1000 samples)

• Surveys– On line surveys– Conjoint analysis– Pricing studies

• Monadic• Multiple monadic• Van Westendorp• … etc

– Customer satisfaction: NPS, P/M fit

• Web testing– A/B split, Multivariate– Web analytics– … etc

Some Qualitative Techniques

Contextural interview: Interview subject where product will be used

Structure the interview for consistency, ask open ended q’s, let the subject lead

Set up gear

Warm up chat

“Tell us the story of sleep in your house”

House tour

Discuss sleep

problems

Probe expect-ations

Show product idea if and only if there’s if time

left over and they ask

Observation: Learning things people cannot articulate

In this project, we were trying to understand people’s showering habits. We contracted the research to an ethnographic research company, who took shower footage of volunteer subjects in a nudist colony.

We were able to measure things like % time people stand facing the showerhead versus the other way, etc. that no one can tell us with words.

Observation: Learning things people will not articulate

In this project, we were trying to understand people’s infotainment needs and multitasking behavior in the car. In this example, the subject (a real estate agent) was talking on the phone and scheduling appointments in her book while driving at 60mph (With our coworkers in the back, who were shadowing her for 8h that day )

Immersion: become the user

In this project, we were trying to understand the usability of competitive CAD programs. We had our in house CAD experts try out 3 or 4 applications. They spent 3 days on each application, designing the same set of parts and documenting their experience learning and mastering each application.

Longitudinal diary studies: learn things over days/weeks, not over 1-2h

Photo essays: use imagery to capture the context

Usability benchmark: Give users a task, watch them struggle and learn why

Focus groups: not a fan. Dynamics hard to manage, expensive to do right

Image credit: msresearchinc.com

Some Quantitative Techniques

First things first: the product concept

In an on line survey, you have 2 pages to make your point about your product’s value proposition and its relevance to your survey respondents. For all practical purposes, you are creating an ad. The effectiveness of your survey depends on this ad. Invest in the creative content to make your

point. Get a marketer to help do it right.

On line surveys – gauging interest

Construct your survey carefully for easy tabulation of results.

Best practice is to use 5 point scales, except for specific methodologies that require 10 point scales (e.g. Net promoter score – see slide in customer satisfaction section)

There are techniques to “code” open ended responses – read more here. http://www4.uwm.edu/cuir/resources/upload/Planning-Council-qualitative-analysis-handout.pdf

Conjoint analysis

• Conjoint analysis measures the value the market places on each feature of your product and predicts the value of any combination of features.

• How it works:– Ask questions that force respondents to make trade-offs among

features– Determine the value they place on each feature based on the

trade-offs they make– Simulate how the market reacts to various feature trade-offs you

are considering

• Read more at: http://www.pragmaticmarketing.com/resources/conjoint-analysis-101#sthash.dqQGvTpS.dpuf

Pricing research: Monadic – a.k.a. “Intent to Purchase at Price”

• Show the product concept, then ask:• How likely would you to be purchase this product at

this price?– Not at all likely– Moderately likely– Neutral– Moderately likely– Extremely likely

• “Top 2 box” gives estimate of purchase intent

Pricing research: Multiple Monadic

• Repeat the Monadic survey with the exact same product concept, but with a different price point

• Tabulate top 2 box responses and see if you see a lift in purchase intent as price goes up or down

• Note: the product concept in a survey is the key. It’s a 2 page ad where you get to explain your value proposition. Invest in the creative content and the marketing messaging - get help from a marketer to do it right.

Pricing research: Van Westendorp (the Kindle team did this and came up with $99 )

• Show the product concept, then ask these 4 questions with a free response:– At what price would you consider the product to be so expensive that

you would not consider buying it? (Too expensive)– At what price would you consider the product to be priced so low that

you would feel the quality couldn’t be very good? (Too cheap)– At what price would you consider the product starting to get expensive,

so that it is not out of the question, but you would have to give some thought to buying it? (Expensive/High Side)

– At what price would you consider the product to be a bargain—a great buy for the money? (Cheap/Good Value)

• Chart cumulative frequencies. Intersection = ideal price.• Read more here.

http://en.wikipedia.org/wiki/Van_Westendorp's_Price_Sensitivity_Meter

Customer satisfaction measure: Net Promoter Score (NPS)

• Show the product concept, then ask this question on a 10 point scale (1 = not at all likely, 10 = very likely):– “How likely is it that you would recommend [your product] to a friend [or

family, or colleague etc]?”

• Categorize responses:– Promoters = 9, 10 – your loyal enthusiasts– Passives = 7, 8 – satisfied but unenthusiastic– Detractors = 0-6 – unhappy, can damage brand by word of mouth

• Calculate the Net Promoter Score as follows: (NPS)– NPS = % Promoters - %Detractors

• Calibrating yourself– 2013 scores: Apple laptop (best computer company) – 76%; Kaiser

(best health insurance company) – 36% - find your industry for comparables

• Read more here: http://www.netpromoter.com/why-net-promoter/know/

Customer satisfaction measure: Product / Market Fit (P/M Fit)

• Show the product concept, then ask this question on a 3 point scale (1 = not at all disappointed, 2 = somewhat disappointed, 3 = very disappointed)– “How would you feel if they could no longer use [the product]?”

• If over 40% of responses = 3 (very disappointed) you have achieved Product Market Fit

• Read more here: http://www.startup-marketing.com/using-survey-io/

A word of caution: know your statistics and experimental design

• The mean of any metric is meaningless without the variance. Learn how to do analysis of variance (ANOVA) properly to interpret your results correctly. Start with the student-t test and build up from there

• The best ANOVA effort with the best experimental design falls down with insufficient sample size. – 500 is the absolute minimum– 5000 is the best practice number– 1000 is a good compromise

A/B split testing

Image credit: http://www.smashingmagazine.com

Multivariate testing

Image credit: http://visualwebsiteoptimizer.com

Web analytics

Image credits: emaginewebmarketing.com, http://www.openwebanalytics.com/

The possibilities are endless

@chenelaine blog.conceptspring.com

Thank you