The Data "Never" Lies: How to Get the Truth from Marketing Data

Embed Size (px)

Text of The Data "Never" Lies: How to Get the Truth from Marketing Data

  • The Data Never LiesHow to Get the Truth from Marketing Data

  • If you torture the data long enough, it will confess.

    - Ronald Coase

    You, the marketer

    The poor,poor data

  • Client Case: Mens Skincare Product

    Holiday Gift Promotion

    Bootstrapped (Small Budget)

    Sell Primarily Online

    Ran Facebook Ads, No Sales

    Whats the problem?

  • Strategy

    Tactics

    Creative

    Types of Problems You Can Have

    or

    Marketing Problems Statistics Problems

    Not Enough Data

    Bad Experiment Design

    Correlation vs. Causation

    These are real problems.Things that we should test.

    Were wrong.But, we dont know were wrong.

  • Resources:

    Todays Full Slide Deck

    On SlideShare

    Open Source Statistics Textbook

    After Todays Talk

    Subscribe at nickrosener.co/breakfast

    Extra Goodies for Subscribers

    Download and Use With Clients:

    Excel Files

    Simulations and Calculations

    Marketing Experiment Report Template

  • Real Marketing Problems

    Strategy

    Tactics

    Creative

    Things Like:

    Segmentation

    Targeting

    Positioning

    Product

  • Real Marketing Problems

    Strategy

    Tactics

    Creative

    Things Like:

    Social Media

    Platform Choice

    Ad Format

  • Real Marketing Problems

    Strategy

    Tactics

    Creative

    Things Like:

    Copy

    Images

    Video

  • The Stats ProblemsWhat were going to cover

    Estimation(basically, measurement)

    Experiments(how to test stuff)

    and

    Up next:

  • Model the Process

    Impressions

    Profit

    Visits to Landing Page

    Purchases

    Impressions:

    Clickthrough Rate:

    Cost / Click: $1

    Conversion Rate:

    Retail Price: $100/item

    Gross Profit Margin: 50%

    Gross Profit: $50/item

  • Model the Process

    Impressions

    Profit

    Visits to Landing Page

    Purchases

    Impressions: Not Important*

    Clickthrough Rate: Not Important*

    Cost / Click: $1

    Conversion Rate:

    Retail Price: $100/item

    Gross Profit Margin: 50%

    Gross Profit: $50/item

  • Model the Process

    Impressions

    Profit

    Visits to Landing Page

    Purchases

    Impressions: Not Important*

    Clickthrough Rate: Not Important*

    Cost / Click: $1

    Conversion Rate:

    Retail Price: $100/item

    Gross Profit Margin: 50%

    Gross Profit: $50/item

    At breakeven, total ad cost = total gross profit

  • Model the Process

    Impressions

    Profit

    Visits to Landing Page

    Purchases

    Impressions: Not Important*

    Clickthrough Rate: Not Important*

    Cost / Click: $1

    Conversion Rate: 2% @ breakeven

    Retail Price: $100/item

    Gross Profit Margin: 50%

    Gross Profit: $50/item

    At breakeven, total ad cost = total gross profit

  • Now, New Questions

    How do we find out if the adhas a conversion rate that is above breakeven?

    (vs. a conversion rate of zero)

    and

    How much money do we needto spend to find out?

  • Estimate The Truth

    Estimating a Proportion

    Lots of thingsin marketing

    are proportions.

    Conversion Rates

    Engagement Rates

    Clickthrough Rates

    Open Rates

    How it Works

    Your Data (Sample)

  • Estimating a Proportion

    To estimate a conversion rate:

    1. Set up an ad

    2. Let it run

    3. Count the # of purchases attributed to the ad

    4. Divide by total # of clicks

  • Estimating a Proportion

    Smaller Sample=

    Higher Probabilityof Being Wrong

    (and not knowing it)

    =

    More Affordable

    Larger Sample

    =

    More ConfidenceThat Youre Right

    =

    More Expensive

    >

  • Spending $5 on an Ad

    90.4% of the time, our estimate of conversion is 0%while the true conversion rate is 2%.

    Estimated Conversion %

    Probability

    0% 20% 40% 60% 80%

    90.4% 9.2%

  • Watch Your Stopping Point

    $100

  • Bigger Budget = More Reliable

    $500

  • Nicks Budget Rule of Thumb

    In our example, thats $250 - $500+ @ $1/click and a 2% conversion rate.

    Spend enough where you would expect to getat least 5 10+ events.

  • The Stats ProblemsWhat were going to cover

    Estimation(basically, measurement)

    Experiments(how to test stuff)

    and

    Lets switch gears:

  • Study Population

    Why Do Experiments?

    Using When to Poston Social Media Studies

    Heres why:

    Your Community

    = , IMHO.

  • Correlation

    Why Do Experiments?

    Using When to Poston Social Media Studies

    Heres why:

    Causation

    = , IMHO.

  • Retrospective

    Why Do Experiments?

    Using When to Poston Social Media Studies

    Heres why:

    Experiment

    = , IMHO.

  • Why Experiments > Retrospective

    Meet John,a Social Media Manager

    When are my customersmost engaged?

    Ill analyze my posts,and see when I got

    the highest engagement!

  • Why Experiments > Retrospective

    Looks like my customers prefer

    to engage at midday.

    This may be true.But there could be

    another explanation.

  • Why Experiments > Retrospective

    Wait. I also eat baconand drink coffee at

    10:00 every day.

    Hungry andTired

    Bacon-InspiredBrilliance

    Post-CoffeeCrash Remember:

    Correlation Causation

  • Typical Problems with Retrospectives

    Hidden Third Factor

    ObservableThing 1

    ObservableThing 2

    HiddenThing

    Causes Also Causes

    Correlated,but not causal

    Stats Concept:Lurking Variable

  • Typical Problems with Retrospectives

    Fishing for Insights

    What

    Stats Concept:Spurious Correlation

    Disclaimer:This is also a problem for poorly-designed

    experiments.

  • Post Afternoon

    Simple Randomized Experiment Design

    Post Early

    Randomizeinto Groups

    Write Social Media Posts

    Post Midday

    MeasureMeasure

    Measure

    Sample Size:At least 5 10 events

    per group.

  • What We Went OverWe covered a ton.

    1. Real Marketing Problems vs. Statistics Problems

    2. Model the Process

    3. Estimate a Proportion

    4. Bigger Sample = More Confident

    5. At Least 5 10+ Events Per Group

    6. Experiments > Retrospective, When Possible

  • Resources:Slides, Excel Files, Report Templateshttp://nickrosener.co/breakfast

    Contact:@nickrosenernick@nickrosener.co

    ?

    Questions