Understanding multivariate testing techniques and how to ... Understanding multivariate testing techniques

  • View

  • Download

Embed Size (px)

Text of Understanding multivariate testing techniques and how to ... Understanding multivariate testing...

  • Multivariate testing

    Understanding multivariate testing techniques and how to apply them to your email marketing strategies

    An Experian Marketing Services white paper

  • Multivariate testing

    Page 1 | Understanding multivariate testing techniques and how to apply them to your email marketing strategies

    Testing should be standard operating procedure Whether it is in search, display, direct mail, email, mobile or social, one would be hard-pressed to find a marketer today who isn’t aware of the concepts of A/B or multivariate testing (MVT) and the value they offer. With the amount of data that is being captured, stored, processed and analyzed in today’s world, combined with the speed at which results boomerang back to the tester, we operate in a time and place where there is no excuse not to continually optimize your marketing campaigns via some sort of testing procedure.

    Even the president tests Testing has indeed become mainstream. Highlighting this fact is a recent article in Businessweek, which details the use of statistical testing by the Obama reelection campaign when trying to solicit donations from their email subscribers. The article describes how the campaign staff would initially experiment with a handful of different subject lines for their emails. After sending these seed emails out to a limited number of subscribers, the team would wait for the results (amount of money donated) to trickle in before it became clear which email subject lines performed best. Now equipped with the knowledge of what emails would perform significantly better than others, the team would send the winning emails out to the rest of the millions of email subscribers on the president’s list, arguably increasing the donations haul from email marketing by millions of dollars.

    What’s being tested? Testing ranges in complexity. A/B and A/B/C testing are the most common forms of testing used in marketing today. These types of tests are fairly straightforward to perform and interpret, as they focus solely on measuring the effects of changing values from only one factor. As shown in the graph on the following page from Experian Marketing Services’ December 2012 Email Market Study, where email marketers across eight verticals were surveyed about their email marketing initiatives, most companies today are leveraging A/B and A/B/C testing to optimize different facets of their email campaigns (subject line, frequency, time of day, day of week, call-to-action, etc.), but very few (11 percent) are leveraging the more complicated, yet more powerful, tool of multivariate testing to optimize their email marketing efforts.

  • Multivariate testing Multivariate testing

    An Experian Marketing Services white paper | Page 2

    What type of testing do you perform on email campaigns?







    MultivariateFriendly from

    Number of


    Product placement

    Call-to- action

    HTML versus


    Day of week

    Time of day

    FrequencyCreativeSubject line










    11% 11%

    Source: Experian Marketing Services’ Email Market Study, Acquisition and engagement tactics, December 2012

    A common A/B testing mistake A common mistake when implementing A/B testing is performing sequential A/B tests in an effort to arrive at optimal levels for multiple factors. These experiments often start with the standard or status quo settings of the key factors to be tested. The levels of the factor that is believed to be the most responsible for performance are tested first, while the other factor levels remain constant. After the responses have been gathered and the optimal level for the first factor is determined, the factor regarded as the second most influential is tested next, with the “optimal” first level factor remaining fixed for the rest of the experiment. This process continues to repeat itself until each factor level has been individually tested.

    To better illustrate this flawed process, consider the following example with two factors, each with just two levels (the simplest case possible). Suppose an organization wants to determine the best image and ad copy to put in an email with the click-to-open ratio being the metric to maximize. We will denote the different images as I1 and I2 and the different ad copy as C1 and C2. In their first email blast to 20,000 customers, the company decides to send half of these customers an email with the combination (I1, C1) and the other half with the combination of (I1, C2).

  • Multivariate testing

    Page 3 | Understanding multivariate testing techniques and how to apply them to your email marketing strategies

    The results are as follows:

    Click to open percent: (I1, C1) = 7.5 percent

    (I1, C2) = 8.5 percent

    Based on these results, the company believes that Copy 2 is the preferred ad copy and fixes the next email blast to 10,000 more customers at this level so that the next test in the sequence will only vary the image level not already tested.

    Click to open percent: (I2, C2) = 9.5 percent

    Seeing these results, the company decides that (I2, C2) is the optimal combination in terms of being able to generate the highest click-to-open ratio.

    The problem is that the company may have missed out on finding the global

    optimum by not testing the fourth combination of (I2, C1), which may have

    yielded a click-to-open ratio even greater than 9.5 percent.

    A/B tests assess one level of one factor versus the control group, but cannot measure the interaction effect across factors.

    The reason this method of sequentially testing one factor at a time fails to find the optimal factor levels is that an interaction effect exists between factors 1 and 2. That is, the factor effects are not additive, but rather the combinations among different factors and their levels produce an additional effect (interaction) when used simultaneously. By not being able to capture interaction effects, this sequential approach may miss the optimum altogether.

    In situations such as these, it is more appropriate to perform a multivariate test (factorial test to be exact), where all factors are changed together and all combinations are accounted for. There is a wide array of different kinds of multivariate tests available, but when the number of factors and the number of factor levels to be tested are limited, the full factorial approach is the best option, as it retains the most amount of information about the factors.

  • Multivariate testing Multivariate testing

    An Experian Marketing Services white paper | Page 4

    Multivariate testing: a brief history Multivariate testing has its roots in the statistical discipline of experimental design. Experimental design is the process of planning a study to meet specified objectives. Planning an experiment properly is important in order to ensure that the right data and sufficient sample sizes are available to answer the research question of interest, as clearly and efficiently as possible. One of the first recorded applications of experimental design comes from the early 20th century and a statistician named R.A. Fisher. Fisher was hired by an agricultural research center to determine the effects that various factors (soil type, sun exposure, rainfall, fertilizer, etc.) had on plant growth. By designing proper, randomized controlled experiments before any seeds had been planted into the ground, Fisher was able to harvest results that allowed him to determine the optimal levels of each factor that would maximize plant growth and crop output. His work proved seminal, and others soon found good use of experimental design in a wide variety of industrial and manufacturing problems. Only recently have brands discovered the power of experimental design (now termed multivariate testing) in optimizing marketing campaigns.

    Advantages of MVT over A/B testing Multivariate testing has several advantages over that of A/B or A/B/C testing. In A/B or A/B/C testing, it is easy to determine which particular creative performed best at generating the most amount of conversions, but this type of testing can’t tell you why one creative performed better than the next. Was the increase in conversions due to the change in background color, the call to action, the main image, etc.? MVT allows you to measure the size of the effect that each of these factors has in generating conversions and provides guidance on where to focus your next round of testing efforts.

  • Multivariate testing

    Page 5 | Understanding multivariate testing techniques and how to apply them to your email marketing strategies

    The graph below is an example of what may be found through MVT testing:

    Factor influence on click rates

    0% 10% 20% 30% 40% 50%



    Call to action

    Main image

    Layout 42.0%

    Fa ct

    or s

    Percent effect







    MVT also saves the tester both time and money, as it tests multiple factor-level settings simultaneously. Instead of being limited to varying the values of only one factor, MVT, as its name implies, allows you to test multiple factors at the same time, thereby eliminating the need to run multiple A/B tests. This increased efficiency gives the tester the ability to find the optimal factor- level settings more quic