CHAPTER 1: When to listen to your customers1

How Smart Businesses Develop Winning Ideas: A Lean Marketer’s Guide

Table of Contents

Welcome! ---------------------------------------------------------- 3

What you’ll learn in this guide --------------------------------- 3

Part I:

Testing is important, so why aren’t people doing it? ------ 4

Testing: It’s not all or nothing ---------------------------------- 5

4 traditional approaches (and their shortcomings) --------- 6

How testing ideas with surveys works ------------------------ 7

How Procter & Gamble comes up with winning concepts - 8

5 Benefits of testing concepts with surveys ----------------- 9

Part II:

How to test concepts with surveys ---------------------------- 10

Step 1: Define your goals --------------------------------------- 10

How do you measure success? -------------------------------- 11

Step 2: Form a hypothesis about the results ---------------- 12

Step 3: Choose your methodology ---------------------------- 13

Step 4: Create your survey ------------------------------------- 14

Step 5: Send your survey to your target market ------------ 15

Step 6: Analyze the results ------------------------------------- 16

Step 7: Track your progress over time ------------------------ 17

Start testing your concepts and creative -------------------- 18

3

Welcome!

What you’ll learn in this guide

Over the next 15 pages, we’ll show you how to use surveys to test

concepts and perform market research. You’ll learn how to take the

guesswork out of your big decisions without delaying your launch

or breaking the bank. Our guide includes:

3What we’ve learned from talking to marketers

3 Insights on research trends

3What to avoid when it comes to testing

3Tips for getting your timing right

3Best practices on writing concept testing surveys

3Real-world examples of how big brands use surveys to make decisions

Are you investing in the right ideas?When you’re responsible for getting new concepts and campaigns to market fast, you need effective ways to test your ideas. Whether you’re developing ads, a product, logo, package design, branding, a landing page—or just about anything else—your big idea is a big investment. Concept testing helps make sure it pays off.

Part I: Testing is important, so why aren’t people doing it?

We wanted to know how much importance professionals place on testing new concepts before they go to market. To find out, we used SurveyMonkey Audience to survey more than 500 product managers, advertisers, and marketing professionals.

While more than 70% feel testing is important, most professionals aren’t conducting market tests before their new ideas go live. For example, 60% of those we surveyed pre–test their ads only once in a while—or not at all!

The majority say they lack either the time, the money, or the know–how to follow through on their desire to pre–test new concepts.

4

In your opinion, how important is it to your ad’s success that you test the ad or an ad concept before launching it?

How often do you or your team test ad concepts or ideas before developing and launching them?

ImportantExtremely

often

0% 0%10% 10%20% 20%30% 30%40% 40%50% 50%60% 60%70% 70%80% 80%

Slightly important

Very often

Moderately often

Not at all important Slightly, not at

all often

5

Testing: It’s not all or nothing

Here’s the thing. Many marketers and product professionals aren’t

testing new concepts because the issue of research seems black

and white: Either use a big chunk of budget on a lengthy, expensive

project to determine the best approach, or just make their best guess.

But you can test more and save money at the same time. The key

is to use surveys to test ideas earlier in the development process.

It’s an approach that costs less than traditional research, it’s faster,

and it allows you more iterations to make sure you’re getting things

right with potential customers. You can also get critical research

early in the development cycle—before you’re too far along to make

major changes.

The wrong way to pick up the paceMoving more quickly doesn’t have to mean making uninformed

business decisions. Some companies sacrifice information for the

sake of speed, and that can cost them. Gap made this mistake in

2010. Under pressure to increase sales, Gap decided to change

their image with a new logo. The process was quick and inexpen-

sive—and a huge fail.

Executives chose a new logo without testing different concepts

with their target market customers. Then Gap rolled it out

almost silently, changing their website masthead one day with

no explanation.

Criticism came fast and furious. Customers were confused, and the

company’s attempt to make the changes seem more organic just

felt sneaky and misguided to the millennials it hoped to attract.

Within a week, the old logo was back. Though Gap hadn’t spent

too much money on the process, they paid the price by damaging

their brand image.

Source: “Closing the Gap’s Brand Fail,” Natalie Zmuda, Advertising Age.

6

4 traditional approaches (and their shortcomings)

1. Testing different versions live In–market testing means putting your concepts in front of consumers in the marketplace to see how they perform. It’s a great way to get reads on concepts out in the wild, so to speak. But there’s a high development cost. Because each concept has to be fully developed before research can begin, you’re only able to test a few concepts.

2. Full–service market research agency A full–service research agency should have the analyti-cal rigor and the access to a test market to get the data you need to make business decisions. But high costs and months–long timelines mean using agencies for research makes it hard to take a lean, iterative approach.

3. Focus groups Gathering feedback via focus groups can influence the evolution of your creative concepts. You can get in–depth feedback, and you’ll have the opportunity to ask follow-up questions and dive really deep into each participant’s opinion. However, you can only get qualitative feedback this way because of all the many costs involved: recruiting participants, facility rental, moderator or agency fees, compensation for partici-pants, and travel to view groups. For all the expenses focus groups incur, the results simply may be more directional than conclusive. And outspoken participants can skew your results by dominating discussions.

4. Gut instincts Okay, yes—you may have lots of experience working on ideas that have worked. But gut instincts will only take you so far. When you combine your inspiration, passion and smarts with quantitative analysis, you get a truly compelling argument for developing an idea.

“People think focus means saying yes to the things you’ve got to focus on. But that’s not what it means at all. It means saying no to the hundred other good ideas.”  — Steve Jobs

7

How testing ideas with surveys works

Basic Testing Framework

“The only way to win is to learn faster than anyone else.” —Eric Ries, The Lean Startup: How Today’s Entrepreneurs Use

Continuous Innovation to Create Radically Successful Businesses

Surveys fit into the creative or product development process very early on. If you still want to use traditional research methods, such as focus groups, it’s smart to test concepts right after you’ve come up with ideas—but before you spend lots of time or money developing and presenting them as final. Here’s an overview. We’ll dive into the nitty gritty in Part II.

8

How Procter & Gamble comes up with winning concepts

In the past, conducting large–scale tests through traditional research agencies worked well for Procter & Gamble (P&G). But when it was time to cut unnecessary costs from their budget, they looked at what traditional research methods cost them. Typically, they’d have their agency test four concepts—and the ideas that met the industry benchmarks for success were taken to production. But in most cases, about half the concepts would come up short.

P&G realized they were throwing away hundreds of thousands of dollars on bad ideas by fully testing them. Today, they use surveys as a creative testing filter to gain more confidence in each of the concepts they bring to full–scale traditional testing. And they’re saving significant amounts of cash.

They’re also able to test more concepts, which means there are more opportunities to find winners (which may have been discarded otherwise).

9

5 Benefits of testing concepts with surveys

1. Test many concepts early You probably have more ideas than you could ever bring to market. A survey allows you to present a wide range of ideas to your target market, and get qualitative feedback fast. You’ll be able to test more of your big ideas, and work through more iterations as you fine tune things.

2. Filter out the bad ideas You know that idea your boss wants you to test, but you just KNOW it’s a bad one? Let your target market decide what’s awesome and what falls flat. It’s easier to relay the bad news when it comes directly from the customer—and it helps you make sure you give your customers what they want (even if it wasn’t the idea you were sold on).

3. Refine your good ideas If you test a bunch of concepts through a survey, you can quickly see which ones are rising to the top. Rather than going into a focus group or a pilot program with half–baked ideas, you can take the survey feedback and noodle on the concepts before it’s show time.

4. Test good ideas with different audiences more easily Need to target a specific audience, or segment feedback based on customer demographics? It’s much easier (and less costly) to use surveys to reach different demograph-ic groups in your target market. For example, you can send a survey to a target audience of women 25–34 who have purchased beauty products online in the past year. Then use the demographics you collect (like income levels or geography) to see how different concepts resonate with different segments

5. Save a lot of money Not convinced that surveys can save you lots of time and money? After you’ve launched your first concept testing survey, take a look at the results and see if any of your ideas tanked. Chances are, you had some that weren’t so great. Now imagine having spent months and thousands of dollars developing those ideas, just to have them shot down by a focus group or fail in A/B testing. Yikes!

“I have not failed 10,000 times. I have successfully found 10,000 ways that will not work.”  — Thomas Edison

Regardless of what you’re developing, you know you’ll need to test throughout the process to make it a success. (At least 70% of you do.) But where do you start? Performing research is tough: Who you ask, what you ask them, and how you ask it dramatically affects the quality of the data you collect.

To make it easier for you, we’ve outlined a framework you can use to get your testing surveys off the ground.

10

Part II: How to test concepts with surveys

Step 1: Define your goals

Imagine you’re the head of marketing for a snack company, Tasty

Treats, interested in advertising your new blue cheese potato chips.

Most consumers have never tasted a blue cheese chip before, so your

ad campaign will need to convince them to try something new. So

how will you know if you have a winning ad on your hands?

To know the answer, you need to define a narrow and concrete research

goal. In the this example, your goal would likely be to determine which

ad drives the highest purchase intent and has the highest brand recall

among your target consumers.

What to considerHere are 4 questions to consider as you define your

research goal:

3Why are you doing this research?

3What do you want to learn?

3What data points will convince you to make a decision?

3What are the attributes of a successful concept?

Our research has revealed six key metrics as success factors for different types of concepts. See how these might apply to your product, service, or ad:

11

How do you measure success?

1. Purchase intent: Whether your concept motivates viewers to go buy your product/service or take another desired action (click, call, etc)

2. Appeal: The overall appeal of your concept is a strong indicator of purchase intent. It’s a less specific metric, but a very useful one

3. Recall: A gauge of how well your target audience remembers your message or brand after seeing the concept

4. Need: Whether your target market sees a need for your product/service in their lives

5. Preference: Relative analysis of how your concepts stack up against one another

6. Loyalty: How strongly your target market prefers your brand over competitors—a strong predictor of repeat purchase and long–term success

12

Step 2: Form a hypothesis about the results

Video: See how experts form hypothesesSee how Brent Chudoba, SVP and GM of SurveyMonkey Audience, formed a hypothesis

and created a survey to understand how Netflix customers responded to their original

series, House of Cards. Watch video >>

It’s critical to form a hypothesis before writing your survey. This step is where your gut instincts come into play. What do you expect your survey to reveal?

Your guess will largely be based on subtle market knowledge born of your professional experience—so use it! Your assumptions don’t have to be right, but having an idea you want to prove (or disprove) helps you zero in on the right success metrics and best questions to ask.

In the Tasty Treats example, your hypothesis may look something like this:

3Advertisement A will create the highest recall because it

focuses on the benefits of the new product

3Advertisement B will have the highest purchase intent

because it creates a coupon

3Advertisements A and B will be more appealing to a young

audience because they use a celebrity endorser

13

Step 3: Choose your methodology

When you’re ready to test your concepts, you’ve got to think about which methodology is best.

Do you want to show your respondents one concept, or several? Here are two strategies testing experts use:

1. Single–exposure testing (also called Monadic testing) means you break your respondents into groups or “cells” so that each cell only sees—and gives feedback—on one concept. By showing just one concept in the abstract, you’ll get a clear view of how your target market views it. Comparisons between different concepts may not be apples to apples, however, because they come from different groups of people.

2. A multi–exposure strategy treats the whole group of respondents as one big cell and shows each of them a randomized collection of several concepts. You’ll get results that show the relative appeal of different concepts. The presence of multiple concepts means you won’t know how each concept would have been rated on its own—and you end up with a so–so winner if all the concepts are bad.

€ Tips for comparing images in the same surveyIf you’re running multiple images by your survey respondents, you

may want to end your survey with a ranking question. This can be

very helpful to see if one or two concepts rise to the top or fall

to the bottom of most viewers’ lists. Just remember that ranking

questions only reveals relative appeal, not overall appeal.

Even if one test group chose Advertisement B over options A, C,

and D, that doesn’t mean you should go with B, as it might have

been better than the other choices—but only mediocre overall. Pair

ranking questions with some about the overall appeal of individual

options, and you might learn that your best concept is one you

haven’t thought of yet.

14

Step 4: Create your survey

When the time comes to write your survey, keep these tips from our survey experts close at hand:

3Keep your goal and hypothesis in mind while formulating your questions. This will keep your questions relevant and your survey succinct because you want to be considerate of your respondents’ time: If a question won’t help you make a decision, don’t include it.

3Use proper scales when including multiple choice questions. In most cases, you’re better off using words than numbers. A numeric scale may seem more rigorous, but it could produce ambiguous results. Do all respondents think “2” on a scale from 1 to 5 means the same thing?

2 Do this d Not this

How likely are you to purchase this product?

Extremely likely

Very likely

Moderately likely

Slightly likely

Not at all likely

How likely are you to purchase this product?

1 (not likely)

2

3

4

5 (extremely likely)

3Ask the same questions for each concept to make your results easy to compare. If you ask participants who view concept A whether they like it, and then ask those who view concept B if it will motivate them to buy your product, you will have to guess at which received a more favorable response.

3Turn rating questions into a scorecard by assigning weights to each answer choice. This will help you determine which concepts did better than others, even when respondents did not directly compare them.

Video: 3 common survey mistakes to avoidWant to make sure you’re getting accurate survey results?

Just spend 2 minutes with Sarah, our resident survey research

scientist. In this video, she reveals 3 of the most common

mistakes she sees survey–creators making every day—and

how you can avoid them. Watch video >>

15

Step 5: Send your survey to your target market

Once your survey is written, it’s time to launch it. But to whom? To

keep your research focused, let market segmentation principles guide

your selection.

As soon as you know who’s in your target market, send your survey

to them—and only them. Don’t waste time or money registering the

opinions of people who will never buy your product. You’ll end up

creating something that won’t resonate with your real customers.

“Founders solicit feedback about their product from friends, family, investors…pretty much anybody they can get their hands on. What they get is a mashup of conflicting advice, none of it from the people who are at all likely to buy the product...stop wasting your time talking to people who are never going to buy your product.”  — Laura Klein, Boxes and Arrows, “The Right Way to Do Lean Research”

Survey targeting tips 3Decide which demographic criteria—such as age, gender, and income—will put someone in your target market.

3Look at behavioral characteristics. This includes patterns related to buying. Who is most likely to purchase your type of product? From your brand in particular? To try an unfamiliar product?

3 Identify your target market’s attitudinal or psychographic

traits. What are their habits, values, and other elements of their lifestyle?

16

Step 6: Analyze the results

Employing these tips will give you clearly usable responses to help

you decide if your best concept is among the set you tested. Does

one (or more) of your ideas pass the bar for success? If so, great work!

What if one concept doesn’t emerge victorious? That could mean

your survey wasn’t focused enough, that several ideas were equally

good, or that none of your concepts will work out in the marketplace.

These different interpretations of the same results could lead you in

opposite directions! However if you have properly planned, written,

and administered your survey, interpreting your results won’t be a

guessing game.

The best outcome for a concept test is finding a clear winner. But sometimes your data can be hard to translate into decisions. Even before receiving your results, you’ve set yourself up for smooth analysis if you have:

3Created a clear definition of success

3Hypothesized the result

3Written focused questions

3Asked the same questions about each product

17

Step 7: Track your progress over time

Your work may not be done after running a survey once. There are two reasons to run multiple surveys:

1. Get your process right: Especially when you begin testing with surveys, it can take a few iterations to find the right framework for your business. If your feedback is unclear, reconsider your definition of success. Maybe your hypothesis underestimated how sticky your concept would be, or maybe you hypothesized about the wrong metric altogether! Don’t be afraid to make major changes to a survey that doesn’t seem to be working.

2. Benchmark your results: Once you’ve got a winner (and you see its success translate to the marketplace) use it to compare future results. For example, you can include an old ad with candidates for your new campaign. Or, you can use the results from a product concept that wound up being successful to assess the performance of new concepts.

But once you find a survey that works, don’t change it. Running the

same survey multiple times is the only way you can reliably benchmark

your progress. Learn more about benchmarking survey results >>

How Cosmo uses surveys to pick a winnerBefore Cosmo started using surveys to test their magazine covers,

they were performing in–market testing. Each month, they’d test

different covers by launching two versions of their magazine

cover—live! And they let sales numbers inform which cover buyers

preferred. As you can imagine, the problem with this approach

was that they were losing money on the inferior cover, and they

still didn’t know why consumers preferred the favorite.

Now Cosmo tests multiple versions of their cover by sending

surveys to readers in their target demographic. And they publish

only the most popular choice. Plus, the insights they gain from

the feedback they collect helps them optimize future designs.

18

Start testing your concepts and creative

Now that you know why testing is important, how research is changing, and how to make your research leaner, get started testing with SurveyMonkey Audience!

SurveyMonkey Audience is home to millions of on–demand survey respondents who are ready to tell you what they think of your ideas. Our Concept Assessment package delivers end–to–end testing in just one week—from survey design to data collection to analysis.

Our survey research experts have designed a testing framework that helps you focus on critical data points and choose only the best ideas to take to market. More than 20 million customers (including 99% of the Fortune 500) use SurveyMonkey to make smarter decisions.

Whatever the size of your company or your goals, we’re here to help you find your winners fast.

Bring your best ideas to market with concept testing Contact us to get fast and reliable testing for creative and product concepts

Contact us about your project —>

What’s included in our Concept Assessment package?