WHAT WERE THEY THINKING Why People Take Part In Market … · What Were They Thinking: Why People Take Part In Market Research Paul Dixon, BrandHook Pip Stocks, BrandHook WHAT WERE

Page 1 of 14

What Were They Thinking: Why People Take Part In Market Research Paul Dixon, BrandHook Pip Stocks, BrandHook

WHAT WERE THEY THINKING Why People Take Part In Market Research

"You can tell whether a man is clever by his answers. You can tell whether a man is wise by his questions." - Naguib Mahfouz

We often wonder that as we work in an industry built on asking questions and interpreting what people say, are we naïve in our thinking that positioning ourselves as a means to let the consumer voice and share their opinions will elicit the honest feedback we want, when in reality, the motivation to take part in market research could be purely financial? By ignoring the main motivations for taking part in market research, are we really capturing and understanding the voice of the customer and regardless of the methodology, are we really uncovering the truth? Even though we may be investigating ourselves out of a job, we decided to go and find out. We started looking at honesty figures:

• "The book The Day America Told the Truth says that 91 percent of those surveyed lie routinely about matters they consider trivial, and 36 percent lie about important matters; 86 percent lie regularly to parents, 75 percent to friends, 73 percent to siblings, and 69 percent to spouses."

• "According to Dr. Michael Lewis of the Robert Wood Johnson Medical School in New

Jersey, "In a single day, most people lie a minimum of 25 times."

• In fact, it's very hard to tell when someone is lying. Robert Feldman, the author of The Liar in Your Life says ‘Some indicators of deception, like facial and body language, could merely be signs of anxiety or nervousness that have nothing to do with lying.’

• The average woman tells two lies a day, whereas the average man tells about three lies (BBC UK Research 2010). According to a recent study by the Science Museum in London, the average woman tells about 728 lies a year! That comes out to around two lies a day. The average man tells about 1,092 lies a year! That comes out to about three lies a day. The lie women say most often is, “nothing’s wrong, I’m fine.” The lie that men tell the most often is, “I didn’t have that much to drink.”

Page 2 of 14


If these are the lies and distorted truth within the real world, what lies are we being told within the market research world? We wanted to examine the truth into why people take part in research as well as what they were actually thinking, rather than saying, in market research activities. To begin with, we simply asked the participants the question, wanting to know why people were motivated to take part in online panels, telephone surveys, focus groups, in home tribe sessions, MROC and online qualitative methodologies. Our hypothesis was that while more dangers and biases exist within the quantitative methodologies, they are more equipped to eradicate these whereas it is much harder within qualitative projects. We believed that while online methodologies would appear to make it easier for people to lie, that perhaps, it was traditional methodologies that saw a prevalence of stretching the truth. We also wanted to explore the possibility that the market research setting could be the catalyst for cases of distorted truth rather than being generated from the individual themselves. So, as consumer insight specialists, we asked, we poked, we prodded, we challenged and we did what we do naturally, we did some market research. Online Panels Our starting place addressed two main hypotheses – why people take part in quantitative research and how truthful they are when they do so. A quick audit of how online panel companies recruit and entice members of the public also reveals the consistent trade off in messaging – share your opinion for financial reward. This was validated when pureprofile allowed us to complete an independent audit of their 550,000 strong consumer and business panel. We interviewed a randomly selected sample to investigate their motivations for joining an online panel and how they view the relationship they’ve entered into.

Page 3 of 14


Table 1.1 Motivations for Taking Part In Research Thinking back to joining pureprofile, what were your motivations for joining? What was your ONE main motivation for joining N=702 Earning money 36% Sharing my opinion 23% Giving feedback to brands and companies 20% To be invited to take part in other market research opportunities like focus groups 10% Was invited by a family member/friend 10% Other (please specify) 1% No specific reason 1% With a random cross section of recent and long time members of the panel, the participants were asked whether their motivations had changed. The below chart indicates that as time goes by, the motivation to provide opinions decreases while the incentive to earn cash increases. Chart 1.1 The Change In Motivation

We don’t see anything wrong with financial incentives being the main motivation for taking part in research as long as it’s clear up front what is expected of the relationship. It should be based on

Page 4 of 14


the fair exchange principle as sometimes we give consumers less credit than they deserve. We shouldn’t be sending 25 minute questionnaires, we shouldn’t be forcing open ended questions, consumers shouldn’t be screened out of surveys unnecessarily. If we are to believe in the fair exchange and understand the motivations of why people are taking part in research, we need to put ourselves behind the two-way mirror and understand how they perceive our business. The below diagram examines the qualitative element in how 700 panellists perceived pureprofile to be: Diagram 1.1 Descriptions of the Platform

Powered by Wordle Q In your own words, what does being an Account Holder of pureprofile require you to do?

Page 5 of 14


The positioning of pureprofile as being a platform where you provide honest opinions and in return, you get paid every time you participate in a survey was seen to be accurate. Chart 1.2 Positioning Fit of Online Panel

When we’re honest about the service that we’re providing and what we expect from our customers, we’re getting the best results and we can’t kid ourselves that opinions are solely enough. When we asked pureprofile members whether they would continue to be members if no financial reward was offered, commitment to providing ongoing opinions diminished. Chart 1.3 Commitment to Panel Without Financial Incentive

Q Wouldyoucontinuetobeamemberofpureprofileiftheydidn’tpayyouforcompletingeachindividualsurvey?

Page 6 of 14


We wanted to explore the possibility that with money as the motivator, members of online panels may be tempted to stretch the truth in order to achieve their goal. Noting that only one panel company was used for the research and not all online fieldwork agencies are the same or have the same quality measures in place, we were pleasantly surprised by the two quality checks we performed. Our first quality audit was to use the vast amount of information that panel companies hold on their members and match this back to the same questions we asked within a survey. An astounding 98% of the sample passed this ‘hidden’ quality check – meaning they were who they said they were. The 2nd quality audit was, based on the fair exchange principle, to simply ask the panel how truthful they are and have been. Results are below: Chart 1.4 Truth Within Market Research Experience (1)

Page 7 of 14


Chart 1.5 Truth Within Market Research Experience (2)

Q Have you ever completed a survey with answers you thought the survey wanted rather than what you actually thought or believed? What were the reasons for completing a survey not as themselves? Well, it’s mainly our fault! “Not all surveys are set up to have a N/A section and sometimes I have had to answer incorrectly to

proceed to the next level”

“The questions were worded that way and you could not pick other or put in your own answers”

“Sometimes the answers in the multiple choice don't provide an answer that actually applies for you. Which means you have to select an option to continue, so you have to select an option that is not

necessarily what fits you”

“I think I did that once or twice when I first started with pureprofile to make sure I got the max money for the survey, but quickly felt bad for being dishonest & answered honestly after that”

Do we necessarily believe what online panel members tell us? Not entirely but we need to give them the benefit of the doubt and assume they are telling us the truth. The good online fieldwork companies will have the tools and resources to do everything they can to provide ‘honest’ sample and we need to ensure that we deliver the rest.

Page 8 of 14


If the motivations and future intentions are clear from online panel members and if the industry is going to continue to use more and more of an online methodology, what do clients and agencies need to ensure they do to get the best quality opinions? The consumer view or recommendations are as follows: Chart 1.6 User Experience Recommendations

Our recommendations are:

1. Choose your panel partner wisely. There’s good and bad online panels out there. Avoid the companies that treat their panel as a commodity.

2. Get them to adhere to your own quality guidelines and don’t be mislead by claimed ESOMAR guidelines. There’s remarkable consistency in each fieldwork providers quality positioning and the fact they are guidelines, rather than rules, means some can be loosely adhered to. Use your own knowledge and research understanding in developing the quality metrics that will deliver you the truthful insight you require.

3. Ask yourself whether you would complete the questionnaire you want your audience to complete.

4. Use existing information wisely. Online panel companies hold a wealth of existing information and this should be used to either reduce questionnaire / screening length or incorporated as part of your quality measures.

5. Develop in survey checks with the panel partner and check data carefully.

Page 9 of 14


But what about the truth?

"There are a terrible lot of lies going about the world, and the worst of it is that half of them are true." Winston Churchill

We know that with online panels, there’s means and ways of providing a platform from honest participants but how can we ensure that we’re hearing the truth from them and outside of online quantitative, how can we understand exactly what the consumers are saying? We asked the general population about the lies they told. Table 2.1 What People Lie About We would like to get your feedback about honesty and the truth. Looking at the below list, what things would you say that most people have lied about? Weight 62.8 % Age 57.3 % At a job interview 50.9 % Their Happiness 50.3 % Number of relationships / partners 50.3 % Finances / Financial Status 49.7 % Drinking 48.6 % Resume/CV 46.2 % Salary 44.2 % Dieting 39.5 % Intentions 38.0 % Relationship Status 29.6 % Education 22.1 % Belief System / Religious Viewpoint 15.7 % None of these 3.1% Table 2.2 What You’ve Lied About And using the same list, which things have you personally lied about in the last 12 months? Your Happiness 34.3 % None of these 33.8 % Weight 19.5 % Finances / Financial Status 13.2 % Dieting 11.7 % Intentions 11.0 % Drinking 8.8 % Age 7.1 % Salary 6.8 % Relationship Status 5.3 % Number of relationships / partners 4.8 % At a job interview 4.6 % Resume/CV 3.7 % Education 2.0 % Belief System / Religious Viewpoint 1.9 %

Page 10 of 14


The average percentage of truth told within a week was 93%. So there’s an element of story-telling within the general population – whether it be the 7% within a week or the 66% within a year, we’re not going to get the whole truth and nothing but the truth all the time. Therefore, how do we as champions of the consumer voice identify what is and what isn’t a lie. When we looked at what science had to say about lying we found there were some research methodologies to ensure honesty were mostly based on questionnaires: 1. The question of faking has led to an ipsative approach being used in many contexts, where the respondents compare two or more desirable options and pick the one which is most preferred. (Ref: www.changingminds.com) 2. Add in implicit measures like Implicit Association Task. The implicit association test (Greenwald, McGhee & Schwartz, 1998), sometimes called the IAT, is often used to assess traits, such as prejudice, self esteem (Bosson, Swann, & Pennebaker, 2000; Greenwald & Farnham, 2000), and personality, while ensuring that participants are oblivious to the purpose of this procedure. (Ref: www.psych-it.com.au) 3. Having a lie scale within the survey which detects how honest the person is answering. (Ref: Effect of Reinforcing ‘Honesty’ Responses to the ‘Lie’ Scale of the CMAS. Schutz and Whittemore) WikiHow also highlights some interesting basic techniques on how you get people to tell the truth: 1. Try to trick the person you want the truth from. For example, you may convince them you already know the truth, and just want them to conspire with you about what you seem to know. 2. Use the person's weakness against them. An adult might use an underhanded method against another adult, by providing alcoholic beverages to the "secret keeper", in an effort to get them inebriated and learn the truth. 3. Threaten them with exposure if the truth is not voluntarily offered. If someone is guilty of some serious transgression, they may be more willing to share the truth if it will lessen the punishment, or make it easier to accept the consequences. These techniques clearly do not adhere to the AMSRS rules, so we do need to look to other things if we’re going to understand the motivations and truthfulness in qualitative research!

Page 11 of 14


Identifying truth in a qualitative setting

“Research is formalized curiosity. It is poking and prying with a purpose” Zora Neale Hurston

We believe that your consumers are trying to tell you something. The water cooler discussions continue to take place and have moved online in a rapid rate. So if you want to grow your brand, it is important that you listen. Unfortunately most of your customers won’t often tell you things directly so it’s our responsibility to find out what they are really ‘saying’. To explore our remaining hypothesis, we retrospectively went back to a number of qualitative projects we had run over the past 12 months which included face to face focus groups, e-groups, customer advisory committees and intercept surveys to understand whether our subjects had told us the truth. We also put the call out for people who had taken part in research activities in the past or were about to take part in qualitative research. In total, we spoke or surveyed approximately 100 people who had taken part in qualitative research in the past 12 months. The objectives of the follow up research was relatively simple (although the depth of the follow up was dependent upon whether Melon Research or BrandHook had conducted the research):

a) Establish what they said at recruitment was in fact correct and probe on their motivation for taking part in the research

b) Talk them through what they said during the session to understand why they said what they did and investigate any differences between the methodology.

Recruitment Findings For the most part, the recruitment was as specified and the demographics and behaviours that they had been recruited on were correct. There were a couple of examples where respondents had been asked to bring friends and now acknowledge them in the session, and one case where the subject of the group was revealed when it shouldn’t have been, but overall, recruitment was solid. With recruitment, it should always be advised that it’s the moderator’s responsibility – whether this be within an online group, ongoing community or face to face session to validate the participant.

Page 12 of 14


We tend to think of any qualitative work as a job interview – you wouldn’t interview a candidate that was going to be responsible for the possible growth, development or refinement of your company or product without reading their CV – so why should you go into qualitative research without knowing exactly who you are speaking with and their background and how this impacts on the research brief. This is where face to face qualitative research has a seeming advantage over online as the moderator will have the advantage of seeing the person. Many of the in-home tribe sessions that BrandHook run not only go into the homes of the research subjects but into their fridges, wardrobes and cars – it’s taking poking, prodding and digging to a new level that has discovered foie gras in the fridge of an ‘ethical eater’ and Dolce & Gabbana sunglasses in a ‘skint’ students car! While not feasible to enter the homes of all the people you are speaking with, we would recommend a part ethnographic approach to validate recruitment – call them up, email them, observe and challenge them in the focus group waiting room. Motivations for Qualitative Research As for uncovering why people take part in quantitative surveys, we needed to uncover the motivations for taking part in the qualitative research, where the financial reward was ever present: Chart 3.1 Motivation for Taking Part in Online Qualitative Research

Page 13 of 14


In Session Findings For BrandHook/Melon Research qualitative participants, each participant was taken through an abridged version of the transcript and final presentation to understand what they were thinking during their session, why they said the things they did and how they felt throughout the qualitative project. Key findings included:

92% maintained what they said throughout the session was honest and truthful to what they believed.

8% said if they had the chance again they would have been more vocal or on reflection would have said something different.

Most agreed the moderator was instrumental to the session and how the participant contributed:

o Good moderators challenged some of the comments people made throughout and had the ability to remember what had been said previously.

o Poor moderators failed to ask their opinion or recognise/understand why their reasoning would be different to others in the session.

35% said that they went into the session thinking they were there as an expert or someone who should be knowledgeable on the subject and felt they needed to ‘live up’ to this expectation.

7% believed that they were just there to make up numbers and their input wasn’t valid. 5% felt intimidated by other members within the qualitative research.

When looking at the same results by the qualitative method people participated in

Online bulletin boards / advisory committees were by far the most favourable o Participants felt less pressure to be an ‘expert’ and were comfortable in posting

their honest opinions without fear of recrimination. In home / ethnographic sessions were most dependent and reliant on the skills of the

moderators but perceived to be the most interesting and once the session had commenced, the 2nd most conduit to truth telling.

Participants in traditional focus groups had the highest prevalence of people who ‘inflated’ their comments / the truth, but seen to offer the highest financial incentive to take part.

Page 14 of 14


What do we know now?

"First get your facts; then you can distort them at your leisure."

Mark Twain The standouts from the research and our conclusions from a large-scale exercise are pretty simple:

1. Regardless of the methodology, as an industry, we need to recognise that while people are happy to take part in multiple market research activities, there’s the expectation and the desire to be financially rewarded for their time and opinions. In fact, it’s the key motivation for taking part in any type of research but it doesn’t necessarily bias their opinions.

a. For any quantitative research project, you need to select the right panel partner who has the quality checks in place to weed out the “professional panel members” – the people who are there for the money only.

b. For qualitative research, you need to conduct your own quality checks, remembering that we’re paying these people a service for their opinions and treat it as you would any service agreement. It’s about the fair exchange and we need to adhere to that.

2. The truth is always going to be distorted at some stage by someone and it’s our responsibility to identify it. For quantitative, this is easier and simple steps within the survey or at data cleaning will identify inconsistencies. For qualitative projects, it’s the responsibility of the moderator / community manager to utilise the data we hold on people for our advantage.

3. For online methodologies, we all know that we can’t see, hear, smell or touch the participants but maybe this is a good thing – our research indicated that people were more comfortable and more honest within the online setting than they were when in a traditional focus group. Inflated opinions, wearing a Marketing Manager’s cap, and all these things that distort the reality that we want to hear are less when the pressure and group setting don’t apply.

4. Regardless of the methodology, try and bring an ethnographic approach to your project. We know that talking to consumer tribes in their own environment means you can watch and probe into the world around them. Participating in, rather than spying on people, changes the whole investigating process. This way you will find the gaps, the disconnects, the things that keep coming up that don’t always make sense. We learn as much from what isn’t said, as what is. Being in their space means we can probe on things we see and not just what is said. This will unearth a more powerful consumer insight.

Documents

WHAT WERE THEY THINKING Why People Take Part In Market … · What Were They Thinking: Why People Take Part In Market Research Paul Dixon, BrandHook Pip Stocks, BrandHook WHAT WERE