Distilling the Crowd

Jon Puleston

VP Innovation

28th Nov 2016

The next evolutionary step in crowd

prediction techniques

This is the story of the development of a research methodology

Market research is not so good at predicting certain

things…

Will you buy

this product?

Does this

ad work?

Who will win

the election?

Political

Polling

• The US election poll error = 2%

• EU referendum error = 4%

• UK election error = 5%

Show me a brand that would not be happy with a 2-5% error margin on

volumising their market?

BEFORE WE BEAT OURSELVES UP TOO MUCH ABOUT THIS LET’S

CONSIDER…

5

No other method not reliant traditional market research polling

information gets anywhere near doing any better: • Social media analytics directionally helpful but proven completely unreliable at predicting

• Betting markets have not only miss-called all 3 votes, they have magnified the error

• What respondents say and what they do is so often different – we are often not

very good at observing our own behaviour

• Often asking people questions they don’t know the answer themselves to, or are

too difficult to work out

• Not always asking the right question in the right way to get the right answer

• Sometimes compounded by not asking the right people and the difficulties of

accessing representative samples

DO I NEED TO EXPLAIN TO A ROOM FULL OF RESEARCHERS WHY

PREDICTION IS SO DIFFICULT?

6

• Well, for the last few years we have been very interested in prediction protocols

THE WISDOM OF THE CROWDS

• Instead of asking people what they will do, will they buy a product, you turn then

tables and ask them to predict the outcome – who will win, will the product be

successful

• We have conducted about 3 years’ worth of experiments looking at the differences

between traditional and predictive research approaches

SO WHAT ARE THE ALTERNATIVES?

7

• 100 year old evidence to show that when used in the right way, it can be

uncannily accurate

• Accesses a different part of the brain “we think”

• You can sculpt a different type of survey using these techniques, that is more fun

• With a prediction you can feed back the answer and turn it into a game

WHAT WE LIKE ABOUT PREDICTION PROTOCOLS

8

• We have had some success at using this approach

• It can be really effective for ad testing…

WHAT WE LIKE ABOUT PREDICTION PROTOCOLS

9

0.89 correlation 5x differentiation

• They can miss fire ~ 15% of the time

• Highly sensitive to a wide array of cognitive biases

WHAT WE DON’T LIKE ABOUT PREDICTION PROTOCOLS

10

35% LIKE BEST

9% PREDICT THIS IS

MOST POPULAR

25% PREDICT THIS MOST POPULAR

Prediction error:

FAMILIARITY DOES NOT EQUAL LIKING

PREDICT IF THIS A GOOD AD?

“Its an Apple ad and

everything they do is great,

right!”

some people find it easier to put themselves in someone else's shoes, some

people find it easier to step back and look at things more objectively.

• Crowds are particularly rubbish at predicting things like elections!

• Our predictions for many people are really proxies for what they want to happen…

WHAT WE HAVE LEARNT…

Predict who will win the election

• But its more complicated than that

…we now all live in social bubbles, so even if we are able to step back from what we

want to happen and look out and all we see we see is that nearly everyone has the

same opinions as us

WHAT HAVE WE LEARNT?

15

ME!

The minority illusion

More social people have more connections but not necessarily the same

points of view as the crowd

This person is a

socialite has lots of

dinner parties. She

also goes to London

a lot where blue is

the most fashionable

colour

Thesr 2 people are

also more social and

aware that blue is the

trend colour

The rest of this crowd are

less social don’t have many

dinner parties so have not

seen many other peoples

bathrooms

The minority illusion: predict what coloured bathrooms people have?

The crowd predicts blue!

SOLUTIONS?

Isolating the good

predictors

Philip Tetlock

THE GOOD JUDGEMENT PROJECT

PHILIP TETLOCK’S GOOD JUDGEMENT PROJECT HAS DEMONSTRATED THAT

SOME PEOPLE ARE A LOT BETTER AT PREDICTING THAN OTHERS

By asking people to make ongoing predictions you can begin to calibrate them and cream off the good

predictors into panels of super predictors

LEARNING FROM THE GOOD JUDGEMENT PROJECT

• Those with an actively open mind-set are better at making predictions

• Knowledge and understanding of the topic is important

• For more complex predictions, numeracy and intelligence are important

• Each participant needed to make 20 predictions before you could tell if they

were any good or not

Are researchers good predictors?

Kantar prediction experiment….examples of what we asked them to predict:

Competitive Brand performance

Global economic growth figures

TV audience figures

Global warming estimates

Assessment of fashion trends

Performance of teams in the Olympics

Outcome of a variety of political events

48%

25%

Kantar staff Consumer

Close Predictions

Who will win the US election: Kantar Staff prediciton

80%

20%

Clinton Trump

How many superforecasters predicted the US election?

Even if they are brilliant at predicting, if the raw data they are working from is flawed

they will simply amplify the error (the minority illusion)

Some poll agitators e.g. Polypoll were predicting Clinton victory with 99% certainty

Solution: Find the people with the best

vantage point on what is going on

C

C

A

B

Predicting what coloured bathrooms people have…

C

C

A

B

Person A, has a blue bathroom and so do all their friends so predicts

blue

C

C

A

B

Person B, has an orange bathroom and so do most of their friends

so predicts orange

C

C

A

B

Solution = Ask the plumbers

ME!

Research people

in this territory

• Find a group of people with good judgement but also with a good vantage point!

Then combine these 2 bits of thinking…

Good

JudgementPlumber

Plumbers with good judgement

Predict the polls experiment

• Regular monthly survey in the 5 months leading up to the UK election

• Each month a group of 1,500 randomly recruited panellists were asked to predict the opinion polls

DISTILLING THE EXPERTS

Scored each respondent each month for the prediction accuracy

Isolated the top 100 predictors and aggregated their predictions

Compared their predictions to the average from the 4 leading polls

It worked…sort of

EU Referendum forecasting leave…

Issue

F#@k me it is a lot of work

to find, calibrate and keep these people!

Issue 2

We had our suspicions that we were just lucky. The only way to validate the method would be to test out on 20

or so similar 50/50 elections which come round once in a blue

moon (we thought!)

Could we apply this technique to something that

does not occur once in a blue moon?

Distilling a crowd who could predict the success of movies

Predicting box office takings…

Results of experiments

• Recruited 1,500 participants in the UK & USA

• Survey experiment ran for 5 months with adapted versions in each

market

• 70% of participants completed all 5 waves

• Collected a total of 38 film predictions

• The average participant correctly predicted the box office

performance (+/-20%) with 40% accuracy

• Isolated a group of the top 100 participants who predicted box

office takings (+/-20%) with an average accuracy of 60%

45

Could we switch to something that

was easier to calibrate?

Price testing experiment

48

A simple test taking 6 minutes

Ad testing experiment

50

51Source: Based on an average of 4 large scale experiments

= 90% better

Crowd accuracy (Assuming no network errors)

Accuracy of individual predictors

100

people

50

people

20

people

10

people

5

people

50% 50% 50% 50% 50% 50%

55% 86% 80% 74% 74% 60%

60% 98% 94% 85% 82% 68%

65% 100% 99% 95% 91% 79%

70% 100% 100% 98% 95% 84%

75% 100% 100% 99% 99% 91%

80% 100% 100% 100% 99% 94%

85% 100% 100% 100% 100% 97%

Sample Prediction maths

• Crowd distillation techniques I believe is the next evolutionary step in crowd prediction methodology

• The maths of expert crowd prediction work on smaller numbers - whether or not the market is ready for a

expert micro-sampling techniques is another matter

• It takes work to distil a crowd – finding and maintaining relationships with good predictor is not easy

• Creativity has been require to think how to apply this methodology within the Market research arena

Summary & learnings

54