Accuracy of Small-Group Estimation

Accuracy of Small-Group Estimation and the Wisdom of Crowds

Jenny ShiMichael D. Lee

Department of Cognitive ScienceUniversity of California, Irvine

Cheese tray knife set

Crafted of sustainable bamboo and sleek stainless steel, these five distinct blades carve up your brie and Roquefort with ease.

Price in Dollars ($1-$50):

Real price: $38

Experimental Stimuli

100 everyday items

• Images, description, prices

• Between $5-$45

• Obtained through shopping websites

Two sets of 50

• Items uniformly distributed by price

• Each item = 1 trial

Wisdom of Crowds

Groups of people can be smarter than the best individuals among them in the right conditions (Surowiecki, 2005)

A crowd can be “wise” when four conditions are met:

Diversity: Each individual has own unique view

Independence: Less relying on others

Decentralization: Draw info from different sources

Aggregation: Turn individual to collective decision

Preliminary Analysis: Individual vs. Crowds

• Examined the mean average deviation of the price estimations of 22 participants.

• Looking at each individual serves a lower bound

• Standard Wisdom of Crowd analysis serves as upper bound.

Current Study

What if there are only small groups available?

• groups of three individuals

• Between subjects design

• Priming, cooperative and competitive settings

Research questions:

• Which of these settings lead to better or worse estimation of the true prices?

• How does the best setting compare to the individuals and standard wisdom of crowd analysis (our preliminary analyses)?

Experimental Conditions

Condition type

1. With two primes (drawn from previous data sets)

2. Cooperate by hearing each other’s bids

3. Cooperate by agreeing on an estimate

4. Compete with each other by playing the Price is Right game

Participants cooperating or competing with each other estimated sequentially and systematically alternated between each trial.

The Price is Right

Rules: To win the game, player must bid closest to the retail price without going over.

Players can bid as high as they want, but they cannot bid the same amount as others or bid less than $1.

Price is Right encourages strategic estimation

Item for bid: Ipod

$150$165 $1

Cognitive model for competition estimate

(Lee & Shi, 2010)

Bottom line: Instead of aggregating the “raw” estimates from participants that competed, we used a cognitive model to infer their latent knowledge.

wx (a,b,c,μ,σ) πc (c | a,b,μ,σ)= w3 (a b,c,μ,σ)p (μ,σ | a,b,c)p (a,b,c | μ,σ ) p (μ,σ)

…blah blah blah.

Results for small group estimates

$9.36 $8.82 $8.79

Competitive Results

Competitive MAD: $8.05

Primed MAD: $9.36

Cooperative Average MAD: $8.82

Cooperative Consensus MAD: $8.79

Competitive estimate was better than both primed and cooperative

Summary of our results

Wisdom of crowds performs best

• Four conditions were present

Competitive outperformed both cooperative and priming.

• Competing participants discouraged to mimic other bids because of winning incentive.

• Participants that cooperate or were primed may be dependent on other participants in the group or additional information given.

Conclusion

Wisdom of crowd analysis is superior to any other aggregation method.

• Resourceful in extracting information from people.

Competition > Cooperation > Individual

• Groups perform better than individuals in estimation tasks.

• Cooperation worse than competition possibly because lack of independence.

Using cognitive models is an efficient way of combining knowledge across individuals.

• Helps us understand both the observed behavior and the reasoning behind it.

Thanks!

References

Lee, M.D., & Shi, J. (2010).  The accuracy of small-group estimation and the wisdom of crowds. In R. Catrambone, & S. Ohlsson (Eds.), Proceedings of the 32nd Annual Conference of the Cognitive Science Society. Austin, TX: Cognitive Science Society.

Surowiecki, J. (2004). The wisdom of crowds. New York: Random House.

Extra Slides

Starbucks stainless steel tumbler

Enjoy your favorite Starbucks brew in this 10-oz. stainless steel tumbler bottle with convenient handles.

Price in Dollars ($1-$50):

Real price: $34

Optimal Price is Right Bidding

For just 3 players, bidding between $1-$50

1 10 20 30 40 500

0.2

0.4

0.6

0.8

1 Player 1

1 10 20 30 40 500

0.2

0.4

0.6

0.8

1 Player 3

1 10 20 30 40 500

0.2

0.4

0.6

0.8

1 Player 4

1 10 20 30 40 500

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1 Winning Probabilities

Toaster Inference

Participants were shown a $28 toaster

• Bid $31, $28, $1

• Mean of data is $20

• Mean of inferred latent pricedistribution is $29

1 10 20 30 40 50Bid

Pro

babi

lity

1 10 20 30 40 50 1 10 20 30 40 50