Information Aggregation: Experiments and Industrial Applications Kay-Yut Chen HP Labs

Information Aggregation:Experiments and Industrial Applications

Kay-Yut ChenHP Labs

Experimental Economics Program

Agenda

• Lessons from HP Information Markets

(Chen and Plott 2002)

• Scoring Rules and Identification of Experts

(Chen, Fine and Huberman 2004)

(Chen and Hogg 2004)

• Public Information (Chen, Fine and Huberman 2004)


HP Information Markets (Chen and Plott)• Summary of Events

– 12 events, from 1996 to 1999

– 11 events sales related

– 8 events had official forecasts

• Methodology & Procedures

– Contingent state asset (i.e. winning ticket pays $1, others $0)

– Sales amount (unit/revenue) divided into (8-10) finite intervals

– Web-based real time double-auction

– 15-20 min phone training for EVERY subject

– Market open for one week at restricted time of the day (typically lunch and after hours)

– Market size: 10-25 people


Event 2

0

0.05

0.1

0.15

0.2

0.25

0.3

0 100 200 300

$

P

IAM Distribution

Actual Outcome

HP OfficialForecast

IAM Prediction


Results

Abs % Errors of IAM Predictions

Last Interval Ignored Last Interval Mass at Lower Bound

Event Absolute % errors of HP

forecasts

Average last 60%

trade

Average last 50%

trade

Average last 40%

trade

Average last 60%

trade

Average last 50%

trade

Average last 40%

trade2 13.18% 4.61% 4.57% 4.68% 5.63% 5.68% 5.80%3 59.55% 57.48% 55.72% 54.60% 59.25% 57.46% 56.32%4 8.64% 7.84% 8.15% 8.52% 6.45% 6.77% 7.13%5 32.08% 30.93% 31.57% 31.83% 29.74% 30.33% 30.48%6 29.69% 24.23% 24.54% 25.30% 22.94% 23.22% 23.93%7 4.10% 7.33% 7.02% 6.71% 5.35% 4.91% 4.55%8 0.11% 2.00% 2.35% 1.83% 1.53% 1.39% 1.00%9 28.31% 23.85% 24.85% 24.39% 17.55% 17.32% 16.54%

T-test P-value 0.079 0.084 0.071 0.034 0.026 0.022

Random variable x=official error – market error

H0: mean of x=0 Alternate: mean of x>0


Business Constraints and Research Issues

• Not allowed to “bet” players’ own money -> stakes limited to an average of $50

per person

• Time horizon constraints -> 3 months to be useful

• Recruit the “right” people

• Asset design affects the results (How to set the intervals?)

• Thin markets (sum of price ~ $1.11 to $1.31 over the dollar)

– Few players

– Not enough participation


Reporting with Scoring Rule

Reports of Probability Distribution

A B C

Outcome

p1 p2 p3

Pays C1+C2*Log(p3)


Information Aggregation Function

ssss

sss

N

N

ppp

pppIsP

...

...|

21

21

If reports are independent, Bayes Law applies …


Two Complications

• Non-Risk Neutral Behavior

• Public Information


Dealing with Risks Attitudes:Two-Stage Mechanism

Event 1

Event 2

Event 3

Event 4

Event 5

Event 6

Event 7

Event 8

Stage 1: Information Market

Call Market to Solicit Risk Attitudes

Stage 2: Probability Reporting & Aggregation

Individual Report of Probability DistributionNonlinear Aggregated Function

Tim

e


Second Stage: Aggregation Function

Bayes Law with Behavioral Correction

i=r(V i / i)c

Holding value/Risk- measure relative risk of individuals

Normalizing constant for individual risks

“market” risk~sum of prices/winning payoff

sNss2s1

Nss2s1

N21

N21

p...pp

p...ppI|sP


Experiments:Inducing Diverse Information

A B C

Outcome

Box of

Balls

A

B

C

C

C

* In actual experiments, there are TEN states

Random DrawsProvide Info

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1 2 3 4 5 6 7 8 9 10

Series1

Series2

Kullback-Leibler = 1.453

Comparison To All Information Probability

Experiment 4, Period 17No Information

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

A B C D E F G H I J

States

Pro

bab

ilit

yOmniscient

No Info


Kullback-Leibler Measure

• Relative entropy

• Always >=0

• =0 if two distributions are identical

• Addictive for independent events

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1 2 3 4 5 6 7 8 9 10

Series1

Series2

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1 2 3 4 5 6 7 8 9 10



Experiment 4, Period 171 Player

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

A B C D E F G H I JStates

Pro

bab

ilit

yOmniscient

IA Mechanism

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1 2 3 4 5 6 7 8 9 10

Series1

Series2

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1 2 3 4 5 6 7 8 9 10

Series1

Series2

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1 2 3 4 5 6 7 8 9 10

Series1

Series2



Experiment 4, Period 172 Players Aggregated

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9


Pro

bab

ilit

yOmniscient

IA Mechanism

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1 2 3 4 5 6 7 8 9 10

Series1

Series2

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1 2 3 4 5 6 7 8 9 10

Series1

Series2

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1 2 3 4 5 6 7 8 9 10

Series1

Series2




0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9


Pro

bab

ilit

yOmniscient

IA Mechanism

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1 2 3 4 5 6 7 8 9 10

Series1

Series2

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1 2 3 4 5 6 7 8 9 10

Series1

Series2

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1 2 3 4 5 6 7 8 9 10

Series1

Series2




0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9


Pro

bab

ilit

yOmniscient

IA Mechanism

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1 2 3 4 5 6 7 8 9 10

Series1

Series2

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1 2 3 4 5 6 7 8 9 10

Series1

Series2

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1 2 3 4 5 6 7 8 9 10

Series1

Series2




0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9


Pro

bab

ilit

yOmniscient

IA Mechanism

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1 2 3 4 5 6 7 8 9 10

Series1

Series2

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1 2 3 4 5 6 7 8 9 10

Series1

Series2

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1 2 3 4 5 6 7 8 9 10

Series1

Series2




0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9


Pro

bab

ilit

yOmniscient

IA Mechanism

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1 2 3 4 5 6 7 8 9 10

Series1

Series2

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1 2 3 4 5 6 7 8 9 10

Series1

Series2

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1 2 3 4 5 6 7 8 9 10

Series1

Series2




0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9


Pro

bab

ilit

yOmniscient

IA Mechanism

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1 2 3 4 5 6 7 8 9 10

Series1

Series2

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1 2 3 4 5 6 7 8 9 10

Series1

Series2

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1 2 3 4 5 6 7 8 9 10

Series1

Series2




0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9


Pro

bab

ilit

yOmniscient

IA Mechanism

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1 2 3 4 5 6 7 8 9 10

Series1

Series2

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1 2 3 4 5 6 7 8 9 10

Series1

Series2

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1 2 3 4 5 6 7 8 9 10

Series1

Series2




0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9


Pro

bab

ilit

yOmniscient

IA Mechanism


Experiment 4, Period 17

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

A B C D E F G H I J

Pro

bab

ilit

yOmniscient

IA Mechanism

market

Best Individual


KL Measures for Private Info Experiments

1.977 (0.312) 1.222 (0.650) 0.844 (0.599) 0.553 (1.057)

1.501 (0.618) 1.112 (0.594) 1.128 (0.389) 0.214 (0.195)

1.689 (0.576) 1.053 (1.083) 0.876 (0.646) 0.414 (0.404)

1.635 (0.570) 1.136 (0.193) 1.074 (0.462) 0.413 (0.260)

1.640 (0.598) 1.371 (0.661) 1.164 (0.944) 0.395 (0.407)

No Information

Market Prediction

Best PlayerNonlinear Aggregation

Function


Group Size Performance


Did the Markets Pick out Experts?

Group Exp 1 Exp 2 Exp 3 Exp 4 Exp 5

Random 1.36 0.93 1.18 1.12 1.15

Payoff 1.45 1.09 1.24 1.13 1.39

Value 0.72 0.91 0.94 1.13 1.22

Optimal 0.53 0.72 0.75 0.83 0.77

•KL measure of all query data

•Pick groups of 3


Did Previous Queries Pick out Experts?

Group Exp 1 Exp 2 Exp 3 Exp 4 Exp 5

Random 1.15 0.92 1.18 1.07 1.21

Query 0.78 0.89 0.71 0.92 0.81

Optimal 0.60 0.59 0.69 0.72 0.72

•KL measure of second half of query data

•Pick groups of 3


Public Information

• Information observed by more than one

• Double counting problem

0.00%

20.00%

40.00%

60.00%

80.00%

100.00%

120.00%

A B C D E F G H I J

States

Pro

bab

ilit

yOmnicient

Public

No Info

Information Aggregation with Public InformationKullback-Leibler = 2.591

Public Info Experiment 3, Period 911 Players Aggregated

0

0.2

0.4

0.6

0.8

1

1.2

A B C D E F G H I J

States

Pro

bab

ilit

yOmnicient

Public

IAM

0

0.2

0.4

0.6

0.8

1

1.2

A B C D E F G H I J

States

Pro

bab

ilit

yOmnicient

Public

IAM

0

0.2

0.4

0.6

0.8

1

1.2

A B C D E F G H I J

States

Pro

ba

bil

ity

Omnicient

Public

IAM

0

0.2

0.4

0.6

0.8

1

1.2

A B C D E F G H I J

States

Pro

ba

bil

ity

Omnicient

Public

IAM

0

0.2

0.4

0.6

0.8

1

1.2

A B C D E F G H I J

States

Pro

ba

bil

ity

Omnicient

Public

IAM


Dealing with Public Information:Add a Game to the Second Stage

Event 1

Event 2

Event 3

Event 4

Event 5

Event 6

Event 7

Event 8




Individual Report of Probability DistributionMatching Game to Recover Public Information

Modified Nonlinear Aggregated Function

Tim

e


Assumptions

• Individuals know their public information

• Private & Public Info Independent

• Structure of Public Info Arbitrary


Matching Game


A B C

Outcome

q11 q12 q13Player 1: q1

q21 q22 q23Player 2: q2

q31 q32 q33Player 3: q3...

.

.

.

.

.

.

.

.

.

Player 1’s Payoff: (match function)*(C1+C2*Log(q33))Match function: f(q1,q2)=(1-0.5*sum(abs(q1i-q2i)))^2

Choose player (3) byMax (match function)


Matching Game

• Any match function f(q1,q2) with property

– Max when q1=q2

• Multiple Equilibria

• Payoff increases as entropy decreases

• Hopefully, individuals report public information


Aggregation Function withPublic Information Correction

Bayes Law with a) Behavioral Correction b) Public Info Correction

s Ns

Ns

s2

s2

s1

s1

Ns

Ns

s2

s2

s1

s1

N21

N21

qp

...qp

qp

qp

...qp

qp

I|sP

i=r(V i / i)c





Public Information Experiments

• 5 Experiments

• Various Information Structures

– All subject received 2 private draws & 2 public draws

– All subject received 3 private draws & 1 public draws

– All subject received 3 private draws & half of the

subjects receive 1 public draws

– All subject received 3 private draws & 1 public draws.

2 groups of independent public information.

• 9 to 11 participants in each experiments

Correcting for Public Information

Public Info Experiment 3, Period 911 Players Aggregated


0

0.2

0.4

0.6

0.8

1

1.2

A B C D E F G H I J

Omnicient

sim aggr

IAM

IAM (true public)


ExptPrivate

InfoPublicInfo No Info

Market Prediction

BestPlayer

Nonlinear Aggregation

Function

Public Info

Correction

Perfect Public Info Correction

1 2 draws for all

2 draws for all

1.332 (0.595)

0.847 (0.312)

0.932 (0.566)

2.095(1.196)

0.825 (0.549)

0.279 (0.254)

2 2 draws for all

2 draws for all

1.420 (0.424)

0.979 (0.573)

0.919 (0.481)

2.911 (2.776)

0.798 (0.532)

0.258 (0.212)

3 3 draws for all

1 draws for all

1.668 (0.554)

1.349 (0.348)

1.033 (0.612)

2.531 (1.920)

0.718 (0.817)

0.366 (0.455)

4 3 draws for all

1 draws for half

1.596 (0.603)

0.851 (0.324)

1.072 (0.604)

0.951 (1.049)

0.798 (0.580)

0.704 (0.691)

53 draws for all

Two groups

of public info

1.528 (0.600)

0.798 (0.451)

1.174 (0.652)

0.886 (0.763)

1.015 (0.751)

0.472 (0.397)

KL Measures for Public Info Experiments


Summary

• IAM with public info correction did better than

best person.

• IAM with public info correction did better than

markets in 4 out of 5 cases.

• IAM corrected with true public info did significant

better than all other methods.


Implied Probabilities of Revenue Bins, September 2003

0%

5%

10%

15%

20%

25%

30%

35%Actual Value

$1053mOfficial Projection


Implied Probabilities of Operating Profit Bins, September 2003

0%

10%

20%

30%

40%

50%

60%

70%

$0 -$37.1m

$37.1 -$46.1m

$46.1m -$54.4m

$54.4 -$62.0m

$62.0 -$69.3m

$69.3 -$73.6m

$73.6 -$77.8m

$77.8 -$82.0m

$82.0 -$86.2m

$86.2 -$90.4m

$90.4 -$94.7m

$94.7 -$102.0m

$102.0 -$109.6m

$109.6 -$117.9m

$117.9 -$126.9m

$126.9mor above

Actual Value$113m

Official Projection


Supplementary


Previous Research

• Academic Studies– Information Aggregation in Markets

• Plott, Sunder, Camerer, Forsythe, Lundholm, Weber,…

– Pari-mutuel Betting Markets

• Plott, Wit & Yang

• Real World Applications– Iowa Electronic Markets

– Hollywood Stock Exchange

– HP Information Markets

– Newsfuture

– Tradesport.com

– …


Risk Attitudes

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

A B C D E F G H I J

Risk Loving

Risk Neutral

Risk Averse


Dealing with Risks Attitudes:Two-Stage Mechanism

Event 1

Event 2

Event 3

Event 4

Event 5

Event 6

Event 7

Event 8




Individual Report of Probability DistributionNonlinear Aggregated Function

Tim

e


Probability Reporting


A B C

Outcome

p1 p2 p3

Pays C1+C2*Log(p3)


Second Stage: Aggregation Function

Bayes Law with Behavioral Correction

i=r(V i / i)c




sNss2s1

Nss2s1

N21

N21

p...pp

p...ppI|sP


Private Information Experiments

• 5 Experiments

• Various Information Conditions

– All subject received 3 draws

– Half received 5 draws, half received 1 draw

– Half received 3 draws, half received random number of draws

• 8 to 13 participants in each experiments


Next Step

• Field Test (Fine and Huberman) …

Documents

Information Aggregation: Experiments and Industrial Applications Kay-Yut Chen HP Labs