Quantifying TasteTesting for Dependence in Choice

Let’s say there’s a survey of the following format:Choose n out of these k objects

For example: Choose your three favorites out of these ten

photographs Of these fifty apps, which ten would you

download to your phone? Which two of these seven movies would you

want to watch?

So if we had a survey like that…

Could we prove that that there is dependence within each person’s choices? For example, do people have a certain “taste” in sushi rolls?

Objectives: We wanted to prove that each person does not

choose randomly. Some items are chosen together more often than they would be otherwise.

In particular, we wanted to find which items are similar to one another. If a person chooses a given object, which other objects is he also more likely to choose?

Dataset SUSHI Preference Data Set -- survey taken by

5000 people in which they were asked to rank ten different types of rolls from best to worst (http://www.kamishima.net/sushi/ )

The ten rolls: shrimp (0), sea eel (1), tuna (2), squid (3), sea urchin (4), salmon (5), egg (6), fatty tuna (7), tuna roll (8), cucumber (9)

We just looked at each respondent’s first three choices and ignored the order in which they listed them. (This way, the data fit our “choose n out of k” format.)

Summary of the Data

0 1 2 3 4 5 6 7 8 9

0 x                  

1 394 x                

2 451 411 x              

3 369 223 299 x            

4 398 519 373 193 x          

5 395 387 412 191 806 x        

6 227 238 144 114 62 140 x      

7 774 925 1412 421 1296 1148 253 x    

8 129 156 404 122 88 134 74 468 x  

9 83 43 42 52 21 47 66 61 35 x

The following is a matrix of how often each pair of sushis appeared together in someone’s top

three:

Most popular

pairs Least popular pair

End of the Story?Doesn’t this answer our questions? The most popular pairings were (2,7)

and (4,7). So those who like roll #7 were more likely to choose roll #2 or #7.

The least popular pairing was (5,9) – only 21 respondents listed them as two of their top three! They must be very dissimilar.

Taste or Popularity?That ignores the fact that some rolls were just more popular overall. It makes sense that (2,7) and (4,7) were chosen together so often since 2, 4, and 7 were popular overall. The reverse is true for 5 and 9.

There’s no clear proof that these pairings tell us anything about people’s taste – they may just reflect each roll’s popularity.

0 1 2 3 4 5 6 7 8 90

5001000150020002500300035004000

Popularity of Each Roll

Roll

# o

f Vot

es

How Should We Adjust For This?

We needed to generate a matrix of how often each pair of rolls would be expected to appear together. We could then compare the actual results to the expected results.

To generate this matrix, we decided to run a simulation.

How Should We Simulate It? Each respondent needs to randomly

choose three rolls The rolls must be chosen without

replacement – each respondent needs to choose three different rolls

Each roll’s overall popularity must be held fixed

Simulation Technique #1 Simply choose three rolls out of ten without replacement,

using sample(0:9,3,replace=FALSE,prob=P1,P2,…)in R Imagine that a number line between 0 and 3 is split up into

10 parts where the size of each part is proportional to the frequency of each subsequent roll.

A random number between 0 and 3 is then generated, corresponding to one of the rolls. For example, if 1.4 was generated, then roll #4 would be chosen.

Simulation Technique #1 con’t

A new number line is then drawn, leaving out whichever roll was chosen the first time, while proportionally increasing the size of each remaining part. For example, this would be the new number line if #4 were chosen:

Once again, a number between 0 and 3 would be chosen, corresponding to the second roll chosen.

This same process would be repeated to choose the third roll.

Problem with Technique #1 We have to redraw the number line after the first choice.

As a result, the probabilities for the second and third choices are not the same as the overall probabilities.

The overall distribution of choices from the simulation is not equal to the overall distribution of choices from the actual survey:

How can we fix this? We somehow need to keep the overall probabilities constant for each choice, while still not allowing for repeats.

  0 1 2 3 4 5 6 7 8 9Actual Frequency 0.107 0.110 0.132 0.066 0.125 0.122 0.044 0.225 0.054 0.015

Simulated 0.111 0.113 0.132 0.072 0.126 0.124 0.049 0.195 0.060 0.017

Simulation Technique #2Hartley and Rao (1962) describe an approach to solve this problem:1. Randomize the order of the rolls. This was

accomplished by calling sample(0:9) in R.2. Split up the number line between 0 and 3 into 10

parts where the size of each part was still proportional to the frequency of each subsequent roll, but using the new order.

For example, when the new order of the roll is [3,7,5,9,1,2,4,0,8,6] we use the following number line:

Simulation Technique #2 con’t

3. A random number between 0 and 1, d, is chosen. 4. The three rolls selected are the ones corresponding

to d, d+1, and d+2.

In the following example d = .95, meaning that rolls 5, 2, and 6 – the rolls corresponding to .95, 1.95, and 2.95 – are chosen.

Simulation Technique #2 = Success!

Our simulation shows that each roll is chosen with the same frequency using this technique as in the actual survey.

  0 1 2 3 4 5 6 7 8 9

Actual Frequency 0.107 0.110 0.132 0.066 0.125 0.122 0.044 0.225 0.054 0.015

Technique #2 0.107 0.110 0.132 0.066 0.125 0.122 0.044 0.225 0.054 0.015

Technique #1 0.111 0.113 0.132 0.072 0.126 0.124 0.049 0.195 0.060 0.017

Expected MatrixUsing this second method, we found our matrix of expected results. The fact that our expectations were so different from the actual data implies that people don’t make their choices independently.

0 1 2 3 4 5 6 7 8 90 x

1 387.57 x

2 470.63 481.2 x

3 221.9 227.63 262.73 x

4 441.5 455.07 552.37 249.87 x

5 432.27 441.6 529.13 245.13 507.77 x

6 145.63 144.43 176.53 81.5 167.7 163.43 x

7 911.9 935.73 1196.3 550.13 1128.4 1080.9 367 x

8 177.03 180.2 214.43 116.4 208.6 200.97 48.367 456.23 x

9 49.1 50.233 51.733 23.767 52.333 51.2 18.667 124.47 20.3 x

Generating the Residual Matrix

We generated the residual matrix using the formula

The residuals serve as measurements of similarity. A large positive residual means that the two rolls are similar and were chosen together more often than would have been expected.

The opposite is true for a large negative residual.

Residual Matrix0 1 2 3 4 5 6 7 8 9

0 x

1 0.327 x

2 -0.905 -3.2 x

3 9.875 -0.307 2.237 x

4 -2.07 2.997 -7.632 -3.598 X

5 -1.792 -2.598 -5.092 -3.458 13.235 x

6 6.742 7.786 -2.449 3.6 -8.162 -1.833 x

7 -4.567 -0.351 6.236 -5.506 4.989 2.041 -5.951 x

8 -3.61 -1.803 12.95 0.519 -8.35 -4.724 3.686 0.551 x

9 4.838 -1.021 -1.353 5.791 -4.331 -0.587 10.96 -5.689 3.26 x

*Remember how 2 and 7 initially seemed to be the most similar pair? It still looks like they are similar, but there are many other pairings which are much more similar. For example, 6 and 9 were chosen together only 66 times yet has a larger residual!

Distance Matrix and Visualization

To convert the residual matrix into a distance matrix, we needed to make all the values positive. We did this by setting distance equal to .

To visualize this matrix, we ran multidimensional scaling (MDS).

MDS attempts to set a point for each roll such that the distance between any two points is proportional to the distance between the corresponding rolls. These points are then plotted on an (x,y) axis so the results can be seen more easily.

Essentially, the n objects are first plotted in (n-1)-dimensional space so that the distances between all points are perfect. This is then “scaled down” to two dimensions.

MDS Results0 - shrimp1 - sea eel2 - tuna3 - squid4 - sea urchin

5 - salmon6 - egg7 - fatty tuna8 - tuna roll9 - cucumber

To further support these results, we re-ran the analysis by looking at each respondent’s top five choices. These were the results of the new multidimensional scaling:

The fact that this plot is so similar to our prior one (see previous slide) proves that our results were not merely a result of the fact that we arbitrarily chose to look at the top three choices and that any value of k and n (where k<n) should work.

The MDS makes sense!The groupings made by the MDS make sense when we look back at what each type of roll was.

Why does it make sense?Look at the clusters it formed: 6 and 9 Egg and Cucumber, the two non-fish

choices 2, 7, and 8 All three are different types of tuna

rolls

Since those clusters make sense on their own, and were confirmed by our statistical analysis, we could also trust the other clusters we formed: 4 and 5 Sea Urchin and Salmon 0, 1, and 3 Shrimp, Sea Eel, Squid

Conclusion In our study, we looked at associations in

choice data using simulations. The simulation was done by sampling without

replacement yet still proportional to size. We showed that people did not make their

choices randomly. MDS and clustering based on the identified

associations revealed the specifics of people’s taste.

This general approach can be readily applied to other choice data.