Collaborative Filtering in Map/Reduce

Collaborative Filteringin

Map/Reduce

Ole-Martin Mørk - Open AdExchange

tirsdag 14. september 2010

Vision

• Learn that Map/Reduce is simple

• Learn that Map/Reduce may be powerful

• Collaborative Filtering is fun!


Agenda

• Map/Reduce

• Collaborative Filtering

• Collaborative Filtering with Map/Reduce

• Amazon Elastic MapReduce


Map/Reduce


Map/Reduce

• Very scalable algorithm

• Inspirered by map and reduce from functional programming.

• Everything is based on key/value


6 phases

• Reader

• Map

• Partition

• Comparison

• Reduce

• Writer


6 phases

• Reader

•Map

• Partition

• Comparison

•Reduce

• Writer


Map


List(“hello”,“dude”).map{x=>x.substring(0,1)}

functional map


Map/Reduce map

• Input is key/value

• Output is key/value


Simple Example, Map

• Count occurences of words in a document

• Input is: <linenumber>, <content of line>

• For each word on the line, the output is <word>, <count>


Map


Reducetirsdag 14. september 2010

functional reduce

val sum=List(32,40,23).reduceLeft{_+_}


Map/Reduce reduce

• Input is key/list of values

• Output is key/value


Simple Example, Reduce

• Reduce input is <word, counts>

• For each value we increase the count

• Output is <word>, <sum of counts>


Reduce


CollaborativeFiltering


Amazon


Last.fm


Sceneami.com


User based

• Useful when we have

• Small number of users

• High correlation between users

• Data that changes often


Item based

• Useful for big sites like Amazon etc..

• Small overlap between users

• Mostly static data


Min

drø

mm

eapp

likas

jon

Pattern Matching in Scala

Euclidean Distance

Rating

Rating

Match

Match


Euclidean Distance

• Alf‘s presentations:1,25,56,57,58,98 (6)

• Kari’s presentations: 2,25,98,99 (4)

• Equal presentations: 25 and 98 (2)

• Unmatched presentations: 6-2 + 4-2 = 6

• Distance score: 1/1+sqr(6)= 0.29


Recommended sessions

• Me:1,2,5,6,7

• Kate (0.31): 5,6,8,9

• Paul (0.41): 1,2,4,5,6

• Mary(0.31):1,5,8,9



• Me:1,2,5,6,7

• Kate (0.31): 5,6,8,9

• Paul (0.41): 1,2,4,5,6

• Mary(0.31):1,5,8,9

• Recommended: 8 (0.62)



• Me:1,2,5,6,7

• Kate (0.31): 5,6,8,9

• Paul (0.41): 1,2,4,5,6

• Mary(0.31):1,5,8,9

• Recommended: 8 (0.62), 9 (0.62)



• Me:1,2,5,6,7

• Kate (0.31): 5,6,8,9

• Paul (0.41): 1,2,4,5,6

• Mary(0.31):1,5,8,9

• Recommended: 8 (0.62), 9 (0.62), 4 (0.41)


Demo


More Map/Reduce


Several iterations

Iteration 1

Iteration 2

Iteration 3


Several iterations

Iteration 3

Iteration 1 Iteration 2


Partitioning

Reducer Reducer

Jeff

Kate

Mary

Ali

Lea

Paul

Paul Mary Kate Lea Jeff Ali


Comparison

Reducer Reducer

Pres 2

Kate

Pres 2 JeffPres 2

Mary

Pres 1

Paul

Pres 1 AliPres 1

Lea

Pres 1Pres 1Pres 1 Pres 2Pres 2Pres 2Paul Lea Ali Jeff Mary Kate


Guidelines

• Never access external sources during computation.

• Your functions should be small and fast

• You might not have all the data available


Hadoop

• Hadoop is reusing objects, so remember to clone if you plan to keep them.

• You can read and write all objects implementing hadoop.WritableComparable

• write(DataOutput)

• readFields(DataInput)

• compareTo(Object)


Collaborative Filtering, the Map/Reduce way


Overview

• Create an application that recommends JavaZone presentations.

• Overall goal: Scalable performance

• 4 iterations

• Reading input from text file


Iteration 1

• Map input: <user>, <presentations>

• Map output: <presentation>, <user>

• Reduce output: <presentation>, <userList>


Iteration 2

• Map input: <presentation>, <userList>

• Map output: <user>, <userList>

• Reduce input: <user>, <list of userList>

• Reduce output: <userTuplet>, <match count>


Iteration 3

• Map input: <userTuplet>, <match count>

• Map output: <userTuplet>, <diff>

• Map output: <userTuplet reversed>, <diff>

• Reduce output: <user>, <similaruser>


Iteration 4

• Map input: <user>, <similaruser>

• Map output: <user>, <presentation with score>

• Reduce output: <user>, <presentations>


Demo


Map/Reduce on EC2


Elastic Map/Reduce

• Same code

• Same input

• Different configuration


Upload files

s3cmd put oax-jz10:jar/oax-jz10.jar target/oax.jz10.jar

s3cmd.rb put oax-jz10:input/data.txt data.txt


Create job flow

elastic-mapreduce --create --alive --log-uri s3n://oax-jz10/log


Register iterations

elastic-mapreduce --jobflow j-1NLAIW45QUN4B --jar s3n://oax-jz10/jar/oax-jz10.jar --arg com.openadex.pres.iterations.Iteration1 --arg s3n://oax-jz10/input --arg s3n://oax-jz10/output1


Download output

s3cmd.rb get oax-jz10:output4/part-00000 out


Demo


Summary

• Map/Reduce may be simple

• Map/Reduce can be really powerful

• Collaborative filtering is fun :-)



Thank you

Ole-Martin Mø[email protected]/olemartin

del.icio.us/olemartin/jz10

All images are licensed with Creative Commons. See http://bit.ly/mr-photos for details,


http://bit.ly/mr-photos

http://bit.ly/mr-photos

Technology

Collaborative Filtering in Map/Reduce