Search as recommendation

Embed Size (px)


When recommendation is described in mathematical terms as a matrix equation, a striking symmetry in the form of the equation becomes apparent.Exploiting this symmetry allows us to build search engines that don't need meta-data and self-organizing web-sites.

Text of Search as recommendation

  • 1. Recommendation as SearchReflections on SymmetryMapR Technologies - Confidential1

2. Company BackgroundMapR provides the industrys best Hadoop Distribution Combines the best of the Hadoop communitycontributions with significant internallyfinanced infrastructure developmentBackground of Team Deep management bench with extensive analytic, storage, virtualization, and open source experience Google, EMC, Cisco, VMWare, Network Appliance, IBM, Microsoft, Apache Foundation, Aster Data, Brio, ParAccelProven MapR used across industries (Financial Services, Media,Telcom, Health Care, Internet Services, Government) Strategic OEM relationship with EMC and Cisco Over 1,000 installsMapR Technologies - Confidential 2 3. What is Hadoop? A new style of computation A new style of combining computation and storage Allows very large computations Used by all large internet companies, many other industries Fundamentally changes the economics of large-scale computationMapR Technologies - Confidential3 4. Why Big Data? Because we can Because we can learn new things Because new economics of computation favors large scale Because big data can be simpler than small dataMapR Technologies - Confidential4 5. Recommendations Often known as collaborative filtering People who bought x also bought y Actors (people) interact (bought) with items (x and y) observe successful interaction We want to suggest additional successful interactions Observations are inherently very sparseMapR Technologies - Confidential 5 6. Examples Customers buying books (Linden et al) Web visitors rating music (Shardanand and Maes) or movies(Riedl, et al), (Netflix) Internet radio listeners not skipping songs (Musicmatch) Internet video watchers watching >30 s (Veoh) iTunes song purchases or plays (Apple)MapR Technologies - Confidential 6 7. Fundamental Algorithm History matrix A has the shape of actors x items Cooccurrence matrix K has the shape of items x items an actor interacted with both x and y sum over all actors A is also a linear operator K tells us users who interacted with x also interacted with yMapR Technologies - Confidential7 8. Warning MapR Technologies - Confidential8 9. Warning Mathematics aheadMapR Technologies - Confidential 9 10. Fundamental Algorithmic Structure CooccurrenceK=A A Tr = AT (Ah) = (AT A)h For very large data-setsr = sparsify(A A)h TMapR Technologies - Confidential10 11. But Wait ...Does it have to be that way?MapR Technologies - Confidential11 12. But why not ...T(A A)hMapR Technologies - Confidential 12 13. But why not ...T (A A)hWhy just dyadic learning?MapR Technologies - Confidential 13 14. But why not ...T (B A)hWhy just dyadic learning? Why not triadic learning?MapR Technologies - Confidential 14 15. But why not ...T (B A)hWhy just dyadic learning? Why not p-adic learning?MapR Technologies - Confidential 15 16. For example Users enter queries (A) (actor = user, item=query) Users view videos (B) (actor = user, item=video) AA gives query recommendation did you mean to ask for BB gives video recommendation you might like these videosMapR Technologies - Confidential16 17. The punch-line BA recommends videos in response to a query (isnt that a search engine?) (not quite, it doesnt look at content or meta-data)MapR Technologies - Confidential 17 18. Real-life example Query: Paco de Lucia Conventional meta-data search results: hombres del paco times 400 not much else Recommendation based search: Flamenco guitar and dancers Spanish and classical guitar Van Halen doing a classical/flamenco riffMapR Technologies - Confidential 18 19. Real-life exampleMapR Technologies - Confidential 19 20. Real-life exampleMapR Technologies - Confidential 20 21. Hypothetical Example Want a navigational ontology? Just put labels on a web page with traffic This gives A = users x label clicks Remember viewing history This gives B = users x items Cross recommend BA = label to item mapping After several users click, results are whatever users think theyshould beMapR Technologies - Confidential21 22. Resources Me @ted_dunning Slides and such: The original paper Accurate Methods for the Statistics of Surprise andCoincidence (check on citeseer) Source code Mahout project contact meMapR Technologies - Confidential22