Industry Report

76 JANUARY • FEBRUARY 2003 Published by the IEEE Computer Society 1089-7801/03/$17.00©2003 IEEE IEEE INTERNET COMPUTING

Amazon.comRecommendations Item-to-Item Collaborative Filtering

R ecommendation algorithms are bestknown for their use on e-commerce Websites,1 where they use input about a cus-

tomer’s interests to generate a list of recommend-ed items. Many applications use only the itemsthat customers purchase and explicitly rate to rep-resent their interests, but they can also use otherattributes, including items viewed, demographicdata, subject interests, and favorite artists.

At Amazon.com, we use recommendation algo-rithms to personalize the online store for each cus-tomer. The store radically changes based on cus-tomer interests, showing programming titles to asoftware engineer and baby toys to a new mother.The click-through and conversion rates — twoimportant measures of Web-based and emailadvertising effectiveness — vastly exceed those ofuntargeted content such as banner advertisementsand top-seller lists.

E-commerce recommendation algorithms oftenoperate in a challenging environment. For example:

• A large retailer might have huge amounts ofdata, tens of millions of customers and millionsof distinct catalog items.

• Many applications require the results set to bereturned in realtime, in no more than half asecond, while still producing high-quality rec-ommendations.

• New customers typically have extremely limit-ed information, based on only a few purchasesor product ratings.

• Older customers can have a glut of information,based on thousands of purchases and ratings.

• Customer data is volatile: Each interaction pro-vides valuable customer data, and the algorithmmust respond immediately to new information.

There are three common approaches to solving therecommendation problem: traditional collabora-tive filtering, cluster models, and search-basedmethods. Here, we compare these methods withour algorithm, which we call item-to-item collab-orative filtering. Unlike traditional collaborativefiltering, our algorithm’s online computation scalesindependently of the number of customers andnumber of items in the product catalog. Our algo-rithm produces recommendations in realtime,scales to massive data sets, and generates high-quality recommendations.

Recommendation AlgorithmsMost recommendation algorithms start by findinga set of customers whose purchased and rateditems overlap the user’s purchased and rateditems.2 The algorithm aggregates items from thesesimilar customers, eliminates items the user hasalready purchased or rated, and recommends theremaining items to the user. Two popular versionsof these algorithms are collaborative filtering andcluster models. Other algorithms — includingsearch-based methods and our own item-to-itemcollaborative filtering — focus on finding similaritems, not similar customers. For each of the user’spurchased and rated items, the algorithm attemptsto find similar items. It then aggregates the simi-lar items and recommends them.

Traditional Collaborative FilteringA traditional collaborative filtering algorithm rep-resents a customer as an N-dimensional vector ofitems, where N is the number of distinct catalogitems. The components of the vector are positivefor purchased or positively rated items and nega-tive for negatively rated items. To compensate for

Greg Linden, Brent Smith, and Jeremy York • Amazon.com

best-selling items, the algorithm typically multi-plies the vector components by the inverse fre-quency (the inverse of the number of customerswho have purchased or rated the item), making lesswell-known items much more relevant.3 For almostall customers, this vector is extremely sparse.

The algorithm generates recommendationsbased on a few customers who are most similar tothe user. It can measure the similarity of two cus-tomers, A and B, in various ways; a commonmethod is to measure the cosine of the anglebetween the two vectors: 4

The algorithm can select recommendations fromthe similar customers’ items using various meth-ods as well, a common technique is to rank eachitem according to how many similar customerspurchased it.

Using collaborative filtering to generate recom-mendations is computationally expensive. It isO(MN) in the worst case, where M is the numberof customers and N is the number of product cat-alog items, since it examines M customers and upto N items for each customer. However, becausethe average customer vector is extremely sparse,the algorithm’s performance tends to be closer toO(M + N). Scanning every customer is approxi-mately O(M), not O(MN), because almost all cus-tomer vectors contain a small number of items,regardless of the size of the catalog. But there area few customers who have purchased or rated asignificant percentage of the catalog, requiringO(N) processing time. Thus, the final performanceof the algorithm is approximately O(M + N). Evenso, for very large data sets — such as 10 million ormore customers and 1 million or more catalogitems — the algorithm encounters severe perfor-mance and scaling issues.

It is possible to partially address these scalingissues by reducing the data size.4 We can reduce Mby randomly sampling the customers or discardingcustomers with few purchases, and reduce N by dis-carding very popular or unpopular items. It is alsopossible to reduce the number of items examinedby a small, constant factor by partitioning the itemspace based on product category or subject classi-fication. Dimensionality reduction techniques suchas clustering and principal component analysis canreduce M or N by a large factor.5

Unfortunately, all these methods also reducerecommendation quality in several ways. First, ifthe algorithm examines only a small customersample, the selected customers will be less similarto the user. Second, item-space partitioningrestricts recommendations to a specific product orsubject area. Third, if the algorithm discards themost popular or unpopular items, they will neverappear as recommendations, and customers whohave purchased only those items will not get rec-ommendations. Dimensionality reduction tech-niques applied to the item space tend to have thesame effect by eliminating low-frequency items.Dimensionality reduction applied to the customerspace effectively groups similar customers intoclusters; as we now describe, such clustering canalso degrade recommendation quality.

Cluster ModelsTo find customers who are similar to the user, clus-ter models divide the customer base into many seg-ments and treat the task as a classification problem.The algorithm’s goal is to assign the user to the seg-ment containing the most similar customers. It thenuses the purchases and ratings of the customers inthe segment to generate recommendations.

The segments typically are created using a clus-tering or other unsupervised learning algorithm,although some applications use manually deter-mined segments. Using a similarity metric, a clus-tering algorithm groups the most similar customerstogether to form clusters or segments. Becauseoptimal clustering over large data sets is imprac-tical, most applications use various forms ofgreedy cluster generation. These algorithms typi-cally start with an initial set of segments, whichoften contain one randomly selected customereach. They then repeatedly match customers to theexisting segments, usually with some provision forcreating new or merging existing segments.6 Forvery large data sets — especially those with highdimensionality — sampling or dimensionalityreduction is also necessary.

Once the algorithm generates the segments, itcomputes the user’s similarity to vectors that sum-marize each segment, then chooses the segmentwith the strongest similarity and classifies the useraccordingly. Some algorithms classify users intomultiple segments and describe the strength ofeach relationship.7

Cluster models have better online scalabilityand performance than collaborative filtering3

because they compare the user to a controllednumber of segments rather than the entire cus-

similarity A B A B

A B

A B

r r r rr r

r r, cos ,*

( ) = ( ) =•

IEEE INTERNET COMPUTING http://computer.org/internet/ JANUARY • FEBRUARY 2003 77

Amazon.com Recommendations

tomer base. The complex and expensive clusteringcomputation is run offline. However, recommen-dation quality is low.1 Cluster models groupnumerous customers together in a segment, matcha user to a segment, and then consider all cus-tomers in the segment similar customers for thepurpose of making recommendations. Because thesimilar customers that the cluster models find arenot the most similar customers, the recommenda-tions they produce are less relevant. It is possibleto improve quality by using numerous fine-grained segments, but then online user–segmentclassification becomes almost as expensive as find-ing similar customers using collaborative filtering.

Search-Based MethodsSearch- or content-based methods treat the rec-ommendations problem as a search for relateditems.8 Given the user’s purchased and rateditems, the algorithm constructs a search query tofind other popular items by the same author,artist, or director, or with similar keywords orsubjects. If a customer buys the Godfather DVDCollection, for example, the system might recom-mend other crime drama titles, other titles star-ring Marlon Brando, or other movies directed byFrancis Ford Coppola.

If the user has few purchases or ratings, search-based recommendation algorithms scale and per-

form well. For users with thousands of purchases,however, it’s impractical to base a query on all theitems. The algorithm must use a subset or summa-ry of the data, reducing quality. In all cases, rec-ommendation quality is relatively poor. The rec-ommendations are often either too general (suchas best-selling drama DVD titles) or too narrow(such as all books by the same author). Recom-mendations should help a customer find and dis-cover new, relevant, and interesting items. Popu-lar items by the same author or in the same subjectcategory fail to achieve this goal.

Item-to-ItemCollaborative FilteringAmazon.com uses recommendations as a targetedmarketing tool in many email campaigns and onmost of its Web sites’ pages, including the high-traffic Amazon.com homepage. Clicking on the“Your Recommendations” link leads customers to anarea where they can filter their recommendations byproduct line and subject area, rate the recommendedproducts, rate their previous purchases, and see whyitems are recommended (see Figure 1).

As Figure 2 shows, our shopping cart recom-mendations, which offer customers product sug-gestions based on the items in their shopping cart.The feature is similar to the impulse items in asupermarket checkout line, but our impulse itemsare targeted to each customer.

Amazon.com extensively uses recommendationalgorithms to personalize its Web site to each cus-tomer’s interests. Because existing recommendationalgorithms cannot scale to Amazon.com’s tens ofmillions of customers and products, we developedour own. Our algorithm, item-to-item collaborativefiltering, scales to massive data sets and produceshigh-quality recommendations in real time.

How It Works Rather than matching the user to similar cus-tomers, item-to-item collaborative filtering match-es each of the user’s purchased and rated items tosimilar items, then combines those similar itemsinto a recommendation list.9

To determine the most-similar match for a givenitem, the algorithm builds a similar-items table byfinding items that customers tend to purchasetogether. We could build a product-to-productmatrix by iterating through all item pairs and com-puting a similarity metric for each pair. However,many product pairs have no common customers,and thus the approach is inefficient in terms ofprocessing time and memory usage. The following

78 JANUARY • FEBRUARY 2003 http://computer.org/internet/ IEEE INTERNET COMPUTING

Industry Report

Figure 1. The “Your Recommendations” feature on the Amazon.comhomepage.Using this feature, customers can sort recommendationsand add their own product ratings.

Figure 2. Amazon.com shopping cart recommendations. The recom-mendations are based on the items in the customer’s cart: ThePragmatic Programmer and Physics for Game Developers.

iterative algorithm provides a better approach bycalculating the similarity between a single prod-uct and all related products:

For each item in product catalog, I1For each customer C who purchased I1

For each item I2 purchased by

customer C

Record that a customer purchased I1and I2

For each item I2Compute the similarity between I1 and I2

It’s possible to compute the similarity between twoitems in various ways, but a common method is touse the cosine measure we described earlier, in whicheach vector corresponds to an item rather than acustomer, and the vector’s M dimensions correspondto customers who have purchased that item.

This offline computation of the similar-itemstable is extremely time intensive, with O(N2M) asworst case. In practice, however, it’s closer toO(NM), as most customers have very few purchas-es. Sampling customers who purchase best-sellingtitles reduces runtime even further, with littlereduction in quality.

Given a similar-items table, the algorithm findsitems similar to each of the user’s purchases andratings, aggregates those items, and then recom-mends the most popular or correlated items. Thiscomputation is very quick, depending only on thenumber of items the user purchased or rated.

Scalability: A ComparisonAmazon.com has more than 29 million customersand several million catalog items. Other majorretailers have comparably large data sources.While all this data offers opportunity, it’s also acurse, breaking the backs of algorithms designedfor data sets three orders of magnitude smaller.Almost all existing algorithms were evaluated oversmall data sets. For example, the MovieLens dataset4 contains 35,000 customers and 3,000 items,and the EachMovie data set3 contains 4,000 cus-tomers and 1,600 items.

For very large data sets, a scalable recommen-dation algorithm must perform the most expensivecalculations offline. As a brief comparison shows,existing methods fall short:

• Traditional collaborative filtering does little orno offline computation, and its online compu-tation scales with the number of customers andcatalog items. The algorithm is impractical on

large data sets, unless it uses dimensionalityreduction, sampling, or partitioning — all ofwhich reduce recommendation quality.

• Cluster models can perform much of the com-putation offline, but recommendation qualityis relatively poor. To improve it, it’s possible toincrease the number of segments, but thismakes the online user–segment classificationexpensive.

• Search-based models build keyword, category,and author indexes offline, but fail to providerecommendations with interesting, targetedtitles. They also scale poorly for customers withnumerous purchases and ratings.

The key to item-to-item collaborative filtering’sscalability and performance is that it creates theexpensive similar-items table offline. The algo-rithm’s online component — looking up similaritems for the user’s purchases and ratings — scalesindependently of the catalog size or the total num-ber of customers; it is dependent only on howmany titles the user has purchased or rated. Thus,the algorithm is fast even for extremely large datasets. Because the algorithm recommends highlycorrelated similar items, recommendation qualityis excellent.10 Unlike traditional collaborative fil-tering, the algorithm also performs well with lim-ited user data, producing high-quality recommen-dations based on as few as two or three items.

ConclusionRecommendation algorithms provide an effectiveform of targeted marketing by creating a person-alized shopping experience for each customer. Forlarge retailers like Amazon.com, a good recom-mendation algorithm is scalable over very largecustomer bases and product catalogs, requires onlysubsecond processing time to generate online rec-ommendations, is able to react immediately tochanges in a user’s data, and makes compellingrecommendations for all users regardless of thenumber of purchases and ratings. Unlike otheralgorithms, item-to-item collaborative filtering isable to meet this challenge.

In the future, we expect the retail industry tomore broadly apply recommendation algorithms fortargeted marketing, both online and offline. Whilee-commerce businesses have the easiest vehicles forpersonalization, the technology’s increased conver-sion rates as compared with traditional broad-scaleapproaches will also make it compelling to offlineretailers for use in postal mailings, coupons, andother forms of customer communication.

IEEE INTERNET COMPUTING http://computer.org/internet/ JANUARY • FEBRUARY 2003 79

Amazon.com Recommendations

IEEE INTERNET COMPUTING

References1. J.B. Schafer, J.A. Konstan, and J. Reidl, “E-Commerce Rec-

ommendation Applications,” Data Mining and KnowledgeDiscovery, Kluwer Academic, 2001, pp. 115-153.

2. P. Resnick et al., “GroupLens: An Open Architecture forCollaborative Filtering of Netnews,” Proc. ACM 1994 Conf.Computer Supported Cooperative Work, ACM Press, 1994,pp. 175-186.

3. J. Breese, D. Heckerman, and C. Kadie, “Empirical Analy-sis of Predictive Algorithms for Collaborative Filtering,”Proc. 14th Conf. Uncertainty in Artificial Intelligence, Mor-gan Kaufmann, 1998, pp. 43-52.

4. B.M. Sarwarm et al., “Analysis of Recommendation Algo-rithms for E-Commerce,” ACM Conf. Electronic Commerce,ACM Press, 2000, pp.158-167.

5. K. Goldberg et al., “Eigentaste: A Constant Time Collabo-rative Filtering Algorithm,” Information Retrieval J., vol.4, no. 2, July 2001, pp. 133-151.

6. P.S. Bradley, U.M. Fayyad, and C. Reina, “Scaling ClusteringAlgorithms to Large Databases,” Knowledge Discovery andData Mining, Kluwer Academic, 1998, pp. 9-15.

7. L. Ungar and D. Foster, “Clustering Methods for Collabo-rative Filtering,” Proc. Workshop on Recommendation Sys-tems, AAAI Press, 1998.

8. M. Balabanovic and Y. Shoham, “Content-Based Collabora-tive Recommendation,” Comm. ACM, Mar. 1997, pp. 66-72.

9. G.D. Linden, J.A. Jacobi, and E.A. Benson, CollaborativeRecommendations Using Item-to-Item Similarity Mappings,US Patent 6,266,649 (to Amazon.com), Patent and Trade-mark Office, Washington, D.C., 2001.

10. B.M. Sarwar et al., “Item-Based Collaborative Filtering Rec-ommendation Algorithms,” 10th Int’l World Wide WebConference, ACM Press, 2001, pp. 285-295.

Greg Linden was cofounder, researcher, and senior manager in

the Amazon.com Personalization Group, where he designed

and developed the recommendation algorithm. He is cur-

rently a graduate student in management in the Sloan Pro-

gram at Stanford University’s Graduate School of Business.

His research interests include recommendation systems, per-

sonalization, data mining, and artificial intelligence. Linden

received an MS in computer science from the University of

Washington. Contact him at Linden_Greg@gsb.stanford.edu.

Brent Smith leads the Automated Merchandising team at Ama-

zon.com. His research interests include data mining, machine

learning, and recommendation systems. He received a BS in

mathematics from the University of California, San Diego,

and an MS in mathematics from the University of Washing-

ton, where he did graduate work in differential geometry.

Contact him at smithbr@amazon.com.

Jeremy York leads the Automated Content Selection and Deliv-

ery team at Amazon.com. His interests include statistical

models for categorical data, recommendation systems, and

optimal choice of Web site display components. He

received a PhD in statistics from the University of Wash-

ington, where his thesis won the Leonard J. Savage award

for best thesis in applied Bayesian econometrics and sta-

tistics. Contact him at jeremy@amazon.com.

AD

VE

RT

ISE

R /

PR

OD

UC

T I

ND

EX

J A N U A R Y / F E B R U A R Y 2 0 0 3

John Wiley & Sons Inside Back Cover

CTIA Wireless Back Cover

Advertising Personnel

Advertiser / Product Page Number

Marion DelaneyIEEE Media, Advertising DirectorPhone:+1 212 419 7766Fax: +1 212 419 7589Email: md.ieeemedia@ieee.org

Marian AndersonAdvertising CoordinatorPhone:+1 714 821 8380Fax: +1 714 821 4010Email: manderson@computer.org

Sandy BrownIEEE Computer Society,Business Development ManagerPhone:+1 714 821 8380Fax: +1 714 821 4010Email: sb.ieeemedia@ieee.org

Debbie SimsAssistant Advertising CoordinatorPhone:+1 714 821 8380Fax: +1 714 821 4010Email: dsims@computer.org

Advertising Sales Representatives

Mid Atlantic (product/recruitment)Dawn BeckerPhone: +1 732 772 0160Fax: +1 732 772 0161Email: db.ieeemedia@ieee.org

Midwest (product)Dave JonesPhone: +1 708 442 5633Fax: +1 708 442 7620Email: dj.ieeemedia@ieee.orgWill HamiltonPhone: +1 269 381 2156Fax: +1 269 381 2556Email: wh.ieeemedia@ieee.orgJoe DiNardoPhone: +1 440 248 2456Fax: +1 440 248 2594Email: jd.ieeemedia@ieee.org

Connecticut (product)Stan GreenfieldPhone: +1 203 938 2418Fax: +1 203 938 3211Email: greenco@optonline.net

Northwest (product)John GibbsPhone: +1 415 929 7619Fax: +1 415 577 5198Email: jg.ieeemedia@ieee.org

Midwest (product)David KovacsPhone: +1 847 705 6867Fax: +1 847 705 6878Email: dk.ieeemedia@ieee.org Midwest/Southwest recruitment)

Tom WilcoxenPhone: +1 847 498 4520Fax: +1 847 498 5911Email: tw.ieeemedia@ieee.org

New England (recruitment)Barbara LynchPhone: +1 401 738 6237Fax: +1 401 739 7970Email: bl.ieeemedia@ieee.org

Northwest (recruitment)Mary TononPhone: +1 415 431 5333Fax: +1 415 431 5335Email: mt.ieeemedia@ieee.org

Southern CA (recruitment)Tim MattesonPhone: +1 310 836 4064Fax: +1 310 836 4067Email: tm.ieeemedia@ieee.org

JapanGerman TajiriPhone: +81 42 501 9551Fax: +81 42 501 9552Email: gt.ieeemedia@ieee.org

Europe (product) Hilary TurnbullPhone: +44 131 660 6605Fax: +44 131 660 6989Email: impress@impressmedia.com

Southeast (product/recruitment)C. William Bentz IIIEmail: bb.ieeemedia@ieee.orgGregory MaddockEmail: gm.ieeemedia@ieee.orgSarah K. WileyEmail: sh.ieeemedia@ieee.orgPhone: +1 404 256 3800Fax: +1 404 255 7942

Southwest (product)Royce HousePhone: +1 713 668 1007Fax: +1 713 668 1176Email: rh.ieeemedia@ieee.org

Southern CA (product)Marshall RubinPhone: +1 818 888 2407Fax: +1 818 888 4907Email: mr.ieeemedia@ieee.org

New England (product)Jody EstabrookPhone: +1 978 244 0192Fax: +1 978 244 0103Email: je.ieeemedia@ieee.org