MapReduce Algorithm Design

Chapter 3

Why this Chapter?

• Everything you didn’t think could be done with MapReduce

Programmer

• Only implements mapper, reducer• Optional: combiner, partitioner• Must express in a small number of rigidly

defined components• Trickiest aspect:

– Synchronization• Shuffle/short, otherwise everything runs in isolation in

MR

But, user can

• Construct complex data structures as keys, values• Execute user-specified initialization/ termination code• Control sort order of intermediate keys• Control partitioning of keys, ie. Set of keys

encountered by particular reducer• May need more than a single job, sequence of jobs

– Iteration– Convergence – may need drivers (counters)

• Preserve state in mappers/reducers – really??

Local Aggregation

• Hadoop writes intermediate results to local disk, then sends over network

• Can have lots of data transmitted to reducers• Instead use Combiners and in-Mapper

combining• Reduce amount to shuffle between M and R

– Without combiners, as many intermediate values as input (k,v)

Combiners

• Combiners – mini-Reducers (implemented in Map)– Input to combiner is output of Mapper – no

sending over network– Combiners run on same machine as Mapper– Output of combiner is much smaller than input

Correctness

• Combiners can reduce running time, must take care to be correct

• Example:– Compute mean of all integers associated with a

key• E.g. key – user ids, value - elapsed time for session• Compute mean on per-user basis• Requires lots of shuffling

Use combiner?

– Requires lots of shuffling– Solution?– Can use mini-REDUCER as combiner:

• But, Mean(1,2,3,4,5) != Mean(Mean(1,2), Mean(3,4,5))

New - Complex Types for v

• Add a combiner– Computes sum of values and count number

encountered– Sum, count packaged as a pair for v, emitted as

output with same key– This means v can be a complex type

• Don’t have to be ints, strings

• Reducer must reduce pairs of partial sums and counts

The Rules

• Combiners must have same input/output type for (k,v)

• Combiner (k,v) type must be same as output of Mapper, input to Reduce

• Reduce output can be different type than input• Computing the mean problem:

– Output of Map is integer, reducer expects list of pairs, not integers – problem!

– Expect combiner to run exactly once• Combiner can run 0 or more times, don’t know – problem!

Fixing the problems

• Mapper must emit: (k, (int, 1)) – Partial count

• Combiner aggregates sum and counts, emits• Reducer computes mean• What if no combiner run• Still correct?

ReduceCopier.fetchOutputs() method is the one that holds the reduce task from running until all map outputs are copied (through the while loop in line 2026 of Hadoop release 1.0.4).

In the MapReduce model, you need to wait for all mappers to finish, since the keys need to be grouped and sorted; plus, you may have some speculative mappers running and you do not know yet which of the duplicate mappers will finish first.

However, as the "Breaking the MapReduce Stage Barrier" paper indicates, for some applications, it may make sense not to wait for all of the output of the mappers. If you would want to implement this sort of behavior (most likely for research purposes), then you should take a look at theorg.apache.hadoop.mapred.ReduceTask.ReduceCopier class, which implements ShuffleConsumerPlugin.

the reduce task in hadoop contains three steps: shuffle, sort and reduce where the sort (and after that the reduce) can only start once all the mappers are done.

New! - Associative ArraysAKA In-mapper combiner

• How to aggregate for word count in the same Mapper?

• Compute partial count of words processed by Mapper

• How do to this?– Include associative array

• Tallies up counts within single document• (k,v) for each unique word, not just (word, 1)

New - Yes, you can preserve state

• With associative array• In Hadoop, mapper object created for each map task

– first call Initialize (API Hook) for associative array for term counts,

– can preserve state across multiple calls of Map method, – can compute counts across multiple documents, until Close– Apply to one map task– Incorporate combiner into Mapper– Why?

Very efficient

Computing Mean - associative arrays

• How to use associate arrays for this?– Must user combiner if no associate array

Preserving State

• But – violates functional programming paradigm of MR

• Algorithm behavior may depend on order of (k,v) input

• Scalability problems – depend on memory to store intermediate results– Solution – can emit intermediate results after every n

(k,v) or after certain memory usage• Can reduce straggler problem

Information Retrieval with MapReduce

Chapter 4

Some slides from Jimmy Lin

Topics

• Introduction to IR• Inverted Indexing• Retrieval

Web search problems• Web search problems decompose into three

components1. Gather web content (crawling)2. Construct inverted index based on web content

gathered (indexing)3. Ranking documents given a query using inverted

index (retrieval)

Web search problems• Crawling and creating indexes share similar

characteristics and requirements– Both are offline problems, no need for real-time– Tolerable for a few minutes delay before content

searchable– OK to run smaller-scale index updates frequently

• Querying online problem – Demands sub-second response time– Low latency high throughput– Loads can very greatly

Web Crawler

• To acquire the document collection over which indexes are built

• Acquiring web content requires crawling– Traverse web by repeatedly following hyperlinks

and storing downloaded pages– Start by populating a queue with seed pages

Web Crawler Issues

• Shouldn’t overload web servers with crawling• Prioritize order in which unvisited pages downloaded• Avoid downloading page multiple times

– coordination and load balancing• Robust when failures• Learn update patterns so content current• Identify near duplicates and select best for index• Identify dominant language on page

How do we represent text?• Remember: computers don’t “understand” anything!• “Bag of words”

– Treat all the words in a document as index terms for that document

– Assign a “weight” to each term based on “importance”– Disregard order, structure, meaning, etc. of the words– Simple, yet effective!

• Assumptions– Term occurrence is independent– Document relevance is independent– “Words” are well-defined

What’s a word?天主教教宗若望保祿二世因感冒再度住進醫院。這是他今年第二度因同樣的病因住院。 - باسم الناطق ريجيف مارك وقال

قبل - شارون إن اإلسرائيلية الخارجيةبزيارة األولى للمرة وسيقوم الدعوة

المقر طويلة لفترة كانت التي تونس،لبنان من خروجها بعد الفلسطينية التحرير لمنظمة الرسمي

1982عام . Выступая в Мещанском суде Москвы экс-глава ЮКОСа заявил не совершал ничего противозаконного, в чем обвиняет его генпрокуратура России.

भारत सरकार ने आर्थि �क सर्वे�क्षण में विर्वेत्तीय र्वेर्ष� 2005-06 में सात फ़ीसदी विर्वेकास दर हासिसल करने का आकलन विकया है और कर सुधार पर ज़ोर दिदया है

日米連合で台頭中国に対処…アーミテージ前副長官提言

조재영 기자 = 서울시는 25 일 이명박 시장이 ` 행정중심복합도시 '' 건설안에 대해 ` 군대라도 동원해 막고싶은 심정 '' 이라고 말했다는 일부 언론의 보도를 부인했다 .

Sample DocumentMcDonald's slims down spudsFast-food chain to reduce certain types of fat in its french fries with new cooking oil.NEW YORK (CNN/Money) - McDonald's Corp. is cutting the amount of "bad" fat in its french fries nearly in half, the fast-food chain said Tuesday as it moves to make all its fried menu items healthier.But does that mean the popular shoestring fries won't taste the same? The company says no. "It's a win-win for our customers because they are getting the same great french-fry taste along with an even healthier nutrition profile," said Mike Roberts, president of McDonald's USA.But others are not so sure. McDonald's will not specifically discuss the kind of oil it plans to use, but at least one nutrition expert says playing with the formula could mean a different taste.Shares of Oak Brook, Ill.-based McDonald's (MCD: down $0.54 to $23.22, Research, Estimates) were lower Tuesday afternoon. It was unclear Tuesday whether competitors Burger King and Wendy's International (WEN: down $0.80 to $34.91, Research, Estimates) would follow suit. Neither company could immediately be reached for comment.…

16 × said 14 × McDonalds12 × fat11 × fries8 × new6 × company, french, nutrition5 × food, oil, percent, reduce,

taste, Tuesday…

“Bag of Words”

Representing Documents

The quick brown fox jumped over the lazy dog’s back.

Document 1

Document 2

Now is the time for all good men to come to the aid of their party.

the

isfor

to

of

quick

brown

fox

over

lazy

dog

back

now

time

all

good

men

come

jump

aid

their

party

00110110110010100

11001001001101011

Term Docu

men

t 1

Docu

men

t 2

Stopword List

Inverted Index• Inverted indexing is fundamental to all IR models• Consists of postings lists, one with each term in the

collection• Posting list – document id and payload

– Payload can be term frequency or number of times occurs on document, position of occurrence, properties, etc.

– Can be ordered by document id, page rank, etc.– Data structure necessary to map from document

id to for example, URL

Inverted Index

quick

brown

fox

over

lazy

dog

back

now

time

all

good

men

come

jump

aid

their

party

00110000010010110

01001001001100001

Term

Doc

1Do

c 2

00110110110010100

11001001001000001

Doc

3Do

c 4

00010110010010010

01001001000101001

Doc

5Do

c 6

00110010010010010

10001001001111000

Doc

7Do

c 8

quick

brown

fox

over

lazy

dog

back

now

time

all

good

men

come

jump

aid

their

party

4 82 4 61 3 71 3 5 72 4 6 83 53 5 72 4 6 831 3 5 7

1 3 5 7 8

2 4 82 6 8

1 5 72 4 6

1 36 8

Term Postings

Process query - retrieval

• Given a query, fetch posting lists of index associated with query, traverse postings to compute result set

• Query document scores must be computed• Top k documents extracted• Optimization strategies to reduce # postings

must examine

Index

• Size of index depends on payload• Well-optimized inverted index can be 1/10 of size of

original document collection• If store position info, could be several times larger• Usually can hold entire vocabulary in memory (using

front-coding)• Postings lists usually too large to store in memory• Query evaluation involves random disk access and

decoding postings– Try to minimize random seeks

Indexing: Performance Analysis

• Creating indexes problem– Must be relatively fast, but need not be real time– For Web, incremental updates are important

• How large is the inverted index?– Size of vocabulary– Size of postings

• Fundamentally, a large sorting problem– Terms usually fit in memory– Postings usually don’t

Vocabulary Size: Heaps’ Law

KnV V is vocabulary sizen is corpus size (number of documents)K and are constants, determineempirically

Typically, K is between 10 and 100, is between 0.4 and 0.6

When adding new documents, the system is likely to have seen most terms already… but the postings keep growing

Heaps' law means that as more instance text is gathered, there will be diminishing returns in terms of discovery of the full vocabulary from which the distinct terms are drawn.

Postings Size: Zipf’s Law• George Kingsley Zipf (1902-1950) observed the following relation

between frequency and rank

• In other words:– A few elements occur very frequently– Many elements occur very infrequently

• Zipfian distributions:– English words– Library book checkout patterns– DB queries– Website popularity (almost anything on the Web)

crf or

rcf f = frequency

r = rankc = constant

Word Frequency in Englishthe 1130021 from 96900 or 54958of 547311 he 94585 about 53713to 516635 million 93515 market 52110a 464736 year 90104 they 51359in 390819 its 86774 this 50933and 387703 be 85588 would 50828that 204351 was 83398 you 49281for 199340 company 83070 which 48273is 152483 an 76974 bank 47940said 148302 has 74405 stock 47401it 134323 are 74097 trade 47310on 121173 have 73132 his 47116by 118863 but 71887 more 46244as 109135 will 71494 who 42142at 101779 say 66807 one 41635mr 101679 new 64456 their 40910with 101210 share 63925

Frequency of 50 most common words in English (sample of 19 million words)

Question to consider

• How to create an inverted index using map reduce?

MapReduce: Index Construction• Map over all documents – does what?

– What is input? • Docid and content

– What is output?• Emit postings: k= term ; v = (docid, frequency) • Emit other information as necessary (e.g., term position)

• Reduce – does what?– What is input?

• (term, (docid, frequency))– What is output?

• Trivial: combines list of postings• Might want to sort the postings (e.g., by docid or term_frequency)

for group by

MapReduce

• Map emits term postings• Reduce performs large distributed group by of

postings by term

1: class Mapper2: procedure Map(docid n; doc d)3: H ← new AssociativeArray4: for all terms t € doc d do5: H{t} ← H{t} + 16: for all terms t € H do7: Emit(term t, posting (n, H{t}))1: class Reducer2: procedure Reduce(term t, postings [(n1, f1), (n2, f2) : : :])3: P ← new List4: for all posting (a, f) € postings [(n1, f1), (n2, f2) : : :] do5: Append(P, (a, f))6: Sort(P)7: Emit(term t; postings P)

Figure 4.2: Pseudo-code of the baseline inverted indexing algorithm in MapReduce. Mappers emit postings keyed by terms, the execution framework groups postings by term, and the reducers write postings lists to disk.

Creating an inverted index – Map

• Input is docids and content• Documents processed in parallel by mappers• Document analyzed and broken down into

component terms• May have to strip off HTML tags, JavaScript code,

remove stop words• Iterate over all terms and keep track of counts• Emit (term, posting) where posting is docid and

payload

Creating Inverted Index - Reduce

• After shuffle sort, Reduce is easy – Just combines individual postings by appending to

initially empty list and writes to disk– Usually compresses list

• How would this be done without MapReduce?

Retrieval

• Have just showed how to create an index, but what about retrieving info for a query?

• What is involved:– Identify query terms– Look up posting lists corresponding to query terms– Traverse posting lists to compute query-document

scores– Return top k results

Retrieval

• Must have sub-second response• Optimized for low-latency• True of MapReduce?• True of underlying file system• Issues?

– Look up postings– Postings too large to fit in memory

• (random disk seeks)– Round-trip network exchanges– Must find location of data block from master, etc.

Issues

• Issues?– Look up postings to find terms in documents– Postings too large to fit in memory

• (random disk seeks)– Round-trip network exchanges

Solution?

• Distributed Retrieval

Distributed Retrieval

• Distribute retrieval across large number of machines

• Break up index – how?• What does it look like?

• Document -chunks (cols) and terms (rows)• Document partitioning• Term partitioning• Need Query Broker and Partition Servers

Partitioning• Document broken up into partitions and assigned to a server

– Document partitioning• Servers hold complete index for a subset of entire collection • Vertical partitioning

– Term partitioning• Each server responsible for subset of terms for entire collection• Horizontal partitioning

– Sharding is a synonym for horizontal partitioning

– Google guy uses word sharding for partitioning (any type)

Term

Document

Document Partitioning• Suppose query contains 3 terms, t1, t2 and t3 • Document Partitioning

– Query broker forwards query to all partition servers– Each server must search for t1, t2, t3 in its subset of

documents– Each server sends partial results to query broker– Query broker merges partial results from each– Query broker returns final result to user

Term Partitioning• Suppose query contains 3 terms, t1, t2 and t3• Term (word) Partitioning

– Typical strategy Uses pipelined strategy – accumulators and route:

• Broker forwards query to server that holds postings for t1 (least frequent term)

• Server traverses postings list, computes partial query scores• Partial scores sent to server with t2 – only has to examine

documents listed in postings list that have t1• Partial scores sent to server with t3 – examines only

documents in which t1 and t2 occur– Then final result passed to query broker

Term Partitioning

• What if did in parallel?– Each term would be processed at a different

server– Each server would have to traverse all documents

and not just those in which previous terms exist – All would have to be combined at the end

Document Partitioning

• Pros/Cons?– Query must be processed by every server– Each partition operates independently, traverse postings in

parallel• shorter query latencies

• Assuming K words, N partitions (shards):• Number of disk accesses - Big O?

– O(K*N)

Term Partitioning• Pros/Cons

– Increases throughput - smaller total # of seeks per query– Better if memory limited system– Load balancing tricky - may create hotspots on servers– Requires higher bandwidth – info spread across machines– Need data in one place about each document – no obvious

home• Number of disk accesses - Big O?

– O(K) K words, at most K partitions (shards) – why?

• only have to search for one term and in documents in which previous terms occurred, maximum documents

Which is best?

• Google adopts which partitioning?– Document partitioning – less bandwidth– Keeps indexes in memory (not commonly done in

search engines)– Quality degrades with machine failure– Servers offline won’t deliver results

• User doesn’t know• Users less discriminating as to which relevant documents

appear in results• Even is no failure may cycle through different partitions

Other aspects of Partitioning

• Can partition documents along a dimension, e.g. by quality

• Search highest quality first• Search lower quality if needed• Include another dimension, e.g. content• Only send queries to partitions likely to contain

relevant documents instead of to all partitions

Replication and Partitioning

• Replication?– Within same partition as well as across

geographically distributed data centers– Query routing problems

• Serve clients from the closest data centers• Must also route to appropriate locations• If single data center, must balance load across replicas

Displaying to User• Postings only store docid• Get a list of ranked docids• Responsibility of document servers (not

partition servers) to generate meaningful output– Title, URL, snippet of result entry

• Caching useful if index not in memory and can even cache results of entire queries– Why cache results?– Zipfian!!

Summary

1. Documents gathered by web crawling2. Inverted indexes built from documents3. Use indexes to respond to queries

• More info about Google’s approach 2009• Jeff Dean 8:06 - http

://videolectures.net/wsdm09_dean_cblirs/