7
BigData – MapReduce - YARN Syed Solutions Engineer - Big Data [email protected] [email protected] +91-9030477368

Map Reduce in Depth

Embed Size (px)

Citation preview

Page 1: Map Reduce in Depth

BigData – MapReduce - YARN

Syed

Solutions Engineer - Big Data

[email protected]

[email protected]

+91-9030477368

mailto:[email protected]
mailto:[email protected]
Page 2: Map Reduce in Depth
Page 3: Map Reduce in Depth

Hadoop MapReduce Data Flow

Page 4: Map Reduce in Depth

Hadoop 1 - Running Jobs

Rack1 Rack2 Rack3 RackN

Hadoop Client

JobTracker

DN | TT

DN | TT

DN | TT

DN | TT

DN | TT

DN | TT

DN | TT

DN | TT

DN | TT

DN | TT

DN | TT

DN | TT

submit job

deploy job

part 0

map

reduce

shuffle

Page 5: Map Reduce in Depth

Hadoop 2 - Running Jobs

RackN

NodeManager

NodeManager

NodeManager

Rack2

NodeManager

NodeManager

NodeManager

Rack1

NodeManager

NodeManager

NodeManager

C2.1

C1.4

AM2

C2.2 C2.3

AM1

C1.3

C1.2

C1.1

Hadoop Client 1

Hadoop Client 2

create app2

submit app1

submit app2

create app1

ASM Scheduler

queues

ASM Containers

NM ASM

Scheduler Resources

.......negotiates.......

.......reports to.......

.......partitions.......

ResourceManager

status report

Page 6: Map Reduce in Depth

Partitions

Page 7: Map Reduce in Depth

Thank you!

www.syedacademy.com

[email protected]

[email protected]

+91-9030477368

http://www.syedacademy.com/
mailto:[email protected]
mailto:[email protected]

CouchDB Map/Reduce

CouchDB Map/Reduce

Business
Map Reduce - cis.csuohio.educis.csuohio.edu/~sschung/cis612/MapReduceJoinAlgorithms.pdf · Map Reduce Basic Map Reduce Join Algorithms Improvements in performance by pre processing

Map Reduce - cis.csuohio.educis.csuohio.edu/~sschung/cis612/MapReduceJoinAlgorithms.pdf · Map Reduce Basic Map Reduce Join Algorithms Improvements in performance by pre processing

Documents
Hadoop map reduce

Hadoop map reduce

Data & Analytics
Main map reduce

Main map reduce

Documents
Topic Map Reduce

Topic Map Reduce

Documents
Reading Material Map Reduce The Map-Reduce Framework · Map-Reduce Steps • Input is typically (key, value) pairs –but could be objects of any type • Map and Reduce are performed

Reading Material Map Reduce The Map-Reduce Framework · Map-Reduce Steps • Input is typically (key, value) pairs –but could be objects of any type • Map and Reduce are performed

Documents
s07 Map Reduce

s07 Map Reduce

Documents
Map Reduce Ppt

Map Reduce Ppt

Documents
Elastic map reduce

Elastic map reduce

Technology
Map Reduce Problem

Map Reduce Problem

Documents
Map Reduce v1.1

Map Reduce v1.1

Documents
Map Reduce - Marquette Universitytschwarz.mscs.mu.edu/Classes/DataAtScale/Presentations/mapreduce2.pdf · Map Reduce in Detail • Reducer • Shuffle and Sort • Part of the map-reduce

Map Reduce - Marquette Universitytschwarz.mscs.mu.edu/Classes/DataAtScale/Presentations/mapreduce2.pdf · Map Reduce in Detail • Reducer • Shuffle and Sort • Part of the map-reduce

Documents
Map Reduce - unipi.itdidawiki.cli.di.unipi.it/.../cpa/aa1516/mapreduce_i.pdf · map reduce map map map map aggregate values by key reduce reduce. MCSN - N. Tonellotto - Distributed

Map Reduce - unipi.itdidawiki.cli.di.unipi.it/.../cpa/aa1516/mapreduce_i.pdf · map reduce map map map map aggregate values by key reduce reduce. MCSN - N. Tonellotto - Distributed

Documents
Map Reduce basics

Map Reduce basics

Data & Analytics
Occlusion-Aware Depth Map Coding Optimization Using ... · Occlusion-Aware Depth Map Coding Optimization Using Allowable Depth Map Distortions Pan Gao and Aljosa Smolic Abstract—In

Occlusion-Aware Depth Map Coding Optimization Using ... · Occlusion-Aware Depth Map Coding Optimization Using Allowable Depth Map Distortions Pan Gao and Aljosa Smolic Abstract—In

Documents
Erlang Map Reduce

Erlang Map Reduce

Documents
Map Reduce ~Continuous Map Reduce Design~

Map Reduce ~Continuous Map Reduce Design~

Technology
Map reduce

Map reduce

Technology
Map/Reduce Programming Model Ahmed Abdelsadek. Outlines Introduction What is Map/Reduce? Framework Architecture Map/Reduce Algorithm Design Tools and

Map/Reduce Programming Model Ahmed Abdelsadek. Outlines Introduction What is Map/Reduce? Framework Architecture Map/Reduce Algorithm Design Tools and

Documents
06 Map Reduce

06 Map Reduce

Technology
Joins Map Reduce

Joins Map Reduce

Documents
Map reduce presentation

Map reduce presentation

Data & Analytics
Exposicion Map Reduce

Exposicion Map Reduce

Documents
%% · Map’2 Reduce’0’ Reduce’1

%% · Map’2 Reduce’0’ Reduce’1

Documents
Cassandra&map reduce

Cassandra&map reduce

Technology
Map Reduce Programming

Map Reduce Programming

Documents
Big Data: A Storage Systems Perspective - Storage ...e.g. Map/Reduce) Shuffle Shuffle Shuffle Tasks (e.g. Map/Reduce) Tasks (e.g. Map/Reduce) Tasks (e.g. Map/Reduce) 2013 SNIA Analytics

Big Data: A Storage Systems Perspective - Storage ...e.g. Map/Reduce) Shuffle Shuffle Shuffle Tasks (e.g. Map/Reduce) Tasks (e.g. Map/Reduce) Tasks (e.g. Map/Reduce) 2013 SNIA Analytics

Documents
Map/Reduce intro

Map/Reduce intro

Technology
An Introduction to Spark Fishel... · 4 A brief review of MapReduce Map MapMap Map Map Map Map Map Map Reduce Reduce Reduce Reduce Key advances by MapReduce: • Data Locality: Automatic

An Introduction to Spark Fishel... · 4 A brief review of MapReduce Map MapMap Map Map Map Map Map Map Reduce Reduce Reduce Reduce Key advances by MapReduce: • Data Locality: Automatic

Documents
UNIT 3 : MAP REDUCE Introduction To Map Reduce Programs

UNIT 3 : MAP REDUCE Introduction To Map Reduce Programs

Documents