HadoopEdit by Cassell Hsu

2013.04.19

Hadoop

Master

Slave

Slave

Slave

Hadoop•Master•NameNode•JobTracker•SecondaryNameNode

NameNode•Where is NameNode?•Master•HDFS(Hadoop Distributed File

System)•What is NameNode?•資料之位置資訊 •資料之屬性

？？

NameNode

•位置資訊 ?•所有資料皆存放在 – DataNode

DataNode•What is DataNode•存放資料

•Where is DataNode•HDFS•Slaves (and Master)

User

DataNode

NameNode

DataNode

DataNode

128Mb

B64Mb

A64Mb

Check hdfs-site.xml

B64Mb

A64Mb

Hadoop

DataNodeNameNode

MasterDataNode

DataNode

DataNode

Slaves

Hadoop•Master•NameNode•JobTracker•SecondaryNameNode

JobTracker•What is JobTracker?•排程工作

•Where is JobTracker?•Master

誰來工作？

JobTracker & TaskTrackerJobTracker TaskTracker

Where Master Slaves

What 排程工作 執行工作

Hadoop

DataNodeNameNode

MasterDataNode

Slaves

JobTracker

TaskTracker

Hadoop•Master•NameNode•JobTracker•SecondaryNameNode

SecondaryNameNode

•What is SecondaryNameNode?•NameNode 發生錯誤時補救

•Where is SecondaryNameNode?•Master

Hadoop

DataNodeNameNode

MasterDataNode

Slaves

JobTracker

TaskTracker

SecondaryNode

MapReduce

User Master

Slave2

Slave1

A

A1

A2Task

NameNode

Task

Result1Result2

ReduceFinal

ResultHDFS

MapReduce

•檔案切割•Hadoop 上區塊切割•程式指定