Upload
jessie-sutton
View
222
Download
2
Tags:
Embed Size (px)
Citation preview
Cloud Computing
Other High-level parallel processing languages
Keke Chen
Outline sawzall Dryad and DraydLINQ (MS, abandoned) Hive
Sawzall Simplify mapreduce programming Filters + aggregator
mapper reducer
Example
mappers
reducers
Convert the input record to float
input Sawzall program works on a single
record As a filter filtering through the data stream
Input can be parsed to Values, e.g., float Data structurex: float = input;(variable : type = input)
aggregators definition
table agg_name of data_type/variable
Examples: c: table collection of string; S: table sample(100) of string; S: table sum of {count: int, revenue: float}
More aggregators Maximum, quantile, top, unique
Indexed aggregators similar to “group by”, the index is group
id Example
t1: table sum[country: string] of intcountry: string = input;Emit t1[country] <- 1;
More example
Proto “querylog.proto”queries_per_degree: table sum[lat: int]
[lon:int] of int;Log_record: queryLogProto = input;Loc: Location = locationinfo(log_record.ip);Emit queries_per_degree[int(loc.lat)]
[int(loc.lon)]<-1
Performance
Single-CPU speed, Also 51 times slower than compiled C++
Performance
Dryad and DryadLINQ Dryad provides a low-level parallel data
flow processing interface Acyclic data flow graphs Data communication methods include pipes,
file-based, message, shared-memory
DryadLINQ A high level language for app developers It hides the data flow details
Job = Directed Acyclic Graph
Processingvertices Channels
(file, pipe, shared memory)
Inputs
Outputs
Runtime
Services Name server Daemon
Job Manager Centralized coordinating process User application to construct graph Linked with Dryad libraries for scheduling
vertices Vertex executable
Dryad libraries to communicate with JM User application sees channels in/out Arbitrary application code, can use local FS
V V V
Graph operators
Hive Developed by facebook (open source) Mimic SQL language Built on hadoop/mapreduce
Hive data model: table etc. Table
Similar to DB table stored in hadoop directories Builtin compression, serialization/deserialization
Partitions Groups in the table Subdirectory in the table directory
Buckets Files in the partition directory Key (column) based partition
/table/partition/bucket1
Hive data model: Column type integers, floating point numbers, generic
strings, dates and booleans nestable collection types: array and
map.
Architecture
Metastore stores the schema of databases. It uses non HDFSdata store
Query processing Steps (similar to DBMS)
Parse Semantic analyzer Logical plan generator (algebra tree) Optimizer Physical plan generator (to mapreduce jobs)
Operations: DDL and DML HiveQL: SQL like, with slightly different
syntax User defined filtering and aggregation
functions Java only
Map/reduce plugin for streaming process Implemented with any language
Example Facebook status updates
Table: status_updates(userid int, status string,ds string) profiles(userid int,school string,gender int)
Operations Load data
LOAD DATA LOCAL INPATH `/logs/status_updates‘ INTO TABLE status_updates PARTITION (ds='2009-03-20')
Count status updates by school and by gender
More query examples
Query examples
Query examples – using hadoopstreaming