dwivedishashwat@gmail.comhttp://helpmetocode.blogspot.com

Well designed tables PartitioningBucketingand well written queries can improve your query speed

and reduce processing cost.

Optimization on Table sidePartitioning Hive Tables:

It is a kind of horizontal slicing of data. This slicing can be on the range, single value or a set of values.

Imagine log files where each record includes a timestamp. If we partitioned by date, then records for the same date would be stored in the same partition.

E.g.: Partition on date. Partition on geography location. Partition on number range.

Defining a table partitionLets take a Apache log file example where we have log

generated by web server on visit of client.These log contains data & time information about browser and

location(IP).So we can create table in hive and partition these log data

using date & time and we can create sub partition of location. Which looks like :

CREATE TABLE alogs (timstamp BIGINT, detail STRING) PARTITIONED BY (date STRING, loc STRING);Lo

g Ta

ble

Directory Structure

/user/hive/warehouse/logs/dt=2010-01-01/country=GB/file1

/file2

/country=US/file3

/dt=2010-01-02/country=GB/file4

/country=US/file5

/file6

Hive Buckets

Bucketing Hive Tables:Bucketing hive table result in more efficient queries.

Bucketing imposes extra structure on the table, which Hive can take advantage of when performing certain queries.

It makes sampling more efficient. The two tables are bucketed in the same way, a mapper

processing a bucket of the left table knows that the matching rows in the right table are in its corresponding bucket, so it need only retrieve that bucket.

Bucket may additionally be sorted by one or more columns. This allows even more efficient map-side joins, since the join of each bucket becomes an efficient merge-sort.

Parallel execution of queries Hadoop can execute map reduce jobs in parallel and several queries executed on

Hive make automatically use of this parallelism. The queries or sub queries which are not interdependent can be execute in

parallel mode,like some Join queries. Following is the example how it is done: SET hive.exce.parallel=true; #Can be used to set this mode on

Main Query

Sub query 1

Sub query 2

Sub query 3

Sub query (1 & 2) Joined

Query (1 & 2) & 3 Joined Join

Join

Final Result1

2

3

4

5

Misc

So in the above flow, 1,2,4 can run in parallel as sub queries and then joined finally to 3 and then to 5 and the final query result.

Since map join is faster than the common join, it's better to run the map join whenever possible. Previously, Hive users needed to give a hint in the query to specify the small table.

For example, select /*+mapjoin(a)*/ * from src1 x join src2 y on

x.key=y.key;

Newer hive automatically converts normal join to map join.

Some examples

Which query is faster?

Select count(distinct(column)) from table.

Or

Select count(*) from (select distinct(column) from table) ??

AnswerM

R

M M

Result

M M M

R R R

M M M

R

Result

2nd one is faster

In first case :Maps send each value to reducerSingle reducer counts them all(over head)

In Second Case:Map splits the values to many reducerEach reducer generated a listFinal job is to count the size of each list

Note : Singleton reducer is not always good.

Tips

Hive does not know whether query is bad.

So try to use “Explain” for queries which you doubt to be bad or even don’t doubt.

Explain tells about followingNumber of jobsNumber of map and reduceWhat job is sorting byWhat are the directories it will read.

So explain will help to see the difference between the two or more queries for the same purpose.

Job configuration and history can be studied for the query performance.