Hive and Pig - VGCWiki juliana/courses/BigData2014/Lectures/hive...Hive and Pig! • Hive: data warehousing application in Hadoop • Query language is HQL, variant of SQL • Tables

  • View

  • Download

Embed Size (px)

Text of Hive and Pig - VGCWiki juliana/courses/BigData2014/Lectures/hive...Hive and Pig! • Hive: data...

  • Hive and Pig!

    Juliana FreireNew York University

    Some slides from J. Lin

  • Need for High-Level Languages!

    Hadoop is great for large-data processing! But writing Java programs for everything is verbose and slow Not everyone wants to (or can) write Java code

    Solution: develop higher-level data processing languages Hive: HQL is like SQL Pig: Pig Latin is a bit like Perl

  • Hive and Pig!

    Hive: data warehousing application in Hadoop Query language is HQL, variant of SQL Tables stored on HDFS as flat files Developed by Facebook, now open source

    Pig: large-scale data processing system Scripts are written in Pig Latin, a dataflow language Developed by Yahoo!, now open source Roughly 1/3 of all Yahoo! internal jobs

    Common idea: Provide higher-level language to facilitate large-data processing Higher-level language compiles down to Hadoop jobs

  • Hive: Background!

    Started at Facebook Data was collected by nightly cron jobs into Oracle DB ETL via hand-coded python Grew from 10s of GBs (2006) to 1 TB/day new data (2007), now 10x


    Source: cc-licensed slide by Cloudera




    (Extract, Transform, and Load)


  • Hive Components!

    Shell: allows interactive queries Driver: session handles, fetch,

    execute Compiler: parse, plan, optimize Execution engine: DAG of stages

    (MR, HDFS, metadata) Metastore: schema, location in

    HDFS, SerDe

    Source: cc-licensed slide by Cloudera

    standard input and write out rows to standard output. Thisflexibility does come at a cost of converting rows from andto strings.

    We omit more details due to lack of space. For a completedescription of HiveQL see the language manual [5].

    2.3 Running Example: StatusMemeWe now present a highly simplified application, Status-

    Meme, inspired by Facebook Lexicon [6]. When Facebookusers update their status, the updates are logged into flatfiles in an NFS directory /logs/status updates which are ro-tated every day. We load this data into hive on a daily basisinto a tablestatus updates(userid int,status string,ds string)

    using a load statement like below.

    LOAD DATA LOCAL INPATH /logs/status_updatesINTO TABLE status_updates PARTITION (ds=2009-03-20)

    Each status update record contains the user identifier (userid),the actual status string (status), and the date (ds) when thestatus update occurred. This table is partitioned on the dscolumn. Detailed user profile information, like the gender ofthe user and the school the user is attending, is available inthe profiles(userid int,school string,gender int) table.

    We first want to compute daily statistics on the frequencyof status updates based on gender and school which the userattends. The following multi-table insert statement gener-ates the daily counts of status updates by school (intoschool summary(school string,cnt int,ds string)) and gen-der (into gender summary(gender int,cnt int,ds string)) us-ing a single scan of the join of the status updates andprofiles tables. Note that the output tables are also parti-tioned on the ds column, and HiveQL allows users to insertquery results into a specific partition of the output table.

    FROM (SELECT a.status,, b.genderFROM status_updates a JOIN profiles b

    ON (a.userid = b.userid anda.ds=2009-03-20 )

    ) subq1INSERT OVERWRITE TABLE gender_summary

    PARTITION(ds=2009-03-20)SELECT subq1.gender, COUNT(1) GROUP BY subq1.genderINSERT OVERWRITE TABLE school_summary


    Next, we want to display the ten most popular memesper school as determined by status updates by users whoattend that school. We now show how this computationcan be done using HiveQLs map-reduce constructs. Weparse the result of the join between status updates andprofiles tables by plugging in a custom Python mapperscript which uses sophisticated naturallanguage processing techniques to extract memes from sta-tus strings. Since Hive does not yet support the rank ag-gregation function the top 10 memes per school can then becomputed by a simple custom Python reduce script

    REDUCE,, subq2.cntUSING AS (school,meme,cnt)

    FROM (SELECT,, COUNT(1) AS cntFROM (MAP, a.status

    USING AS (school,meme)

    Figure 1: Hive Architecture

    FROM status_updates a JOIN profiles bON (a.userid = b.userid)

    ) subq1GROUP BY, subq1.memeDISTRIBUTE BY school, memeSORT BY school, meme, cnt desc

    ) subq2;

    3. HIVE ARCHITECTUREFigure 1 shows the major components of Hive and its in-

    teractions with Hadoop. The main components of Hive are:

    External Interfaces - Hive provides both user inter-faces like command line (CLI) and web UI, and appli-cation programming interfaces (API) like JDBC andODBC.

    The Hive Thrift Server exposes a very simple clientAPI to execute HiveQL statements. Thrift [8] is aframework for cross-language services, where a serverwritten in one language (like Java) can also supportclients in other languages. The Thrift Hive clients gen-erated in dierent languages are used to build commondrivers like JDBC (java), ODBC (C++), and scriptingdrivers written in php, perl, python etc.

    The Metastore is the system catalog. All other com-ponents of Hive interact with the metastore. For moredetails see Section 3.1.

    The Driver manages the life cycle of a HiveQL state-ment during compilation, optimization and execution.On receiving the HiveQL statement, from the thriftserver or other interfaces, it creates a session handlewhich is later used to keep track of statistics like exe-

  • Data Model!

    Tables: analogous to tables in RDBMS Typed columns (int, float, string, boolean) Structs: {a INT; b INT}. Also, list, arrays : map (for JSON-like data)

    Partitions For example, range-partition tables by date

    Buckets Hash partitions within ranges (useful for sampling, join optimization)

    Source: cc-licensed slide by Cloudera

    Hive - A Warehousing Solution Over a Map-ReduceFramework

    Ashish Thusoo, Joydeep Sen Sarma, Namit Jain, Zheng Shao,Prasad Chakka, Suresh Anthony, Hao Liu, Pete Wyckoff and Raghotham Murthy

    Facebook Data Infrastructure Team

    1. INTRODUCTIONThe size of data sets being collected and analyzed in the

    industry for business intelligence is growing rapidly, mak-ing traditional warehousing solutions prohibitively expen-sive. Hadoop [3] is a popular open-source map-reduce im-plementation which is being used as an alternative to storeand process extremely large data sets on commodity hard-ware. However, the map-reduce programming model is verylow level and requires developers to write custom programswhich are hard to maintain and reuse.

    In this paper, we present Hive, an open-source data ware-housing solution built on top of Hadoop. Hive supportsqueries expressed in a SQL-like declarative language - HiveQL,which are compiled into map-reduce jobs executed on Hadoop.In addition, HiveQL supports custom map-reduce scripts tobe plugged into queries. The language includes a type sys-tem with support for tables containing primitive types, col-lections like arrays and maps, and nested compositions ofthe same. The underlying IO libraries can be extended toquery data in custom formats. Hive also includes a systemcatalog, Hive-Metastore, containing schemas and statistics,which is useful in data exploration and query optimization.In Facebook, the Hive warehouse contains several thousandtables with over 700 terabytes of data and is being used ex-tensively for both reporting and ad-hoc analyses by morethan 100 users.

    The rest of the paper is organized as follows. Section 2describes the Hive data model and the HiveQL languagewith an example. Section 3 describes the Hive system ar-chitecture and an overview of the query life cycle. Section 4provides a walk-through of the demonstration. We concludewith future work in Section 5.


    2.1 Data ModelData in Hive is organized into:

    Tables - These are analogous to tables in relational

    Permission to copy without fee all or part of this material is granted provided

    that the copies are not made or distributed for direct commercial advantage,

    the VLDB copyright notice and the title of the publication and its date appear,

    and notice is given that copying is by permission of the Very Large Data

    Base Endowment. To copy otherwise, or to republish, to post on servers

    or to redistribute to lists, requires a fee and/or special permission from the

    publisher, ACM.

    VLDB 09, August 24-28, 2009, Lyon, FranceCopyright 2009 VLDB Endowment, ACM 000-0-00000-000-0/00/00.

    databases. Each table has a corresponding HDFS di-rectory. The data in a table is serialized and stored infiles within that directory. Users can associate tableswith the serialization format of the underlying data.Hive provides builtin serialization formats which ex-ploit compression and lazy de-serialization. Users canalso add support for new data formats by defining cus-tom serialize and de-serialize methods (called SerDes)written in Java. The serialization format of each tableis stored in the system catalog and is automaticallyused by Hive during query compilation and execution.Hive also supports external tables on data stored inHDFS, NFS or local directories.

    Partitions - Each table can have one or more parti-tions which determine the distribution of data withinsub-directories of the table directory. Suppose datafor table T is in the directory /wh/T. If T is partitionedon columns ds and ctry, then data with a particulards value 20090101 and ctry value US, will be