Crunching Data with Google BigQuery. JORDAN TIGANI at Big Data Spain 2012

Crunching Data with BigQuery Fast analysis of Big Data

Jordan Tigani, Software Engineer

01000001011011100111001101110111011001010111001

00010000001110100011011110010000001110100011010

00011001010010000001010101011011000111010001101

00101101101011000010111010001100101001000000101

00010111010101100101011100110111010001101001011

01111011011100010000001101111011001100010000001

00110001101001011001100110010100101100001000000

11101000110100001100101001000000101010101101110

01101001011101100110010101110010011100110110010

10010110000100000011000010110111001100100001000

00010001010111011001100101011100100111100101110

100101110011001000000011010000110010...........

Big Data at Google

72 hours

100 million gigabytes

SELECT

kick_ass_product_plan AS strategy,

AVG(kicking_factor) AS awesomeness

FROM

lots_of_data

GROUP BY

strategy

+-------------+----------------+

| strategy | awesomeness |

+-------------+----------------+

| "Forty-two" | 1000000.01 |

+-------------+----------------+

1 row in result set (10.2 s)

Scanned 100GB

Regular expressions on 13 billion rows...

13 Billion rows

1 TB of data in 4 tables

FAST! AST

Google's Internal Technology:

Dremel

MapReduce is Flexible but Heavy

Master

Mapper Mapper

• Master constructs the plan and

begins spinning up workers

Distributed Storage

• Mappers read and write to

distributed storage

• Map => Shuffle => Reduce

Reducer • Reducers read and write to

distributed storage

Master

Reducer

Mapper Mapper

Stage 2

MapReduce is Flexible but Heavy

Stage 1

Master

Mapper Mapper

Distributed Storage

Reducer

Dremel vs MapReduce

• MapReduce

o Flexible batch processing

o High overall throughput

o High latency

• Dremel

o Optimized for interactive SQL queries

o Very low latency

Mixer 0

Mixer 1 Mixer 1

Leaf Leaf Leaf Leaf

Distributed Storage

Dremel Architecture

• Columnar Storage

• Long lived shared serving tree

• Partial Reduction

• Diskless data flow

SELECT

state, COUNT(*) count_babies

FROM [publicdata:samples.natality]

WHERE

year >= 1980 AND year < 1990

GROUP BY state

ORDER BY count_babies DESC

LIMIT 10

Simple Query

Mixer 0

Mixer 1 Mixer 1

Leaf Leaf Leaf Leaf

Distributed Storage SELECT state, year

O(Rows ~140M)

COUNT(*)

GROUP BY state

WHERE year >= 1980 and year < 1990

O(50 states)

LIMIT 10

ORDER BY count_babies DESC

COUNT(*)

GROUP BY state

COUNT(*)

GROUP BY state

O(50 states) O(50 states)

Modeling Data

Example: Daily Weather Station Data

weather_station_data

station lat long mean_temp humidity timestamp year month day

9384 33.57 86.75 89.3 .35 1351005129 2011 04 19

2857 36.77 119.72 78.5 .24 1351005135 2011 04 19

3475 40.77 73.98 68 .35 1351015930 2011 04 19

etc...

Example: Daily Weather Station Data

station, lat, long, mean_temp, year, mon, day

999999, 36.624, -116.023, 63.6, 2009, 10, 9

911904, 20.963, -156.675, 83.4, 2009, 10, 9

916890, -18133, 178433, 76.9, 2009, 10, 9

943320, -20678, 139488, 73.8, 2009, 10, 9

CSV

Organizing BigQuery Tables

Your Source

Data

October 22

October 23

October 24

Modeling Event Data: Social Music Store

logs.oct_24_2012_song_activities

USERNAME ACTIVITY Cost SONG ARTIST TIMESTAMP

Michael LISTEN Too Close Alex Clare 1351065562

Michael LISTEN Gangnam Style PSY 1351105150

Jim LISTEN Complications Deadmau5 1351075720

Michael PURCHASE 0.99 Gangnam Style PSY 1351115962

logs.oct_24_2012_song_activities

USERNAME ACTIVITY Cost SONG ARTIST TIMESTAMP

Michael LISTEN Too Close Alex Clare 1351065562

Michael LISTEN Gangnam Style PSY 1351105150

Jim LISTEN Complications Deadmau5 1351075720

Michael PURCHASE 0.99 Gangnam Style PSY 1351115962

Users Who Listened to More than 10 Songs/Day

SELECT

UserId, COUNT(*) as ListenActivities

FROM

[logs.oct_24_2012_song_activities]

GROUP EACH BY

UserId

HAVING

ListenActivites > 10

How Many Songs Listened to Total by Listeners of PSY?

SELECT

UserId, count(*) as ListenActivities

FROM


WHERE UserId IN (

SELECT

UserId

FROM


WHERE artist = 'PSY')

GROUP EACH BY UserId

HAVING

ListenActivites > 10

Modeling Event Data: Nested and Repeated Values

{"UserID" : "Michael",

"Listens": [

{"TrackId":1234,"Title":"Gangam Style",

"Artist":"PSY","Timestamp":1351075700},

{"TrackId":1234,"Title":"Alex Clare",

"Artist":"Alex Clare",'Timestamp":1351075700}

]

"Purchases": [

{"Track":2345,"Title":"Gangam Style",

"Artist":"PSY","Timestamp":1351075700,"Cost":0.99}

]}

JSON


"Listens": [

{"TrackId":1234,"Title":"Gangam Style",




]

"Purchases": [

{"Track":2345,"Title":"Gangam Style",


]}


"Listens": [

{"TrackId":1234,"Title":"Gangnam Style",




]

"Purchases": [

{"Track":2345,"Title":"Gangnam Style",


]}

Which Users Have Listened to Beyonce?

SELECT

UserID,

COUNT(ListenActivities.artist) WITHIN RECORD

AS song_count

FROM

[logs.oct_24_2012_songactivities]

WHERE

UserID IN (SELECT UserID,

FROM [logs.oct_24_2012_songactivities]

WHERE ListenActivities.artist = 'Beyonce');

What Position are PSY songs in our Users' Daily Playlists?

SELECT

UserID,

POSITION(ListenActivities.artist)

FROM

[sample_music_logs.oct_24_2012_songactivities]

WHERE

ListenActivities.artist = 'PSY';

SELECT

AVG(POSITION(ListenActivities.artist))

FROM

[sample_music_logs.oct_24_2012_songactivities],

[sample_music_logs.oct_23_2012_songactivities],

/* etc... */

WHERE

ListenActivities.artist = 'PSY';

Average Position of Songs by PSY in All Daily Playlists?

Summary: Choosing a BigQuery Data Model

• "Shard" your Data Using Multiple Tables

• Source Data Files

• CSV format

• Newline-delimited JSON

• Using Nested and Repeated Records

• Simplify Some Types of Queries

• Often Matches Document Database Models

Developing with BigQuery

Google Cloud Storage

Upload Your Data

BigQuery

Load your Data into BigQuery

"jobReference":{

"projectId":"605902584318"},

"configuration":{

"load":{

"destinationTable":{

"projectId":"605902584318",

"datasetId":"my_dataset",

"tableId":"widget_sales"},

"sourceUris":[

"gs://widget-sales-data/2012080100.csv"],

"schema":{

"fields":[{

"name":"widget",

"type":"string"},

...

POST https://www.googleapis.com/bigquery/v2/projects/605902584318/jobs

"jobReference":{

"projectId":"605902584318"},

"configuration":{

"load":{

"destinationTable":{

"projectId":"605902584318",

"datasetId":"my_dataset",

"tableId":"widget_sales"},

"sourceUris":[

"gs://widget-sales-data/2012080100.csv"],

"schema":{

"fields":[{

"name":"widget",

"type":"string"},

...

Query Away!

"jobReference":{

"projectId":"605902584318",

"query":"SELECT TOP(widget, 50), COUNT(*) AS sale_count

FROM widget_sales",

"maxResults":100,

"apiVersion":"v2"

}

POST https://www.googleapis.com/bigquery/v2/projects/605902584318/jobs

"jobReference":{

"projectId":"605902584318",

"query":"SELECT TOP(widget, 50), COUNT(*) AS sale_count

FROM widget_sales",

"maxResults":100,

"apiVersion":"v2"

}

Libraries

• Python

• Java

• .NET

• Ruby

• JavaScript

• Go

• PHP

• Objective-C

Libraries - Example JavaScript Query

var request = gapi.client.bigquery.jobs.query({

'projectId': project_id,

'timeoutMs': '30000',

'query': 'SELECT state, AVG(mother_age) AS theav

FROM [publicdata:samples.natality]

WHERE year=2000 AND ever_born=1

GROUP BY state

ORDER BY theav DESC;'

});

request.execute(function(response) {

console.log(response);

$.each(response.result.rows, function(i, item) {

...

Custom Code and the Google Chart Tools API

Google Spreadsheets

Commercial Visualization Tools

Demo: Using BigQuery on BigQuery

• Full table scans FAST

• Aggregate Queries on Massive Datasets

• Supports Flat and Nested/Repeated Data Models

• It's an API

BigQuery - Aggregate Big Data Analysis in Seconds

Get started now:

http://developers.google.com/bigquery/

SELECT questions FROM audience

SELECT 'Thank You!'

FROM jordan

http://developers.google.com/bigquery

Schema definition

birth_record

parent_id_mother

parent_id_father

plurality

is_male

race

weight

parents

id

race

age

cigarette_use

state

Schema definition

birth_record

mother_race

mother_age

mother_cigarette_use

mother_state

father_race

father_age

father_cigarette_use

father_state

plurality

is_male

race

weight

Tools to prepare your data

• App Engine MapReduce

• Commercial ETL tools

• Pervasive

• Informatica

• Talend

• UNIX command-line

Schema definition - sharding

birth_record_2011

mother_race

mother_age


mother_state

father_race

father_age


father_state

plurality

is_male

race

weight

birth_record_2012

mother_race

mother_age


mother_state

father_race

father_age


father_state

plurality

is_male

race

weight

birth_record_2013

birth_record_2014

birth_record_2015

birth_record_2016

Visualizing your Data

BigQuery architecture

“ If you do a table scan over a 1TB table,

you're going to have a bad time. ”

Anonymous

16th century Italian Philosopher-Monk

•

• Reading 1 TB/ second from disk:

• 10k+ disks

• Processing 1 TB / sec:

• 5k processors

Goal: Perform a 1 TB table scan in 1 second

Parallelize Parallelize Parallelize!

Data access: Column Store

Record Oriented Storage Column Oriented Storage

Distributed Storage (e.g. GFS)

BigQuery Architecture

Mixer 0

Mixer 1

Shard 0-8

Mixer 1

Shard 17-24

Mixer 1

Shard 9-16

Shard 0 Shard 10 Shard 12 Shard 24 Shard 20

Running your Queries

SELECT COUNT(foo), MAX(foo), STDDEV(foo)

FROM ...

BigQuery SQL Example: Simple aggregates

SELECT ... FROM ....

WHERE REGEXP_MATCH(url, "\.com$")

AND user CONTAINS 'test'

BigQuery SQL Example: Complex Processing

SELECT COUNT(*) FROM

(SELECT foo ..... )

GROUP BY foo

BigQuery SQL Example: Nested SELECT

BigQuery SQL Example: Small JOIN

SELECT huge_table.foo

FROM huge_table

JOIN small_table

ON small_table.foo = huge_table.foo

Distributed Storage (e.g. GFS)

BigQuery Architecture: Small Join

Mixer 0

Mixer 1

Shard 0-8

Mixer 1

Shard 17-24

Shard 0 Shard 24 Shard 20

Other new features!

Batch queries!

• Don't need interactive queries for some jobs?

• priority: "BATCH"

• API

• Column-based datastore

• Full table scans FAST

• Aggregates

• Commercial tool support

• Use cases

That's it

SELECT questions FROM audience

SELECT 'Thank You!'

FROM ryan

http://developers.google.com/bigquery

@ryguyrg http://profiles.google.com/ryan.boyd

Data access: Column Store

Record Oriented Storage Column Oriented Storage

A Little Later ...

Row wp_namespace Revs

1 0 53697002

2 1 6151228

3 3 5519859

4 4 4184389

5 2 3108562

6 10 1052044

7 6 877417

8 14 838940

9 5 651749

10 11 192534

11 100 148135

Underlying table:

• Wikipedia page revision records

• Rows: 314 million

• Byte size: 35.7 GB

Query Stats:

• Scanned 7G of data

• <5 seconds

• ~ 100M rows scanned / second

Mixer 0

Mixer 1 Mixer 1

Leaf Leaf Leaf Leaf

Distributed Storage

SELECT wp_namespace, revision_id

10 GB / s

COUNT (revision_id)

GROUP BY wp_namespace

WHERE timestamp > CUTOFF

ORDER BY Revs DESC

COUNT (revision_id)


COUNT (revision_id)


"Multi-stage" Query

SELECT

contributor_id,

INTEGER(LOG10(COUNT(revision_id))) LogEdits

FROM [publicdata:samples.wikipedia]

SELECT

contributor_id,

INTEGER(LOG10(COUNT(revision_id))) LogEdits


GROUP EACH BY contributor_id)

SELECT

LogEdits, COUNT(contributor_id) Contributors

FROM (

SELECT

contributor_id,

INTEGER(LOG10(COUNT(*))) LogEdits


GROUP EACH BY contributor_id)

GROUP BY LogEdits

ORDER BY LogEdits DESC

Mixer 0

Mixer 1 Mixer 1

Leaf Leaf Shuffler Shuffler

Distributed Storage

SELECT contributor_id

ORDER BY LogEdits DESC

COUNT(contributor_id)

GROUP BY LogEdits


GROUP BY LogEdits


GROUP BY LogEdits

SELECT LE, Id

COUNT(*)

GROUP BY contributor_id

Shuffle by

contributor_id

N^2

GB/s

When to use EACH

• Shuffle definitely adds some overhead

• Poor query performance if used incorrectly

• GROUP BY

o Groups << Rows => Unbalanced load

o Example: GROUP BY state

• GROUP EACH BY

o Groups ~ Rows

o Example: GROUP BY user_id