A (very) short history of big data

Copyright (c) 2008 Scale Unlimited, Inc. All Rights Reserved. Reproduction or distribution of this document in any form without prior written permission is forbidden.

Lightening Talk

1

A Very Short History of Big Data

phot

o by

: exf

ordy

, flic

kr

Monday, December 19, 2011


The First Big Data Problem

1880 Census

50 Million People

Age, gender, number of insane peoplein household

2



The First Big Data Solution

Hollerith Tabulating System

Punched cards - 80 variables

Used for 1890 Census

6 weeks instead of 7+ years

3



What is Big Data?

I Know It When I See It

More than you can handle withthe computer you’ve got

And scaling up isn’t an option

4



Big Science == Big Data

Weather predictions

Super-collider data

Astronomy images

5



A Data Explosion

OK, there’s a lot of data

Increased to 800 billion gigabytes in 2009.

If every person on earth tweeted continuously for a century...

6

“Every two days now we create as much information as we did from the dawn of civilization up until 2003. That’s something like five exabytes of data”-- Google CEO Erik Schmidt

Te xt

Gigabyte = 10^9 = 1,000,000,000Terabyte = 10^12 = 1,000,000,000,000Petabyte = 10^15 = 1,000,000,000,000,000Exabyte = 10^18 = 1,000,000,000,000,000,000



Search

Analyzing lots of data

Important pages are those thatimportant pages link to

Solving Satan’s spreadsheet100 billion rows x 100 billion columns

7



Advertising

Specifically online advertising

Lots of data in the form of log files

Lots of value if you increase sales

8



Advertising




Targeted advertising can be good

9



Advertising




Targeted advertising can be good

But scary, when they know too much

10

Satisfy your Barney Fetish

Pictures of Barney being drop-kicked off bridges. Discrete shipping. Noquestions asked.


Sports

A (very) short history of big data