Data, Data, Data,…….

Nov 05, 2012

Size of all Internet

data 2011

Size of all Internet

data 2020

Size of all Internet

data 2011

Size of all Internet

data 2020

70% Packaged

Goods Media

90% UGC/Senso

r

a future view……

User Generated Content (UGC)

Source - DOMO

what will enable growth of user

generated content?

some enabling technologies…….

network bandwidth cheap storage cheap compute power user friendly devices

network bandwidth

software defined network google fiber innovations related to:- SwitchesRoutersPackets sizecompressions

cheap storage

• cheap storage - a forcing function• storage companies provide free storage

• in return, they have access to user data

• raw data is turned into boutique data• sold at premium to interested companies and advertisers

cheap compute power

• Innovations on rack space• cheap, baremetal hardware• lowers TCO of servers• operational tasks become easier• allows companies to offer cloud

user friendly devices

buttons free WYSIWYS(tore) connectivity – most important and a “given”

tendency to track family

some data sources…..

reality

show

s

sensor data…

10 TB of Data/Engine/30 minutes 6 hour flight from NY to LA for Twin Engine 737 = 240 TB of Data/flight 28,537 Airliners in US Skies/day 6.5 Exabytes (6688 Petabytes/day)

“……..within the next five years, sensor data will hit the crossover point with unstructured data generated by social media. From there, the sensor data will dominate by factors 10-to-20 times that of social media……

” - Stephen Brobst, CTO, Teradata

online games

Pic – coolarcade.org

• ~225 million seventh-generation game consoles sold worldwide by early 2012• ~700 million Wii games, • 425 million PlayStation 3 games• 600 million Xbox 360 games. 

GPS data

Innovations in Transportation ApplicationsMultiple sources:

• Computers Embedded in Vehicle• In-vehicle navigation systems• Drivers’ cell phones. • Communication networks• Third-party data like weather• Traffic

Pic – www.bmwusa.com

intelligent roads (INTRO*)

• roads with sensors• determine traffic patterns• sustainable ways to route traffic• generate data for:-

• law enforcement• transportation• insurance companies• medical agencies

* INTRO – INTelligent ROads – a project of European Commission

mobile devices of tomorrow……

user generated content

• curated content • mashed content (pinterest like)• blogs• videos (own shows, personal videos, etc)• pics• collaboration – emails/IMs/ “Likes” etc• microblogs (twitter like)

another perspective………

How much is ZB, anyway?

all this leads to…..

Pic source – bigdatabytes.com

BIG DATA

big data characteristics…

3 V’s*

• volume

• velocity

• variety

* coined by Doug Laney of Gartner Inc

big data problems…

3 I’s

• immediate – do something now!!

• intimidating – what if you don’t?

• ill-defined – what is it, anyway - Vance Loiselle, CEO, Sumo Logic

big data skills……..

analytics – no more an afterthought….

analytic.NEXT

• near real time • new data sources• mobile • immediately actionable• big• agile• core of business

impact on us…• data scientists lead the “Data Orchestra”• developers/product mgrs/DBAs/Ops will merge

• Data Techs will emerge• “behavior”, “intent” and “thought” targeting• hourly trends will be considered “Jurassic” old

problems….

storage…..• store Exabytes (Petabytes)• huge compression ratio (80% compression)• cheap storage (~ 10 cents/GB/month)• MTTF rate (High failure 8%)• distributed storage • storage over software defined networking• read compressed data• ETL

servers…..• servers and storage merge?• special CPUs to handle compression?• encryption?• better cpu• bus speed

analytics

• understand data• analytical skills• discover new ways of looking at data• new containers for data warehouses incldg data warehouses on cloud

• backup and recovery (should not be an issue)