Big data - from consumers and patients, to the sea and stars

Ubiquitous, affordable devices and connectivity

Big data: The next frontier for innovation, competition, and productivity

May 2011 | byJames Manyika, Michael Chui, Brad Brown, Jacques Bughin, Richard

Dobbs, Charles Roxburgh, Angela Hung Byers, McKinsey Global Institute (MGI)

CloudBig data

Aggregation

MachineLearning Analytics

“Snow White Portraits”

http://www.mypersonality.org/wiki

http://www.pnas.org/cgi/doi/10.1073/pnas.1218772110

http://youarewhatyoulike.com/Kosinski, M., Stillwell D.J., Graepel T (2013) Private traits and

attributes are predictable from digital records of human behavior.

Proceedings of the National Academy of Sciences (PNAS).)

Data

Acquisition &

modelling

Collaboration

and

visualisation

Analysis &

data mining

Dissemination

& sharing

Archiving and

preserving

fourthparadigm.org

Data-intensive Research

X-Info

• Data ingest

• Managing a petabyte

• Common schema

• How to organize it

• How to reorganize it

• How to share with others

• Query and Vis tools

• Building and executing models

• Integrating data and Literature

• Documenting experiments

• Curation and long-term

preservation

The Generic Problems

Experiments &Instruments

Simulations

Literature

Other Archives

facts

facts

facts

facts

Questions

Answers

The Cloud

democratizes

access to scale &

economies of scale

Genomics Revolution

Genomics

Manual Measurement

Automated Measurement

Sample Collection

Historical Photographs

Counting

Ubiquitous

Motes

Aircraft SurveysModel Output

Typing

Monitoring

Collation

Quality assurance

Aggregation

Analysis

Reporting

Forecasting

Distribution

Done poorly,but a few notablecounter-examples

Done poorly to moderately,not easy to find

Sometimes done well,generally discoverable and available,

but could be improved

Integration

(I. Zaslavsky & CSIRO, BOM, WMO)

Web search:

“open weather

data azure”

http://fetchclimate2.cloudapp.net/

World Wide Telescope

www.worldwidetelescope.org http://www.worldwidetelescope.org/planck.html

www.tryfsharp.org

Datacenter

Edge point

http://github.com/windowsazure

Big Data Cloud Ecosystem

Windows Azure for Research Group

@azure4research

www.azure4research.com

Windows

Azure

Urban

Science

Machine

Learning

Environmental

Science

Life

Sciences

Computer

science

Engineering

British Library Labs cloud

analysis of digital catalogues,

including 19th Century books

scanned by Microsoft.@MechCuratorBot

mechanicalcurator.tumblr.com

www.azure4research.com

Use laptops &

desktop computers

Overwhelmed by

data

Finding analysis

ever more difficult;

sharing even

harder

www.azure4research.com