Big Data in Science(Lessons from astrophysics)

Michael Drinkwater, UQ & CAASTRO

1. Preface Contributions by Jim Grey

Astronomy data flow

2. Past Glories Why it was easy to be world-leading

3. Future challenges Why really big data makes us worry!

CSIRO Parkes radio telescope

1. Preface: Jim Grey (Microsoft eScience)

› Much of what I discuss was already said by the late Jim Grey:

› “I have been hanging out with astronomers for about the last 10 years… I look at their telescopes… $15-20M worth of capital equipment with about 20-50 people operating the instrument… millions of lines of code are needed to analyse all this information. In fact the software cost dominates the capital expenditure!”

› Jim Grey on eScience, in The Fourth Paradigm, eds Hey, Tansley & Tolle, 2009. (emphasis added)

research.microsoft.com

Jim Grey,Microsoft Research

1. Preface: Astronomy Data Flow

Telescope Raw Images Output Image

Science Database Catalogues

2. Past Glories

› 20 years ago

- Easy to lead the world!

› UKST photographic all sky survey

- 1 image = 1 GB

- All-sky image = 1 TB

- All-sky catalogue = 100 MB

- Put online with two summer student projects

2. Past Glories

› Why did astronomy lead the way with (old) big data?

› 1) Telescopes are expensive so only a few data sources

- Data complex so only a few software packages, especially for national projects

- => easy to adopt a common data file format

› 2) Astronomers had strong computing skills

- => easy to search relatively large discovery space

CSIRO's ASKAP radio telescope with its innovative phased array receiver technology. (Image: Dragonfly Media)

2. Past Glories

› Problems with the old approach in astronomy

- Most team projects underestimate or ignore database budget

- Astronomers too independent – skeptical of computer science expertise

- Bespoke solutions not scalable or sustainable

The Anglo-Australian Telescope (Image: AAO) – used for many team projects

2. Past Glories

› WiggleZ Dark Energy Survey

- 5 year observing project

- $5M facility time + $1.5M grants + 20 team salaries

- Database $40k (donated by host as not funded)

› Success!

- 4 tests proving Einstein’s General Relativity correct

- Many other results

- 1425 citations

› Failure!

- Database failed as not supported

3. Future Challenges

› New projects so large astronomy must change…

- 1995 Schmidt photographic survey: 1 TB

- 2006 Sloan Digital Sky Survey: 25 TB

- …

- 2022-32 Large Synoptic Survey Telescope 130 PB in 10 years

- 2030-? Square Kilometre Array radio telescope: 10 PB per day!

- More data per day than entire internet per year

The LSST: 8.4 m telescope mirror, 3.2Gpixel camera

3. Future Challenges

› Challenges we know how to solve (Jim Gray predicted most of these)

- Realistic funding

- Scalable database structure: how to avoid i/o limits

- Must move the query to the data

- Efficient database design (Jim’s 20 questions to define functionality)

3. Future Challenges

› Nasty challenges we are yet to solve…

- Complex data mining way beyond SQL

- “Teaching software engineering to the whole community”1

- Real-time analysis for transient events

- Cross-matching different large databases in different locations

“The data collected by the SKA in a single day would take nearly two million years to play back on an iPod.” skatelescop.org

1. Mario Juric, LSST Data Management Project Scientist

Postscript: Jim Grey (Microsoft eScience)

› Jim Gray’s rules for large data design:

- Scientific computing is increasingly data intensive

- Solution is a “scale-out” architecture

- Bring computations to the data, rather than data to the computations

- Start the design with the 20 top questions

- Go from "working to working"

- From “Gray’s Laws: Database-centric Computing in Science”, Szalay & Blakeley, , in The Fourth Paradigm, eds Hey, Tansley & Tolle, 2009.

research.microsoft.com

Jim Grey,Microsoft Research