Virtual Laboratory for Earth Observation · 2011. 11. 28. · 2 VLEO eResearch 2011-11-08 Divided, we struggle on… SENSOR Day-time Night-time Terrestrial Marine Timeliness Historical

1

Virtual Laboratory for Earth Observation

Edward King, Matt Paget (CMAR)Lesley Wyborn, Wenjun Wu, Leo Lymburner, Peter Tan, Fei Zhang, Medhavy Thankappan (GA/NEO)Ben Evans, Joseph Antony, Muhammad Atif (NCI/ANU)Vittorio Brando, Thomas Schroeder (CLW)Stefan Maier (CDU) & Mark Gray (CurtinU)

2011-11-08

Wealth from Oceans

VLEO eResearch 2011-11-08

What’s the use?

Ocean dynamics

Cropping in northern WA wheat belt

0

0.1

0.2

0.3

0.4

0.5

0.6

0 23 46 69 92 115 138 161 184

Time

En

han

ced

Veg

etat

ion

Ind

ex

Crop

2000 2001 2002 2003 2004 2005 2006 2007

agronomy

fire history

forecasting

Soil conservation

marine bio-geochemistry

fish ecology

climate change

Ocean productivity

The Earth system SENSOR

Day-time

Night-time

Terrestrial

Marine

Timeliness

Continental

Daily

Annual

Local

Historical Record

2


Divided, we struggle on…

SENSOR

Day-time

Night-time

Terrestrial

Marine

Timeliness Historical Record

Continental

Daily

Annual

Local

Acquisition, Storage & Processing In Common (eg. Unpacking, Geolocation, Calibration)

Specialised Processing(eg. NDVI, LAI, SST, Chl, Compositing…)

Data Volume + Process specialisation = Community Fragmentation

Generic ToolsSubsetting, process control, visualisation


NCRIS/EIF Paradigm Change is an Opportunity

1. Funding science domain research infrastructure • Marine – IMOS has a National focus• Terrestrial – TERN has a National focus• Participants in BOTH domains are thinking at large scale AND

getting to know each other…

2. Support for enabling domains - ICT: “Platforms for Collaboration”• National Computational Infrastructure (NCI) provides

1. The scale to handle these data sets (12k CPU + storage)2. The shared access - It is outside all our institutional firewalls so it

overcomes one of the biggest practical obstacles to collaboration

3. Seeding uptake through NeAT (ARCS+ANDS)• MACDDAP Crawler+Aggregator, Bioflow, AusCover Workflows

(RS-YABI)

4. A roadmap in the future – RDSI + NeCTAR

3


Phase 1 - National Datasets for Key Sensors

SENSOR

Day-time

Night-time

Terrestrial

Marine

Timeliness Historical Record

Continental

Daily

Annual

Local

Storage & ProcessingIn Common

Partners:

• NCI + IMOS + TERN

• Geoscience Australia

• CMAR + CLW

• CDU + Curtin

Concept:

• Improve storage efficiency

• Establish basic processing tools

• Build selected specialised tools

• Start with MODIS + Landsat


Phase 1a. MODIS (2000+)

NCI .

Long Term Store

Level-0, 35+TB

Short Term Store,

<0.5TB

WA

DAR

VICTAS

QLD

Alice

Springs.

AARNET

NASAArchive(US East Coast)

Internet

National Standard

Production Facility(IMOS,TERN)

Experimental/Development

Local Users & Projects

Remote Users & Projects

ReceptionStations

Real-TimeData

HistoricalArchive

Com

mon P

rocessing Tools

Specialised A

pplications

Ocean

Vegetation

Land-Air

Weather

Fire

Coastal

4


Phase 1b. Landsat (1979+)

• 300+ TB Australasian archive, every 16 days

• Geoscience Australia leads this work

• See Poster “Unlocking the Landsat Archive”

• ~2 years to build this

Figures c/o Simon Oliver, GA


Coming Soon (3-6 months)

• NOAA/AVHRR (1980s+)• 10 TB, 200000 scenes

• SeaWIFS (1997-2010)• 0.5 TB• re-uses MODIS software

5


Phase 2 – Community Tools

• Data services (DAP, OGC protocols, for Google Earth)

• http, Hyrax, Thredds Data Server…

• Catalogues and data access• IMOS/AODAAC + NeAT MACDDAP Aggregator

• Generic processing management• NeAT Bioflow + NeAT AusCover Workflows

• Presentation, query and analysis tools (Twinkleware)


IMOS/AODAAC – Automatic Cataloguing & Access for Gridded Data

• IMOS Project, NeATaggregator cpt.

• Deployed to support • IMOS portal

• Tern/Auscover

• Interactive/batch mode• Supports

• mass data access• individual scenes OPeNDAP Data

Servers

Web Crawler and

OPeNDAPHarvester

SpatialData-base

Web Query

Service

Aggre-gatorInternet

End User

orAppli-cation

(model)

1

2

3

4

5

0

6

T

6


Workflow Tools: RS-YABI

• RS workflows typically linear & per-scene: big automation opportunity• YABI engine originally developed in Bioinformatics domain to exploit an

abstraction of HPC processing and storage (NeAT Bioflow)

• Adapted for remote sensing under a NeAT project with TERN/AusCover

• Strong HPC capability• Easy to incorporate heterogeneous processing

steps (different languages, code etc)• User friendly interface

• Developed @iVEC, being installed @NCI• See 2 Posters – YABI & RS-YABI


Decrease in Annual Minima

2000

2500

3000

3500

4000

4500

5000

5500

6000

6500

0 23 46 69 92 115 138 161 184

Time (16 Day Intervals)

En

han

ced

Veg

eta

tio

n In

dex

EVI time seriesAnnual MinimaLinear (Annual Minima)

2000 2001 2002 2003 2004 2005 2006 2007

Time series for that pixel

Comparison to other land covers within the region

Land Cover

Alpine

Open Sed

ges

Closed

For

est

Open F

ores

t

Woo

dland

Rainfed

Pas

ture

Tre

nd in

the

annu

al m

inim

um E

VI v

alue

s

-0.03

-0.02

-0.01

0.00

0.01

0.02

Comparison to other land covers within the region

Land Cover

Close

d Fo

rest V

IC

Close

d Fo

rest S

M

Close

d Fo

rest A

ust

Ope

n Fo

rest V

IC

Ope

n Fo

rest SM

Ope

n Tr

ees Au

st

Woo

dlan

d VI

C

Woo

dlan

d SM

Woo

dlan

d Au

st

Rainfed Pa

sture VI

C

Rainfed

Pas

ture S

M

Rainfed Pa

sture

Tren

d in

the

annu

al m

inim

um E

VI v

alue

s

-0.03

-0.02

-0.01

0.00

0.01

0.02

Tools to query large spatial time series (example, currently twinkleware)

Credit: Leo Lymburner @ GA

These rely on standards-based ways to represent data, such as netCDF/CF conventions and the (emerging) raster storage architectures and the National nested grid

7


Where have we got to?

According to NeCTAR, virtual laboratories are…

“Connecting researchers with existing and new research facilities, data repositories and computational tools, NeCTAR’s Virtual Laboratories will streamline research workflows and enable new opportunities for research innovation. The Virtual Laboratories will enable more collaborative research and new research efficiencies on a national scale by connecting researchers with each other and with research infrastructure.”

This doesn’t look too different from what we are building.


Key Points

• Enabled by NCRIS/EIF support across multiple research sectors and evolution of research community attitudes

• IMOS + TERN + PfC + NeAT

• No specific funding for this initiative but now attracting support through projects - we need a plan!

• Enable trans-domain science (ESS) by bringing together data sets not normally used by only one community

• Share with partners: data assembly - data production - software maintenance

• Easier to collaborate• Consistency and standards

• VIIRS – launched 11 days ago –will provide a good test to see if we can do a better job nationally than we did with MODIS

• Reduce duplication• Reduce duplication

Documents

Virtual Laboratory for Earth Observation · 2011. 11. 28. · 2 VLEO eResearch 2011-11-08 Divided, we struggle on… SENSOR Day-time Night-time Terrestrial Marine Timeliness Historical