Embed Size (px)
Citation preview
Sentinel 2 and the Australian Geoscience Data
Cube
Simon Oliver – National Earth and Marine Observation
National Computational Infrastructure
Commonwealth Scientific and Industrial Research Organisation
Dr Adam Lewis and Medhavy Thankappan
Acknowledgements
Sentinel 2 and the Australian Geoscience Data Cube
Geoscience Australia
Department of Industry, 673 staff
Located in Canberra
4PB of geoscience data and growing
Sentinel 2 and the Australian Geoscience Data Cube
Overview
• Challenges
• Data volume, growth rate and variety
• User expectations
• The Australian Geoscience Data Cube
• What makes it work?
• Example applications
• Sentinel-2 integration
Sentinel 2 and the Australian Geoscience Data Cube
Population
Italy 62 million
Australia 24 million
Sentinel 2 and the Australian Geoscience Data Cube
The Challenge of Big Spatial Data
• 35 years of Earth Observation data
• >200TB of Level 0 Landsat
TM,ETM+,OLI-TIRS
• + processed products >1PB
• If every pixel is an observation
• ~40*1012 pixels in all available
datasets
The Challenge of Big Spatial Data
Sentinel 2 and the Australian Geoscience Data Cube
The Challenge of Big Spatial Data
• Earth Observation data is both spatially and temporally sparse
and irregular.
• EO data collection is dynamic: growing in time, and also subject
to modification (existing data) and insertion (new data).
Sentinel 2 and the Australian Geoscience Data Cube
• Need to incorporate ancillary
data for exact acquisition time
(e.g. tides for shallow-water
bathymetry)
• Often have two satellites
observing the same area in a
given 24h period, so we need a
much finer temporal resolution
than one day.
Expectations for On-Demand Integration
Sentinel 2 and the Australian Geoscience Data Cube
Community safety
Information for decision support
Traditional remote sensing product process
Sentinel 2 and the Australian Geoscience Data Cube
Traditional remote sensing product process
Sentinel 2 and the Australian Geoscience Data Cube
• Raijin @ National Computational Infrastructure
• 57,472 cores (Intel Xeon Sandy Bridge technology, 2.6 GHz) in
3592 compute nodes;
• 160 TBytes (approx.) of main memory;
• 10 PBytes (approx.) of usable fast filesystem (for short-term scratch
space).
Powering the Data Cube
Sentinel 2 and the Australian Geoscience Data Cube
*http://top500.org/
Developing the scientific processes and
workflows
Sentinel 2 and the Australian Geoscience Data Cube
*Landsat Product Generation System (USGS and
Pinkmatter)
- Level 1 processor / precision geometric correction
Landsat/EO Processor (L5/7/8 – adaptable to Sentinel2)
- Nadir BRDF Adjusted Reflectance (MODTRAN 5, MODIS
BRDF) Li et al 2010,2012
- Pixel Quality Assessment
- Fractional Cover (Joint Remote Sensing Research
Program)
gVerify
- Compare geometry with reference
Pixel Quality Assessment
Sentinel 2 and the Australian Geoscience Data Cube
Test Bit Value
Cumulative
Sum
Saturation band 1 0 1 1
Saturation band 2 1 2 3
Saturation band 3 2 4 7
Saturation band 4 3 8 15
Saturation band 5 4 16 31
Saturation band 61* 5 32 63
Saturation band 62* 6 64 127
Saturation band 7 7 128 255
Contiguity 8 256 511
Land/Sea 9 512 1023
ACCA 10 1024 2047
Fmask 11 2048 4095
Cloud Shadow (ACCA) 12 4096 8191
Cloud Shadow
(Fmask) 13 8192 16383
Topographic Shadow ** 14 16384 32767
To be determined ** 15 32786 65535
Fmask – Zhu and Woodcock, 2012
ACCA – Irish et al, 2006
The Australian Geoscience Data Cube
• The AGDC arranges 2D (spatial) data temporally and spatially to
allow flexible but reasonably efficient large-scale analysis.
• “Dice’n’Stack” method used to subdivide the data into spatially-
regular, time-stamped, band-aggregated tiles which can be
managed as dense temporal stacks.
Sentinel 2 and the Australian Geoscience Data Cube
Dice… …and Stack
Current Tile Contents (Landsat TM, ETM+)
Sentinel 2 and the Australian Geoscience Data Cube
* PQA Geoscience Australia
** Joint Remote Sensing Research Program, Currently only a 3x2 path/row test area of FC data held in AG-DC. Planned to
complete load by end June 2014
Level 1 precision terrain correction
1. LS5-B60 – Thermal Infrared
or
1. LS7-B61 – Thermal Infrared Low
Gain
2. LS7-B62 – Thermal Infrared High
Gain
(Byte type)
ARG-25 (Surface Reflectance)
1. LS5/7-B10 – Visible Blue
2. LS5/7-B20 – Visible Green
3. LS5/7-B30 – Visible Red
4. LS5/7-B40 – Near Infrared
5. LS5/7-B50 – Middle Infrared 1
6. LS5/7-B70 – Middle Infrared 2
(Int16 type)
Pixel Quality (PQA)*
1. PQ – Bit-array of PQ tests
(UInt16 type)
Fractional Cover (FC)**
1. Photosynthetic Veg. (PV)
2. Non-Photosynthetic Veg. (NPV)
3. Bare Soil (BS)
4. Un-mixing Error (UE)
(Int16 type)
Digital Surface Model (DSM)***
1. Elevation
2. Slope
3. Aspect
(Float32 type)
The prototype AGDC implementation
Sentinel 2 and the Australian Geoscience Data Cube
AGDC database – provides
indexing and filtering capability to
enable tile grouping
AGDC API – facilitates user
interaction
Written in Python and based on
Geospatial Data Abstraction Library
- Virtual Raster Transforms
provide data stacks for analysis
Data grid specification adheres to
the ANZLIC National Nested Grid
Specification Guide
Sentinel 2 and the Australian Geoscience Data Cube
Surface water
Menindee Lakes
time series
1998-2012
Total observations
per grid cell
~600-1200
4000*4000 grid
cells
Water Observations
from Space
• 15 Years of data
from LS5 &
LS7(1998-2012)
• 25m Nominal Pixel
Resolution
• Approx. 133,000
individual ARG-25
scenes in ~12,400
passes
• Entire archive of
1,312,087 ARG25
tiles => 21x1012
pixels visited
• 3 hrs at NCI
(elapsed time) to
compute.
Sentinel 2 and the Australian Geoscience Data Cube
Continental Surface Water
Intertidal morphology and extent
Sentinel 2 and the Australian Geoscience Data Cube
Intertidal morphology and extent
Sentinel 2 and the Australian Geoscience Data Cube
CSIRO invariant target analysis
CSIRO Minerals
Downunder
Flagship
Caccetta, Ong
and Lau
Sentinel 2 and the Australian Geoscience Data Cube
Platform for Collaboration
Preserve each observation
What makes the AGDC work?
Sentinel 2 and the Australian Geoscience Data Cube
Massive data volume with deep time-series
Common data structures / consistent gridding
Sensor-agnostic reflectance correction
Per pixel quality assessment
High Performance Computing
High Performance Data
Common interface to data
Massive data volume with deep time-series
Common data structures / consistent gridding
Sensor-agnostic reflectance correction
Per pixel quality assessment
High Performance Computing
High Performance Data
Common interface to data
Preserve each observation
Platform for Collaboration
AGDC short term work programme
- Further automating workflows
- Optimise DB schema
- Move from stacked 2D files to dense, contiguous, indexed NetCDF
files
- ASTER, MODIS and other data uploaded (June 2014)
- Ingest full Landsat TM/ETM/OLI -1986 to present (June 2014)
- Evaluate Discrete Global Grid System options to further optimise and
simplify the statistical engine
- Open source the project (June 2014) – GitHub
Sentinel 2 and the Australian Geoscience Data Cube
Common Analytical Platform
Sentinel 2 and the Australian Geoscience Data Cube
National / International
Common Analytical
Platform Organisational
Repositories
Integrating Sentinel 2 into AGDC
- Flexible architecture enabling addition of new sensors
- TM, ETM+, OLI-TIRS, ASTER and MODIS
- Fit to the nested hierarchy for interoperability i.e. Landsat is on a
25m grid (scale up, scale down)
- Ability to rapidly integrate Sentinel 2 with other “like” data
- Reduction of technical barriers to uptake of the data
Sentinel 2 and the Australian Geoscience Data Cube
Network of Interoperable Data Cubes?
Sentinel 2 and the Australian Geoscience Data Cube
Data Cubes for:
Africa, Antarctica, China,
Europe, North America, … http://www.opengeospatial.org/pressroom/
pressreleases/1980
Thank you!
Phone: +61 2 6249 9770
Web: www.ga.gov.au
Email: [email protected]
Address: Cnr Jerrabomberra Avenue and Hindmarsh Drive, Symonston ACT 2609
Postal Address: GPO Box 378, Canberra ACT 2601
