Discussion: Coverage data and WCS implementationinspire.ec.europa.eu/events/conferences/inspire_2016/pdfs... · 2016-10-04 · Issue 1 - How to deal with huge volume of coverage data

www.jrc.ec.europa.eu

Serving societyStimulating innovationSupporting legislation

Discussion:Coverage data and WCS implementation

INSPIRE Thematic Cluster onElevation, Orthoimagery, Reference systems and Geographical grids

Peter Baumann, Dominique Laurent, Jordi Escriu

Topics / Issues selected for discussion• Issue 1

How to handle the huge volume of coverage data

• Issue 2How to implement the concept of coverage aggregation

• Issue 3How to implement the concept of mosaicking

• Issue 4INSPIRE Coverage extensions vs. OGC standard coverages

• Issue 5Purpose of the metadata hook

Issue 1 - How to handle the huge volume of coverage data

Description

• It is generally difficult to transfer huge volume of coverage data through network services and data producers must analyse how to handle it.


Server side:big data

Client side: user requests

(various sizes)

(small) Internet pipe


Server side:big data

Client side: user requests

(various sizes)

(small) Internet pipe

???


Choice of download services

predefined data sets WCS

Make small coverages (tile,

small AU)

Other solution at encoding

level?

Server may handle big data and « cut »it

Client side?

Issue 1 - How to deal with huge volume of coverage data

Open questions

Case 1 - Delivery through Predefined data sets• Experiences in making “small” coverages (at conceptual

level)? Issues?• Alternative approaches at encoding level?

Case 2 - Delivery through WCS• Feed-back from experience?• Are there any issues on client side?


Case 1 - Delivery through Predefined data sets• Most data providers consider each tile / map sheet as

a coverage - general trend among NMCAs.• The coverage over a certain territory is split in several

pieces, for both organizational and efficiency purposes. • To be implemented using the concept of tiling. • Need for guidelines to select: The most appropriate tiling approach.

Maximum size for each tile.

• Tiling approach may have a direct impact on efficiency of delivery.


Case 1 - Delivery through Predefined data sets

• When each tile is considered as a coverage The coverage identifier depends on the tiling system and

so on the chosen CRS. Is it an issue?

What should be a data set? A unique coverage/tile or a set of coverages/tiles?

If a data set is composed of one coverage/tile, it implies a lot of data sets. Might be difficult to be handled both by producer (many metadata to be produced) and by user (how to find relevant tiles? => is there need for user friendly-tools?


Case 2 - Delivery through WCS• A coverage is the natural response of a WCS to a data

request (GetCoverage operation).• However, some Efficiency issues has been identified

when using such kind of services: WCS implementations build the final output of the

GetCoverage request first totally in the RAM of the server before sending it out to the client, e.g. MapServer / GeoServer [Jukka Rahkonen].

• How to solve this efficiency issues?

Issue 1 - How to deal with huge volume of coverage data• Big Data = “ship code to data”, extract what you want Who wants images > RAM?

• Coverage image For user convenience, pack many images in one coverage Tentatively no tiling prescribed, WCS is interface spec Many tilings possible for different use cases

– cf PhD thesis Paula Furtado WCS implementers can exercise ingenuity in That said, CIS 1.1 supports flexible tiling

– Space, time, range, …– Can exploit data format tiling (TIFF, NetCDF, …)

• Predefined data sets: no such restriction in WCS Any request, anytime – with tools existing today!

• Proven implementation evidence of WCS scalability WCS/WCPS : 145 TB space/time datacubes @ ESA (rasdaman)

WCS Core GetCoverage• Download a coverage (or a subset thereof), values guaranteed unaltered• Ex: „download coverage c001“

http://www.acme.com/wcs ? SERVICE=WCS & VERSION=2.0& REQUEST=GetCoverage & COVERAGEID=c001

http://standards.rasdaman.com

http://www.acme.com/wcs ? SERVICE=WCS & VERSION=2.0& REQUEST=GetCoverage & COVERAGEID=c001& SUBSET=Long(100,120) & SUBSET=Lat(50,60)& SUBSET=time("2009-11-06T23:20:52")

Ex: „coverage c001, lat/long cutout, time slice t=2009-11-06T23:20:52”

http://www.acme.com/wcs ? SERVICE=WCS & VERSION=2.0& REQUEST=GetCoverage & COVERAGEID=c001 & FORMAT=“image/tiff“

Ex: “coverage c001, in GeoTIFF”

Issue 2 - How to implement the concept of coverage aggregationDescription• The concept of tiling is introduced in the INSPIRE

TG on Elevation and Orthoimagery. • Tiles describe logical structures (e.g. map sheets,

administrative units like regions or districts, etc.).• Each tile is implemented as an individual Coverage.• This splitting approach is helpful to handle raster

data in manageable units.• Coverages can be aggregated (associated) to

indicate all of them cover certain areas of territory.

Issue 2 - How to implement the concept of coverage aggregation


A number of Tiles

Their aggregation and corresponding BBOX


Implementation of each Tile

Aggregation of a number of Tiles

covering a certain area


Contributing footprints corresponding to each Orthoimage coverage / Tile which participates in the aggregation

The aggregated orthoimage coverage (blue footprint)

Issue 2 - How to implement the concept of coverage aggregationOpen questions• Is it clear how to implement coverage / tile

aggregations in GMLCOV?

• Might be probably different ways in practice!

• TC #3 discussion thread:https://themes.jrc.ec.europa.eu/discussion/view/50412/how-to-implement-tiling-model-mosaic-elements-coverages-and-coverage-aggregations-in-gmlcov-files

• Are coverage aggregations really needed?


• Why are coverage aggregations really needed? Requested only by ESA early in TG OI [Markus Seifert] Doubt usefulness for standard agency cases

• Adds complexity [Dominique Laurent, Peter Baumann] For users: what you get what is advertised For service ops: tedious to keep consistent

• Recommendation optional advanced feature Simple core coverage model

– Coverage = seamless mosaic (many files!)– Coverage = image timeseries– Coverage = weather datacube– Etc.

Inset: WaterML 2.0 Time Handling• WaterML 2.0: timeseries = time slices layout hardwired

• OGC Coverages: time just another axis Implementation can choose efficient layout

Tiling• Goal: faster tile loading by adapting storage units to access patterns• Approach: partition n-D array into n-D partitions („tiles“)• Tiling classification based on degree of alignment [ICDE 1999]

chunking [Sarawagi, Stonebraker, DeWitt, ... ]

Tiling in rasdaman• Any partitioning („tiling“)

145+ TB datacubes[IJDE 2015]

• Why irregular tiling?

[OpenStreetMap]

Tiling in rasdaman

• tiling strategies as service tuning [Furtado]:regular directional ... area of interest

• rasdaman storage layout languageinsert into MyCollection values ...tiling area of interest [0:20,0:40], [45:80,80:85]tile size 1000000index d_index storage array compression zlib

Issue 3 - How to implement the concept of mosaickingDescription

• The concept of mosaicking is introduced in the INSPIRE data specification on Orthoimagery.

• Mosaic elements are intended to identify the contributing area and the acquisition time of one or several input images used to generate an orthoimage coverage, i.e. precisely describe the temporal characteristics of a mosaicked product.

• Their modelling corresponds to a thematic need. • Not clear how to implement mosaicking in

GMLCOV.• Might be probably different ways in practice.

Issue 3 - How to implement the concept of mosaicking



Use of‘SingleMosaicElement’

objects

Use of ‘AggregatedMosaicElement’

objects


Open questions• Again! - Is it clear how to implement mosaic

elements in GMLCOV?

• Might be probably different ways in practice!

• Are there any other ways to add mosaic information to each individual coverage, without providing ‘MosaicElement’ objects?


• No mosaicking in Google Maps! ;-)• Tiling vs mosaicking? As before: complex concept!

• Internal implementation detail,should not be visible to users Similar to tiling WaterML vs Coverage

• Good tools will do it automatically Should not be constrained

Issue 4 – INSPIRE Coverage extensions vs. OGC standard coverages

• INSPIRE models based on coverages partially extends the standard way of modelling and implementing OGC coverages.

• Such INSPIRE extensions corresponds to thematic needs identified by TWGs.

• Danger: They are ignored by WCS interface.• Agreement: Minimize INSPIRE extensions as

possible - Need to examine them case-by-case. • Worth to align this task with CIS v1.1, if this new

standard is adopted by INSPIRE in the future.


INSPIRE Coverage extensions (Elevation)


INSPIRE Coverage extensions (Orthoimagery)

Issue 4 – INSPIRE Coverage extensions vs. OGC standard coverages• EXAMPLE - Identification of coverage extent –

There are 2 attributes to deal with it: boundedBy (GML): Optional / Approximate / Does not

allow identification of discontinuous spatial extents.

domainExtent (INSPIRE extension): It supports the identification of discontinuous spatial extents (needed e.g. when coverage aggregations are identified) / Not recognized by WCS.

• Largely discussed in TC #3: https://themes.jrc.ec.europa.eu/discussion/view/12901/domainextent-vs-gmlboundedby-el-oi-coverages-encoding

Issue 4 – INSPIRE Coverage extensions vs. OGC standard coveragesOpen questions• Are INSPIRE extensions really needed?

• For each of them, is there any equivalent element foreseen in the implementations standards? e.g. in GMLCOV, CIS v1.1?

• Is there specific GMLCOV component to implement additional items not covered by OGC standards?

• If an INSPIRE extension which is not considered / foreseen in the implementation standards proves to be useful, could it be considered in a future update of such standards?

Issue 4 – INSPIRE Coverage extensions vs. OGC standard coverages• Problem 1: INSPIRE coverages have strange

definitions domainSet: <any>; rangeType: <any>

• Problem 2: INSPIRE coverages add in wrong place <metadata> slot foreseen for this purpose

• Suggestions: Reinspect each item Use metadata for extensions Make voidable (allow vanilla WCS to serve INSPIRE) INSPIRE to contribute to further OGC CIS/WCS work

Issue 5 - Purpose of the metadata hook

• Coverages may have metadata at different levels: Data set level (xml metadata file of the data set) –

For discovery purposes (CSW).

Image file (header) – Required by specific format.

Coverage level (optional metadata hook, as proposed in GMLCOV).

• Guidelines about the purpose and content of the metadata hook are needed.


Metadata hook


Open questions

• Which metadata elements are expected in the metadata hook?

• Why? This content is expected to cover any WCS functionality?

• Is there any need to include in the metadata hook specific metadata elements?


• …to transport any additional “meta”data Coverage metadata = CoverageDescription

Discovery (catalog): <metadata>

• Image “header” ~ covg descr + <metadata>• Example: CIS 1.1 00_metadata.gml

• Recommendation: own schema per unit ISO 19115, EO Metadata Profile, … Can coexist

Documents

Discussion: Coverage data and WCS implementationinspire.ec.europa.eu/events/conferences/inspire_2016/pdfs... · 2016-10-04 · Issue 1 - How to deal with huge volume of coverage data