Page 1Session 4: State of the art technology and standards, 12/10/2012 Web Processing Service Models for scientific exploitation of EO Data, Session 4:

Page 1Session 4: State of the art technology and standards, 12/10/2012

Web Processing Service

Models for scientific exploitation of EO Data, Session 4: State of the art technology and standards

Yves Coene, Spacebel s.a.Patrick Jacques, Spacebel s.a.

Peter Baumann, Jacobs University

Frascati - 12 October 2012


Overview

• Introduction

• Web Processing Service

• Case Studies

– Decision Support (DREAM)

– Lunar Landing (LandSAfe)

– Processing On Demand (G-POD, SSEGrid, ngEO)

• Open-source implementations

• Issues and future work

• Standardisation work

– Ongoing

– Planned

• Conclusion


Introduction

• HMA Standardisation

– Service Oriented Architectures

– For GSCDA and Science

• Service Taxonomy (HMA)

– Identity Management Services

– Discovery Services

– Invoke Services

• Feasibility Analysis

• Ordering

• Geospatial Processing: §3.4.7, §5.5.5

– Online Data Access Services



• WPS is an OGC standard for:

– describing and executing a geoprocessing operation via a service,

– standardizing the requests and responses for a particular geospatial processing service.

• WPS is a generic model of a Web service supporting WSDL and REST, for which profiles need to be developed and standardised to support interoperability:

– Syntax and semantics of input parameters

– Input and output data format



• Operations:

– GetCapabilities

• Discover which processes are offered by the service*

– DescribeProcess

• Discover input/output of a specific process

• Response can be used to build the user interface for capturing the input parameters**.

– Execute

• Execute a process

• Storing response or returning response document

(*) e.g. ProcessCoverages is a process that accepts WCPS request language as per OGC 09-045.

(**) similar to DescribeTasking in OGC 10-135 Earth Observation Satellite Tasking Extension of SPS.


Web Coverage Processing Service

• The Web Coverage Processing Service (WCPS) specifies an expression language allowing for server-side filtering and processing of multi-dimensional raster coverages.

• It defines syntax and semantics of expressions for specifying extraction and analysis of coverages containing sensor, image, and statistics data.

• Not tied to a particular transmission protocol, the query paradigm can be embedded into any service framework, such as:

– OGC Web Coverage Service (WCS)

– OGC Web Processing Service (WPS).


Web Coverage Processing Service

• Use Cases:

– On-demand retrieval

• “thresholded NDVI where threshold value is obtained from user input”

• Ad-hoc queries

– Processing

• “Fourier transform of scene X”

– Filtering

• “Scenes with less than x% cloud coverage”

– Summary data

– Sensor fusion and pattern mining

Slide 7


WCPS By Example

• "From MODIS scenes M1, M2, and M3, the absolute of the difference between red and nir, in HDF-EOS"

for $c in ( M1, M2, M3 )return encode( abs( $c.red - $c.nir ), "hdf“ )

(hdfA,

hdfB,

hdfC)

8


WCPS By Example


– …but only those where nir exceeds 127 somewhere

for $c in ( M1, M2, M3 )where some( $c.nir > 127 )return encode( abs( $c.red - $c.nir ), "hdf“ )

(hdfA,

hdfC)

9


WCPS By Example


– …but only those where nir exceeds 127 somewhere

– …inside region R

for $c in ( M1, M2, M3 ), $r in ( R )where some( $c.nir > 127 and $r )return encode( abs( $c.red - $c.nir ), "hdf“ )

(hdfA)

10



• OGC 05-007r7, Web Processing Service, OGC Implementation standard.

• OGC 08-091r6, Corrigendum for ... Web Processing Service.

• OGC 08-068r2, Web Coverage Processing Service (WCPS) Language Interface Standard, version 1.0.0.

• OGC 12-029, Web Processing Service Best Practices, Discussion Paper, OGC Implementation standard.

• OGC 08-123: WPS-T Discussion Paper

• OGC 09-045, Web Processing Service – Application Profile for Coverage Processing. Version 0.2.0 (status: discussed in WPS.SWG).

*

(*) OGC Standards.


Case Study 1: DREAM

• DREAM Project

– Dedicated to exploitation of EO data for Decision Support

– Based on use cases of European Institutions: EUSC and EMSA

– WPS used for Quality Assessment of ortho-images and Orthorectification of raw images:

• Horizontal Accuracy Maps

• Vertical Accuracy Maps

Slide courtesy DREAM Project (ASTRIUM-GEO)


O1 – Online Data Access ClientO2 – Online Data Access Server

Q1 - Data Quality Proxy

Data Quality Service

Photogrammetry Library

Compute AOI WPS Processing

Reference3D Database

Reference Orthoimages

DEM

Compute RPCWPS Processing

OG

C WPS Service Interface

Web Application

Server

Compute Quality WPS Processing

OG

C WPS Client

OrthoRectifyWPS Process

AssessQualityWPS Process

Ortho-ImageProducer

OGC WPS Service Interface

Web JS Plug-in

ViewerWMS

….

File System

• Raw Acquisition• Ortho-image• Shift Shapefile• QualityMask Coverage

Q1 – Data Quality Client

WMS Server

WCS ServerIF-DREAM-

Q-DataQualityProxy

IF-DREAM-Q-AssessQuality,IF-DREAM-Q-OrthoRectify

IF-DREAM-Q-FileIO

IF-DREAM-O-RegisterData

[IF-HMA-Download]

IF-OGC-View

Slide courtesy DREAM Project (ASTRIUM-GEO)

Raw ImageHandler


Case Study 1: DREAM

• DREAM, involving end users with massive data sets, allows to establish relevant demo scenarios. Sample WCPS use cases:

– „all coverages with timestamp younger than T within bbox B where cloud percentage is below P“

• User input: T, B, P

• Web client to allow input of T and P via sliders (or similar), B graphically

– „Bit mask of fires detected in coverage C“

– „band ratio between B1 and B2 for given coverage C“

• User input: B1, B2, C

– Scenario cases to be refined with users.


Case Study 2: LandSAfe

• Landing Site risk Analysis software framework (ESTEC)

– ESA Lunar Lander Mission

– Vision-based Navigation technologies for safe precision landing.

– Hazard detection and avoidance.



• Software modules for mission preparation stage and use during descent stage (live footage)



• Processing requests sent over HTTP/SOAP to the WPS service which executes the request by running Python scripts from the LandSAfe Core.

pyWPS

See modules on previous

slide

See modules on previous

slide


Case Study 3: G-POD

• ESA Grid - Processing On Demand

• WPS access point of G-POD installation is http(s)://<gpod_host>/wps/

• WPS process

– Refers to G-POD service

– Has identifier, title and abstract

– Output is RDF document referring to actual results

• G-POD service parameters are LiteralData (no ComplexData).


Case Study 3: G-POD

• WPS as interface for ngEO Hosted ProcessingFacilities

WPS

e.g.G-POD


Case Study 4: SSEGrid

• SSEGrid Project

– Introduce a Grid Processing on Demand model at VITO to be integrated with Vegetation processing chains

– Portal to integrate multi-Grid processing services to be used by the Principal Investigator (PI) for the systematic processing or re‑processing of EO data in a single or multi-mission environment

– On-demand Processing of EO data

• Large EO data size processing must be done where data is located

• Re-processing by PI processing chains must be deployable by PI


Demonstration Scenario (FAPAR)

On VITO grid

For each product For each product

On ESRIN grid

For each product

ImportHDF_to_ENVI (5)

FAPAR (11)

Calculatedifference (21)

Projection (9)

Get list of SPOT-VGT products (3)

ImportMERIS_to_ENVI

(8)

Composite (10) Composite (10)

UI assign values to inputparameters (1)

Get list of MERISproducts (3)

MERIS FRarchive

(raw segm)

SPOT-VGTarchive

(P-products)

MODISarchive

(Level 1)

Get list of MODISproducts (3)

Projection (7)

ImportMODIS_to_ENVI

(6)Unzip script (4)

VIP MapProjection (15)

VIP Imageextractor (14)

SMAC (12)

NDVI (13)

SMAC (12)

NDVI (13)

(2)

(16)

(17)

(18)

(19)

(20)

(19)(19)

(20)

FAPAR (11) FAPAR (11)

(20)

Composite (10)

… (22)

Fraction of Absorbed Photosynthetically Active Radiation

for each = parallel

execution

data transfer


SSEGrid WPS Hierarchy

PortalFederatingWorkflow

Engine

WPS-G

GridWorkflow

Engine

WPS-G

GridWorkflow

Engine

WPS-G

Grid

WPS-G

Grid

WPS-G

UserProcessing

Infrastructure(UPI)

FederatingProcessing

Infrastructure(FPI)

MissionProcessing

Infrastructure(MPI)

GridAccess

Infrastructure(GAI)

Grid

JavaSAGA

JavaSAGA

PI

One per mission Ground SegmentHierarchy of

WPS-G services


SSEGrid WPS-G Interface

• Encapsulates all forms of processing– Grid Processes

– Workflows

WPS-G

Workflow Engine

Workflow

Workflow

DeploymentMonitoring

Execute

WPS-G

Grid

Met

a-sc

hedu

ler

DeploymentMonitoring

GRAM

MDS

RFT Scheduler

...

Nodes

job queue

Compute Elements

Execute


Highlighting of thresholded data

• Date• Threshold• AOI• Target PNG (as URL)

Image thresholding

Display results

AOI & threshold selection

Image highlighting

• Highlight colour, fixed to (RGB=200,0,0)

• URL of thresholded PNG, from previous step

• Highlighted PNG, from previous step

Case Study 5: SSE



Open-source Implementations

• FP7 GENESIS Toolbox

– Evolution of SSE Toolbox

– Supports WPS 1.0.0 and corrigendum

– Multiple bindings + asynchr.

– http://code.google.com/p/toolboxenvironment/

• PyWPS

– Implementation in Python

• 52°North

– Supports WPS 1.0.0

– Supports WPS-T

– Extended with WPS-G On-demand Processing in SSEGrid


Open-source Implementations

• SSEGrid contribution:

– Asynchronous Execute operation with callback (no polling)

– New WPS-G specific operations

• GetStatus, Cancel

• GetAudit, DeployData, UndeployData, DescribeData

– Two new WPS-T backends

• Apache ODE Process Manager (workflows)

• JavaSAGA Process Manager (Grid processes)

– Backend callback operation

• Available as Open source at 52°North repositoryhttps://svn.52north.org/svn/geoprocessing/main/WPS/branches/WPS-Spacebel


Issues and future work

• Lack of support for:

– Asynchronous operations (not polling)

– Status reporting (Audit)

– Cancel, GetAudit, …

– Quota Management,

– Discovery of SLA, QoS, Performance, availability

• Integrating Authentication/Authorisation and security

– Restrict visibility/access to deployed Processing resources

– Discover access restrictions

• Harmonising data model across services:

– DescribeTasking (SWECommon) versus DescribeProcess



• Consistency with INSPIRE Spatial Data Services and Invoke Service (Draft) Implementing rules

– May or may not retain OGC WPS.

– Oriented towards WSDL and SOAP.



• Completion of standardisationof various interfaces defined in ESA TM-21 « HMA Cookbook »

No yet OGC standards.

Partly underway in WPS 2.0 SWG


Ongoing Standardisation Activities

• SSEGrid Project:

– Two Change Requests submitted for inclusion into WPS 2.0

– Processing On-Demand Extension (WPS-G) to be proposed when WPS 2.0 available

WPS-G

Processing

On-Demand

Extension

BindingWPS-T

Asynchronous



• SSEGrid Project:

– “Processing On-Demand WPS Extension”

– Covers changes and additions to WPS 1.0

– Available at

https://wiki.services.eoportal.org/tiki-download_wiki_attachment.php?attId=1645


WPS with On-Demand Extension

• Feature/operations– Discovery of service metadata (WPS GetCapabilities operation)

– Discovery of process metadata (WPS DescribeProcess operation)

– Execution of processes (WPS Execute operation)

– Deployment of processes (WPS-T DeployProcess operation)

– Undeployment of processes (WPS-T UndeployProcess operation)

– Deployment of (auxiliary) data (WPS-G DeployData operation)

– Undeployment of data (WPS-G UndeployData operation)

– Discovery of data metadata (WPS-G DescribeData operation)

– Query of process status (WPS-G GetStatus operation)

– Cancellation of process (WPS-G Cancel operation)

– Query of process execution trace (WPS-G GetAudit operation)

– Process completion/failure notification (callback operation)

WPS-G

Workflow

1: Execute

2: Callback

Addressed by WPS 2.0

SWG



• DREAM Project:

– “Identify the requirements and scenarios, extend the WCPS to cover the needed multidimensional queries in particular for what concerns space/time phenomena related to the joint exploitation of S-2, optical high resolution and other data including geo-statistical data as the one routinely provided by EUROSTAT and other international organisations. “

– WCPS work items:

• WCPS 1.0 based on WCS 1.1.2. Standardisation effort required to align with GMLCOV 1.0 and WCS 2.0 (Core)

• ATS to be updated

• ETS to be created


Planned Standardisation Activities

• HMA-S Project:

– Revise WPS Specification taking into account:

• ESA project’s technical report “SSEGrid Processing On-Demand Extension (WPS-G)”

• ngEO needs in terms of process description and access to the processing outputs

• Relevant third party initiatives, about cloud computing standardisation, such as IEEE P2301.

• WPS 2.0 SWG ongoing work

• ESE Project:

– Create new WPS-G deployment profile for Map-Reduce processing environment on dynamic resources (cloud).


Conclusion

• Two main « Processing » specifications: WPS and WCPS

• WPS:– Widely adopted in ESA projects

– Several open-source implementations available

– Many subjects to be resolved, improved or not yet a formal standard

• E.g. asynchronous, WPS-T, …

– Definition of OGC WPS 2.0 underway but slow process

• WCPS:– Less widely adopted

– To be used in DREAM and ESE projects

• Various ongoing or planned ESA projects contribute to standardisation effort of both specifications.


Conclusion

• Further Reading:

– HMA Cookbook: http://esamultimedia.esa.int/multimedia/ publications/TM-21/TM-21.pdf

– http://www.opengeospatial.org/standards/wps

– http://www.opengeospatial.org/standards/wcps

– http://inspire.jrc.ec.europa.eu/documents/Network_Services/position_positionpaper_06062011pd__EN_1.pdf

Documents

Page 1Session 4: State of the art technology and standards, 12/10/2012 Web Processing Service Models for scientific exploitation of EO Data, Session 4: