Embed Size (px)
Citation preview
PROBA-VMISSION EXPLOITATION PLATFORM
ERWIN GOOR, VITO
PROBA-VMISSION EXPLOITATION PLATFORM
ERWIN GOOR, VITO
VITO – COMPANY PROFILE
VITO - Flemish Institute for Technological Research, Belgium
750 employees, 26 nationalities, 140 mio€ turnover in 2014
Remote Sensing unit: 80+ employees
4 October, 20162
Energy Materials Chemistry
Health Land use
VITO REMOTE SENSING PRODUCTS AND SOLUTIONS
4 October, 20163
Infrastructure
+150 Tbytes/year
Time series since 1999Global extend
PROBA-V MEP Proposal | 31/03/2015 | Slide 5
ESA UNCLASSIFIED – For Official Use
Future: “bring the users to the data”
Mission operator
data
Current: “bring the data to users”
data download
data request
BorderGround Segment User Segment
Long feedback loop
"bring the users to the data”
results
discussion/needs Community(science, services, others)
user
+processing resources
dataMission operator
Easier for userMore dataMore resultsBetter serviceMore community spirit
user
results
discussion/needsProcessing & data requests
Community(science,
service prov.others)
Shorter feedback loopUser segment is within the Ground segment
+processing resources
PROBA-V MEP CONCEPT
04 October 20166
Bring the users to the data by offering a platform with» complete archive of PROBA-V & SPOT-VEGETATION
& derived products from PV_MEP users& HR EO/reference data to support Cal/Val activities on the platform
» Scalable ICT resources (private cloud) & processing middleware (Hadoop)» Users develop applications & can offer this as on-demand service
» A growing set of tools & software from various contributors» Documentation & collaborative tools» User support
Future links with other
Exploitation Platforms [federation]
Develop Operate Support users
proba-v-mep.esa.int
MEP user groupsMEP application flexibility
DIVERSE USERS & APPLICATIONS
04 October 20168
Viewers, Information Dashboards
Notebooks, On-demand
Processing
Virtual Research Environment, API’s
Any user
EO data Expert
Developers
# UsersFl
exib
ility
FULL RESOLUTION VIEWINGTIME SERIES VIEWINGPRE-DEFINED ON-DEMAND
PROCESSING SERVICEVIRTUAL
RESEARCH ENVIRONMENTNOTEBOOKS - WEB-BASED
INTERACTIVE COMPUTINGINFORMATION DASHBOARDS
4 October, 20169
Core Platform
Data Layer
Applications
GeoViewer Timeseries Viewer JupyterHUB
Libraries
Geotrell is Spark GRASS Geotools GDAL
Composite On Demand
Through Data ManagerFull Archive Full ArchiveFull Archive Cache via DataMgrCache via DataMgr
Aux. Data…
Aux. Data…
04 October 201610
(1) Black box
YOUR ALGORITHM ON THE MEP: SEVERAL POSSIBLE APPROACHES
YOUR ALGORITHM
Input tiles
Auxiliarydata
Output filesOutput files
(2) Tile based processing
Write Spark (Python/Java) script to parallelize processing:
myTiles.forEach(myAlgorithm)
Input tile2
YOUR ALGORITHM
Input tile1
Aux. data
Output file1 Output file2
Per tile processing
(3)Maximise use of Spark
Write Spark program, reuse of existing code is possible if it is sufficiently modular.
Advantages:• Best performance• All Hadoop features available
• Distributed Machine learning• Streaming• In memory distributed
processing
Input tile2
YOUR SPARK ALGORITHM(Python/Java/Scala)
Input tile1
Aux. data
Output file1 Output file2
EXAMPLE DASHBOARD ARCHITECTURE
04 October 201612
Data sources Batch Jobs Backend Client Side
Geoserver
Elasticsearch ElasticSearch SecurityEnforcement REST
OLAP Data Cube REST - MDX
Spark ETL
Image processingChain EO Dashboard
Drupal
Raw Satellite/UAVData
Meteo Data
Ground Data
OGC Web CoverageService
OGC Web Map TileService
AngularJS
Widgets
OpenLayers
DEVELOPMENT PLAN (SCHEDULED DATES)
11 January 201613
Nov 2015 Mid 2018
Operations (*)Operations (*)
Early delivery(pre-operations)
Iteration 1 Iteration 2 Iteration 3
26 Jan 2016
July 2016
Iteration 1Iteration 1
June 2017
Iteration 2Iteration 2
January 2018
Iteration 3Iteration 3
Transfer to operationsTransfer to operations
(*) includes continuous user support & data/service integration
04 October 201614
State-of-the-art tools for
BIG Data
Processing & analytics
OperationsRelevant EO-data …
growing in size
04 October 201615
Open solution that can grow easily
Open Source approach
Runs on own metal
Users can develop-debug-run on ‘private cluster’ on Openstack
Support for both human & machine users
Automated provisioning and Configuration Mgt of machines and middleware – DevOps approach
FIRST PILOT ‘USER’ APPLICATIONS
» First pilot user applications on the PROBA-V MEP: Oct 2016 - 2017:
» PROBA-V Toolbox (also as plug-in to existing toolboxes) » Land Cover Monitoring » Snow Mapping » Detection of Disturbances in Natural Vegetation » Detection of Fires / Burned Areas» Desert Locust
» Looking for further collaborations
» Applications, tools, Software components, … develop pre-operational service» More EO data: data copy OR links with other Exploitation Platforms» Focus on machine-to-machine interfaces: discovery – viewing – access – analytics –
processing - …» Expertise on Hadoop – OpenStack – Data analytics - …
04 October 201616Geo application platform with open access to data via partnerships
EXPERIENCES FROM PROBA-V MISSION EXPLOITATION PLATFORM
» When services are available as RESTful i/f mimimal effort to integrate.» Still little services are available/known e.g. meteo data at global scale» Federating infrastructures we are ready to do! but no theoretical exercise is needed!» In-situ data exercise by VITO in SIGMA project – STAC DB
1. Lesson(s) learned on the use of data/information from heterogeneous sources
4 October, 201617
4 October, 201618
EXPERIENCES FROM PROBA-V MISSION EXPLOITATION PLATFORM
» When services are available as RESTful i/f mimimal effort to integrate.» OGC WPS? … does not really ensure interoperability» Still little services are available/known e.g. meteo data at global scale» Federating infrastructures we are ready to do! but no theoretical exercise is needed!» In-situ data exercise by VITO in SIGMA project – STAC DB
» Notebooks potential to share scientific results» https://proba-v-mep.esa.int/sites/proba-v-mep.esa.int/files/notebooks/trend_analysis.htm
» Sharing/Export of dashboards ‘state’» requirements are often to export as png, csv, … of single components
1. Lesson(s) learned on the use of data/information from heterogeneous sources2. Lesson(s) learned on the user needs for sharing of data/information
4 October, 201619
EXPERIENCES FROM PROBA-V MISSION EXPLOITATION PLATFORM
» When services are available as RESTful i/f mimimal effort to integrate.» OGC WPS? … does not really ensure interoperability» Still little services are available/known e.g. meteo data at global scale» Federating infrastructures we are ready to do! but no theoretical exercise is needed!» In-situ data exercise by VITO in SIGMA project – STAC DB
» Notebooks potential to share scientific results» https://proba-v-mep.esa.int/sites/proba-v-mep.esa.int/files/notebooks/trend_analysis.htm
» Export of dashboards requirements are often to export as png, csv, … of single components
» User needs ‘easy-to-use’ & performing Web services for M2M communication INSPIRE as clearstimulus and good guidance … but sometimes complex and on-demand processing/analytics is not fully addressed.
1. Lesson(s) learned on the use of data/information from heterogeneous sources2. Lesson(s) learned on the user needs for sharing of data/information 3. Lesson(s) learned on the user needs for INSPIRE compliance and the benefits of INSPIRE
4 October, 201620
EXPERIENCES FROM PROBA-V MISSION EXPLOITATION PLATFORM
» Two measures» Focus on open standards which are easy to use: ‘openSearch’, easy A&A» Exploitation Platform needed for Copernicus global land (*) since time series are
involved» Web services & dashboards for data analytics» Hadoop for parallel processing» VM’s for prototyping & development backend can run on the platform
(*) not foreseen yet!
4. Solution(s)/recommendation(s) on how to integrate Copernicus data/information in existing business/working processes
4 October, 201621
EXPERIENCES FROM PROBA-V MISSION EXPLOITATION PLATFORM
» Two measures» Focus on open standards which are easy to use: ‘openSearch’, easy A&A» Exploitation Platform needed for global land since time series are involved
» Web services & dashboards for data analytics» Hadoop for parallel processing» VM’s for prototyping & development backend can run on the platform
(*) not foreseen yet!
» Open architecture, based on open-source components.
4. Solution(s)/recommendation(s) on how to integrate Copernicus data/information in existing business/working processes
5. How can/could the solution presented today be possibly applied in a completely different sector?
4 October, 201622
proba-v-mep.esa.int
