Upload
others
View
9
Download
0
Embed Size (px)
Citation preview
1 ACT Luxembourg - INSPIRE roadmap and architecture: lessons learned
INSPIRE roadmap and architecture: lessons learned INSPIRE 2017
Stijn Goedertier – GIM Thierry Meessen – GIM Jeff Konnen – ACT Luxembourg Patrick Weber – ACT Luxembourg
2
Administration du cadastre et de la topographie (ACT) Luxembourg
ACT Luxembourg - INSPIRE roadmap and architecture: lessons learned
• Responsible for:
• National cadastral plan
• Differential GPS network
• Aerial photo’s and maps
• Open data portal: data.public.lu
• Geoportal: geoportail.lu
• INSPIRE: catalog.inspire.geoportail.lu
• …
3
#1 Make a plan
ACT Luxembourg - INSPIRE roadmap and architecture: lessons learned
• ACT is responsible for the INSPIRE data
harmonisation for the whole of Luxembourg
• 253 datasets from 22 government entities
• Made a 2016-2021 roadmap
• Tasks, roles & responsibilities, estimated timing and
effort
4
#2 Review the application architecture
ACT Luxembourg - INSPIRE roadmap and architecture: lessons learned
Data harmonisation FME Desktop / FME Server 2017
Metadata & discovery service GeoNetwork
View service GeoServer
Download service GeoNetwork (ATOM/OpenSearch)
Data store Postgres/PostGIS
File store AWS S3 (via uData)
Validation ETF-WebApp, GDI-DE Testsuite, INSPIRE validator
Components were replaced by their open-source
counterparts (where feasible)
5
#3 Keep things as simple as possible
ACT Luxembourg - INSPIRE roadmap and architecture: lessons learned
• e.g. Metadata: one language only: English
• e.g. Metadata: only mandatory metadata elements
• e.g. Open Data: no access restriction means no
authentication and authorisation
• e.g. Service capability document: only minimal
extension (scenario 1)
6
#4 Go for a file download service
ACT Luxembourg - INSPIRE roadmap and architecture: lessons learned
• Minimally a predefined dataset download service: i.e.
a file download service ATOM/OpenSearch or WFS2.0
stored queries)
• The user must download the entire dataset as a file
• Requires no effort from data publisher (when
ATOM/OpenSearch generated from metadata)
• Where practicable a direct access download service
(i.e. a WFS2.0 query service)
• The user can query and get the feature(s) of interest
• Implementations like GeoServer App-Schema have
many limitations
• QoS (availability, response, throughput) is more
difficult to guarantee
7
#4 Go for a file download service
ACT Luxembourg - INSPIRE roadmap and architecture: lessons learned
predefined dataset download service:
• Implementation: GeoNetwork was enhanced to
support local atom feeds generated automatically on
the basis of the service and dataset metadata
records.
• Conformance testing: Implementation was tested
with the GDI-DE test suite.
• Benefits:
• No additional effort required from data
publisher (just correct metadata records)
• QoS easy to guarantee
8
#4 Go for a file download service
ACT Luxembourg - INSPIRE roadmap and architecture: lessons learned
direct access download service:
• Implementation: GeoServer Application Schema.
• Conformity: good results with ETF-Webapp
• Drawbacks:
▶ Requires transformation of source database
▶ Not possible to distinguish multiple datasets
(only feature types) (multiple virtual WMS not
possible with App-Schema)
▶ QoS more difficult to guarantee
9
#5 Speed up deployment with Docker
ACT Luxembourg - INSPIRE roadmap and architecture: lessons learned
• Dockerfiles / docker images for:
• FME Server
• PostgreSQL,
• GeoServer,
• GeoNetwork,
• ETF-Webapp
• NGINX
• INSPIRE dashboard.
• Building images and starting-up containers in concert
using Docker-compose or Rancher
10
#6 Automate conformance testing
ACT Luxembourg - INSPIRE roadmap and architecture: lessons learned
• Metadata: INSPIRE Geoportal Validator
• Data: ETF-Webapp (ARE3NA ETS repository)
• Network Services: GDI-DE TestSuite
11
#7 Automate the workflow
ACT Luxembourg - INSPIRE roadmap and architecture: lessons learned
FME
GeoServer
View (WMS) and download (WFS) services
ReST API
GeoNetwork
Data Catalog
CSW-T
API
uData
Data Portal
uData
API
INSPIRE
Validation service
ReST
API
1. Harmonise dataset
2. Upload harmonised dataset
3. Configure view service
4. Update metadata records
5. Test conformance of data, metadata, and services
12
#7 Automate the workflow - 1. Harmonise data with FME
ACT Luxembourg - INSPIRE roadmap and architecture: lessons learned
• XML Schema: write GML according to 34 built-in
schemas… or your own
• supports INSPIRE complexities: multiple
geometries, nested features, object referencing,
repeating elements, nillable elements, grids …
How to map features into the right schema?
13
#7 Automate the workflow - 1. Harmonise data with FME
ACT Luxembourg - INSPIRE roadmap and architecture: lessons learned
What is the SchemaMapper?
restructures the existing schema (data model) of
incoming features to a new schema, based on mappings
defined in an external lookup table.
very useful
▶ when the mappings are potentially complex,
▶ when they need to be maintained by someone
unfamiliar with FME.
14
#7 Automate the workflow - 1. Harmonise data with FME
• source_feature_type target_feature_type: mappings of
features types.
• source_attr_name target_attribute_name: mappings of
attributes.
• Attribute filters: new attribute-value combinations
(filter_target_attribute_name – filter_target_attribute_value),
conditional to the presence of attribute-value combinations in the
source feature (filter_source_attribute_name –
filter_source_attribute_value).
source_feature_type target_feature_type source_attr_name target_attr_name
UNESCO_vieux_quartiers_fortifications_LUX_2zones ProtectedSite
UNESCO_vieux_quartiers_fortifications_LUX_2zones
UNESCO_vieux_quartiers_fortifications_LUX_2zones
UNESCO_vieux_quartiers_fortifications_LUX_2zones
UNESCO_vieux_quartiers_fortifications_LUX_2zones
UNESCO_vieux_quartiers_fortifications_LUX_2zones Id inspireID.Identifier.localId
UNESCO_vieux_quartiers_fortifications_LUX_2zones
UNESCO_vieux_quartiers_fortifications_LUX_2zones Lien_Link1
UNESCO_vieux_quartiers_fortifications_LUX_2zones Lien_Link2 gml_descriptionReference.xlink_href
UNESCO_vieux_quartiers_fortifications_LUX_2zones Nom_Name siteName{0}.GeographicalName.spelling{0}.SpellingOfName.text
UNESCO_vieux_quartiers_fortifications_LUX_2zones
UNESCO_vieux_quartiers_fortifications_LUX_2zones
UNESCO_vieux_quartiers_fortifications_LUX_2zones Type_Typ
UNESCO_vieux_quartiers_fortifications_LUX_2zones
UNESCO_vieux_quartiers_fortifications_LUX_2zones
UNESCO_vieux_quartiers_fortifications_LUX_2zones Ville_Stad
UNESCO_vieux_quartiers_fortifications_LUX_2zones Zone
UNESCO_vieux_quartiers_fortifications_LUX_2zones Zone
15
#7 Automate the workflow - 1. Harmonise data with FME
ACT Luxembourg - INSPIRE roadmap and architecture: lessons learned
A simple workspace looks as follows
(sometimes more operations required)
16
#7 Automate the workflow - 2. Upload dataset with FME
ACT Luxembourg - INSPIRE roadmap and architecture: lessons learned
Upload datasets to a data portal using an Application
Programming Interface (API)
https://www.data.gouv.fr/en/apidoc/#!/dataset
s/create_resource
17
#7 Automate the workflow - 2. Upload dataset with FME
ACT Luxembourg - INSPIRE roadmap and architecture: lessons learned
import requests
url = 'https://data.public.lu/api/1/datasets/57c07be'
headers = {'X-API-KEY': 'eyJhb'}
data = {
'title': 'Dataset title',
'description': 'Dataset abstract',
'tags': 'keywords',
'private': 'true',
'spatial': {'geom': 'null', 'granularity': 'other',
'zones': []}
}
response =
requests.put(url,headers=headers,json=data)
print(response.text)
18
#7 Automate the workflow - 3. Configure view service with FME
ACT Luxembourg - INSPIRE roadmap and architecture: lessons learned
Use the GeoServer ReST API to create workspaces, stores,
layers, styles, and WMS services.
http://docs.geoserver.org/latest/en/user/rest/api/index.html
19
#7 Automate the workflow - 3. Configure view service with FME
ACT Luxembourg - INSPIRE roadmap and architecture: lessons learned
HTTPCaller: general-purpose ReST client
20
#7 Automate the workflow - 4. Update metadata records
ACT Luxembourg - INSPIRE roadmap and architecture: lessons learned
Data and service metadata records are managed with
GeoNetwork.
Standards: ISO19139, CSW, INSPIRE TG metadata &
discovery service
21
#7 Automate the workflow - 4. Update metadata records
ACT Luxembourg - INSPIRE roadmap and architecture: lessons learned
• Use Xquery update statements to update the
metadata record
• Use CSW-T API to post the updated metadata record
to GeoNetwork
• Also for service metadata (e.g.
<srv:coupledResource> <srv:operatesOn>)
22
#7 Automate the workflow - 5. Test conformance
ACT Luxembourg - INSPIRE roadmap and architecture: lessons learned
▶ ETF-webapp validator: FME HTTPCaller used to execute tests
23
Conclusion: lessons learned
ACT Luxembourg - INSPIRE roadmap and architecture: lessons learned
• #1 Make a plan
• #2 Review the application architecture
• #3 Keep things as simple as possible
• #4 Go for a file download service
• #5 Speed up deployment with Docker
• #6 Automate conformance testing
• #7 Automate the publication workflow