Upload
gustavo-stapp
View
215
Download
0
Tags:
Embed Size (px)
Citation preview
Kick-off meetingDelft, April 9-11 2014
7FP – SPACEno. 607131
Data management
@ Fast First step in data management – repository
Gerrit Hendriksen
Gerben J. de Boer
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
Preview
tailored data> WCS> WFS> SOS> SOAP
> netCDF-CF- OPeNDAP> ISO SQL– PostGIS> SVN
> GIT> http> ftp
> KML> WMS> WFS
> CSWgraphics of data
standard dataraw
data
catalogueof data
work done on server
work done on client
Exchange, develop
standards
spatio-temporal
standards
database standards
(lab and field data)
ISO standards
whatwherewhenwho whyhow…data URLs
smart phone & tablet usersscientists professionals
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
In relation with WP’s/ Tasks
D4.3D4.3
D4.1D4.1
D4.5D4.5
T5.2T5.2
D4.8D4.8
D3.3D3.3
datamanagementdatamanagement
DisseminationDissemination
D4.8D4.8D4.1D4.1 T5.2T5.2D3.3D3.3
T5.1T5.1
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
Techniques in data management
Version control (SubVersion, GIT,…)
OGC netCDF+ OPeNDAP
OGC CF Conventions
OGC PostGIS
GeoServer
CSW
ncWMS, ADAGUC
Data & tools as-isvia version control
General purposedata storage:syntax standard
Geospatialsemantic standard
Server side OGCtailoring & graphics
Discoverymeta-data catalog
ISO SQL(PostgreSQL
)
geospatial semantics
servlets
ETL
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
Data management systeem
(DAS)
modelling
(DDS)
(DAS)(DAS)(DMS)
Mobile applications
Monitoring Network
EO/RS
EO/RS tools
Version control
DAS – Data Acquisition Sub-system DMS - Data Management Sub-system DDS - The Data Dissemination Sub-system
Catalogueservice
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
Open source and - standards
• Software• PostgreSQL• Geoserver• Geonetwork• Mapnik• (QGIS)• Python• OpenStreetMap
• OGC (exchange of data)
• Semantic vocubularies• CF• WORMS (PESI)
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
Reason
?
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
Quality assurance
Vessel visualisation of location and speed of fish tracks carried out during monitoring.
The red lines show deviation from the protocol (maximum speed between 5-10 km/hour). It seemed that this was due
to input errors.
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
Clear standards
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
For FAST
• Data will be available on an opensource platform:– Scientists– Basis of dissemination- and calculation platform
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
First step in generalization
• Repository• Web-based system for version control• Allows a worldwide community of people
to collaborate online• Everyone (with proper credentials) can
add/alter/remove and retrieve data• Nothing can be added/altered/removed
without being noticed• Makes sure that changes are merged
without conflicts
• Bear in mind, it is password controlled, so not fully open
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
Frequently used commands
Commit
Checkout (sparse)
Update
Add/Delete/Modify
Log
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
Checkout (sparse)
Create a local copy (only once) Checkout Depth (Windows)
•Fully recursive•Immediate children, including folders•Only file children•Only this item
svn checkout --depth (Command line)•Infinity•Immediates•Files•Empty
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
Update
Download: Server -> Local copy
Note: update before working on a file!
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
Commit
Upload: Local copy –> server Always add a log message!
Command line: svn commit -m”[insert log message]”
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
Add/delete
• Svn addUsed when a new file or folder is added to the repository, before committing. Otherwise, the file will only stay local
• Svn deleteUsed when a file or folder should be deleted. Otherwise, the file or folder is only deleted locally and will reappear at the following update
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
• Language: English (folder names, file names, code)
• Don’t use spaces in file/folder names, preferably no capitals
• Always add a log message (also in English)
• Update regularly!
conventions
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
In practice
• Install subversion client– Tortoise (http://tortoisesvn.net/)
• Sparse checkout of the repository of you country• Identify some data• Commit data• Checkout the data
• http://repos.deltares.nl/repos/FAST (mind the capitals)
• temporary username: oegast• temporary password: 0terra
Let’s do it!!!
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
Meta-data for cataloguing: save *.xml
• Think about copyright and ownership already now!• Think about life cycle after end of FAST: shift to library / Datacentre• DOI
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
OpenEarth wiki (openearth.eu)