Embed Size (px)
Citation preview
The CERA2 Data Base
Data input – Data output
Hans Luthardt
Model & Data/MPI-M, Hamburg
Services and Facilities of DKRZ and Model & DataHamburg, 29.3.2006
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 2
Data Input
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 3
Including data into CERA
Data sets from users can be included into the data base and madeaccessible (by internet)
Inclusion is done by Model & Data if following conditions are met:
• data are of general interest• data should become available to a broader community• data size is small relative to the data archive size • a complete set of meta data (fitting the CERA standard) must be
provided
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 4
Store datadata: data must be provided in a structure that can be
transferred into the data base without further processing
meta data: XML-files for the experiment and each of the data setsmust be provided
XML-templates are available from :
http://cera-www.dkrz.de/Meta_Fill/ orhttp://cera-www.dkrz.de/Meta_Fill/mod_metadata.html
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 5
Preferred storage structure for 3-d gridded data:
• single variable• single level • time series• grib formatted / netCDF formatted (/ others)
Data structures
Application related data structure (2-D)(180 KByte/blob)
original data structure (4-D)(3 MByte/blob)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 6
Retrieve data sets
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 7
Retrieve data sets
Database content:
220 TByte data volume
650 experiments80 000 data sets
Database content:
220 TByte data volume
650 experiments80 000 data sets
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 8
Data and Meta dataCERA Metadata Model
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 9
Search strategies
Keyword Project
Experiment
Dataset
Information
Download
Variable
specific Dataset(by name)
Postprocessing
Code
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 10
Access to the data base
Access to the CERA data base:
• by java servlet - using an internet browser• by java applet (java runtime required) - obselet!• by java command line tool (jblob )
Unrestricted browsing for meta data
User account required for data download
Start: http://www.mad.zmaw.de/
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 11
Access to the data base
Retrieval by script :
• Java based – for Unix/Linux and Windows
• http and JDBC connection
examples :jblob –datasetname <name> -optionsjblob –datasetid <id> -optionsjblob –showdatasets ”<search_string>” (% as wildcard)
examples :jblob –datasetname <name> -optionsjblob –datasetid <id> -optionsjblob –showdatasets ”<search_string>” (% as wildcard)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 13
Access to the data base
Access by java applet:(obsolet!)
e.g.hierarchical topic search
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 14
Browse/Retrieve Metadata
see wini.wdc-climate.de
Metadata access via WWW:
• xsql query to DB
• xml output from DB
• xsl mapping to any metadata format
raw xml
xhtmlISO xml
DC xml
... variousmetadataformats
http:XML
xsl –mappi
ngxs
ql –
quer
ylnternetApplicati
onServer
request: URL
user applications
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 16
Search for data
The servlet interface currently allows search for:
• Project – experiment – data set (keyword)
• Data set names (or part of it)
• Data set associated with codes
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 17
Hierarchical search for data sets
Project
Experiment
data set
keywords
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 18
Access to the data base
Access to the CERA data base:
• by java servlet - using an internet browser• by java applet (java runtime required) - obselet!• by java command line tool (jblob)
Unrestricted browsing for meta data
User account required for data download
Start: http://www.mad.zmaw.de/
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 19
Search for data (hierarchical)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 20
Search for data (hierarchical)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 21
Search for data (hierarchical)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 22
Search for data (hierarchical)
List of data sets
Metadata output
Experiment decsription
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 23
Search for data (hierarchical)
List of data sets
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 24
Search for data (hierarchical)
Data set description
Parameter/Data organisation
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 25
Search for data (hierarchical)
Data set description:data organisation
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 26
Download data sets
x
Selecting data setfor download(login required)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 27
Search data sets (by Code)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 28
Search data sets (by Code)
Specifying code table
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 29
Search data sets (by Code)
Specifying code number
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 30
Search data sets (by Code)
Selecting data set
Add to process list
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 31
Search data sets (by dataset name)
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 32
Search data sets (by dataset name)
Specifyingname elemets
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 33
Search data sets (by dataset name)
Selecting data set
Add to process list
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 34
Download data sets
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 35
Download data sets
Process list:initiating data download
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 36
Download data sets
Specifying timerange for download
Time range
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 37
Download data sets
Specifying processing options
region
format
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 38
Download data sets
Startingdownload
H. Luthardt (M&D/MPI-M, Hamburg) 29.3.06 / 39
Contact
URLs:
M&D : http://www.mad.zmaw.de/Data Base Portal : http://cera-www.dkrz.de/CERA/WDCC : http://www.wdc-climate.de
email: [email protected]
