Using the Network Common Data Form for storage of atmospheric data

Using the Network Common Data Form for storage of atmospheric data

Maarten Plieger Maarten Plieger

Royal Netherlands Meteorological Institute

The NetCDF4 data format (Network Common Data Form) is used within the ADAGUC project for the storage of atmospheric data

The NetCDF4 data format (Network Common Data Form) is used within the ADAGUC project for the storage of atmospheric data

http://a

daguc.k

nm

i.nl

plieger@

knm

i.nl

Euro

pean G

eosciences U

nion 2

009

Introduction

• The ADAGUC project

• Adaguc file standard

• NetCDF4 data format

• Climate & Forecast (CF) conventions

• GDAL/OGR Drivers

• OGC Webservices WMS, WFS, WCS

• ADAGUC files in GIS

http://a

daguc.k

nm

i.nl

plieger@

knm

i.nl

Euro

pean G

eosciences U

nion 2

009

ADAGUC

Atmospheric Data Access for the Geospatial User Community

• Bridging different sciences

• Atmospheric Datasets in GIS systems

• Standard data format for all data

• Open Standards � OGC

• Connect with GIS clients through OGC standards

This project is sponsored bySpace for Geo-Information

www.rgi.nlhttp://adaguc.knmi.nl/

http://a

daguc.k

nm

i.nl

plieger@

knm

i.nl

Euro

pean G

eosciences U

nion 2

009

DatasetsDatasets

• Instruments:

• Sciamachy

• GOME

• OMI

• AMSR

• Precipitation radar

• Instruments:

• Sciamachy

• GOME

• OMI

• AMSR

• Precipitation radar

Goal: Store data in a generic format and provide access to the data by using standard OGC Services

Goal: Store data in a generic format and provide access to the data by using standard OGC Services

• Datasets:• Soil Moisture• FRESCO• NO2

• CH4

• CO• O3

• Precipitation

• Datasets:• Soil Moisture• FRESCO• NO2

• CH4

• CO• O3

• Precipitation

• Current data types:• Raster• Vector

• Current data types:• Raster• Vector

RasterRaster VectorVector

http://a

daguc.k

nm

i.nl

plieger@

knm

i.nl

Euro

pean G

eosciences U

nion 2

009

Adaguc file standardAdaguc file standard

• NetCDF 4 data format

• Metadata conventions:

• INSPIRE compliant: ISO-19115

• NL kernset Metadata Standard

• Climate and Forecast metadata conventions 1.4

• Data and metadata reside in the same file

• ISO metadata describes the data on dataset level

• Climate and forecast conventions describes the data level

• NetCDF 4 data format

• Metadata conventions:

• INSPIRE compliant: ISO-19115

• NL kernset Metadata Standard

• Climate and Forecast metadata conventions 1.4

• Data and metadata reside in the same file

• ISO metadata describes the data on dataset level

• Climate and forecast conventions describes the data level

ISO core setISO core set

EU core set (CEN TC 287)EU core set (CEN TC 287)

NL core setNL core set

ADAGUC metadataADAGUC metadataCF Conventions 1.4CF Conventions 1.4

http://a

daguc.k

nm

i.nl

plieger@

knm

i.nl

Euro

pean G

eosciences U

nion 2

009

Network Common Data Form 4 (netCDF 4)

Network Common Data Form 4 (netCDF 4)

• Joint project between Unidata and HDF Group

• NetCDF 4 uses HDF5 as the storage layer of NetCDF

• Programming interfaces are backward compatible with the netCDF3 programming interface

• Joint project between Unidata and HDF Group

• NetCDF 4 uses HDF5 as the storage layer of NetCDF

• Programming interfaces are backward compatible with the netCDF3 programming interface

NetCDF3popular, simple, lots of tools,

multiple implementations

HDF5powerful, high-performance,

storage efficiency, extensibility

NetCDF4

Best of both formats combined!

http://a

daguc.k

nm

i.nl

plieger@

knm

i.nl

Euro

pean G

eosciences U

nion 2

009

Climate & forecast (CF) conventions

Climate & forecast (CF) conventions

• Standard names, standard units…

• Identify and compare data

• Locate data in space–time and as a function of other independent variables

• Coordinate variables:

• Standard names, standard units…

• Identify and compare data

• Locate data in space–time and as a function of other independent variables

• Coordinate variables:

http://a

daguc.k

nm

i.nl

plieger@

knm

i.nl

Euro

pean G

eosciences U

nion 2

009

Structure of an ADAGUC fileStructure of an ADAGUC file

attributes with metadata coordinate variable data

iso_dataset: The same for each file from this dataset

product: Can be different for each file

http://a

daguc.k

nm

i.nl

plieger@

knm

i.nl

Euro

pean G

eosciences U

nion 2

009

Unique identifiers (UUID)Unique identifiers (UUID)

• Unique identifiers to make the data distinguishable

• Google on UID provides the description of the data!

• Unique identifiers to make the data distinguishable

• Google on UID provides the description of the data!

http://a

daguc.k

nm

i.nl

plieger@

knm

i.nl

Euro

pean G

eosciences U

nion 2

009

GDAL/OGR driversGDAL/OGR drivers

• GDAL / OGR

• Open source translator library for raster geospatial data formats

• Used in many software products

• UMN MapServer, GeoServer, Quantum GIS, ArcGIS, GRASS, OpenEV etc..

• GDAL - Geospatial Data Abstraction Library

• Access raster data

• OGR - Simple Feature Library

• Access vector data

• GDAL/OGR extensions available with support for the ADAGUC data standard

• GDAL / OGR

• Open source translator library for raster geospatial data formats

• Used in many software products

• UMN MapServer, GeoServer, Quantum GIS, ArcGIS, GRASS, OpenEV etc..

• GDAL - Geospatial Data Abstraction Library

• Access raster data

• OGR - Simple Feature Library

• Access vector data

• GDAL/OGR extensions available with support for the ADAGUC data standard

http://http://trac.osgeo.org/gdal/wiki/ADAGUCtrac.osgeo.org/gdal/wiki/ADAGUC

http://a

daguc.k

nm

i.nl

plieger@

knm

i.nl

Euro

pean G

eosciences U

nion 2

009

OGC WebservicesOGC Webservices

• Why OpenGIS services?

• Data discovery

• Multiple output formats

• Resampling/ interpolation

• Spatial/temporal selections

• Connection to GIS programs

• Enabled by UMN MapServer

• GDAL � UMN MapServer

• Web Mapping Service (WMS) for visualization

• Web Feature Service (WFS) for feature data access

• Web Coverage Service (WCS) for raster data access

• Why OpenGIS services?

• Data discovery

• Multiple output formats

• Resampling/ interpolation

• Spatial/temporal selections

• Connection to GIS programs

• Enabled by UMN MapServer

• GDAL � UMN MapServer

• Web Mapping Service (WMS) for visualization

• Web Feature Service (WFS) for feature data access

• Web Coverage Service (WCS) for raster data access

http://a

daguc.k

nm

i.nl

plieger@

knm

i.nl

Euro

pean G

eosciences U

nion 2

009

ADAGUC Web PortalADAGUC Web Portal

WMS GetCapabilities

WMS GetMap

WMS GetFeatureInfo

WMS GetLegendGraphic

http://a

daguc.k

nm

i.nl

plieger@

knm

i.nl

Euro

pean G

eosciences U

nion 2

009

Download data from the PortalDownload data from the Portal

• Download the original file,or convert to your favorite GIS format using OpenGIS

• Download the original file

• Raster files are downloaded using WCS• Vector files are downloaded using WFS• Depending on the datatype, which service is needed?

• The right service is automatically selected

• Download the original file,or convert to your favorite GIS format using OpenGIS

• Download the original file

• Raster files are downloaded using WCS• Vector files are downloaded using WFS• Depending on the datatype, which service is needed?

• The right service is automatically selected

WCS DescribeCoverage

WFS MetadataURL

WFS or WCS?

http://a

daguc.k

nm

i.nl

plieger@

knm

i.nl

Euro

pean G

eosciences U

nion 2

009

ADAGUC in GIS programsADAGUC in GIS programs

ArcGISArcGIS ExplorerExplorer Google EarthGoogle Earth Quantum GISQuantum GIS

gvSIGgvSIG ArcGISArcGIS DesktopDesktop UDIGUDIG

IDVIDV HDF ExplorerHDF Explorer NASANASA’’S PanoplyS Panoply