SDMX TOOLSWorkshop on Data Transmission

Becici – Montenegro17- 19 June 2009

Vincenzo PATRUNOISTAT

Overview

There are several tools that help us working with SDMX files

Released under GPL or EUPL license

http://sdmx.org/?page_id=13http://sdmx.org/?page_id=13

http://www.osor.eu/communities/oss-and-statistics/wiki/sdmx-toolshttp://www.osor.eu/communities/oss-and-statistics/wiki/sdmx-tools

SDMX Converter

http://sdmx.org/?page_id=13http://sdmx.org/?page_id=13

SDMX-IM

From SDMX-IM to messages

SDMX-EDI SDMX-ML

• Based on a common Information Model– SDMX-EDI (GESMES/TS)

• EDIFACT syntax• Time series oriented – One format for Data Sets

– SDMX-ML• XML syntax• Four different formats for Data Sets• Easier validation (XML based)

From SDMX-IM to messages

SDMX-ML

XML format for the exchange of SDMX-structured data and metadata.

Name of message Short description Schema file

1 Structure Definition Message Contains a data structure definition Fixed

2 Generic Data MessageConveys data in a form independent of a data structure definition. It is designed for data provision on websites and in any scenario where applications receiving the data may not have detailed understanding of the data set's structure before they obtain the data set itself.

Fixed

3 Compact Data Message Exchange of large data sets in a data structure definition-dependent form

Derived from data structure definition message

4 Utility Data Message For schema-based functions, such as validation, in a data structure definition-dependent form

Derived from data structure definition message

5 Cross-sectional Data Message

Exchange of many observation types in a data structure definition-dependent form

Derived from data structure definition message

6 Query message To query a database to obtain an SDMX-ML message as the result Fixed

SDMX-ML: Six standard messages

Metadata

Structural metadata must be associated with the data, otherwise it becomes impossible to identify, retrieve and browse the data.

The simplest explanation of a data structure definition is that it is a set of descriptor concepts, associated with a set of data, which allow us to understand what that data means.

Data Structure Definition

SDMX-ML “Model-Driven” XML Approach

SDMX Converter Main Activities

Reading the input message parsing of the message populating the data model of the tool (based on the SDMX v2.0

information model ) Writing the converted message

uses the data model to write the output message in the target format.

Supported formats SDMX-ML (Generic, Compact, Utility Compact) Gesmes (TS, 2.1. DSIS) Flat files (CSV, FLR)

DSD retrieved from the Registry that is mandatory for completing a conversion

http://bms.istat.it/sodidownload/download.aspxhttp://bms.istat.it/sodidownload/download.aspx

ISTAT SDMX Framework

SDMX Framework It is based on the SDMX Information Model It covers all the dissemination business process It can be used entirely

or Only some building bocks can be used, in order to integrate them in an already existing information system

Available APIs SDMX Query Parser

It allows to set up an SDMX architecture in a short time It can be used for didactic purpose It will be available as Open Source

Technical requirements windows applications:

MS Windows XP or sup. Microsoft .Net framework 2.0

web applications MS Windows 2003 or sup. Internet Information Server 6.0 or sup. Microsoft .Net framework 2.0 Microsoft ASP.NET 2.0

Database Oracle 9 or sup MS SQL Server 2005 or sup. MS Access

NSI

1

2

3

Data Sharing Model

Query SDMX

DataSDMX-

ML

RSS

WSNSI

Data Sharing Model

Link to SDMX

file

DataSDMX-

ML

RSS

WSNSI

Data Sharing Model

Query Message

This is another special-purpose message. It is used to convey a query to a database which then returns an SDMX-ML message

For web services and database-driven applications

Data Structure Wizard

http://sdmx.org/?page_id=13http://sdmx.org/?page_id=13

SDMX Registry A repository for keeping

Structural metadata (e.g. CodeLists, ConceptSchemes, DSDs, MSDs)

Provisioning information (e.g. Dataflows, Provision Agreements)

Repository is accessible via a Web Service accepting SDMX-ML messages

GUI for user interaction over the Web Authentication using CIRCA accounts DSW – “standalone” Java GUI

https://webgate.ec.europa.eu/sdmxregistry/

SDMX Registry

Thank You for Your Attention

vincenzo.patruno@istat.it

....and Good SDMX-ing