Embed Size (px)
Citation preview
2014-10-10 SDMX Technical Working Group 1
SDMX Technical Working Group
Activity Report
Sami Airo
7th Meeting of the Expert Group on SDMX
27-30 October 2014, Seoul, Korea
Background & composition
• Provides the forum for the active participation of the SDMX community in the management and development of the SDMX technical standards
• Started in May 2011, with the SDMX Global Conference in Washington
• Currently 24 members
–Central banks, statistical offices & international organisations
–16 countries and 7 international organisations
–4 regions
=> Representative of the SDMX technical community
2
Tasks
• Improves the SDMX Technical Standards
• Owns the SDMX technical standards issues list
• Reviews the issues list and sets priorities
• Prepares proposals for revised versions of the standard
• Oversees a set of SDMX reference tools
• Cooperates with the SDMX Statistical Working group
3
SDMX Technical Standards
4
Humans Computers
1. Framework Document
2. Information Model Specification (UML)
3. Exchange Formats
3.1 SDMX-ML Specification XML Schema
3.2 SDMX-EDI Specification
SDMX-JSON Specification JSON Schema
4. Services (=APIs)
4.1 Registry Services Specification
4.2 Web Services Guidelines WADL + WSDL
5. Technical Notes
Validation & Transformation Language Specification BNF Grammar
Active Work Packages
• WP2 - Interoperability of IT tools
• WP4 - Improved technical documentation
• WP5 - Interaction of SDMX with other technical standards (i.e. DDI)
• WP6 - Expressions and calculations (VTL)
• WP7 - SDMX-JSON
• WP8 - SDMX Technical Issues List
5
WP2: Interoperability of IT Tools
6
SDMX Software Components
The main aim is to foster the interoperability of the SDMX software tools. This allows adopters of the SDMX standards to combine various tools developed by different organisations and build the system they need.
WP2: Interoperability of IT Tools
• Report to be published on www.sdmx.org,
• Set of normative interfaces to be fulfilled by SDMX-compliant tools (“SDMX Common APIs”) would help in the development of custom implementations.
–Current “de-facto” SDMX common API is implemented in the SdmxSource framework (Java and .NET versions).
–Creating and maintaining APIs as technical standards would not be practical as programming platforms are substantially different, and the software development work is substantial.
–Open-source implementations of technical standards will continue to be important for the SDMX initiative and their further development needs stable funding.
• Areas for further work
– Investigate automated tools for testing web service compatibility with SDMX 2.1 Web Services specifications (“compatibility tool kit”).
–Maintain mapping of the component list to existing software and “state of art” implementations (in the report appendix) when new software applications and new versions of existing applications are released.
7
WP4: Improved technical documentation
• Often pointed out that the SDMX technical documentation is complex
• Need to suggest possible improvements and enhancements
• Focused on SDMX 2.1
• Close co-operation with the SWG
• Some deliverables
–Updated user guide
– Implementation model
–Technical vocabulary
8
WP5: Interaction of SDMX with other technical
standards
• Contribute to the discussion of the possible interaction of SDMX and other standards (such as DDI, GSIM, etc.)
–Assess current capabilities (SDMX vs. other standards)
–Suggest how SDMX could evolve
–Discuss the possible integration with other standards
• Final report in preparation, some preliminary conclusions
–SDMX standard is broadly able to represent both unit and dimensional data in a homogeneous way, by using the DSD.
–Representing unit and dimensional data through the same basic artefacts may give significant benefits.
–Some areas for improvements: representation of multiple measures, relationships and integrity constraints, time validity of data and metadata
9
WP6: Validation and Transformation
Language (VTL)
• Allows definition and exchange of data validation and transformation rules
• Based on the existing SDMX Information Model framework and the GSIM Data Model
• Usable by statisticians
• Supports dimensional and unit-record data
• Two main parts: general description and library of operations
• Draft version published on the www.sdmx.org, public review period until 8 November
10
VTL - Example
11
D1 := get( “POP/2011-2013.US”, keep(REF_DATE, BIRTHS, DEATHS) )
D2 := get( “POP/2011-2013.EU”, keep(REF_DATE, BIRTHS, DEATHS) )
DR := D1 + D2
DS := put( DR, “POP/2011-2013.ALL” )
This simple VTL program retrieves two data
sets, sums the data sets and stores the result.
VTL - Data Retrieval Example
12
Result Data Sets
D1 := get( “POP/2011-2013.US”, keep(REF_DATE, BIRTHS, DEATHS) )
D2 := get( “POP/2011-2013.EU”, keep(REF_DATE, BIRTHS, DEATHS) )
VTL - Simple Transformation Example
13
Source Data Sets Result Data Set
DR := D1 + D2
VTL - Data Storage Example
14
Input and Result Data Sets
DS := put( DR, “POP/2011-2013.ALL” )
VTL - Transformation Examples
15
DR := D1 + D2
DR := log( D1 - D2 )
DR := ln( abs(D1) + D2 ) - abs(D1)
DR := union( D1, D2 )
DR := diff( D1, D2)
DR := D1 <> D2
DR := D1 between 1000 and 2000
DR := isnull(D1)
DR := check( D1 > D2, imbalance( abs(D1 - D2) , erlevel(8) )
DR := if D1 > 4 then D1 else D2 + 3
DR := D1[ filter REF_DATE = 2013 ]
DR := D1[keep(BIRTHS)][aggregate sum(BIRTHS)]
WP7: SDMX-JSON Exchange Format
• New exchange format for data dissemination on the Web
• Conforms to the JSON standard specification, and supports the SDMX 2.1 Information Model
• Supports all features of the SDMX RESTful API for data queries
• Allows web developers to write software that requests data responses in JSON from SDMX RESTful API and processes data and metadata in the JSON response
16
WP8: Issue List
• Issue list is for bug fixes and minor improvements
• Examples
– “StructureMap defines wrong multiplicity for ComponentMap”
– “Ability to reference a Code List Map not contained in the same Structure Set”
• Current status
–Bug reports: 1 open, 5 resolved
– Improvements: 7 open, 2 resolved
–New features: 2 resolved
• Minor updates to 2.0 and 2.1 Technical Standards
17
WP8: Updated Processes and New Tools for
the Standards Maintenance
• Updated Technical Standards revision process
–Types of changes: bug fix, improvement, new feature
–Semantic version numbering: xx.yy.zz
–Process steps: working draft, candidate, proposed, final
• Testing new tools (Github) for the maintenance of standards documents and related source code files
–Github provides efficient source code management and collaborative features such as bug tracking
–Should also improve transparency and openness of the standards maintenance process
–Example: http://github.com/sdmx-twg/
18
