European approaches -

- for ensuring high quality of European statistics

EMOS Open Day Luxembourg – 25-26.01.2018

Zsuzsanna Kovács

Eurostat

Eurostat

Overview

What is quality and quality in statistics?

Quality framework (fw) of the ESS, its relation with the legal fw

The ES Code of Practice and Quality Assurance Framework of the ESS – 3 levels of quality assurance

The Quality Declaration of the ESS

The Quality function of the ESS

4th level, centralised quality assurance

Eurostat's Quality Reviews' exercise

Quality reporting in the ESS

Quality-conference

From quality assurance to a holistic view of quality

2 Zsuzsanna Kovács

Eurostat

What is quality?

A relative, multi-dimensional concept

Different definitions and approaches exist

The most general and succinct definition: "fitness for purpose/use"

Definitions of international standards:

• ISO 9000: "Degree to which a set of inherent characteristics fulfils requirements"

• ISO 8402:1986: "Totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs"

3 Zsuzsanna Kovács

Eurostat

What is quality in statistics?

Encompasses all aspects of how well statistical processes and outputs fulfil expectations of all of their users and stakeholders =>

Good quality of statistics is:

• Not just meeting user needs but also

• Addressing respondent concerns regarding reporting burden and confidentiality

• Ensuring institutional environment is independent, impartial, objective, comprising sound methodology and cost-effective procedures

4 Zsuzsanna Kovács

Eurostat

Quality framework of the ESS

The 'common' quality framework of the ESS* is composed of (cf. in the Preamble of the ES Code of Practice)

• The European Statistics Code of Practice

• The Quality Assurance Framework of the ESS, as the 'extension' / complement of the ES Code of Practice

• The general quality management principles – quality for the institutions as a whole, QUAL Action 3

The quality framework of the ESS complements the legal framework and they both constitute the robust framework in which high-quality European statistics are developed, produced & disseminated

5 Zsuzsanna Kovács

Legal and quality fw of the ESS – Relation, inter-action, roles

Legal framework – Quality aspects: 7 Principles anchored in 338/TreatyFEU (Primary law) – Those Principles repeated and defined in amended Regulation 223/2009, together with the introduction of the CoP (Art 11) & definition of the output quality criteria and quality reporting (Art 12) – Quality provisions of the sector-specific legislation (Secondary law)

Common Quality Framework: Complements the legal framework: 'dichotomy' (difference in nature)

Zsuzsanna Kovács 6

Eurostat

Legal and quality fw of the ESS – Relation, inter-action, roles

Their 'meeting points': the 7 Principles & the inclusion of CoP, quality reporting and output quality criteria in the Statistical law

Their differences:

• Coverage – the CoP extends the list of Principles and elevates the status of the output quality 'criteria' to 'Principles'

• Approach (stemming from the nature) – in addition to the law's definition, the CoP provides practical guidance, best practices for the implementation of the Principles

Importance of a '1:1 correspondance / matching'? No – they serve different purposes & have their own rationale

Question of the language, wording used – Clarity, exactitude

Zsuzsanna Kovács 7

Eurostat

Eurostat

ES Code of Practice – aim

Ensures public trust and confidence in the independence, integrity and accountability of the statistical authorities as well as in the credibility and quality of the statistics they develop, produce and disseminate

Sets the standards for developing, producing and publishing European statistics

Promotes the application of best international statistical practices, principles and methods by all producers of European statistics

8 Zsuzsanna Kovács

ES Code of Practice – the ESS concept of quality in statistics

• Institutional environment

• Professional independence • Coordination and cooperation • Mandate for data collection • Adequacy of resources • Commitment to quality • Statistical confidentiality • Impartiality and objectivity

• Statistical processes

• Sound methodology • Appropriate statistical

procedures • Non-excessive burden on

respondents • Cost effectiveness

• Statistical output

• Relevance • Accuracy and reliability • Timeliness and punctuality • Coherence and comparability • Accessibility and clarity 9

Eurostat

Zsuzsanna Kovács

Eurostat

ES Code of Practice – mechanism

Self-regulatory

Applies to Eurostat, National Statistical Institutes of the ESS and other national authorities producing European statistics

For each Principle, there are Indicators showing how compliance can be demonstrated

10 Zsuzsanna Kovács

Eurostat

ES Code of Practice – example

Principle 8

Appropriate statistical procedures, implemented from data collection to data validation, underpin quality statistics

Indicator 8.6

Revisions follow standard, well-established and transparent procedures

11 Zsuzsanna Kovács

Eurostat

ES Code of Practice – monitoring

National and Eurostat self-assessments in 2005 http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/code_of_practice/ess_system_compliance

ESS peer reviews in 2006-2008 (1st round) and in 2013-2015 (2nd round) http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/code_of_practice/peer_reviews http://ec.europa.eu/eurostat/web/quality/peer-reviews

Reports and improvement actions per country and Eurostat; Final report to the Parliament and Council

Annual monitoring and reporting of improvement actions (Eurostat & ESGAB); implementation deadline end-2019

12 Zsuzsanna Kovács

Eurostat

Quality Assurance Framework (QAF) of the ESS – aim

3rd level of quality assurance

Applicable across all statistical domains

But it is not part of the Code

Provides methods and tools at institutional and process level

Contains also links to relevant reference documentation

Can also be considered as guidance to compliance assessors

13 Zsuzsanna Kovács

Eurostat

Quality Assurance Framework of the ESS – example

Principle 8

Appropriate statistical procedures, implemented from data collection to data validation, underpin quality statistics

Indicator 8.6

Revisions follow standard, well-established and transparent procedures

Methods of implementation

Guidelines on revision of published statistics exist, are applied and made known to users

Revisions accompanied by explanations made available to users

Quality indicators on revisions are calculated and published

14 Zsuzsanna Kovács

The common quality framework of the ESS

Eurostat

Zsuzsanna Kovács 15

The Quality Declaration of the ESS

The 'crowning' of many years' work –

To demonstrate the competitive advantage of European statistics

What makes them different from other data

Zsuzsanna Kovács

Eurostat

16

Why in the form of a 'Declaration'

Because

It is a self-commitment, a continuous self-engagement of all members of the European Statistical System (ESS)

It builds on all the efforts that the ESS deployed in order to guarantee the development, production and dissemination of high-quality European statistics and services (amended Regulation 223, second round of peer reviews,…)

Zsuzsanna Kovács

Eurostat

17

Its main message: awareness… … and official guarantee: 'ESS made'

A very high level of quality awareness exists in the ESS

All members of the ESS partnership do the same in terms of quality &

The standards are set very high, as defined in the world-class quality framework: European Statistics Code of Practice =>

Therefore, the product: European Statistics equals high quality that can be trusted, is decided upon in a democratic manner & provides high value for money for the societies

Zsuzsanna Kovács

Eurostat

18

The Quality function of the ESS

Governance

Zsuzsanna Kovács 19

Eurostat

Eurostat

4th level, centralised Q assurance – methods & tools, contin. improvement

Labelling

User

satisfaction

surveys

Production

processes

Statistical

products

User

perception

User

requirementsStandards

1- Documentation and measurement

2- Evaluation

3- Conformity

Process

descriptions

Quality reports

and indicators

Quality

assessments

Quality

improvements

20 Zsuzsanna Kovács

Eurostat's Quality reviews – 4th level quality assurance

2nd round of screening / all statistical production processes

In the light of the standards set by the CQF

Planned roll-out: 4-5 years

Main objective: continuous process improvement, independent and objective re-assurance

Basic tool: shortened and pre-filled questionnaire

4 sub-processes:

1. Risk assessment and planning

2. Quality reviews at process level

3. Annual follow-up of improvement actions and reporting

4. Annual process improvement, presentations, training courses

Zsuzsanna Kovács 21

Eurostat

Eurostat's Quality reviews – workflow of sub-process n° 2

Zsuzsanna Kovács 22

Eurostat

Quality reporting in the ESS

• Data need metadata

Member States have to provide Eurostat with reports on the quality of the data transmitted &

Eurostat has to assess the quality of the data and has to prepare and publish reports on the quality of European statistics

• Quality reports aim at describing and quantifying the quality of the statistics produced, based on defined quality criteria

• In Nov 2015 the ESSC approved the Single Integrated Metadata Structure (SIMS) v2.0 as the ESS standard for quality reporting:

• Euro SDMX Metadata Structure (ESMS): user-oriented

• ESS Standard for Quality Reports Structure (ESQRS): producer-oriented

23

Eurostat

Zsuzsanna Kovács

Eurostat

ESS Handbook for Quality Reports

Describes the structure and content of a standard, detailed ESS quality report

Contains practical examples as well

Explaines the standard ESS quality indicators and their use

Is now being revised, to include the latest developments

http://ec.europa.eu/eurostat/documents/3859598/6651706/KS-GQ-15-003-EN-N.pdf

24 Zsuzsanna Kovács

Eurostat

SIMS, version 2.0

25 Zsuzsanna Kovács

Quality conference

'European conference on Quality in official statistics'

Biennial event, organised by Eurostat and a NSI of the ESS

Exchange of methods and good practices between statistical offices, international organisations, researchers and academics

Last conference: www.q2016.es (Madrid) 492 participants from 119 institutions and 60 countries

Next conference: 26-29 June 2018, Krakow (Poland)

26

Eurostat

Zsuzsanna Kovács

Eurostat

Quality for the institutions as a whole – difference between managing and assuring quality…

Quality management is the set of systems and frameworks which are in place within an organisation to manage the quality of products and processes – a broad range of activities

Quality assurance is the guarantee of an organisation that the product or service it offers meets the accepted quality standards; according to the ISO, it is a part of quality management – a more technical part, about the core business

27 Zsuzsanna Kovács

Eurostat

…in their respective frameworks

Quality management is implemented via the Quality Management Systems: a set of interrelated or interacting elements that organisations use to direct and control how quality policies are implemented and quality objectives achieved

Quality assurance is implemented via the Quality Assurance Frameworks (QAF): an umbrella for quality work at a statistical office, a place for recording quality concepts, policies and practices and a basis for creating and promoting quality culture

28 Zsuzsanna Kovács

Conclusions

Quality work set up in a robust and coherent system in Eurostat and the ESS

Importance of continuous development and remaining relevant, up-to-date in a fast evolving context

Importance of managing quality within the institution's overall management framework – with a clear view and understanding of the links and inter-relations, scope and nature of the various elements

Zsuzsanna Kovács 29

Eurostat

Thank you for your attention!

Do you have questions or comments?

Visit us:

http://ec.europa.eu/eurostat/web/quality/overview

Eurostat