Implementation of quality indicators in the Finnish statistics production process Kari Djerf...

Implementation of quality indicators in the Finnish statistics production process

Kari DjerfStatistics FinlandQ2008, Rome Italy

2

Contents

1. Current situation

2. Challenges

3. Steps to proceed and time schedule

3

1. Current situation

Data exist - but:

lots of data are collected for various reporting purposes which does not necessarily serve this purpose, and

(too) much of data stored in inconsistent form

3

4

Current situation – Strategic and/or performance indicators

Collected centrally for all follow-up operations(now 2/4 times a year)

Partly at the agency level, partly at department level

Many of these indicators can be retrieved from a general planning and performance database

5

Current situation – Strategic and/or performance indicators - 2

Examples of indicators:Public confidence (every 2nd year)Delays in publications and releasesNonresponse rates of some key sample surveysShare of electronic data collection of all data collectionResponse burden

→ Not very suitable for continuous follow-up of quality of individual statistics!

6

Quality indicators to be collected and compiled

Product quality indicators

Process indicators

Very often these two are inseparable – a matter of opinion to which category an indicator actually belongs

Goal: accumulate information, do not add unnecessary burden on subject-matter departments!

7

Quality indicators to be collected and compiled - 2

Structure to follow the ESS quality dimensions:

RelevanceAccuracyTimeliness and punctualityAccessibility and clarityComparability Coherence

8

Quality indicators to be collected and compiled - 3

Obviously focus on accuracy and timeliness/punctuality but probably all dimensions will be covered

Most ESS standard quality indicators are suitable as such, some may not be directly applicable

9

Traditional indicators from sample surveys

Unit response/nonresponse rateshousehold surveys: long time-seriesbusiness surveys: incomplete data

Unit nonresponse rates divided into some key domains or classifications:

Reason for nonresponseDemographics (gender, age, region, education,

industry, size…)

10

Traditional indicators from sample surveys - 2

Item response/nonresponse ratesnot sufficiently calculated in household and business

surveys

Evaluation of both types of nonresponse effects on survey results of key parameters:

some results exist both on household and busness surveys

11

Traditional indicators from sample surveys - 3

Reliability estimates (standard errors, CV’s or CI’s)have been reported from most household surveyslesser extent in business surveys

(cut-off samples problematic) Survey specific indicators most probably to be included Editing and imputation indicators

currently under development: indicators to be retrieved from the validation and editing process

e.g. edit failure rates, imputation rates and their effect(esp. important in business statistics)

12

Traditional indicators from sample surveys - 4

Response burden of household surveys measured since 1970s as interview time + occasional evaluation of self-completeted questionnaires or diaries

Measurement of response burden of business and institutional surveys is currently under development

Cost model to be developed

13

Traditional indicators from censuses and administrative data

Coverage rates – to be evaluated with respect to critical contents!

Measurement errors, esp. correspondence between administrative and statistical concepts – important but normally they stay stable unless changes occur

Editing (and imputation) rates

13

14

2. Technical challenges

Periodicity

Requirements by various stakeholders

Metadata standard(s)

Various data sources

14

15

Technical challenges - periodicity

Example: Labour Force Survey

In Finland a monthly survey since 1959: many indicators in comparable since 1984

Current EU-LFS regulations: quarterly with annual combination of data

→ Indicators must be calculated monthly, quarterly and annually – some can be aggregated, mostly not

15

16

Technical challenges - stakeholders

EU regulations, IMF, OECD etc. different in definitions and requirements

EU regulations differ VERY much from each other on the extent of quality reporting and derivation of the indicators(EU-SILC, LFS, PEEIs etc.)

→ New Statistical law may improve the situation in general, but some very subject-dependent indicators might be left aside

16

17

Technical challenges – different types of statistics

Sample surveys Censuses and other total enumeration Administrative sources and registers Indices National accounts

→ Technical solutions must be flexible to allow different types of indicators

17

18

Technical challenges - metadata

SDMX standard to take over

New ESMS is to include some indicators which may or may not be similar between different statistical domains

→ Technical allowance to retrieve directly as many of the required indicators as possible

18

19

Technical challenges – existing data sources

Obviously the biggest challenge!

Subject-matter statistics do not compile and store data in a similar manner: many data warehouse systems were developed for one purpose only. New harmonised statistics production model will improve it gradually.

Next proper database tools must be found to store data and facilitate easy reporting

19

20

3. Steps to proceed - Cross-sectional data collection

A self-assessment of all statistics in next autumn:Quality reportsAvailable indicatorsAvailable metadata

Obviously it will resemble the DESAP approach in contents

Analysis of indicators to include important ones and exclude redundancies

20

21

Cross-sectional data collection - 2

Find a ”good cocktail” of indicators and start retrieving them

Database construction 2008/2009

Programs for reporting

… and system to work in 2-3 years!

22

THANK YOU VERY MUCH

FOR YOUR ATTENTION !