1 National Quality Assurance Frameworks Mary Jane Holupka and Iliana Vaca Trigo Learning Centre on National Quality Assurance Frameworks 43 rd Session

1

NationalNational QualityQuality Assurance FrameworksAssurance Frameworks

Mary Jane Holupka and Iliana Vaca TrigoLearning Centre on National Quality Assurance Frameworks

43rd Session of the UN Statistical Commission 27 February 2012

2

National Quality Assurance Frameworks (NQAF)National Quality Assurance Frameworks (NQAF)- Outline -- Outline -

I. Background: NQAF and Expert Group

II. The generic NQAF Template

III. NQAF Guidelines (section 3 examples)

IV. NQAF Website and Glossary

V. Future challenges

3

I. BACKGROUND – 2009-2010I. BACKGROUND – 2009-2010

•UN Statistical Commission: Quality on the agenda for the 1st time

•Programme review report for the 2010 Commission prepared by Statistics Canada; 1st draft underwent a global consultation

•Final Statistics Canada report presented to the 2010 Statistical Commission

4

• Statistical Commission decision, based on Canada’s reports’ recommendations, supported the:

- development of a generic NQAF template** with guidelines. The template should: focus on national statistical systems; use existing frameworks; and be flexible so as to take national circumstances into consideration.(**generic template rather than a framework per se in recognition that a one-size-fits-all framework was not feasible)

- establishment of an expert group to take on the work (template, guidelines, common glossary, mapping, website with links to quality assurance tools and guidelines, and guidance and training)

• Expert Group work began Sept. 2010; 17 countries invited to be members; Eurostat, IMF, WB, regional commissions (chair: South Africa)

I. BACKGROUND - 2010I. BACKGROUND - 2010

5

• All work (template, guidelines, mapping, glossary, references & website) carried out entirely via e-mail exchanges 1st year

• Sept. 2011 - EGM in NY to discuss pending issues

• Nov. 2011 - global consultation; feedback & suggestions from NSOs; taken into account in the final StatComm report & guideline document (guideline doc submitted as a background doc)

I. BACKGROUNDI. BACKGROUND

6

The framework template is: (page 4 of the Guidelines document)

•Arranged in five sections

(1) quality context

(2) quality concepts and frameworks

(3) quality assurance guidelines

(4) quality assessment and reporting

(5) quality & other management frameworks•Has been based on the 3 proposals Statistics Canada had made in its report to the 2010 StatCommission

- The resulting NQAF template is a combination with some additions

“Fitness for use” / “Fitness for purpose”

Quality is all about providing goods & services that meet the needs of users

II. The generic national quality assurance II. The generic national quality assurance framework (NQAF) framework (NQAF) templatetemplate

Main focus of the work of the Expert Group

7


8

II. The generic NQAF II. The generic NQAF template: template: The NQAF template’s section 3 has been based on Statistics Canada’s 2010 The NQAF template’s section 3 has been based on Statistics Canada’s 2010

Statistical Commission report’s 3 proposals (cols 1-3) Statistical Commission report’s 3 proposals (cols 1-3)

9

•So as not to “re-invent the wheel”, the EG drew heavily upon the work of Eurostat, IMF and StatCan

The template is aligned with (mapped to) the other well-

known quality frameworks – European Statistics Code of Practice (CoP)– International Monetary Fund’s Data Quality Assessment

Framework (DQAF)– Statistics Canada's quality assurance framework

– (and the newer) Proposal for the Structure of a Regional Code of Good Statistical Practice for Latin America and the Caribbean

II. The generic NQAF II. The generic NQAF template template - and - and mappingmapping to existing frameworks to existing frameworks

10

II. The generic NQAF II. The generic NQAF templatetemplate - and - and mappingmapping to existing frameworks to existing frameworks

http://unstats.un.org/unsd/dnss/docs-nqaf/MAPPING%20OF%20THE%20NQAF%20.xls

11

•Voluntary, not mandatory•Flexible to permit national circumstances to be taken into consideration; application of ALL components of the template not necessarily expected; not prescriptive

•A starting point on which to build/modify as necessary

•A useful organizing framework• A framework created by the national agency for the national agency


12

•Guidelines – a kind of checklist, focusing on section 3, the numbered “NQAF lines” (other sections intentionally briefer, more general)

•For each numbered NQAF line (1-19):– Description (what? “… agencies should minimize delays in making data available…” )– Elements to be assured (which?)

(roughly ordered by levels or stages) – Supporting mechanisms (how?)– Selected references (where?)

III. NQAF template III. NQAF template GUIDELINESGUIDELINES

Some repetition across different NQAFs - underscores the multi-dimensional aspect of quality and allows users to use parts of the framework independently.(e.g.use of sample surveys instead of censuses, when possible & appropriate – in cost-effectiveness (11) and managing respondent burden (13))

Are the data

fit for the purpose they are intended to be used?

•Helpful to:– Data providers in designing a statistical collection

or product or reviewing existing ones– Data users in making informed decisions about

the statistics produced

13

Quality assurance guidelinesQuality assurance guidelines (click on hyperlinks for details and examples for selected NQAFs, i.e. 3,7,8,11,13,16)(click on hyperlinks for details and examples for selected NQAFs, i.e. 3,7,8,11,13,16)

3a. Managing the statistical system[NQAF 1] Coordinating the national statistical system [NQAF 2] Managing relationships with data users and data providers[NQAF 3] Managing statistical standards3b. Managing the institutional environment [NQAF 4] Assuring professional independence [NQAF 5] Assuring impartiality and objectivity [NQAF 6] Assuring transparency[NQAF 7] Assuring statistical confidentiality and security[NQAF 8] Assuring the quality commitment [NQAF 9] Assuring adequacy of resources3c. Managing statistical processes[NQAF 10] Assuring methodological soundness[NQAF 11] Assuring cost-effectiveness [NQAF 12] Assuring soundness of implementation[NQAF 13] Managing the respondent burden 3d. Managing statistical outputs [NQAF14] Assuring relevance[NQAF15] Assuring accuracy and reliability [NQAF16] Assuring timeliness and punctuality [NQAF17] Assuring accessibility and clarity[NQAF18] Assuring coherence and comparability[NQAF19] Managing metadata

III. NQAF template III. NQAF template GUIDELINESGUIDELINES

14

Description: Description:

NQAF 3: Managing statistical standards NQAF 3: Managing statistical standards

Standards:•a comprehensive set of statistical concepts and definitions used to achieve uniform treatment of statistical issues - within a survey or across surveys, and across time and space•assist in maximising the effectiveness of statistical outputs and the efficiency of the production process in terms of inter-temporal, national and international comparability and coherence (i.e. the capacity for integration) of the statistics.

While comparability and coherence are important for any dataset, they are particularly important where data are obtained from multiple sources and have to be combined or where outputs are used in a wide variety of contexts. The use of standard collection units (families, households, businesses, etc.) helps the compilation, comparison and dissemination of statistics for these standardised units.

Statistical agencies should aim to use consistent names and definitions for populations, statistical units, concepts, variables, and classifications in their statistical programmes/domains.

15


Agency works towards developing statistical standards

Collaboration: with other agencies in developing, reviewing, promoting & implementing statistical standards

Person(s) responsible: for leading development of standards & supporting programmes to develop new or updated ones; staff assigned responsibility has the appropriate level of authority

Monitoring: extent that statistical standards are used by the programmes; senior management is given periodic reports

Informing staff and users: about statistical standards & changes made to them

Collaboration: data users & data providers & the agency’s own statistical programmes are involved in creating, developing & approving statistical standards

Inclusion of (in standards): a statement of conformity to corresponding international or national standards

Divergences: from the corresponding international or national statistical standards are documented & explained

Detailed concordances to corresponding int’l & national standards

Detailed concordances to previous standards Agency levelProgramme

designImplementation

programmePost-collection

evaluation

Elements to be assured: Elements to be assured: Mechanisms/examples: Mechanisms/examples:

Central organizational units or senior level groups responsible to lead & coordinate the development, implementation, maintenance & use of statistical standards

Active participation with other national & international organizations in the development, review, promotion & implementation of statistical standards (e.g. employees attend workshops, conferences &seminars at the national and international levels) (hyperlink to examples: UNSD and Mexico )

Notice of introduction of a new aggregation structure for the classification of imports and exports of goods (hyperlink to example: Canada)

Active participation of both data users & providers in the development & approval of statistical standards

Correspondence tables for classifications exist and are kept up-to-date & made available to the public with explanatory information (hyperlink to example: UNSD)

16


Conceptual frameworks used: e.g. SNA, that provide a basis for consolidating statistical information about certain sectors or geographical entities (hyperlink to example: UNSD)

Integrated statistics programmes are developed that require statistical standards

Compliance with required application: programmes are held accountable to apply the standards

Non-compliance with required application: programmes have to obtain exemptions from standards if they do not apply them

Informing statistical programmes/domains: plans (and deadlines) for the development & application of new statistical standards are communicated well in advance (even several years)

Level of information: to provide maximum flexibility in aggregation & to facilitate retrospective reclassification,statistical programmes collect & retain information at the fundamental or most detailed level of each standard classification, to the extent possible

Informing users and the public: all potential data users & the public

Review and revision: standards are regularly reviewed & revised, if necessary, to ensure their quality, notably their relevance, coherence & clarity

Agency levelProgramme

designImplementation

programmePost-collection

evaluation


Statistical programmes based on conceptual frameworks or data integration frameworks

that rely heavily on statistical standards (hyperlink to example: Australia)

Upcoming Reviews (hyperlink to example: Australia)

Documentation: on the statistical standards used is included in statistical products or explicitly referred to and is readily accessible (hyperlink to example: Canada)

17

Examples: Examples: Statistics Canada Statistics Canada (http://www.statcan.gc.ca/concepts/consult-napcs-scpan-eng.htm)

NQAF 3: Managing statistical NQAF 3: Managing statistical standardsstandards

http://www.statcan.gc.ca/concepts/consult-napcs-scpan-eng.htm

18

ExamplesExamples:: Global Inventory of Statistical Standards - - Global Inventory of Statistical Standards - - UNDER DEVELOPMENTUNDER DEVELOPMENT


19

Examples: Examples: INEGI’s Inventory on International Statistical StandardsINEGI’s Inventory on International Statistical Standards(http://mapserver.inegi.org.mx/estandares/Index1.cfm)

NQAF 3: Managing statistical standardsNQAF 3: Managing statistical standards

http://mapserver.inegi.org.mx/estandares/Index1.cfm

20

Examples: Examples:


21

Examples: Examples: Statistics Canada Policy on standardsStatistics Canada Policy on standards(http://www.statcan.gc.ca/about-apercu/policy-politique/standards-normes-eng.htm)


http://www.statcan.gc.ca/about-apercu/policy-politique/standards-normes-eng.htm

22


Examples: Examples: System of National Accounts 2008 - 2008 SNASystem of National Accounts 2008 - 2008 SNA

(http://unstats.un.org/unsd/nationalaccount/sna2008.asp)

http://unstats.un.org/unsd/nationalaccount/sna2008.asp

23

NQAF 3: Managing statistical standardsNQAF 3: Managing statistical standardsExamples: Examples: Australian Bureau of Statistics Upcoming ReviewsAustralian Bureau of Statistics Upcoming Reviews(http://www.abs.gov.au/websitedbs/D3310114.nsf/home/Upcoming+Reviews)

24


Examples: Examples: ABS National Statistical Service (NSS) HandbookABS National Statistical Service (NSS) Handbook(http://www.nss.gov.au/nss/home.nsf/NSS/35BFD39E0E2A8597CA25763F000B622C?opendocument#10.2.1)

http://www.nss.gov.au/nss/home.nsf/NSS/35BFD39E0E2A8597CA25763F000B622C?opendocument#10.2.1

25


NQAF 7: Assuring statistical NQAF 7: Assuring statistical confidentiality and security confidentiality and security

Statistical agencies should guarantee that the privacy of data providers (persons, households, enterprises, administrations and other respondents) will be protected and that the information they provide will be kept confidential, will not be able to be accessed by unauthorized internal or external users, and will be used for statistical purposes* only.

Statistics shall be considered confidential when they allow statistical units to be identified, either directly or indirectly, thereby disclosing individual information.

*Examples of purposes not strictly statistical are those for:

administrativelegaltax purposes.

26

NQAF 7: Assuring statistical NQAF 7: Assuring statistical confidentiality and securityconfidentiality and security


Legal arrangements are in place to protect confidentiality (Statistics Acts, Privacy laws) (hyperlink to example: Japan)

The staff signs legal confidentiality agreements or declarations covering their obligations - upon appointment or on a more regular basis (hyperlink to example: Canada)

Legally enforceable contracts regarding the use of microdata/public use files exist

Policies and procedures that elaborate the legal arrangements are made known to staff and users (hyperlink to example: Canada)

Anonymization procedures (hyperlink to example: International Household Survey Network)

Fines or imprisonment may be imposed on persons who wilfully violate the law by breaching statistical confidentiality

Laws: or some other formal statistical confidentiality provision or code of practice that guarantees the proper management, with regard to privacy and security, of information received from data providers.

[Where the statistics law provides for exceptions to the general confidentiality provisions, clear policies and procedure are in place and are made public to operationalize the exceptions] (hyperlinks to examples: Canada)

Dissemination policy: how statistics are to be disseminated to users and under what circumstances microdata (i.e. statistical information relating to individual respondents) may be made available for research and further analysis

Microdata: appropriate procedures and processes (e.g. anonymization) in place to ensure that individual respondents cannot be identified from the data

Penalties: for statistical staff or other personnel found guilty of activities leading to the release of confidential data


designProgramme

implementationPost-collection

evaluation

27



Guidelines and instructions for staff on the protection of statistical confidentiality in the production and dissemination processes (e.g. cell suppression, aggregation, etc.) (hyperlink to example: France)

Documentation: of procedures taken to eliminate or reduce the risk of identification. It is made available as part of the metadata related to the statistical dataset

Informing users: to make them aware that procedures to eliminate the risk of identification have been implemented and that this could lead to a loss of information

Procedures in place: appropriate physical, administrative and IT security to ensure the protection of data

Monitoring procedures: when microdata sets are used, to identify any circumstances in which data confidentiality may be breached, e.g. through file matching. Immediate action is taken to redress the situation


designProgramme


evaluation

28


Examples: Examples: Japan’s Statistical Act (Act No. Japan’s Statistical Act (Act No. 53 of May 23, 2007), Articles 41, 42 and 43, pp. 53 of May 23, 2007), Articles 41, 42 and 43, pp. 16 - 18 16 - 18 (http://www.stat.go.jp/english/index/seido/pdf/stlaw.pdf)

http://www.stat.go.jp/english/index/seido/pdf/stlaw.pdf

http://www.stat.go.jp/english/index/seido/pdf/stlaw.pdf

29


Examples: Examples: Statistics Canada Policy on informing survey respondents Statistics Canada Policy on informing survey respondents (http://www.statcan.gc.ca/about-apercu/policy-politique/info_survey-enquete-eng.htm)

http://www.statcan.gc.ca/about-apercu/policy-politique/info_survey-enquete-eng.htm

30


Examples: Examples: Canada’s Statistics Act Canada’s Statistics Act (http://www.statcan.gc.ca/about-apercu/act-loi-eng.htm)

http://www.statcan.gc.ca/about-apercu/act-loi-eng.htm

31


Examples: Examples: Canada’s Canada’s Statistics Act Statistics Act (http://www.statcan.gc.ca/about-apercu/act-loi-eng.htm)



32


Examples: Examples: France’s Guide to statistical confidentialityFrance’s Guide to statistical confidentiality(http://www.insee.fr/en/insee-statistique-publique/statistique-publique/guide.pdf)

33


Examples: Examples: International Household Survey Network’s Tools and Guidelines on Data AnonymizationInternational Household Survey Network’s Tools and Guidelines on Data Anonymization(http://www.internationalsurveynetwork.org/HOME/index.php?q=tools/anonymization)

34


NQAF 8: Assuring the quality NQAF 8: Assuring the quality commitmentcommitment

Statistical agencies should be dedicated to assuring quality in their work, and systematically and regularly identify strengths and weaknesses to continuously improve process and product quality.

Processes, staff and facilities should be in place for ensuring that the data produced are commensurate with their quality objectives.

35


Agency level

Programme design

Programme implementation

Post-collection evaluation


Quality policy, declaration or commitment statement Staff awareness “campaign” to emphasize

commitment (hyperlink to example: Lithuania)

Quality manager, committee, unit, coaches or advisers

Guidelines, methodological manuals, etc.; GSBPM - for guidance on managing & monitoring the quality of all stages of the statistical process (hyperlink to examples: Eurostat, GSBPM)

Use of TQM, ISO 9000, quality initiatives of the ESS, independent evaluations and/or IMF ROSC evaluations (hyperlink to IMF)

Programmes set up for carrying out quality reports, self-assessments, audits, user satisfaction surveys - to monitor & report on quality over time

Work plans, schedules & standard forms or templates - for facilitating the consistent updating of the documentation

Regularly held training courses designed to support quality policy

Commitment to quality: policy or message is made public

Person(s) responsible

Guidelines for implementing quality management w/in the statistical production process: describing a) entire process & documentation; b) methods for monitoring the quality of stages; c) identification of indicators (quality measures) for evaluating the quality of the main stages

Guidelines made available to external users (at least a summary)

Externally-recognized processes or activities that focus on quality are followed

Procedures put in place for monitoring and reporting on product quality; top management gets informed of the results in order to define improvement actions; quality reviews of key products regularly conducted

Documentation on quality is required & regularly updated

Training & development programmes: agency’s quality policy & how quality may be achieved

36


Agency level

Programme design

Programme implementation

Post-collection evaluation


IT staff, methodologists and other specialists (e.g. in questionnaire design) participate in assisting subject matter units; appropriate software is provided

Validation techniques are widely promoted and applied

Expert groups are established & meetings held regularly

Documentation on methods, concepts and definitions is made available for all major fields of statistics

User satisfaction surveys carried out; reports on the results are made publicly available

User-oriented quality reports are produced & made available to the public; producer-oriented reports are produced; can be used by agency to monitor quality over time

External experts may conduct quality reviews of key statistical domains (e.g. IMF’s ROSCs, peer reviews, external audits & rolling reviews)

Support: to subject matter units by specialized (methodological & IT) units to help implement improvements in data development, production & dissemination

Benchmarking: of key statistical processes with other agencies carried out to identify good practices

Quality assurance plan or similar mechanism for planning and monitoring the quality of different stages of the process; describes the working standards, formal obligations (laws & internal rules) & quality control actions to prevent/ monitor/evaluate errors & to control different points at each stage. Taken into account here are: • users’ needs and relevance of the statistical operation • examination of possible trade-offs among quality dimensions (hyperlink to above example: France)

• assurance of the quality of data collection (incl. use of admin. data) & data editing

Metadata & quality indicators or measures: prepared & provided to users to help them assess the quality of the released data (hyperlink to example: Eurostat ESMS)

Evaluation: quality reviews are conducted

Users’ reactions and feedback: collected; to be used as inputs to action plans

37


Examples: Examples: Statistics Lithuania Statistics Lithuania

Quality PolicyQuality Policy(http://unstats.un.org/unsd/dnss/docs-nqaf/Lithuania-Quality_policy.pdf

)

http://unstats.un.org/unsd/dnss/docs-nqaf/Lithuania-Quality_policy.pdf

http://unstats.un.org/unsd/dnss/docs-nqaf/Lithuania-Quality_policy.pdf

38

Examples: Examples: Generic Statistical Business Process Model (GSBPM)Generic Statistical Business Process Model (GSBPM)


Structure of the ModelStructure of the Model

Process

Phases

Sub-processes

(Descriptions)

39

Examples: Examples: Eurostat’s HandbooksEurostat’s Handbooks


40


Examples: Examples: IMF’sIMF’s Reports on the Observance of Reports on the Observance of

Standards and Codes (ROSCs) Standards and Codes (ROSCs) (http://www.imf.org/external/NP/rosc/rosc.aspx)

http://www.imf.org/external/NP/rosc/rosc.aspx

http://www.imf.org/external/NP/rosc/rosc.aspx

41


Examples: Examples: INSEE Quality plan INSEE Quality plan (http://www.insee.fr/en/insee-statistique-publique/default.asp?page=qualite/plans_action.htm)

http://www.insee.fr/en/insee-statistique-publique/default.asp?page=qualite/plans_action.htm



42


Examples: Examples: INSEE Quality plan INSEE Quality plan

43


Examples: Examples: Quality metadata following ESS standards (ESMS) Quality metadata following ESS standards (ESMS)(http://epp.eurostat.ec.europa.eu/portal/page/portal/statistics/metadata/metadata_structure )

http://epp.eurostat.ec.europa.eu/portal/page/portal/statistics/metadata/metadata_structure

44


NQAF 11: Assuring cost-effectiveness NQAF 11: Assuring cost-effectiveness

Statistical agencies should assure that resources are effectively used. They should be able to explain to what extent the set objectives were attained and that the results were achieved at a reasonable cost consistent with the principal purposes for which the statistics will be used.

45

NQAF 11: Assuring cost-effectivenessNQAF 11: Assuring cost-effectiveness

Agency level Programme design Programme implementation Post-collection evaluation


Standardization programmes and procedures are defined and implemented in key areas according to the business process model

Indicators of human and financial resources - monitored centrally and regularly reported to management

Accounting systems allow allocation of resources to statistical operations

Human resources (allocation, performance and staff training needs) are evaluated annually in line with office-wide guidelines

Staff opinion surveys - conducted regularly IT infrastructure - reviewed regularly Ex-ante cost calculation procedures are available for

statistical operations Appropriate arrangements (e.g. SLAs agreements or

national legislation) are signed with owners of administrative data collections and regularly updated

An assessment of possible administrative data sources is carried out prior to launching any new survey

Data linking and integration methods are proactively pursued subject to data security considerations

Quality indicators are developed and compiled to improve the use of administrative data for statistical purposes

Procedures are in place to measure and manage the respondent burden

Standardization: standardized solutions that increase effectiveness & efficiency are promoted and implemented

Monitoring: of the agency’s use of resources, both by internal and independent external measures (hyperlink to example: France)

Costs: of producing the statistics, at each stage of statistics production, are regularly reviewed and documented to assess their effectiveness

Cost–benefit analyses: carried out to determine the appropriate trade-offs in terms of data quality

Administrative data: proactive efforts made to improve the statistical potential of administrative data and to limit recourse to direct surveys

Administrative data: - instead of sample surveys – are used when it is appropriate and possible

Respondent burden: is managed

New data collection: when contemplated - review whether current data sources can be utilized instead with minimal impact on their current purpose and quality

46



designProgramme


evaluation

Elements to be assured:Elements to be assured: Mechanisms/examples: Mechanisms/examples:

Centralized IT and methodological units provide possibilities for the pooling of resources and investments and the identification of innovation/modernization potentialAn appropriate IT architecture and strategy exists and is regularly updatedPolicies, procedures and tools exist to promote automatic techniques for data capture, data coding and validationThe use of automated processing techniques is regularly reviewed

Procedures are in place to measure and manage the respondent burden

User satisfaction surveys

Review ongoing programmes: to consider whether a particular programme is still operating in the most cost-effective way to meet its stated requirements

IT: review whether its productivity potential being optimized for data collection, processing and dissemination

Reporting burden minimization: keeping in mind the principal purposes for which the statistics will be used

Automation: of routine clerical operations (e.g. data capture, coding, validation) where possible

Cost-effectiveness assessment: undertaken for every statistical survey

User feedback: to verify whether outputs produced continue to meet the needs of the key users so as to justify the collection of the data

47

Examples: Examples: INSEE is audited by the French INSEE is audited by the French Court of Auditors and the General Finance Court of Auditors and the General Finance Inspectorate: on this subject, see the "INSEE Inspectorate: on this subject, see the "INSEE International Comparative Analysis" ReportInternational Comparative Analysis" Reporthttp://www.insee.fr/en/insee-statistique-publique/qualite/

report_igf.pdf


48


NQAF 13: Managing the respondent NQAF 13: Managing the respondent burden burden

Individuals, households or businesses who provide data, upon which statistical products are based, are fundamental contributors to the quality of data and information. The requirement to collect information (needs) should be balanced against production costs and the burden placed on respondents (supplier costs). Mechanisms to maintain good relationships with individual providers of data and to proactively manage the response burden are essential for improving quality.

This difficult challenge is particularly topical with declining response rates in surveys. This decline lowers quality and increases the cost of surveys. Improving response rates requires a multi-dimensional strategy that addresses the issue of non-response at different stages of the survey process. This includes an assessment of the need to collect the information, the use of data from administrative sources or other surveys, and the use of sound statistical and survey methods to keep the burden to a minimum.

49


Respondent burden/respondent relations programme: for actively managing the response burden

Respondent rights and responsibilities are made clear

Promotion of the value and use of statistics

Assessment of the necessity to undertake a new statistical survey

A respondent relations officer or advocacy position (not part of the data collection processes)

A provider charter Continuous efforts to develop techniques that reduce the burden Costs of complying with surveys are measured and reported each year &

taken into account when designing new surveys or redeveloping existing ones

Value of administrative data in producing statistics is recognised & statistical purposes are promoted in the design of administrative systems

Burden is measured & taken into account in regular comprehensive reviews of surveys & their methods (hyperlink to example: Austria)

Information packages (demonstrating the value of official statistics) (hyperlink to

example: New Zealand)Initiatives with community groups, business associations & advocates,

schools, etc. to raise awareness of the value of official statisticsInternet-based products – for providing statistical information to businesses

& individuals; these products are promoted through initiatives with communities & respondents

Social media presence is set up (providing e.g. key population and economic indicators)

Users’ needs & the range of data items involved are assessed against the

burden;Statistics are sought from some other sourceCheck if the required data can be produced by modifying an existing survey

rather than instituting a new one Check if an existing survey can be eliminated or reduced in size or content

when users’ requirements, needs or priorities might be changingElimination of already collected data items (or similar ones) in another

survey

NQAF 13: Managing the respondent NQAF 13: Managing the respondent burdenburden


designProgramme

implementation Post-collection

evaluation

50


Response burden reduction: sound survey methods are applied to reduce or distribute the burden

Response burden reduction: statistical standards to make responding to surveys easier

Respondent relations: standard practices to respond to respondents’ requests and complaints

Questionnaire design and implementation: assessments undertaken to ascertain if there were problematic aspects

Good quality frames are usedSample surveys instead of censuses are used, as well as

advanced sampling techniques to minimize sample sizes to achieve the target level of accuracy

Multiple modes of collection are offered Electronic reporting initiatives are introduced (where cost-

effective from a respondent perspective)Data collection is done at the most appropriate time of the

day/yearSurveys are conducted from central registers or other

common frames to better record, assess & control the burden

Questionnaires are pre-testedStandard frameworks, concepts, questions and

classifications are used - while respondents are still allowed to easily complete questionnaires from readily available information or bookkeeping records

Questionnaires are tested to ensure minimal intrusion on privacy and to respect public sensitivities and overall social acceptability

The legal obligation, policies and practices to assure the confidentiality and security of all respondent-provided data are made known

Support centres, “contact us”, “hotlines” Feedback from data providers



designProgramme


evaluation

51

Examples: Examples: Statistics Austria’s Reducing respondents’ burden websiteStatistics Austria’s Reducing respondents’ burden website(http://www.statistik.at/web_en/about_us/responsibilities_and_principles/reducing_respondents_burden/index.html )


http://www.statistik.at/web_en/about_us/responsibilities_and_principles/reducing_respondents_burden/index.html

52

Examples: Examples: Statistics Austria’s Reducing respondents’ burden websiteStatistics Austria’s Reducing respondents’ burden website


53

Examples: Examples: Statistics New Zealand’s Respondent Load StrategyStatistics New Zealand’s Respondent Load Strategy(http://www.stats.govt.nz/about_us/policies-and-protocols/respondent-load-strategy/demonstrate-the-value-of-official-statistics-to-respondents.aspx)


http://www.stats.govt.nz/about_us/policies-and-protocols/respondent-load-strategy/demonstrate-the-value-of-official-statistics-to-respondents.aspx

54


Examples: Examples: Statistics New Zealand’s Respondent Load StrategyStatistics New Zealand’s Respondent Load Strategy

55


NQAF 16: Assuring timeliness and NQAF 16: Assuring timeliness and punctuality punctuality

Statistical agencies should minimize the delays in making data available.

Timeliness refers to how fast - after the reference date or the end of the reference period - the data are released or made available, whether for dissemination or for further processing.

Punctuality refers to whether data are delivered on the dates promised, advertised or announced (for example, in an official release calendar).

56


Release policy: to describe release procedures and define timeliness targets

Dissemination standards or other relevant timeliness targets: for released data to comply with

[Action plans: developed and followed if the timeliness targets are not met]

Release calendar: published to announce in advance the dates that (major) statistics are to be released

[User notifications: about any divergences from the advance release calendar; to announce the new release time and explain the reasons for the delays]

Monitoring and evaluation procedures: for regularly tracking and evaluating the punctuality of each release as per the release calendar

User requirements: to be taken into account when the periodicity of the statistics is being decided

Released data: to be made available to all users at the same time [If not, and privileged pre-release access is given, it is limited, controlled and publicized]



designProgramme


evaluation

A publicly available written release or dissemination policy , distinguishing between different kinds of statistical outputs (press releases, statistics-specific reports or tables, general publications, etc.)(hyperlink to example: Chile)

International/national standards, e.g. IMF data dissemination standards (hyperlink to example: IMF)

A law or other formal provisions exist that requires the setting of a release calendar

The public is informed about the statistics being released via release calendars; calendars also include information on how the data can be accessed (e.g. through the Internet, in publications, etc.) (hyperlink to example: Italy)

Quality indicators on timeliness and punctuality are regularly calculated, monitored and disseminated.

Procedures for consulting with users about periodicity are in place.

57

NQAF 16 Assuring timeliness and NQAF 16 Assuring timeliness and punctuality punctuality

Elements to be assured: Elements to be assured: Mechanisms/examples: Mechanisms/examples:Trade-offs’ consideration: between timeliness and other dimensions of quality (e.g. accuracy, cost and respondent burden)

Preliminary data release: consideration is given to its possibility and usefulness, while at the same time taking into account the data's accuracy

Contingency planning: for handling emerging problems that could delay the release of data

Schedules and timing: attainable schedules for the production processes are defined

Schedules and timing: for specifying and making known to staff and users the maximum acceptable amount of time that can elapse - between the end of the reference period and the availability of the data

Agreements with data providers: on the planned delivery dates

Procedures in place: to ensure the effective and timely flow of data from providers

Following-up with data providers: to ensure timely receipt of data

Preliminary data: when released, are clearly identified as such; users are provided with appropriate information to be able to assess the quality of the preliminary data

Revision policy: to describe the revisions for those key outputs that are subject to scheduled revisions

Quality indicators: on timeliness and punctuality are regularly calculated, monitored, published and followed up

Guidelines are available on how to deal with delays when using administrative data for statistical purposes.

Respondents are made aware and reminded of the deadlines set for reporting.

A published revision policy exists (hyperlink to example: Portugal)

Quality indicators on timeliness and punctuality are regularly calculated, monitored and disseminated (hyperlink to example: Eurostat)


designProgramme


evaluation


58

Examples: Examples: Chile's Policy on Data DisseminationChile's Policy on Data Dissemination


59

Examples: Examples: IMF's Dissemination IMF's Dissemination

Standards Bulletin Board Standards Bulletin Board


60

Examples: Examples: Italy's release calendarItaly's release calendar


61


Examples: Examples: ESS Quality and Performance ESS Quality and Performance

Indicators 2010Indicators 2010

62


Examples: Examples: Portugal’s Revisions Policy, December 2008 Portugal’s Revisions Policy, December 2008 (http://www.ine.pt/ngt_server/attachfileu.jsp?look_parentBoui=70086429&att_display=n&att_download=y)

http://www.ine.pt/ngt_server/attachfileu.jsp?look_parentBoui=70086429&att_display=n&att_download=y%22%20target=%22_blank%22





63

IV. NQAF IV. NQAF Website and Online GlossaryWebsite and Online Glossary

• Website - http://unstats.un.org/unsd/dnss/QualityNQAF/nqaf.aspx

• Glossary - http://unstats.un.org/unsd/dnss/docs-nqaf/NQAF%20GLOSSARY.pdf

64

65

V. FUTURE CHALLENGESV. FUTURE CHALLENGESIdentification of practices that constitute desirable minimum standards• Institutional arrangements & experiences of countries are too different to allow for

meaningful recommendations; at this point in time, could be premature

Development of practical measurable quality indicators • Detailed supporting questions contained in the guidelines do allow for a comprehensive

qualitative description of the degree of quality of a given statistical process or product; more work on the further identification of indicators would be beneficial

Development of specific training materials based on the tools developed • This exercise could and should build on existing materials available at the national and

international levels• The comprehensive inventory of materials as contained at the United Nations National

Quality Assurance Frameworks website will be a valuable starting point • As a matter of practicality, member countries continue to be encouraged to share their

national documentation and experiences as generously as possible

66

Countries are encouraged to express their needs & expectations

Feedback: What do countries consider to be the positives and negatives of the international organizations’ work vis-à-vis quality?

Guidance: what are countries’ priorities regarding NQAFs and quality assurance work?

V. FUTURE CHALLENGESV. FUTURE CHALLENGES

67

THANK YOU for your attentionTHANK YOU for your attention

Contact UNSD at: [email protected] UNSD at: [email protected]

Documents

1 National Quality Assurance Frameworks Mary Jane Holupka and Iliana Vaca Trigo Learning Centre on National Quality Assurance Frameworks 43 rd Session