Stop the Madness: Use Quality Targets

Stop the Madness:Use Quality Targets

Laurie Reedman

Scope Aspects of quality

• Timeliness and accuracy

Mechanisms to manage quality• Indicators and pre-set targets

Survey processes• Computer assisted interviewing

• Collection follow-up

• Manual processing

23-04-21Statistics Canada • Statistique Canada2

Statistics Canada’s dimensions of quality Relevance Accuracy Timeliness Accessibility Interpretability Coherence






How can a survey manager manage both process and product quality of data collection and manual processing activities?

Interviewer Monitoring

Computer assisted interviewing Monitor observes and grades samples of

interviewer work Frequency of monitoring sessions geared to

attain desired average outgoing quality level


Responsive Collection Design (RCD)

An adaptive approach to survey data collection Uses information prior to and during data

collection to adjust the strategy for the remaining in-progress cases (Groves and Herringa, JRSS 2006)

Can use RCD to:• Control quality (response rate, representativeness)

• Control cost (time and resources spent)


Responsive Collection Design (RCD) RCD was piloted on 2 surveys at Statistics Canada

(Laflamme and St-Jean, JSM 2011). Three distinct phases during data collection

• Early in collection – attempt all cases

• Mid collection – increase response rates

• Late collection – reduce variability of response rates between domains of interest

Key to success is changing from one phase to the next at the optimal time


Responsive Collection Design (RCD) The turning point decisions are based on the

comparison of quality indicators to pre-set target levels Indicators are derived from paradata from current and

previous collection activity• If targets are too high or too low the turning points will not be

effective at improving quality

• Targets need to reflect the priorities, for example to reduce costs, improve response rates, or optimize both simultaneously


Selective Editing and Top-Down Approach Data editing is a quality assurance activity, not a data correction

activity (John Kovar, 199?) Goals:

• Make data fit for use (not perfect) - effectiveness

• Use as few resources as necessary - efficiency Human resources to do telephone follow-up calls and manual

analysis and data modification are costly Managers need a mechanism to improve efficiency and

effectiveness of manual processes without significantly impacting accuracy


Selective Editing and Top-Down Approach Focus effort where it will do the most good (Hedlin, UNECE

2008) Tackle efficiency and effectiveness from two angles:

• Choose certain units or domains of units for further processing, cease processing of the rest

• Arrange the units requiring further processing in priority order

Pro-actively control the impact on quality by basing turning points and priorities on comparisons of quality indicators to pre-set targets


Selective Editing

When to stop processing• Quality indicator could be mean squared error,

coefficient of variation, response rate, calculated for key variables at cell or domain level

Targets need to be set carefully• If too high, might never be reached, end of processing

will never be triggered, costs will not be reduced

• If too low, resulting data might not be fit for use


Top-Down Approach

How to prioritize units needing more processing• Score function – to get a single rank incorporating

several different criteria simultaneously

• “Biggest” based on some size measure (prior knowledge)

• “Biggest” based on a measure of impact (relative to what has already been collected)

• Most outrageous errors (outlier detection)


An Example: Edit process Canadian Census of Population edit and imputation

process 110 modules grouped into 43 processes Underwent a “Quality Assurance Review” in 2013

(Reedman and Julien, FCSM 2013). 65-70% of time was spent on manual data verification Recommended increased use of automation, and pre-set

quality thresholds to limit activity that amounts to “polishing the apple”


An Example: Edit process Pro-active quality management could include:

• Derive quality indicators for key variables, compare to pre-set targets, and direct satisfactory records onwards to the next processing step, while only retaining unsatisfactory records for appropriate intervention

• Use a top-down prioritization method to further restrict manual intervention to only records having a significant impact

The effect on data accuracy and potential time (cost) savings could be estimated using Census 2011 data


Conclusions Many sources of error in statistical processes We looked at four ways to manage accuracy and timeliness in data

collection and manual processing

• Interviewer monitoring

• Responsive Collection Design

• Selective editing

• Top-down prioritization

• Using paradata• Feasibility and effectiveness demonstrated• Can be used separately or together


Thank-you!

For more information, please contact:

Laurie Reedman

Statistics Canada

[email protected]


Documents

Stop the Madness: Use Quality Targets