Improving efficiency by introducing macro editing in

Statistics New Zealand business performance surveys

Paper by Tim Hawkes,

presented by Emma BentleyMay 2011

Background

Suite of business performance (BP) surveys, many started around 2005

Quality review recommendations: • Reduce manual micro edits, cut processing costs• More emphasis needed on macro editing• ‘Big picture’ perspective needed for analysts

New approach to editing developed for 2010 survey round

2

Previous editing approach

Micro editing – time consuming (2 FTE!)• Validity edits• Consistency edits• Statistical edits

Very few automatic edits, lots of manual intervention

Macro editing – limited• Pressure of publication deadlines• Not standard across the suite of surveys

3

Recommended new approach

BP analysts reviewed macro practices in-house and editing strategies internationally

Decided to improve:• Automatic micro editing

– Consistency edits– Validity edits

• Macro editing– Create more time for this– Wider range of strategies available to BP team

4

A routing question with:

Both Yes and No then the section is blank Auto edit to No

Both Yes and No then the section is answered Auto edit to Yes

No then the section is answered Auto edit to Yes

Yes then section is blank Auto edit to No or leave for imputation

Automatic micro editing

5

Sum of components not equal to given totals

Routing and validity edits

Macro editing

Calculating estimates during the data collection process• Produce initial estimates when 50-60% of target

response rate achieved• Early detection of influential observations for

validation• Problems with processing or estimation system can

be identified and resolved early in production process• Issue if insufficient responses for estimation purposes• Analysts gain better understanding of estimation

process6

Macro editing

Top-down editing• E.g. drill down to industry and stratum level estimates• Identify unusual components, drill down more• May provide evidence for estimate, or identify error

Use of sampling errors to identify suspicious estimates

• Compare with expectations and previous results• Unusual responses that have effect on sample errors,

may also effect estimates

7

Macro editing

Top contributor method• Ranked lists of top contributors e.g. for level

estimates or movements• Compare lists to previous years

Graphical analysis• Not widely used by BP surveys

Processing checks• Monitoring template developed containing these

checks

8

Implementation

Interactive training provided by Statistical Methods division

Analysts asked to identify potential questions suitable for automatic edits

Up to the lead analyst to implement the new approach and determine their macro editing techniques

BP team able to take more ownership of editing strategy now that the surveys are well established and topic knowledge developed

9

Review: Quality indicators

Quality indicators for business performance surveys

• No decline in quality detected

Quality Indicator

Business Operations

SurveyR&D Survey ICT Supply Energy Use

2009 2010 2008 2010 2008 2010 2009 2010

Average number of edit failures per unit 5.72 6.66 5.00* 4.23* 1.34 1.00

Clerical edits 4130 2497 2619 787 4000* 900* 1206 740

Resource usage (staff hours) – Micro 337.5 187.5 375* 150* 375* 100*

Resource usage (staff hours) - Macro 15 40.5 112.5* 150* 188** 375**

* Estimate** Estimate which also includes resource usage on imputation.

10

Review: Qualitative feedback

Generally positive

Reduction in amount of manual micro editing – popular with analysts!

Timing for incorporating changes a challenge• Set up of automatic editing, will be reusable in future• Saved time from micro editing taken up by

implementation of other methodology changes

11

Review: Qualitative feedback

More time allocated to macro editing

Better macro strategies helped analysts better understand the processes and data, this in turn helps subsequent analysis stage

Conservative approach for first cycle of new editing strategy, room for more efficiencies

Improved ability to calculate quality indicators would help assess efficiencies in future

Reviews are useful!

12