A Panel Pilot Study for English Business Survey

Presented by Yi Zhang

Department for Business, Innovation and Skills

Overview

• Introduction to EBS – Why, Who, How – Dissemination

• Panel Pilot Study – Pros and Cons of Panel approach– Objective– Methodology

• Timescale • Sample Selection and profile • Fieldwork

– Results • Analysis of responses • Mode effects

– A propensity weighting approach

– Conclusions and Next steps

• Comments and Questions

Introduction to EBS

Introduction to EBS

EBS - Why ?

• Need to maintain regular flow of timely sub-national data after close-down of RDA

• EBS seen as best way to provide this

• Survey intended to complement existing sources, not replace them

• In accordance with transparency agenda, data will be freely available for users to conduct own analysis

Introduction to EBS

EBS - Who?

• Data collected by TNS-BMRB – independent contractor commissioned by BIS

• Three-thousand workplaces surveyed every month

• Covers all nine English regions

• Sample drawn from Inter-Departmental Business Register

• Covers all sectors, including public sector

Introduction to EBS

EBS- How?• Short telephone-based survey

• Designed to be light-touch as possible• Approximately 10-12 minutes to complete• Voluntary

• Directional questions - able to complete without detailed material

• Currently use a cross-sectional approach• Panel pilot study undertaken to explore the

possibility of moving to a panel from year 2 onwards

EBS Dissemination

• Monthly Statistical Releases

• Planned Quarterly Statistical Releases

• Online reporting tool (under development)

Balance statistics (i.e. higher minus lower) by workplace size)

Jan ’12 vs. Oct ‘11

-30%-20%-10%

0%10%20%30%Output

Stocks

Employment

Investment

Labour costs

Output prices

Less than 10 employees

England average

Panel Pilot Study

Pros of a Panel Approach

• Increase analytical capability – Track individual workplaces over time and across

samples – Enhance indicators of economic activity and

behaviour

• Improve quality and reliability – More larger businesses can be interviewed – Less variation in the sample between waves

• Cost effectiveness – Cheaper to re-contact the same respondents

Cons of a Panel Approach

• Potentially increase respondent burdens - a maximum of four interviews per year for each workplace as opposed to one currently

• May cause a break in the series so far – size of break is unknown but could be investigated in early stages of panel

• Expected to be small given the questions remain the same and the panel gradually built up

Panel Pilot Study - Objectives

Objectives of the Pilot Study

• To estimate the proportion of workplaces to participate in a Panel survey (by size, region and sector)

• To estimate the proportion of workplaces to respond online

• To explore any mode effects between the online and telephone data

Panel Pilot Study – Method

Timescale for the Pilot Study

Month ActionNovember 2011 Main fieldwork starts

January 2012 Select sample for Panel re-contact

February 2012 Run Panel experiment (contact 1) (online invite followed by telephone contact)

March 2012 Interim analysis of results

April 2012 Run Panel experiment (contact 2) (online invite followed by telephone contact)

Early May 2012 Analyse results

May- June 2012 Discussion of panel results and assessment of pros and cons of a changed approach

By end of June 2012 Decision taken as to whether to adopt a panel approach

Sample Selection

3081 interviews in Nov

main fieldwork 2357 agreed to be

re-contacted

1000 selected in Re-contact 1

(957 continued in Re-contact 2)

1,357 remain

A1: workplaces with email address

Re-contact 1 – 688

Re-contact 2 – 662

A2: workplaces who provided no email - to be contacted by phone

Re-contact 1 – 312

Re-contact 2 – 295

B: workplaces with email addresses (contacted to boost online response)

Re-contact 1 – 926

Re-contact 2 – 914

Sample Profile by Region and Sector

• Similar profiles of selected sample, those who agreed to recontact, and the Nov fieldwork sample within each region and sector

Sample profile by Region

0% 5% 10% 15% 20%

North East

North West

Yorkshire and The Humber

East Midlands

West Midlands

East of England

London

South East

South West

Sample profile by Sector

0% 20% 40% 60% 80%

Production

Construction

Services

Education, Health, Public admin & Defence

Main fieldwork Nov Agreed to re-contact Selected for panel

Sample Profile by Workplace Size

• Profile of those who agreed to re-contact mirrored very closely to the main Nov sample by workplace size

• Relatively higher proportion of 50+ selected for the panel – to encourage more larger workplaces to participate

Sample profile by workplace size

0% 10% 20% 30% 40% 50%

0-9

10-49

50-249

250+

Main fieldwork Nov Agreed to re-contact

Selected for panel

Panel Fieldwork

• Two re-contact panels– Each one calendar month: Feb and Apr 2012 – Overlap of respondents between two panels : ~40% – Same design for each follow up:

• two groups: A1: mixed modes ; A2: CATI only

• two modes: online (CAWI), telephone (CATI)

A1:Supplied email address & Phone # CAWI CATI & CAWI

A2:Supplied Phone # only

CATI

wk1 wk2 wk3 wk4

Panel Pilot Study – Results

Results 1: Conversion rates and online take up

• Conversion rates consistently lower in Panel Apr across workplaces of different size, regions and sectors

• No consistent pattern - conversion rate higher for the CATI only (A2) compared to the mixed mode (A1) in Panel Feb, but conversion rates for A1 and A2 similar in Panel Apr

–Due to a lower conversion rate for large workplaces from A2 group in Panel Apr

• Rates for the online mode much lower than CATI for both panels

A1: tel , 44.90%

A1:online, 8.00%

A1:online, 7.10%

A1: tel , 49.70%

Conversion rate by group

0% 10% 20% 30% 40% 50% 60% 70% 80%

Panel Apr

Panel Feb

Overall conversion rate Conversion rate (A2 ) Conversion rate (A1)

Results 2: Conversion rates by workplace size and industry

• Workplaces with 250+ employment were less likely to respond for both panels

• Those in Education, Health Public Admin and defence were more willing to participate in both panels

Conversion rate by workplace size

0% 10% 20% 30% 40% 50% 60% 70%

0-9

10-49

50-249

250+

Apr Panel

Feb Panel

Conversion rate by industry

0% 10% 20% 30% 40% 50% 60% 70%

Production

Construction

Services

Education, Health and Publicadministration and defence

Apr Panel

Feb Panel

Results 3: Conversion rates by region

• Regional variation for both panels

– London and the North West were consistently less likely to participate

Conversion rate by region

0% 20% 40% 60% 80%

London

North West

South West

South East

West Midlands

East of England

East Midlands

Yorkshire and The Humber

North East

Feb Panel Apr Panel

Mode Effect – Propensity Weighting Approach

• To investigate whether there was a online (treatment) vs. telephone (control) effect

• Propensity score weighting – Aim: Control for differences in profile between two groups– Method:

• A logistic regression model to produce the propensity weights • Applied weights to the telephone group to match the profile of

workplaces that completed the survey online

Y: mode of completion

X: Workplace characteristics - employment size,

region, sector, single/multi sites, Age

Weight:

Mode Effect – Propensity Weighting Approach (2)

• Only workplace size predictive of being in the online sample– The online sample has a higher proportion of

small workplaces compared with the telephone sample

• After weighting, workplace sizes in the telephone sample closely matches those in the online panel, while region and industry profiles remain similar

Results: Mode Effect

• Difference between two modes but not consistent pattern– small on-line sample – samples not aligned as closely as they need to be after

propensity score matching

• Significant different outcomes of key questions between online and weighted telephone samples:– Those responding by telephone were more likely to agree to

re-contact in future and to have their data linked

– Different responses to the key questions, but not consistently negative or positive

• E.g. Those responding by telephone were less likely to say ‘The same’ at level of business activity or volume of output last

month compared with 3 months before

Panel Pilot Study – Conclusions and Next Steps

Conclusions

• Workplaces would engage with a panel survey - conversion rates where higher than predicted. If a panel adopted, ~50% of the interviews come from re-contact sample in the first follow up month

• Online is not a popular mode - Only a small proportion of any mixed mode panels would be online respondents

• Indications of a possible mode effect - exist but not coherent. More agreements to re-contact and data linkage via telephone mode than online mode

Next Steps

• Results point to a telephone only panel approach

• Further analysis of pilot results

• Put to EBS SG for a decision about whether to change the design for November 2012 fieldwork

Thanks for your attention!

Comments or Questions?