Don’t Stop Me NowPreparing for an ageing population

Mystery Shopper Survey Findings and Methodology

Contents

Part 1Why a mystery shopper survey is valuable

Part 2Don’t Stop Me Now methodology – What we did

Part 3Don’t Stop Me Now findings – What we found

Part 4Running your own mystery shopper survey - checklists

Mystery shopper survey

Part 1 : Why is a mystery shopper survey valuable?

Why is a mystery shopper survey valuable?

A Mystery Shopper Survey will enable you to:

Capture an accurate and realistic picture of service user experiences

Assess whether service users are able to easily obtain information about their local area

Mystery shopper surveys and age-proofing

Working with older mystery shoppers:

• Is a quick and cost effective method of assessing whether local services are age-proofed and accessible to older residents

• Will enable councils to benchmark their performance against the findings in Don’t Stop Me Now

• Will enable councils to measure their own direction of travel by repeating the survey on a regular basis

Mystery shopper survey

Part 2: Methodology from Don’t Stop Me Now – What we did

Part 2: Methodology from Don’t Stop Me Now

Part 2 covers:

The purpose and remit of the Don’t Stop Me Now mystery shopper survey

How the Don’t Stop Me Now survey was conducted

Purpose and remit of the surveyThe purpose of the survey was to:

• Evaluate how well councils are able to direct older people to a range of local services designed to maintain their well-being and independence; and

• Ascertain if councils are enabling older people to play a full and active role in the community

The focus was on:

• Services and initiatives outside of social care; and

• Local opportunities to keep fit, well, active and engaged

How the survey was conducted

The mystery shoppers

• Aged 50 – 70

• 2 male, 2 female

• All lived actively and independently with no social care needs

• All made the calls from their own homes

• Calls were spread over Monday – Friday and were made at different times of the day

• Each used a common scenario and tested 10 – 12 councils each (49 councils in total)

How the survey was conducted: Making the calls

The mystery shoppers:

• used a local phone directory to find the number(s) they felt were most appropriate

• additionally used the internet to find numbers and information if they were comfortable doing so

• asked to be transferred to the relevant person or department if they could only locate a call centre number for the council

• used their own personal details (name, age, address)

• prompted and probed if information was not forthcoming

How the survey was conducted: Scenario and script

• The common scenario: “My name is x, I currently live in # and I am moving to your area in the next few months. I am # years old / recently retired and I am looking for information on #”.

• The mystery shoppers were briefed to prompt and probe in all calls to assess the “best case” scenario. For example, if the council was struggling to provide information on opportunities to engage in voluntary work, the mystery shopper would provide some suggestions .e.g. “Is there a local Age Concern that you could put me in touch with?”

• If the council asked for the mystery shopper’s new address or postcode the response was “I don’t have a new address yet, I am just seeking general local information at the moment”.

How the survey was conducted: Areas tested

• In each call the shoppers asked for local information on one of the following areas:

Opportunities to keep fit and active Opportunities to socialise Transport Learning Voluntary work and employment

• If a council was struggling to refer the call to the appropriate person or department then the mystery shopper would ask “Do you have a team or a person that specifically deals with issues for older people?”

Mystery shopper survey

Part 3: Findings from Don’t Stop Me Now – What we found in 2007

Survey findings – November 2007

• In 82% of calls Mystery Shoppers needed to prompt and probe

• It was easiest to obtain information on leisure services for older people and most difficult to obtain information on employment and volunteering opportunities

• In 69% of calls shoppers were referred to a website and no information was sent in the post

• 56% of calls were dealt with on first contact, 35% were transferred once and the remainder transferred more than once

• Most transferred calls were to adult social care

Mystery shoppers needed to prompt and probe in 82% of calls overall

0% 20% 40% 60% 80% 100%

Employment

Voluntary Work

Learning

Transport

Socialise

Keep fit and activeInformation providedspontaneously

Had to probe to getinformation

No informationprovided, even afterprobing

Was information sent in the post?

31%

69%

Yes

No

* 71%of people over 65 have never used the internet

69% were referred to the internet for more information.

31% were sent information in the post

Good information provided on leisure, socialising and transport

They told me about a leisure card based on ability to pay – it gives various discounts. They produced a directory for adults over 50

and will send it out. Plenty of sports facilities for the older person – Mystery shopper

Try U3A – they do classes, nights out, pub visits, meals out - Council Go to George Lawson hall – it’s

the time of your life for the over 50s! - Council

I’ve actually learned a lot from doing this project and speaking to all the different councils. I never realised how many possible services are out there for people my age – mystery shopper

We have a very good transport system – buses and trains - all you need to get round. We have

a transport map and it will contain all the information that you need. Freedom passes are

available to all over sixty - council

Reliance on the internet and transferred calls

If you haven’t access to a computer you will spend hours on the phone! I

asked the switchboard if they had leaflets they could send me and had

no reply – mystery shopper

The lady had to ask a colleague. I felt sure I had been put through to the wrong department. I was

kept on hold for several minutes….she really couldn't help me with anything else and didn't put me through to anyone else who might have been able to answer my questions – mystery shopper

The council said “Services for older people? I've never heard of this. I have been here for 2 years but never heard of that. We have social care and social welfare - go

onto our website.” – Mystery Shopper

I felt shunted around and that there was no single place where I could get all the

information – Mystery shopper

Mystery shopper survey

Part 4: Running your own mystery shopper survey: Methodology checklists

Part 4: Methodology checklists

Checklists cover:

Project management

Remit, scenarios and scripts

Areas to test

Selecting and training the mystery shoppers

Data capture

Analysis and Presentation

Using the results

Checklist 1 – Project Management

Appoint a project manager

Project manager will need awareness and expertise about local older people and services available

Project manager should consult other local experts to shape survey and analyse data (e.g. local analysts / local voluntary and community / older community

Agree timeframe for survey

Agree overall cost of survey

Checklist 2 – Remit, scenario and scripts

Agree the remit of the survey at the outset (e.g. ease of finding appropriate contact in council / ease with which information was provided / type and quality of information provided)

Agree a common scenario (e.g. I am about to move to your area / I live in this area and have recently retired)

Decide on method(s) of contact (e.g. phone calls / face to face visits / e-mail)

Decide on number and nature of contacts needed and over what period

Ensure contacts are made across a range of days and times (e.g. spread across all working days / opening hours)

Checklist 3 - Selecting the areas to test

Possible options are:

Opportunities to keep fit and active / leisure services

Opportunities to meet new people / socialise

Transport options in the area

Opportunities to study / learn new skills

Opportunities to get involved in volunteering

Employment options in the area

This is not an exhaustive list. The range of area tested can be expanded or contracted depending on local issues.

Checklist 4 - selection and training of mystery shoppers

Select shoppers with a range of ages from 50 upwards

Ensure a gender spilt

Ensure diverse representation to reflect local community

Provide a training session to brief mystery shoppers on the purpose and remit of the work

Take on board suggestions from the mystery shoppers, and adapt methods and script accordingly

Work with local voluntary and community sector partners to help select / train / brief mystery shoppers

Checklist 5 - Data Capture

Consult the mystery shoppers on their preferred method of data capture (e.g. written pro-forma / IT solution)

Consider recording calls to capture full conversation and to provide accurate quotes. Consult mystery shoppers on preferred methods of data recording

Ensure the method of data capture reflects each aspect of the remit of the survey (e.g. number of times call transferred / department / person transferred to / number of referrals to the intranet / offers to send written information / quality of information provided)

Allow mystery shoppers to record overall experiences / any other information on each call

Part 6 – Analysis and presentation of the data

Use expertise from in-house analysts

Provide quantitative data (e.g. on number of calls / number of transferred calls / number of calls that provided spontaneous information / number of calls that needed prompting and probing)

Separate data into areas tested e.g leisure / transport etc

Use quotes from calls and comments from mystery shoppers to illustrate the content and usefulness of calls

Use Audit Commission report and findings as a guide on presentation (part 3)

Checklist 6 - Using the results

Compare results against Audit Commission findings from November 2007 to benchmark performance (part 3)

Repeat survey on a regular basis (quarterly / bi-annual / annual) to benchmark the direction of travel of council

Report on results as part of scrutiny and performance process

For more information onDon’t Stop Me Now

www.audit-commission.gov.uk/nationalstudies/