Contents
Part 1Why a mystery shopper survey is valuable
Part 2Don’t Stop Me Now methodology – What we did
Part 3Don’t Stop Me Now findings – What we found
Part 4Running your own mystery shopper survey - checklists
Why is a mystery shopper survey valuable?
A Mystery Shopper Survey will enable you to:
Capture an accurate and realistic picture of service user experiences
Assess whether service users are able to easily obtain information about their local area
Mystery shopper surveys and age-proofing
Working with older mystery shoppers:
• Is a quick and cost effective method of assessing whether local services are age-proofed and accessible to older residents
• Will enable councils to benchmark their performance against the findings in Don’t Stop Me Now
• Will enable councils to measure their own direction of travel by repeating the survey on a regular basis
Part 2: Methodology from Don’t Stop Me Now
Part 2 covers:
The purpose and remit of the Don’t Stop Me Now mystery shopper survey
How the Don’t Stop Me Now survey was conducted
Purpose and remit of the surveyThe purpose of the survey was to:
• Evaluate how well councils are able to direct older people to a range of local services designed to maintain their well-being and independence; and
• Ascertain if councils are enabling older people to play a full and active role in the community
The focus was on:
• Services and initiatives outside of social care; and
• Local opportunities to keep fit, well, active and engaged
How the survey was conducted
The mystery shoppers
• Aged 50 – 70
• 2 male, 2 female
• All lived actively and independently with no social care needs
• All made the calls from their own homes
• Calls were spread over Monday – Friday and were made at different times of the day
• Each used a common scenario and tested 10 – 12 councils each (49 councils in total)
How the survey was conducted: Making the calls
The mystery shoppers:
• used a local phone directory to find the number(s) they felt were most appropriate
• additionally used the internet to find numbers and information if they were comfortable doing so
• asked to be transferred to the relevant person or department if they could only locate a call centre number for the council
• used their own personal details (name, age, address)
• prompted and probed if information was not forthcoming
How the survey was conducted: Scenario and script
• The common scenario: “My name is x, I currently live in # and I am moving to your area in the next few months. I am # years old / recently retired and I am looking for information on #”.
• The mystery shoppers were briefed to prompt and probe in all calls to assess the “best case” scenario. For example, if the council was struggling to provide information on opportunities to engage in voluntary work, the mystery shopper would provide some suggestions .e.g. “Is there a local Age Concern that you could put me in touch with?”
• If the council asked for the mystery shopper’s new address or postcode the response was “I don’t have a new address yet, I am just seeking general local information at the moment”.
How the survey was conducted: Areas tested
• In each call the shoppers asked for local information on one of the following areas:
Opportunities to keep fit and active Opportunities to socialise Transport Learning Voluntary work and employment
• If a council was struggling to refer the call to the appropriate person or department then the mystery shopper would ask “Do you have a team or a person that specifically deals with issues for older people?”
Survey findings – November 2007
• In 82% of calls Mystery Shoppers needed to prompt and probe
• It was easiest to obtain information on leisure services for older people and most difficult to obtain information on employment and volunteering opportunities
• In 69% of calls shoppers were referred to a website and no information was sent in the post
• 56% of calls were dealt with on first contact, 35% were transferred once and the remainder transferred more than once
• Most transferred calls were to adult social care
Mystery shoppers needed to prompt and probe in 82% of calls overall
0% 20% 40% 60% 80% 100%
Employment
Voluntary Work
Learning
Transport
Socialise
Keep fit and activeInformation providedspontaneously
Had to probe to getinformation
No informationprovided, even afterprobing
Was information sent in the post?
31%
69%
Yes
No
* 71%of people over 65 have never used the internet
69% were referred to the internet for more information.
31% were sent information in the post
Good information provided on leisure, socialising and transport
They told me about a leisure card based on ability to pay – it gives various discounts. They produced a directory for adults over 50
and will send it out. Plenty of sports facilities for the older person – Mystery shopper
Try U3A – they do classes, nights out, pub visits, meals out - Council Go to George Lawson hall – it’s
the time of your life for the over 50s! - Council
I’ve actually learned a lot from doing this project and speaking to all the different councils. I never realised how many possible services are out there for people my age – mystery shopper
We have a very good transport system – buses and trains - all you need to get round. We have
a transport map and it will contain all the information that you need. Freedom passes are
available to all over sixty - council
Reliance on the internet and transferred calls
If you haven’t access to a computer you will spend hours on the phone! I
asked the switchboard if they had leaflets they could send me and had
no reply – mystery shopper
The lady had to ask a colleague. I felt sure I had been put through to the wrong department. I was
kept on hold for several minutes….she really couldn't help me with anything else and didn't put me through to anyone else who might have been able to answer my questions – mystery shopper
The council said “Services for older people? I've never heard of this. I have been here for 2 years but never heard of that. We have social care and social welfare - go
onto our website.” – Mystery Shopper
I felt shunted around and that there was no single place where I could get all the
information – Mystery shopper
Part 4: Methodology checklists
Checklists cover:
Project management
Remit, scenarios and scripts
Areas to test
Selecting and training the mystery shoppers
Data capture
Analysis and Presentation
Using the results
Checklist 1 – Project Management
Appoint a project manager
Project manager will need awareness and expertise about local older people and services available
Project manager should consult other local experts to shape survey and analyse data (e.g. local analysts / local voluntary and community / older community
Agree timeframe for survey
Agree overall cost of survey
Checklist 2 – Remit, scenario and scripts
Agree the remit of the survey at the outset (e.g. ease of finding appropriate contact in council / ease with which information was provided / type and quality of information provided)
Agree a common scenario (e.g. I am about to move to your area / I live in this area and have recently retired)
Decide on method(s) of contact (e.g. phone calls / face to face visits / e-mail)
Decide on number and nature of contacts needed and over what period
Ensure contacts are made across a range of days and times (e.g. spread across all working days / opening hours)
Checklist 3 - Selecting the areas to test
Possible options are:
Opportunities to keep fit and active / leisure services
Opportunities to meet new people / socialise
Transport options in the area
Opportunities to study / learn new skills
Opportunities to get involved in volunteering
Employment options in the area
This is not an exhaustive list. The range of area tested can be expanded or contracted depending on local issues.
Checklist 4 - selection and training of mystery shoppers
Select shoppers with a range of ages from 50 upwards
Ensure a gender spilt
Ensure diverse representation to reflect local community
Provide a training session to brief mystery shoppers on the purpose and remit of the work
Take on board suggestions from the mystery shoppers, and adapt methods and script accordingly
Work with local voluntary and community sector partners to help select / train / brief mystery shoppers
Checklist 5 - Data Capture
Consult the mystery shoppers on their preferred method of data capture (e.g. written pro-forma / IT solution)
Consider recording calls to capture full conversation and to provide accurate quotes. Consult mystery shoppers on preferred methods of data recording
Ensure the method of data capture reflects each aspect of the remit of the survey (e.g. number of times call transferred / department / person transferred to / number of referrals to the intranet / offers to send written information / quality of information provided)
Allow mystery shoppers to record overall experiences / any other information on each call
Part 6 – Analysis and presentation of the data
Use expertise from in-house analysts
Provide quantitative data (e.g. on number of calls / number of transferred calls / number of calls that provided spontaneous information / number of calls that needed prompting and probing)
Separate data into areas tested e.g leisure / transport etc
Use quotes from calls and comments from mystery shoppers to illustrate the content and usefulness of calls
Use Audit Commission report and findings as a guide on presentation (part 3)
Checklist 6 - Using the results
Compare results against Audit Commission findings from November 2007 to benchmark performance (part 3)
Repeat survey on a regular basis (quarterly / bi-annual / annual) to benchmark the direction of travel of council
Report on results as part of scrutiny and performance process