Care Quality Commission (CQC) Widget User Testing February 2015
Version 1.1
Slide 2
Introduction These are the findings and recommendations from a
day of testing the Widget, February 2015. The individual moderated
sessions (one user / one facilitator) were conducted with a total
of FIVE users who had been recruited to an agreed specification in
particular having themselves or a relative spent time in hospital
within the last two years. Each session covered a subset of
specified tasks as set out in the approved discussion guide
previously circulated. Sessions lasted for 1 hour. Sessions were
conducted using a Windows PC running Tobii eye tracking and Morae
session recording with a remote observer adding notes in real time.
They were remotely observed live locally and via web streaming, and
recorded to enable subsequent further analysis.
Slide 3
The severity assigned to the issues listed below has been
agreed by both of the consultants involved in the testing. There
were no serious issues to report. Serious Major user experience
issues that stop users from completing tasks and would likely lead
them to abandon the site. Significant Issues that interrupt task
progress or are detrimental to the users experience. Minor Smaller
issues that irritate or distract users but do not impact progress.
Positive something of note Severity Ratings
Slide 4
Summary of Results 1 Description of issueRelevant section
Severity Widget Further work required to reduce widget appearing
(and being overlooked) like an Ad 14 Significant Improve the
prominence of this KEY.22 Significant Define your criteria and
ensure it is clearly visible to those that wish to read it. 23
Significant To recommend that all institutions required to display
the widget do so above the fold to improve the prominence of the
CQC widget. 9 Significant Progress the development of the longer
widget with domain field rankings suggested as it looks less like
an advertisement. 27 Significant To suggest that providers display
the widget on other key pages of their site associated with
performance (especially one labeled performance). 10 Minor If
possible, the widget should be included on the pages relevant to
service level performance and rating. 12 Minor
Slide 5
Summary of Results 2 Description of issueRelevant section
Severity Widget contd This reinforces the efforts already under way
by the CQC to raise its public profile and increase general
awareness of the official role that it has. 13 Minor Include Report
Summaries wherever possible. Highlight to those involved in report
generation of their usefulness. 20 Minor Users wanted to see what
the Hospital response was, what are they doing about it? What is
their Action Plan? If possible, these should be included. 20 Minor
Draw together some best practice for widget display. This would
include, positioning, and avoiding contrasting against other boxes
on the side (with images for example). 25 Minor Provide some best
practice examples of how the widget should be displayed on hospital
websites in order to encourage hospitals to improve the visibility
of the performance widget. 17 Minor
Task 1 Introduction. Looking for health info P1 Spent a long
time working in the Middle East, was thinking about retiring to the
Philippines but was worried about health care provision. Ended up
back in the UK and looking at quality of care services, having to
use them quickly on his return for prostate issues and suffered
tuberculosis. P2 - she has bad experiences observing her parents
care and was due for an operation herself, and so she wanted to
check the level of service in advance. Says she wanted to be
prepared for what to expect rather than finding out on the spot. P3
- has previously looked for herself and family regarding health
care, Google is her first port of call and then usually ends up on
a Hospital website. P4 - explains he researches for his mother;
care homes near where he lives, all private no NHS (Google was his
starting point). P5 She trusts the system, but would read things
like OfStead reports if the school was bad and would do the same
for healthhad a previous incident where she had to force the
hospital to take action.
Slide 8
WIDGET
Slide 9
Task 2 Provider Widget (Trust) The red line marks the bottom of
the screen. Participants were looking for information to help them
rate the Trust and were sidetracked by the information in front of
them. Most not scrolling down until prompted. Recommendation To
recommend that all institutions required to display the widget do
so above the fold to improve the prominence of the CQC widget. If
the site is going to be honest, that should be higher up, its good
for them but not good for the patient. Kirti
Slide 10
Task 2 Provider Widget (Trust) cont. The green boxes highlight
the areas that were distracting participants from their task. In
particular the section on the right that takes the user to the page
offering an overview of their performance. Areas on the right-hand
menu included Case Studies and Improving the Patient Experience.
Recommendation To suggest that providers display the widget on
other key pages of their site associated with performance
(especially one labeled performance).
Slide 11
Quotes It feels that the site seems to try to sell itself. I
would expect to see statistics such as number of operations, number
of patients admitted, survivals etc. within the our performance, I
would also expect to see death rate, who sets the targets.
Presumably the trust itself has written the performance page? C I
would immediately look for a yearly performance report, also would
look for forum/feedback and minimum standards and objectives, the
'our performance' and 'case studies' are standing out for me. S
This Our Performance is the page is based on the Trust, they would
be trying to bury anything bad. I guess its from managers, I would
hope there is something from a third party. (She does not scroll
down) S I would read the report to find out more, but I would want
a second opinion as I have not heard of CQC before. M. I would look
at patients and visitors as well as our services, I think it is
lower down because it is negative! K
Slide 12
If possible, the widget should be included on the pages
relevant to service level performance and rating.
Slide 13
Task 2 Provider Widget (Trust) Three of the participants who
did not notice the widget described glancing over it and assuming
it was an advertisement. This will be heavily influenced by none of
the participants knowing the role of the CQC has in determining
service level quality. Recommendation This reinforces the efforts
already under way by the CQC to raise its public profile and
increase general awareness of the official role that it has.
Slide 14
The Widget looks rather like an Ad. It looks like an intrusion,
like Google showing ads. I tend to not pay close attention to it...
it looks like an advertisement, the requires improvement stands out
in orange, too much so and that it's like an ad. C I think this is
a really good idea but I wouldnt have clicked there it didnt catch
my eye. M. It feels that the positioning of it is not too good. I
would also expect to find more information on other pages. M. The
smaller one, I could easily go past itit looks like an advert.
Looking at a service, I would want to see it there regardless as
it's helpful. M. It looks like an advert.. its like one of them
popups, the orange colour is not as vibrant as the red, which is a
lot more popup-like. MK. Recommendation Further consideration
required to reduce the chances of being overlooked as an Ad.
Slide 15
Task 2 Provider Only the first two participants took the
Provider task and it was decided to skip over and straight into the
Hospital example instead. The overview of the hospitals within the
Trust, as seen here. Was favorably received.
Slide 16
Task 2 Provider (smaller) Widget for Trust example As we
dropped the Trust example we did not explore the preferences for
the different widgets within this specific context, however this
was picked up when looking at the Hospital and Service
examples.
Slide 17
Task 3 Hospital Widget Again, the widget falls below the fold
proving difficult to spot (apart from the first two participants).
Recommendation Provide some best practice examples of how the
widget should be displayed on hospital websites in order to
encourage hospitals to improve the visibility of the performance
widget.
Slide 18
Task 3 Hospital Widget The whole widget is one link that
surprised participants expecting to be taken to the sub-section
they are interested in. So, they would be interested in seeing why
well-led is considered inadequate. There was some confusion as to
what the criteria actually meant e.g. safe and how they would find
out how it was being defined. Recommendation Having discussed this
with the team present on the day, I dont think a specific
recommendation can come from this as it is not advisable to bury
someone deep into the report. Criteria descriptions are best dealt
with in situ within the report rather than on the hospital
pages.
Slide 19
Task 3 Hospital Widget (REPORT) These reports were favorably
received by participants. They are clearly set out at first glance
and not too overwhelming. However, there were a few issues to
report.
Slide 20
Task 3 Hospital Widget (REPORT) Looking for why the Hospital
was considered inadequately led, users expect to see a summary
under summary not a link to the full report. Recommendations I
understand there is a gradual move away from report summaries due
to the time taken to draft them, but this is an example of where
they are highly valuable (on the website). This should be
highlighted to those involved in report generation. Users wanted to
see what the Hospital response was, what are they doing about it?
What is their Action Plan? If possible, these should be
included.
Slide 21
Quotes What does well-led mean? S I want to see a very brief
summary [tagline] to know information before diving in. S Wo! The
definition of inadequate, what it says, Im very surprised and
worried to see that inadequate means what it means inadequate is
not a strong enough word! S Im not sure what the expressions mean
and guess I might have to download the inspection report? M. I dont
know why, but its the actual light system that caught my attention
first, Im not sure what safe means, is it secure when youre there?
Or is it your treatments are safe with them? M K. Where's
cleanliness [within the ratings]? M K.
Slide 22
Task 3 Hospital Widget (REPORT) The Key The Click for Key went
unnoticed in these sessions and was only spotted after heavy
prompting. The key only describes the grades not the criteria.
Recommendations Improve the prominence of this key. Each grade can
have a roll-over linking to the key within the report that follows
in order to reinforce the definitions being used. Define your
criteria and ensure it is clearly visible to those that wish to
read it.
Slide 23
Quotes I'd like to know what 'requires improvement' means
[within report]. C It [widget] should be more prominent and I would
also want to see a response from the organisation itself. C (Do the
organisations tell you what their action plan is?) I didnt notice
that at that time, I guess it gives you a bit more in depth
information as to what the icons mean it needs to be a bit more
user friendly. K Im clicking on the word safe, it doesnt tell me
what does it mean by safe? M K. Okay thats interesting but it [the
Key] was too small for me to see it.. I didnt see it. M K.
Slide 24
Task 4 Hospital Widget (Small) Again, it is difficult to
consider the best widget design from the preferences of the five
test participants (that did not offer much in terms of preference).
The widgets are considered after the next example looking at a
Service example.
Slide 25
Task 4 Service Widget This widget in particular looked like an
advertisement to most of the participants. This is because it is on
the side, brightly coloured and it is not helped by the image above
it. Recommendation Draw together some best practice for widget
display. This would include, positioning, and avoiding contrasting
against other boxes on the side (with images for example). I want
to see patient reviews on the website for reassurance. M.
Slide 26
Awareness of CQC and role very low The CQC have made an
assumption that people know who they are! C I dont know if there
are any externals involved in auditing. C I have heard of the CQC
but dont know what they do. S
Slide 27
Widget Considerations Opinions on the widget were mixed between
the small sample (5 participants). However, as there was a familiar
theme emerging, warning that the widget could be mistaken for an
advert the following considerations are offered based on the
comments of the participants and a common sense approach moving
forwards. The smaller widget without the breakdown across domain
fields loses the context of the rating (and is more likely to be
overlooked as an advert). The larger widget, containing the traffic
light system for the relevant domain fields, gives a better
indication that it is a considered (official) rating. The traffic
light system presented here, reinforces the ratings used across the
CQC landscape. It is recommended that the longer widget be the
design developed moving forwards.
Slide 28
Final Reflections The system is a good idea and would help
patients. S I think the explanations are really good.. and the way
they present their information is clear and straightforward. M I
think it is relevant to me - it's quite clear for most people its
like the Food Standards Agency the CQC, I think it is clear that
CQC are official. M. I would trust CQC... especially coming from
the NHS site. K I prefer the small one [widget], it doesnt give you
enough detail so Im enticed to click on read report, I dont want to
be too put off at the start! K Would I look for the ratings? How
would someone like my parents know to look for this stuff? For me,
theres not enough information out there telling me that this
information is at our hands. I didnt know who the Care Quality
Commission were. M K.
Slide 29
REDEYE 29 Dr Jon Rimmer Senior Research Consultant RedEye
[email protected]