IVR Caller Experience Metrics World IA Day 2013
March 2
What is Caller Experience?
2
• Processes and Design Principles that put
the caller at the center of attention, to
ensure that an IVR is:
– Useful
– Usable
– Aesthetically Pleasing
3 Dimensions of Design
3
Determine Caller Needs
Design To Fit Needs
Engaging
Artful
Clever
Pleasant
Make Obvious
Reduce Errors
Save Time
Equation for Caller Experience Measure
P(Successful Caller Experience) = P(Useful) x
P(Usable) x P(Aesthetically Pleasing)
• Equation is more about people than technology
• Even if the caller experience is useful, it will not
succeed if people cannot use it or do not like it.
3 Key Processes for Caller Experience
1 Know the Caller
3 Evaluate
Appropriately
2 Employ Iterative Design
Caller Needs Analysis 1
Very
Detailed
More
Specific
General
Design
Requirements
• Demographics - age, education, location…
• Call Environment - home, car, office…
• Concurrent Activities - walking, driving,
working
• Needs – get something done
• Wants – quickly and reliably complete
• Enjoys – no “what ifs”
• Exact procedures - step-by-step
• Automation delegation - the caller acts, the
system responds
• How to think - mental models
Iterative Design & Analysis 3 2
Requirements
Compare to needs analysis, standards,
lessons learned, best practices, etc.
High Level Design
Building the functional pieces, placement, order
Detailed Design
Dialog design matches persona, error recovery, transitions, natural traversals
Heuristic Evaluations
UI peers reviewing the design – different peers find different issues
Usability Testing
Compare caller performance & perceptions/preferences to CE Thresholds
Caller Experience Design Principles
• Differentiate confusable functions
and commands
• Know human cognitive and auditory
limits
• Allocate tasks that people are
better at to people, and tasks
machines are better at to machines
• Fit technology to people,
not visa versa
• Minimize what must be learned
• Build in performance support
including reminders & examples
• Keep it simple
• Consistency
• Error recovery
• Cultural norms
• Obviousness
• Menu option grouping
• Caller-controlled pace
• Flexible order of actions
• Consider caller’s environment
• Consider caller’s other activities
• Familiar language & terminology
Measuring Caller Experience
Performance
(Success on
tasks)
Best Success
Best Rating Perceptions/Preferences
(Opinion Ratings)
Caller Experience
Score
Measuring Performance
•Performance Success - metric for how usable a product is.
•Behavioral Data - measuring speed and accuracy
collected by having callers do tasks.
•Scenario-Based Tasks - information-gathering or problem-
solving descriptions are commonly used.
•Design Performance Data - compared to target thresholds
for performance.
Performance Examples
Task Scenario Examples
Task 1: Take money from your checking account
and deposit it into your savings account.
Task 2: Report a death in the family and identify
what needs to be done to collect benefits.
Threshold Examples
Gold Medal Performance: 90% of
participants perform each task error-free.
Silver Medal Performance: 80% of
participants perform each task error-free.
Summarizing Performance
• Performance is summarized for each task.
• Performance on each task can be compared to acceptance thresholds.
• Overall performance is measured by averaging across all measured tasks. This number summarizes how usable the product is.
100
90
80
70
60
Task 1
Task 2
Task 3
Overall
Gold Medal
(outstanding)
Silver Medal
(acceptable) P
erf
orm
an
ce
Su
cc
es
s
(%)
Measuring Perceptions/Preferences
•Opinions form the metric for callers’ Perceptions and
Preferences.
•Questions are asked to determine opinions about
usefulness, usability, and aesthetics.
•A Numerical Scale with anchoring verbal labels is used to
make the opinions quantifiable.
•Deductive interviewing is used to probe callers on
opinions and why is that data?
Perceptions/Preferences Examples
“Useful” example
The system was missing some
important features.
1 2 3 4 5
Strongly
Disagree
Strongly
Agree
“Usable” example
I think most people would learn to
use this system quickly.
1 2 3 4 5
Strongly
Disagree
Strongly
Agree
“Aesthetics” example
The system voice was pleasant like
an agent that I might speak with.
1 2 3 4 5
Strongly
Disagree
Strongly
Agree
Threshold Examples
Gold Medal Preference:
Average rating is 4 or higher.
Silver Medal Preference:
Average rating is 3 or higher.
Summarizing Perceptions/Preferences
• Perceptions/Preferences are summarized for the three dimensions of design, usefulness, usability and aesthetics.
• Ratings for each dimension can be compared to acceptance thresholds.
• An overall rating can be obtained by averaging across the three dimensions. This number is the Perceptions/Preferences opinion rating.
5
4
3
1
Useful Usable Aesthetics Overall
Gold Medal
(outstanding)
Silver Medal
(acceptable)
2
Perc
ep
tio
n/p
refe
ren
ces R
ati
ng
s
Caller Experience Score Matrix
Performance Success (%) (Combined results
for all measured tasks)
Qualitative Ratings
(Combined opinions about usefulness, ease of use and aesthetics)
The Caller Experience Score
Matrix combines the
quantitative data
(callers’ performance) with
the subjective data (callers’
perceptions/preferences)
into a single summary
statistic about the overall
Caller Experience.
GOLD MEDAL CALLER EXPERIENCE
SILVER MEDAL CALLER EXPERIENCE
100
90
80
70
60 1 2 3 4 5
Caller Experience Score Matrix Examples
Performance Success (%) (Combined results
for all measured tasks)
Qualitative Ratings
(Combined opinions about usefulness, ease of use and aesthetics)
The Caller Experience Score
Matrix combines the
quantitative data
(callers’ performance) with
the subjective data (callers’
perceptions/preferences)
into a single summary
statistic about the overall
Caller Experience.
GOLD MEDAL CALLER EXPERIENCE
SILVER MEDAL CALLER EXPERIENCE
100
90
80
70
60 1 2 3 4 5
Auto & Home
Auto & Home
Dental
Dental
Life
Life
Disability
Disability
Questions ?