Cost-Justifying Usability

Javad Sadeghi

CPSC 681

Nov 5, 2007

Outline

Motivation Background Discount usability Samples Methodology Worked example

Motivation

Dennis Wixon, SIGCHI Bulletin, Volume 27, Number 2, April (1995), p94

View from the other side of the table For each dollar a company invests in developing

the usability of a product, the company receives $10-$100 in benefits and wins customer satisfaction and continued business

For each dollar spent to fix a problem during product design, $10 are spent to fix the same problem in product development, and $100 or more are spent to fix the same problem after product release

Claire Marie Karat, A business case approach to usability cost justification. In, R. Bias and D.

Mayhew, Eds. Cost-Justifying Usability, Academic Press, NY, 1994. 

Background

Cost-benefit of usability (Marilyn Mantei and Toby Teorey)

Business case approach (Clare-Marie Karat )

Cost-justifying usability (Bias and Mayhew)

Discount usability (Jacob Nielsen)

Web-based applications

Bias R. G., Mayhew D. J. Cost-Justifying Usability: An Update for the Internet Age, 2nd Edition. Morgan

Kaufmann Publishers, 2005

Discount usability

“Le mieux est l’ennemi du bien” Focuses on “the good”:

Reduces the costs Requires less experience and knowledge

Three techniques: Scenarios Simplified thinking aloud Heuristic evaluation

Nielsen J., Guerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier

http://www.useit.com/papers/guerrilla_hci.html

Scenarios

Cuts down complexity by eliminating parts of full system

Small and cheap to design and implement Paper mock-ups

Nielsen J., Guerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier

http://www.useit.com/papers/guerrilla_hci.html

Simplified thinking aloud

Thinking aloud studies, videotape subjects, perform detailed protocol analysis

Simplified version Take notes 3~5 test user per test

Nielsen J., Guerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier

http://www.useit.com/papers/guerrilla_hci.html

Heuristic evaluation

Finding usability problems Evaluator inspect alone Current usability guidelines: 1000 rules 10 basic usability principles Easy to learn Experts best Non-experts can find most problems Remaining, using simplified thinking aloud

Nielsen J., Guerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier

http://www.useit.com/papers/guerrilla_hci.html

Nielsen’s surveys on cost-justifying usability

Usability costs ~ 10% of the budget of each project Cost does not increase linearly with the budget

Usability benefits 135% usability increase after redesign

Return on investment (ROI) It is not linear with the costs The more users, the bigger ROI

Jakob Nielsen's Alertbox, http://www.useit.com/alertbox/20030107.html

Samples of cost-justifying usability Australian insurance company

Usability-enhancing project, costing < A$ 100,000 Average of 7.8 errors per form More than one hour to repair the errors of each form

Annually saving as a result > A$ 530,000 Redesigning application forms

IBM employees login system Usability project, financing $ 20,700

Making sign-on attempts faster Savings in work time during the first day of the new

system $ 41,700

Nielsen J., Usability Engineering. Academic Press, 1993.

Methodology

Cost-justifying usability is A conservative approach Part of project planning

Mayhew and Mantei’s framework A basic framework for cost-justifying usability Estimating costs Estimating benefits Justifying the assumptions made for benefits

Randolph G. Bias, Deborah J. Mayhew, Cost-Justifying Usability. Academic Press, 1994.

Estimating costs

Steps Personnel hours and

equipment costs

Needs experience Additional one time

setup Lab setup

Development of survey Pilot testing Distribution and collection Responding Coding and entering data Analyzing results Computer time Supplies and duplicating costs

Usability tasks (e.g. user profile) Techniques (e.g. user questionnaire)

Randolph G. Bias, Deborah J. Mayhew, Cost-Justifying Usability. Academic Press, 1994.

Estimating benefits

Relevant audience Internal development

organization Vendor company

Relevant categories of benefits

Estimation Best measurement unit Assuming magnitudes

Increased user productivity Decreased user errors Decreased training costs Savings gained from making

changes earlier in design Decreased user support

Increased sales Decreased customer support Savings gained from making

changes earlier in design Reduced cost of providing

training

Randolph G. Bias, Deborah J. Mayhew, Cost-Justifying Usability. Academic Press, 1994.

Example

Data entry system for 250 users Two to four primary screens Process 60 of them per day Users work 230 days a year Total cost per hour of $25 The benefit of 1 second speedup in year 250 users * 60 screens * 230 days * 1/3600

hours * $25 = $23,958 / year

Randolph G. Bias, Deborah J. Mayhew, Cost-Justifying Usability. Academic Press, 1994.

Justifying the assumptions made for benefits

Reasonable and likely minimum benefits Not precise, proven and specific Basing the predictions on known facts

Published research Actual case histories Previous experience

Randolph G. Bias, Deborah J. Mayhew, Cost-Justifying Usability. Academic Press, 1994.

Cost-benefit analysis of heuristic evaluation: a case study Case study: integrating system

For internal telephone company 11 evaluators Find 44 usability problems:

40 core problems 4 from other part of the interface

Estimating the costs (evaluation time) Estimating the benefits (increased usability)

Nielsen J., Guerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier

http://www.useit.com/papers/guerrilla_hci.html

Estimates of the times spent

Assessing appropriate ways to use heuristic evaluation, 4 people @ 2 hours 8

Having outside evaluation expert learn about the domain and scenario 8

Finding and scheduling evaluators, 1.8 hours + 0.2 hours per evaluator 4

Preparing the briefing 3

Preparing scenario for the evaluators 2

Briefing, 1 system expert, 1 evaluation expert, 11 evaluators @ 1.5 hours 19.5

Preparing the prototype (software and its hardware platform) for the evaluation 5

Actual evaluation, 11 evaluators @ 1 hour 11

Observing the evaluation sessions, 2 observers @ 11 hours 22

Debriefing, 3 evaluators, 3 developers, 1 evaluation expert @ 1 hour 7

Writing list of usability problems based on notes from evaluation sessions 2

Writing problem descriptions for use in severity-rating questionnaire 6

Severity rating, 11 evaluators @ 0.5 hours 5.5

Analyzing severity ratings 2

Total 105

►

►

►

Nielsen J., Guerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier

http://www.useit.com/papers/guerrilla_hci.html

Time expenditure

time(i) = 47.8 + 5.2 i (Eq1) i: number of evaluators Not exact for larger i

Reduction: reducing 2 observers to 1 Revised: time(i) = 37.3 + 4.2 i (Eq2) Estimated hourly loaded cost: $100 Cost estimation: $10,500

Nielsen J., Guerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier

http://www.useit.com/papers/guerrilla_hci.html

Benefit estimation

Estimations of evaluators for improvements

Nielsen J., Guerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier

http://www.useit.com/papers/guerrilla_hci.html

Benefit estimation (cont.)

Mean values: 0.8 days -learning time reduction 18%-expert speedup

To be conservative: 0.5 days - learning time reduction 10%- expert speedup (3.3% of total work time)

Assuming 2000 users (from 3000) User-days saved 1,000 User-years saved 67 = 13,000 user days Total user days saved = 14,000

Nielsen J., Guerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier

http://www.useit.com/papers/guerrilla_hci.html

Convert to monetary terms User-day cost of $100 With actual saving of 50% One-year saving $540,000 Net present value (benefit): $500,000

Benefit-cost ratio: 48

Nielsen J., Guerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier

http://www.useit.com/papers/guerrilla_hci.html

Thanks!

Questions?