© 2

008

The

Mat

hWor

ks, I

nc.

® ®

Using Instant Messaging to Usability Test an API

Rachel Cobleigh

Donna Cooper

2

® ®

Our task: evaluate the usability of an API

API = Application Programming Interface Used for writing software code Programmers (our users) have different

expectations than point-and-click users

This API was in the design phase, so no live prototype was available

3

® ®

Our product: MATLAB®

4

® ®

Example command-line interaction

5

® ®

Challenge #1: No GUI

6

® ®

Partial Solution: Use the documentation

?

7

® ®

Another solution: Language Norming

Describe a task and ask the participant to suggest an API name

Provide a list of API names and ask the participant to choose the one that best describes a task

Show the participant an API name and ask what they think it does

Provide help documentation and observe adoption of API names

Describe a task and ask the participant to suggest an API name

Provide a list of API names and ask the participant to choose the one that best describes a task

Show the participant an API name and ask what they think it does

8

® ®

How does API testing compare to GUI testing?

What's different Users will need to rely more heavily on documentation Prepare at least a first pass at error and warning messages and

bare-bones help documentation

What’s the same Recruit as usual Prepare a scenario and user tasks to structure the test session Have a developer act as the “computer” Do walkthroughs before the first real usability test

9

® ®

Challenge #2: Realistic API interaction when there’s no live prototype

Paper prototype? Awkward participant experience Difficult to manage test results

10

® ®

Challenge #2 continued...

Word processor? Participant experience still awkward Pre-written “computer” responses could

bias test results Difficult to manage test results

?? Undefined function or variable ‘c’.

Warning: site vector should be arrow Vector with integer elements

??? Error using ==> imshow>preParseInputs at 340IMSHOW expected at least 1 input argument but was called instead with 0 input arguments.

>> d = zeros(0.5)Warning: site vector should be arrow vector with integer elements

d = [ ]

d = zeros(c)?? Undefined function or variable ‘c’.

11

® ®

Solution: Use Instant Messaging (IM) to simulate the command-line interface!

IM interaction closely resembles command-line interaction

Blank screen interface

Text-based input and responses

Immediate interactive responses to input

12

® ®

Example of an IM prototype interaction

13

® ®

Set up with a local participant

14

® ®

Set up with a remote participant

15

® ®

Mechanics of a successful test session

Customize Use copy-and-paste Have responses ready Agree on a protocol Provide help and/or documentation Ensure that “computer” realistically simulates the

command-line

[>>] try looking at other methods[>>] command mostly worked

16

® ®

Observations & tips for success

IM interactions closely resemble command-line interactions

Participants quickly understood what was expected of them because IM conversations “felt like” the command-line interface

Pre-defined text made the IM session flow quickly so that the illusion of really interacting with the software could be maintained

Delays while “computer” is typing are good opportunities to ask participants questions

17

® ®

Benefits of using IM

Improved participant experience

Easier to manage test results

Transcripts are easy to save and disseminate

18

® ®

Limitations of using IM

It's not a perfect simulation

It can have a slightly different look-and-feel

There can be security issues if testing proprietary designs over the internet

Avoid using personal IM accounts: minimize outside interruptions

Hey hon, can u stop for milk on the way home?

19

® ®

Conclusions

We improved the usability of our API

The techniques were simple and effective

20

® ®

Acknowledgements

We are grateful for the input given by the following people at The MathWorks, without whom this work would not have been possible:

Mary Beth Rettger Amy Kidd Jennifer Lymneos Jeff Goodwin Ellen Mangan

21

® ®

Related work

Steven Clarke has published work on API usability testing Clarke, Steven, Measuring API Usability, Dr. Dobb’s Journal

Special Windows/.NET Supplement, May 2004, pp. S6-S9 Focus is put on analyzing test data in terms of the Cognitive

Dimensions framework http://blogs.msdn.com/stevencl/

Testing was done on a live prototype, in an integrated development environment (i.e., a specialized text editor)

Laura Chessman introduced Language Norming to The MathWorks