Skype For Business Usability Study

Usability Study Nathan Hedin | Clara Jauquet | Andre Wyatt

Skype for Business

Table of Contents

Introduction 4

Method and Structure 5

Participant Profiles 7

Severity Ratings 8

Executive Summary 9

Scorecard Results 12

Detailed Findings 15

Conclusions 101

Appendix 105

2

About the Study

3

About the Study

What

Skype for Business on Android

Where

Microsoft Usability Labs, Redmond, WA on 2/20, 2/22, 2/26

Why

Ongoing series of usability tests to evaluate the current release of Skype for Business

How

10 participants worked through 10 scenarios mapped to experience outcomes (tasks). Task success rates, subjective

ratings, and qualitative data about the participant’s experience were collected.

Research Objectives

Has the newest release of SfB improved from a usability and customer satisfaction standpoint?

Can users use multiple features at once?

How valuable do users think the meetings feature is?

Which key features are easily discoverable and learnable by users, and which features do they struggle with?

4

Method and Structure

Pre-test questions

Open questions

about workplace

culture and comm

tools

Ovo Software

Android screen and

participant recorded

Scenarios and tasks

10 scenarios, each

with multiple tasks

within SfB, Success

rates collected

Experience Ratings

Collected after each

scenario with probing

questions

Qualitative Data

About participant task

experience

Post-test questions

Open questions

about experience

with Skype for

Business app

Test Time

90-120 minutes

1 2 3

5

Sample Scenario

Scenario 2b: Once you figured out what the

lunch plan is with each colleague, combine them

into one group conversation to finalize a place to

go for lunch. At the end of the task, return to

main screen view.

Key Tasks:

1. Add a contact to the ongoing chat to create

group chat

2. Have a group chat

3. Return to the main view

Experience Rating:

6

Participant Profiles

P1 P2 P3 P4 P5 P6 P7 P8 P10 P9

Occupation Communication Tools Age Range

7

Severity Ratings

Each finding was given a severity rating based on how it impacts the application usability

Low

Minor usability problem: fixing this should be given

low priority

High

Usability catastrophe: imperative to fix before

product can be released

Medium

Major usability problem: important to fix, should be

given high priority

8

Executive Summary

9

Positive Findings

Overall look and feel of the application

Participants mentioned that it was “familiar” [to Skype desktop], “easy to

navigate,” and liked the “app arrangement” and “UI”

Individual IMs and Notifications

Participants encountered no issues searching for a contact, IMing them,

acknowledging notifications, and responding to IMs

Voice Calls

Participants encountered no issues making a voice call, muting/unmuting

the call, and ending the call

Video Calls

Participants encountered no issues receiving a video call

Meetings

Participants encountered no issues joining meetings

Availability Management

Participants encountered no issues changing their availability status

...clear and intuitive UI, I

liked the look, clean,

colors and circular icon,

look and feel and

arrangement of things.

10

Opportunities

Multiple video windows during group video calls

● Most users independently mentioned they would like to see, or expected to see, video windows

of each of the participants on a video call

Calls

● Error messages were persistent and obscured notifications

● Incoming IM notifications were frequently unseen during phone and video calls (each have their

own issues)

● It was easy for users to accidently drop a call when attempting to add participants to it

Meetings

● Several tasks did not receive top scores from users

● User confusion was observed with in-call buttons, video switching during active speaker mode,

and figuring out who was and was not in the meeting

● Half of all users expressed interest in seeing a shared desktop feature made available for the

Android app

11

Scorecard (% of Top-Two Box Scores*)

12

*Top-two box score: % of satisfaction scores rated a 4 or 5 by participants

Unable to share historical ratings and top box scores

Scorecard (% of Top-Two Box Scores) cont.

13


Scorecard (% of Top-Two Box Scores) cont.

14


Detailed Findings

15

IM Conversations Scenario 2a: You want to chat with your colleagues. Search for your colleague Nathan and send him an IM conversation

about his lunch plans for today. Then IM Andre to see if he is free for lunch today.

Run a search for your colleague

Have an IM conversation

Have two IM conversations

Be aware of messages from another conversation

Be able to respond to messages from another conversation

16

Search for Contact

17

Search for Contact

Perfectly easy. It was direct to see

how to get there.

- P9

18

Average

% Top-Two Box

Exceeds Criteria

% Change from SfB 2015

1

2

3

4

5

Satisfa

ction L

evel

Satisfaction Scores

Unable to share historical

ratings and top box scores

Findings

• Participants encountered no major issues searching for a colleague

• The search bar was the first place all users went to when searching for a

colleague

• The magnifying glass was appreciated by users

• The search result clears and reloads every time the user types a new letter,

which one user complained about

• Searched colleague name showed up twice in one participant's search results

Consider:

• Removing the clear state after each letter entry

• Looking into why a contact might show up twice

Search for Contact

19

Have an IM Conversation

20


Very simple and intuitive…

-P5

21

Average

% Top-Two Box

Exceeds Criteria


1

2

3

4

5

Satisfa

ction L

evel

Satisfaction Scores



Findings

• Participants encountered no major issues when having an IM conversation

• When exiting out of a chat with a partially-typed message, the app says

“Message saved as draft” but the message is not there when the user returns

• One user mentioned she likes to type messages before she sends them when

she multitasks

Consider:

• Saving user’s partially drafted messages for when the user returns to app or

conversation


22

Have two IM Conversations

23


...didn’t even have to think about it…

-P2

24

Average

% Top-Two Box

Exceeds Criteria


1

2

3

4

5

Satisfa

ction L

evel

Satisfaction Scores



Findings

• Met expectations

• Users liked the notifications for incoming IMs

• The back button was appreciated by many users


25

Be aware of Messages from Another Conversation

26


...was looking to see if someone answered

and just saw ‘available’...

-P1

27

Average

% Top-Two Box

Exceeds Criteria


1

2

3

4

5

Satisfa

ction L

evel

Satisfaction Scores



Findings

• For new messages, most users encountered no major issues noticing

messages

• Sometimes the first line of the most recent IM shows up below a user’s name,

and sometimes it just says “Available”

Consider:

• Consistently showing the first line of the last message sent or received


28

Respond to Messages from Another Conversation

29


I want to be able to respond to a message and

not leave the application I was in…

- P6

30

Average

% Top-Two Box

Exceeds Criteria


1

2

3

4

5

Satisfa

ction L

evel

Satisfaction Scores




Findings

• Within the app, no users had issues responding to a message

• The only user who gave a 4 had attempted to respond from the Android pull-

down notification bar (outside of the Skype app)

Consider:

• Allowing users to respond from the Android pull-down notification bar

31

IM Conversations

Scenario 2b: Once you figured out what the lunch plan is with each colleague, combine them into one group conversation to

finalize a place to go for lunch. At the end of the task, return to main screen view.

Add a contact to the ongoing chat to create group chat

Have a group chat

Return to the main view

32

Create a Group Chat

33

Create a Group Chat

Create a Group Chat

34

Average

% Top-Two Box

Exceeds Criteria


1

2

3

4

5

Satisfa

ction L

evel

Satisfaction Scores



Create a Group Chat

Findings

• Group chat was easy to set up - click a name and you’re returned to the chat

with them in it

• “Impressively simple”

• The tradeoff is it would be hard to set up group chats with many people

• “If I wanted to add multiple people he noted he would have to go in and out to

add people”

Consider:

• Allowing users to select several people to chat at once

Create a Group Chat

35

Voice Calls

Scenario 3: You want to have a voice call with your colleague Andre. Ask him about the status of Project X. While Andre is

responding, mute the call, cough, and then unmute the call. When you finish your conversation, end the call.

Make a voice call

Mute and unmute the call

End the call

36

Make/End Call

37

Make/End Call

Make/End Voice Call

So similar to my own phone that it didn’t take much thought… -

- P3 38

Average

% Top-Two Box

Exceeds Criteria


1

2

3

4

5

Satisfa

ction L

evel

Satisfaction Scores



Findings

• Making and ending calls were very easy

• Two users complained that they couldn’t find speakerphone

Consider:

• Changing “audio” to “speaker”

Make/End Call

39

Mute/Unmute Call

40

Mute/Unmute Call

*n=9 (one score not captured)

Icon was big and clear and

right there, very well labeled…

- P7

41

Average

% Top-Two Box

Exceeds Criteria


1

2

3

4

5

Satisfa

ction L

evel

Satisfaction Scores



Findings

• Every user was able to mute and unmute with ease

Mute/Unmute Call

42

Voice Calls

Scenario 4: You are going to receive a call from your colleague Andre. During the call, you might get an IM message from

Nathan; ignore his IM message. At the end of this task, return to the main screen view.

Receive a call from the first screen

Receive an IM message during the call

End the call

43

Receive a Call from the Homescreen

44


That was fine, just like a regular call…

-P6

45

Average

% Top-Two Box

Exceeds Criteria


1

2

3

4

5

Satisfa

ction L

evel

Satisfaction Scores



Findings

• No users had any issue answering a call

• Based on the feedback we received, it seems like the call receiving notification

is close enough to a regular conventions that it is easy to tell users are

receiving a call, but different enough to make it clear that they are answering a

Skype call and not a system phone call


46

Receive and Ignore an IM In-Call

47


I didn't know I got an IM during the call, until after

I was off the phone.

- P6

48

Average

% Top-Two Box

Exceeds Criteria


1

2

3

4

5

Satisfa

ction L

evel

Satisfaction Scores



Findings

• The notification was missed by half of the users

• Many had the phone up to their ear and never saw the banner that came down

on screen. Since there is no audio or vibration cue, it was impossible to tell

something had occurred

Consider:

• Making IM notifications more obvious

• Adding an audio cue

• Make notifications appear over errors

• Visually differentiating notifications from the app


49

Video Calls Scenario 6a: Start a video call with Andre. Make sure your face is centered and the lighting is good for the video call.

During the call, your colleague Nathan might contact you. Ignore him.

Make a video call

Make sure your video looks professional

While in a call, ignore an incoming video call

Stay on the call

50

Make a Video Call and Check Your Video

51

Make a Video Call

Very satisfied with that…

- P5

52

Average

% Top-Two Box

Exceeds Criteria


1

2

3

4

5

Satisfa

ction L

evel

Satisfaction Scores



Check Your Video

53

Average

% Top-Two Box

Exceeds Criteria


1

2

3

4

5

Satisfa

ction L

evel

Satisfaction Scores



Findings

• No users had any issue making a video call

• No users had any issue checking their video

• Many users expressed discomfort looking at their own face for such a long time

Consider:

Easing user’s discomfort while waiting for the other party to start their video

Check Your Video

54

Ignore an Incoming Call

55


I didn’t see an “x” right away...I wanted to

look at the person I was talking to rather

than the top of the screen.

- P10

56

Average

% Top-Two Box

Exceeds Criteria


1

2

3

4

5

Satisfa

ction L

evel

Satisfaction Scores



Findings

• The phone didn’t stop vibrating for 3 users

• “The vibration is very distracting while trying to hold a conversation”

• The notification wasn’t seen by 5 of the users

• Two users requested a polite decline option

• Users wanted to focus on the ongoing call, not the incoming call

Consider:

• Having the phone automatically stop vibrating after a short amount of time

• Increasing the length of time the notifications were on the screen for and make

it simpler to understand

• Fewer buttons, clearer affordances, or button labels

• Adding a polite decline option


57

Video Calls

Scenario 6b: During your call with Andre, you decide you need to include Nathan in the discussion. Please add him to the

call. At the end of the task, return to the main screen view.

Add person to 1:1 call

Have a group video call

End the call

58

Add a Person to the Call

59

Did not see add participant button right

away…maybe have more icons, so it is

more obvious.

-P10


60

Average

% Top-Two Box

Exceeds Criteria


1

2

3

4

5

Satisfa

ction L

evel

Satisfaction Scores




61

Unable to share usability

videos

Findings

• Three users dropped the original call while trying to add the second person

Consider:

• Unnestling “add a participant” from the submenu

• When making a new call while in another call, adding a “Merge” or “End

Ongoing Call” option post click


62

Have a Group Video Call

63

...switching back and forth between the users… I think if

you have too many participants and if the video keeps

switching back and forth it's going to drive people mad or

give them a seizure.

- P7


64

Average

% Top-Two Box

Exceeds Criteria


1

2

3

4

5

Satisfa

ction L

evel

Satisfaction Scores



Multiple People Video Highlight


videos

Findings

• Eight users independently requested having either split screen or thumbnail

video options

Consider:

• Adding a way to see everyone’s video at once


66

Meetings

Scenario 7a: Confirm if you have a meeting. Join the meeting and get your bearings by confirming who is in the meeting

and who hasn’t joined yet. View the contact card details of one of the attendees.

Check for and join a scheduled meeting

Know who is in the meeting (and who hasn’t joined yet)

Know who is speaking in the meeting

View contact card details of one of the attendees

Stay in the meeting

67

Check for and Join a Scheduled Meeting

68


*n=9 (technical issue, P7)

Even if it’s a bad day before

coffee, you aren’t going to

miss that [join] button...

- P2

69

Average

% Top-Two Box

Exceeds Criteria


1

2

3

4

5

Satisfa

ction L

evel

Satisfaction Scores




Findings

• All participants saw scheduled meeting right away and experienced no

problems joining the meeting

• Participants gave this task a top-box score, 8/9 rated the experience a 5

• 9/9 participants succeeded in completing the task

70

Know Who is in the Meeting (and who hasn’t joined yet)

71


*n=9 (technical issue, P7)

I know who is in the meeting but I wasn't

sure who hadn't joined yet that was part

of meeting. I'd probably want to see their

name on the list, but I didn't see that.

- P5 72

Average

% Top-Two Box

Exceeds Criteria


1

2

3

4

5

Satisfa

ction L

evel

Satisfaction Scores




Findings

• 4/9 participants mentioned they would like a list of who is present and who is

absent from the meeting “roll call”

• 2/9 participants failed to complete the task

• Most participants were able to find the attendee list by clicking on the number of

participants in the top left hand corner of the meeting screen (participants banner)

• Meeting is separated presenter vs attendee, no one appeared as attendee

Consider:

• A way to represent invitees who are not in the meeting in the participants banner

• Giving the “participants banner” a visual affordance

73

Know Who is Speaking in the Meeting

74


*n=8 (technical issue P7, & score not collected P2)

People in meetings love to talk over each other….I don’t

know how it [the app] would handle multiple people

talking at the same time.

- P8

75

Average

% Top-Two Box

Exceeds Criteria


1

2

3

4

5

Satisfa

ction L

evel

Satisfaction Scores



Participants Video Highlight


videos


Findings

• 4/8 participants could not determine who was speaking in the meeting

• 2 participants mentioned preferring a split pane option, seeing all meeting

attendees, over active speaker mode

• Delay as view switches between attendees bothered some users

• Primary differential to Skype’s competitors

Consider:

• Incorporating a version of the split pane for meetings as a visual cue

77

View Contact Card Details of One of the Attendees

78


*n=9 (technical issue P7)

I expected to just click on his name to

see his contact info, but I had to do a

couple more clicks to find that. But it

wasn’t that big a deal.

- P5

79

Average

% Top-Two Box

Exceeds Criteria


1

2

3

4

5

Satisfa

ction L

evel

Satisfaction Scores




Findings

• 1/9 participants could not locate a meeting attendees contact card – too many

steps masking the information

• 2 participants expressed they expected to find the contact card by clicking on

another user’s image

• 1 participant did not understand what the contact card is

Consider:

• Explore ways to make the banner more discoverable

• Other issues may be the root cause of discoverability (previous task: meeting

attendees)

80

Meetings

Scenario 7b: One of the attendees will present and share their desktop. During the presentation, you may receive side IM

messages. Respond to Andre, but ignore others. Leave the meeting; Rejoin momentarily; then leave for good.

View the shared desktop to participate in the presentation

Have side IM conversations with a colleague.

Leave the meeting and rejoin the meeting (then leave)

81

View the Shared Desktop to Participate in the Presentation

82


*Desktop not shareable on Android App (task not possible to complete)

n=9 (technical issue P6)

Don’t see any alerts, which I

assume is going to happen...

- P5

83

Average

% Top-Two Box

Exceeds Criteria


1

2

3

4

5

Satisfa

ction L

evel

Satisfaction Scores



Shared Desktop Video Highlights


videos


Findings

• 9/9 participants could not view the desktop

• 5/8 users expressed interest in the shared desktop feature on an Android

• 3/8 users mentioned they expected to receive notifications about a shared

desktop

Consider:

• Addition of the shared desktop feature for the Android platform

• Include an affordance or notification when there is an incoming shared desktop

or another person is presenting

85


When I have to go into the office at 4 am and talk to the UK I

could just do it from home, that would be wonderful... When it's,

you know, it's not planned when when you have a workaholic boss like

mine sending you messages at midnight and he genuinely wants to

call across seas and hash it out right now. I put on a blouse but I am

sitting in my pjs, sitting at home in bed, I can do it all from my phone. I

am one of those people, I've got my smart tv, my smart tablet. I have

got everything you're supposed to have, but for some freakin

reason I use my phone. I watch my movies from my phone, I take

into the bath and I set it up on a little stand. I got a tablet, it doesn't

need a stand, it's got a larger screen, it's got better sound, but I don't

because it's ten feet away and I'm tired...I've got my phone with me. I

always have it with me. I love when it can do everything.

- P2

86

Have Side IM Conversations with a Colleague

87

...did not read as a text conversation...

- P8


88 *n=9 (technical issue P6)

Average

% Top-Two Box

Exceeds Criteria


1

2

3

4

5

Satisfa

ction L

evel

Satisfaction Scores




Findings

• No significant findings with actual task, but secondary issues were found

• 2 participants mentioned that the IM view “did not read as a text conversation”

and felt cluttered with interaction history

• 1 participant thought the shared meeting IM window was a side conversation

Consider:

• Simplifying IM window so it just displays IM conversations instead of an entire

history

89

Leave the Meeting and Rejoin the Meeting (then leave)

90

Here’s a rejoin button, how about that?

- P8



Average

% Top-Two Box

Exceeds Criteria


1

2

3

4

5

Satisfa

ction L

evel

Satisfaction Scores




Findings

• Participants experienced no difficulties leaving the meeting; hang up icon was

very straightforward

• Participants experienced no difficulties rejoining the meeting

• Rejoin button was obvious and easy to find

92

Manage Conversations

Scenario 8: During the last meeting, you remember you had ignored some IM messages. Find those missed messages and

respond to them. At the end of the task, return to the main screen view.

Find the missed conversations

Follow up on the missed communications

93

Find and Follow up on the Missed Communications

94

Bolding is a very common visual cue that

something new is there.

- P2


Would be nice if new messages came to

the top...

- P2


Average

% Top-Two Box

Exceeds Criteria


1

2

3

4

5

Satisfa

ction L

evel

Satisfaction Scores




Findings

• 4/9 referenced the bolded text as a common “visual cue” for an unread

message

• 8/9 participants succeeded in completing the task

• 1 participant found the entire history view in the IM conversation to be

“Disorienting”

• Chronological order?

• “all these things look the same except for the highlights…but I didn’t know if

that is new, if I missed it, or ignored it.” –P1

Consider:

• Simplify IM window

96

Status and Availability

Scenario 9: Change my availability status to Do Not Disturb. At the end of the task, return to the main screen view.

Change my availability status to: Do Not Disturb

97

Change my Availability Status to: Do Not Disturb

98


Pretty obvious that clicking on the face

would bring up options about yourself.

- P1

99

Average

% Top-Two Box

Exceeds Criteria


1

2

3

4

5

Satisfa

ction L

evel

Satisfaction Scores




Findings

• 10/10 participants succeeded in completing the task and rating it

• 7/10 participants immediately went to avatar image and changed their status

• 3/10 participants first checked contacts and calendar before touching the profile

picture

• Profile picture as personal settings seems to be the industry standard

Consider:

• Keep as is, profile picture as link to profile settings

Had a moment of hesitation... that

would be the least intuitive, but i

don’t disagree where it was, it must

be under me.

-P5

100

Conclusions

101

Recap

• Participants found IM conversations easy to navigate

• Participants found making and receiving calls to be easy

• Incoming IM notifications were missed by many users

• Ignoring an incoming call was a distraction for users

• Participant information was nestled within submenus

• Split pane view of users on a video call or in a meeting requested over active

speaker mode

102

Next Steps

• Implement improvements on high severity findings, most notably:

• Affordance to the “participant banner”

• Split pane screen of all users in a call/meeting, with the speaker or selected video taking over

the majority of the real estate

• Change the incoming IM/call notification

• Match the desktop sharing experience to iPhone and desktop

• Run another benchmark study after implementation of changes to monitor

satisfaction ratings increase

• Make process improvements

103

Process Improvement and Food for Thought

• Add more company contacts for a realistic work environment

• Add a task between leaving and joining the meeting

• Side IM with a colleague during the meeting was not a realistic experience with

realistic challenges as viewing shared desktop was not possible

• Even though meeting is separated as presenter vs attendee, no one ever

appeared as attendee

• Keeping profile settings under profile picture?

• History landscape

104

Appendix

105

Documents

Screener:

Data not available to share

Usability Test Kit:


Data Collection:


106