User studiesUser studies

Why user studies?Why user studies?

How do we know security and How do we know security and privacy solutions are really usable?privacy solutions are really usable?

Have to observe users!Have to observe users!– you may be surprised by what users you may be surprised by what users

really doreally do– you are not your usersyou are not your users

Typical Security Typical Security EvaluationEvaluation

Does indicator behave correctly when Does indicator behave correctly when notnot under attack?under attack?– No false positives or false negativesNo false positives or false negatives

Does it behave correctly when under Does it behave correctly when under attack?attack?

Can it be spoofed or obscured?Can it be spoofed or obscured?

Wrong indicatorAttacker redirects

Correct indicator

Usability evaluation Usability evaluation questionsquestions Do users notice it?Do users notice it?

– ““What lock icon?”What lock icon?”

Do users know what it Do users know what it means?means?

Netscape SSL iconsCookie flag

IE6 cookie flagFirefox SSL icon

Do users know what to Do users know what to do when they see it?do when they see it?

Other usability Other usability questionsquestions Are they motivated to take Are they motivated to take

action?action? And do they actually do it?And do they actually do it? How about over the long term?How about over the long term?

Why Johnny Can’t Why Johnny Can’t EncryptEncrypt Whitten and Tygar, 1999Whitten and Tygar, 1999

A Usability Evaluation of PGP 5.0A Usability Evaluation of PGP 5.0– Pretty Good PrivacyPretty Good Privacy– Software for encrypting and signing dataSoftware for encrypting and signing data– Plug-in provides “easy” use with email Plug-in provides “easy” use with email

clientsclients– Modern GUI, well designed by most Modern GUI, well designed by most

standardsstandards

Evaluation Evaluation MethodologyMethodology Motivation: Security software may Motivation: Security software may

require additional usability require additional usability considerationsconsiderations

Question: Is PGP usable by everyday Question: Is PGP usable by everyday users?users?

Method: Cognitive Walkthrough + Method: Cognitive Walkthrough + User StudyUser Study

Goal: demonstrate usability problemsGoal: demonstrate usability problems

Question: is method appropriate?Question: is method appropriate?

Defining usable Defining usable security softwaresecurity software Security software is usable if the people who Security software is usable if the people who

are expected to use it:are expected to use it:

1.1. are reliably made aware of the security tasks are reliably made aware of the security tasks they need to perform.they need to perform.

2.2. are able to figure out how to successfully are able to figure out how to successfully perform those tasks perform those tasks

3.3. don't make dangerous errorsdon't make dangerous errors

4.4. are sufficiently comfortable with the interface are sufficiently comfortable with the interface to continue using it.to continue using it.

The studiesThe studies

Cognitive Walkthrough:– Tasks: encrypting and signing email, decrypting, etc.

User Study– PGP 5.0 with EudoraPGP 5.0 with Eudora

– 12 participants all with at least some college and 12 participants all with at least some college and none with advanced knowledge of encryptionnone with advanced knowledge of encryption

– Participants were given a scenario with tasks to Participants were given a scenario with tasks to complete within 90 mincomplete within 90 min

– Tasks built on each otherTasks built on each other

– Participants could ask some questions through Participants could ask some questions through emailemail

Cognitive Walkthrough Cognitive Walkthrough resultsresults Visual metaphorsVisual metaphors

– Public vs. Private keysPublic vs. Private keys– Signatures and verificationSignatures and verification

Key serverKey server– Hidden? What is it doing?Hidden? What is it doing?– Revocation not automaticRevocation not automatic

Several irreversible actionsSeveral irreversible actions– Can cause serious errorsCan cause serious errors

ConsistencyConsistency Too much informationToo much information

– More unneeded confusionMore unneeded confusion

User Study ResultsUser Study Results

3 users accidentally sent the message in clear text3 users accidentally sent the message in clear text

7 users used their public key to encrypt and only 2 7 users used their public key to encrypt and only 2 of the 7 figured out how to correct the problemof the 7 figured out how to correct the problem

Only 2 users were able to decrypt without Only 2 users were able to decrypt without problemsproblems

Only 1 user figured out how to deal with RSA keys Only 1 user figured out how to deal with RSA keys correctly.correctly.

A total of 3 users were able to successfully A total of 3 users were able to successfully complete the basic process of sending and complete the basic process of sending and receiving encrypted emails.receiving encrypted emails.

One user was not able to encrypt at allOne user was not able to encrypt at all

ConclusionConclusion

None of their defined usability None of their defined usability goals were met.goals were met.

Question: Is this a failure in the design of Question: Is this a failure in the design of the PGP 5.0 interface or is it a function the PGP 5.0 interface or is it a function of the problem of traditional usable of the problem of traditional usable design vs. design for usable secure design vs. design for usable secure systems?systems?

Kazaa File Sharing Kazaa File Sharing StudyStudy

Motivation: Lots of people use P2P file Motivation: Lots of people use P2P file sharingsharing

Problem: Seems like lots of people sharing Problem: Seems like lots of people sharing files accidentally. Why?files accidentally. Why?

MethodMethod– Cognitive walkthroughCognitive walkthrough– User studyUser study

12 users, 10 had used file sharing before12 users, 10 had used file sharing before Questionnaire for file sharing understandingQuestionnaire for file sharing understanding Task: figure out what files are being shared Task: figure out what files are being shared

by Kazaa by Kazaa (Answer: Download files set to C:\ so all files (Answer: Download files set to C:\ so all files on the C:\ drive)on the C:\ drive)

Their usability criteriaTheir usability criteria

Peer-to-peer file sharing software is Peer-to-peer file sharing software is safe and usable if users:safe and usable if users:– Are clearly made aware of what files are Are clearly made aware of what files are

being offered for others to downloadbeing offered for others to download– Are able to determine how to share and Are able to determine how to share and

stop sharing files successfullystop sharing files successfully– Do not make dangerous errors that can Do not make dangerous errors that can

lead to unintentionally sharing private fileslead to unintentionally sharing private files– Are comfortable with what is being shared Are comfortable with what is being shared

with others and confident that the system with others and confident that the system is handling this correctlyis handling this correctly

Cognitive Walkthrough Cognitive Walkthrough ResultsResults

Multiple names for similar thingsMultiple names for similar things– My Shared Folder, My MediaMy Shared Folder, My Media , My Kazaa, , My Kazaa,

Folder for downloaded filesFolder for downloaded files Downloaded files are also sharedDownloaded files are also shared Kazaa recursively shares sub-foldersKazaa recursively shares sub-folders Easy to add directories to share, difficult Easy to add directories to share, difficult

to removeto remove

User Study ResultsUser Study Results

5 people thought it was “My Shared Folder” 5 people thought it was “My Shared Folder” – which one UI did suggestwhich one UI did suggest

2 people used Find Files to find all shared 2 people used Find Files to find all shared filesfiles– This UI had no files checked, thus no files This UI had no files checked, thus no files

shared?shared? 2 people used help, said “My Shared 2 people used help, said “My Shared

Folder”Folder” 1 person couldn’t figure it out at all1 person couldn’t figure it out at all Only 2 people got it rightOnly 2 people got it right

Generalizing resultsGeneralizing results

Design suggestions:Design suggestions:– Only allow sharing of multimedia filesOnly allow sharing of multimedia files– Better feedforwardBetter feedforward– Allow exceptions to recursively Allow exceptions to recursively

shared foldersshared folders

A very different studyA very different study

Motivation: Online social networking Motivation: Online social networking widespread.widespread.

Problem: People sharing large amounts Problem: People sharing large amounts of personal information, which puts of personal information, which puts them at risk for variety of problemsthem at risk for variety of problems

Questions:Questions:– how and why do users share and protect how and why do users share and protect

their information? their information? – how do they form impressions of other how do they form impressions of other

profiles?profiles? Goal: Identify requirements, issues and Goal: Identify requirements, issues and

challenges in improving privacy in challenges in improving privacy in online communitiesonline communities

MethodMethod

User study of Facebook.comUser study of Facebook.com– 16 college participants from psych pool16 college participants from psych pool– Logged into own profile, information and Logged into own profile, information and

privacy settings notedprivacy settings noted– Interviewed about their own profile:Interviewed about their own profile:

their motivations for entering information, their motivations for entering information, how they formed their social networks, their how they formed their social networks, their concerns over others viewing their profiles, concerns over others viewing their profiles, etc.etc.

– View 4 other profiles, and interviewed View 4 other profiles, and interviewed about impressionsabout impressions

Big picture resultsBig picture results

Users thinking about their privacy mainly during Users thinking about their privacy mainly during initial activation – while filling out initial profile initial activation – while filling out initial profile informationinformation

Neglect privacy implications of later interactions Neglect privacy implications of later interactions – interacting with friends, not thinking about the – interacting with friends, not thinking about the broader audience at that point…broader audience at that point…

……Until a negative experience occursUntil a negative experience occurs Need new mechanisms to increase awareness Need new mechanisms to increase awareness

of the accessibility of their profile and their risks of the accessibility of their profile and their risks – especially during everyday activities.– especially during everyday activities.

Need new ways to more easily adjust privacy Need new ways to more easily adjust privacy settings during those everyday activities.settings during those everyday activities.

Summing it UpSumming it Up

Examples of how to run user Examples of how to run user studiesstudies– Not the most rigorous studies, but Not the most rigorous studies, but

good enough to demonstrate main good enough to demonstrate main pointpoint

Tradeoffs of various methods?Tradeoffs of various methods? How to choose methods?How to choose methods?

Your ObservationsYour Observations

Where did you observe?Where did you observe? What were some general What were some general

observations?observations? What problems did people have?What problems did people have? Any privacy or security implications?Any privacy or security implications?

What did you think of being an What did you think of being an observer?observer?

Now let’s practiceNow let’s practice

4 groups:4 groups:– Voice recorderVoice recorder– CameraCamera– PDA (2)PDA (2)

Take a few minutes to design a simple Take a few minutes to design a simple user studyuser study– What questions to you have?What questions to you have?– What are your usability goals?What are your usability goals?– What methods?What methods?– Use one member as tester if you canUse one member as tester if you can

User TestUser Test

Results!?Results!?