Evaluating Your Prototype with Users Class 30. Agenda 3:00-3:05 Announcements 3:05-3:50 Evaluation with Users

Evaluating Your Prototype with Users

Class 30

Agenda• 3:00-3:05 Announcements

• 3:05-3:50 Evaluation with Users

Overview of Evaluation Methods

With tasks Without tasks

With

out

user

sW

ith

user

s

Testing with Users

• Task-specific– think alouds – wizard of oz

• Task-independent– alpha and beta testing – Bugs vs usability

Conducting a User Study

• Planning• Preparing• Doing• Analyzing

The Challenge

2 mi.

5 mi.

• Within 2 miles there are no playgrounds, safe areas to run around, or supermarkets. But there are fried food joints.

• Within 5 miles there is a supermarket, but no playgrounds or YMCAs.

• The schools in the immediate area got rid of gym, music, and art to get the resources needed to fulfill the No Child Left Behind requirements. Recently, they cut down on recess time for more classroom time!

• Children under 5 have 3 of the 10 factors that go into heart disease risk (obesity, high blood pressure, and tobacco use - second hand). The single parents have 7 of the 10 factors!

How can we solve this challenge?

How can we solve this challenge?

Obesity

High Blood Pressure

Tobacco Use

Exercise

Nutrition

Education

Walking…in regular life

Informed decisions about what you eat

Just in time messages instead of lecture, nag, zzzz…

Low Tech:

Pedometer, information cards, pamphlets

High Tech:

Pedometer or Mobile Phone

Possible Ideas…

My food phone

Bones in Motion

Planning: Developing a Task

• Describes what the user wants to accomplish, not how

• Describes a “complete” job

• Describes who the user is

• Very specific

Example Task for our Challenge

• Tammy wants to find out how many miles she walked today and see if she could increase her step count for tomorrow. She wants to see a map of where she walked and see if there are safe alternatives to increase her step count.

o What?

o Complete Job?

o Who?

o Specific?

Example Task for our Challenge

• To calculate how far you walked today, turn on your phone and walk throughout the day. To see how far you walked in total, go to the menu screen, select BMI, and view your results on the main BMI screen.

o What?

o Complete Job?

o Who?

o Specific?

Develop a task for My Food Phone

o What?

o Complete Job?

o Who?

o Specific?

Planning: 80% of usability problems are uncovered with 5 participants

00.10.20.30.40.50.60.70.80.91

1 5 10 20

Proportion of Problems Uncovered

Number of Participants in Study

Virzi, R. A. (1990). Streamlining the design process: Running fewer subjects. Proceedings of the Human Factors Society 34th Annual Meeting, 291-294. Santa Monica, CA: HFES.

Virzi, R. A. (1992). Refining the test phase of usability evaluation: How many subjects is enough? Human Factors, 34, 4, 457-468.

Be mindful…

• Environmental Factors– Know the rules & regulations

– Visit the site

– Be mindful of safety

• Developing Relationships– Trusting relationship– Dress appropriately

• Participant’s Ability– Be aware of comfort

– Take sensitivities into account

• Time Management– Design adaptable studies

– Multitask

Lessons Learned Conducting User Studies in a Dialysis Ward. Katie A. Siek and Kay H. Connelly. In Extended Abstracts of CHI 2006: Workshops - Reality Testing.

How could this affect your data?

How could this affect your data?How much useful data will you draw out from a hospital visit?

I was just in the hospital for 2 months and it was not a good time for me - I was disoriented and confused most of the time. [Hospitals] are not a good place for people to reflect. - Older Adult


I was just in the hospital for 2 months and it was not a good time for me - I was disoriented and confused most of the time. [Hospitals] are not a good place for people to reflect. - Older AdultOur entire healthcare system is based on the idea assuming that everyone receiving care in a hospital unit are 100% coherent and understanding of their condition…even if this may not be the case. - Doctor


I was just in the hospital for 2 months and it was not a good time for me - I was disoriented and confused most of the time. [Hospitals] are not a good place for people to reflect. - Older Adult

Compromise: For this particular study, we visit the patients while they are in the hospital to see how care can be improved and 2-4 weeks after their hospital visit to give them time to reflect.

Our entire healthcare system is based on the idea assuming that everyone receiving care in a hospital unit are 100% coherent and understanding of their condition…even if this may not be the case. - Doctor

How should information be given?

Did you say 4th grade reading?

• Harvard Health Literacy Studies: http://www.hsph.harvard.edu/healthliteracy/materials.html

• MedLine Plus “How to Write Easy to Read Health Materials” http://www.nlm.nih.gov/medlineplus/etr.html

• Doak and Doah http://www.hsph.harvard.edu/healthliteracy/doak.html

SMOG (< 30 sentences):

1.Count the total number of sentences in the material.

2.Count the number of words with 3 or more syllables.

3.Find the total number of sentences and the corresponding conversion number in SMOG Conversion Table II.

4.Multiply the total number of words with 3 or more syllables by the conversion number. Use this number as the word count to find the correct grade level from Table I.

Wd ct. Grade # of Sent. Conversion

0-2 4 29 1.03

3-6 5 28 1.07

…

11 2.7

10 3.0







Hello. My name is …, and I am a computer science Master's student. I am currently conducting a study for a Medical Informatics class project on smoking factors and what would motivate people to quit. I would like to thank you for participating in this study and would like to express my gratitude with this chocolate. For the following hour, I would like to ask you some questions about your smoking history and experience. After the question answer session, we will be doing a small 5-minute exercise. Please feel free to stop me at any time you wish to. There are no correct or incorrect answers to the questions, so please do not worry and express everything you want. If you have any questions, please do not hesitate to ask. You are helping me a lot by contributing your experience to my study. Any information you give me will only be shared between my teacher and I, and all data will be anonymized.







Wd ct. Grade # of Sent. Conversion

0-2 4 29 1.03

3-6 5 28 1.07

…

31-42 9 24 1.25

43-56 10 23 1.3

57-72 11 22 1.36

…

11 2.7

10 3.0

Planning: What to ObserveQualitative Quantitative

Planning: What to Observe

• Process Data (Qualitative)– How humans go about completing tasks– Why a design does/does not work

• Bottom Line (Quantitative)– Quantitative measures

• Task completion times• Error rates

– How well/poorly a design performs

Preparing: What tools will you need?

Preparing: Practice, Practice, Practice

• Schedule participants and rooms• Prepare forms

•NDA - Non-Disclosure Agreement•Voluntary informed consent

• Plan the procedure•Pre-session survey•Tasks

•which ones and in what order•adapt your tasks

• Post-session: exit survey and debriefing

Doing: Think Aloud Process

• Reduce participant stress level• Your key behaviors

– No excuses– No justifications for the software

• Record observations– Note-taking– Audio/Video– Instrumentation

• De-briefing– Ask questions you think of along the way– Give compensation (if possible)

What would you do a think aloud on?

What would you do a think aloud on?

Low Fi High Fi

Doing: Wizard of Oz

What is it?

When do we use it?

Examples shown in Class or in Papers?

How could we address our challenge with WoO method?

…to give automatic feedback on food intake/calories…

…to give just in time educational messages about tobacco use

Looking Forward

• Week 12• EMRs (Chris)• Healthcare for Underserved• Presenting Data

• Week 13• MI Standards (Danish)• Guest Speaker Sheana Bull• NO CLASS – Users Studies

• Week 14 – Fall Break• Week 15

• NO CLASS – User Studies• Examine User Study Data• What’s Next

Documents

Evaluating Your Prototype with Users Class 30. Agenda 3:00-3:05 Announcements 3:05-3:50 Evaluation with Users