User Research for Everyone

Aras BilgenGaranti @arasb

@arasb #uxsg

User research is asking deliberate questions about a problem you care about, working with your users methodologically to find out answers and needs, discussing findings with your team to design possible solutions.

@arasb #uxsg

Find a problem and the questions

Work with users to discover

answers

Analyze & discuss with the team

@arasb #uxsg

Define the problem

Formulate questions

Plan inquiry

Carry it out

Analysefindings

Share

@arasb #uxsg Adapted from Christian Rohrer

Attitude

Behavior

Quantitative Qualitative

Focus groups

Interviews

Multivariate testing

Clickstream analysis

Contextual interviews

Usability lab studies

Ethnography

Diary studies

Desirability studies

Eye-tracking

First click tests

Instrumentation

True intent

Off-task surveys

Intercept surveys

Guerilla testing

Remote testing

Noticeability studies

@arasb #uxsg

User research is not “asking” people.

@arasb #uxsg

@arasb #uxsg

User research does not delay things.

@arasb #uxsg

User research is not pure science.

@arasb #uxsg

User research is not witchcraft.

@arasb #uxsg

User research is not expensive.

@arasb #uxsg

No user research is expensive.

@arasb #uxsg

@arasb #uxsg

No user research is expensive.

@arasb #uxsg

User research is not a room.

@arasb #uxsg

@arasb #uxsg

@arasb #uxsg

@arasb #uxsg

User research is not a room.

@arasb #uxsg

User research is not a department.

@arasb #uxsg

We cannot find the right number of users.

This is not the right time for research.

We really need these features, we can do the research-y part later.

The procedures are too complex for our project.

We prefer to innovate.

But there is nothing to test!

We can figure it out in the pilot phase/beta.

But we have Analytics?

@arasb #uxsg

We cannot find the right number of users.

This is not the right time for research.

We really need these features, we can do the research-y part later.

The procedures are too complex for our project.

We prefer to innovate.

But there is nothing to test!

We can figure it out in the pilot phase/beta.

But we have Analytics?

Research procedures are usually very flexible.

Scientific rigor is not required in many methods. It is not only OK, but desired to tailor the procedures to project needs.

@arasb #uxsg

We cannot find the right number of users.

This is not the right time for research.

We really need these features, we can do the research-y part later.

The procedures are too complex for our project.

We prefer to innovate.

But there is nothing to test!

We can figure it out in the pilot phase/beta.

But we have Analytics?

Find a problem and the questions

Work with users to discover answers

Analyze & discuss with the team

@arasb #uxsg

We cannot find the right number of users.

This is not the right time for research.

We really need these features, we can do the research-y part later.

The procedures are too complex for our project.

We prefer to innovate.

But there is nothing to test!

We can figure it out in the pilot phase/beta.

But we have Analytics?

Define the problem

Formulate questions

Plan inquiry

Carry it out

Analysefindings

Share

@arasb #uxsg

We cannot find the right number of users.

This is not the right time for research.

We really need these features, we can do the research-y part later.

The procedures are too complex for our project.

We prefer to innovate.

But there is nothing to test!

We can figure it out in the pilot phase/beta.

But we have Analytics?

Define the problem

Formulate questions

Plan inquiry

Carry it out

Analysefindings

Share

@arasb #uxsg

We cannot find the right number of users.

This is not the right time for research.

We really need these features, we can do the research-y part later.

The procedures are too complex for our project.

We prefer to innovate.

But there is nothing to test!

We can figure it out in the pilot phase/beta.

But we have Analytics?

Define the problem

Formulate questions

Plan inquiry

Carry it out

Analysefindings

Share

@arasb #uxsg

We cannot find the right number of users.

This is not the right time for research.

We really need these features, we can do the research-y part later.

The procedures are too complex for our project.

We prefer to innovate.

But there is nothing to test!

We can figure it out in the pilot phase/beta.

But we have Analytics?

Formulate questions

Plan inquiry

Carry it out

Analysefindings

Share

@arasb #uxsg

We cannot find the right number of users.

This is not the right time for research.

We really need these features, we can do the research-y part later.

The procedures are too complex for our project.

We prefer to innovate.

But there is nothing to test!

We can figure it out in the pilot phase/beta.

But we have Analytics?

Define the problem

Formulate questions

Plan inquiry

Carry it out

Analysefindings

Share

@arasb #uxsg

We cannot find the right number of users.

This is not the right time for research.

We really need these features, we can do the research-y part later.

The procedures are too complex for our project.

We prefer to innovate.

But there is nothing to test!

We can figure it out in the pilot phase/beta.

But we have Analytics?

Questions for the interview

Carry it out

Analysefindings

Share

@arasb #uxsg

We cannot find the right number of users.

This is not the right time for research.

We really need these features, we can do the research-y part later.

The procedures are too complex for our project.

We prefer to innovate.

But there is nothing to test!

We can figure it out in the pilot phase/beta.

But we have Analytics?

Questions for the interview

Carry it out

Analyse findings in greatstrategic detail

Share

@arasb #uxsg

We cannot find the right number of users.

This is not the right time for research.

We really need these features, we can do the research-y part later.

The procedures are too complex for our project.

We prefer to innovate.

But there is nothing to test!

We can figure it out in the pilot phase/beta.

But we have Analytics?

Research procedures are usually very flexible.

Scientific rigor is not required in many methods. It is not only OK, but desired to tailor the procedures to project needs.

@arasb #uxsg

We cannot find the right number of users.

This is not the right time for research.

We really need these features, we can do the research-y part later.

The procedures are too complex for our project.

We prefer to innovate.

But there is nothing to test!

We can figure it out in the pilot phase/beta.

But we have Analytics?

Validity is more important than statistical significance.

Many methods do not require statistical significance to produce useful insight.

@arasb #uxsg

We cannot find the right number of users.

This is not the right time for research.

We really need these features, we can do the research-y part later.

The procedures are too complex for our project.

We prefer to innovate.

But there is nothing to test!

We can figure it out in the pilot phase/beta.

But we have Analytics?

Quantitative and qualitative methods can be combined.

One is not better than the other. They produce different types of insight.

@arasb #uxsg

We cannot find the right number of users.

This is not the right time for research.

We really need these features, we can do the research-y part later.

The procedures are too complex for our project.

We prefer to innovate.

But there is nothing to test!

We can figure it out in the pilot phase/beta.

But we have Analytics?

It is useful before, during, and after the project.

Generative, descriptive and evaluative research reveal different insights.

@arasb #uxsg

We cannot find the right number of users.

This is not the right time for research.

We really need these features, we can do the research-y part later.

The procedures are too complex for our project.

We prefer to innovate.

But there is nothing to test!

We can figure it out in the pilot phase/beta.

But we have Analytics?

GenerativeDescriptiveEvaluative

@arasb #uxsg

We cannot find the right number of users.

This is not the right time for research.

We really need these features, we can do the research-y part later.

The procedures are too complex for our project.

We prefer to innovate.

But there is nothing to test!

We can figure it out in the pilot phase/beta.

But we have Analytics?

Planning:What is the need?

What drives it?Is this real?

Execution:Are we meeting the need?

Are we getting close?Can there be alternatives?

Live product:Have we missed anything? What are the new needs?

How do people feel?

@arasb #uxsg

We cannot find the right number of users.

This is not the right time for research.

We really need these features, we can do the research-y part later.

The procedures are too complex for our project.

We prefer to innovate.

But there is nothing to test!

We can figure it out in the pilot phase/beta.

But we have Analytics?

Pilots/Betas are too late in the product cycle.

You may have already built something that no one cares about.

I thought we were going to build a better Facebook. But within 30 minutes after the release, I realized, Oh my God, it's broken. Holy shit, we totally fucked up!

-Bill Nguyen

@arasb #uxsg

We cannot find the right number of users.

This is not the right time for research.

We really need these features, we can do the research-y part later.

The procedures are too complex for our project.

We prefer to innovate.

But there is nothing to test!

We can figure it out in the pilot phase/beta.

But we have Analytics?

Pilots/Betas are too late in the product cycle.

You may have already built something that no one cares about.

@arasb #uxsg

We cannot find the right number of users.

This is not the right time for research.

We really need these features, we can do the research-y part later.

The procedures are too complex for our project.

We prefer to innovate.

But there is nothing to test!

We can figure it out in the pilot phase/beta.

But we have Analytics?

Research accelerates innovation

Serving unmet needs is essential for innovation. User research finds those needs.

@arasb #uxsg

We cannot find the right number of users.

This is not the right time for research.

We really need these features, we can do the research-y part later.

The procedures are too complex for our project.

We prefer to innovate.

But there is nothing to test!

We can figure it out in the pilot phase/beta.

But we have Analytics?

Your competition is not taking a vacation.

Benchmarking yourself against others guarantees that you remain ahead.

@arasb #uxsg

We cannot find the right number of users.

This is not the right time for research.

We really need these features, we can do the research-y part later.

The procedures are too complex for our project.

We prefer to innovate.

But there is nothing to test!

We can figure it out in the pilot phase/beta.

But we have Analytics?

End states give us only partial information.

Correlation does not mean causation, “why” is just as important as “what.”

@arasb #uxsg

We cannot find the right number of users.

This is not the right time for research.

We really need these features, we can do the research-y part later.

The procedures are too complex for our project.

We prefer to innovate.

But there is nothing to test!

We can figure it out in the pilot phase/beta.

But we have Analytics?

@arasb #uxsg

@arasb #uxsg

@arasb #uxsg

@arasb #uxsg

Quick Do it in less than 10 days

Affordable No big budget planning

Correct Methodologically sound

Together Not over-the-wall

@arasb #uxsg

User Tracks

Benchmarking

Stakeholder Interviews

User TracksUsers leave tracks in data.

Look at these traces to find

potential problems.

@arasb #uxsg

@arasb #uxsg

User Content

• Anonymized form fields

• Complaints• Significant outbound calls

• User comments (app stores, customer sites, social)

Analytics

• Top entry and exit paths• Top referrers• Search keywords

• Bounce rates• Time on page• Session length• Visitor flow/Funnels

Usage Data

• Top features• Collection sizes

• Time of use• Frequency of use

• Device distribution

Prioritize HypothesizeQuick wins

User TracksUsers leave tracks in data.

Look at these traces to find

potential problems.

@arasb #uxsg

What isn’t working? What can we do better?Day 1: Identify the numbers that describe

user behavior, work with the team

to pull that data.

Day 2: Understand the “what”, create

hypotheses, get more data to see

how strong they are.

BenchmarkingA fact-based snapshot of

how the product/service

compares with others.

@arasb #uxsg

MS Office

Google Docs

OpenOffice

@arasb #uxsg

WordProcessor?

Online Collaboration? Forms? Price

$100 & up

Free for personal use

Free

Business• Monthly new customers• Monthly revenue• Monthly profit• Commissions and fees• Main competitors• Risk profile• Cust. Satisfaction scores• Daily digital traffic• Monthly downloads

Brand• Attributes• Segments• Primary colors• Taglines• Aspiration• Last revision

Products offered• Savings• Checking• Loans• Mortgages• Pension plans

Customer channels• Branches• In-branch kiosks• Partner kiosks• ATM• Mini-branches• Web• Mobile app• Wearable app• Companion apps• Call center

Content• Tone of voice• Timeliness• Detail• Accuracy• Authenticity• Content types• Update frequency• Reuse

Visual design• Layout• Colors• Typography• Legibility• Consistency• Photo quality• Responsiveness

@arasb #uxsg

Business• Monthly new customers• Monthly revenue• Monthly profit• Commissions and fees• Main competitors• Risk profile• Cust. Satisfaction scores• Daily digital traffic• Monthly downloads

Brand• Attributes• Segments• Primary colors• Taglines• Aspiration• Last revision

Products offered• Savings• Checking• Loans• Mortgages• Pension plans

Customer channels• Branches• In-branch kiosks• Partner kiosks• ATM• Mini-branches• Web• Mobile app• Wearable app• Companion apps• Call center

Content• Tone of voice• Timeliness• Detail• Accuracy• Authenticity• Content types• Update frequency• Reuse

Visual design• Layout• Colors• Typography• Legibility• Consistency• Photo quality• Responsiveness

@arasb #uxsg

@arasb #uxsg

A* B C D E F

@arasb #uxsg

@arasb #uxsg

Content• Tone of voice• Timeliness• Detail• Accuracy• Authenticity• Content types• Update frequency• Reuse

BenchmarkingA fact-based snapshot of

how the product/service

compares with others.

@arasb #uxsg

How do we honestly compare against others?Day 1: Establish criteria, identify competitors.

Day 2: Divide amongst the team, fill and grade.

Day 3: Fact-checking and cross-ranking.

Use existing data where available.

Ask finance, call center and field support.

@arasb #uxsg

Stakeholder InterviewsMaking sure that you and

the sponsors are hearing

each other correctly.

@arasb #uxsg

Preparation

• Talk with your champion.• Study the org chart.• Inquire about organizational

dynamics.

• Find out about recent successes and failures.

• Pay attention to ego-boosters and sore spots.

• Distill 10-15 questions, as a guide for yourself.

Interviews

• Be yourself.

• Ask questions selectively.• Go in ascending stratified order.

@arasb #uxsg

CEO

GM 1

VP 1 VP 2 VP 3

GM 2

VP 4 VP 5 VP 6

@arasb #uxsg

Preparation

• Talk with your champion.• Study the org chart.• Inquire about organizational

dynamics.

• Find out about recent successes and failures.

• Pay attention to ego-boosters and sore spots.

• Distill 10-15 questions, as a guide for yourself.

Interviews

• Be yourself.

• Ask questions selectively.• Go in ascending stratified order.

• Avoid interrogation and sales.• Avoid masochism, servitude and

cordial hypocrisy.

@arasb #uxsg

What is at stake?Do they know that?Day 1: Prepare for the interview, chat

with your champion.

Day 2: Carry out your interviews.

Day 3: Analyze and prepare a pithy,

crystal-clear executive summary.

Stakeholder InterviewsMaking sure that you and

the sponsors are hearing

each other correctly.

@arasb #uxsg

Field ImmersionUser Tracks

Benchmarking

Stakeholder Interviews

Field ImmersionA shocking, fast-acting

calibration on prejudices

and assumptions.

@arasb #uxsg

@arasb #uxsg

@arasb #uxsg

@arasb #uxsg

@arasb #uxsg

@arasb #uxsg

@arasb #uxsg

@arasb #uxsg

@arasb #uxsg

@arasb #uxsg

@arasb #uxsg

ÇiçekSepeti Headquarters Florist doing the real work

@arasb #uxsg

@arasb #uxsg

Field guide for beginners

• Dress a bit less casual than normal.• Pair if possible.• Think with a beginners’ mind.• Ask and shut up.

@arasb #uxsg

“What do you think of this feature?”“How would you design this?”

“My name is Albert, I built this.”“Wow I didn’t realize it was THIS hard!”

@arasb #uxsg

Focus on Behavior, Not Attitude

“When was the last time you did X?”

“Can you show me how you normally do X?”

“You mentioned X, can you talk a bit about it?”

@arasb #uxsg

Field guide for beginners

• Dress a bit less casual than normal.• Pair if possible.• Think with a beginners’ mind.• Ask and shut up.

• Don’t forget the research question.• Don’t take notes verbatim, listen first.

@arasb #uxsg

@arasb #uxsg

How do customers pay their bills in the branch?

Observation Thoughts Actions

Unsure about which window to go to

Show upcoming tickets in each window

Field guide for beginners

• Dress a bit less casual than normal.• Pair if possible.• Think with a beginners’ mind.• Ask and shut up.

• Don’t forget the research question.• Don’t take notes verbatim, listen first.• Take pictures/video with permission.

@arasb #uxsg

Watch out for

• Desire to help• Confirmation bias• Anchoring effect• Personal agendas

• Organizational dynamics• Sample selection

Field ImmersionA shocking, fast-acting

calibration on prejudices

and assumptions.

@arasb #uxsg

What are they doing? Why are they doing it?Day 1: Identify questions, let your users know

that you are coming for a visit.

Day 2: Visit the site in the morning, complete

and organize notes in the afternoon.

Day 3: Prepare for analysis by going through

your notes and organizing themes.

@arasb #uxsg

Field ImmersionUser Tracks

Benchmarking

Stakeholder Interviews

Snowball Meetings

Narrative Prototypes

Shared Analysis

Get everyone involved in

analysis to get a very rich

understanding.

@arasb #uxsg

Shared Analysis

@arasb #uxsg

@arasb #uxsg

@arasb #uxsg

@arasb #uxsg

@arasb #uxsg

Thou shall change the login flow.

“Ivory Tower” Analysis

• Opaque and arrogant• Hard to comprehend the underlying

reasons• Closed to contribution• Not helping others improve the insights• If wrong, creates a big loss of time

@arasb #uxsg

Shared Analysis

• Open and collaborative• Easy to understand and challenge• If wrong, caught very early

• Team annotations

@arasb #uxsg

How do customers pay their bills in the branch?

Observation Thoughts Actions

Unsure about which window to go to

Show upcoming tickets in each window

“Ivory Tower” Analysis

• Opaque and arrogant• Hard to comprehend the underlying

reasons• Closed to contribution• Not helping others improve the insights• If wrong, creates a big loss of time

@arasb #uxsg

Shared Analysis

• Open and collaborative• Easy to understand and challenge• If wrong, caught very early

• Team annotations• Focused analysis techniques

@arasb #uxsg

SCAMPER

SubstituteCombineAdaptModifyPut to other useEliminateReverse

Observation or ThoughtNot an action or a solution

@arasb #uxsg

SCAMPER

SubstituteCombineAdaptModifyPut to other useEliminateReverse

Use Facebook, Twitter, Google loginMake it a part of checkoutChange the login promptAsk for less info, simplify passwordsAsk for personal info each timeDon’t ask for login, rely on a cookieForce login, make logging out harder

Visitors are not logging in.

@arasb #uxsg

Visitors are not logging in.

Why? / How might we?

Not an action or a solution

@arasb #uxsg

Visitors are not logging in.

They are not seeing the link.

Links are low-contrast b/c

brand guidelines.

Our brand is too bland and boring.

Why?

@arasb #uxsg

Visitors are not logging in.

They are not seeing the link.

Links are low-contrast b/c

brand guidelines.

Our brand is too bland and boring.

They don’t want to give us info.

They don’t trust us.

We share their info with our

partners.

@arasb #uxsg

Visitors are not logging in.

Show the benefits of logging in.

Talk about our loyalty program.

Make a banner near login form.

How might we?

@arasb #uxsg

Why? / How might we?

“Ivory Tower” Analysis

• Opaque and arrogant• Hard to comprehend the underlying

reasons• Close to contribution• Not helping others improve the insights• If wrong, creates a big loss of time

@arasb #uxsg

Shared Analysis

• Open and collaborative• Easy to understand and challenge• If wrong, caught very early

• Team annotations• Focused analysis techniques• Open discussion and consensus

Get everyone involved in

analysis to get a very rich

understanding.

@arasb #uxsg

Shared AnalysisHere’s what we didn’t know.Here’s what we can do!

Day 1: Take time to work on analysis, explore

your data, call your participants if you

need to clarify points.

Day 2: Meet with the entire team, share your

why/HMW work. Discuss in detail,

identify 5-10 areas to work on as a team.

Snowball MeetingsShare findings openly and

let every team absorb as

much as they can.

@arasb #uxsg

Presentation structure

• Always relate back to the original research question!

• Start with the problem• Briefly tell what has been done

• Share 5-8 findings• Show the research material• Be transparent about your process

@arasb #uxsg

Make it impactful and compact

• Group recurring themes.

• Items that confirm you and your team’s assumptions.

• Items that completely destroy the project.

PiezoBankLoan Experience on Mobile

@arasb #uxsg

@arasb #uxsg

High Bounce Rates

Very low Q3 Performance

Increase in Phone Inquiries

Question:How do our customers in mid and premium segments learn about loans on mobile?

@arasb #uxsg

Assumptions

Recruitment

Methods

Questions / Hypotheses

Loan pages are not inviting the users to explore.

@arasb #uxsg

@arasb #uxsg

“I yawn when I look at these”Mid-segment, 34 years old, customer since 1997

Loan pages are not inviting the users to explore.

Possible solutions:

1

2

3

4

@arasb #uxsg

Presentation structure

• Always relate back to the original research question!

• Start with the problem• Briefly tell what has been done

• Share 5-8 findings• Show the research material• Be transparent about your process

@arasb #uxsg

Make it impactful and compact

• Group recurring themes.

• Items that confirm you and your team’s assumptions.

• Items that completely destroy the project.

• Ask for new audiences!

Snowball MeetingsShare findings openly and

let every team absorb as

much as they can.

@arasb #uxsg

Help us talk to all teamsto improve our product.

Day 1: Review your analysis with the team.

Prepare a rough outline, storyboard

your narrative. Use pen/paper/walls.

Day 2: Create your presentation. Include

the data that lead to the research

questions. Compile a strong set of

quotes, videos, photos from the field.

@arasb #uxsg

Narrative PrototypesInfuse solution reviews

with your research process

and insights.

PiezoBankLoan Experience on Mobile

@arasb #uxsg

@arasb #uxsg

@arasb #uxsg

ResearchReport

13-11-2016

(Contains page after page)

@arasb #uxsg

Aras Bilgen

Summary of our research

Perona Iner (BBVA Compliance Manager)

Hi Perona,

Please find our research report attached. You can view the prototype showing our suggestions at the following link.

http://prototype.link/dkJd2ao

Regards,

Aras

User research report 13-11-2016.docx

[Auto-saved]

@arasb #uxsg

@arasb #uxsg

@arasb #uxsg

Narrative PrototypesInfuse solution reviews

with your research process

and insights.

Here’s what we should do, and here is why.

Day 1: Design a few screen, flow and

content improvements. Work in

high-fidelity, if possible.

Day 2: Add the rationale highlights from

your research findings.

@arasb #uxsg

Field ImmersionUser Tracks

Benchmarking

Stakeholder Interviews

Snowball Meetings

Narrative Prototypes

Shared Analysis

Focus groups? Hard to moderate, strong group dynamics,

potential for bias, lack of focus.

Targeting is hard, self-report may

not be accurate.

Surveys?

Recruitment is very hard for a wide user

base, relatively easy for niche services,

enterprise apps.

Interviews?

Getting to the right users is hard, different

motivations, hard to go below UI level.

Guerilla tests?

@arasb #uxsg

Why not…

Why not…

@arasb #uxsg

Remote testing? Too opaque to demonstrate good

research practices, may be hit/miss

based on the tasks and domain.

Can be too isolated and artificial.

Not very easy to ramp up in a week.

Lab studies?

Many potential confounding factors, hard

to get good behavioral insights.

Intercept surveys?

Hard to setup correctly across segments,

takes time for conclusive results.

A/B testing?

Why not…

@arasb #uxsg

Heuristics? No direct exposure to the user. Quality

depends too much on the domain and

the quality of the administering expert.

No users involved. Useful but

artificial way of finding problems.

Task analysis?

@arasb #uxsg

User research is asking deliberate questions about a problem you care about, working with your users methodologically to find out answers and needs, discussing findings with your team to design possible solutions.

@arasb #uxsg Adapted from Christian Rohrer

Attitude

Behavior

Quantitative Qualitative

Focus groups

Interviews

Multivariate testing

Clickstream analysis

Contextual interviews

Usability lab studies

Ethnography

Diary studies

Desirability studies

Eye-tracking

First click tests

Instrumentation

True intent

Off-task surveys

Intercept surveys

Guerilla testing

Remote testing

Noticeability studies

@arasb #uxsg

@arasb #uxsg

We already know what our users want.

Research will delay our project.

We are not scientists.

We are engineers, not researchers.

Research is fairly expensive.

@arasb #uxsg

We cannot find the right number of users.

This is not the right time for research.

We really need these features, we can do the research-y part later.

The procedures are too complex for our project.

We prefer to innovate.

But there is nothing to test!

We can figure it out in the pilot phase/beta.

But we have Analytics?

@arasb #uxsg

Quick Do it in less than 10 days

Affordable No big budget planning

Correct Methodologically sound

Together Not over-the-wall

@arasb #uxsg

Field ImmersionUser Tracks

Benchmarking

Stakeholder Interviews

Snowball Meetings

Narrative Prototypes

Shared Analysis

Find a problem and the questions

Work with users to discover answers

Analyze & discuss with the entire team

Everyone can do user research

Ask deliberate questions about a problem you care about,

Work with your users to find out answers and needs,

Discuss with your team to design possible solutions.

@arasb

/arasbilgen

Keep an open mind, keep your cycles short too.

Accept anyone who you come across this

journey as a fellow team member.

Thank you for your visit!

See my LinkedIn profile and follow me on Twitter for my new presentations.

@arasb

/arasbilgen