Introducing a New UX Maturity Metric: Team Engagement ......#UXPA2017 Download our mobile app UXPA.org/2017app to fill out our conference survey! Introducing a New UX Maturity Metric:

#UXPA2017

www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!

Introducing a New UX Maturity Metric: Team Engagement Score

(TES) During Usability Testing

Bill Albert

Executive Director, Bentley University User Experience Center

Josh Rosenberg

User Experience Researcher, DonorsChoose.org

June 6, 2017

@BentleyUXC | @UXMetrics

#UXPA2017


Two Short Stories


#UXPA2017


Why Team Engagement Matters

• Adoption of UX recommendations

• Efficiency in the design process and product development

• Culture of UX throughout the organization

• Sparking innovation

• Value to co-workers


#UXPA2017


Our Motivation

• Formalize our thinking through development of a team engagement metric

• Launch discussions internally – explore the value of a team engagement metric

• Share with others in the UX community

• Promote team engagement

#UXPA2017


TES: Team Engagement Score

TES = Observation + Collaboration + Behavior

• Observation = Observing usability sessions

• Collaboration = Being part of the team during testing

• Behavior = Being actively engaged when observing


#UXPA2017


Component 1: Observation


# Actual Observed Sessions

# Total Observable Sessions

#UXPA2017


25 Actual Observed Sessions100 Total Observable Sessions

Component 1: Observation


0.25=

#UXPA2017


Component 2: Collaboration

# Feasible In-Person Observations

# In-Person Observations

#UXPA2017


Component 2: Collaboration

2 In-Person Observers

5 Possible Observers

= 0.40

#UXPA2017


Component 3: Behavior


10

1-10 Rating

#UXPA2017




Took Notes

Follow-up Questions

Paid Attention

Brainstormed Solutions

Etc.

Rating: 8

✓

✓

✓

Rating: 8

10

= 0.80

#UXPA2017




Component 3: B


• Attention (0 -2)• Low = meetings, multi-tasking; High = undivided attention

• Conversation (0-2)• Low = none; High = creative brainstorming based on insights

• Notes (0-2)• Low = no notes; High = took detailed notes

• Attitude (0-2)• Low = discredits participants; High = empathizes with participants

• Questions (0-2)• Low = no questions; High = a lot of questions

#UXPA2017


Calculating TES

• TES = Observation + Collaboration + BehaviorTES = ((# actual observations/ # total possible observations) +

(# in-person observers / # feasible in-person) +

(engagement score/10)) / 3

• Average the three numbers• TES = (0.25 + 0.4 + 0.8) = 1.45, divided by 3 = 0.48

• TES runs from 0 – 1 (higher the better)

• All three dimensions weighted equally, but can apply different weights

• TES works for large or small teams


#UXPA2017


Scenario 1: High Engagement


• Observation = 10 sessions (2 days), 8 attend all 10 sessions, 2 attend the first day only (5) = 90/100 = 0.90

• Collaboration = 5 people local, all attend in person = 5/5 = 1.0

• Behavior = highly engaged = 9/10 = 0.9

• TES = 0.9 + 1.0 + 0.9 = 2.8/3.0 = .93

#UXPA2017


Scenario 2: Low Engagement


• Observation = 10 sessions (2 days), 4 attend day 1 only, 6 don’t attend any sessions = 20/100 = 0.20

• Collaboration = 5 people local, 2 attend in person = 2/5 = 0.4

• Behavior = low engagement = 3/10 = 0.3

• TES = 0.2 + 0.4 + 0.3 = 0.9/3.0 = .30

#UXPA2017


Scenario 3: Remote Only Studies


• Observation = 10 sessions (2 days) with 10 invited - 3 attend both days, 5 attend day 1 only, 2 don’t attend any sessions = 45/100 = 0.45

• Collaboration = all remote, so this component is not considered

• Behavior = moderate engagement = 6/10 = 0.6

• TES = 0.45 + 0.6 = 1.05/2.0 = .525

#UXPA2017


Changing Weights


• Change weights if you value one component more than another

• Observation = 0.5 x 2 = 1.0 (weighted twice)

• Collaboration = 0.4 x 0.5 = 0.2 (weighted half)

• Behavior = moderate engagement = 0.6 (no change)

• TES = 1.0 + 0.2 + 0.6 = 1.8/3.0 = 0.6

#UXPA2017


Interpreting TES


0.1 0.2 0.3 0.4 0.60.5 0.7 0.90.8

Big Problem Fully EngagedOpportunity to ImproveWork to Do

Take it Seriously

#UXPA2017


Improving TES

• Formalize the process

• Bake into design schedule

• Senior-level support

• Correlate metrics to individual/team performance reviews

• Free food is no longer viable!


#UXPA2017


How Would You Measure Team Engagement?


#UXPA2017


Value of TES

• Easy to collect and measure

• Track team engagement over time

• Report with other UX metrics

• Roll into larger UX maturity score

• Is this something you would use?


#UXPA2017


Limitations of TES

• Not a validated metric, no normative data

• Engagement is more than participating in usability testing

• Not all observers are the same, not all sessions are the same

• Potential political ramifications

• Other dimensions of engagement are hard to measure


#UXPA2017


Delivering Value


#UXPA2017


Questions?

Bill Albert

Executive Director

Bentley University User Experience Center

[email protected]

www.bentley.edu/uxc


http://www.bentley.edu/uxc

Documents

Introducing a New UX Maturity Metric: Team Engagement ......#UXPA2017 Download our mobile app UXPA.org/2017app to fill out our conference survey! Introducing a New UX Maturity Metric: