Fluxx-The Tested User event-011112-main presentation

© 2012 – All rights reserved Fluxx Ltd.© 2012 – All rights reserved Fluxx Ltd.


© 2012 – All rights reserved Fluxx Ltd.

Why this event?


The 5 ages of User Experience

1. Total ignorance

2. Blind obedience (The Jakob years)

3. User-Centred Design & Usability

4. Let’s go crazy – multi-touch is here!

5. Anything goes, so long as users say it’s ok(AKA nobody gets fired for doing what customers ask you to do)

And most recently:

Let’s test everything. It will make our lives so much easier. We will never have to make a decision again.


What is ‘User Testing’ anyway?

Has it evolved?

Have we evolved our understanding and usage of it?

Is it always the right thing to do?


?

Designed a phone

that inspired a

generation

Re-invented a

familiar household

object for a new era

Pioneered a focus

on user testing

Made the world feel

like a smaller place


Henry Dreyfuss, 1904-1972

Designed a phone

that inspired a

generation

Re-invented a

familiar household


Pioneered a focus

on user testing

Made the world feel



Henry Dreyfuss, 1904-1972

Designed a phone

that inspired a

generation

Re-invented a

familiar household


Pioneered a focus

on user testing

Made the world feel




“If I’d asked

people what they wanted, they

would have said faster horses”

Henry Ford


So our questions for tonight are:

• When should I place emphasis on customers’ opinions, and

when should I put down the shutters and just show them

when we’re ready?

• What sorts of means do we have at our disposal for

establishing what customers will and won’t do?

• What is the future of validating our approaches with

customers?


Rajiv Arjan

City

Interaction Lab

Gavin Gordon

Mintel UK

Paul Dawson

Fluxx


Importance of user

testing

Rajiv Arjan

City Interaction Lab

Importance of user testing Raj Arjan, City Interaction Lab

Agenda

• Introduction of City Interaction Lab and Me

• User Centered design process

• Usability testing process

• Usability testing enhancements

• Benefits of user testing• Questions

@rajarjan

About Me

• Managing the interaction lab since 2008.

• Responsible for;

– Business development

– Managing UX projects

– Supporting Academics/Students

– Promoting Interaction Lab and HCID

@rajarjan

City University London

• Courses Taught

– MSc Human Centred Systems

– MSc Creativity Innovation and Leadership

– Can be undertaken Part time/ Full time

@rajarjan


• Facility used for student and commercial projects in areas of;– Usability

– Accessibility

– Interaction design

– User research

@rajarjan

User Centered Design

Source : kitdigital - http://www.kitd.com/what-we-do/services/design/@rajarjan

Usability Testing Inputs

Wireframe Live

Site

Paper PrototypePowerpoint/Keyn

ote@rajarjan

Usability Testing Process

Morae Usability Testing Apps

Tobii Studio – eye tracking

Silverback - for MAC OS

@rajarjan

Usability Testing Outputs

• Usability problems

• User opinions

• Task completion times

• Other Metrics (eye tracking, mouse clicks,

key presses)

@rajarjan

Paying attention to what users do

@rajarjan

“If I had asked

people what they

wanted, they would

have said faster

horses.”

@rajarjan

@rajarjan

Paying attention to how users react

@rajarjan


• Movement

• Skin Conductance

• Temperature@rajarjan


Opinions

Observations

Emotional

Response

@rajarjan

Take Away’s

• User Testing is always important

• Observing what users do is often more important than what they say.

• The user is not the designer.

• Maximise the value of user testing by asking the right questions.

• What are you trying to test? The interface or the concept?

@rajarjan


Thank you

Rajiv Arjan


@rajarjan


Facts, not opinions

Paul Dawson

Fluxx Labs

labs

Fluxx Labs is our methodical approach to validating new product and service propositions, using lean

startup techniques.

labs

Why?

In 15 years of designing and delivering digital products

big and small, we’ve seen a lot of great ideas and

realised a few things.

In 15 years of designing and delivering digital products

big and small, we’ve seen a lot of great ideas and

realised a few things.


1 : BIG BETS

Don’t always pay offDon’t always pay off

labs

2 : There is no shortage of big

ideas in our client base

labs

3 : Some people are pretty

opinionated

And they often spell the end to a strong conceptAnd they often spell the end to a strong concept

labs

http://www.youtube.com/watch?v=E1IarxtP4aAhttp://www.youtube.com/watch?v=E1IarxtP4aA

labs

labs


insight idea feasibility

design &

developmentmanufacture distribution


man

ufac

turedesign &

development

feasibility

Idea

d

i

s

t

r

i

b

u

t

i

o

n

insight

Easy to do, and no risk Hard to do, and a lot of riskStarting to invest time & money


feasibility

Idea

insight

Easy to do, and no risk Hard to do, and a lot of riskStarting to invest time & money

man

ufac

turedesign &

development

d

i

s

t

r

i

b

u

t

i

o

n

labs

labs

The MVP is at the heart of Fluxx Labs

In ‘Lean Startup’ land, the fastest route to getting facts is to

launch a ‘minimum viable product’

Q: What is the absolute minimum we can do to see if people

actually do value our product?

In ‘Lean Startup’ land, the fastest route to getting facts is to

launch a ‘minimum viable product’

Q: What is the absolute minimum we can do to see if people

actually do value our product?


distribution

Dim and distant past

insight

concept

feasibility

design &

development

manufacture

Painful, but we got there…. Dragon’s Den

Give it

to the

press

Bankrupt

factory

Dodgy

catches

People

bought

it

Licensee

liquidated

labs

labs

He believed in it so much

because he generated facts by

putting his product into the

hands of potential customers

The toughness of his journey

was testament to the fact that his

MVP could not be that minimal

He believed in it so much

because he generated facts by

putting his product into the

hands of potential customers

The toughness of his journey

was testament to the fact that his

MVP could not be that minimal

labs

Fortunately…

We live in a world where you can make anything seem

real, to make the learnings as real as they can be.

We set Fluxx Labs up to generate evidence-based

learning, to help our clients get their ideas right and

make good decisions based on facts.

We live in a world where you can make anything seem

real, to make the learnings as real as they can be.

We set Fluxx Labs up to generate evidence-based

learning, to help our clients get their ideas right and

make good decisions based on facts.

labs

labs

Basic Insight

We find it really hard to f ind a whiteboard the

size we want and think that other people will

have the same problem.

labs

Insight leads to a value hypothesis that we can begin to test

“We think that people will value a premium product that makes and installs whiteboards to a precise set of dimensions.”

labs

1. Make a site that

calculates the cost and puts customers in touch

with us.

2. Get a phone!

If people value this

service as much as we do.

• Volume of people

searching for it on Google• Requests for pricing

• People who buy

labs

The first MVP

(2 days)

labs

The 2nd MVP

labs

The Lab’s Learnings

This is a viable, small, local

business.

It would be a good idea to make

the whiteboards erasable!

It is possible in a week to prove

out a digital & physical

business in a lean way.

This is a viable, small, local

business.

It would be a good idea to make

the whiteboards erasable!

It is possible in a week to prove

out a digital & physical

business in a lean way.

labs

The High Street Bank

A real life case studyA real life case study

labs

Amongst others, we created one

experiment designed to test

something that had been debated

for ages at the bank

labs

It was a thorny subject.

It involved using customers’

personal data to do something we

thought they would like and value.

labs

But would they accept it, or would

they go nuts?

We put it l ive in a few weeks in an MVP to

several thousand real customers who didn’t

know it was just an experiment.

labs

But before we did…

labs

It was usability tested

It panned!

Only 24% of people said they would do it

labs

How many ACTUALLY did it in

real life?

96% of people just jumped right on in!


“What people say they will do and

what they actually do is often

very different and can be greatly

affected by their level of

awareness of the task at hand”


Thank you, and see you in the Lab

Paul Dawson - @poleydee

[email protected]


Mintel’s approach to

research

Gavin Gordon

Mintel

mintel.com67

A typical Mintel Oxygen report contains…

Market Drivers Size of the Market and Forecast

Brands, Companies and Innovations

Advertising and Marketing Activity

Distribution Trends Consumer Insight

mintel.com68

Desk research

Company reports

Press releases

Media reports

Legislative and regulatory developments

Statistical forecasting

Mintel’s market size database

Other data ONS, Treasury, Bank of England etc.

Trade research

Trade input – formal and informal

Industry events

Consumer research

Mintel’s own consumer research

TGI data

Qualitative research

Trade

Understanding a market

Mintel and analyst experience

Different elements of research

mintel.com69

Desk research

• Press reports – trade publications and national media

• Regulatory/legislative developments

• Gathering and analysing reliable and trusted data sources

• Typical sources include ONS, Ofcom, trade associations and other industry data

• Company reports and press releases

mintel.com70

Trade research

• Adds relevance and understanding, as well as a source of data

• Trade interviewing and attending industry events

• How is the market performing? Checking market size and our forecasts.

• What challenges and issues is the market facing?

• Future trends, innovations, possible developments

• What consumer insight do industry players want?

mintel.com71

Forecasting – Statistical modelling

Analysis of relationships between, Mintel market sizes and a selection of

key economic and demographic factors to

identify those which have the most influence on the

market.

Mintel’s own market size database is supplemented

by macro- and socio-economic data sourced

from organisations such as the Office for National

Statistics, HM Treasury and the Bank of England.

5-year forecasts based on an advanced statistical

technique known as ‘multivariate time series

auto-regression’ using the statistical software

package SPSS

mintel.com72

Forecasting – Tackling uncertainty

The fan chart illustrates the probability of various outcomes for the market value/volume over the next five years.

At a 95% confidence interval, we are saying that 95 out of 100 times, the forecast will fall within these outer limits.

The Mintel fan chart

Equity Release Schemes, 2011-16

mintel.com73

Forecasting - Qualitative insight

Forecast of games consoles and games software sales by value, 2012-17

(£m)1,651

Best case (£m)

3,792

Worst case (£m)

2,074

Mintel forecast (£m)2,811

0

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017

Mark

et

valu

e (

£m

)

95%

Confidence intervals

90%

70%

50%

Actual Forecast

Est.

0

Wii U

Sony/ Microsoft next gen

Slimline, cheaper models

mintel.com74

Consumer research planning

• Deciding what to cover – insight from desk and trade research

• Questionnaire design – collaborate with Research Services Team

• Looking at ways to provide interesting analysis

• Qualitative consumer research

mintel.com75

Qualboard:

1. To add context to rigorous quant data2. To point to emergent trends3. To provoke additional quant analysis

mintel.com76

Face-to-faceOnline Specialist panel

Consumer research planning

• Internet users

• Increasingly popular form of research

• More detailed exploration of issues

• Nationally representative

• Reaching those groups not engaged with the internet

• Annuities / Equity Release Schemes

• Travel Agents

• IFA research

• HNW groups (Affluent Banking or SIPPs)

• Youth surveys

mintel.com77

Desk research

Company reports

Press releases

Media reports

Legislative and regulatory developments

Statistical Forecasting

Mintel’s market size database

Other data ONS, Treasury, Bank of England etc.

Consumer research

Mintel’s own consumer research

TGI data

Qualitative research

Trade research

Trade input – formal and informal

Industry events

Understanding a market

Mintel and analyst experience

Bringing it all together

mintel.com78

Transforming information into actionable insight: What makes it work

• Understanding/evolving the role of syndicated research

• Faster

• More accessible

• More consultative

• More disruptive.

mintel.com79

“The reports were too out-of-date

- I wanted something more

current and relevant”

Updating our Model

“To ensure Mintel remains timely”

� Move away from the 2-year publishing cycle?

� Modified approach to report writing and publishing?

� Analyst updates (market data/company information/consumer)

� Auto updates (background data and existing databases)

� Consistency with US?

“There's a real lag between

some reports being updated”

“Found relevant report

easily but not

recent enough”

mintel.com80

Communication“Building relationships and tapping into client knowledge”

� Making clients one of the first ports of call:

� Modifications to the current system?

� Developing the analyst’s role to become more than a picture on the website

� Listening to clients on a regular, on-going basis

� Pro-active development led by research.“More consultation

with clients

for their specific needs”

‘I didn’t realise I

could talk to the analysts’

– 46%

mintel.com81

Consistent and Insightful“Usage is high, but frequency is not high enough”

� Driving frequent usage up:

� Grounding Inspire in more ‘facts’ and making What’s Hot to be more than just a visual treat

� We’re guilty of ‘hiding away’ insight (reports too long, too much text)

� More flexible approach to report scheduling (hot topic reports)

� Consistent approach across Oxygen (both within UK and UK/US).

“More raw data

where possible.

Let me make decisions!”

“Opinions/viewpoints always

interesting/thought provoking”

“Keep the same level of detail

and analysis but just make it

clearer and easier to find

what you are looking for”

mintel.com82

3.1

20.1

21.2

30.2

40.4

51.5

63.0

67.6

Other

When I cannot find data elsewhere

When somebody recommends a report internally

When I need a quick answer

For a strategy meeting

To prep for a client meeting/ presentations

As evidence to back up our ideas

When I have been given a brief for a project

Base: 646 Mintel Oxygen clients

Thinking about your needs, why do you typically use Mintel?


Thank you

Gavin Gordon

Mintel



Is the User

aware of the test?

Product/Service

Maturity Level

Unaware Aware

Idea/Concept

Partial/Fully

Developed

The fluxx ‘Tested User’ Framework


What are people’s opinions,

expectations and ideas?

How well does our

user interface

support our product?

Which is the best way to

implement this?

Will people find the

product/service of value?

Market

Intelligence

Unaware Aware

Product/Service

Maturity Level

Idea/Concept

Partial/Fully

Developed

Is the User

aware of the test?



Market Research

companies

Usability Testing

e.g. City Interaction Lab

Product Development

e.g. Fluxx

MVP/Lean

e.g. Fluxx Labs

Market Intelligence

e.g. Mintel

Unaware Aware

Product/Service

Maturity Level

Idea/Concept

Partial/Fully

Developed

Is the User

aware of the test?



Questions Trying to Answer:

Will people find the product of value?

When to use:

When there is a perceived barrier or high degree of uncertainty about willingness of consumers to adopt

Where the product paradigm is substantially different from the norm

Technique(s) Used:

Minimum Viable Product (MVP)

Lab-like

validation

Where product is at an early concept stage, and users

are unaware of the test itself.



What are people’s opinions, expectations and ideas?

When to use:

When there is a wide range of possible concepts that you want to understand more

Where you want early stage insight into how specific consumer groups behave and think

You want consumer ideas and input

Technique(s) Used:

Focus Groups, Market Research

Opinions

& ideas

An early stage concept, and users are aware they are

being asked for their opinions and input


Question Trying to Answer:

Which is the optimal way to implement this?

When to use:

Where there are clearly defined potential solutions with no strong supporting evidence as to which is the best approach

Technique(s) Used:

A/B Testing, Multi-variate testing (MVT)

Where product is at a later stage, but you don’t make

consumers aware that they are part of a test

A/B

& MVT



How well does our user interface support our product? Can we make it better?

When to use:

When you are clear on your proposed design solution and want to uncover any potential barriers to adoption

Technique(s) Used:

Usability Testing, prototype testing, eye tracking

User

Testing

Where product is well developed and people know they

are being asked to test it


Questions Trying to Answer:Is this the right thing to do?

Is there a market for what we want to do?

When to use:When you need clarity around target segments or market

To provide external evidence of key adoption assumptions

To inform and validate the business case

Technique(s) Used:Surveys, Polls, Intelligence Gathering

Market

Intelligence

Intelligence and data on consumer activities, behaviours and

opinions in specific sectors, industries and markets


Appendix

of common

Testing Techniques


Fluxx Rapid Start

• A multi-day multi-disciplinary hugely intensive exercise unique to Fluxx, where business teams are taken from early stage ideas to launchable products ready for circulating to consumers

• Typically this is done in two days and key side effects include– Rebuilding teams’ belief in their ability to deliver

– Promotes collaborative design and ways of working

– Incorporating the voice of the customer into the build process

– Most rapid way to get something into the hands of customers


Focus Groups

• Definition:

– A focus group is a form of qualitative research in which a group of

people are asked about their perceptions, opinions, beliefs, and

attitudes towards a product, service, concept, advertisement, idea, or

packaging. Questions are asked in an interactive group setting where

participants are free to talk with other group members.


Usability Testing

• Definition:

– Usability testing is a technique used in user-centered interaction

design to evaluate a product by testing it on users. This can be seen as

an irreplaceable usability practice, since it gives direct input on how

real users use the system.


Ethnographic Research

Ethnography is technically about observing a particular society or culture through a passive participation in it, and without impacting on the society itself, and this is still the strictest definition of it.

However, nowadays, it is more widely used to categorise techniques that are about trying to understand how people live their lives. Typically this involves visiting people in their home or work environments using a series of highly practical questions designed to uncover what is happening and why, Their goal is to see people’s behaviour on their terms, not in an artificial way.

The latter approach is often confused or used interfchangeably to mean what is known as a ‘contextual inquiry’ or ‘contextual analysis’ , the definition of which is observivng someone’s interactions with your product in their usual context or surroundings.


A/B Testing

• A/B testing or split testing compares the effectiveness of two

versions of a product/service in order to discover which has

better response rate or better sales conversion rate

• Two alternate designs or approaches are randomly served to

people, and the resultant outcomes of those differently

served customer bases compared to each other, and a control

group.


Heuristic Evaluation

• A Heuristic Evaluation, or Usability Audit, is an evaluation of an interface by one or more Human Factors experts. Evaluators measure the usability, efficiency, and effectiveness of the interface based on 10 usability heuristics.

• There are many heuristics available, but a popular one was created by Jakob Nielsen in 1994. Many argue that this heuristic is now outdated by modern interaction techniques, and takes no account of user attitudes.


‘Depths’

• Structured, one-on-one interviews help researchers learn

about users’ attitudes and beliefs surrounding a website or

application and specific tasks that the website or application

supports. In order to prevent the researcher from introducing

bias and to ensure that every participant is asked the same

questions, the researcher prepares a Discussion Guide,

which is a list of questions that will be asked of participants

in a particular order.


Card Sorting

• Card sorting is a useful tool to determine how users

categorize the information that will appear on a website. The

name “card sort” comes from the original exercise, which

used index cards with a word or phrase written on one side,

and in some cases, a definition or additional information on

the reverse side


Electroencephalography (EEG)

• EEG involves measuring electrical activity in different parts of

the brain in response to certain stimuli

• It can now used for the assessment of user experiences to

see how people react at an emotional level to stimulus or a

particular experience

• Detailed and specific design recommendations can be made

bbbby recording reactions at different stages of interaction with

a product/service


Minimum Viable Product (MVP)

• Minimum Viable Product is a strategy used for fast and

quantitative market testing of a product or product feature

• The product is typically deployed to a subset of possible

customers, such as early adopters that are thought to be

more forgiving, more likely to give feedback, and able to

grasp a product vision from an early prototype or marketing

information

• This is the technique employed by the Fluxx Labs method


Multi-variate testing (MVT)

• Multivariate testing is a technique for testing hypotheses on

complex multi-variable systems, especially used in testing

market perceptions

• It is a process by which more than one component of a

product/service may be tested in a live environment

• When automated it can be very powerful in finding the

optimal combination of multiple component parts


Prototype Testing

• Prototypes can be tested for aspects like design flaws and

ease of use, two things that are critical if your product is

going to be a success

• You need to make sure everything works the way it should -

and that your customers can figure out how to make it work

too


Eye Tracking

• Eye tracking measures the point of gaze and eye movement

• It can detect the direction in which users are looking and the

duration for which they are focusing on a specific part of a

product e.g. web page

• This is helpful knowledge if you are trying to find out if users

are actively focusing on particular content on your page or if

they miss it all together


Quantitative Market Research

Market research is the systematic gathering and interpretation of information about individuals or organisations using the statistical and analytical methods and techniques of the applied sciences to gain insight or support decision making

For example:

• Mass participation surveys– Usually aimed at specifically targeted segments. This type of research relies on volume of

participants to give its results significance. Questions usually have binary or multiple choice answers.

• Omnibus Survey: – A regular survey designed to determine what the general population thinks on a range of topics and

issues. Brands can pay for questions on this survey alongside the other questions. Thought to have derived from the phrase “The Man on the Clapham Omnibus’ which was used as a euphemism for ‘the common man’.


Leading global market intelligence company with more than 500 employees worldwide

Guide for leaders in marketing, product development and research for more than 35

years

Clients rely on our authoritative offering to drive strategic decisions

About MintelAbout MintelAbout MintelAbout Mintel

What we do

Provide fresh insight into your marketplace and across industries

Identify opportunities for profit and growth

Formulate insights into consumer behavior, product innovation and competitive

marketing strategies

Create high-quality data, evocative analysis and tailored consulting


“What people say they will do and

what they actually do is often

very different and can be greatly

affected by their level of

awareness of the task at hand”


Documents

Fluxx-The Tested User event-011112-main presentation