View
110
Download
0
Category
Tags:
Preview:
DESCRIPTION
The main presentation from our "The Tested User" event held at Fluxx Studios on 1st November 2012.
Citation preview
© 2012 – All rights reserved Fluxx Ltd.© 2012 – All rights reserved Fluxx Ltd.
© 2012 – All rights reserved Fluxx Ltd.© 2012 – All rights reserved Fluxx Ltd.
© 2012 – All rights reserved Fluxx Ltd.
Why this event?
© 2012 – All rights reserved Fluxx Ltd.
The 5 ages of User Experience
1. Total ignorance
2. Blind obedience (The Jakob years)
3. User-Centred Design & Usability
4. Let’s go crazy – multi-touch is here!
5. Anything goes, so long as users say it’s ok(AKA nobody gets fired for doing what customers ask you to do)
And most recently:
Let’s test everything. It will make our lives so much easier. We will never have to make a decision again.
© 2012 – All rights reserved Fluxx Ltd.
What is ‘User Testing’ anyway?
Has it evolved?
Have we evolved our understanding and usage of it?
Is it always the right thing to do?
© 2012 – All rights reserved Fluxx Ltd.
?
Designed a phone
that inspired a
generation
Re-invented a
familiar household
object for a new era
Pioneered a focus
on user testing
Made the world feel
like a smaller place
© 2012 – All rights reserved Fluxx Ltd.
Henry Dreyfuss, 1904-1972
Designed a phone
that inspired a
generation
Re-invented a
familiar household
object for a new era
Pioneered a focus
on user testing
Made the world feel
like a smaller place
© 2012 – All rights reserved Fluxx Ltd.
Henry Dreyfuss, 1904-1972
Designed a phone
that inspired a
generation
Re-invented a
familiar household
object for a new era
Pioneered a focus
on user testing
Made the world feel
like a smaller place
© 2012 – All rights reserved Fluxx Ltd.
© 2012 – All rights reserved Fluxx Ltd.
“If I’d asked
people what they wanted, they
would have said faster horses”
Henry Ford
© 2012 – All rights reserved Fluxx Ltd.
So our questions for tonight are:
• When should I place emphasis on customers’ opinions, and
when should I put down the shutters and just show them
when we’re ready?
• What sorts of means do we have at our disposal for
establishing what customers will and won’t do?
• What is the future of validating our approaches with
customers?
© 2012 – All rights reserved Fluxx Ltd.© 2012 – All rights reserved Fluxx Ltd.
Rajiv Arjan
City
Interaction Lab
Gavin Gordon
Mintel UK
Paul Dawson
Fluxx
© 2012 – All rights reserved Fluxx Ltd.© 2012 – All rights reserved Fluxx Ltd.
Importance of user
testing
Rajiv Arjan
City Interaction Lab
Importance of user testing Raj Arjan, City Interaction Lab
Agenda
• Introduction of City Interaction Lab and Me
• User Centered design process
• Usability testing process
• Usability testing enhancements
• Benefits of user testing• Questions
@rajarjan
About Me
• Managing the interaction lab since 2008.
• Responsible for;
– Business development
– Managing UX projects
– Supporting Academics/Students
– Promoting Interaction Lab and HCID
@rajarjan
City University London
• Courses Taught
– MSc Human Centred Systems
– MSc Creativity Innovation and Leadership
– Can be undertaken Part time/ Full time
@rajarjan
City Interaction Lab
• Facility used for student and commercial projects in areas of;– Usability
– Accessibility
– Interaction design
– User research
@rajarjan
User Centered Design
Source : kitdigital - http://www.kitd.com/what-we-do/services/design/@rajarjan
Usability Testing Inputs
Wireframe Live
Site
Paper PrototypePowerpoint/Keyn
ote@rajarjan
Usability Testing Process
Morae Usability Testing Apps
Tobii Studio – eye tracking
Silverback - for MAC OS
@rajarjan
Usability Testing Outputs
• Usability problems
• User opinions
• Task completion times
• Other Metrics (eye tracking, mouse clicks,
key presses)
@rajarjan
Paying attention to what users do
@rajarjan
“If I had asked
people what they
wanted, they would
have said faster
horses.”
@rajarjan
@rajarjan
Paying attention to how users react
@rajarjan
Paying attention to how users react
• Movement
• Skin Conductance
• Temperature@rajarjan
Paying attention to how users react
Opinions
Observations
Emotional
Response
@rajarjan
Take Away’s
• User Testing is always important
• Observing what users do is often more important than what they say.
• The user is not the designer.
• Maximise the value of user testing by asking the right questions.
• What are you trying to test? The interface or the concept?
@rajarjan
© 2012 – All rights reserved Fluxx Ltd.© 2012 – All rights reserved Fluxx Ltd.
Thank you
Rajiv Arjan
City Interaction Lab
@rajarjan
© 2012 – All rights reserved Fluxx Ltd.© 2012 – All rights reserved Fluxx Ltd.
Facts, not opinions
Paul Dawson
Fluxx Labs
labs
Fluxx Labs is our methodical approach to validating new product and service propositions, using lean
startup techniques.
labs
Why?
In 15 years of designing and delivering digital products
big and small, we’ve seen a lot of great ideas and
realised a few things.
In 15 years of designing and delivering digital products
big and small, we’ve seen a lot of great ideas and
realised a few things.
© 2012 – All rights reserved Fluxx Ltd.
1 : BIG BETS
Don’t always pay offDon’t always pay off
labs
2 : There is no shortage of big
ideas in our client base
labs
3 : Some people are pretty
opinionated
And they often spell the end to a strong conceptAnd they often spell the end to a strong concept
labs
http://www.youtube.com/watch?v=E1IarxtP4aAhttp://www.youtube.com/watch?v=E1IarxtP4aA
labs
labs
© 2012 – All rights reserved Fluxx Ltd.
insight idea feasibility
design &
developmentmanufacture distribution
© 2012 – All rights reserved Fluxx Ltd.
man
ufac
turedesign &
development
feasibility
Idea
d
i
s
t
r
i
b
u
t
i
o
n
insight
Easy to do, and no risk Hard to do, and a lot of riskStarting to invest time & money
© 2012 – All rights reserved Fluxx Ltd.
feasibility
Idea
insight
Easy to do, and no risk Hard to do, and a lot of riskStarting to invest time & money
man
ufac
turedesign &
development
d
i
s
t
r
i
b
u
t
i
o
n
labs
labs
The MVP is at the heart of Fluxx Labs
In ‘Lean Startup’ land, the fastest route to getting facts is to
launch a ‘minimum viable product’
Q: What is the absolute minimum we can do to see if people
actually do value our product?
In ‘Lean Startup’ land, the fastest route to getting facts is to
launch a ‘minimum viable product’
Q: What is the absolute minimum we can do to see if people
actually do value our product?
© 2012 – All rights reserved Fluxx Ltd.
distribution
Dim and distant past
insight
concept
feasibility
design &
development
manufacture
Painful, but we got there…. Dragon’s Den
Give it
to the
press
Bankrupt
factory
Dodgy
catches
People
bought
it
Licensee
liquidated
labs
labs
He believed in it so much
because he generated facts by
putting his product into the
hands of potential customers
The toughness of his journey
was testament to the fact that his
MVP could not be that minimal
He believed in it so much
because he generated facts by
putting his product into the
hands of potential customers
The toughness of his journey
was testament to the fact that his
MVP could not be that minimal
labs
Fortunately…
We live in a world where you can make anything seem
real, to make the learnings as real as they can be.
We set Fluxx Labs up to generate evidence-based
learning, to help our clients get their ideas right and
make good decisions based on facts.
We live in a world where you can make anything seem
real, to make the learnings as real as they can be.
We set Fluxx Labs up to generate evidence-based
learning, to help our clients get their ideas right and
make good decisions based on facts.
labs
labs
Basic Insight
We find it really hard to f ind a whiteboard the
size we want and think that other people will
have the same problem.
labs
Insight leads to a value hypothesis that we can begin to test
“We think that people will value a premium product that makes and installs whiteboards to a precise set of dimensions.”
labs
1. Make a site that
calculates the cost and puts customers in touch
with us.
2. Get a phone!
If people value this
service as much as we do.
• Volume of people
searching for it on Google• Requests for pricing
• People who buy
labs
The first MVP
(2 days)
labs
The 2nd MVP
labs
The Lab’s Learnings
This is a viable, small, local
business.
It would be a good idea to make
the whiteboards erasable!
It is possible in a week to prove
out a digital & physical
business in a lean way.
This is a viable, small, local
business.
It would be a good idea to make
the whiteboards erasable!
It is possible in a week to prove
out a digital & physical
business in a lean way.
labs
The High Street Bank
A real life case studyA real life case study
labs
Amongst others, we created one
experiment designed to test
something that had been debated
for ages at the bank
labs
It was a thorny subject.
It involved using customers’
personal data to do something we
thought they would like and value.
labs
But would they accept it, or would
they go nuts?
We put it l ive in a few weeks in an MVP to
several thousand real customers who didn’t
know it was just an experiment.
labs
But before we did…
labs
It was usability tested
It panned!
Only 24% of people said they would do it
labs
How many ACTUALLY did it in
real life?
96% of people just jumped right on in!
© 2012 – All rights reserved Fluxx Ltd.© 2012 – All rights reserved Fluxx Ltd.
“What people say they will do and
what they actually do is often
very different and can be greatly
affected by their level of
awareness of the task at hand”
© 2012 – All rights reserved Fluxx Ltd.© 2012 – All rights reserved Fluxx Ltd.
Thank you, and see you in the Lab
Paul Dawson - @poleydee
embrace@fluxx.uk.com
© 2012 – All rights reserved Fluxx Ltd.© 2012 – All rights reserved Fluxx Ltd.
Mintel’s approach to
research
Gavin Gordon
Mintel
mintel.com67
A typical Mintel Oxygen report contains…
Market Drivers Size of the Market and Forecast
Brands, Companies and Innovations
Advertising and Marketing Activity
Distribution Trends Consumer Insight
mintel.com68
Desk research
Company reports
Press releases
Media reports
Legislative and regulatory developments
Statistical forecasting
Mintel’s market size database
Other data ONS, Treasury, Bank of England etc.
Trade research
Trade input – formal and informal
Industry events
Consumer research
Mintel’s own consumer research
TGI data
Qualitative research
Trade
Understanding a market
Mintel and analyst experience
Different elements of research
mintel.com69
Desk research
• Press reports – trade publications and national media
• Regulatory/legislative developments
• Gathering and analysing reliable and trusted data sources
• Typical sources include ONS, Ofcom, trade associations and other industry data
• Company reports and press releases
mintel.com70
Trade research
• Adds relevance and understanding, as well as a source of data
• Trade interviewing and attending industry events
• How is the market performing? Checking market size and our forecasts.
• What challenges and issues is the market facing?
• Future trends, innovations, possible developments
• What consumer insight do industry players want?
mintel.com71
Forecasting – Statistical modelling
Analysis of relationships between, Mintel market sizes and a selection of
key economic and demographic factors to
identify those which have the most influence on the
market.
Mintel’s own market size database is supplemented
by macro- and socio-economic data sourced
from organisations such as the Office for National
Statistics, HM Treasury and the Bank of England.
5-year forecasts based on an advanced statistical
technique known as ‘multivariate time series
auto-regression’ using the statistical software
package SPSS
mintel.com72
Forecasting – Tackling uncertainty
The fan chart illustrates the probability of various outcomes for the market value/volume over the next five years.
At a 95% confidence interval, we are saying that 95 out of 100 times, the forecast will fall within these outer limits.
The Mintel fan chart
Equity Release Schemes, 2011-16
mintel.com73
Forecasting - Qualitative insight
Forecast of games consoles and games software sales by value, 2012-17
(£m)1,651
Best case (£m)
3,792
Worst case (£m)
2,074
Mintel forecast (£m)2,811
0
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000
2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017
Mark
et
valu
e (
£m
)
95%
Confidence intervals
90%
70%
50%
Actual Forecast
Est.
0
Wii U
Sony/ Microsoft next gen
Slimline, cheaper models
mintel.com74
Consumer research planning
• Deciding what to cover – insight from desk and trade research
• Questionnaire design – collaborate with Research Services Team
• Looking at ways to provide interesting analysis
• Qualitative consumer research
mintel.com75
Qualboard:
1. To add context to rigorous quant data2. To point to emergent trends3. To provoke additional quant analysis
mintel.com76
Face-to-faceOnline Specialist panel
Consumer research planning
• Internet users
• Increasingly popular form of research
• More detailed exploration of issues
• Nationally representative
• Reaching those groups not engaged with the internet
• Annuities / Equity Release Schemes
• Travel Agents
• IFA research
• HNW groups (Affluent Banking or SIPPs)
• Youth surveys
mintel.com77
Desk research
Company reports
Press releases
Media reports
Legislative and regulatory developments
Statistical Forecasting
Mintel’s market size database
Other data ONS, Treasury, Bank of England etc.
Consumer research
Mintel’s own consumer research
TGI data
Qualitative research
Trade research
Trade input – formal and informal
Industry events
Understanding a market
Mintel and analyst experience
Bringing it all together
mintel.com78
Transforming information into actionable insight: What makes it work
• Understanding/evolving the role of syndicated research
• Faster
• More accessible
• More consultative
• More disruptive.
mintel.com79
“The reports were too out-of-date
- I wanted something more
current and relevant”
Updating our Model
“To ensure Mintel remains timely”
� Move away from the 2-year publishing cycle?
� Modified approach to report writing and publishing?
� Analyst updates (market data/company information/consumer)
� Auto updates (background data and existing databases)
� Consistency with US?
“There's a real lag between
some reports being updated”
“Found relevant report
easily but not
recent enough”
mintel.com80
Communication“Building relationships and tapping into client knowledge”
� Making clients one of the first ports of call:
� Modifications to the current system?
� Developing the analyst’s role to become more than a picture on the website
� Listening to clients on a regular, on-going basis
� Pro-active development led by research.“More consultation
with clients
for their specific needs”
‘I didn’t realise I
could talk to the analysts’
– 46%
mintel.com81
Consistent and Insightful“Usage is high, but frequency is not high enough”
� Driving frequent usage up:
� Grounding Inspire in more ‘facts’ and making What’s Hot to be more than just a visual treat
� We’re guilty of ‘hiding away’ insight (reports too long, too much text)
� More flexible approach to report scheduling (hot topic reports)
� Consistent approach across Oxygen (both within UK and UK/US).
“More raw data
where possible.
Let me make decisions!”
“Opinions/viewpoints always
interesting/thought provoking”
“Keep the same level of detail
and analysis but just make it
clearer and easier to find
what you are looking for”
mintel.com82
3.1
20.1
21.2
30.2
40.4
51.5
63.0
67.6
Other
When I cannot find data elsewhere
When somebody recommends a report internally
When I need a quick answer
For a strategy meeting
To prep for a client meeting/ presentations
As evidence to back up our ideas
When I have been given a brief for a project
Base: 646 Mintel Oxygen clients
Thinking about your needs, why do you typically use Mintel?
© 2012 – All rights reserved Fluxx Ltd.© 2012 – All rights reserved Fluxx Ltd.
Thank you
Gavin Gordon
Mintel
© 2012 – All rights reserved Fluxx Ltd.© 2012 – All rights reserved Fluxx Ltd.
© 2012 – All rights reserved Fluxx Ltd.
Is the User
aware of the test?
Product/Service
Maturity Level
Unaware Aware
Idea/Concept
Partial/Fully
Developed
The fluxx ‘Tested User’ Framework
© 2012 – All rights reserved Fluxx Ltd.
What are people’s opinions,
expectations and ideas?
How well does our
user interface
support our product?
Which is the best way to
implement this?
Will people find the
product/service of value?
Market
Intelligence
Unaware Aware
Product/Service
Maturity Level
Idea/Concept
Partial/Fully
Developed
Is the User
aware of the test?
The fluxx ‘Tested User’ Framework
© 2012 – All rights reserved Fluxx Ltd.
Market Research
companies
Usability Testing
e.g. City Interaction Lab
Product Development
e.g. Fluxx
MVP/Lean
e.g. Fluxx Labs
Market Intelligence
e.g. Mintel
Unaware Aware
Product/Service
Maturity Level
Idea/Concept
Partial/Fully
Developed
Is the User
aware of the test?
The fluxx ‘Tested User’ Framework
© 2012 – All rights reserved Fluxx Ltd.
Questions Trying to Answer:
Will people find the product of value?
When to use:
When there is a perceived barrier or high degree of uncertainty about willingness of consumers to adopt
Where the product paradigm is substantially different from the norm
Technique(s) Used:
Minimum Viable Product (MVP)
Lab-like
validation
Where product is at an early concept stage, and users
are unaware of the test itself.
© 2012 – All rights reserved Fluxx Ltd.
Questions Trying to Answer:
What are people’s opinions, expectations and ideas?
When to use:
When there is a wide range of possible concepts that you want to understand more
Where you want early stage insight into how specific consumer groups behave and think
You want consumer ideas and input
Technique(s) Used:
Focus Groups, Market Research
Opinions
& ideas
An early stage concept, and users are aware they are
being asked for their opinions and input
© 2012 – All rights reserved Fluxx Ltd.
Question Trying to Answer:
Which is the optimal way to implement this?
When to use:
Where there are clearly defined potential solutions with no strong supporting evidence as to which is the best approach
Technique(s) Used:
A/B Testing, Multi-variate testing (MVT)
Where product is at a later stage, but you don’t make
consumers aware that they are part of a test
A/B
& MVT
© 2012 – All rights reserved Fluxx Ltd.
Questions Trying to Answer:
How well does our user interface support our product? Can we make it better?
When to use:
When you are clear on your proposed design solution and want to uncover any potential barriers to adoption
Technique(s) Used:
Usability Testing, prototype testing, eye tracking
User
Testing
Where product is well developed and people know they
are being asked to test it
© 2012 – All rights reserved Fluxx Ltd.
Questions Trying to Answer:Is this the right thing to do?
Is there a market for what we want to do?
When to use:When you need clarity around target segments or market
To provide external evidence of key adoption assumptions
To inform and validate the business case
Technique(s) Used:Surveys, Polls, Intelligence Gathering
Market
Intelligence
Intelligence and data on consumer activities, behaviours and
opinions in specific sectors, industries and markets
© 2012 – All rights reserved Fluxx Ltd.© 2012 – All rights reserved Fluxx Ltd.
Appendix
of common
Testing Techniques
© 2012 – All rights reserved Fluxx Ltd.
Fluxx Rapid Start
• A multi-day multi-disciplinary hugely intensive exercise unique to Fluxx, where business teams are taken from early stage ideas to launchable products ready for circulating to consumers
• Typically this is done in two days and key side effects include– Rebuilding teams’ belief in their ability to deliver
– Promotes collaborative design and ways of working
– Incorporating the voice of the customer into the build process
– Most rapid way to get something into the hands of customers
© 2012 – All rights reserved Fluxx Ltd.
Focus Groups
• Definition:
– A focus group is a form of qualitative research in which a group of
people are asked about their perceptions, opinions, beliefs, and
attitudes towards a product, service, concept, advertisement, idea, or
packaging. Questions are asked in an interactive group setting where
participants are free to talk with other group members.
© 2012 – All rights reserved Fluxx Ltd.
Usability Testing
• Definition:
– Usability testing is a technique used in user-centered interaction
design to evaluate a product by testing it on users. This can be seen as
an irreplaceable usability practice, since it gives direct input on how
real users use the system.
© 2012 – All rights reserved Fluxx Ltd.
Ethnographic Research
Ethnography is technically about observing a particular society or culture through a passive participation in it, and without impacting on the society itself, and this is still the strictest definition of it.
However, nowadays, it is more widely used to categorise techniques that are about trying to understand how people live their lives. Typically this involves visiting people in their home or work environments using a series of highly practical questions designed to uncover what is happening and why, Their goal is to see people’s behaviour on their terms, not in an artificial way.
The latter approach is often confused or used interfchangeably to mean what is known as a ‘contextual inquiry’ or ‘contextual analysis’ , the definition of which is observivng someone’s interactions with your product in their usual context or surroundings.
© 2012 – All rights reserved Fluxx Ltd.
A/B Testing
• A/B testing or split testing compares the effectiveness of two
versions of a product/service in order to discover which has
better response rate or better sales conversion rate
• Two alternate designs or approaches are randomly served to
people, and the resultant outcomes of those differently
served customer bases compared to each other, and a control
group.
© 2012 – All rights reserved Fluxx Ltd.
Heuristic Evaluation
• A Heuristic Evaluation, or Usability Audit, is an evaluation of an interface by one or more Human Factors experts. Evaluators measure the usability, efficiency, and effectiveness of the interface based on 10 usability heuristics.
• There are many heuristics available, but a popular one was created by Jakob Nielsen in 1994. Many argue that this heuristic is now outdated by modern interaction techniques, and takes no account of user attitudes.
© 2012 – All rights reserved Fluxx Ltd.
‘Depths’
• Structured, one-on-one interviews help researchers learn
about users’ attitudes and beliefs surrounding a website or
application and specific tasks that the website or application
supports. In order to prevent the researcher from introducing
bias and to ensure that every participant is asked the same
questions, the researcher prepares a Discussion Guide,
which is a list of questions that will be asked of participants
in a particular order.
© 2012 – All rights reserved Fluxx Ltd.
Card Sorting
• Card sorting is a useful tool to determine how users
categorize the information that will appear on a website. The
name “card sort” comes from the original exercise, which
used index cards with a word or phrase written on one side,
and in some cases, a definition or additional information on
the reverse side
© 2012 – All rights reserved Fluxx Ltd.
Electroencephalography (EEG)
• EEG involves measuring electrical activity in different parts of
the brain in response to certain stimuli
• It can now used for the assessment of user experiences to
see how people react at an emotional level to stimulus or a
particular experience
• Detailed and specific design recommendations can be made
bbbby recording reactions at different stages of interaction with
a product/service
© 2012 – All rights reserved Fluxx Ltd.
Minimum Viable Product (MVP)
• Minimum Viable Product is a strategy used for fast and
quantitative market testing of a product or product feature
• The product is typically deployed to a subset of possible
customers, such as early adopters that are thought to be
more forgiving, more likely to give feedback, and able to
grasp a product vision from an early prototype or marketing
information
• This is the technique employed by the Fluxx Labs method
© 2012 – All rights reserved Fluxx Ltd.
Multi-variate testing (MVT)
• Multivariate testing is a technique for testing hypotheses on
complex multi-variable systems, especially used in testing
market perceptions
• It is a process by which more than one component of a
product/service may be tested in a live environment
• When automated it can be very powerful in finding the
optimal combination of multiple component parts
© 2012 – All rights reserved Fluxx Ltd.
Prototype Testing
• Prototypes can be tested for aspects like design flaws and
ease of use, two things that are critical if your product is
going to be a success
• You need to make sure everything works the way it should -
and that your customers can figure out how to make it work
too
© 2012 – All rights reserved Fluxx Ltd.
Eye Tracking
• Eye tracking measures the point of gaze and eye movement
• It can detect the direction in which users are looking and the
duration for which they are focusing on a specific part of a
product e.g. web page
• This is helpful knowledge if you are trying to find out if users
are actively focusing on particular content on your page or if
they miss it all together
© 2012 – All rights reserved Fluxx Ltd.
Quantitative Market Research
Market research is the systematic gathering and interpretation of information about individuals or organisations using the statistical and analytical methods and techniques of the applied sciences to gain insight or support decision making
For example:
• Mass participation surveys– Usually aimed at specifically targeted segments. This type of research relies on volume of
participants to give its results significance. Questions usually have binary or multiple choice answers.
• Omnibus Survey: – A regular survey designed to determine what the general population thinks on a range of topics and
issues. Brands can pay for questions on this survey alongside the other questions. Thought to have derived from the phrase “The Man on the Clapham Omnibus’ which was used as a euphemism for ‘the common man’.
© 2012 – All rights reserved Fluxx Ltd.
Leading global market intelligence company with more than 500 employees worldwide
Guide for leaders in marketing, product development and research for more than 35
years
Clients rely on our authoritative offering to drive strategic decisions
About MintelAbout MintelAbout MintelAbout Mintel
What we do
Provide fresh insight into your marketplace and across industries
Identify opportunities for profit and growth
Formulate insights into consumer behavior, product innovation and competitive
marketing strategies
Create high-quality data, evocative analysis and tailored consulting
© 2012 – All rights reserved Fluxx Ltd.© 2012 – All rights reserved Fluxx Ltd.
“What people say they will do and
what they actually do is often
very different and can be greatly
affected by their level of
awareness of the task at hand”
© 2012 – All rights reserved Fluxx Ltd.© 2012 – All rights reserved Fluxx Ltd.
Recommended