Upload
guest205a5430
View
1.773
Download
0
Embed Size (px)
DESCRIPTION
Presentation by Tim Bosenick, SirValUse, Germany for the Usability Marathon 2009
Citation preview
Bild
Measuring User ExperienceCombining the “Qualitative” with the “Quantitative”Some Examples from the Web
Tim Bosenick, SirValUse Consulting GmbH
What do I know about my site and the users?
I know the serverlogs, PIs and
sessions.
Sometimes a apply surveys
on my site.
I have a Web Analytics System and collect loads
of data.
But...
I have some data of users given during
order.I know that some users order.
I run regularely qualitative User
Experience Tests.
But still there are many questions…
Who exactly are my users? Do they act effectively on my
website?
Are my competitos better? Why?(scope, CR, ..)
How does the business
competition develop?
Does my marketing reach the people I want
to reach?
Where is my target group on the way in internet?
Where can I appeal to them?
Who are my most important
competitors?Do I reach my target groups?
Conclusion:
We need a 360 degree view on our costumers
User experience, usability and marketing (the whole “experience chain”) come
closer together
We often know the “what happens”, we sometimes know the
“why” – a combination would be great
The Idea
5WebWerte
Combining qualitative and quantitative observational and survey data.
Further requirements:
» Single source data collection for high validity
» Scalable to large numbers of participants
» Possibility of true-intent and task-based user experience tests, also experiments
» Measurement without cooperation of website owners or publishers(only Add-On)
StandardisedSurveys
FreeUser Feedback
User TrackingDetailed
analysis of user sessions
Quantitative Qualitative
Observation
Survey Data
for one specific website,site centric,
short time frame
Whole internet, user centricFeatures of the LEOtrace® Browser Add-Ons
Features:
» Single source measurement of reactive and non-reactive data
» Data collection without cooperation of website owners or publishers
» Allows data collection on third party sites
» Flexible setup» Customising of design and branding
» Remote control of all functions
» Extended research designs possible:» Task-based designs
» Experimental designs (manipulation of client-side HTML code)
» Data collection in SSL-encrypted areas possible (usually disabled)
Deliverable data:
» Non-reactive data ("Tracking")» Session information (e.g. duration)
» Visited URL’s
» Search queries (Google, ...)
» Precise ad impressions
» Screenshots
» Clicks (and mouse movements)
» Reactive data» Standardised surveys (also event-
triggered) with any survey software
» Free ad-hoc feedback
» Client information» Operating system, web browser, …
» User structure data provided from access panel
Whole internet, user centric Installation process
1DownloadAdd-On
2InstallAdd-On
Easy setup – justlike any other
browser add-on 3
Optional:Registration onfirst browser start
4
Optional: Identificationon every
browser start
Specific website, site centricThe user are invited on your website …
» To elevate the problems occuring while using the website,the LEOtrace® Remote Test has been developed:
… and use the website according to your natural performance.
Invitation by a layerPreliminary interviewOnline questionnaire
• "What is your today main reason of visit?"
Follow-Up InterviewOnline questionnaire
• "Did you achieve everything today you planned to?"
• "Please evaluate the website on the basis of following items."• User-Behaviour
• Clickway-Analysis
• User-Feedback
Remote Session
• "Please surf the website as you would do it normally."
Participants and website-providers don´t need to install software .Only the Invitationlayer has to be linked. Adjustments have only to be neccessary in the LEOtrace®-System itself
Specific website, site centric The Technology
Usability Expert
Recalls monitored actions of participants
Surveyserver
Give free and scaled feedback
Utility target-website
Recalls comments and evaluations of participants
Website
Content enquieries through
Participants
Proxy-Server
3
2 1
Via feedback-buttons the users are always able to call up a short survey or to give a positive or negative feedback .
Specific website, site centric The Feedback-Bar
To the finaly surveyCall up feedback-
sessions
Types of Studies
12WebWerte
Typical StudiesTasked based Benchmarking
Recruitmentfrom online access panel
(screening according totarget group criteria);
Installation of LEOtrace®Browser-Add-On
Initial survey(e.g. favourite search
engines, …)
Taskwith detailed recording of
user behaviour (clicks,screenshots) as well as free
feedback and event-triggeredquestionnaires
Final survey(e.g. rating of websites used);
Deinstallation of Add-On
1
2 3
4
Typical StudiesTasked based Benchmarking & Usage Monitoring
Recruitmentfrom online access panel
(screening according totarget group criteria);
Installation of LEOtrace®Browser-Add-On
Initial survey(e.g. favourite search
engines, …)
Taskwith detailed recording of
user behaviour (clicks,screenshots) as well as free
feedback and event-triggeredquestionnaires
Final survey(e.g. rating of websites used);
Deinstallation of Add-On
1
2 3
4
Non-reactivemonitoring of online usage
during several weeks(before receiving task)
3a
Typical StudiesDigital Behaviour Studies
Recruitmentfrom online access panel
(screening according totarget group criteria);
Installation of LEOtrace®Browser-Add-On
Initial survey(e.g. favourite search
engines, …)
Final survey(e.g. rating of websites used);
Deinstallation of Add-On
1
2
4
Non-reactivemonitoring of online usage
during several weeks(before receiving task)
3
Typical StudiesDigital Behavior Studies with Event based Surveys
Recruitmentfrom online access panel
(screening according totarget group criteria);
Installation of LEOtrace®Browser-Add-On
Initial survey(e.g. favourite search
engines, …)
Final survey(e.g. rating of websites used);
Deinstallation of Add-On
1
2
4
Non-reactivemonitoring of online usage
during several weeks(before receiving task)
3
Event based surveys,e.g. having used a
certain function,leaving a certain website,
leaving a certain page
Typical StudiesWeb Efficiency Panel
Internet userin WEP
Browser Add-On sendsURLs
Ad contactsSearch queries
Internet user buys online
Panellist scans purchases
Fusion & analysis
Offliner buys offline
Internet user buys offline
Offlinerin GfK Consumer Scan panel
Generating single source consumer & Internet usage data:
» Purchases(Consumer Scan)
» Internet usage PageImpressions
Visits
AdImpressions
AdClicks
Queries with relevant search engines
» Structural data
Typical StudiesIndustry Benchmarking
» We use the data from the Web Efficiency Panel to calculate benchmarking KPIs for certain industries (e.g. e-commerce, automobile, …)
» Possible analyses: Target group
Cross usage
Previous and following websites
Usage of functions / areas
Conversion rates
…
Typical StudiesTrue Intent Experience Reports
Recruitmentonline (real users)
(screening according totarget group criteria);
no installation needed
Initial survey(e.g. reason for
the visit, …)
Final survey(e.g. rating of website, satisfaction, reason for
leaving, …)
1
2
4
Non-reactivemonitoring of website usage
during this sessionoptional: free feedback
3
The Importance of Measurements
Quantitative studies are an ideal enhancement of qualitative studies
Qualitative Approach Quantitative Approach
Focus on “usability” Focus on “user experience”
Derive recommendations Measure key performance indicators
Focus on product management / UX departments
Focus on market research / general management
Sometimes “artificial” lab situation, no “real” behavior can be observed
“True intent” studies possible, observation of “real behavior”
Combination with qualitative marketing research possible
Combination with marketing KPIs possible
Testing of “offline” applications possible
Test object must be “online”
Why is quantitative measurement important?
» Users tend to misjudge their behaviour Social request, known brands are over-rated, problem to correctly quantify their own behavior
Example: User were asked: What kind of websites did you use while looking for a product? Answer: 88% Google. In the actual measurement phase, only 68% really used Google.
» “User Experience” gets more and more important “Usability” still is an important factor of the overall user experience, but e.g. “joy of use” and
“design” come into the focus of testing – also in the early development phase.
With this, classical market research methodologies and “real measurement” become central for the usability testing practice – or else market research companies will gain bigger parts of the market.
» “Usability” and Marketing come closer together It is nowadays not only important to get as much traffic as possible to a website (online
marketing) and to optimize the conversion rate (usability), it is also important to combine both views so that the whole shopper experience can be optimized.
Automobile Industry Monitor 2009
23WebWerte
Methods
Internet Tracking
Surveys» Surveys completed by the WEP panelists regarding the
topics automotive ownership, automotive purchase, brand affinity and advertising awareness.
To satisfactorily address the questions at hand, we have employed all of the following methods:
» Permanent data collection regarding all surfing behaviour by means of the Web Efficiency Panel (WEP) designed by GfK during Q1 2009 (01/01 - 31/03).
» Quantitative evaluation of internet use with regard to relevant automotive websites.
» Determination of indicators and modelling of navigation behaviour.
Market Analysis» Expert analysis of the most important manufacturer
websites with regard to user experience. Identification of best practices.
In Q1 2009, one-fifth of internet users visited atleast one automotive manufacturer website.
Internet users in Germany: 42,540,000
Total users of "Auto-Websites": 16,380,824
Users of manufacturers' websites: 8,308,536
Users of sales portals: 7,692,277
Users info portals: 5,423,271
Users of auto club websites: 4,153,947
Users of online community websites: 2,678,939
Users of media websites: 2,352,151
Users of auto group websites: 223,034 1%
Data base: Internet users in WEP designed by GfK between 01/01.–31/03/2009
In the first quarter, manufacturer websites werevisited around 41 million times, and on average,9.5 PIs were generated.
(in mil.)
9.5 PIs per Visit11.1 5.0 8.9 5.9 3.8 4.6
Data base: Internet users in WEP designed by GfK between 01/01.–31/03/2009
The VW website has by far the most users,followed by Opel, Ford and Audi.
Data base: Internet users in WEP designed by GfK between 01/01.–31/03/2009
Users ofmanufacturerwebsites(in thousands)
Page 1 of 2
The Mercedes and VW websites have the highestrate of repeat visits.
1.9MercedesVW
OpelAudi
SkodaFiat
CitroënToyota
MitsubishiMazda
SeatPorscheRenault
JaguarSuzukiSmartHonda
DaciaFord
PeugeotBMW
NissanAlfa Romeo
DaihatsuHyundai
SubaruKia
VolvoChevrolet
Visits per user of the givenmanufacturer websites (mean)
Data base: Internet users inWEP designed by GfKbetween 01/01.–31/03/2009
Overview of the most significant indicators formanufacturer websites
Unique Users(in thousands)
PIs per Visit
Visits per User
User Proportionfor Showroom
use
User Proportionfor
Configuratoruse
Proportion ofended
configurations
User Proportionfor Financing
pages
User ProportionFinancingCalculator
User Proportionfor Dealer
Search
Mean Value* 899 9.6 2.2 52% 28% 36% 11% 6% 13%
Audi 1,051 10.6 2.3 52% 23% 45% 1% 1% 0%
BMW 615 6.1 1.5 43% 26% n. a. 5% 5% 4%
Chevrolet 306 15.7 1.3 70% 23% 37% 14% n. a. 16%
Citroën 692 9.0 2.0 62% 29% 59% 8% 0% 8%
Dacia 756 9.3 1.6 89% n. a. n. a. 29% n. a. 31%
Fiat 545 14.9 2.1 40% 45% 45% 0% 13% 13%
Ford 1,053 6.4 1.6 51% 27% n. a. 9% 3% 20%
Honda 895 6.4 1.6 24% 12% 50% 4% 0% 11%
Mercedes 771 7.8 5.0 39% 24% 24% 5% 8% 31%
Mitsubishi 274 5.7 2.0 54% 27% 6% 0% 6% 18%
Opel 1,109 16.6 2.5 59% 48% 50% 3% 16% 10%
Peugeot 923 5.0 1.5 37% 14% n. a. 27% n. a. 11%
Renault 882 9.4 1.8 61% 43% 27% 3% 1% 10%
Skoda 722 14.6 2.2 40% 36% n. a. 4% 5% 4%
Toyota 860 6.5 2.0 75% 24% n. a. 10% 12% 16%
VW 2,933 9.2 3.7 34% 20% 12% 59% 2% 5%
Data base: Internet users in WEP designed by GfK between 01/01.–31/03/2009 Info: The best three providers are marked in green, the worst three in red.
* Refers to the 16 manufacturers represented
Showrooms, configurators and information aboutfinancing are of particular interest to users who intend to purchase – the dealer search is somewhat less used by those interested in purchasing.
Data base: Internet users in WEP designed by GfK between 01/01.–31/03/2009
Info: In the calculation of the mean,only those manufacturer websitesare represented for which use dataare available for the given area.
Fewer than half of manufacturer website usersvisit showrooms – if one is visited, then others arelikely to be also.
Showroom users (Q1): 3,680,249
1 Showroom visited: 1,373,005
2 Showrooms visited: 626,044
3 Showrooms visited: 438,454
4 Showrooms visited: 330,935
5 Showrooms visited: 190,848
More than 5 Showrooms visited: 720,963
Number of showrooms visited for allusers of manufacturer websites (mean):
1.0
Number of showrooms visited for allusers of showrooms (mean):
4.2
Data base: Internet users in WEP designed by GfK between 01/01.–31/03/2009
Overview of the most significant indicators formanufacturer websites
Unique Users(in thousands)
PIs per Visit
Visits per User
User Proportionfor Showroom
use
User Proportionfor
Configuratoruse
Proportion ofended
configurations
User Proportionfor Financing
pages
User ProportionFinancingCalculator
User Proportionfor Dealer
Search
Mean Value* 899 9.6 2.2 52% 28% 36% 11% 6% 13%
Audi 1,051 10.6 2.3 52% 23% 45% 1% 1% 0%
BMW 615 6.1 1.5 43% 26% n. a. 5% 5% 4%
Chevrolet 306 15.7 1.3 70% 23% 37% 14% n. a. 16%
Citroën 692 9.0 2.0 62% 29% 59% 8% 0% 8%
Dacia 756 9.3 1.6 89% n. a. n. a. 29% n. a. 31%
Fiat 545 14.9 2.1 40% 45% 45% 0% 13% 13%
Ford 1,053 6.4 1.6 51% 27% n. a. 9% 3% 20%
Honda 895 6.4 1.6 24% 12% 50% 4% 0% 11%
Mercedes 771 7.8 5.0 39% 24% 24% 5% 8% 31%
Mitsubishi 274 5.7 2.0 54% 27% 6% 0% 6% 18%
Opel 1,109 16.6 2.5 59% 48% 50% 3% 16% 10%
Peugeot 923 5.0 1.5 37% 14% n. a. 27% n. a. 11%
Renault 882 9.4 1.8 61% 43% 27% 3% 1% 10%
Skoda 722 14.6 2.2 40% 36% n. a. 4% 5% 4%
Toyota 860 6.5 2.0 75% 24% n. a. 10% 12% 16%
VW 2,933 9.2 3.7 34% 20% 12% 59% 2% 5%
Data base: Internet users in WEP designed by GfK between 01/01.–31/03/2009 Info: The best three providers are marked in green, the worst three in red.
* Refers to the 16 manufacturers represented
The showrooms on the websites of Dacia, Toyotaand Chevrolet are visited especially often.
Proportion of total users whovisited the showroom on anymanufacturer website (mean):
52%
Number of users of a website who visited theshowroom on thatwebsite.
Data base: Internet users in WEP designed by GfK between 01/01.–31/03/2009
Images of the models displayed on the start pagespur users on to visit the showrooms.
» The first-ranked websites of Dacia, Toyotaand Chevrolet feature links to the showroom directly on the start page, including images of the models.
» Websites that only offer a text link (e.g. Peugeot, VW) create less interestfor showrooms. If the links to the models are not prominently placed,
the showrooms are used significantly less often (see Honda).
Showrooms with a high degree of multimedia concentration often may notexploit their potential because they are less able to be located.
DaciaToyota
Honda
eCommerce Branchenmonitor 2008
35WebWerte
Analysis question
» Why does a website sells good – and better than the competition?
» Why do customers decide for one special online-shop?
» Where (also online/offline) and how do customers inform themselves?
» Which products are bought out of what reasons?
Focus: Consuming goods in low to middle pricesegment, which are bought regularly
Method- Mixing
1. LEOtrace® Behaviour Tracking of online-shopper
» Multi-Method Study for wholistic mapping and analysis of the onlineshopping-process, from wording the shoppingintention until delivery.
» Monitoring of users during productresearch, measuring of detailled data-characteristics and different interviews before, during and after the monitoring phase.
2. Expert assessment by the SirValUse E-Commerce-Team
» Focus: Best Practice-Analysis.
Fact sheetLEOtrace® Behaviour Tracking
Method» Several weeks long Add-On-Study for Internet Explorer and
Firefox.
Sample
» 440 Users were recruited via an Online-Access-Panel for the study.
– All participants planned to purchase online-products during field time.
– 59% Women, 41% Men.
– 52% under 30 years, 26% 30-39 years, 22% 40 years & older.
Field time » From Juli 1st until September 8th 2008.
Tested websites
» Testing of all websites which could be important for research-and shoppingprocess.
– Focus on onlineshops.
– Furthermore: price comparing-websites, searchengines and portals, manufacturer-websites and community-websites.
Expiry of Behaviour Trackings for Users
First survey:
Screening
Installation of AddOn
Questions to online-shopping
Field time:
Handle the shoppinglist
Monitoring of surf behaviour
Event-Interviews
Final survey:
Afterexploration
Imagemeasurement of onlineshops
Deinstallation of AddOn
2 weeks 7 weeks 1 week
TechnologieBrowser Extension LEOtrace® AddOn
The Shoppinglist
» Origin- and endpoint of all productresearch.
» The Users were told to keep following aspects up to date:
– Productcategory
– Productname
– (contemplated) price of product
– State of research
» Furthermore they were told before a research session, what product had to been researched.
Devolution of a research-session
Shoppinglist: Instruction forproduct to beresearched for
Shoppinglist: If so matching ofprice andresearch-status
Post survey: Evaluation ofresearch-session
Eventinterview "general"
Eventinterview"detailled product
website
Eventinterview"searchfunction
Eventinterview "ordering process"
Research-session
Example: Research- & Shoppingsession (1)
User: 244756
male, 29 years
Product for which exists a buying interest:
Nintendo Wii
Research-sessions:
July 20th (21:12 - 21:57)
July 21st (18:20 - 18:41)
On July 20th following websites were selected:
Geizkragen.de
Search for "Nintendo Wii"
cyberport.com
discount24.de
amazon.de
Example: Research- & Shoppingsession (2)
User: 244756
male, 29 years
On July 21st amazon.de was accessed:
search for "Wii"
Game console put into shopping basket
Search for "Mark Medlock"
CD "Cloud Dancer" by Mark Medlock put into shopping basket
Analysis of research:
Both products were purchased online at amazon.de
Comment about order:"Great that I could bought the CD here as well"
Research-Status at end of field time
Cases QuotaCases
per user
Still in research-phase 800 36% 1,8
Bought online 920 42% 2,1
Bought offline 189 9% 0,4
No shopping interest anymore
289 13% 0,7
Total 2.198 5,0
Basis: Total (N=440).
Used Onlineshops during field time
Basic: Total (N=440).
Rating of Onlineshops
Question: How helpful were following Onlineshops during research and purchase of the varied products?
Basic = User of each website
Rating on a scale of 6: 1=Very helpful 6=Not helpful at all.
Only a section of the answering scale is shown .
Amazon (N=342)
eBay (N=281)
Bonprix (N=50)
Weltbild (N=57)
Neckermann (N=53)
Tchibo (N=65)
Buecher.de (N=46)
Esprit (N=35)
Ikea (N=33)
Quelle (N=80)
Otto (N=81)
Buch.de (N=42)
H&M (N=36)
Conrad (N=64)
Used shops per product following productcategories
Computerhardware / -equipment
Toys
Books
housewares / -tools
Fitments
Consumer electronics
CDs / DVDs
Other products
Telecommunicationproducts
Fashion / Shoes
Videogames
Sports goods / -tools
Jewellery/Watches/Accessoiries
Healthproducts
Perfume / Cosmetiques
Tickets
2,3
Basic: products being researched for (N=1.766).
Rating of experiences during research-phase
Incomplete delivery Couldn´t find anything fitting
Unsufficient productdescription/-mapping
No wanted payment options
no response on E-Mail-request Complicated navigation
Delivery problems pricing not transparent
Product not available unsufficient searchfunction Oversupply
Speedtransfer Too expensive
Und zwar:
How do you rate the experiences you made during the last 6 weeks according to research and shopping in internet?
Tag Cloud View
Basic: Total (N=440).
True Intent Study
50WebWerte
Question 10040: Did you succeed in what you wanted to do on the website today?
in percent
Area: Shop – 'Handys & Shop'Success
37
52
63
48
successful not successful
Buy something (N=50)
Buy something online (N=50)
Results generated from log data.Participants who started from the page "Handys & Shop": N=100. Only navigation steps with N > 1.
NavigationCancel of remote-session
"Handys & Datengeräte"
"Aktions-Angebote"
"Vertrags-Verlängerung"
8 28 12 6
"Prepaid-Angebote"
10
Login area("https")
5
"Tarife & Optionen" *
Handsetdetails
8
24
30
27
9 24
Area: Shop – 'Handys & Shop'Structure of navigation
* First page of the area (Overview page)
in percent
Results generated from log data.Participants with buying intention: N=100. Only navigation steps with N > 3.
NavigationCancel of visit
Area: Shop – 'Handys & Shop' Structure of navigation – Buyers
"Handys & Shop" *
"Aktions-angebote"
25
37 Homepage23
"Handys & Datengeräte"
Handsetdetails
35
18
20
9
11
Login-area("https")
20
23
Pricepop-up
18
20
20
* First page of the area (Overview page)
in percent
Area: Shop – 'Handys & Shop'General
The homepage of the section 'Handys & Shop' was praised as clearly arranged and pleasantly designed.
The range of devices in 'Handys & Shop' was well liked. The choice of mobile phones was stated as complete and up-to-date. Particularly the phone offers for 1€ were liked by the participants.
"Great, that there are mobile phones for one Euro, which look o.k. and have many features."
If the preferred phone model was not offered on the website, the range of products was often criticized as too small. In this case the choice was also described as out of date. Participants always expect that the latest models are available on the website.
"I searched for the Samsung E 840, thinking that XXX as market leader always offers the most up-to-date phone models, even before other providers offer them."
" 'Top-Angebote' (top-offers) are on offer - latest mobile phones are missing."
Area: Shop – 'Handys & Shop'Mobile phones detail-page
Understanding the meaning of the several "function-icons" was a problem.
"Icons completely inoperative!!! They don't work for a comparison and are not understandable without explanations. Very, very bad."
The icons did not suggest that further explanations were available and participants could therefore not find them.
The offered alt-tags were often not clearly understandable (e.g. Voice, Orga).
The explanations were sometimes very long and it was necessary to read carefully through them in order to understand which function was described. Most users only wanted to know which function was included, not its
purpose.
? (radio)
Time for Questions
56WebWerte
SirValUse Consulting GmbH | Schlossstraße 8g | 22041 Hamburg
Tel. +49 (40) 68 28 27-0 | Fax +49 (40) 68 28 27-20
www.sirvaluse.de
WebWerte