UX Testing using Functional Crowd Testing

Jack Dillon | Head of Crowd Delivery, QualiTest11th Nov 2015

Agenda

* Not a sales pitch; practice, developed and now presented

| Overview of what Crowd Testing Delivers

| Service Evolution

| Functional / UX focus

| How It Has Been Delivered

| Some nice images

27

Crowd Testing| Harnessing the crowd for client / front end testing

| Functional Exploratory, by the crowd, not by your testers

| Scripted testing using specific demographics/ geographies

| Uncontrolled environment (in a good way)

| For Retail, Telco, Media, Gambling, Gaming – testers represent end-customer but now working for you, protecting brand

| Testing not for coverage but to complement traditional testing, making product more predictable

28

Service Evolution| Crowd Testing to Managed Crowd Testing

29

| Single point to manage the crowd | Harnessing testing best practices, not only crowd| Adds and covers the end to end aspects, such as test planning, design and

reporting | Internally advocates the service – making sure service is utilized optimally

within your organisation| Helps unobtrusively embed the service into existing WoW (Agile or

Waterfall)

On-demand testing

On-demand Feedback

On-demand Ux FeedbackNO-

BRAINE

R

Wiki

| Traditional Usability Testing| Usability testing is a technique used in user-centered interaction

design to evaluate a product by testing it on users. This can be seen as an irreplaceable usability practice, since it gives direct input on how real users use the system. This is in contrast with usability inspection methods where experts use different methods to evaluate a user interface without involving users. Usability testing focuses on measuring a human-made product's capacity to meet its intended purpose.• (Wikipedia: https://en.wikipedia.org/wiki/Usability_testing, 2nd November 2015)

Why?

DON’T REALLY

CARE

30

User - Experience (Ux) Testing| Specialist lead practice, not always delivered by specialist

| Relies on scores rather than one’s opinion

| Repetitive, not a one-off activity

| Uses Benchmark of similar apps/competitors to determine quality

| Benchmark to previous functionality to determine improvement

| Ability to correlate with feature release

| Questionnaire building and raw-data analysis

| Complements App store reviews/Applause Analytics

31

Examples of when it goes wrong

| In USA on the 30th September 2015 Facebook deployed an update to how their news feed operated. Users now see most popular rather than most recent updates. This had a impact of 1216 negative reviews on day of release and equated to and increase of negative reviews between 100% – 500% in October.

| At the end of September 2015 BBC iPlayer removed the ability to save your place within any media content when navigating away from the application. This is accountable for most of the negative reviews over the coming weeks.

32

Optimal Usage of Crowd Testing

DeploymentFunctional Acceptance Devices

3rd party

R&D

In-country/ Localisation

UAT

Go livePost go- live

testing

20-50%Full functional testing, Scripted and exploratory

80-100%Access to 25 testers with 30-50 devices and browsers

80-100%Testing for content, context, geo-restrictions/ local integration

100%Smoke testing in production after go live

Smoke

20-60%Enables an effective Quality gate, prevents over-the-fence delivery

End User Experience

40-70%Ux feedback from real users

33

Functional Testing with Ux Focus

Deployment Devices and OSes Go live Post go-live testing

Monitoring of Ux, introduction of in-feature changes, competitors innovations and platforms changes

Continuous Improvement across releases

Pre-Test

Capture Ux snapshot/ Baseline

End User Experience

Create Independent Benchmark

Hybrid Load /Hybrid Performance

Embed usability elements to each testing activity

Specific Usability constraints around the different platforms and versions

Perform functional testing under load

Use different networks providers and qualities to determine impact of usability

34

What Does the Ux Feedback Look Like?| Create in-stream/micro-team awareness, define roles and list expectations

| Target Crowd demographics for analysis, qualification of pre-test cycle condition

| Building the questionnaire: 5-5/10-0

| 4 Stage Implemented Functional Usability Process| FUT Design Phase

| Recruitment and Execution Phase

| Reporting and Closure Phase• Relevant output per stakeholders• Trends, internal and external benchmark

| Retrospective and Education

| Incorporates seamlessly into Agile Practice

35

The 7-Day Process

• Day 6• Update of reports &

documentation for continuous monitoring and evaluation

• Day 7• Continuous Improvement on questions, focus

groups, results and outputs discovered to improve future engagements

• Day 3 - 5• Recruitment profile aligned to fit

requirements and customer base, Questionnaire and templates wrap up

• Testers will be required to fill in a Survey and follow user journeys created

• Day 1 and 2• CDM learning, coordinating, design and planning

the cycle with Marketing and Test Manager

FUT Design Phase

Recruitment and Execution

Phase

Reporting and Closure Phase

Retrospective and Education

Continuous effort, trend analysis and internal and external benchmarks

36

Feedback vs. Feature Release| Timeline| Define tracking of

features added and compare with results

| Add and remove questions on a cycle-by-cycle basis dependent on priority

| Results broken down by product

37

Trending

| Feature Trending| Individual tracking of

features and functionality on demand

| Results broken down by feature/question

| Trend-line reporting

38

Realistic Usable Recommendations

| Suggested outputs may include:| Functional changes

| Device Fragmentation Cycles

| Additional or updated internal / external test case scripts

| Specific cycles focusing on Ux Wireframes

| Non-traditional output may include: Additional focus required on Localisation/Accessibility

39

Adaptable Questions

40

Competition Analysis

| Applause Analytics:| Analysis of industry competition

| Focus on areas where your competition thrive

| Secondary validation of results

| Track competition’s positive and negative reviews in real time

41

www.QualiTestGroup.com

Thank You