Running a SuccessfulBeta Test Program

Adam Long (CTO)Darren Gage (QA Mgr) Sam Soubra (Dev Mgr)

September 2010

Agenda

• About Us• Running a Beta Test Program• Runtime Intelligence Service– Overview– Code Demo

• Questions

About our Company

Established in 1995

Independent Software Vendor

based in Melbourne, Australia

90% of revenue from exports

> 400,000 customers in 150 countries

About our Software

• Qualitative Data Analysis– Unstructured Data

• Documents• Audio / Video• Images• Spreadsheets

• What MS Excel does for structured numerical data, we do for unstructured non-numerical data

About our Software

• NVivo 9– .NET 3.5

• Win Forms• WPF• WCF

– SQL Server 2008 R2 Express

• NVivo Server 9– Silverlight 3– SQL Server 2008 R2

Standard

Beta Test

What• Beta testing is conducted by

a sample of customers to identify flaws in feature complete software

• Generally occurs after alpha testing (unit & system)

Why• Validate objectives

– Market– Features– Quality

• Improve outcomes– Collaboration / Innovation

• Assist marketing– Quotes / Case Studies

• Plan future direction– Opportunities

7 Steps of a Beta Test Program

Plan

2 weeks

Promote

4 weeks

Select

2 weeks

Collect

4 weeks

Analyse

2 weeks

Prioritize

1 week

Recognize

1 week

1. Plan

• Scope– Goals & Success Criteria

• Approach– Open vs. Closed

• Responsibilities– Management, Communications, Defects, Suggestions

• Schedule– Balance between having enough features available to

test versus enough time to incorporate feedback

2. Promote

• Internal– Sales, Marketing, Training

• External– Existing Customers– Potential Customers

• Website & Social Media announcements

• Customer Newsletter• Email Select Customers

• Best Beta Testers– Interested– Independent– Vocal

• Avoid pressure to include– Sales Leads– Evaluators

3. Select

• Online Application– Outline Program Expectations– 20 Questions

• Technical Environment• Market Segment• Research Background• Competitor Software• Motivation

• Result– 523 Applicants– 78 Testers Selected

• Selection Process– Top Previous Beta Testers– Top Forum Posters– Subject Matter Experts– Random Applicants

• Simulation Program to Optimize Spread of Answers to Questions

• Obtain Signed Agreement

4. Collect

• Online Webcasts (Citrix GoToWebinar)– 5 recorded videos & 88 feedback forms

• Online Forums (IP.Board)– 412 suggestion posts & 335 defect posts

• Online Survey (Survey Monkey)– 58 survey responses

• Application Analytics (RIS)– 2,448 hours of usage statistics

5. Analyse

• Identify– Defects– Suggestions

• Understand– Most liked features– Most disliked features– Most important

improvements needed

• Dogfooding– Used NVivo Beta to

analyse feedback• Videos• Forums• Surveys

– In conjunction with• Application Analytics

6. Prioritize

• Prioritize Suggestions– 1 Must Address for RTM– 2 Preferably Address for RTM– 3 Consider for Service Pack– 4 Defer for Future Release– 5 Rejected

• Raise Defects in Bug Tracking

7. Recognize

• Active Participants (85%)– Activated the software & provided some feedback– Average of 37 hours testing over 4 weeks– Receive free copy of software

• Top 20 Participants (25%)– Also receive Amazon voucher

Beta Test Tips

• Allocate sufficient time & resources• Observe usage patterns using Application

Analytics and encourage testers to explore specific features

• Reprioritize effort based on final feedback• Hold weekly status meetings• Communicate outcomes to internal

stakeholders

Runtime Intelligence – What is it?

• The purpose of runtime intelligence as a component of SDLC management is to support better development, support and R&D investment decision making.

• Leverages technologies, managed services and practices for the collection, integration, analysis, and presentation of application usage levels, patterns and practices

Runtime Intelligence – Why use it?Gauge

Effectiveness of the beta

test

Collection Phase

Feature usage

Analysis Phase

Platform usage

Beta tester effectiveness

Runtime Intelligence – Why use it?

Improve Decision Making

Usage context while Analysing

feedback

Prioritization PhaseAnalysis

Phase

Usage context while Prioritising

feedback

Understand Quality

Runtime Intelligence – Tool Choice

• PreEmptive RIS– Features matched

those in competitor’s offerings

– We had used other PreEmptive tools (Dotfuscator)

– Integrated into Visual Studio 2010

• Other tools out there;– EQATEC Analytics– Gibraltar Software

Runtime Intelligence – Architecture

11

2

2

3

3

Instrumented App is run

Usage data sent to Data Repository

Dashboard and Reports

Runtime Intelligence – Implementation

• Credentials, Entry point, Endpoint & Security

• Features & Custom data

• System and Performance

• Opt-in policy

Configure analytics

• Inject Instrumentation

• Package and Deploy

Application distribution

• Beta Mgmt., UX program, Support

• Business Intelligence & ROI

• Operations Mgmt.

Runtime Intelligence Services

Runtime Intelligence – Services

Preemptive Cloud

Yours

Aggregate

Validate

ManagePublish

Runtime Intelligence endpointRuntime Intelligence repository

Aggregate, validate, manage, & publishManaged or on-premise

For internal dev. and multi-tenant constituencies

Runtime IntelligenceReports & Dashboards

Owner, application, feature, user, & customRESTful API and export

IDE, CRM, ERP, & IT Operations

Runtime Intelligence – Reports and Dashboard

• PreEmptives RIS Portal

Runtime Intelligence – Export

• Data is rich

• You will need to spend time to mine it!

Runtime Intelligence – Export

Circle Graph Dendrogram Horizontal Dendrogram Vertical Mds 2d Mds 3d

Total 36 65 52 82 92

5

15

25

35

45

55

65

75

85

95

Similarity Diagram Type

# us

es o

f fea

ture

s

Runtime Intelligence – Export

Windows 7 Enter-

prise

Windows 7 Home

Premium

Windows 7 Profes-

sional

Windows 7 Ultimate

Windows XP Home Edition

Windows XP Pro-fessional

Windows(R) XP Pro-fessional x64 Edi-

tion

Windows Vista

Business

Windows Vista Home

Premium

Windows Vista

Starter

Windows Vista Ul-timate

Not De-fined

Total 6 9 14 13 1 27 1 8 9 1 5 1

2.5

7.5

12.5

17.5

22.5

27.5

OS Edition

# w

orks

tatio

ns

Runtime Intelligence – Where to next?

• Move into Production

• Improve Development Decision Making for future releases

Development• P

erformance tuning

• Usability

QA• R

isk based testing

Documentation• M

aterial coverage

• Help usage

Application Usage

Code Demo

Dotfuscator by PreEmptive Solutions• Runtime Intelligence defines several message

types:– Application and Session Start – Application and Session Stop – Feature – Performance Probe – System Profile – Tamper Detected

Runtime Intelligence Portal’s dashboards

To have your application send these messages, you must:

• Be a Runtime Intelligence Service subscriber• Activate Runtime Intelligence from within

Dotfuscator• Attribute your application with Runtime Intelligence

attributes, including Setup and Teardown• Run your application through Dotfuscator with the

Send Analytics Messages option turned on

Business Attribute

Application Attribute

Setup Attribute

Teardown Attribute

Feature Attribute

Sample Application

Sample Code

ILDASM Before

ILDASM After

Analytical Data

Questions?