What You Need to Know about Beta Management

What You Need to Know

Brought to you by the experts at Centercode

v1.1

Beta Management

WWW.CENTERCODE.COM

WHAT YOU NEED TO KNOW • BETA MANAGEMENT

2

About the SeriesCentercode’s “What You Need to Know” series is designed to walk readers through the basics of different kinds of beta tests. This chapter will provide a foundation on beta testing, while future chapters will focus on specific types of beta tests such as hardware or mobile tests.

AudienceThis series is intended for quality, marketing, project, and product managers who’ve been tasked with running their first beta test. If you’re an experienced beta manager, it may provide useful tips to help further improve your beta process.

GoalsThis chapter will explore the core issues and terms found in beta test management, providing you with a basic understanding of how to manage a beta test. It will also provide the framework for future chapters in the series.

Introduction

Download a simpler, print-friendly version of this chapter.

http://www.centercode.com

http://www2.centercode.com/rs/centercode/images/Need-To-Know-Beta-Management-Print.pdf

WWW.CENTERCODE.COM


3

• We’ve been focused on delivering great Beta Test Management Solutions since 2001.

• Our offerings include Centercode Connect™, a hosted beta test management platform that’s been used to run thousands of beta tests globally, fully outsourced Managed Beta Tests, and Onlinebeta.com, a community of more than 60,000 highly profiled test candidates.

• Feel free to contact us with feedback or questions at [email protected].

• Our customers run the gamut of products, markets, and company size, including:

About Us


http://www.centercode.com/connect/

http://www.centercode.com/managed/

http://www.onlinebeta.com

mailto:info%40centercode.com?subject=Need%20to%20Know%20-%20Beta%20Management

WWW.CENTERCODE.COM


4

Table of ContentsIntroduction 2

About Centercode 3

Definition of a Beta Test 6Other Terms for Beta 7

Beta Details 8

Types of Beta Testing 9

Benefits of Beta Testing 10

The Beta Message 11

Major Beta Challenges 12

Top 10 Beta Mistakes 13

The Tools of Beta Testing 15Productivity Tools 16

Non-Beta Software & Services 17

In-House Beta Systems 18

Commercial Beta Management Tools 19

Other Useful Tools for Beta 20

Planning 21The Beta Plan 22

Writing a Beta Plan 23

FAQ: How long to beta? 25

Recruiting 26Tester Recruitment 101 27

Why Do People Beta Test? 28

Friends, Family, and Employees 29

Advertising For Testers 30

Selecting Great Testers 31

FAQ: How Many Testers? 32

Kickoff 33Getting Ready for Beta 34

Product Readiness Checklist 35

Tester Readiness Checklist 36

Team Readiness Checklist 37

Legal Agreements 38

Managing 39Beta Activities 40

Bug Reports 41

Feature Requests and Forums 42

Tasks and Journals 43

Surveys and Testimonials 44

Wikis, Calls, and Visits 45

Tips for Tester Management 46

FAQ: Tester Time Dedicated? 47

Closing 48Keys to Closing a Beta Test 49

Incentives! 50

Developing a Closure Report 51

Improve 52Elements for Beta Success 53

Next Steps 54

Contact Us 55


WWW.CENTERCODE.COM


5

BasicsToolsProcessImprove

Beta Management

{ 1. Planning2. Recruiting3. Kicking Off4. Managing5. Closing

{


WWW.CENTERCODE.COM


6

“A test of a product performed by real users in real environments”

(basically your customers testing instead of your QA team)

Definition of a Beta Test


WWW.CENTERCODE.COM


7

Beta testing goes by many names. Depending on the company, sometimes they have slightly different applications, but they all focus on customer feedback prior to product release. Here are a few:

• User Acceptance Testing (UAT)

• Customer Acceptance Testing (CAT)

• Customer Validation (CV)

• Field Trials (common term in Europe)

• Pre-Release

Note: While we use the term “beta” throughout this series, the same principles, processes, and benefits apply, no matter what you call it.

Also Known As...


WWW.CENTERCODE.COM


8

• Is the final test prior to shipping a product

• Generally begins after an internal alpha test

• Begins when the product is at or near feature complete

• Includes existing and/or targeted prospective customers

• Commonly prioritizes quality improvement (i.e. squashing bugs)

• Establishes a final assessment of the complete product prior to release

• Extends QA testing (but should never replace it)

And Most Importantly...

• Differs greatly in challenges, execution, and benefits based on tools, product type, target market, beta goals, and development phase

A Beta Test... (the details)


WWW.CENTERCODE.COM


9

Types of Beta TestingThere are many different types of beta tests, each with its own set of challenges, goals, and benefits. Any individual product test is likely comprised of a number of these, based on the type of product, its target market, the goal the test is attempting to achieve, and the phase at which it’s happening. For example, the following illustrates a traditional performance-based hardware beta test aimed at the business market.

Each of these defining attributes (hardware, enterprise, etc.) will be covered in individual chapters of this series, focusing on their unique challenges, goals, and benefits.

F Software

F Hardware

F Video Game

F Mobile

F Web

F Consumer

F Children

F Business

F Enterprise

F Government

F Requirements

F Performance

F Focused

F Stress

F Awareness

F Pre-Alpha

F Alpha

F Pre-Release

F Agile

F Post-Release

1. Product Type 2. Target Market 3. Primary Goal 4. Test Phase


WWW.CENTERCODE.COM


10

• Beta testing provides true customer validation and improves product quality beyond the inherent constraints of the quality testing phase, resulting in:

1. Reduced Development Costs

2. Increased Sales

3. Reduced Support Costs

4. Reduced Returns

5. Increased Customer Satisfaction

6. Improved Product Planning

• Other benefits are unique to product type, market, beta goals, and phase (each will be covered in further chapters in this series)

• Incredible return based on minimal investment (see our Beta Test ROI Kit for details)

Benefits of Beta Testing


http://www.centercode.com/roi/

http://www.centercode.com/roi/

WWW.CENTERCODE.COM


11

Everyone will see the value of beta differently. To gain buy-in, it helps to communicate how beta will help achieve each group’s unique goals.

• Product Management • Beta testing uniquely covers every aspect of a product, including support and documentation.

• Marketing • Beta testing is an opportunity to gain initial impressions from your target market. This feedback can be used to improve the launch of the product.

• Engineering • Beta testing ensures the product is performing exactly as expected in the hands of real customers.

• Executive Level • Beta testing provides a high level view of the entire product. This information can be used to make crucial decisions about the launch of the product.

• Quality • Beta directly extends quality efforts to reach real-world environments that are difficult or impossible to simulate in a test lab.

• Sales • Beta testing can generate testimonials and other assests useful in early sales efforts. In addition, beta quality enhancements can lead to higher reviews resulting in more sales.

The Beta Message


WWW.CENTERCODE.COM


12

Major Beta Challenges1. Dedicating Qualified Resources • Beta testing generally happens at the least

opportune time, right when the product team is frantically trying to close up every loose end, and has little available time or energy to focus on beta.

2. Recruiting Appropriate Beta Testers • Not every interested applicant is the right beta tester for your product. Finding an adequate pool of targeted but unbiased testers is a difficult task.

3. Maintaining User Participation • The single most common challenge in beta is achieving high participation levels from beta testers. The industry average is lower than 30%, meaning that 70% or more will fail to provide any meaningful feedback.

4. Collecting Relevant Feedback • Not all feedback is valuable. Relevant feedback is any feedback that either results in a direct improvement in the quality of your product, or meets another specific goal set for your beta test.

5. Organizing and Distributing Feedback • Successful beta tests generate an enormous amount of feedback. Distributing the right feedback to the right people is a complicated and time consuming task.


WWW.CENTERCODE.COM


13

1 Lack of universal beta program support

2 No well defined plan

3 Underestimating ramp-up time requirements

4 Releasing a non-viable product to beta test

5 Too few or too many beta testers

6 Test period is too short or too long

7 Poorly motivated and/or managed testers

8 Ineffective tools resulting in lost time and feedback

9 Failing to manage feedback effectively

10 Badly managed incentives

Top 10 Beta Mistakes


WWW.CENTERCODE.COM


14


Beta Management


{


WWW.CENTERCODE.COM


15

The Tools of Beta TestingBeta testing is a highly unique process that includes individuals ranging a wide variety of disciplines, including your customers. Effective tools can contribute greatly to the success of a beta test.

1. Shorten beta cycles • A pre-profiled pool of candidates can reduce your recruiting and ramp-up period substantially, allowing you to achieve more, faster.

2. Reduce time investment • Great tools can cut the time you spend coordinating, communicating, and reviewing feedback of your beta test by more than half.

3. Increase tester participation • Providing a clear path for feedback reduces the friction of providing feedback, resulting in higher participation rates.

4. Impove feedback quality • Dedicated tools can help produce structured feedback that greatly improves its clarity, providing much more actionable data.

5. Increase visibility • Great beta tools will integrate with your existing systems, ensuring that your team is aware of feedback and able to react accordingly — fixing bugs and improving plans.


WWW.CENTERCODE.COM


16

Examples• Email, spreadsheets, word processors

Pros• Extremely inexpensive (you probably already have them)

• No learning curve

Cons• Can’t scale (more than ~10 participants is extremely difficult to manage)

• No means for communication among beta testers

• Not sufficient for a team environment (little to no visibility or accountability)

Productivity Tools


WWW.CENTERCODE.COM


17

Examples• BugZilla, Jira, Survey Monkey, Get Satisfaction, phpBB, WordPress, etc.

Pros• Inexpensive

• Easily accessible

• Many choices

Cons• Utilizing multiple systems is disjointed for testers (reducing participation)

• Difficult to manage / Produces fragmented data

• Participation is difficult to track

Non-Beta Software & Services


WWW.CENTERCODE.COM


18

Examples• A tool developed within an organization for managing beta tests

Pros• Built for beta, customized for specific processes and needs

• Dedicated experience for testers reduces friction and increases feedback

• May include other custom system integrations

Cons• Immensely expensive to develop and maintain

• The platform doesn’t evolve without ongoing investment

• Risks going end-of-life if internal support wanes

In-House Beta Systems


WWW.CENTERCODE.COM


19

Examples• Centercode Connect, VOConline FMS, BetaEasy

Pros• Built for beta (community, agreements, CMS, surveys, feedback, forums, reporting)

• Low learning curve for customers/testers

• Flexibility to meet evolving requirements, integration with existing tools

Cons• Niche offering with few vendor options

• Likely more costly than single-function tools

• New application to adopt

Commercial Beta Management Tools


WWW.CENTERCODE.COM


20

Other Useful Tools for BetaProject Planning Tools (ex. MS Project) – A beta is a project like any other, with milestones, activities, and resources.

Spreadsheets (ex. Excel, Google Docs) – Spreadsheets contain everything needed to extract actionable data from any beta test.

The Social Web (Portals, Social Networks, Forums) – Mined correctly, the Internet offers an endless collection of great beta candidates.

The Centercode Library – We’ve produced a number of educational resources and useful kits to help run successful beta tests, all available for free at www.centercode.com/library.


http://www.centercode.com/library/

WWW.CENTERCODE.COM


21


Beta Management


{


WWW.CENTERCODE.COM


22

The Beta Plan• Every good project begins with a plan. In the case of

beta, the plan ensures that the project parameters, goals, milestones, activities, and participant demographics are well-defined and communicated to everyone involved.

• We’ve developed two kits which include everything needed to develop a comprehensive beta plan. Each kit includes a complete walkthrough, a detailed sample, and a template to create your own plan.

Download the free kits:

• Hardware Beta Test Planning Kit

• Software Beta Test Planning Kit

Levantra Confidential Aurora 2.1 Beta Project

BETA TEST PLAN EXAMPLE 1

EXAMPLE HARDWARE BETA TEST PLAN TEMPLATE

Mobile Phone Beta Test Plan

Levantra, Inc.

Aurora 2.1 Beta Test

Prepared by Austin Stevens

ALL DATA IS FICTITIOUS


http://www.centercode.com/resources/hardwareplan

http://www.centercode.com/resources/softwareplan

WWW.CENTERCODE.COM


23

Writing a Beta Plan (1)While there are countless ways to structure a beta plan, at a very minimum any plan should include the following components.

1. Budget • Identify any costs which will be associated with the beta, including tools, beta units, shipping logistics, and incentives.

2. Product Definition • Construct an outline of the product itself, including information regarding its current state (alpha, beta, beta2, etc.), high-level modules, etc.

3. Test Parameters • Outline the basic scope of the project, including total number of beta testers, project schedule, and milestones.

4. Target Market Definition • Detail the target market for the product. If the market is segmented, break down the types of users into profile pools. Include the tester requirements (time, hardware, etc.) in this definition.


WWW.CENTERCODE.COM


24

Writing a Beta Plan (2)5. Test Goals • List the unique goals of the project, such as general quality improvement,

interface acceptance, testing real-world performance, testing support infrastructure, collecting customer suggestions, and recording testimonials. Also include the areas (modules) of the product that will be tested.

6. Project Stakeholders • Define the responsibilities of those involved in the project. Define how feedback will be managed as it changes hands and progresses.

7. Initial Testing Activities • Plan out the initial activities which will be performed throughout this beta (activities will likely change in response to the project progression). This may include items such as bug reporting, feature requests, forum discussions, survey completion, and assigned tasks.

8. Incentives • Plan out the awards for participation, as well as the participation levels which they will be based on.


WWW.CENTERCODE.COM


25

Q

A

How long should my beta last?

Beta tests generally run anywhere from 4-8 weeks of test time, often split between multiple phases with short breaks in between. With effective processes and tools, betas can produce great results in as little as 2 weeks. Beta length should be determined by the set of goals you are intending to accomplish. Depending on those goals you may also be able to reduce your test length by increasing your tester pool.

Frequently Asked Question


WWW.CENTERCODE.COM


26


Beta Management


{


WWW.CENTERCODE.COM


27

Your tester team is the most valuable resource in your beta test. This team can make or break your entire project. Therefore, finding the right testers is absolutely crucial.

Recruitment is generally split into two phases:

1. Advertise via various means to recruit a large pool of interested candidates. Candidates are generally asked to complete a qualification survey. If possible, it’s best for this pool to be 5-10x the size of your intended tester team.

2. Use whatever data you have available to select the most qualified best testers from the pool of candidates. The better job you do selecting effective testers, the fewer total participants you’ll actually need in your test.

Tester Recruitment 101


WWW.CENTERCODE.COM


28

Knowing why people become testers will help both recruit great participants and keep them engaged. Here are some of the primary reasons people test:

Problem Solvers • Many products (or new revisions of previous products) exist to solve a problem or fill a void of some sort. People feeling the pain of those problems are often eager to be the first to have them solved.

Early Adopters • Some people are enthusiasts who enjoy experiencing early versions of products. These testers love technology and are excited for the opportunity to carry even a minor role in its development process.

Raving Fans • Many people are passionate about a brand or product line, and are interested in having a chance to influence where those products go. Beta provides a rare opportunity to do so.

Career Minded • Business focused beta tests offer beta testers an opportunity to expand their skillset to include knowledge and use of products that aren’t even on the market, gaining them a competitive edge. Similarly, betas can provide an opportunity to network with peers.

Free Product • The opportunity to earn free products is one the most commonly recognized reasons to participate (but generally not what you’ll use for recruitment).

Why Do People Beta Test?


WWW.CENTERCODE.COM


29

The most readily available beta testers are often those closest to you. Unfortunately, these are generally not the best choices.

EmployeesYour co-workers are generally a bad choice for testers as they either have too much insight into the product itself or are impartial because of their relationship with the company.

FriendsUsually friends make bad testers because they are inclined to feel that their insider relationship negates the requirement to participate at the same level as other testers.

FamilyFamily members are also not the best choice for testers as their connection to you or your team may inhibit them from being objective.

Friends, Family, and Employees


WWW.CENTERCODE.COM


30

• The Internet opens the door to an endless supply of beta testers. It’s important to manage this source effectively to collect the right ones.

• Find portals/networks that interest the target market (every market has them). Ask an administrator to help advertise for you.

• Clearly indicate the technical requirements for each tester. Wading through unqualified candidates is a waste of time.

• Incentives deserve mention in a tester call, but be vague. Testers are most effective when selected based on excitement rather than reward.

• It is not necessary to advertise specific features of the product (especially anything confidential). Focus on the product category being tested.

• Be very clear and honest about the time requirements to participate in the project.

Advertising For Testers


WWW.CENTERCODE.COM


31

• Always keep your target market in mind. Selecting testers from the wrong market is the easiest way to produce useless results.

• Look for responsive and communicative behaviors. These traits will likely remain constant in the actual beta phase.

• Look for detail-oriented applications. Select candidates whose applications resemble the style that would ultimately be most valuable in feedback.

• Recognize those who are excited by your product.

• Avoid those who refer to or question test incentives immediately.

• If possible, include a few previously successful beta testers. They’ll appreciate the recognition and positively influence other testers.

Selecting Great Testers


WWW.CENTERCODE.COM


32

Q

A

How many testers should I use?

It varies. The size of your tester team should depend on your product, target market, prior experience, and the tools available. As a baseline, consumer products should generally be tested with anywhere from 50-200 people, while business products commonly include only 10-50. Many variables can change these numbers. Keep in mind that internal resources should drive tester team size as well.



WWW.CENTERCODE.COM


33


Beta Management


{


WWW.CENTERCODE.COM


34

Kicking off a successful beta test requires alignment of the following three key components:

• The product is in a state that is ready to be tested. The product is stable and near feature complete. General functionality should work, with the majority of testers not discovering unknown bugs in their first hour or so of testing.

• Testers are both ready and able to test your product. Testers have been selected, understand the schedule and what’s expected of them, and meet any technical or knowledge requirements to successfully test.

• The team is ready to engage and react in real-time. Stakeholders have been identified and educated, and are ready to interact with testers and feedback.

Failing to align any of these areas will greatly reduce feedback quantity and quality, ultimately reducing the value and accuracy of the beta test.

Free Resource: The following checklists are summarized from our Getting Ready for Beta Testing whitepaper.

Getting Ready for Beta

WWW.CENTERCODE.COM 1

For many companies, beta test preparation is fraught with uncertainty. Even knowing when you’ll start can be complicated. You might want to base the decision on QA milestones, ensuring the product is stable enough for users. But if you’re facing a product release window that can’t be missed, your beta schedule might be set by counting backwards on the calendar. The goal of this whitepaper is to establish a set of best practices for beta readiness (private beta tests, specifically). That way, you can be confident that you’re ready to launch an effective beta test regardless of any looming uncertainty.

In our experience, the best way to get ready for beta testing is to establish concrete guidelines in three areas: product readiness, team readiness, and tester readiness. If these areas are out of sync or underprepared, your beta will most likely suffer. But when they come together, they enable higher participation, easier management, and better results.

GETTING READY FORBETA TESTING


http://www.centercode.com/readiness/

WWW.CENTERCODE.COM


35

F The engineering team has verified that all components of the product are ready to begin beta testing.

F Auxiliary components (documentation, etc.) have been assembled into a single package which represents exactly what will be sent to testers.

F The out of the box experience has been successfully reviewed, including setup, installation, and documentation.

F Basic product functionality has been successfully reviewed (all key features are working) by product management.

F Known bugs which could not be addressed prior to beta have been clearly documented and communicated to testers.

F The uninstall process (if applicable) has been successfully verified.

Product Readiness Checklist


WWW.CENTERCODE.COM


36

F A sufficient number of testers meeting each required target market segment have been selected and notified.

F Accurate contact information and addresses (if testing hardware) have been verified for all testers.

F Non-disclosure and beta participation agreements have been explained in plain English, signed, and returned by all testers.

F Responsibilities and the project schedule have been clearly communicated to the tester team.

F Testers understand how to use the systems provided for feedback. If they aren’t dead simple, training or documentation has been provided.

F All resources needed by testers to carry out their responsibilities are easily accessible and easy to understand.

Tester Readiness Checklist


WWW.CENTERCODE.COM


37

F Core parameters and processes (e.g. testing goals, strategies and mechanisms for collecting feedback, categories for bugs, incentives, etc.) have been defined and communicated to all stakeholders and contributors.

F Milestones and deadlines have been communicated and understood and all necessary resources are readily available.

F Stakeholders have delivered all pre-test deliverables (e.g. tools, documentation, surveys, packaging, product keys, NDAs, beta units, etc.)

F Contingency plans have been defined for any stakeholders with limited availability (e.g. planned vacation, potential birth of a child, etc.)

F Any infrastructure that will be relied upon during the beta (beta test management tools, customer support, bug tracking, content delivery, servers, etc.) is accessible and has been tested.

Team Readiness Checklist


WWW.CENTERCODE.COM


38

Given the pre-release nature of beta products, most tests require a Non-Disclosure Agreement (NDA) forbidding participants from openly discussing their experiences with the product. Beta NDAs are similar to traditional NDAs, but generally include a few attributes unique to beta.

In addition to NDAs, it is common to execute a separate (or combined) agreement known as a Beta Participation Agreement (BPA). This agreement outlines the expectations for participating in the project.

Free Resource: We’ve developed a comprehensive Beta Test Agreement Kit, which includes NDA and BPA templates, along with an explanation of their contents.

Legal Agreements

Beta Test Agreement Kit Presented by Centercode


http://www.centercode.com/resources/nda/

WWW.CENTERCODE.COM


39


Beta Management


{


WWW.CENTERCODE.COM


40

There are a wide variety of activities your beta testers would be more than willing to participate in with the goal of improving your product. The activities most relevant to your product will vary based on the objectives outlined in the beta plan. Some of the most common activities are:

• Bug Reports

• Feature Requests

• Forum Discussions

• Assigned Tasks

• Daily Journals

• Surveys and Polls

• Testimonials

• Wiki Collaboration

• Tester Calls

• Site Visits

Over the next few pages we’ll cover each of these in more detail.

Beta Activities


WWW.CENTERCODE.COM


41

Bug reports are collected in nearly all beta tests. This is generally considered to be the most valuable aspect of beta testing, as correcting defects will lead to higher quality. The following are tips for collecting beta bugs:

• Bug report forms for beta testers are generally simpler than those completed by trained QA staff, but at a minimum should include a title, description and/or steps to reproduce, and severity (the tester’s view of severity may not match your own definition, but it’s important to capture the impression).

• Bug reports should allow testers to attach files (for screen shots, logs, etc.). These files often describe the bug more clearly or concisely than the tester may be able to (saving a great deal of time).

• If details about the tester’s environment (operating system, hardware configuration, etc.) are not available, this information should be collected. Note that collecting this information on every bug report will require repetition, ultimately reducing feedback (i.e. if possible, collect it up front and reference it).

• A comment log should be used to communicate back and forth with the tester (to collect other details).

• Testers should be encouraged to report bugs not only about the product, but also the documentation and any other auxiliary systems or services they are experiencing as part of the beta (support system, etc.). This feedback should be distributed to the appropriate stakeholders.

• Bug reports should flow into your development team’s defect tracking system (Jira, BugZilla, etc.). Automating this process saves a great deal of time, allows engineering to use familiar tools, and reduces the opportunity for crucial feedback to be lost.

Bug Reports


WWW.CENTERCODE.COM


42

Feature Requests• Given the near feature complete state of most beta tests, the goal of feature requests is to capture early

insight into customer requirements for future revisions of the product. In this sense feature requests are most useful for product planning.

• If an overwhelming majority of your testers are asking for a specific feature, it may be worth considering including that feature in the final version of the product. This is especially true in business product tests.

• It’s a good idea to include a disclaimer on a feature request form which claims ownership of any ideas generated by testers as part of the beta test.

Forum Discussions• Social forums in beta tests (1) Increase participation by providing testers with a sense of community, and a

reason to keep using and discussing the product; (2) Reduce beta management and support requirements by allowing testers to support each other; (3) Provide a simple mechanism for hosting directed discussions (e.g. an online focus group); and (4) Help reduce confidentiality liabilities by providing testers with a controlled outlet to express their excitement, reducing their urge to do so on the public Internet.

• While forums are a great tool in beta testing, they’re a very inefficient means of managing bugs and feature requests. It’s important to educate testers on the proper channels for different types of feedback.

Feature Requests and Forums


WWW.CENTERCODE.COM


43

Assigned Tasks• Assigning specific tasks to groups of testers is an excellent way to both achieve goals and increase

participation. Tasks can initially kickstart a project and encourage exploration. This can ensure coverage while giving testers an initial push to get started. Once the project is up and running, assigned tasks can be used for focused regression testing or to assess the impact of an issue.

• If used incorrectly, tasks can backfire and stifle exploration, more closely emulating QA testing as opposed to true customer beta testing. Early tasks should be focused on things that are valuable for all users to do, while for regression, tasks should be assigned to as few testers as possible to get the job done.

Daily Journals• Daily journals are simple forms that testers are expected to complete each day. Each one provides a short

overview of the tester’s recent experience with the product. If possible these should include a rating scale that allows you to focus on the highly negative or positive experiences.

• Daily journals are a great way to increase participation, as they give less social users an outlet to discuss their experiences with your product. They also provide a reason to return to your beta site, which will keep the tester engaged with updates, builds, etc.

Tasks and Journals


WWW.CENTERCODE.COM


44

Surveys and Polls• Surveys and polls are a great way to both collect feedback and keep testers engaged throughout

your beta test. These are most commonly used to capture opinions (about features, functionality, or performance), or to collect specific feedback about issues (bugs, etc.) identified throughout the project.

• Generally surveys should be limited to once per week, and less than 20 questions. If essay questions require a great deal of effort this should further limit the number of questions. Polls are simpler and may run multiple times per week.

Testimonials• Most beta testers are more than willing to share their excitement for your product. These forms should

include instructions encouraging the tester to be open and honest, followed by a freeform text box. If you’re looking for a specific length, it’s important to make this clear, as some testers will naturally be very short, while others will be overly verbose.

• Testimonials are generally collected toward the end of a beta test (possibly part of a final closure survey) and should include a disclaimer that describes how the testimonial will be used. Depending on your audience, it may also be useful to include a checkbox confirming use of the users name and/or company when referencing the testimonial.

Surveys and Testimonials


WWW.CENTERCODE.COM


45

Wikis• Collaborative wikis are a very contemporary beta activity where testers are allowed to either directly

edit or suggest improvements for future customer facing assets, most commonly including support (documentation, FAQs, etc.) and marketing materials. These are an excellent way to give customers another means of feedback, while directly contributing to the improvement of your product.

Tester Calls• Customer conference calls (including either single or multiple testers) offer direct real-time feedback,

similar to a focus group. These are generally scheduled both early and late in a beta test, offering the product team the ability to converse directly with customers prior to release. Customer calls also increase participation rates by demonstrating the high value of the beta tester and their feedback.

Site Visits• When possible, visiting a beta customer is a great way gain a first-hand understanding of the customer

experience. Beyond the natural benefits of a face-to-face conversation, customer visits allow product teams to watch customers perform tasks in their natural environment, providing valuable insight into real-world usage. Similar to customer calls, customer visits can increase tester participation.

Wikis, Calls, and Visits


WWW.CENTERCODE.COM


46

Achieving great beta tester participation rates is widely considered to be the most frustrating challenge when running a beta test. The following tips will help raise your participation rates.

1. Clearly express expectations early on and throughout

2. Consistently encourage confidentiality (require NDAs, etc.)

3. Balance gratitude with expectations

4. Keep testers involved with ongoing activities

5. Keep testers current regarding project progress

6. Offer a simple method for testers to provide feedback

7. Allow testers to communicate amongst themselves

8. Respond quickly to all issues and requests

9. Contact inactive testers directly (by phone if possible)

Free Resource: Check out our eBook on Reaching 90% Beta Participation.

Tips for Tester Management

Reaching 90% Beta ParticipationA Best Practices eBook for Getting the Most Out of Your Beta Program

by Centercode

www.centercode.com v1.1


http://www.centercode.com/resources/participation

WWW.CENTERCODE.COM


47

Q

A

How much time should testers dedicate?

This depends highly on the complexity of the product and goals of the project. Simple products may only require a few hours a week, while more complex products can require 10-20 hours a week or more. Keep in mind that a properly targeted beta tester should be fitting this product into their daily life, therefore testing shouldn’t feel like “work”.



WWW.CENTERCODE.COM


48


Beta Management


{


WWW.CENTERCODE.COM


49

We’re in the home stretch now. The following steps will help close out your test.

1. Give your testers time to submit final issues. Generally it’s best to give your testers at least three days’ notice before the end of a project.

2. Cut-off tester access to submit issues. Once your end date has been reached it’s important to cut off feedback.

3. Close all open issues. Review and close any remaining feedback. This should be done quickly in case additional information is needed before you lose touch with your testers.

4. Offer testers a simple means to return the product. If the product is hardware based, be sure to provide return instructions (testers should never bear any expense).

Keys to Closing a Beta Test


WWW.CENTERCODE.COM


50

Incentives are incredibly important to your beta test. You want to make sure the testers that provided you with all that amazing and critical feedback are well rewarded. Doing so is not only fair, but also sets a good example for the next time you need their help.

• Include multiple levels of incentives based on performance. Participation will likely vary heavily between your beta testers. It’s important to recognize and thank those who went above and beyond.

• Ensure incentives match participation requirements. If you expected a great deal of time out of your participants, the incentive should match that effort.

• Reward all who meet those levels. Generally a “top 10” approach is not as successful as levels. If everyone provided incredible feedback, everyone deserves an award.

• Distribute incentives quickly. The golden rule applies here: You expected your testers to participate immediately, so it’s only fair for you to do the same. If possible, have your incentives sent within a week of test closure.

• If possible, thank testers individually. A personal touch goes a long way.

Incentives!


WWW.CENTERCODE.COM


51

Now that your testers are happy and on their way, it’s time to go back to your stakeholders. A closure report provides an overview of the results of the beta test.

• Develop an executive summary of the project

• Document key issues found

• Document frequently requested features

• Document survey results (with pretty charts)

• Document top testers & incentives rewarded

• Ensure all departments obtain a report

Developing a Closure Report


WWW.CENTERCODE.COM


52


Beta Management


{


WWW.CENTERCODE.COM


53

1. A Detailed Plan • A solid plan provides a clear roadmap for everyone involved.

2. Universal Buy-In • A team that believes in beta will be much more willing to help.

3. Great Testers • The wrong testers can produce useless results.

4. Communication • Provide regular updates on timelines, responsibilities, and progress.

5. Responsiveness • Make testers feel involved as key members of the team.

6. Effective Tools • The right tools increase the efficiency of everyone involved.

7. Organization • Effectively manage the enormous amount of data generated in beta.

8. Great Incentives • Great testers deserve great rewards.

Free Resource: Check out our eBook, 100 Tips for Better Beta Tests, for a comprehensive collection of practical tips from Centercode and beta managers at top technology companies.

8 Elements for Beta Success


http://www.centercode.com/resources/100tips/

WWW.CENTERCODE.COM


54

We hope this resource has helped you on your quest to understanding beta. If you’d like to improve your betas further, we’d recommend taking a free trial run of our complete beta test management platform, Centercode Connect™.

• Recruit great candidates and select the best beta testers who meet your own specific target market.

• Collect and manage the feedback you need (including bugs, suggestions, tasks, surveys, forums, and wikis).

• Generate and automatically distribute custom reports based on your own beta objectives.

We offer a fully-featured free trial, available at www.centercode.com/trial

Next Steps


http://www.centercode.com/contact/trial/

WWW.CENTERCODE.COM


55

www.centercode.com • 800-705-6540 • [email protected] • @centercode

Banner image courtesy of Flickr user xjrlokix (Thanks!)


http://www.centercode.com/

mailto:info%40centercode.com?subject=

http://twitter.com/centercode

http://www.flickr.com/photos/xjrlokix/

Technology

What You Need to Know about Beta Management