6 A/B Tests You Should Be Running In Your App

2

IntroductionEveryone in mobile believes in A/B testing. But if you’re wasting your days testing nothing more than various colors or screen layouts: you’re doing it wrong!

But we’re here to help. This practical guide lists six key tests that will deliver real ROI for your mobile business. Of course six is by no means the limit - we hope you’ll be inspired with ideas for A/B tests well into the future (our customers certainly are…!)

Anything can be A/B tested. But the fundamental question is what should be tested? We’ve worked with some of the world’s most successful app companies. We’ve created, deployed and analyzed thousands of A/B tests. And now we’re sharing our knowledge about the tests that can make a real difference to your app performance.

This guide will help you deliver ROI on the back of A/B testing - fast. You’ll be developing support for the testing philosophy and creating a truly data-driven business. And you’ll be implementing changes to the mobile app experience that will drive engagement, retention and revenues.

A huge number of organizations test at random - and see no results that make a material difference to their business. This guide, based on our own experience, will help you avoid that trap.

Best of luck

1 OnboardingThe statistics are startling. Something like 19% of all app downloads are used precisely once. And as most mobile businesses know, things don’t get much better from there on in. We live in an ecosystem in which throwing money into a leaky sieve is par for the course - but that has to change.

The first way to make that happen is to start A/B testing your onboarding process.

‘Onboarding’ in this context refers to the initial experience during which we aim to answer the three simple questions almost every user, in every app, has when they first fire it up. Those questions are:

How do I use it? Why am I using this app?

What other features are available to me?

Sounds relatively simple, but a surprising number of mobile apps don’t effectively answer those questions - and if users don’t get those answers, they leave.

On the basis of that conclusion, smart app developers know to create first-time up experiences that, at least in theory, provide the answers to these questions quickly - before mobile app users start leaking away!

3

4

But do they test them? In many cases, the answer is almost certainly ‘no’, which is a shame because A/B testing can provide guidance around a huge range of questions including:

How many screens are most effective inan onboarding process

What content and navigation works best within those screens

Whether delaying certain elements of the process might be a smarter approach, and whether an onboarding process is required at all

At Swrve, we worked with one customer on an onboarding test and found that the innovative, dynamic tutorial experience they had created was in fact less effective than a more traditional approach - to the tune of a 7% increase in user retention rates. That’s the kind of result you won’t get without testing. Make it happen!

Smart, helpful onboarding in Google Translate

“ “

2 Subscription Sales MessagesA large range of apps, whether in me-dia, entertainment, lifestyle - or indeed many other verticals - rely on converting users from a free trial period to a paid subscription to their mobile service. The ‘conversion rate’ from the first state to the second is the one single number that truly defines success and failure for these businesses.

So what’s the right way to move that number?

Well, before we go any further let’s quickly discuss the wrong way: sending a generic and un-tested message to all free trial users, every day until the trial runs out. That’s a recipe for simply irritating so many of them that they leave well before the trial is over, and even for those that remain, how effective is a generic marketing message anyway?

Clear and effective upselling in Path

“ “

5

6

Instead, a smart organization is identifying user segments within trial users, and creating offers and campaigns that speak to their specific behaviors and interests. In the case of a fitness app, for example, those who have logged a number of runs during trial would see campaigns focused on the deeper features for runners enjoyed by full members.

And these campaigns would be tested. Not just in terms of content (although content is important), but also in terms of creative and, perhaps most importantly, timing. Is it best to wait until 24 hours before a trial expires? Is it best to act early? How many campaigns is enough - without becoming an irritant?

These are precisely the kinds of questions that can only truly be answered with user data. And an A/B testing program will give you that data, and make a real difference to the number of trial users who sign up to the real thing.

an A/B testing program will give you that data, and make a real difference to the number of trial users who sign up to the real thing.

“

“

any given campaign can only ever reach half its potential audience and your push campaigns are only ever at half their potential effectiveness.

7

3 Push Notification Opt-In Timing (And Content)The effectiveness of your push campaigns is directly correlated with how many of your users are willing to receive them. That isn’t an academic point: at present opt-in rates for push are hovering at around the 50% mark, which means any given campaign can only ever reach half its potential audience and your push campaigns are only ever at half their potential effectiveness.

What causes such low opt-in rates? Well, the single biggest ‘killer’ in this context is asking for push opt-in before the app has fully launched, and without providing any information around how push will be used and most importantly what’s in it for the user. To use a direct analogy, this is equivalent to visiting a website for the first time only to see a pop-up obscuring your view and asking for permission to send you email. Would you agree to that? Precisely.

“

“

8

1 2

Deliver a ‘two-step’ opt-in process. Users are significantly more likely to opt-in to receive notifications if you first show an in-app messages that explains why you are asking them and what’s in it for them. The best examples both explain why you should opt-in, but also appear at a relevant point in the app, when a push notification will help the user achieve a specific goal.

Wait until users are familiar with your app before requesting a push opt-in. This builds trust, and makes it more likely the user has experienced enough value to give you push permission.

The good news is there’s two relatively straightforward ways to combat this issue and drive your opt-in rates upwards, namely:

Well-timed and relevant push opt-in request in Uber

“ “

How do you know when exactly to send these opt-in invitations? How do you know what content precisely will be most effective in delivering higher opt-in rates? You test!

9

4 Your App Store Rating CampaignThe rating of your app heavily influences its position in the App Store and users’ expectations of app quality and utility - and thus acquisition. So it should be a primary goal for you to identify your engaged and happy users and solicit a positive review from them.

The default method - simply asking for an app store rating from your entire user base at a random moment in time, really isn’t good enough. Think about it: do you really want to send people who may be having a bad time to the app store to deliver negative feedback in public? Of course not.

That’s why we recommend a two-step campaign. The first step is delivering a message asking users how they are enjoying their experience to date. Those users who indicate they are happy are then asked to rate the app. Those who are not can deliver that same negative feedback in private (which is smarter anyway - as you are far likely to see it and act on it that way).

do you really want to send people who may be having a bad time to the app store to deliver negative feedback in public? Of course not.

“

“

10

Of course the timing and content of that message are of vital importance. The former in particular should be heavily influenced by proximity to key ‘success moments’ in the app that - in theory at least - will leave users with a lovely warm glow!

And just as with the push notification opt-in message earlier, that content, and that timing, should be one of the first A/B tests on your list. Making these campaigns more effective means more reviews, and more positive reviews, on the app store. And that means a greater average rating and more acquisition.

In fact, working with social network if/we Swrve managed to deliver an increase from 2.3 stars to 4.2 - in less than two weeks! That can have a huge impact on any app business.

A textbook two-phase rating campaign in Hipmunk“

“

11

5 PricingMany app business rely on in-app purchases to deliver revenues. That’s obviously the case in games, but is also true in a number of other verticals, with dating and media being other areas experimenting with access via credits.

With that in mind, it is no surprise that the pricing of these in-app offers is often the first port-of-call for those taking a look at A/B testing in mobile apps. The logic is pretty straightforward - by testing various price points it is possible to find the ‘goldilocks’ spot that this not too much, nor too little, but ‘just right’ when it comes to maximize revenues.:

To get the perfect prices - test!“

“

12

Whilst that is true, there are some caveats to keep in front of mind when A/B testing prices, including

It can make more sense to test offers rather than prices. This avoids creating the perception that different users are being charged different prices for the same thing. Although this is unlikely to be a serious issue, it can arise, and by testing whether to offer 20% off or 30% off, or whether a 50% bonus or a 60% bonus is more effective, you make this less likely to be an issue.

Remember to judge the whole-app revenue result of an A/B test. It is obvious that by increasing the revenue spent on one item (the one being tested) you may simply cause a similar fall in revenues relating to others. Obvious - but you’d be surprised how many tests are conducted without checking these effects.

Try not to combine price tests with other campaigns. It’s OK to show offers in the store, but if you combine these with in-app messages you have multiple variables in play, and it can be hard to isolate those variables that are truly making a difference.

Don’t forget, of course, that offers and prices can be delivered only to certain segments of users. In fact, most sophisticated app businesses will take this approach, even if it is just to offer deep discounts to long-term users strangely immune to your charms!

6 Social Sharing InvitesIn the mobile app ecosystem, acquisition is expensive - and getting more expensive every day. So it makes sense to get the highest levels of organic acquisition (the kind you don’t pay for) that you possibly can.

Ultimately word-of-mouth isn’t something you can directly control, but you can certainly give it a helping hand by making social sharing something that is easy and even fun for users to do. At Swrve, we helped entertainment app Beamly increase social shares by over 500%, simply by changing the language with which they were requested! And it’s possible for anyone to do the same.

It’s that combination of a direct effect on key metrics and being responsive to change that marks out a real A/B testing candidate. And this one most definitely is!

Clear, direct sharing prompts in Evernote

“ “

13

14

Remember, there are many A/B tests you can run on social sharing invites. They might include:

When to show them - ask users to share when they’ve just achieved something, rather than just during the app experience. Check which is more effective.

Whether to offer an incentive - some apps offer credit or other rewards for social sharers. Experiment with this approach and see if it works for you (whilst remembering to factor the cost of those incentives into your ROI calculations!)

How to show them, and which social channels to include as options

The language used - we’ve seen that more direct requests can be more effec-tive than ‘softer’ approaches - but test and see what works for you

After a series of tests along these lines, you should be in a position to deliver killer social sharing functionality, that will make a real difference to your (free) acquisition.

ABOUT SWRVESwrve is mobile marketing automation.

Swrve is the world’s leader in driving engagement, retention and revenue in mobile apps and games.The Swrve platform delivers everything product and monetization managers need to ensure their mobile apps succeed where it counts. We do this by building long-lasting, profitable relationships with their users and driving the bottom line as a result.

Swrve includes:A/B testing - to optimize user experience and improve retention rates

Targeted in-app campaigns - delivering the right offers, to the right audience, at the right time

Push notifications - driving engagement even when users are outside the app

Real-time optimization - automatedcampaigns reacting in real-time to user behavior in the app

All the analytics and segmentation you’ll ever need

Swrve is trusted by some of the world’s largest and most successful mobile app businesses. We handle billions of events a day and process data in real-time for multi-million DAU titles.

If you’d like to join them, drop us a line at sales@swrve.com

www.swrve.com