2TestCompleteby SMARTBEAR

Introduction ............................................................................................4

User Expectations .................................................................................5

Methods of Testing for Video Games ....................................................7

Black Box Testing ..................................................................................8

White Box Testing ..................................................................................8

Automated Testing’s Role in Video Game Development .......................9

Video Game Testing on a Mobile Platform .......................................... 11Mobile ............................................................................................................. 11

Data ......................................................................................................... 11Network Availability .................................................................................. 12Gestures .................................................................................................. 12GUI ........................................................................................................... 12Portrait and Landscape Views ................................................................. 12Processing Power .................................................................................... 12Memory .................................................................................................... 13Battery ...................................................................................................... 13Fragmentation of devices ......................................................................... 13

Gaming ........................................................................................................... 14Audio ........................................................................................................ 14Graphics ................................................................................................... 14Accessibility ............................................................................................. 14Databases ................................................................................................ 14

Conclusion ...........................................................................................15

Contents

3TestCompleteby SMARTBEAR

IntroductionWe’ve all heard the stories of video game testers—recent high school graduates sitting in a room, playing video games for twelve hours straight, drinking Mountain Dew and documenting any bugs they can find. Many kids think of this as a dream job. But what they don’t understand is that it’s probably one of the most tedious, mind-numbing jobs one might fall into. As a software tester, it’s not par-ticularly the scene you want to think of when you imagine your day-to-day work life, yet this is the culture of testing video games. There isn’t any test automation tool that can help minimize the tedium. It’s all manual user acceptance testing and, the truth is, video game testing probably ruins video games for most of the people who attempt this voca-tion.

This eBook is about the exploding industry of mobile gam-ing and the reality of testing video games for mobile de-vices. If the nuances of mobile app testing and video game testing weren’t bad enough, combine the two and you have a perfect storm for any software tester. Even the most seasoned software tester might be hesitant to approach mobile gaming.

If you go into the Apple store or the Google Play store, ev-ery day there is a new list of games that have come to the market. A recent study by eMarketer showed that by 2015, approximately 162 million people will be playing games on mobile devices. To put that into perspective, it amounts to about half of the entire US population playing mobile games in some fashion. So, despite the fact that mobile gaming can be a testing nightmare, the benefits of produc-ing a hit game could outweigh the risks. The eMarketer

4TestCompleteby SMARTBEAR

study also claims that user growth could push revenues for the mobile gaming market to $1.78 billion in 2013. Clearly, there is money to be made!

User ExpectationsSo you’ve decided that developing a mobile game is a great way to make money. Before jumping right into game development, you must understand who your end us-ers are and what genre and platform caters to the users’ gaming needs. As with all software development, the user will be your best friend and worst enemy. By listening to your user’s feedback on design and mechanics you increase the chances of being successful. However, by ignoring the needs of your users, you risk alienating them and destroying your credibility as a video game developer.

Here is an example:

The main point of a Diablo game—what users came to love the Diablo games for—was the constant slaughtering of monsters and demons with the sole purpose of level-ing up, customizing your character, and finding cool loot to which you could sell off or trade with other players.

Yes, Diablo III had cool loot, the graphics were better, performance was fine, but there was something very wrong with this iteration of Diablo. With Diablo III, Blizzard intro-duced the auction house as a way to make real life and in-game money off of the rare loot a player could find, with

5TestCompleteby SMARTBEAR

a small fraction of all real-money transactions going into Blizzard’s pockets.

The issue here was that the gear a character needed to be strong enough to progress further in the game became a bottleneck when you had no choice but to use the auction house to get the gear your character needed to progress. Essentially, Blizzard’s attempt to add a feature that was supposed to be optional became the sole basis of the game. Users found themselves spending more time look-ing for gear on the auction house instead of going out and fighting monsters.

We are writing tell you about an important change to Diablo III: we’re going to be removing the gold and real-money auction house system from the game. When we initially designed and implemented the auction house system, the driving goal was to provide a convenient and secure system for trades. But after much review and player feedback, it became increasingly clear that despite the benefits of the AH system and the fact that many play-ers around the world use it, it ultimately undermines Dia-blo’s core game play: kill monsters to get cool loot. We’re working out the details of how the auction house system will be shut down, but we wanted to share the news as soon as we made the decision in order to give everyone as much advance notice as possible. Please note that the final shutdown will occur on March 18, 2014. We will keep everyone informed as we work through this process, but feel free to check out our blog post on the subject, and stay tuned to Diablo 3.com for further details.

The Diablo III team

6TestCompleteby SMARTBEAR

The point is that you must be in touch with whom you are creating the game for and the game must be tested not only for bugs but for fundamental issues, such as un-wanted features and/or requirements that are irrelevant to stakeholders. Sometimes, simplicity is the best approach. In the case of mobile games, you can always add features later via patching once you have a better idea what your users really want. However, this should not be an excuse for releasing buggy software and patching later since this will eliminate your chances at being reputable. Those bad reviews don’t go away after you release version 2.

Methods of Testing for Video GamesYou’ve built game requirements based on feedback from all stakeholders. The development team is about to start developing the game and the testing team is about to start testing the developers’ work. Welcome to the world of video game testing! Your users are going to pick apart your game and tell you everything that is wrong with it, so test for anything that would ruin the user experience.

Every gaming company tests their game differently, be-cause there are so many different genres of games and since software testing methods are always dependent on the context of the project. Despite the differences in testing methodologies from team to team there are two main cat-egories of testing that always seem to be used for testing games.

¿ Black box testing – Testing functionality without checking the code

¿ White box testing - Testing and examining code as opposed to functionality

7TestCompleteby SMARTBEAR

It is important to understand that these are generic test “strategies” and not “types” of testing. For instance, func-tional testing would be a specific type of testing that falls under the black box testing strategy. Likewise, unit testing would fall under white box testing.

Black Box TestingBlack box testing is the method of choice for testing video games. This testing strategy is done without access to the code. It is done by the tester to interpret how a user would play the video game in normal use. When you hear about the scenarios and picture a group of recent high school graduates in a room drinking Mountain Dew and playing and testing video games, they are black box testing.

Not only is black box testing an easy way to emulate a user, it is also a cheap way. Black box testers need mini-mal technical knowledge, but good hand-eye coordination from heavy gaming is highly sought after. You need testers who can play a game all the way through quickly and then play again going a different path through the game.

Black box testing is advantageous in that it can separate the player’s perspective from that of the developers. How-ever, since the tester does not see the code, if a defect is found it is harder for the developer to pinpoint the segment of code that is erroneous.

White Box TestingWhite box testing is testing with access to the code and is the technical side of video game testing. For example, unit testing falls under this testing strategy. White box testing

8TestCompleteby SMARTBEAR

is advantageous since it’s easier to see what inputs can be effective in testing. Cleaning up code that could cause bugs is done in this strategy.

Even though only a small amount of white-box testing directly relates to game play or user experience, it is still important in optimizing the code for better game perfor-mance and having better coverage overall for the gaming application.

Automated Testing’s Role in Video Game DevelopmentSo how about automated testing? Wouldn’t implementing a test automation tool such as TestComplete help increase velocity amongst video game testers? Well, yes and no.

Maybe your game has a database, like Diablo, in which there is a list of items and the possible properties and rarity those items can have. Using TestComplete for data-driven testing in a video game database would work wonders and save time for your team. It’s not that test automation can’t be used, but rather the context determines how it is used. Simply put, because there are so many variables in a video game such as user interaction, 3D environments and AI, a test automation tool could not easily emulate gameplay. Some companies even have used automated bots (AI to emulate users) to speed up the testing process. The problem with this approach is that, for example, a bot goes to click on an item in the game but then a creature runs in front of the bot as it goes to click on that item, the test will fail. A user would just move and click the item if the crea-ture got in the way.

9TestCompleteby SMARTBEAR

In another situation, you could request what amounts to a “temporary easter egg” from the dev team that your tool can use to “warp” right to the shop. This would allow you to set variables such as “player level” so your tool can check to see if the correct items are available for you based on those variables.

So it is important to realize that automated testing may not be applicable for all projects but that does not mean there aren’t ben-efits to using test automation for game development. Some of the main benefits are to understand the risks of game components and finding defects faster and earlier. Since video games are becoming massive in scale, test automation will also be a primary approach for regression testing as game code gets larger with each update.

Then there is always the case of the platform in which your game is developed for. TestCom-plete has been proven as a terrific tool for automated func-tional testing for HTML5 and Flash applications, and video games for those web technologies are no exception.

In actuality, developing games for HTML5 is becoming more popular due to the advancements of the technol-ogy as a multi-platform approach to gaming. For example,

10TestCompleteby SMARTBEAR

instead of porting a web game from the web to a mobile device, you could simply develop a game for HTML5. Since HTML5 is supported across almost every popular platform – from web to mobile - you would have a multiplat-form game without the hassle of porting the code over for each platform.

Video Game Testing on a Mobile PlatformVideo games and mobile devices have a lot in common when it comes to testing; they both have many variables the limit how testers can approach them. You can imagine that developing a game on a mobile platform is twice as difficult. Not only are you dealing with the nuances of video game testing, you basically throw a wrench at software testing with the variables involved in mobile development, for example, having no idea what the attributes are of the device your game will be deployed to—since the “most popular devices” on the day your game is released may not have even been announced when you start your test-ing. This is the main reason that mobile gaming is a work in progress and many games are based off of a simplistic GUI.

Below is a list of variables that need to be considered when developing a game for a mobile platform.

Mobile

Data

¿ How much data? What data?

¿ What are your privacy policies?

¿ What are you accessing on their device?

¿ What is your end users’ tolerance for privacy

11TestCompleteby SMARTBEAR

encroachment?

Network Availability

¿ How does your app perform on various carrier networks as opposed to Wifi?

¿ Will your app be a data hog that causes users to watch their bandwidth usage as a result?

¿ Is the performance on Wifi so markedly improved as to discourage the end user from using the app in any other environment?

Gestures

¿ Does your app give the user gesture options that are confusing or that create conflict with other apps?

¿ If they are not willing to adapt to the gestures you have incorporated in your app is the app navigable without them?

GUI

¿ Does your app have a command line feel to it or is it appealing to even the most non-technical of users (who will be a vast majority of most app audiences anyway)?

Portrait and Landscape Views

¿ What happens to your app when your end user turns his or her device on its side or on its head?

¿ Does it respond quickly?

¿ Is it responsive at all?

¿ Are there any fields that suddenly become impossible to navigate because of the limited space due to the keyboard expanding for use?

Processing Power

12TestCompleteby SMARTBEAR

The last thing you want your app to be associated with is making your user’s device so slow because it is a tre-mendous resource hog. If within those first few hours of downloading your app your customer notices that noth-ing is working as well on their device as it did before they downloaded your app then your app will be immediately removed.

Memory

Let’s be honest. Most mobile device users are pushing their memory limits to the nth degree on any given day.

¿ How will your app impact device memory?

Battery

What impact will your app have on the battery life of a mobile device? If an end user downloads your app then suddenly sees their battery life fall through the floor you may be viewed as a troublemaker. Not good.

Fragmentation of devices

There are an ever growing list of devices, screen sizes, form factors, etc. that make up the mobile marketplace.

It’s not cheap to make design alterations for every make and model. Also, once you make accommodations for one you may end up breaking something with another.

Will you need to pick and choose which devices are going to represent the largest portion of your target market then go from there? This is a critical decision that must be taken seriously.

13TestCompleteby SMARTBEAR

Gaming

Audio

¿ Do sound effects play at the correct time?

¿ Does music play at the correct time?

¿ Do sound effects and music combined cause cancellation at certain frequencies?

¿ Is the sound overall clear on the device in which it is playing?

Graphics

¿ Are devices that supposedly meet hardware requirements able to render graphics at a stable frame rate?

¿ Are there unwanted graphical errors appearing on the screen, such as, artifacts?

¿ Is the layering correct?

Accessibility

¿ Is your game accessible to your target market?

Databases

¿ Is the game accessing the correct data from the correct server?

14TestCompleteby SMARTBEAR

ConclusionI hope you now have a better idea what to look for when developing a mobile game. As stated before, there isn’t a right or wrong approach to testing a mobile game but you need to at least try to consider all the possible variables both physically and virtually. Sometimes it’s best to try to list every possible situation a player would be in when playing a game. Even if you do manage to identify every possible variable, scenario and their interactions, testing them will be a combination of targeting the moving parts of mobile apps and games. With practice and determination, you should be able to accomplish great testing on great new mobile games without breaking the budget or letting your competitors beat you to market, ensuring your games are easy to understand for the players… and easy on the eyes.

About SmartBear Software

More than one million developers, testers and operations profession-als use SmartBear tools to ensure the quality and performance of their APIs, desktop, mobile, Web and cloud-based applications. SmartBear products are easy to use and deploy, are affordable and available for trial at the website. Learn more about the company’s award-winning tools or join the active user community at http://www.smartbear.com, on Facebook or follow us on Twitter @smartbear and Google+.

SmartBear Software, Inc. 100 Cummings Center, Suite 234N Beverly, MA 01915 +1 978.236.7900 www.smartbear.com ©2013 by SmartBear Software, Inc. Specifications subject to change EBOK-TC-20131004-V1-GAMETESTING

Get started with automated testing

today

Try TestComplete