Embed Size (px)
DESCRIPTION
Gaming is a multibillion-dollar industry, and good testing is critical to any game’s success. Game testing has traditionally been black-box through the client—a method clearly insufficient with increasingly more complex software incorporating 3D physics, thousands of linked and interacting assets, large databases, and client-server architecture. Automation is an obvious answer, but how do you automate when the user interface is an immersive virtual environment, the data is as vital a part of the software as the code (and actually more likely to create bugs), and the games themselves are often built specifically to prevent automation? Brett Roark describes how Blizzard Entertainment is meeting this challenge by using automation to tame complex asset pipelines and building custom tools that make each tester more efficient. Take away a deeper understanding of the unique complexities of modern game testing, see why they require fresh and creative solutions, and consider how these solutions might apply to non-game testing.
Citation preview
T6 Concurrent Class
10/3/2013 9:45:00 AM
"Test Automation Challenges in
the Gaming Industry"
Presented by:
Brett Roark
Blizzard Entertainment
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Brett Roark
Blizzard Entertainment
A gaming industry professional with more than fifteen years as a tester, developer, and
producer, Brett Roark has shipped multiple AAA titles as well as numerous casual games.
Before joining Blizzard Entertainment in 2010, Brett had spent more than a decade at Microsoft
as a software design engineer in test. While working on Windows 7, he also taught game design
and production at DigiPen.
9/6/2013
1
Test Automation Challenges
In The Gaming Industry
©2013 Blizzard Entertainment, Inc. All rights reserved.
9/6/2013
2
Brett Roark
• Senior Producer, Test
Automation
• 15 years game industry
experience
– Tester
– Developer
– Producer
Agenda
• Introduction
• Why testing games is
challenging
• Which test functions
should be automated
• How Blizzard built an
automation test team
9/6/2013
3
Blizzard Product Lines
9/6/2013
4
Core Values
Blizzard “polish” doesn’t just
refer to our gameplay
experiences, but to every
aspect of our jobs. Approach
each task carefully and
seriously. Seek honest feedback
from your peers and use it to
improve the quality of your
work. At the end of the day, no
one will remember if the game
was late – only that it was great.
9/6/2013
5
Why Testing Games is
Challenging
Software Complexity
• Large development teams
• Long maintenance periods
• Shared technologies
• Computer science problems
– Distributed computing
– 32/64 bit conversions
– Complex math for physics and 3D
graphics
– Artificial intelligence
9/6/2013
6
Software Complexity
• 3D environments
• Audio and cinematics
• Localization
• Allowing user interface
customization
Multiplayer Interactions
• Adds complexity similar to multi-
threaded vs. single-threaded
programming
• Consistency of experience
• Latency sensitivity
9/6/2013
7
Data
• Games often data heavy
• Designers solve problems by using
game systems in unexpected ways
• Data source control difficulties
Which Test Functions
Should Be Automated?
9/6/2013
8
Benefits of Automation
• Increase coverage to
mitigate risk
• Improve process
efficiency
• Find bugs earlier
Content that’s Difficult to
Automate
• Usability
• Graphic style
• Lore and story
• Game balance
• Difficulty
• FUN!
9/6/2013
9
Areas to Consider
• Clients
• Servers
• Data
• Tools
Client Automation
• Challenges
– Frequent user interface changes
– 3D environments
– Automating around anti-cheat code
• Useful targets
– Client performance
– Compatibility suites
– Regression for expansions
9/6/2013
10
Server Testing
• Functionality and APIs
• Error injection
• Stress testing
– Performance under stress
– User experience under stress
– Fault tolerance and recovery
Data Validation
• Content pipelines
• Data diffs
• Delivery and patching
• Graphics systems
– Shaders
– Lighting
– Hardware driver updates
9/6/2013
11
Tools
• Help make QA and/or development
processes more efficient
• Examples:
– Quest tracker
– Arena teams
– In-game bug reporting
How Blizzard Built an
Automation Team
9/6/2013
12
Blizzard Structure
• Multiple development teams
• Shared platform and technologies
through Battle.net
• QA is a separate division that
supports all development teams
Automation Team Structure
Each automation team consists of:
– An engineering lead
– A producer
– Automation engineers
9/6/2013
13
Building Momentum
• Hired talented engineers who are
a cultural fit
• Started with Battle.net
• Expanded to products in early
development
• Built tools to increase efficiency
Talent Development
• Internal training team
• Robust onboarding program
• Multi-tiered training programs
– Leadership
– Project Management
– Testing
• Professional certifications
– ISTQB Foundation and Advanced Levels
– Scrum Master/Product Owner
9/6/2013
14
Summary
• Games have become incredibly complex
• Some content is difficult to automate
• Identify key areas for automation
• Build momentum with initial successes
• Invest in your people
Questions?
