Upload
sungmin-kim
View
233
Download
3
Tags:
Embed Size (px)
Citation preview
“Software Engineer in Test” in Game De-velopment
How can TERA be verified via test automation?
Sung Min Kim([email protected])
Lead Software Engineer in Test / Bluehole Studio
• I haven’t converted all contents in the original document into this English version yet, so many contents are not shown in this document, compared to the original version due to my laziness
• I don’t have a good English skill, so please understand if wrong sentences or grammars in this document. Thanks in advance for your understanding
• This document is an English version of the original presentation document used at KGC 2014 written in Ko-rean (http://www.slideshare.net/SungminKim12/kgc-2014-software-enginner-in-test-in )
• Hope that this document can help the people who are interested in the test automation development espe-cially for games for developing your test automation system
• Feel free to let me know if you have any questions on the document (Email: [email protected])
Session 1 (15:50 ~ 16:50)
1. About me2. Software Engineer in Test?
• Test Automation• SET at Bluehole Studio
3. TERA Test Framework• TeraPuppet• TeraTest• Selenium Integration
4. Q & A
Agenda
Session 2 (17:00 ~ 18:00)
1. Test Scenarios2. CI & Build System3. Performance Measurement4. Summary 5. Q & A
Agenda
• LG Electronics : 3 yearsProject: 4 mobile devices
About me
• A programmer who loves new developments ( Email: [email protected] )
Me
• Realtime Worlds, Client Programmer : 1 yearProject: My Worlds (Data Generation)
• Microsoft, SDET and SDE 2 : 5 yearsProject: Office 2010, Office 2013 (SharePoint Designer)Award: Gold Star Stock Awards (2011) with SPD Test Automation
• Bluehole Studio, Lead SET (2013.08 ~ Now) : 1.5 yearsProject: TERA (Test Automation )
Software Engineer in Test
Software Engineer in Test
• General jobs in many global game companies, but likely un-
common in Korea yet– Global game companies like Blizzard, Riot Games, and EA are now hiring
many SETs or similar names like that
– SET @ Blizzard, EA, Google, …
– SET @ Bluehole Studio
– SDET @ Epic Games, Microsoft Game Studios,…
• Programmer on Test Area (TEST ≠ QA)– Test Automation
– Verifying features via programming
– Finding code defects as early as possible at the development stage
– Detailed performance and security measurement
– Issue management on CI system
– Share various info with Devs and QA as a evangelist on test automation area
• A specialist working only on test area as a programmer like
server programmer and client programmer
Software Engineer in Test ?
Software Engineer in Test
• Test automation is a more efficient way if some features can
be verified via computer instead of people– Verification on regression areas
– Measure and verify exact values inside game logics like packet data be-
tween client and server
– Can find performance, security issues, code defects in detail
• Can have totally different directions on test automation system
according to development status, organization, project speci-
fication, and so on* Bluehole Studio
– Only one testable project now: TERA
– Organization: about 150 employees based on TERA
– Game Genre: MMORPG
– The number of countries where TERA is working on live mode : 6 (Korea,
China, Taiwan, Japan, Europe, and US)
– The number of SET at Bluehole Studio: only 1 (me.)
Software Engineer in Test ?
For software?
Software Engineer in Test
• Not possible to verify all features in Games via Test Auto-mation only
– Fun– UX– Complicated multi-player behaviors
• Co-work between developers and QAs is very important
• Need to always consider 1) Development efficiency, 2) Cov-erage improvement, 3) Development Process improvement
– Reduce development costs– Test Efficiency – Sustaining– Better development process
Software Engineer in Test ?
Fun
UX
SET at Bluehole Studio
Software Engineer in Test ?
• s
Efficient Test Automation
• Improve Quality• Easy to understand the system under test• Reduce the risk• Easy to run, and maintain• Easy to create scenarios• Minimum maintenance cost
Current point (TERA)
SET
QA and developers
Not to be like this
My SET History (2013.08 ~)
SET at Bluehole Studio
• What I have done so far alone as a SET
– Create SET and SET’s role at Bluehole studio
– Developing Tera Puppet 1.0 for TERA
– Developing TeraTest for TERA
– Selenium integration system development for Web automation
– Adopt JIRA into Bluehole Studio for better BTS system
– Tool development for test automation for QA
– CI system development for running test automation scenarios all the time
– Running test automation scenarios for various countries using the CI system
– Development Process improvement
– Game Client performance measure
:
SET at BHS
Software Engineer in Test ?
Designing and developing all test automation framework and tools for all games at Bluehole Studio
Directing test automation strategy with QA and Devs for better test automation works
Implement test automation libraries and tools for games
Develop and support unit tests for DEVs
Measure detailed performance based on source codes
Find code defects via security / performance test and related tool developments
Measure code coverage based on server / client source codes to find out exact performance issues in detail
12
Gathering requirements and Planning test automation framework
Designing and developing test automation framework
Executing test cases and supporting QA and developers
Maintenance
TERA vs Office 2010
Game (TERA) Application (MS Office 2010)
When started Live Service Before finalizing specs
Logic flow Asynchronous Synchronous
Verification Source Code + Data Source Code + Data
Target Fun + functionalities Functionalities only
Interaction between users
Need to verify with many players No
SET scale 1.6 years * 1 person More than 10 years * ? people
The number of re-lease countries
6 countries 30 countries
OS Windows 7 or Windows XP All kinds of OS including Mac
Purpose For fun For business
When accident happens
Path + reward Lawsuit (possibly)
Update iteration Fast Slow
SET at Bluehole Studio
TERA Test Framework
In-house vs External Solution
• Of course in-house development is much better than external solutions
• TERA Test framework is a in-house framework working only for TERA
• So, we are developing a test framework on actual product codes after fully under-standing source codes on both client side and server side
• when using external solution
• TERA 의 서버 & 클라이언트 코드를 완전히 이해하고 ,
실제 Product 코드 위에 테스트 자동화 시스템을 구축하자– 실제 Product 코드에서 개발
– 누구나 시나리오를 만들 수 있는 , 시나리오 제작 & 재생 방식의 통합된 프레임웍
– Unit Test 및 다양한 테스트 라이브러리 개발을 통한 추후 확장 용이
– .NET 을 이용하여 개발 생산성을 높이자
– 필요한 경우 공개 라이브러리를 활용하자 (e.g. Selenium)
TERA Test Framework
IN-HOUSE
EXTERNAL SOLUTION
Framework Overview
TERA Test Framework
TeraPuppet(Record & Playback for TERA)
- Scenario creation and run via recording & playback
- Everyone can easily create and review scenarios
- Easy to maintain scenarios- But, need to update and stabi-
lize grammars for new func-tionalities
Selenium Integration(Record & Playback for Web)
- Developed libraries and provided various templates
- Integrated with TeraPuppet- Need to implement more libraries- Useful for creating scenarios based
on Web
TeraTest(C# Library & Script)
- Working based on c# test library- Can use other fluent and strong
C# library- Useful for very complicated scenar-
ios using huge data sheets- Useful for the scenarios difficult to
be created by Tera Puppet- But, need to develop separated
libraries for specific scenarios
Puppeteer (BlueholeTestWeb) - Scenarios management and execution via Web
CI with the test automation (TeraTestAgent)- In-house CI system only for Tera
TERA
TeraPuppet
TERA Test Framework
17
• Recording & playback, using 1) sever-client packet, 2) UI event, 3) key event, 4) mouse event, and 5) custom actions
• XML based scenario
• COM(Component Object Model) server and client to communicate between TeraPuppet and Tera-Client
• May help developers debug issues when check-ing events and packets between server and client
• Need to know various grammars to use all func-tionalities
TeraPuppet
TERA Test Framework
18
DEMO
TeraTest
TERA Test Framework
19
• Control and verify TeraClient with C# scripts and li-braries
• Script based scenarios
• Need to implement Test APIs in the managed world (C#) from the native world (C++)
• Useful when verifying w/ various datasheets and complicated logics
TeraTest
TERA Test Framework
20
• WPF and MVVM to show UIs and C++/CLI to communicate between TeraClient and TeraTest
• May help developers create Unit Tests
• Various scenarios can be implemented though it’s not possible by TeraPuppet
• Under implementation with various libraries– BOT– Game UI verification based on Direct X
TeraTest
TERA Test Framework
21
DEMO
AutoIt vs Selenium
• Bluehole Studio decided to use Selenium because it’s more flexible with C#
TERA Test Framework
AutoIt Selenium
Difficulty - -
Expandability Middle High
Community Middle High
Good• Easy to use• Own libraries• Easy to modify
• Firefox IDE• Expandability!! using C#• Can create scenarios in various languages
(VBA, C#, Autoit..)
Bad• Not easy to integrate with C++ /
C#• Need to learn a language like C#
Selenium Integration
TERA Test Framework
• Selenium is a test automation library for Web
• Because Web is based on HTML and DOM, it’s easy to verify scenarios by automation, other than JavaScript area
• SET developed test automation libraries integrated with Tera-Puppet, and now it’s used on TERA web sites for administra-tors
• QA creates scenarios with the Selenium integration framework
• Because Selenium can run on Tera Puppet, we can create vari-ous complicated scenarios using both selenium and Tera Pup-pet
Selenium Integration
TERA Test Framework
DEMO
Q & A
Test Scenarios
Scenario Status for TERA Test Automation
Test Scenarios
• More than 400 scenarios are now working in 5 countries– Countries: Korea, China, Japan, Europe, US
– Purpose: Build Verification Test, Issue verification test, Release veri-
fication test, content verification test
– Target: Contents at live stage
– Tools: TeraPuppet, TeraTest, Selenium Integration
• QA is generating and sustaining scenarios continuously
• Roles– QA: Scenario generations
– SET: Scenario tool development, and provide sample scenarios and
libraires
• The scenarios can be executed by both manual and CI system TeraTestAgent (CI tool for Test Automation) regularly
FYI, How SET currently works with QA and Devs ?
Test Scenarios
• SET– Design and develop test frameworks, tools, and libraries– Provide scenario samples and templates using the developed libraries– Perform research to acquire new knowledge and provide solutions– Review scenarios created by QA– Share knowledge if needed through studies like C# Study
• QA– Create scenarios, and investigate scenario failures– Log and handle issues happening with the test automation scenarios– Review and verify the test frameworks, and provide valuable feedbacks
• DEV– Use scenario results before DEVs submit codes
– Using Puppet and TeraTest as their debugger tools
Scenario Example (TeraPuppet)
Test Scenarios
Scenario Example (TeraTest)
Test Scenarios
Scenario Example (Selenium)
Test Scenarios
Code Coverage
Coverage Measurement
• A way to know how many features and codes are covered exactly by test scenarios
• Need to measure exact coverages to figure out if the production can be released or not
Code Coverage
Performance Measurement
At Microsoft, measuring and improving code coverage is a key to decide if a product can be
released to market or not
Code Coverage in Game development
Performance Measurement
• Limitation only with Code Coverage– Code + Contents Data
– Client + Server + Contents Data
• Packet Coverage
– How many packets are verified between server and client commu-
nication
– Measuring not only Code Coverage but also packet coverage for
getting exact coverage
• Data Coverage
– Need to find out a way to measure how many contents data are
covered by scenarios Packet Coverage
Are you interested in Software Engineer in Test ?– [email protected] / [email protected]