Game ProgrammingAutomated Testing in Games
Nick Prühs
Objectives
• To learn how to properly set up automated testing for your games
• To get an overview of common unit testing frameworks and tools
2 / 49
Unit Testing
• Method by which individual units of source code are tested to
determine if they are fit for use
• Unit of source code is the smallest testable part of an application
(e.g. method)
• Created by programmers during the development process
• Usually split up into three parts:
▪ Arrange
▪ Act
▪ Assert
3 / 49
Unit Testing
• Ideally, each test case is independent from the others
• Substitutes such as mocks can be used to assist testing a module in
isolation (e.g. database, mails)
• Can be implemented as part of automated builds
4 / 49
Unit Testing with NUnit
• Unit Testing framework for all .NET languages
• Initially ported from JUnit
• Written entirely in C#
• Stand-alone tools and R# integration
5 / 49
Setting up NUnit
1. Add new Class Library project to the solution.
2. Add reference to bin/framework/nunit.framework.dll.
6 / 49
NUnit Test Class Design
• Public
• Default constructor
7 / 49
NUnit Test Method Design
• [Test] attribute
• Return void
• No parameters
8 / 49
NUnit Test Example
C#
9 / 49
namespace LevelEditor.Tests{
using LevelEditor.Model;
using NUnit.Framework;
public class MapTest{
[Test]public void TestMapConstructor(){
// Arrange.const int Width = 32;const int Height = 16;
// Act.Map map = new Map(Width, Height);
// Assert.Assert.AreEqual(Width, map.Width);Assert.AreEqual(Height, map.Height);
}}
}
NUnit Assertions
• AreEqual, AreNotEqual
• AreSame, AreNotSame
• IsTrue, IsFalse
• Greater, GreaterOrEqual
• Less, LessOrEqual
• IsEmpty, IsNotEmpty
• IsNull, IsNotNull
• Contains
• Fail, Inconclusive
10 / 49
Expected Exceptions
C#
11 / 49
[Test]
[ExpectedException(typeof(ArgumentOutOfRangeException))]
public void TestNegativeWidth()
{
Map map = new Map(-10, 20);
}
SetUp and TearDown
C#
12 / 49
public class MapTest{
private const int Height = 16;private const int Width = 32;
private Map map;
[SetUp]public void SetUp(){
this.map = new Map(Width, Height);}
[Test]public void TestTileIndexer(){
// Arrange.const int X = 1;const int Y = 2;
// Act.var mapTile = new MapTile(X, Y, "Desert");this.map[X, Y] = mapTile;
// Assert.Assert.AreEqual(mapTile, this.map[X, Y]);
}}
Hint
Override Equals in types whose objects you need to compare!
13 / 49
NUnit Test Tool
14 / 49
NUnit Console
Console Output
15 / 49
D:\Dev\Repositories\SAE-ToolDevelopment\Vendor\NUnit-2.6.3\bin>nunit-console-x86.exe ..\..\..\Source\LevelEditor.Tests\bin\Debug\LevelEditor.Tests.dll
NUnit-Console version 2.6.3.13283
Copyright (C) 2002-2012 Charlie Poole.
Copyright (C) 2002-2004 James W. Newkirk, Michael C. Two, Alexei A. Vorontsov.
Copyright (C) 2000-2002 Philip Craig.
All Rights Reserved.
Runtime Environment -
OS Version: Microsoft Windows NT 6.2.9200.0
CLR Version: 2.0.50727.7905 ( Net 3.5 )
ProcessModel: Default DomainUsage: Single
Execution Runtime: net-3.5
...
Tests run: 3, Errors: 0, Failures: 0, Inconclusive: 0, Time: 0.046059608766156 seconds
Not run: 0, Invalid: 0, Ignored: 0, Skipped: 0
NUnit & Visual Studio
16 / 49
NUnit & Visual Studio
17 / 49
Advantages of Unit Testing
✓ Finds problems early
▪ Test Driven Development
✓ Facilitates changes
▪ Can be run before each commit or build
▪ In combination with source control, can identify the revision (and
originator) that broke the code
18 / 49
Limits of Unit Testing
• Won’t catch every error in the program
• Won’t catch integration errors
• Combinatorial problem
▪ Every boolean decision statement requires at least two tests
• Can’t test non-deterministic or concurrency problems
19 / 49
Limits of Unit Testing
• Setting up realistic and useful tests is a challenge
• Test case failures need to be reviewed daily and addressed
immediately
• Embedded system software presents a unique challenge
▪ Software is being developed on a different platform than the one
it will eventually run on
20 / 49
Test Driven Development
1. Write an (initially failing) automated test case that defines a desired
improvement or new function.
2. Produce the minimum amount of code required to pass that test.
3. Refactor the new code to acceptable standards.
21 / 49
Advantages of TDD
✓ Client-first development
✓ Taking small steps
✓ All written code is covered by at least one test
✓ Can lead to more modularized code
22 / 49
Limits of TDD
• Support of the entire team is essential
• Test are typically created by the developer who is writing the code
being tested, and may therefore share the same blind spots with the
code
• Maintenance overhead
23 / 49
Unity Test Tools
• Released by Unity in December 2013
• Partly integrated since Unity 5.3
• Completely open-source
• Available on the Asset Store and Bitbucket
• Based on NUnit
24 / 49
Unity Test Tools
• Unit tests are discovered using reflections
• Can be run automatically on recompile
26 / 49
Unity Test Tools
27 / 49Unity NUnit Test Runner Window
Unity Test Tools
• Integration tests allow testing integration of components, game
objects and assets
• Each test suite is a separate scene containing a game object with a
TestRunner attached
• Each test is a separate game object with a TestComponent attached
▪ Everything beneath that object in the hierarchy is considered to
belong to that test
• The CallTesting behaviour can modify the test result without having
to actually write additional code
28 / 49
Unity Test Tools
When you run the tests, the following steps are performed, in order:
1. Play mode is enabled.
2. The first or next test object is activated.
3. Wait until the test has finished (or a timeout has occurred).
4. The current active test gets deactivated.
5. If there are more tests in the queue, go to step 2.
6. Report results and finish test run.
29 / 49
Unity Test Tools
30 / 49Unity Integration Test Scene
Unity Test Tools
• Assertions check invariants – conditions you expect to be always
true
• You can specify when to check these conditions
• If any assertion fails, an exception is thrown
• You can automatically cause the game in that case by enabling
Error pause in the Unity console window
31 / 49
Unity Test Tools
32 / 49
Unity Assertion Component
Unity Test Tools
Just like with NUnit, Unity integration tests can be run from command
line:
"C:\Program Files (x86)\Unity\Editor\Unity.exe“
–batchmode
-projectPath D:\Temp\UnityTest
-executeMethod UnityTest.Batch.RunIntegrationTests
-testscenes=NpruehsScene
-targetPlatform=StandaloneWindows
-resultsFileDirectory=D:\Temp\Results
33 / 49
NSubstitute
• Creates substitutes for interfaces
• Saves you from having to use stubs, mocks, spies, test doubles
34 / 49
public interface ICalculator{
int Add(int a, int b);}
NSubstitute
• Creates substitutes for interfaces
• Saves you from having to use stubs, mocks, spies, test doubles
35 / 49
ICalculator calculator = Substitute.For<ICalculator>();
NSubstitute
• Allows you set up return values to method calls.
• Great for mocking database connections or email plugins.
36 / 49
calculator.Add(1, 2).Returns(3);
37 / 49
Unit Testing in C++
• googletest is a platform-independent framework for writing C++ tests
• Used for a variety of Google projects, including Chromium and
Google Protocol Buffers
• googlemock allows you to write and mock C++ mock classes
38 / 49
Unit Testing in C++
1. Create a new Win32 console application.
2. Add the projects you want to test to the additional include
directories.
3. Add the gtest root folder and gtest-a.b.c/include folder to the
additional include directories.
4. In your test source file, #include "src/gtest-all.cc“.
5. Provide a main entry point like this:
39 / 49
GTEST_API_ int main(int argc, char**argv){
testing::InitGoogleTest(&argc, argv);return RUN_ALL_TESTS();
}
Unit Testing in C++
Test method signatures are automatically generated by the TESTmacro:
40 / 49
TEST(TestSuiteName, TestName){
// Arrange.
// Act.
// Assert.}
Unit Testing in C++
Assertions are made using macros as well.
41 / 49
TEST(TestArithmetic, TestAdd){// Arrange.auto i = 3;auto j = 4;
// Act.auto k = i + j;
// Assert.EXPECT_EQ(7, k);
}
Unit Testing in C++
Run the tests by just starting the console application…
42 / 49
Unit Testing in C++
… or by using the UI of gtest-gbar.
43 / 49
Unit Testing in C++
googlemock provides macros for mocking methods.
44 / 49
#pragma once
#include "gmock/gmock.h"
class ICalculator{public:virtual int Add(int x, int y) = 0;
};
class MockCalculator : public ICalculator{public:MOCK_METHOD2(Add, int(int x, int y));
};
Unit Testing in C++
Just like NSubstitute, it allows you to specify results:
45 / 49
#include "src/gtest-all.cc"#include "src/gmock-all.cc"
#include "MockCalculator.h"
using ::testing::Return;
TEST(TestArithmetic, TestCalculator){
// Arrange.MockCalculator calculator;EXPECT_CALL(calculator, Add(3, 4))
.WillRepeatedly(Return(7));
// Act.auto sum = calculator.Add(3, 4);
// Assert.EXPECT_EQ(7, sum);
}
Unit Testing in UE 4
Unreal Engine 4 defines unit tests with macros, too:
46 / 49
IMPLEMENT_SIMPLE_AUTOMATION_TEST(FSetResTest, "Windows.SetResolution", ATF_Game)bool FSetResTest::RunTest(const FString& Parameters){
FString MapName = TEXT("AutomationTest");FEngineAutomationTestUtilities::LoadMap(MapName);
int32 ResX = GSystemSettings.ResX;int32 ResY = GSystemSettings.ResY;FString RestoreResolutionString =
FString::Printf(TEXT("setres %dx%d"), ResX, ResY);
ADD_LATENT_AUTOMATION_COMMAND(FEngineWaitLatentCommand(2.0f));ADD_LATENT_AUTOMATION_COMMAND(FExecStringLatentCommand(TEXT("setres 640x480")));ADD_LATENT_AUTOMATION_COMMAND(FEngineWaitLatentCommand(2.0f));ADD_LATENT_AUTOMATION_COMMAND(FExecStringLatentCommand(RestoreResolutionString));
return true;}
Unit Testing in UE 4
Tests can be run from the Unreal Frontend:
47 / 49
References
• Wikipedia. Unit testing. http://en.wikipedia.org/wiki/Unit_testing, April 25, 2017.
• Poole. NUnit Quick Start. http://www.nunit.org/index.php?p=quickStart&r=2.6.4, 2015.
• Wikipedia. Test-driven development. http://en.wikipedia.org/wiki/Test-driven_development, April 21, 2017.
• Petersen. Unity Test Tools Released. http://blogs.unity3d.com/2013/12/18/unity-test-tools-released/, December 18, 2013.
• Unity Technologies. Unity Test Tools. https://bitbucket.org/Unity-Technologies/unitytesttools, March 9, 2017.
• NSubstitute. NSubstitute. http://nsubstitute.github.io/help/getting-started/, May 2017.
• Google. Google C++ Testing Framework. https://code.google.com/p/googletest/, May 19, 2017.
• Google. Google Test UI. https://code.google.com/p/gtest-gbar/, May 2, 2017.
• Epic Games. Unreal Engine Automation Technical Guide. https://docs.unrealengine.com/latest/INT/Programming/Automation/TechnicalGuide/, May 2017.
• Epic Games. Unreal Engine Automation User Guide. https://docs.unrealengine.com/latest/INT/Programming/Automation/UserGuide/, May 2017.
48 / 49
5 Minute Review Session
• What are unit tests?
• What are the main advantages of unit testing?
• What are the most important limits of unit tests?
• Explain the process of Test Driven Development!
• What are the upsides and downsides of TDD?
• What are integration tests?
• How do you test functionality that depends on external
communication (e.g. database, mails)?
Recommended
