Game Programming 06 - Automated Testing

Embed Size (px)

Text of Game Programming 06 - Automated Testing

  • Game ProgrammingAutomated Testing in Games

    Nick Prhs

  • Objectives

    To learn how to properly set up automated testing for your games

    To get an overview of common unit testingframeworks and tools

    2 / 49

  • Unit Testing

    Method by which individual units of source code are tested to determine if they are fit for use

    Unit of source code is the smallest testable part of an application (e.g. method)

    Created by programmers during the development process

    Usually split up into three parts: Arrange

    Act

    Assert

    3 / 49

  • Unit Testing

    Ideally, each test case is independent from the others

    Substitutes such as mocks can be used to assist testing a module in isolation (e.g. database, mails)

    Can be implemented as part of automated builds

    4 / 49

  • Advantages of Unit Testing

    Finds problems early Test Driven Development

    Facilitates changes Can be run before each commit or build

    In combination with source control, can identify the revision (and originator) that broke the code

    5 / 49

  • Limits of Unit Testing

    Wont catch every error in the program

    Wont catch integration errors

    Combinatorial problem Every boolean decision statement requires at least two

    tests

    Cant test non-deterministic or concurrency problems

    6 / 49

  • Limits of Unit Testing

    Setting up realistic and useful tests is a challenge

    Test case failures need to be reviewed daily and addressed immediately

    Embedded system software presents a unique challenge Software is being developed on a different platform than

    the one it will eventually run on

    7 / 49

  • Unit Testing with NUnit

    Unit Testing framework for all .NET languages

    Initially ported from JUnit

    Written entirely in C#

    Stand-alone tools and R# integration

    8 / 49

  • Setting up NUnit

    1. Add new Class Library project to the solution.

    2. Add reference to bin/framework/nunit.framework.dll.

    9 / 49

  • NUnit Test Class Design

    Public

    Default constructor

    10 / 49

  • NUnit Test Method Design

    [Test] attribute

    Return void

    No parameters

    11 / 49

  • NUnit Test Example

    C#

    12 / 49

    namespace LevelEditor.Tests{

    using LevelEditor.Model;

    using NUnit.Framework;

    public class MapTest{

    [Test]public void TestMapConstructor(){

    // Arrange.const int Width = 32;const int Height = 16;

    // Act.Map map = new Map(Width, Height);

    // Assert.Assert.AreEqual(Width, map.Width);Assert.AreEqual(Height, map.Height);

    }}

    }

  • NUnit Assertions

    AreEqual, AreNotEqual

    AreSame, AreNotSame

    IsTrue, IsFalse

    Greater, GreaterOrEqual

    Less, LessOrEqual

    IsEmpty, IsNotEmpty

    IsNull, IsNotNull

    Contains

    Fail, Inconclusive

    13 / 49

  • Expected Exceptions

    C#

    14 / 49

    [Test]

    [ExpectedException(typeof(ArgumentOutOfRangeException))]

    public void TestNegativeWidth()

    {

    Map map = new Map(-10, 20);

    }

  • SetUp and TearDown

    C#

    15 / 49

    public class MapTest{

    private const int Height = 16;private const int Width = 32;

    private Map map;

    [SetUp]public void SetUp(){

    this.map = new Map(Width, Height);}

    [Test]public void TestTileIndexer(){

    // Arrange.const int X = 1;const int Y = 2;

    // Act.var mapTile = new MapTile(X, Y, "Desert");this.map[X, Y] = mapTile;

    // Assert.Assert.AreEqual(mapTile, this.map[X, Y]);

    }}

  • Hint

    Override Equals in types whose objects you need to compare!

    16 / 49

  • NUnit Test Tool

    17 / 49

  • NUnit Console

    Console Output

    18 / 49

    D:\Dev\Repositories\SAE-ToolDevelopment\Vendor\NUnit-2.6.3\bin>nunit-console-x86.exe ..\..\..\Source\LevelEditor.Tests\bin\Debug\LevelEditor.Tests.dll

    NUnit-Console version 2.6.3.13283

    Copyright (C) 2002-2012 Charlie Poole.

    Copyright (C) 2002-2004 James W. Newkirk, Michael C. Two, Alexei A. Vorontsov.

    Copyright (C) 2000-2002 Philip Craig.

    All Rights Reserved.

    Runtime Environment -

    OS Version: Microsoft Windows NT 6.2.9200.0

    CLR Version: 2.0.50727.7905 ( Net 3.5 )

    ProcessModel: Default DomainUsage: Single

    Execution Runtime: net-3.5

    ...

    Tests run: 3, Errors: 0, Failures: 0, Inconclusive: 0, Time: 0.046059608766156 seconds

    Not run: 0, Invalid: 0, Ignored: 0, Skipped: 0

  • NUnit & Visual Studio

    19 / 49

  • NUnit & Visual Studio

    20 / 49

  • Test Driven Development

    1. Write an (initially failing) automated test case that defines a desired improvement or new function.

    2. Produce the minimum amount of code required to pass that test.

    3. Refactor the new code to acceptable standards.

    21 / 49

  • Advantages of TDD

    Client-first development

    Taking small steps

    All written code is covered by at least one test

    Can lead to more modularized code

    22 / 49

  • Limits of TDD

    Support of the entire team is essential

    Test are typically created by the developer who is writing the code being tested, and may therefore share the same blind spots with the code

    Maintenance overhead

    23 / 49

  • Unity Test Tools

    Released by Unity in December 2013

    Completely open-source

    Available on the Asset Store and Bitbucket

    Based on NUnit

    24 / 49

  • Unity Test Tools

    DEMO

    25 / 49

  • Unity Test Tools

    Unit tests are discovered using reflections

    Can be run automatically on recompile

    26 / 49

  • Unity Test Tools

    27 / 49Unity NUnit Test Runner Window

  • Unity Test Tools

    Integration tests allow testing integration of components, game objects and assets

    Each test suite is a separate scene containing a game object with a TestRunner attached

    Each test is a separate game object with a TestComponent attached Everything beneath that object in the hierarchy is

    considered to belong to that test

    The CallTesting behaviour can modify the test result without having to actually write additional code

    28 / 49

  • Unity Test Tools

    When you run the tests, the following steps are performed, in order:

    1. Play mode is enabled.

    2. The first or next test object is activated.

    3. Wait until the test has finished (or a timeout has occurred).

    4. The current active test gets deactivated.

    5. If there are more tests in the queue, go to step 2.

    6. Report results and finish test run.

    29 / 49

  • Unity Test Tools

    30 / 49Unity Integration Test Scene

  • Unity Test Tools

    Assertions check invariants conditions you expect to be always true

    You can specify when to check these conditions

    If any assertion fails, an exception is thrown

    You can automatically cause the game in that case by enabling Error pause in the Unity console window

    31 / 49

  • Unity Test Tools

    32 / 49

    Unity Assertion Component

  • Unity Test Tools

    Just like with NUnit, Unity integration tests can be run from command line:

    "C:\Program Files (x86)\Unity\Editor\Unity.exe

    batchmode

    -projectPath D:\Temp\UnityTest

    -executeMethod UnityTest.Batch.RunIntegrationTests

    -testscenes=NpruehsScene

    -targetPlatform=StandaloneWindows

    -resultsFileDirectory=D:\Temp\Results

    33 / 49

  • NSubstitute

    Creates substitutes for interfaces

    Saves you from having to use stubs, mocks, spies, test doubles

    34 / 49

    public interface ICalculator{

    int Add(int a, int b);}

  • NSubstitute

    Creates substitutes for interfaces

    Saves you from having to use stubs, mocks, spies, test doubles

    35 / 49

    ICalculator calculator = Substitute.For();

  • NSubstitute

    Allows you set up return values to method calls.

    Great for mocking database connections or email plugins.

    36 / 49

    calculator.Add(1, 2).Returns(3);

  • 37 / 49

  • Unit Testing in C++

    googletest is a platform-independent framework for writing C++ tests

    Used for a variety of Google projects, including Chromium and Google Protocol Buffers

    googlemock allows you to write and mock C++ mock classes

    38 / 49

  • Unit Testing in C++

    1. Create a new Win32 console application.

    2. Add the projects you want to test to the additional include directories.

    3. Add the gtest root folder and gtest-a.b.c/include folder to the additional include directories.

    4. In your test source file, #include "src/gtest-all.cc.

    5. Provide a main entry point like this:

    39 / 49

    GTEST_API_ int main(int argc, char **argv){testing::InitGoogleTest(&argc, argv);return RUN_ALL_TESTS();

    }

  • Unit Testing in C++

    Test method signatures are automatically generated by the TEST macro:

    40 / 49

    TEST(TestSuiteName, TestName){

    // Arrange.

    // Act.

    // Assert.}

  • Unit Testing in C++

    Assertions are made using macros as well.

    41 / 49

    TEST(TestArithmetic, TestAdd){

    // Arrange.auto i = 3;auto j = 4;

    // Act.auto k = i + j;

    // Assert.EXPECT_EQ(7, k);

    }

  • Unit Testing in C++

    Run the tests by just starting the console application

    42 / 49

  • Unit Testing in C++