LambdaTest

Functional Scala Testing:LambdaTest

John Nestor 47 Degrees

www.47deg.com

https://github.com/47deg/LambdaTest

December 13, 2016

147deg.com

http://www.47deg.com

https://github.com/47deg/LambdaTest

http://47deg.com

47deg.com © Copyright 2016 47 Degrees

Outline

• Introduction

• Concepts

• Exceptions

• Immutable and Mutable

• ScalaCheck

• Options

• Tags

• Timing

• Running

• Extending

• Coming Soon: Asynchronous testing

2

http://47deg.com

Introduction

3


My Testing Philosophy

• Not a big system that tries to provide everything for everybody

• Instead a small clean system that is easy to extend and tailor from small simple customization through major custom testing systems

• Testing for developers not managers

• Test code should be high leverage

• Testing is programming not a list of specific simple tests

• Scala and functional programming is my focus

4

http://47deg.com


Existing Scala Scala Test Frameworks

• Java testing (JUnit)

• ScalaTest (400K lines)

• Specs2 (50K lines)

• UTest (< 1K lines, supports ScalaJS)

• ScalaCheck (property based testing)

5

http://47deg.com

http://junit.org/junit4/

http://www.scalatest.org/

https://etorreborre.github.io/specs2/

https://github.com/lihaoyi/utest

https://www.scalacheck.org/


Goals of New Scala Testing Framework

• Designed for developers not managers or PM’s

• Pure Scala. (No Java code)

• Purely functional (no mutable state)

• Small, simple, clean (~1K lines of Scala code)

• Easy to extend and customize

• Supports ScalaCheck

6

http://47deg.com


A Simple Example

• “com.fortyseven” % “lambda-test” % “1.1.2” % “test”

• import com.fortysevendeg.lambdatest._

• class Example extends LambdaTest { val act = … }

• Quickly show and run demo Example.scala. We will come back to it later to explain in detail how it works. For now, get a general feel.

7

http://47deg.com

Concepts

8


OO Versus Functional

• OO. Object contains mutable state. Operations change that state.

• Functional. Object contains only immutable state. Operations produce a new immutable state object.

9

http://47deg.com


Major Types

• States. An immutable value that represents each state during testing. Type LambdaState. There is an initial state before all tests and a final state after all tests are run.

• Transformations. A function that maps an old state to a new state. Type LambdaState => LambdaState.

• Actions. A sequence of zero or more transformations. Type LambdaAct. In simple case contains exactly one transform. Can be combined using infix +.

• Reporters. A way to report test results. Type LambdaReporter. Each state has a reporter. Reporter state is also immutable (but may have output side effects, see later).

• Options. Type LambdaOptions. Each state has its own options. Options are also immutable.

10

http://47deg.com


Overall State Transitions

11

S0 S1 S2 S3 S4 S5

S0 S1 S2 S3 S4 S5

Incremental Output

Held Output

http://47deg.com


Major Kinds of Actions

• Primitive actions

• Assertions: assert assertEq assertEx assertSC

• Exec: exec

• Compound actions (contain other actions)

• Test: test

• Other: label, nest

• Timing: timer, assertTime, assertPerf

• User defined: wrappers, …

12

http://47deg.com


Actions State Transitions

13

S1 S2

S1 S2

SA SB SC

Primitive Action

Compound Action

S1

http://47deg.com


Revisit Example

• Examine and run Example.scala. Now with more knowledge of how it works.

14

http://47deg.com

Exceptions

15


Handling Exceptions

• Expected exception. Use assertEx action. Flexible check option.

• Unexpected exception. Testing system, detects, reports and when possible recovers from unexpected exceptions and errors (for example ???).

16

http://47deg.com


Exception Example

• Examine and run Except.scala

17

http://47deg.com

Immutable and Mutable

18


Immutable and Mutable

• LambdaCheck is itself functional/immutable

• It is ideal for testing functional/immutable code.

• But it can also be used to test code that is mutable.

19

http://47deg.com


Putting Code Around a Sequence of Assertions

• Before all. In the body of a test or label. Can contain declarations visible in assertions.

• Between (mutable). Use an exec action. Can modify the state of declarations from before.

• Between (immutable). Use nesting.

• After all. User defined wrappers (more on this later).

20

http://47deg.com


Mutable and Immutable Code Examples

• Examine and run demo Mutable.scala.

• Examine and run demo Immutable.scala

21

http://47deg.com


Wrappers

• Replace the Before and After features of ScalaTest and Specs2.

• Purely functional/immutable. No need for the global mutable state needed in ScalaTest and Specs2.

• A wrapper is a user defined compound action.

• Wrappers can catch exceptions (for cleanup) and can hide some of its enclosing state from the assertions it wraps.

22

http://47deg.com


Wrapper Example

• Makes use of the eval operation on its body (of type LambdaAct).

• Examine and run Wrap.scala.

23

http://47deg.com

ScalaCheck

24


ScalaCheck Concepts

• Generators (generate a set of values)

• Random

• Specified

• Combined

• Properties (test an assertion on a set of values)

• forAll

• exists

• http://noelmarkham.github.io/practical-scalacheck/index.html#/

25

http://47deg.com

http://noelmarkham.github.io/practical-scalacheck/index.html#/


ScalaCheck Example

• Wrap in action assertSC

• Examine and run demo ScalaCheck.scala

26

http://47deg.com

Options

27


Options

• Default state: look at reference.conf

• Options can be overriden for any action.

• Look at and run Off demo

28

http://47deg.com

Tags

29


Tags

• Tags enable the selection of subsets of actions to be run

• Can be specified on label and test actions

• Are controlled by options

• Look at and run Tag demo

30

http://47deg.com

Timing

31


Timing

• timer: times a block of code

• assertTime: checks that code is fast enough

• assertPerf: runs code multiple times and reports mean, max and standard deviation. Can set limits on mean and max

• Look at and run Timing demo

32

http://47deg.com

Running

33


Running

• Directly. Companion object with main method calls run function

• Uses the StdoutLambdaReporter

• SBT. Can use any of the SBT test commands.

• Uses the SbtLambdaReporter

34

http://47deg.com


Reporters and Side-Effects

• A pure functional program should not have side effects.

• It is possible to defer all the report output side effects until after a purely functional testing core.

• But it is often nice to see report output incrementally as it is produced. So we sacrifice a little purity for convenience.

• There is however another special purely-functional side-effect-free reporter: HoldLambdaReporter. We will see a couple of its uses later in the talk.

35

http://47deg.com


Parallel Execution

• By default, tests are run and reported serially in the order written.

• The run function, label action and test action support an optional parallel parameter.

• Parallel actions are run on separate Scala futures but with the results still being reported in order written.

• To prevent mixing of output from different actions, all but the first are run using the hold reporter.

• Parallel logic is contained inside the eval method of LambdaAct.

36

http://47deg.com


Options

• Testing options are specified using Typesafe config.

• Examine reference.conf

• Change via API

37

http://47deg.com

Extending

38


Kinds of Extensions

• Generating actions

• Defining new actions

39

http://47deg.com


Generating Actions

• Implicit conversion Seq[LambdaAct] => LambdaAct

• Examine and run generate.scala

40

http://47deg.com


Layer Diagram

41

LambdaStateLambdaAct

+ Actions

User Tests

Testing Core

eval LambdaState API

http://47deg.com


Defining new Actions

• API to change state. Should not be called directly in user test code. Only used for extensions.

• LambdaAct eval changes state

• LambdaState method

• Examine ScalaDoc

• Examples

• Wrappers

• Examine predefined actions in lambdatest package object.

42

http://47deg.com

Coming Soon

43


Coming Soon: LambdaTestExpect

• Test that something that is expected happens

• Handles asynchronous actions

• Testing

• Futures

• Actor messages

• Akka events

• Log messages

• …

44

http://47deg.com


Expect

• Extends LambdaTest (no modifications needed!)

• It can itself be customized/extended

• Uses Akka Actors

• Two modes

• Poll: Expect a condition to become true

• Subscribe: Expect an event or pattern of events

• In either case there is a timeout

45

http://47deg.com


Expect

• Polling demo

• Log demo

• Actor demo

46

http://47deg.com

Questions

47

Software

LambdaTest