Model-based Testing:Taking BDD/ATDD to the Next Level

Embed Size (px)

DESCRIPTION

Acceptance Test Driven Development (ATDD) and Behavior Driven Development (BDD) are well-established Agile practices that rely on developer intuition to identify and then translate statements into test suites. But the resulting test suites typically cover only a minuscule slice of happy-path behavior. And, as the test code base grows over time, work to maintain it either crowds out new development/testing or, typically, is simply ignored. Either is high-risk. That’s how Agile teams get eaten by the testing BackBlob. Using the Chat Server case study, this presentation provides an overview of a typical BDD tool (SPecFlow). Using a demo of SpecExplorer, a free model-based testing tool, suggests how model-based testing can both achieve better testing and greatly reduce the testing backblob.

Citation preview

  • Model-based Testing: Taking BDD/ATDD to the Next Level

    Robert V. Binder System Verification Associates

    [email protected]

    #MoreModelsLessTests

    Chicago Quality Assurance Association February 25, 2014

  • Overview

    Model Driven Development: Taking BDD/ATDD to the Next Level 1

    The Incredible

    Vastness of

    Testing

    Eaten Alive by the

    Testing Backblob

    Tell Me

    A Story

    More Models,

    Less Tests

    Model-

    Based

    Testing

  • THE INCREDIBLE VASTNESS OF TESTING

    The Incredible

    Vastness of Testing

  • Saved by Revealing Sub-domains

    Grains of sand to fill the Grand Canyon

    ~ 1.68 1025

    ~ 10 septillion

    Trivial Program

    Data space: 21024

    Sequences: 21024

    Execution space: 22048

    1.34 10616

    Large subsets of any programs execution space are revealing: if a bug exists in any member of that space, the space is reached it will trigger an observable failure. But, we rarely know exactly which sub-domains are revealing until we test.

    Model Driven Development: Taking BDD/ATDD to the Next Level 4

  • Its release time. Do you know where your bugs are?

    Average bugs per thousand equivalent source code lines, post-release

    Its release time. Do you know where your bugs are?

    Mobile

    IT

    Medical

    Aerospace

    6.0

    4.4

    1.8

    0.8

    6.0

    5.0

    1.9

    0.7 Other Agile

    Best In Class 4.7

    Capers Jones statistics: 5 defects/FP, 95% Removal. Assumes 53 Java LOC/FP .

    Model Driven Development: Taking BDD/ATDD to the Next Level 5

  • Predicted total latent bugs, 500 KLOC Java

    IT

    Medical

    Aerospace

    2202

    875

    375

    2511

    925

    325 Other Agile

    Its release time. Do you know where your bugs are?

    Its release time. Do you know where your bugs are?

    Best In Class 2350

    Capers Jones statistics: 5 defects/FP, 95% Removal. Assumes 53 Java LOC/FP .

    Model Driven Development: Taking BDD/ATDD to the Next Level 6

  • Driven to Distraction

    What is it?

    How does it work?

    What can it do for me?

    Test Driven Development TDD

    Behavior Driven Development BDD

    Acceptance Test Driven Development ATDD

    Model-based Testing MBT

    Model Driven Development: Taking BDD/ATDD to the Next Level 7

  • EATEN ALIVE BY THE TESTING BACKBLOB

    Eaten Alive

    by the

    Testing

    Backblob

  • Where does the time go? Total

    Number of Test Cases

    Sprint 1 Sprint 2 Sprint 3 Sprint 4

    Available Dev/TestTime

    Manual Test Cases not executed

    Automated Test Cases not

    maintained

    Model Driven Development: Taking BDD/ATDD to the Next Level 9

  • The Attack of the BackBlob

  • TELL ME A STORY

    Tell Me a Story

  • The Chat Server

    Logon

    List active users

    Post a message

    Broadcast message

    Logoff

    Client

    Endpoint

    Client

    Endpoint

    Chat

    Server

    !!!

    Model Driven Development: Taking BDD/ATDD to the Next Level 13

  • TDD BDD ATDD

    Programming strategies

    Test suites are scaffolding

    Test each method/feature at least once

    Rerun after any add/change

    Not testing strategies

    Test Driven Development

    Behavior Driven Development

    Acceptance Test Driven Development

    Model Driven Development: Taking BDD/ATDD to the Next Level 14

  • The TDD Red-Green Cycle

    Logon

    GetStatus

    Logoff

    Logon

    GetStatus

    Logoff

    Imagine a class and its responsibilities

    Code its API only

    Model Driven Development: Taking BDD/ATDD to the Next Level 15

  • The Red-Green Cycle

    TestLogon Logon

    GetStatus

    Logoff

    Imagine a class and its responsibilities

    Code its API only

    Write and run a simple test it should fail

    Model Driven Development: Taking BDD/ATDD to the Next Level 16

  • The Red-Green Cycle

    TestLogon

    GetStatus

    Logoff

    Logon

    Imagine a class and its responsibilities

    Code its API only

    Write and run a simple test it should fail

    Code implementation

    Rerun all tests expect pass

    Model Driven Development: Taking BDD/ATDD to the Next Level 17

  • The Red-Green Cycle

    TestLogon

    TestGetStatus

    Logon

    GetStatus

    Logoff

    Imagine a class and its responsibilities

    Code its API only

    Write and run a simple test it should fail

    Code implementation

    Rerun all tests expect pass

    Repeat until feature complete

    Model Driven Development: Taking BDD/ATDD to the Next Level 18

  • The Red-Green Cycle

    TestLogon

    TestGetStatus

    Logon

    GetStatus

    Imagine a class and its responsibilities

    Code its API only

    Write and run a simple test it should fail

    Code implementation

    Rerun all tests expect pass

    Repeat until feature complete

    TestLogoff Logoff

    Model Driven Development: Taking BDD/ATDD to the Next Level 19

  • TDD Limitations

    No explicit linkage for user/customer needs, requirements, or stories

    Gaps

    UI automation

    Feature testing

    Components, Test Objects

    Process, Activities

    Model Driven Development: Taking BDD/ATDD to the Next Level 20

  • Components

    *Unit

    Test Objects

    *DD Tool

    Behavior Driven Development

    Feature Files Scenario A

    Scenario B

    Scenario C

    Features, Scenarios

    Components, Test Objects Process,

    Activities

    Model Driven Development: Taking BDD/ATDD to the Next Level 21

  • System Under Test

    *Unit

    Test Objects

    Test Tooling

    Test Scripts

    *DD Tool

    Acceptance Test Driven Development

    Feature Files Scenario A

    Scenario B

    Scenario C

    Features, Scenarios

    UI Test Cases

    Process, Activities

    Model Driven Development: Taking BDD/ATDD to the Next Level 22

  • Features and Scenarios

    Feature: story

    Scenario: instance of a story

    Steps:

    Given: pre-condition

    When: trigger

    Then: verification condition

    Feature: Logon In order to use the chat server, As a validated chat client, I want to submit my credentials and establish a session

    Scenario: Submit Good Credentials Given A connection to the server is open When I send the Login message Then I should see the "Hi Chatter" message

    Model Driven Development: Taking BDD/ATDD to the Next Level 23

  • Overview Enter features, scenarios

    Generates skeleton test code for each step Extracts step names

    Can run immediately

    Developer adds step code

    Test run reports

    CI integration

    Model Driven Development: Taking BDD/ATDD to the Next Level 24

    Components

    Nunit Visual Studio

    Test Objects

    SpecFlow

    Feature Files Scenario A

    Scenario B

    Scenario C

    http://www.specflow.org

  • 1. Create a Feature

    As a

    I want to

    So that

    Model Driven Development: Taking BDD/ATDD to the Next Level 25

  • 2. Add Scenarios and Steps

    Given Precondition

    Data State

    When Trigger

    API call, GUI action

    Then Verification

    conditions (expected results)

    Model Driven Development: Taking BDD/ATDD to the Next Level 26

  • 3. Add code for steps

    Model Driven Development: Taking BDD/ATDD to the Next Level 27

  • THE VERY MODEL OF A MODERN

    Model-

    based

    Testing

  • What is a Model?

    Focus

    Smaller, simpler

    In IT systems

    Behavior

    Structure

    Content

    Model Driven Development: Taking BDD/ATDD to the Next Level 29

  • IT Models

    Behavior, Structure, Content

    Technology Process, Activities

    Model Driven Development: Taking BDD/ATDD to the Next Level 30

  • IT Models

    Notional (design)

    As built (documentation)

  • Test Models Focus on aspects that

    support testing goals

    Behavior

    Sequence and time

    Transformation

    Input, output function

    Verification (oracle)

    Coverage

    Risk Management

    Model Driven Development: Taking BDD/ATDD to the Next Level 32

  • Chat Server Logon

    Must be first Cant be repeated

    GetStatus Only after logon Returns number of calls since

    logon

    Logoff Only after logon

    Typical RFC status codes

    Chat Server

    Logon

    GetStatus

    Logoff

    Model Driven Development: Taking BDD/ATDD to the Next Level 33

  • Simple Test Model nc = 0

    GetStatus

    Logoff

    Logoff

    Logoff

    assert (401) assert (401) assert (200)

    assert (200)

    assert (200)

    nc++ GetStatus

    assert (200) assert (n == nc)

    Logon Logon

    assert (401)

    System Under Test (SUT)

    Logon

    GetStatus

    Logoff

    Model Driven Development: Taking BDD/ATDD to the Next Level 34

  • Chat Server Message Sequence

    Login a 1234

    Hi Chatter a

    Login b secret

    Hi Chatter b

    List Users

    Users: a, b

    Client Endpoint Chat Server

    !!!

    Model Driven Development: Taking BDD/ATDD to the Next Level 35 Client Endpoint

  • Chat Server Message Sequence

    Ack a1 Pumpkin Latte #mypics

    Post Pumpkin Latte #mypics

    List Users

    Users: a, b

    Client Endpoint Client Endpoint Chat Server

    Bcast a1 Pumpkin Latte #mypics

    Logoff b

    Ack Logoff b !!!

    Model Driven Development: Taking BDD/ATDD to the Next Level 36

  • Spec Explorer Demo Model-based testing tool

    Microsoft Research

    Used to test all MSFT APIs

    Robust and stable

    Visual Studio power tool

    C# code, not cartoons

    Generates standalone executable test suite

    Model Driven Development: Taking BDD/ATDD to the Next Level 37

  • 1. Create Test Model Actions

    [Rule] tags a method for exploration

    Requires defines a exploration precondition must be true to explore

    Model Driven Development: Taking BDD/ATDD to the Next Level 38

  • 2. Define Accepting State(s)

    Tag condition(s) that terminate an exploration

    Model Driven Development: Taking BDD/ATDD to the Next Level 39

  • 3. Create Cord File

    Set options

    Define machines

    Limits Actions used in an exploration

    Similar to regular expression

    Composable

    Very powerful!

    Model Driven Development: Taking BDD/ATDD to the Next Level 40

  • 4. Explore the Model

    Select any explorable machine Explore

    Generate test suite

    Run on-the-fly test

    Model Driven Development: Taking BDD/ATDD to the Next Level 41

  • 5. Review Exploration Graph

    Shows action/event sequence

    Ellipse: model state computed by model program machine

    Diamond: observation event get something from SUT (input to the model program)

    Right click to inspect any item

    Model Driven Development: Taking BDD/ATDD to the Next Level 42

  • 6. Define machine(s) to generate test suites

    Set options to generate test code

    Use any machine or composite machine

    Model Driven Development: Taking BDD/ATDD to the Next Level 43

  • 7. Explore the Test Suite machine

    Each root to leaf path becomes a test suite

    Model Driven Development: Taking BDD/ATDD to the Next Level 44

  • 8. Generate Test Code

    Explores the test machine

    Writes out test code using VS testing framework

    Model Driven Development: Taking BDD/ATDD to the Next Level 45

  • 9. Generate Test Code

    Not intended to be maintained

    Re-generate any time

    Add helper code in

    the model to extend/customize Model Driven Development: Taking BDD/ATDD to the Next Level 46

  • 10. Run the Test Suite

    Uses built-in Visual Studio test runner

    Adapters

    start/stop chat clients and chat server

    Other SUT setup/cleanup

    Test code sends messages, checks replies

    Model Driven Development: Taking BDD/ATDD to the Next Level 47

  • 11. Review Test Results

    Detailed run log VS test runner

    States

    Events

    Requirements

    Add any other programmable function

    Model State (test case inputs)

    Event Handler (pass/fail)

    Requirement Trace

    Model Driven Development: Taking BDD/ATDD to the Next Level 48

  • Exploration Interactive model development Generate test suites

    Spec Explorer Machines define composable

    Action sequences

    Options control exploration and data generation

    Actions are explored

    Rules define exploration conditions

    Events define expected SUT response

    Data structures support exploration

    Adapters abstract, control, and observe any interface, any platform

    Model Program

    Cord Declarations

    Machines

    Options

    Actions Rules

    Events

    Model Data Structures

    Adapters

    Model Driven Development: Taking BDD/ATDD to the Next Level 49

  • MBT Driven Development

    SUT

    Requirements, Stories, Scenarios...

    Bug

    Ambiguous, missing, contradictory, incorrect, redundant, incomplete

    Missing, incorrect

    Expected Outputs (Test Oracle)

    Inputs (Test Sequences)

    Run

    Control Observe

    Evaluate

    Generate

    Test Model

    Test Suite

    Coverage Requirements Model

    Model error, omission

    Explore

    Model Driven Development: Taking BDD/ATDD to the Next Level 50

  • MORE MODELS, LESS TESTS

    More Models,

    Less Tests

  • Model-based Testing in the Real World

    Model Driven Development: Taking BDD/ATDD to the Next Level 52

    Requirements Study Modeling Adapter Coding Test Coding Test Execution

    42% Less Time Per Requirement

    Model-based Testing 1.4 Days/Requirement

    Traditional Testing 2.4 Days/Requirement

    Windows protocol compliance (Web services, RPC, DCOM)

    Total effort: 250 person years (mostly junior SDETs)

    Saved 50 person years with model-based testing

  • Scope of Test Asset Maintenance

    Model

    Test Code

    Adapters

    Model-based Testing Behavior Driven Development

    Model Driven Development: Taking BDD/ATDD to the Next Level 53

  • More Models, Less Tests

    More better testing-MBT

    Rule-based

    Systematic and deep exploration

    Generate 1000s of scenarios in seconds

    Icing the BackBlob

    Maintain model

    Regenerate test suites

    Models grow much more slowly than test code

    Model Driven Development: Taking BDD/ATDD to the Next Level 54

  • Icing the Test BackBlob

  • Q & A

    #MoreModelsLessTests

    Model Driven Development: Taking BDD/ATDD to the Next Level 56

    [email protected]

  • Learn More - MBT UCCAT (User Conference on Advanced Automated Testing) Presentations

    http://ucaat.etsi.org/2013/program_conf.html

    2012 Model-based Testing User Survey

    http://robertvbinder.com/wp-content/uploads/rvb-pdf/arts/MBT-User-Survey.pdf

    Open Source MBT tools

    http://www.robertvbinder.com/open-source-tools-for-model-based-testing/

    Overview of MBT, list of commercial MBT tools

    http://www.slideshare.net/robertvbinder/model-basedtestingignite

    Spec Explorer

    http://visualstudiogallery.msdn.microsoft.com/271d0904-f178-4ce9-956b-d9bfa4902745

    http://msdn.microsoft.com/library/ee620411.aspx

    Microsoft Open Protocol Project

    http://cacm.acm.org/magazines/2011/7/109908-microsofts-protocol-documentation-program/fulltext#!

    Model Driven Development: Taking BDD/ATDD to the Next Level 57

  • Learn More: BDD, etc. SpecFlow

    http://www.specflow.org/

    Using SpecFlow with WatiN

    http://msdn.microsoft.com/en-us/magazine/gg490346.aspx

    SpecFlow Quick Start

    http://www.youtube.com/watch?v=SVwBz-ZTliw

    Writing Cucumber Feature Files

    https://github.com/cucumber/cucumber/wiki

    Model Driven Development: Taking BDD/ATDD to the Next Level 58

  • Content and Image Sources Grand Canyon Time Lapse. Ezra Kamer http://www.youtube.com/watch?v=NF6O6HuNJ0k The Blob, Theatre Scene. Fairview Productions, 1958. http://www.youtube.com/watch?v=GODDLgM1gKo http://www.imdb.com/title/tt0051418/ Defect densities: Capers Jones, private communication. Robert E. Park. Software Size Measurement: A Framework for

    Counting Source Statements. Technical Report CMU/SEI-92-TR-020. Software Engineering Institute, Carnegie Mellon University, September 1992.

    SpecFlow http://www.specflow.org/

    Contrast (Order and Chaos). M. C. Escher, 1950. National Gallery of Canada http://www.gallery.ca/en/see/collections/artwork.php?mkey=29270 Order and Chaos II. M. C. Escher, 1955. National Gallery of Canada http://www.gallery.ca/en/see/collections/artwork.php?mkey=41980 Spec Explorer. Microsoft http://visualstudiogallery.msdn.microsoft.com/271d0904-f178-4ce9-956b-d9bfa4902745 Spec Explorer Chat Model. Microsoft. The Blob, Final Scene. Fairview Productions, 1958. http://www.youtube.com/watch?v=0GEJgR-bX0w

    All other content Copyright Robert V. Binder, 2014.

    Model Driven Development: Taking BDD/ATDD to the Next Level 59