Model-Based Testing: Concepts, Tools, and Techniques

W14 Concurrent Class

10/2/2013 3:00:00 PM

"Model-Based Testing:

Concepts, Tools, and

Techniques"

Presented by:

Adam Richards

Critical Logic

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Adam Richards

Critical Logic

Adam Richards is the director of consulting services at Critical Logic, a consulting firm

specializing in model-based testing tools and services. With nearly two decades in the industry,

Adam has experience as a developer, business analyst, requirements manager, test analyst,

and QA manager. He brings first-hand experience implementing model-based testing

approaches on systems in a wide array of industries, across the entire gamut of development

methodologies.

9/20/2013

1

2013 Critical Logic

Model-Based Testing

Concepts, Tools, and Techniques

Presented by Adam Richards

Starwest, 2013

2013 Critical Logic

2

If the shoe fits…

Model-Based Testing offers something for everyone.

9/20/2013

2

2013 Critical Logic

3

Who uses MBT, how, and for what

Survey says…

2013 Critical Logic

4

Domains

Binder, Robert V. 2011 Model-based Testing User Survey: Results and Analysis. 2012

http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf

9/20/2013

3

2013 Critical Logic

5

Interfaces



2013 Critical Logic

6

Methodologies



9/20/2013

4

2013 Critical Logic

7

Test Level At what scope do you use MBT?



2013 Critical Logic

8

Test Aspects Which aspects of the SUT do you test with MBT?



9/20/2013

5

2013 Critical Logic

9

Benefits

On average, respondents report MBT:

• Reduced escaped bugs by 59%

• Reduced testing costs by 17%

• Reduced testing duration by 25%



2013 Critical Logic

10

Okay…

But what is it???

9/20/2013

6

2013 Critical Logic

11

What we Mean by “Model”

Model Soup

2013 Critical Logic

12

Structural Models

• Static

• ERD, Class, Component, Data Model…

• Can define test environment (e.g. UTP)

• Foundation for Behavioral models

• Used for both Rqmts & System Models

9/20/2013

7

2013 Critical Logic

13

Behavioral Models

• Describe flow

• BPM, Activity, State, Cause-Effect…

• Can specify abstract test scenarios

(no data, test environment)

• Used for both Rqmts & System Models

2013 Critical Logic

14

Test Models

• Behavioral model + inputs + environment

• Specify tests at executable level

• Automatable

• May or may not include test generator

9/20/2013

8

2013 Critical Logic

15

Hanging the model types together

*Substitutions welcome

MBT Process Menu*

2013 Critical Logic

16

Rqmts

Model

Flavor 1: Model-Centric

• Rare today (yet)

• Generate all

• Big investment

• Big returns

• Maint. costs

• Human error

• Defect escapes

• Defect

introduction

Rqmts

Model

9/20/2013

9

2013 Critical Logic

17

Flavor 2a: Model-Driven

• Less uncommon

• No rqmts model

• Generate

executable tests

from test model

2013 Critical Logic

18

Flavor 2b: Model-Driven (Agile)

• Test model only

• Generate

acceptance criteria

• Maintain model,

not tests

• Supports CIT

9/20/2013

10

2013 Critical Logic

19

Flavor 3: Model-Based

• No test model

• Walk rqmts model

• Validation later

• Add data, env.

2013 Critical Logic

20

MBT Flavor Rainbow

Model “Centric”

• Model-generated tests & code

Model “Driven”

• Model-generated tests

Model “Based”

• Tests based on rqmts models, built by hand

9/20/2013

11

2013 Critical Logic

21

What Test Modeling Tools Can Do

Tool Ingredients

2013 Critical Logic

22

Test Model Tool Functions

Test Data Generators

Test Case Editors

Test Case Generators >> Test Coverage Criteria

9/20/2013

12

2013 Critical Logic

23

Test Coverage Criteria Sys/Rqmt Model Coverage

• Control Flow, Data Flow

• Transition-Based (FSM, statecharts)

• UML-Based (AD, Class, Seq, etc.)

Data Coverage • Boundary Value Analysis

• Classification Tree

• Statistical, Pairwise

Fault-Based Coverage • Mutation, Stuck-At Fault Model, MC/DC

Requirements-Based Coverage • All Requirements Coverage

Utting, M.; Legeard, B.: "Practical Model-Based Testing - A Tools Approach". Morgan Kaufmann Publ, Amsterdam, 2007

2013 Critical Logic

24

Test Modeling Tool Examples

Commercial

• RBT/DTT (Bender RBT, Critical Logic)

• Rhapsody ATG (IBM)

• Others (Conformiq, AETG, Leirios…)

Open Source

• ParTeG (Eclipse Plugin)

• Spec Explorer (Visual Studio)

• Others (JUMBL, Torx, Uppaal TRON…)

9/20/2013

13

2013 Critical Logic

25

More Survey

What kind of MBT tools were used? • 17% Open Source

• 30% Custom

• 53% Commercial

Testing modes used on project (avg)? • 33.6% MBT

• 36.7% Traditional

• 29.7% Manual



2013 Critical Logic

26

Coverage What kind of coverage is measured for your MBT test runs?



9/20/2013

14

2013 Critical Logic

27

More Survey (last one)

How do you connect MBT output, SUT control interfaces, & SUT observation interfaces?

• 06% Manual input & observation

• 15% Manual scripting required

• 15% Tool generates code/data + manual processing

• 26% Tool generates code/data for the SUT interface

• 38% Tool generates code/data for an adapter that abstracts the SUT interface



2013 Critical Logic

28

Tool Challenges Opportunities

Tool Integration • Automation, RMS, TMS, SDLC, TestIF

Coverage limited to source model • Can it synthesize sources to generate tests?

Constraints & infeasibles • Good coverage <> executable tests

Environment Independence • What if the scripting language changes?

9/20/2013

15

2013 Critical Logic

29

Model-Centric Example - CEM

1.Multi-Source

Synthesis

2.RMS

Traceability

3.Manual &

Auto

4.TMS

integration

5.MCDC tests

that run

6.Language

independence

“Test

Mo

del”

2013 Critical Logic

30

Advantages, Challenges, & Benefits

Yeah but…

9/20/2013

16

2013 Critical Logic

31

MBT Advantages 1. Flexibility (env, interface, model basis)

2. Low barrier to entry (80-100 hrs)

[Binder]

3. Measurable, repeatable test coverage

4. Basis for effort estimation

5. Review models, not tests

6. Move validation forward in SDLC

7. Store test definitions as data

8. Integrated test automation

9. Maintain models, not tests

2013 Critical Logic

32

MBT Challenges

• Education of test modelers

• MBT strategy is an intellectual task

• Expectations: models not “automatic”

• Selecting and integrating tool suite

• Integration with existing SDLC • People

• Tools

• Processes

9/20/2013

17

2013 Critical Logic

33

MBT Benefits

1. Requirements improvement

2. Efficient test generation, automation

3. Reduce maintenance costs

4. Models as basis for effort estimation

5. Reduce technical debt

6. Lower cost of quality from rework

7. Reduce defect escapes

8. Reduce defect introduction

9. Data-based result analysis and reporting

2013 Critical Logic

34

Tips to Get Started

• Start small • Pilot low-risk app/change

• Rev. eng. an old complex function

• Parallel = ideal

• Use a vendor • Planning, objectives, strategy

• Delivery, training & mentoring

• Identify demonstration objectives • Test Coverage improvement

• Requirements improvement

• Reduced defect introduction/escape rates

• Scalability, strategy

9/20/2013

18

2013 Critical Logic

35

Discussion

2013 Critical Logic

36

Thank You!

Adam Richards

Critical Logic

Tel: 415-814-9556

[email protected]

9/20/2013

19

2013 Critical Logic

37

References & Further Reading

Ambler, Scott W. Agile Modeling: Effective Practices for Extreme Programming and the Unified Process.

2002

Utting, M.; Legeard, B.: "Practical Model-Based Testing - A Tools Approach". Morgan Kaufmann Publ,

Amsterdam, 2007

Web site of Robert V. Binder, software engineer, expert on testing and automation, source of important

research information in this presentation: http://www.robertvbinder.com/

http://www.robertvbinder.com/open-source-tools-for-model-based-testing/


Web site of Stephan Weissleder, MBT theoretician, expert, and innovator: http://model-based-testing.de/

Please feel free to contact me directly if you have any questions or simply to further the discussion. Model-

Based Testing is the future!



























http://model-based-testing.de/







Technology

Model-Based Testing: Concepts, Tools, and Techniques