Multi-Paradigmatic Model-Based Testing Wolfgang Grieskamp Microsoft Research

Multi-Paradigmatic Multi-Paradigmatic Model-Based TestingModel-Based Testing

Wolfgang Grieskamp Wolfgang Grieskamp Microsoft ResearchMicrosoft Research

Wolfgang Grieskamp Microsoft Research 2

AgendaAgenda

I. I. Model-Based Testing at MicrosoftModel-Based Testing at Microsoft

II. II. Lessons LearnedLessons Learned

III. III. A Multi-Paradigmatic Approach & ToolA Multi-Paradigmatic Approach & Tool


AgendaAgenda





Model-Based Testing at Model-Based Testing at MicrosoftMicrosoft

► Applied since around 1999Applied since around 1999► Mailing list > 700 members (~10 % of MS Mailing list > 700 members (~10 % of MS

testers)testers)► Number of internal tools aroundNumber of internal tools around► Microsoft Research tools:Microsoft Research tools:

AsmL-T (2002)AsmL-T (2002) Spec Explorer 1 (2004)Spec Explorer 1 (2004) Spec Explorer 2 (in development)Spec Explorer 2 (in development)

A success story, but MBT is not yet A success story, but MBT is not yet mainstreammainstream


How current MS tools work How current MS tools work

► Based on EFSM or ASMBased on EFSM or ASM A set of parameterized actionsA set of parameterized actions A set of parameterized guarded update rules A set of parameterized guarded update rules

associated with actionsassociated with actions Update rules fire in states where they are enabledUpdate rules fire in states where they are enabled

► Try all parameter combinations as defined by the userTry all parameter combinations as defined by the user

State transitions are labeled with associated State transitions are labeled with associated actionsactions

► EFSM vs ASM (Spec Explorer): state space EFSM vs ASM (Spec Explorer): state space can grow dynamicallycan grow dynamically


A Glance of Spec ExplorerA Glance of Spec Explorer

[Campbell, Grieskamp, Nachmanson, Schulte, Tillmann, Veanes 2005][Campbell, Grieskamp, Nachmanson, Schulte, Tillmann, Veanes 2005]

► Actions and update rules are defined as Actions and update rules are defined as parameterized methods of an AsmL or Spec# parameterized methods of an AsmL or Spec# model model programprogram

► Actions are partitioned into Actions are partitioned into controlled controlled and and observed observed onesones Controlled = Input, Observed = OutputControlled = Input, Observed = Output

► Alternating refinement provides the conformance Alternating refinement provides the conformance notionnotion SUT must simulate all control steps of ModelSUT must simulate all control steps of Model Model must simulate all observation steps of SUTModel must simulate all observation steps of SUT

► Quick demoQuick demo


AgendaAgenda





Lessons LearnedLessons Learned

1.1. Adoption ProblemsAdoption Problems• AuthoringAuthoring• Executable Specification LanguagesExecutable Specification Languages• Scaling up to Model-Based DevelopmentScaling up to Model-Based Development• Education and DocumentationEducation and Documentation

2.2. Technological ProblemsTechnological Problems• State ExplosionState Explosion• Test SelectionTest Selection• Test ManagementTest Management



1.1. Adoption ProblemsAdoption Problems

2.2. Technological ProblemsTechnological Problems


AuthoringAuthoring

► ““Where is my IntelliSense?”Where is my IntelliSense?”

Context-sensitive editing, incremental Context-sensitive editing, incremental compilation, refactoring, etc. ubiquitous todaycompilation, refactoring, etc. ubiquitous today

Environments that support this probably more Environments that support this probably more relevant than the actual languagesrelevant than the actual languages

If you provide your own modeling notation, If you provide your own modeling notation, you better support decent authoringyou better support decent authoring

ButBut: Building decent authoring support for a language is : Building decent authoring support for a language is probably by an order of magnitude more complex than writing probably by an order of magnitude more complex than writing its compiler its compiler


Executable Specification Executable Specification LanguagesLanguages

► ““Why do I have to learn this new language?”Why do I have to learn this new language?”

High-level language constructs hard to digest for High-level language constructs hard to digest for many testersmany testers

Mainstream programming languages catch upMainstream programming languages catch up► Let the language guys market thisLet the language guys market this

Don’t mix up evangelizing MBT and your idea Don’t mix up evangelizing MBT and your idea of language designof language design

ButBut: some stakeholders like those high-level languages, in : some stakeholders like those high-level languages, in particular outside of test organizations; therefore it is best to particular outside of test organizations; therefore it is best to be agnosticbe agnostic regarding choice of languages.regarding choice of languages.


Scaling up to Model-Based Scaling up to Model-Based DevelopmentDevelopment

► ““Isn’t this very low-level?”Isn’t this very low-level?”► ““Where is the test plan?”Where is the test plan?”

Scenario-oriented (=interaction-based) modeling Scenario-oriented (=interaction-based) modeling is ubiquitous in the processis ubiquitous in the process

Need to support scenarios, best using Need to support scenarios, best using diagram notations (UML)diagram notations (UML)

ButBut: Scenarios need to be combined (composed) with state-: Scenarios need to be combined (composed) with state-based models.based models.


Education and Education and DocumentationDocumentation

► ““I do not have time to learn this!”I do not have time to learn this!”► ““Where is the textbook for this?”Where is the textbook for this?”

People in the industry have not enough time to educate People in the industry have not enough time to educate themselvesthemselves

Practical education material for MBT is poorPractical education material for MBT is poor

Need better coverage at universities, text books, Need better coverage at universities, text books, and individual tool documentationand individual tool documentation

ButBut: chicken-and-egg problem: mainstream in university education : chicken-and-egg problem: mainstream in university education requires proof of feasibility in industryrequires proof of feasibility in industry



1.1. Adoption ProblemsAdoption Problems

2.2. Technological ProblemsTechnological Problems


State ExplosionState Explosion

► ““Cool! That stuff generates a large number of Cool! That stuff generates a large number of tests on the click of a button! tests on the click of a button! … … ””

Test suite execution time is a significant Test suite execution time is a significant productivity factorproductivity factor

On-the-fly only works in particular test applicationsOn-the-fly only works in particular test applications

Need better techniques to reduce the state Need better techniques to reduce the state spacespace

Partly solved by test purposes, but more work is required (apply Partly solved by test purposes, but more work is required (apply POR, state symmetries, etc. from MC)POR, state symmetries, etc. from MC)


Test SelectionTest Selection

► ““I want to have I want to have these these tests from my model”tests from my model”

Typical request: find the shortest path to a given Typical request: find the shortest path to a given state; from then on, apply link coveragestate; from then on, apply link coverage

Give users more fine-grained control over Give users more fine-grained control over test selection, in particular, composition of test selection, in particular, composition of test selection strategiestest selection strategies

Partly solved by test purposes, provided context sensitive Partly solved by test purposes, provided context sensitive selection strategies can be controlled with test purposesselection strategies can be controlled with test purposes


Test ManagementTest Management

► ““How does this integrate with my test How does this integrate with my test management tool X?”management tool X?”

Test generation only part of the testing problemTest generation only part of the testing problem A diversity of test management tools and A diversity of test management tools and

processes aroundprocesses around

Cannot prescribe test management from an Cannot prescribe test management from an MBT tool, but must be able to plug-in a MBT tool, but must be able to plug-in a variety of such frameworksvariety of such frameworks


AgendaAgenda





Multi-Paradigmatic?Multi-Paradigmatic?

par·a·digm (plural par·a·digms) noun

1. typical example: a typical example of something

2. model that forms basis of something: an example that serves as a pattern or model for something, especially one that forms the basis of a methodology or theory

3. relationship of ideas to one another: in the philosophy of science, a generally accepted model of how ideas relate to one another, forming a conceptual framework within which scientific research is carried out


PrinciplesPrinciples

► Do not stick to a single paradigmDo not stick to a single paradigm State-based and scenario-based modelingState-based and scenario-based modeling Textual and diagrammatic notationsTextual and diagrammatic notations Programming languages and specification Programming languages and specification

languageslanguages

► Support combination of paradigmsSupport combination of paradigms Model compositionModel composition

► Fully integrate into a modern development Fully integrate into a modern development environmentenvironment IntelliSense, refactoring, and so onIntelliSense, refactoring, and so on


Quick GlanceQuick Glance

Model

Explore

Traverse

Test

Model with Charts



Model

Explore

Traverse

Test

Model with Programs



Model

Explore

Traverse

Test

Explore Individually or in Composition



Model

Explore

Traverse

Test

Explore for Model-Checking



Model

Explore

Traverse

Test

Traverse for Test-Selection



Model

Explore

Traverse

Test

Test Execution


DemoDemo


Foundations: Action Foundations: Action MachinesMachines

[Grieskamp, Kicillof, Tillmann 2005/2006][Grieskamp, Kicillof, Tillmann 2005/2006]

► A combination of LTS and automataA combination of LTS and automata► Given:Given:

a set of terms with variables to represent values and a set of terms with variables to represent values and action labelsaction labels

a set of a set of environments environments mapping locations to termsmapping locations to terms► Special locationSpecial location containing the action labelcontaining the action label► forms a complete lattice with meet and joinforms a complete lattice with meet and join

► Action machine M = (C,A,I,T):Action machine M = (C,A,I,T): C a set of C a set of control pointscontrol points A A C a set of C a set of accepting control pointsaccepting control points I I C initialization relation: e C initialization relation: e (e’,c) (e’,c) I I T T CCC stepping relation: (e,c) C stepping relation: (e,c) (e’,c’) (e’,c’) T T

► RemarkRemark: the action label is contained in the special location : the action label is contained in the special location


Given Action MachinesGiven Action Machines► Guarded-update machineGuarded-update machine (encapsulates model (encapsulates model

program)program) Defined by a set of guarded-update rulesDefined by a set of guarded-update rules Steps for each rule in each environment where the guard is Steps for each rule in each environment where the guard is

truetrue Step updates the environment Step updates the environment Two control points: accepting and non-acceptingTwo control points: accepting and non-accepting

► Determined by a predicate over the environmentDetermined by a predicate over the environment

► Single-step machineSingle-step machine (building block for scenarios): (building block for scenarios): Defined by an action termDefined by an action term Two control points: before-step and after-stepTwo control points: before-step and after-step

► Before-step: enables transition into after-step labeled with Before-step: enables transition into after-step labeled with action term; is non-acceptingaction term; is non-accepting

► After-step: nothing enabled; is acceptingAfter-step: nothing enabled; is accepting Completely parametric over environmentCompletely parametric over environment


Composed Action MachinesComposed Action Machines► Synchronized Parallel MachineSynchronized Parallel Machine

Control points are product of control points of Control points are product of control points of composed machines composed machines

Steps if both machines step into an environment which Steps if both machines step into an environment which has a non-empty join: has a non-empty join:

► (e,(c1,c2)) (e,(c1,c2)) (e1’ (e1’ e2’,(c1’,c2’)) e2’,(c1’,c2’)) iffiff (e,c1) (e,c1) (e1’,c1’) (e1’,c1’) (e,c2) (e,c2) (e2’,c2’) (e2’,c2’) e1’ e1’ e2’ e2’ ≠≠

► Alternating Refinement MachineAlternating Refinement Machine Similar as product machine, but: Similar as product machine, but:

► (e,(c1,c2)) (e,(c1,c2)) (e’,(c1’,c2’)) (e’,(c1’,c2’)) iffiff (e,c1) (e,c1) (e1’,c1’) (e1’,c1’) (e,c2) (e,c2) (e2’,c2’) (e2’,c2’) control stepcontrol step e1’ e1’ e2’ , e’ = e1’ e2’ , e’ = e1’ observation stepobservation step e1’ e1’ e2’ , e’ = e2’ e2’ , e’ = e2’


Composed Action MachinesComposed Action Machines► Sequential machineSequential machine

Control points are disjoint union of control Control points are disjoint union of control points of composed machinespoints of composed machines

Each step of the first machine leading into an Each step of the first machine leading into an accepting state is duplicated to lead also into accepting state is duplicated to lead also into an initial state of the second machine an initial state of the second machine ((simplified description)simplified description)

Justifies the need of the initialization relation: a Justifies the need of the initialization relation: a machine can be initialized in the environment machine can be initialized in the environment reached by another machinereached by another machine


Composed Action MachinesComposed Action Machines

► Interleaving parallel compositionInterleaving parallel composition►RepetitionRepetition►Action translationAction translation►DeclarationDeclaration►ConstraintsConstraints►HidingHiding►TraversalsTraversals►……


Implementation: XRTImplementation: XRT

[Grieskamp, Tillmann, Schulte 2005][Grieskamp, Tillmann, Schulte 2005]

►Virtual execution framework for .NET Virtual execution framework for .NET intermediate codeintermediate code

► Implements mixed concrete/symbolic Implements mixed concrete/symbolic state space explorationstate space exploration

►Action machines a thin layer on top of Action machines a thin layer on top of XRTXRT


AgendaAgenda





ConclusionsConclusions► MBT successful but not mainstreamMBT successful but not mainstream

► Application at MS suggests that tools need to be more Application at MS suggests that tools need to be more powerful and simpler at the same timepowerful and simpler at the same time Let users model in mainstream languages (making modeling Let users model in mainstream languages (making modeling

simpler) and different paradigms (adapting to roles in the simpler) and different paradigms (adapting to roles in the process)process)

Use model-composition as the cornerstone to provide additional Use model-composition as the cornerstone to provide additional powerpower

► Multi-paradigmatic MBT is a premier case study for Multi-paradigmatic MBT is a premier case study for model-based development in general model-based development in general

► Current plans at MSR:Current plans at MSR: Ship the new Spec Explorer tool to internal customersShip the new Spec Explorer tool to internal customers Possibly ship as emerging technology externallyPossibly ship as emerging technology externally Push integration and/or spin-off for mainstream Visual Studio Push integration and/or spin-off for mainstream Visual Studio


The EndThe End

►Checkout Checkout http://research.microsoft.com/projects/specexplorerhttp://research.microsoft.com/projects/specexplorer for the current Spec Explorer tool and for the current Spec Explorer tool and updates regarding the new toolupdates regarding the new tool

►Checkout my website for papersCheckout my website for papers

►Thanks!Thanks!

Documents

Multi-Paradigmatic Model-Based Testing Wolfgang Grieskamp Microsoft Research