Model Based Technology for Functional Testing to · PDF fileModel Based Technology for Functional Testing ... Test script Test framework Test ... Model_Based_Technology_for_Functional_Testing_to_Conformiq

Model Based Technology for Functional TestingBusiness case for BSO Budapest R&D

Company confidential – for internal use only1 © Nokia Siemens Networks

Business case for BSO Budapest R&D

MBT Concept

Model Based Testing (MBT) comes from the following idea:If we have an abstract model of the system’s behavior then we can generate all possible test-cases. The limit is defined only by the abstraction level.

Specification Function Model

Test generation

model from system perspective


Test script

Test framework

Test execution engine

System under test

Test generation

Milestones in Model Based Testing Adoption for Test Design• 1990s: 1st generation MBT tools become commercially available

– strong link to Model Driven Development– described system operation from user perspective

• 2006: 2nd generation MBT tools came on market– modeled system operation from specification perspective

• 2007: Start of ITEA2 D-MINT (Deployment of Model Based Technologies to Industrial Testing) project


to Industrial Testing) project– 25 partners, 6 countries, 8 case studies, 5 application areas– http://www.d-mint.org/

• 2010: 1st industrial standard on MBT approved at ETSI – ES 202 951 ”Requirements for Modeling Notation”– Active participation from 15 companies incl. MBT tool makers, users &

research institutes• 2011: 1st industrial conferences in EU and India dedicated to MBT

Evolution of Software Testing

FrameworksKeyword Driven

MBTManually written scriptsExecution and analysis are

Generated scriptsExecution and analysis are automatic


Manual

Scripts-BasedCapture/Replay

Keyword Driven

No scriptsExecution is manual

Manually written scriptsExecution and analysis are manual

Execution and analysis are automatic

Pilot project in Budapest R&D

Content: Early interface implementation in HLRWhy?• New interface introduced to HLR � we can start from scratch• Well documented: 3GPP standards, specification, fault reports, etc…• Feature is already on the market � passed through all test phasesResources:• 2 cooperative students, with the help of a full-time employee


• 2 cooperative students, with the help of a full-time employee• Training and remote support from Conformiq

Comparison parameters

In order to say definitely that MBT is good for us, we need to take the following things into consideration:

• Effort– How many man hours were needed in the original testing, and with MBT?

• Quality


• Quality– Number of fault reports: How many new faults were found with MBT, and how

serious are those?– Coverage: full requirement coverage is expected from MBT

• Competence ramp-up• Is accelerated knowledge acquisition possible?

Traditional way of functional testing

Specification writing

Test plan writing

Review OK?

Specification

Test plan

Y

Y

N

N


Manual execution

Manual macro writing

Review OK?

TC OK?

Framework writing / updating

Y

Y N

N

Macro

Execution

Automatized way of functional testing with TTCN


Test plan writing

Review OK?

Specification

Test plan

Y

N

N


Manual macro writing

Review OK?

TC OK?


Y

Y

N

N

Macro

Execution

Functional testing with MBT


Model writing/updating

Review OK?

Specification

Model

Y

N

N


Scripter writing/updating

Exporting macros

Review OK?


Y

N

Macro

Execution

Efforts in man hoursTTCN automation testing MBT

Test plan writing 150 hours Specification study 25 hours

Creating/updating model

225 hours

Model review 79 hours

Creating/updating scripter

62 hours

Manual macro writingand automation

500 hours

Generate test-cases 2 hours for all generations


and automation hours generations7 minutes for 1 generation

Validation 100 hours

Validate generatedtest-cases

16 hours

Update test framework

Execution 400 hours Execution 452

Sum 1150 hours Sum 860 hours

25% savings on effort

Product Quality – Fault Reports

32 faults found during original testing (FT + SyVe + Pet + Customer )• Only 9 were found in FT (out of 32)• 8 faults were found later (out of 23) that could have been found in FT

6 additional faults found with MBT (2 critical, 1 i n Implementation Spec)


39%

35%

26%

found in FT

could be found in FT, but found later

found with MBT only

Product Quality - Coverage

Requirement coverage:• All explicit requirements are covered by the model.• The most important implicit requirements (e.g.: feature interworking) are

also covered.

Coverage achieved by MBT is greater than by manually written tests, because it is much less effort to increase it after a certain threshold.


because it is much less effort to increase it after a certain threshold.

Automated requirement traceability; Forces application level implicit requirement to be covered by the model

Learning from model (Competence ramp -up)

• 1 newcomer:– only basic GSM training before (SYSTRA)

• Training material: – QML model + code– Abbreviations handout

• Learning time: 21 hours• Support time: 6 hours


• Support time: 6 hours

• Competence test:– General understanding of the functionality– Detailed knowledge of message sequences and triggering events– Test result: 83%

• Useful productive capacity can be raised to 100% much faster than the usual capacity planning process. The newcomer from the pilot project is already a part of the fault report solving team.

Subjective impressions

• With a few clicks, you can quickly draft many test-cases

• Easy export of test-cases in desired format

• Parameters from model go easily to exported test-cases

• Tool does the boring work of trying out multiple values

• With built-in HTML scripter we can produce test documentation in minutes

• The technology is exciting AND works in the real world


• The technology is exciting AND works in the real world

Conclusion

+ shorter time to market: model design and maintainenance time is shorter+ easy to maintain regression set+ full requirement coverage+ better understanding of the product needs and the functionality, increases

quality+ faults can be found in earlier phases (even in IS)+ much better structured documentation, easy to learn, more useful for


+ much better structured documentation, easy to learn, more useful for Innovation improvement

- time and competence for TC review and reduction- model creation competence- initial investment: license cost + time of initial model creation- follow up the changed FT set (tc history, QC)- time and competence for model review

Dept. / Author / Date

Let’s put it into use

Questions?


Documents

Model Based Technology for Functional Testing to · PDF fileModel Based Technology for Functional Testing ... Test script Test framework Test ... Model_Based_Technology_for_Functional_Testing_to_Conformiq