Embed Size (px)
Citation preview
Model-Based Black Box Testingand its Standardization Layers
THE SOLUTION TO BOOST
YOUR TEST EFFICIENCY
MaTeLo
Since 1998
French: 45 experts
Turnover 2010: 4 M€
Expertise offer:PROCESS IMPROVEMENT
SAFETY ENGINEERING
SYSTEM ENGINEERING
OPTIMIZED TESTING
SOFTWARE RELIABILITY
EMBEDDED SYSTEMS
ALL4TEC
OUR COMPETENCES OFFER
Training
Consulting
Engineering
Tooling
MaTeLo
EXAM
Safety Architect
M Elopée
AGENDA
THE TEST PROBLEM
THE EXPECTED SOLUTION
THE MODEL-BASED TESTING SOLUTION
A PRAGMATIC INDUSTRIAL USE-CASE
A MORE STANDARDIZED TOOLS INTEGRATION
THE TESTING PROBLEM
Test Availability
SystemComplexity
Testing Effort
Time
Reality
Product Life
Reliability Market Expectation
TestNeeds
UsageIntensity
DevelopmentPhase
Customer Validation
NormalUsage
Test Resources Lack
Test Focus Deviation
HOW TO SOLVE THIS ISSUE?
Test the SUT usage before product release
Use best in class engineering process
Widely use automatic generation
Use dedicated tools that decuple engineering productivity
Use formalization and traceability
TestEnvironment
MODEL-DRIVEN ENGINEERING
FunctionalRequirements
UsageModel
DesignModel
Test CasesCode
Test ScriptsEXE
DESIGN TEST
GenerateGenerate
DeriveCompile
Implement Scenarize
MaTeLoModel Based Testing
WHAT IS MaTeLo?
Formal TestSpecification
FullyMeasured
Process
Product UsagePoint of View
Markov Test Logic
MaTeLo
Test ExecutionCompatibility
SystematicEngineeringProductivity
MoreProductiveCoverage
BidirectionalRequirements
Traceability
MATELO SCREENSHOT
INTEGRATED TEST WORKFLOW
MaTeLo Usage Model Editor
Test Design intoUsage Models
MaTeLoTest Campaign Analysis
ConfidenceAnd Coverage
Reporting
MaTeLoTestor
Automatic Test Cases Generation
PLAN
EXAM
Test Automation
DO CHECK
ACT
MaTeLo EDITORUsage Model Design
Test Cases
SystemRequirements
UsageModel
Test ExecutionPlatform
SystemUnderTest
Adaptor
Test Scripts
Requirements Traceability
Model Coverage
Reliability
Mean TimeTo Failure
Intensityof Test
Test CasesTest
Cases
SystemRequirements
SystemRequirements
UsageModelUsageModel
Test ExecutionPlatform
Test ExecutionPlatform
SystemUnderTest
SystemUnderTest
Adaptor
Test Scripts
AdaptorAdaptor
Test Scripts
Test Scripts
Requirements Traceability
Model Coverage
Requirements Traceability
Model Coverage
Reliability
Mean TimeTo Failure
Intensityof Test
Reliability
Mean TimeTo Failure
Intensityof Test
MATELO USAGE MODELS
BEGINNING
END
STATE
TRANSITION
MACRO CHAIN
Probability on every Alternatives Association of Requirements Configuration of Test Operations Setup Transition as Test Step
Stimulations
Inputs StimulationEquivalence Classes
Timing
Verifications
Expected OutputsTiming
MODEL TRANSITION = TEST STEP
RequirementsTest Operations
Select_Gear(5)Accelerate(100%)Check_Speed(185)
Test Oracle
Outputs = f(Inputs)
MATELO PROFILE
Profiles can be embedded to qualify the usage model Operational profiles
Test profiles
347
0-100-150-200-500
87 (+/- 25)
Data distribution Usage path probability
MaTeLo EDITORRequirements ManagementTest
Cases
SystemRequirements
UsageModel
Test ExecutionPlatform
SystemUnderTest
Adaptor
Test Scripts
Requirements Traceability
Model Coverage
Reliability
Mean TimeTo Failure
Intensityof Test
Test CasesTest
Cases
SystemRequirements
SystemRequirements
UsageModelUsageModel
Test ExecutionPlatform
Test ExecutionPlatform
SystemUnderTest
SystemUnderTest
Adaptor
Test Scripts
AdaptorAdaptor
Test Scripts
Test Scripts
Requirements Traceability
Model Coverage
Requirements Traceability
Model Coverage
Reliability
Mean TimeTo Failure
Intensityof Test
Reliability
Mean TimeTo Failure
Intensityof Test
REQUIREMENTS MANAGEMENT
IBM Rational Doors MaTeLo Usage Model
Association
(N,M)
MaTeLo Requirements Library
Import
Requirement UpdateNew Baseline
Update
QuickModelUpdate
RequirementsChangeAnalysis
ModelImpact
Highlighting
Available for other ReqMgt tools
with import of XML or CSV files
MaTeLo TESTORTest Cases GenerationTest
Cases
SystemRequirements
UsageModel
Test ExecutionPlatform
SystemUnderTest
Adaptor
Test Scripts
Requirements Traceability
Model Coverage
Reliability
Mean TimeTo Failure
Intensityof Test
Test CasesTest
Cases
SystemRequirements
SystemRequirements
UsageModelUsageModel
Test ExecutionPlatform
Test ExecutionPlatform
SystemUnderTest
SystemUnderTest
Adaptor
Test Scripts
AdaptorAdaptor
Test Scripts
Test Scripts
Requirements Traceability
Model Coverage
Requirements Traceability
Model Coverage
Reliability
Mean TimeTo Failure
Intensityof Test
Reliability
Mean TimeTo Failure
Intensityof Test
MaTeLo TEST STRATEGY
DEFINE THE TEST STRATEGY, BY CHOOSING Test Algorithm Test Profile Part of model to test
Most probableapproach
Start
Close
FREQUENCYFOCUS
Start
Close
Risk basedapproach
CustomTest profile
CRITICALITY, COMPLEXITYUPDATE FOCUS
Arcs coverageapproach
Close
Start
REQUIREMENTSCOVERAGE
UsageTest profile
Usageapproach
Start
Close
OPERATIONALCOVERAGE
Generate
MaTeLo TESTOR: HTML TEST PLAN
Input Expected Result
State
Requirement
Test Function Verdict
Example: MaTeLo FOR EXAM
Test Repository SUT Interface Test Operations
Stimulation Measurement Administration
Sub Test Sequences
Automatic Test Cases Automatic call of Test Operation Automatic parameterisation Requirements association Test Case description generation Usage model respect
Association
Association
Generate
Requirements Name UUID Description
MaTeLo TCATest Campaign Analysis
Test Cases
SystemRequirements
UsageModel
Test ExecutionPlatform
SystemUnderTest
Adaptor
Test Scripts
Requirements Traceability
Model Coverage
Reliability
Mean TimeTo Failure
Intensityof Test
Test CasesTest
Cases
SystemRequirements
SystemRequirements
UsageModelUsageModel
Test ExecutionPlatform
Test ExecutionPlatform
SystemUnderTest
SystemUnderTest
Adaptor
Test Scripts
AdaptorAdaptor
Test Scripts
Test Scripts
Requirements Traceability
Model Coverage
Requirements Traceability
Model Coverage
Reliability
Mean TimeTo Failure
Intensityof Test
Reliability
Mean TimeTo Failure
Intensityof Test
TEST CAMPAIGN PROCESS
Test Cases
SystemRequirements
UsageModel
Test ExecutionPlatform
SystemUnderTest
Adaptor
Test Scripts
Requirements Traceability
Model Coverage
Reliability
Mean TimeTo Failure
Intensityof Test
Test CasesTest
Cases
SystemRequirements
SystemRequirements
UsageModelUsageModel
Test ExecutionPlatform
Test ExecutionPlatform
SystemUnderTest
SystemUnderTest
Adaptor
Test Scripts
AdaptorAdaptor
Test Scripts
Test Scripts
Requirements Traceability
Model Coverage
Requirements Traceability
Model Coverage
Reliability
Mean TimeTo Failure
Intensityof Test
Reliability
Mean TimeTo Failure
Intensityof Test
Test Cases
SystemRequirements
UsageModel
Test ExecutionPlatform
SystemUnderTest
Adaptor
Test Scripts
Requirements Traceability
Model Coverage
Reliability
Mean TimeTo Failure
Intensityof Test
Test CasesTest
Cases
SystemRequirements
SystemRequirements
UsageModelUsageModel
Test ExecutionPlatform
Test ExecutionPlatform
SystemUnderTest
SystemUnderTest
Adaptor
Test Scripts
AdaptorAdaptor
Test Scripts
Test Scripts
Requirements Traceability
Model Coverage
Requirements Traceability
Model Coverage
Reliability
Mean TimeTo Failure
Intensityof Test
Reliability
Mean TimeTo Failure
Intensityof Test
Test
Case
s
Syst
em
Requ
irem
en
ts
Usage
Model
Test
Execu
tion
Pla
tform
Syst
em
Un
der
Test
Adapto
r
Test
Scr
ipts
Requ
irem
en
ts
Tra
ceab
ilit
y
Model
Covera
ge
Reliab
ilit
y
Mean
Tim
eTo F
ailu
re
Inte
nsi
tyof
Test
Test
Case
sTest
Case
s
Syst
em
Requ
irem
en
tsS
yst
em
Requ
irem
en
ts
Usage
Model
Usage
Model
Test
Execu
tion
Pla
tform
Test
Execu
tion
Pla
tform
Syst
em
Un
der
Test
Syst
em
Un
der
Test
Adapto
r
Test
Scr
ipts
Adapto
rA
dapto
r
Test
Scr
ipts
Test
Scr
ipts
Requ
irem
en
ts
Tra
ceab
ilit
y
Model
Covera
ge
Requ
irem
en
ts
Tra
ceab
ilit
y
Model
Covera
ge
Reliab
ilit
y
Mean
Tim
eTo F
ailu
re
Inte
nsi
tyof
Test
Reliab
ilit
y
Mean
Tim
eTo F
ailu
re
Inte
nsi
tyof
Test
Target Quality Criteria1. Coverage2. Reliability
Confidence Indicators
Test
C
ases
Sys
tem
Req
uire
men
ts
Usa
geM
odel
Test
Exe
cutio
nPl
atfo
rm
Sys
tem
Und
erTe
st
Ada
ptor
Test
S
crip
ts
Req
uire
men
ts
Trac
eabi
lity
Mod
el
Cov
erag
e
Rel
iabi
lity
Mea
n Ti
me
To F
ailu
re
Inte
nsity
of T
est
Test
C
ases
Test
C
ases
Sys
tem
Req
uire
men
tsS
yste
mR
equi
rem
ents
Usa
geM
odel
Usa
geM
odel
Test
Exe
cutio
nPl
atfo
rmTe
st E
xecu
tion
Plat
form
Sys
tem
Und
erTe
st
Sys
tem
Und
erTe
st
Ada
ptor
Test
S
crip
ts
Ada
ptor
Ada
ptor
Test
S
crip
tsTe
st
Scr
ipts
Req
uire
men
ts
Trac
eabi
lity
Mod
el
Cov
erag
e
Req
uire
men
ts
Trac
eabi
lity
Mod
el
Cov
erag
e
Rel
iabi
lity
Mea
n Ti
me
To F
ailu
re
Inte
nsity
of T
est
Rel
iabi
lity
Mea
n Ti
me
To F
ailu
re
Inte
nsity
of T
est
BugsNew
Package Test
C
ases
Sys
tem
Req
uire
men
ts
Usa
geM
odel
Test
Exe
cutio
nPl
atfo
rm
Sys
tem
Und
erTe
st
Ada
ptor
Test
S
crip
ts
Req
uire
men
ts
Trac
eabi
lity
Mod
el
Cov
erag
e
Rel
iabi
lity
Mea
n Ti
me
To F
ailu
re
Inte
nsity
of T
est
Test
C
ases
Test
C
ases
Sys
tem
Req
uire
men
tsS
yste
mR
equi
rem
ents
Usa
geM
odel
Usa
geM
odel
Test
Exe
cutio
nPl
atfo
rmTe
st E
xecu
tion
Plat
form
Sys
tem
Und
erTe
st
Sys
tem
Und
erTe
st
Ada
ptor
Test
S
crip
ts
Ada
ptor
Ada
ptor
Test
S
crip
tsTe
st
Scr
ipts
Req
uire
men
ts
Trac
eabi
lity
Mod
el
Cov
erag
e
Req
uire
men
ts
Trac
eabi
lity
Mod
el
Cov
erag
e
Rel
iabi
lity
Mea
n Ti
me
To F
ailu
re
Inte
nsity
of T
est
Rel
iabi
lity
Mea
n Ti
me
To F
ailu
re
Inte
nsity
of T
est
New version
Test
C
ases
Sys
tem
Req
uire
men
ts
Usa
geM
odel
Test
Exe
cutio
nPl
atfo
rm
Sys
tem
Und
erTe
st
Ada
ptor
Test
S
crip
ts
Req
uire
men
ts
Trac
eabi
lity
Mod
el
Cov
erag
e
Rel
iabi
lity
Mea
n Ti
me
To F
ailu
re
Inte
nsity
of T
est
Test
C
ases
Test
C
ases
Sys
tem
Req
uire
men
tsS
yste
mR
equi
rem
ents
Usa
geM
odel
Usa
geM
odel
Test
Exe
cutio
nPl
atfo
rmTe
st E
xecu
tion
Plat
form
Sys
tem
Und
erTe
st
Sys
tem
Und
erTe
st
Ada
ptor
Test
S
crip
ts
Ada
ptor
Ada
ptor
Test
S
crip
tsTe
st
Scr
ipts
Req
uire
men
ts
Trac
eabi
lity
Mod
el
Cov
erag
e
Req
uire
men
ts
Trac
eabi
lity
Mod
el
Cov
erag
e
Rel
iabi
lity
Mea
n Ti
me
To F
ailu
re
Inte
nsity
of T
est
Rel
iabi
lity
Mea
n Ti
me
To F
ailu
re
Inte
nsity
of T
est
New TestCases
Generation
MetricsQuality/Version
Operational Confidence Needs Coverage Functional Coverage
From Pragmatic Tools Integration
ToIts Standard Adoption
Test Cases
SystemRequirements
UsageModel
Test ExecutionPlatform
SystemUnderTest
Adaptor
Test Scripts
Requirements Traceability
Model Coverage
Reliability
Mean TimeTo Failure
Intensityof Test
Test CasesTest
Cases
SystemRequirements
SystemRequirements
UsageModelUsageModel
Test ExecutionPlatform
Test ExecutionPlatform
SystemUnderTest
SystemUnderTest
Adaptor
Test Scripts
AdaptorAdaptor
Test Scripts
Test Scripts
Requirements Traceability
Model Coverage
Requirements Traceability
Model Coverage
Reliability
Mean TimeTo Failure
Intensityof Test
Reliability
Mean TimeTo Failure
Intensityof Test
THEORETICAL MBT WORKFLOW
AutomationTest Framework
Test Scheduler
Simulator
SystemUnderTest
Test Execution
Log
UsageModel
Test Cases
FunctionalRequirements
AtomicTest
Action
Test Cases
Verdicts
Confidence& Coverage
Reports
Test Scripts
AutomationTest Framework
A 1ST INDUSTRIAL TOOLS INTEGRATION
UsageModel
Test Cases
Simulator
SystemUnderTest
Test Execution
Log
Test Cases
Verdicts
Confidence& Coverage
Reports
FunctionalRequirements
AtomicTest
Action
Test Scheduler
Test Scripts
EXAM
UML
Python
dSpace
MicroNova
Carts
Vision CAN
IOs
Doors
MaTeLo
AutomationTest Framework
A 2nd INDUSTRIAL TOOLS INTEGRATION
UsageModel
Test Cases
Simulator
SystemUnderTest
Test Execution
Log
Test Cases
Verdicts
Confidence& Coverage
Reports
FunctionalRequirements
AtomicTest
Action
Test Scheduler
Test Scripts
dSpace
MicroNova
Carts
Vision CAN
IOs
Doors
TestStand
Proprietary
Opal-RT
NI
Clemessy
Diagnostic Calibration
Radio
Reqtify
MS Excel
MaTeLo
EXAM
UML
Python
AutomationTest Framework
VARIOUS TOOLS INTEGRATION
UsageModel
Test Cases
Simulator
SystemUnderTest
Test Execution
Log
Test Cases
Verdicts
Confidence& Coverage
Reports
FunctionalRequirements
AtomicTest
Action
Test Scheduler
Test Scripts
TestStand
Proprietary
dSpace
Opal-RT MicroNova
NI
Clemessy Carts
Vision
Diagnostic Calibration
CAN
IOs
Radio
Doors Reqtify
MS Excel
EXAM
UML
Python
PROVEtech Proprietary
CANoe EXACT
TTCN-3
VB
C, C#CSV
Java
Simulink CANoe
Flexray
Temperature
XML
CSVHP QC
MaTeLo
AutomationTest Framework
VARIOUS TOOLS INTEGRATION
UsageModel
Test Cases
Simulator
SystemUnderTest
Test Execution
Log
Test Cases
Verdicts
Confidence& Coverage
Reports
FunctionalRequirements
AtomicTest
Action
Test Scheduler
Test Scripts
TestStand
Proprietary
dSpace
Opal-RT MicroNova
NI
Clemessy Carts
Vision
Diagnostic Calibration
CAN
IOs
Radio
Doors Reqtify
MS Excel
EXAM
UML
Python
PROVEtech Proprietary
CANoe EXACT
TTCN-3
VB
C, C#CSV
Java
Simulink CANoe
Flexray
Temperature
XML
CSVHP QC
MaTeLo
AutomationTest Framework
MBT AND ITS STANDARDS
UsageModel
Test Cases
Simulator
SystemUnderTest
Test Execution
Log
Test Cases
Verdicts
Confidence& Coverage
Reports
FunctionalRequirements
AtomicTest
Action
Test Scheduler
Test Scripts
TestStand
Proprietary
dSpace
Opal-RT MicroNova
NI
Clemessy Carts
Vision
Diagnostic Calibration
CAN
IOs
Radio
Doors Reqtify
MS Excel
EXAM
UML
Python
PROVEtech Proprietary
CANoe EXACT
TTCN-3
VB
C, C#CSV
Java
Simulink CANoe
Flexray
Temperature
XML
CSVHP QC
RIF
ATX
ATX ATX
ODS
OTX HIL API
ODX
FIBEX XCP
GDI
ATX
MaTeLo
Items Tools Standard ApplicationFeeling
Requirements Doors RIF, ReqIF 3/5
Test Specification Model MaTeLo, Word - (DSL) -
Test Cases Specification Doors, Word, EXAM ATX, OTX 0/5 , 2/5
Test Campaign, Management MaTeLo, Quality Center ATX 0/5
Test Case Implementation EXAM, ECU TEST, TestStand, vb, python… HIL API, TTCN3 1/5 , 2/5
Test Execution - - -
•Physical Signal NI, dSPACE, Vector… HIL API 1/5
•Network Signal CAN, LIN, FlexRay… FIBEX 3/5
•ECU parameter CANape, INCA XCP 5/5
•Diagnostic CANdela, ODX, UDS 5/5
•Simulator Signal Simulink, AMEsim… HIL API 1/5
•Test Device Signal Agilent, LeCroy… GDI 3/5
Test Case Execution Log, Trace NI, EXAM, dSPACE, ETAS… ODS, ATML 2/5, 2/5
Test Case Verdict Quality Center, MaTeLo ATX 0/5
Issue BugZilla, Jira, Quality Center ISSUE ?
Requirement Coverage Doors, Quality Center, MaTeLo ATX, ReqIF 0/5 , 2/5
Confidence Indicator Quality Center, MaTeLo ATX 0/5
QUESTION ?
www.all4tec.netSales contact
[email protected] +33 6 80 88 40 59
Wiki & Forum & Documentationwww.all4tec.net
ABSTRACTION LAYERS
Usage Scenario Description
Test Flow ControlDevice Driver
Test Case Specification
System Under Test Hardware In the Loop Test Bench
PythonPrecompiled Application
UML Sequence Diagram
Markov-Chain Usage Model
<< derive >>
<< generate >>
<< control >>
MaTeLo
Third Party Equipment
ATX
HIL API
ODX, FIBEX, XCP…
EXAM ABSTRACTION MANAGEMENT
HIL Plateform
dSpaceNovaSim
CartsASAM HIL API
…Proprietary
Network
CAN_1CAN_2LIN_X
FlexRay_1
…FlexRay_2
Functions
Manual GearAuto GearHand Free
ACC
…StartStop
Test Tools
CANoeCANape
INCAMS Excel
…Diag Tool
Env. Model
GasolineDieselTurboHybrid
…Electric
Available Test Configurations
Test Case 1 Test Case 2 Test Case n
Strategic Test Cases
Possible Usage Scenarios
ATX
HIL APIHIL API ODX, FIBEX, XCP…
