1
if you use this format
with a picture in
the vertical-stripe
format, then adjust
the RH edge of the title bar to be just L of the stripe.
Test Automation –How far should it go?
Ian Gilchrist, IPL
2
if you use this format
with a picture in
the vertical-stripe
format, then adjust
the RH edge of the title bar to be just L of the stripe.
20 minutes
15 minutes
10 minutes
Contents 1. Context and Introduction2. C++ Example (‘Reverse
String’) with script generation, coverage analysis, wrapping
3. C example (‘Reactor’) with table-driven testing, coverage optimisation, robustness testing
3
Context• C and C++ software• Unit and Integration tests• ‘High integrity’ systems
• Up to and including ‘safety-critical’
4
The Back StoryPersonal history• I entered the software industry in 1983 as a
programmer working mainly in Assembler code• Testing then consisted of ‘suck and see’• By 1985 we were using test facilities based on use of
emulator, but requiring much manual set up, and interpretation of results from hex output dump of memory.• Not pleasant to do!• Not repeatable!
5
SystemRequirements
ArchitecturalDesign
UnitDesign
DetailedDesign
Code
Software Development Lifecycle
SystemTest
AcceptanceTest
UnitTest
IntegrationTest
Source code Analysis
Code
Does code do what it should?Is code tested sufficiently? Code Coverage requirements
Dynamic testing proof
6
Why Unit Test?
Regulated Industries (avionics, medical, etc)• Mandated or highly recommended• Structural code coverage• Variation by SIL (System Integrity Level)
Business-critical• Making a profit!• Protecting reputation • Avoidance of litigation
7
Unit/Integration Test - Technical Challenges Test harness creation
Finding time to create reusable test harnesses to test all functionality
Control of interfacesMaking it simple to test object interactions with the rest of the system
Flexibility of Black-Box/White-box and OO techniquesHaving a full tool-kit to test efficiently
Selection/Generation of test data for different purposeFunctional tests – does code do what it should?Robustness tests – does code survive large data sets?Baseline tests – we know code’ works’ but want a baseline against which to test changes
Code coverageConfidence that tests cover the important code Evidence generated is sufficient to meet standards
8 8
Test Harness Creation• Test driver harness for units • Structured auto-repeatable test cases • Checking of Outputs• Results production
Automation can do this:• Parse of project code• Generate structured test script
• case per function/ method• parameters, call order, global data, returns• test case independence
• Incorporate checks according to variable type• Positive and negative checking of global data• Results production
9 9
Control of Interfaces• Isolation of units from system • Order of calls• Different behavior on calls• Checking/modifying parameters
Simulate
Intercept
Intercept
Automation can do this• Knowledge of calls made• Automated generation of
• Stubs to simulate calls• Wrappers to intercept calls
• Programmable instances for Stubs and Wrappers• Checks on parameters based on types
10
Stubfor External
Object
SourceCode External
Object
A function/method in test script with programmable instances
Replaces call to external software, firmware or hardware
Check ParametersCheck Call Sequences
Choose Return Parameter value(s)
Stub is a dummy function replacinginterface to theExternal Object
Stubs
11 11
Wrapper for External Object
BEFOREWrapper
AFTERWrapper
SourceCode
ExternalObject
Modify Out ParametersCheck Out ParametersCheck Call sequences
Modify In Parametersand Return
Check In Parametersand Return
WrappingA function/method in test script with programmable instances using Before-After or Before-Replace wrapper pairs
Intercepts call to external software, firmware or hardware
12 12
Set + Set + Check
PrivatePublic
Test Driver Data
Checks
White (clear) Box Testing• Call private methods/static functions• Set/check private/static data • Control of internal calls
Automation can do this:• Automated accessibility instrumentation
• Call private methods and static functions• Set/check data which is private, in unnamed
namespaces and declared static• Wrapping calls internal to compilation unit• Wrapping OS library calls
13 13
Black Box Testing• Large data input sets • Checking large output sets• Robustness tests
Automation can do this:• Table-driven test generation
• Multiple values per parameter• Ranges of values• Functions calculating values• Combinatorial effect calculator• Checks on call sequences and returns
• Robustness rule sets for data types
Public
Test Driver Data
Checks
14 14
Object Oriented Code• Test case re-use aligned with code• Support for Templates• Support for Inheritance
Automation can do this:• Parallel hierarchy of code and test case re-use• C++ template instantiation and testing• C++ inheritance and factory method testing • Object oriented code coverage
15 15
Integrated Code Coverage• Set coverage target in tests• Diagnostics over stages / test runs• Sensible coverage metrics• Coverage redundancy
Automation can do this:• Coverage Rule Sets• Powerful drill-down views, filters and reports• Coverage Metrics
• Entry- Point Coverage• Statement Coverage• Decision Coverage• Call-return Coverage• Condition Coverage (including MC/DC)
• Test case coverage optimization
16
Large Legacy Code Base• Automatic generation of unit tests• Reducing reliance on system regression tests• Identifying testability issues
Automation can do thisAutomatic generation of passing C unit test baseline Automatic regression suite of makefiles Knowing testability limitations in code
Dynamically unreachable codeCrash scenariosData uninitialised or function staticImplicit function declarationsCompiler type truncation
17
if you use this format
with a picture in
the vertical-stripe
format, then adjust
the RH edge of the title bar to be just L of the stripe.
C++ Example – ‘Reverse String’C Example – ‘Reactor’C Example – ‘Airborne’
End of Introduction
18
if you use this format
with a picture in
the vertical-stripe
format, then adjust
the RH edge of the title bar to be just L of the stripe.
If you have any questions, please email:[email protected]
THANK YOU FOR ATTENDING!
End of Presentation
Recommended
