Winter 2007 SEG2101 Chapter 12 1

Chapter 12

Verification and Validation

Winter 2007 SEG2101 Chapter 12 2

Contents

• What is verification and validation?• Reviews• Testing SDL systems• Dynamic analysis of interfaces

Winter 2007 SEG2101 Chapter 12 3

Verification and Validation • Verification:

– to establish the truth of correspondence between a software product and its specification

– are we building the product right– scope of verification is process quality

• Validation– to establish the fitness or worth of a software product

for its operational mission– are we building the right product– scope of validation is system quality

Winter 2007 SEG2101 Chapter 12 4

Techniques for Verification and Validation

• Manual review• Testing SDL systems• Dynamic analysis of interfaces

Winter 2007 SEG2101 Chapter 12 5

Reviews• Reviews may be performed on several levels in a

project.• As a minimum, the quality of milestone results

should be assessed.• Two purposes of formal review:

– to provide feedback to management on the status of products under review

– to stimulate the development team towards producing quality results

• Commenting

Winter 2007 SEG2101 Chapter 12 6

Testing SDL systems

• The purpose of testing• Test environment• Testing a single SDL process• Test complete SDL system

Winter 2007 SEG2101 Chapter 12 7

The Purpose of Testing

• Two distinct aspects to test:– the functionality– the implementation

• Purpose of functionality testing:– verify the functional design against the functional requirements– detect possible internal inconsistencies and errors in functional design– validate the functional design against the users and owners needs

• Purpose of implementation testing:– verify the functionality of the implementation against functional design– verify the non-functional properties of the implementation against the non-

functional requirements

Winter 2007 SEG2101 Chapter 12 8

Test Environment• Requirements for testing environment:

– Inputs, including time-outs, may be generated and replayed.– The responses can be traced.– The execution can be controlled.

• The signal interfaces of SDL hide the internal structures of blocks and processes from each other. There is no difference between the test environment and the real environment as seen from the processes under test.

• Two kinds of problems may remain after a test in simulated time:– the implementation may be too slow to handle the traffic load or meet

response time requirement– there may be time-dependent synchronization problems with the real

environment or with real timers

Winter 2007 SEG2101 Chapter 12 9

Testing a Single SDL Process• A beautiful property of pure FSM is that the effort needed to test an

implementation exhaustively is proportional to the number of transition in the state transition diagram.

• Every possible behavior is explicitly described by states and transitions. The state transition diagram itself may serves as the test plan.

• Work through the diagram transition by transition and check that the specifies outputs are generated that the next state is reached.

• This must be done for some characteristic values. Values must be selected so that each branch of a decision is carefully tested.

• Each asterisk transition and each save must be tested.• Normally there is no point in testing units smaller than an SDL process.

Winter 2007 SEG2101 Chapter 12 10

Testing Complete SDL Systems• Most SDL systems consists of many processes operating

concurrently and interacting with each other through a network of channels and signal routes.

• Implementation testing at this level means to test the implementation of concurrency support, synchronization, signal transfer mechanisms and timers.

• To verify that the functionality is right we need to compare the actual behavior on the system interfaces with the required behavior. MSC

• Before SDL processes are integrated it is well worth the effort of inspecting the signal interfaces manually.

• Dynamic problems: deadlock, livelock

Winter 2007 SEG2101 Chapter 12 11

Testing Complete SDL Systems (cont’)

• In principle, we must test all possible combinations of states and transitions in all the processes of the system.

• Test cases– Normal behavior - the system behaves as intended in all the

normal situations.– Exceptional behavior/situations- try to provoke errors by giving

inputs that are not specified and check the exception handling is correct.

– Try to reveal dynamic interworking problems by running input sequences concurrently on different interfaces.

– When can we stop testing? (cost of testing, cost of undetected bug)

Winter 2007 SEG2101 Chapter 12 12

Validating Interfaces

• The validation of interfaces has two aspects:– static checks on signals– dynamic behavior analysis

• By inspecting the states and transitions of the global behavior graph one may find errors. Deadlock will appear as states in the graph without further transitions. - reachability analysis.

Winter 2007 SEG2101 Chapter 12 13

Two SDL Processes

Winter 2007 SEG2101 Chapter 12 14

Global Behavior Graph

Winter 2007 SEG2101 Chapter 12 15

Example: Transition Coverage Tree

Winter 2007 SEG2101 Chapter 12 16

Example: Symbol Coverage Tree