Exit, Cry TearsDealing with Testing Review BoardsPaul GerrardGerrard ConsultingPO Box 347MaidenheadBerkshireSL6 2GU UK

e: paul@gerrardconsulting.comw: http://gerrardconsulting.comt: 01628 639173

Slide 3

Are Test Exit Reviews, and Testing Review Boards a

Challenge?

Exit Criteria Textbook approach is to set exit (or

acceptance) criteria e.g…- All tests executed- All tests passed- All incidents resolved, re-tested, signed-off- All outstanding incidents waived

This isn’t enough…- Need some expression of coverage as a

target- Theoretically, we can use test design

techniques

Coverage Targets Unlikely to use white box targets for

system, acceptance tests Black box targets:- Equivalence partitions- Boundary values- State transitions etc.

But how often are these enforced? How do you know it’s adequate? What if the TRB ask, “How do you

know?”

Coverage targets 2 Often, we have to use ad-hoc targets- 100% Branches in the business process

- 100% Transactions in end-to-end scenarios

- 100% Data variations in fields that are significant in processing flows

(OK 100% of SOME equivalence partitions)

But how do you know these targets are met unless you do an audit?

The pressure to accept, and press-on regardless Later test phases are on critical path Overrunning a test phase slips the whole

project but…- Is overlapping test phases an attractive

option?

- We could de-scope tests

- Demote high severity defects to lower severity?

More likely…- Just start the next phase anyway…

Demoting high severity defects to lower severity Severity classification should be based on

business viewpoint “Does this failure adversely affect the

acceptability of the system?” Business users may feel pressurised into

relaxing their original assessment- Their decision may cause a project slippage

Can you keep the project “honest”? Do you have to remind the team of the meaning of severity?

Starting the next phase anyway Faults found (and fixed) in one phase may

invalidate tests in the overlapping phase Known faults may block tests in the later

phase Resource limits, (in dev or test) slows

progress – so no advantage may be gained Environmental availability may block

approach and the required compromises may not be safe

Again, can you keep the project “honest”.

The challenges of ‘exit’ Are your test execution metrics misleading? Do you reports tests, not steps? Where’s the bottleneck – execution, checking, dev? Do managers focus on the good news or bad news? Does environment downtime make it look worse?- Do testers find other things to do or just sit around?- Or do they work late to compensate for other folk?

Are your managers over zealous, bullying? Are your managers pessimistic, over cautious? Are you asked to make judgements (on their

behalf)?