Non-standard Software Testing: from Process Algebras to Programmable Hardware.

John Clark

(Material drawing on talks and work with Susan Stepney, Jill Srivratanakul, Fiona Polack, Rob Hierons)

FORTEST Jan 6-7 2003. Sheffield.

• To describe and categorise uses of fault injection.

• To indicate how some analysis techniques form the safety domain can enable us to be a little more rigorous in the way we go about fault injection.

• To indicate some non-standard attempts.

• Some is work already done but in general wish to talk about work in progress and ideas for future work.

Objectives

• Fault-based testing – a sort of hypothetical fault injection.

• E.g. assume that the programmer has ‘injected’ the fault and seek tests to discover whether this is in fact the case.

• Clearly very common.

• Most obviously as the basis for finite state machine testing for protocols.

Forms of Fault Injection I

• Mutation testing. A good deal of research has been carried out into this.

• Gives rise to a well understood coverage criterion (adequacy).

• Most mutation has been carried out on code.

• Mutation operator sets for C and derivatives, FORTRAN, and large set suggested for Ada.

• Question – where exactly do these sets come from?

• Some specification mutation work (starting with Budd).

Forms of Fault Injection II

• Fault injection for dependability.

• For good reasons, specifications give the functionality that is required – omitting failure conditions (e.g. of hardware).

• However, things fail (over-heat, wear out, exercised outside pre-conditions etc.)

• For dependable systems (safety, security etc) we need to acknowledge this and take it into account

• e.g. aircraft are typically required to be resilient to single point failure.

Forms of Fault Injection III

• One use of fault injection is therefore to act as putative component failures.

• Can assess whether the described system and all its single point mutants uphold particular properties.

• Thus, we have a family of systems, each of which must be safe/secure.

• Each must be subject to analysis – model checking would appear to come into its own here.

Families of Failing Systems

• Hypothetical behavioural fault injection. Here deviations from expected behaviour are hypothesised and their effects considered.

• Origins in the safety and security industries:

• Security: flaw hypothesis method (FHM)

• Safety: functional failure analysis, HAZOPs.

• Generally referred to as “what if” analyses. Very widely used.

Forms of Fault Injection IV

• How do we decide which faults are to be injected (either for real or hypothetically)?

• For simple cases there may be obvious faults to consider, e.g. for real Finite State Machines state transfer errors suggest themselves immediately.

• Other cases, e.g. error based testing at the boundaries seems driven by (in many cases well-founded) established practice.

Where do Faults Come From?

• For much code based mutation there would appear to be little in the way of ‘method’ giving rise to mutation operator sets.

• Have found references to ‘analysis of the syntax’ of language under consideration.

• Many mutation operators are familiar across the various languages.

• Swapping of + - / * etc.

• Off by 1

• Variable replacements etc.

• But there would appear to be little guidance for non-standard elements.

Where do Faults Come From?

• We wanted to develop a mutation tool for Java. OO features are interesting.

• However, despite its popularity, error taxonomies were fairly thin on the ground and so we didn’t have much of a basis on which to build a set.

• General aim was to come up with an independent adequacy measure for OO testing methods (there seemed to be quite a number and it wasn’t clear what they actually achieved).

• Needed a method – HAZOPS.

Mutation Operators for Java

HAZOPs• Hazard and Operability Studies HAZOPS: A

flexible team-based activity, originally developed by ICI, can be thought of as structured brainstorming:

The application of a formal systematic critical examination to the process and engineering intentions of new facilities to assess the hazard potential of mal-operation or malfunction of individual items of equipment and the consequential effects on the facility as a whole.

British Chemical Industries Safety Council publication “Safety Audits”

Technical Principles• Take a design for the system.

• Consider each part of the design to discover how DEVIATIONS from the design INTENT can occur and whether these can lead to hazards.

• Formulation of questions directed at each part of the design is motivated by the use of GUIDEWORDS

• CONSEQUENCES of the deviations are investigated and if deemed to be hazardous their CAUSES are sought.

Example System

Overflow

Material B

Material A

Product C

Explosion may occur if concentration of B exceeds that of A Attribute: Flow of AGuideword: NO (NOT)“No FLOW of A”Consequences: potential explosionCauses? Empty supply tankpump failure, peipeline fracture, isolation valve closed

Attribute: Flow of AGuideword: MORE“MORE FLOW of A”Consequences: reaction produces contaminated C. Possible overflow. Are these hazardous?Causes? Pump working too hard...

Guidewords

• NO or NOT or NONE. The complete negation of the intention.

• MORE and LESS. Quantitative increase or decrease.

• AS WELL AS. A qualitative increase.

• PART OF. A qualitative decrease.

• REVERSE. The logical opposite of the intention.

• OTHER THAN. Complete substitution.

HAZOPS ProcessDefine Objectives

and scope

Select the team

Prepare for study

Carry out study

Follow up and record results

Purpose of resultsNature of hazardsRelevant legislationWhere in the lifecyclePhysical boundariesDesign representationsLevel of detail

Study leaderRecorder/secretaryDesignersUsersSpecialistsSize

Technical, e.g nodes and attributes, guidewords and interpretations

Administrative: team management , supporting documentation, rooms etc.

Methodical examinationTry not to be too negative

Emphasis varies on purpose

Issues identified must be investigated

• Classes:

• Class modifiers – access modifier (public), abstract , final.

• Class names; Components – package declarations, imported types, members (fields and methods both declared and inherited)

Java Constructs and Attributes

MethodInvocation:

MethodName (ArgumentListopt)

Primary.Identifier(ArgumentListopt)

super.Identifier(ArgumentListopt)

Applying OTHER THAN leads to consideration for example of wrong method selection.

• MethodName may be wrong

• Primary expression incorrect

• Super class not the intended one

• Identifier wrong

• Argument list wrong.

Method Invocation Expression

MethodInvocation:

MethodName (ArgumentListopt)

Primary.Identifier(ArgumentListopt)

super.Identifier(ArgumentListopt)

Consider Argument list wrong. An arg list has a set of atrtributes, e.g. type, size and order of arguments. Using OTHER than with ‘argument order’ leads to consideration of permutations of arguments. Consideration of LESS leads to argument deletion.

Swapping arguments may well produce semantically valid mutants and the more generic (Object obj) the formal parameters the more likely this is to be the case.

An argument has attributes too, e.g. a run-time type. OTHER THAN leads to consideration of coercion to another type.

Method Invocation Expression

Name:

SimpleName

QualifiedName

SimpleName:

Identifier

QualifiedName:

Name.Identifier

Looking at the accessibility and WIDENING leads to public being used instead of private or protected (or none being specified where it should be private etc.)

Similarly for NARROWING.

Accessibility Example

• Have implemented a CSP mutation tool. Since not that much around decided also to use HAZOPs to determine possible mutants.

CSP Mutation

• Event Drop:

• CLOCK=tick->tock->CLOCK

• CLOCK=tick-->CLOCK

• Event Swap:

• CLOCK=tock->tick->CLOCK

• Event Replacement Operator:

• VCMT=coin->(choc->VCMT|toffee->VCMT)

• CUST=coin->toffee->CUST

• VCMT=coin->(choc->VCMT|toffee->VCMT)

• CUST=coin-> coin ->CUST

CSP Mutation

• Event Insertion:

• CLOCK=tick->tock->tock->CLOCK

• Choice operator replacement:

• P |_| Q

• P [] Q and vice versa

• Tool allows selective mutation. This is important since in practice some mutant behaviours are more likely than others to actually occur in practice. Thus for example event drop may correspond to

CSP Mutation

• Field programmable gate arrays (FPGAs) are beginning to catch on big time.

• Essentially 2D arrays of programmable cells with primitive functionality communicating Boolean values with neighbouring cells NSEW.

• Outputs are functions of inputs and current cell state.

• MASSIVE PARALLELISM – all cells execute in parallel.

• Recent emergence of high level software languages that can be compiled to hardware.

• Handel C based on Occam/CSP model

• Many of the usual C constructs

• But allows for parallel tasks, channels, flexible data types (e.g. 5 bit integers)

Handel C

• Object code (Java Byte Code?)

• Very little at mutation at low level of detail.

• Need to consider what such mutation would mean.

• Coverage would need to be thought through for FPGAs since certain areas on the chip would be ‘spare’. This would perhaps be for mutating netlists, the low level programming of cells.

Other Things to Mutate

• Z specifications.

• Have tried manually mutating Z specs and checking whether proofs still hold.

• Good but very tedious way of validating the spec, it forces you to think about the spec in detail.

• Can also mutate assertions in code, e.g. the falsification testing of Tracey attempts t find data satisfying PRE and NOT(POST)

• More generally – mutate the actual assertions/conjectures to be proved and ask “Why not this” – what good is proof if you are not proving the right thing.

Other Things to Mutate

• Descriptions of secure systems. Has proved a very effective way of teasing out requirements from use cases.

• E.g. the user is required to supply identification details to register with the e-commerce system.

• By homing in on details (in detail) we can ask what happens if many attempts are made (MORE) and leads to questioning what it means to supply details, do you care if several slightly different details are supplied (JA Clark, John A Clark etc). You might if you have a one per customer offers.

• In many designs elements such as acknowledgements are omitted (this may be a security problem!) E.g. users will resend ‘buy’ request or simply repeat.

Other Things to Mutate

• Have been involved in various mutation oriented projects – developed from scratch every time.

• But what about a Generic Mutation Tool:

• Grammar-grammar

• Manipulation language (involving syntactic and semantic elements)

• Handle process algebras, Z, B, code etc.

• Hooks needed for language specific elements.

Finally