50.530: Software Engineering Sun Jun SUTD 1. Week 12: Software Model Checking 2

1

50.530: Software Engineering

Sun JunSUTD

2

Week 12: Software Model Checking

Determining whether a program satisfies a property by the means of exhaustive searching.

3

Software Model Checking

Program

Model Checker

PropertyCounterexample!

What is “property”?

4

How Model Checking Works?

System behaviors

PropertyProgram

• Three researchers won Turing Award 2007 for their pioneer work on model checking!

• Intel i7 processor is verified by symbolic model checking completely without a single test case!– 8 cores, millions of registers; functional verification!

• The Slam project from Microsoft successfully detected many bugs in many driver software!– Dozens of K lines of C codes; debugging.

5

Model Checking Works

6

FUNDAMENTALS OF MODEL CHECKING

7

Model: Kripke Structure

A Kripke structure is a tuple (S, R, L, I) where • S is a set of states; • R is a set of transitions; • I is the nonempty set of initial states; • L labels each state by a set of atomic

propositions.

8

Model Example: Microwave Oven

1

2 3 4

5 6 7

{start,error}

{close} {close, heat}

{start,close}

{start,close,heat}

{start,close,error}

start oven

open doorclose door

resetstart oven

warmup

start cooking

cook

open door

done

open doorclose door

The transition labels are not part of the Kripke Structure.

9

Model Example

A program can be transformed to a Kripke Structure.

L0 x = 0;L1 while (x < n) {L2 x++;L3 }L4 if (x <= 0) {L5 error();L6 }

Each state is represented by the (l,x,n) where l represent the line number; x is the value of variable x; and n is the value of n. The set of labels are: {error}. Question: how many states are there?

0,0,-1 1,0,-1 4,0,-1 5,0,-1{error}

0,0,0 1,0,0 4,0,0 5,0,0{error}

0,0,1 3,1,11,0,1 2,0,1……

0,0,2 3,1,21,0,2 2,0,2……

………………………………………………………….

10

Property: Temporal Logic

• Temporal logic (CTL, LTL, CTL* among many others) extends propositional logic with temporal operators.

• Proposed to specify properties about programs (in particular, program paths).

Turing award 1996 for his work on introducing temporal logic.

11

Linear Temporal Logic

LTL is built up from a finite set of propositions, the logical operators ¬ and ∨, and the temporal modal operators (X, G, and F).– p: p holds at the current state

– X p: p holds at the state after one transition

– G p: p holds on every state in the path

– F p: p holds on some future state in the path

…p

…p

…pp p p

…p

…

12

LTL Examples

• G !error– an error should never occur.

• G (!heat close)∨– it is never that case that the microwave oven is heating

and not closed.

• G (error => F heat)– from a state labelled with error, it will eventually reach a

state labeled with heat.

13

LTL Verification• A trace of a Kripke Structure is a sequence of

labels obtained by traversing through a path in the structure.

• A Kripke Structure satisfies an LTL formula iff every path in the structure satisfies the formula. – G !error ?– G (!heat close) ?∨– G (error => F heat) ?

14

LTL Verification Algorithm

Example: G p• Model checking G p works by traversing through every

state of the Kripke Structure (typically using BFS or DFS)

Example: GF p• Model checking GF p works by finding

– a loop in the Kripke Structure such that no state in the loop is labelled with p• Standard loop finding algorithms are like Nested DFS, Tarjan’s

Strongly Connected Component algorithm.

– a deadlocking state not labeled with p

15

Counterexamples

Example: G p• A counterexample is a finite path in the Kripke

structure which ends with a state not satisfying p.

Example: GF p• A counterexample is a path which leads to a

loop such that p is never satisfied during the loop.

16

Counterexample Examples

• G !error– <3, 1, 2>

• GF heat– <3,1>*

17

State Space Explosion

State Space Explosion is perhaps the single most important problem of model checking.

L0 x = 0;L1 while (x < n) {L2 x++;L3 }L4 if (x <= 0) {L5 error();L6 }

0,0,-1 1,0,-1 4,0,-1 5,0,-1{error}

0,0,0 1,0,0 4,0,0 5,0,0{error}

0,0,1 3,1,11,0,1 2,0,1……

0,0,2 3,1,21,0,2 2,0,2……

………………………………………………………….

18

Parallel Composition

The following models a traffic light system.

The light model The car model

19

Parallel Composition

The overall model (where one transition of the light model and one of the car model always occurs synchronously)

Parallel composition often leads to state space explosion.

20

Abstraction: Example

Concrete Kripke Structure

Abstract Kripke Structure

0 1 2

3 54

6

03

1 5

42

6

{p} {q}{p,q}

{p}

{p,q}

{p}

{p} {p,q}

21

Abstraction: Example

L0 x = 0;L1 while (*) {L2 x++;L3 }L4 if (x < 0) {L5 error();L6 }

0,0

1,0

2,0 3,1

Abstraction: For each control location, let’s group the states into two groups.• One contains all states which satisfy x >=0.• One contains all states which satisfy x < 0.

1,1 4,1

2,1 3,2 1,2 4,2

2,23,3

1,3 4,3

……

4,0

22

Abstraction: Example

L0 x = 0;L1 while (*) {L2 x++;L3 }L4 if (x < 0) {L5 error();L6 }

0,0

1,0

2,0 3,1 1,1 4,1

2,1 3,2 1,2 4,2

2,23,3

1,3 4,3

……0, x>=0

1, x>=0 2, x>=0

4,0

3, x>=04, x>=0

23

Abstraction: Definition

A Kripke Structure A = (Sa, Ra, La, Ia) is an abstraction of a Kripke Structure C = (S, R, L, I) if• Sa is a set of subset of S.• Ra contains a transition (s,s’) where s and s’ are

in Sa if and only if there exists x in s and x’ in s’ such that (x,x’) is in R.

• La(s) for any s in Sa is the union of L(x) for all x in s.

• Ia is a subset of S containing I.

24

Theorem

Theory: If A satisfies an LTL formula, then C satisfies the formula too.

Proof: Every trace of C is a trace of A. Ergo.

25

Question

Is G !(r && d) satisfied or not?

26

Exercise 1

Abstract the model by grouping state green and yellow into one.

27

Exercise: Solution

Abstract the model by grouping state green and yellow into one.

Is G !(r && d) satisfied or not?

28

Abstract Programs

It does not make sense to construct the concrete Kripke Structure first and then the

abstraction. Right, we need a systematic way of generating abstraction from the

program syntax, and never construct the concrete Kripke

Structure.

29

AUTOMATIC PREDICATE ABSTRACTION OF C PROGRAMS

Thomas Ball et al. PLDI 2001, most influential paper award

30

Predicate Abstraction

Ordinary C programs Boolean C programs

Given n predicates and a C program, C2BP automatically construct a C program which only contains n Boolean variables, each of which corresponds to a predicate. It is guaranteed that the Kripke Structure of the Boolean program is an abstraction of that of the original program.

31

Predicate Abstraction: Example

where • the set of predicates is {x>=0};• assume(b) means that we assume that b is true there and we

would ignore the cases where b is not true.

L0 x = 0;L1 while (*) {L2 x++;L3 }L4 if (x < 0) {L5 error();L6 }

L0 b=true;L1 while (*) {L2 if (b) {b=true} else {b=*};L3 }L4 if (*) { assume(!b);L5 error();L6 }

32

Predicate Abstraction: ExampleL0 x = 0;L1 while (*) {L2 x++;L3 }L4 if (x < 0) {L5 error();L6 }

L0 b=true;L1 while (*) {L2 if (b) {b=true} else {b=*};L3 }L4 if (*) { assume(!b);L5 error();L6 }

0,0 1,0

2,0 3,1 1,1 4,1

2,1 3,2 1,2 4,2

2,23,3

1,3 4,3

……

4,0

0,b

1,b 2,b 3,b4,b

33

Predicate Abstraction: Assignment

Let b1,b2,…,bk be the Boolean variables corresponding to the predicates p1,p2,…,pk. A cube is a formula c1 c2 … ck ⋀ ⋀ ⋀(where ci is either !bi or bi).

Ideally, an assignment x := exp is translated into if (p) {b := true}if (n) {b := false}if (u) { b = *}

for any b; for any cube p such that {p} x := exp {b} holds; for any cube n such that {n} x := exp {!b} holds; and for any cube u such that neither {u} x := exp {!b} nor {u} x := exp {b} holds. In reality, we often abstract this so that we don’t have to check all cubes.

34

Predicate Abstraction: Conditional

A conditional if (cond) { … } else { … }

is translated toif (*) { assume(c); //c is any bi or !bi such that cond => c …}else { assume(nc); //nc is any bi or !bi such that !cond => nc …}

35

Predicate Abstraction: While• A while loop is interpreted as a goto

statement plus a conditional– The goto statement is simply copied– The conditional statement is translated as

explained in the last slide.

while(cond) { …}

while(*) { assume(cond); …}

do {L1: … } while(cond)

L1: … if(cond) goto L1

36

37

1: if (*) { 2: do {3: got_lock = 0;4: if (*) {5: lock();6: got_lock ++;7: }8: if (got_lock) {9: unlock();10: }11: } while (*) ;12: }13: do {14: lock();15: old = new;16: if (*) {17: unlock();18: new ++;19: }20: } while ( new != old);21: unlock ();

Exercise

Assume that we know

Question: Is error reachable?

lock()

unlock()unlock() lock()

{error}

38

ExerciseAssume that we know

Property: G !((pc=5⋁pc=14) locked) && ⋀ G !((pc=9 pc=17 pc=21)⋁ ⋁ !locked)⋀

lock()

unlock()unlock() lock()

{error}

1: if (*) { 3: got_lock = 0;4: if (*) {5: lock();6: got_lock ++;7: }8: if (got_lock) {9: unlock();10: }11: if (*) {goto 3;}12: }14: lock();15: old = new;16: if (*) {17: unlock();18: new ++;19: }20: if (new != old) {goto 14;}21: unlock ();

39

Exercise

Construct a Boolean program using predicates {locked} where locked is predicate denoting whether it is locked.

1: if (*) { 3: got_lock = 0;4: if (*) {5: lock();6: got_lock ++;7: }8: if (got_lock) {9: unlock();10: }11: if (*) {goto 3;}12: }14: lock();15: old = new;16: if (*) {17: unlock();18: new ++;19: }20: if (new != old) {goto 14;}21: unlock ();

Assume lock() is implemented by simply assigning locked to true; unlock() is implemented by simply assigning locked to false.

40

Exercise Solution

Property: G !((pc=5⋁pc=14) locked) &&⋀G !((pc=9 pc=17 pc=21)⋁ ⋁ !⋀locked)

Is this property satisfied or not based on the abstraction?

1: if (*) { 3: skip;4: if (*) {5: locked=true6: skip;7: }8: if (*) {9: locked=false;10: }11: if (*) {goto 3;}12: }14: locked=true;15: skip;16: if (*) {17: locked=false;18: skip;19: }20: if (*) {goto 14;}21: locked=true;

41

COUNTER-EXAMPLE GUIDED ABSTRACTION-REFINEMENT

Clarke et al. Journal of the ACM 2003

42

Theorem

Theory: If A satisfies an LTL formula, then C satisfies the formula too.

What if A does not?

43

Exercise 3

Abstract the light model by grouping state green and yellow into one state.

44

Parallel Composition

• Then construct the parallel composition and check if the composition satisfies the property.

45

Spurious Counterexamples

If a counterexample is found while model checking A, it doesn’t mean that C doesn’t satisfy the property, i.e., the counterexample could be spurious.• e.g., is G !(r && d)

satisfied by this model?

46

Spurious Counterexamples

Is G !(r && d) satisfied by this model? A spurious example:

<rs, ws, rd>

47

Analyzing Spurious Counterexample

Step 1: from rs to ws

48

Analyzing Spurious Counterexample

Step 2: from ws to rd

This step is broken in the concrete system!

An abstraction where green and yellow are separated will not have this spurious counterexample!

49

If a counterexample is spurious, the counterexample must be broken at some step!

We can always get rid of a spurious counterexample by refining the abstraction!

50

The Problem

The least abstractThe most abstract

Very small and easy to check;

Lots of spurious counterexamples

Very big and hard to check; No

spurious counterexamples

51

Can we find the right abstraction so that it is not very big and we can find real a counterexample or show there is none?

52

CEGAR

Construct the initial abstraction

Model check the abstraction

Report “system verified”

If no counterexample is found

Check spuriousness

If a counterexample is found

Report counterexample

If it is not spurious

Refine the abstraction

If it is spurious

53

CEGAR: Example

1. do {2. lock();3. old=new;4. if (*) {5. unlock();6. new++;7. }8. } while (new != old)

Property: G !(pc=2 locked) ⋀

54

Initial Abstraction

We should group two states only if• they have the same truth value for all atomic

subformulae in the property, and• they are at the same control location.

In other words, we apply predicate abstraction with an initial set of predicates containing only the atomic subformulae of the property.• For instance, {x > 0, x+y=4} if the property is

(G x > 0 => F x+y=4).

55

CEGAR: Example

1. do {2. lock();3. old=new;4. if (*) {5. unlock();6. new++;7. }8. } while (new != old)

Property: G !(pc=2 locked) ⋀

1. 2. locked=true;3. skip;4. if (*) {5. locked=false;6. skip;7. }8. if (*) {goto 2;}

abstract with {locked}

56

CEGAR: Example

1. 2. locked=true;3. skip;4. if (*) {5. locked=false;6. skip;7. }8. if (*) {goto 2;}

2

3

4

5

6

7

8

{!locked}

{locked}

{locked}

{locked}

{!locked}

{!locked}

{!locked} 8 2{locked} {locked}

Is the property satisfied, with this

abstraction?

57

Check Spuriousness

• Given a counterexample, i.e. a path of the Boolean program, we can check whether it is spurious using symbolic execution.

58

SpuriousnessSymbolic execution:locked=false //initial condition ⋀locked1=true* //line 2⋀old = new //line 3⋀new!=old //condition from line 8⋀

*for simplicity, assume that lock() is locked = true and unlock() is locked = false.

1. do {2. lock();3. old=new;4. if (*) {5. unlock();6. new++;7. }8. } while (new != old)

2

3

4

{!locked}

{locked}

{locked}

8

2

{locked}

{locked}

Unsat

59

Abstraction Refinement

If the counterexample is spurious, it must be broken somewhere. Abstraction refinement is to find a new predicate such that the spurious counterexample is removed.

l1 l2 l3 l4

60

Refinement: Weakest Precondition

wp(prog3, l4) would be such a predicate.

l1 l2 l3 l4prog3prog1 prog2

61

Refinement: Weakest Precondition

Example:

1. 2. lock();3. old=new;4. if (*) {5. unlock();6. new++;7. }8. if (new != old) {2. lock(); }

What is the weakest precondition at line 8 for reaching line 2 (after line 8) with the following post-condition?

locked = true

Answer: locked=true new!=old⋀

Since locked=true is already used for abstraction, the new predicate is new!=old.

62

Refinement: Interpolant

An interpolant based on the path condition of the spurious counterexample would be such a predicate.

The interpolant at line 8 is old=new.

locked=false //initial condition ⋀locked1=true* //line 2⋀old = new //line 3⋀new!=old //condition from line 8⋀

63

Refinement: Example

1. do {2. lock();3. old=new;4. if (*) {5. unlock();6. new++;7. }8. } while (new != old)

Property: G !(pc=2 locked) ⋀

1. 2. locked=true;3. ne=false;4. if (*) {5. locked=false;6. if(!ne) {ne=true} else {ne=*;};7. }8. if (ne) {goto 2;}

Abstract with {locked, new!=old}. Let ne be a Boolean which is true iff new!=old.

64

Exercise

Property: G !(pc=2 locked) ⋀

1. 2. locked=true;3. ne=false;4. if (*) {5. locked=false;6. if(!ne) {ne=true} else {ne=*;};7. }8. if (ne) {goto 2;}

Draw the Kripke Structure of the following program and check whether the property is satisfied or not.

65

Exercise: Result

2

3

4

5

6

7

8

{!locked, ne}

{locked, ne}

{locked, !ne}

{!locked, !ne}

{!locked, ne}

{!locked, ne} 8{locked, !ne}

Is the property satisfied, with this

refined abstraction?

1. 2. locked=true;3. ne=false;4. if (*) {5. locked=false;6. if(!ne) {ne=true} else {ne=*;};7. }8. if (ne) {goto 2;} {locked, !ne}

Property: G !(pc=2 locked) ⋀

66

Recap

67

State-of-the-Art

SLAM2• Part of Static Driver Verifier (SDV) 2, released with the Windows

7 WDK.

• Is capable of verifying (falsifying) programs with dozens of thousands of lines of codes.

• For SDV 2.0, the true bugs/total bugs ratio is 90-98% on Windows 7 Microsoft drivers, depending on the class of driver.

• The number of non-useful results (timeouts, “don’t know” results) for drivers shipped as WDK samples, is 3.5% for WDM drivers and 0.02% for KMDF drivers.