Improving Error Discovery using Guided Search
Neha Rungta & Eric MercerComputer Science Department
Brigham Young University,
Provo UT
Verification and Validation, CS Dept, BYU 2
Software Model Checking Motivation
Ariane 5 Comair debacle
Verifying Software Models A transition graph for the model is created A predefined property is verified ex. Reachability
Problem Number of behaviors is exponential with every increment This causes a state explosion problem
Verification and Validation, CS Dept, BYU 3
Approaches Traditional techniques to counter it
Parallel or Distributed Model Checking Predicate Abstraction Disk based Algorithm Heuristics for Guided search
Heuristics Find a counterexample before memory runs out Property based heuristics Structure based heuristics
Structure of program can be use to guide the search
Verification and Validation, CS Dept, BYU 4
Current Structural heuristics Stefan Edelkamp and Tilman Mehler Finds a short and easy to understand Error trail Minimal operations to reach g from s is FSM distance This distance is admissible and consistent Build control flow graph (CFG) with just PC values
Willem Visser and Alex Groce Specific only to Java
Verification and Validation, CS Dept, BYU 5
01mainmain:01: ldx #102: call foo03: add x,104: call foo05: check for errorfoo:06: pshx07: pulx08: rts
Underestimation Example
Verification and Validation, CS Dept, BYU 6
01
02
main
Underestimation Example
main:01: ldx #102: call foo03: add x,104: call foo05: check for errorfoo:06: pshx07: pulx08: rts
Verification and Validation, CS Dept, BYU 7
01
02
main
foo
Underestimation Example
06
main:01: ldx #102: call foo03: add x,104: call foo05: check for errorfoo:06: pshx07: pulx08: rts
Verification and Validation, CS Dept, BYU 8
01
02
03
04
05
main
foo
error
Underestimation Example
06
07
08
main:01: ldx #102: call foo03: add x,104: call foo05: check for errorfoo:06: pshx07: pulx08: rts
Verification and Validation, CS Dept, BYU 9
error
Underestimation Example
01
02
03
04
05
main
foo
error
06
07
08
main:01: ldx #102: call foo03: add x,104: call foo05: check for errorfoo:06: pshx07: pulx08: rts
Edelkamp’s FSM Heuristic:Shortest Distance from current state to checking for error in the CFG
Verification and Validation, CS Dept, BYU 10
error
Underestimation Example
01
02
03
04
05
main
foo
error
06
07
08
main:01: ldx #102: call foo03: add x,104: call foo05: check for errorfoo:06: pshx07: pulx08: rts
Edelkamp’s FSM Heuristic:Shortest Distance from current state to checking for error in the CFG
Verification and Validation, CS Dept, BYU 11
error
Underestimation Example
01
02
03
04
05
main
foo
error
06
07
08
main:01: ldx #102: call foo03: add x,104: call foo05: check for errorfoo:06: pshx07: pulx08: rts
Edelkamp’s FSM Heuristic:Shortest Distance from current state to checking for error in the CFG
Verification and Validation, CS Dept, BYU 12
Underestimation Example
main:01: ldx #102: call foo03: add x,104: call foo05: check for errorfoo:06: pshx07: pulx08: rts
Edelkamp’s FSM Heuristic:Shortest Distance from current state to checking for error in the CFG
error
01
02
03
04
05
main
foo
error
06
07
08
Verification and Validation, CS Dept, BYU 13
Underestimation Example
foo
3 steps
main:01: ldx #102: call foo03: add x,104: call foo05: check for errorfoo:06: pshx07: pulx08: rts
Edelkamp’s FSM Heuristic:Shortest Distance from current state to checking for error in the CFG
error
01
02
03
04
05
main
foo
error
06
07
08
Verification and Validation, CS Dept, BYU 14
True Distance should be ….
error
01
02
03
04
05
main
foo
error
06
07
08
main:01: ldx #102: call foo03: add x,104: call foo05: check for errorfoo:06: pshx07: pulx08: rts
Verification and Validation, CS Dept, BYU 15
errorerror
01
02
03
04
05
main
foo
error
06
07
08
main:01: ldx #102: call foo03: add x,104: call foo05: check for errorfoo:06: pshx07: pulx08: rts
True Distance should be ….
Verification and Validation, CS Dept, BYU 16
errorerror
01
02
03
04
05
main
foo
error
06
07
08
main:01: ldx #102: call foo03: add x,104: call foo05: check for errorfoo:06: pshx07: pulx08: rts
True Distance should be ….
Verification and Validation, CS Dept, BYU 17
errorerror
01
02
03
04
05
main
foo
error
06
07
08
main:01: ldx #102: call foo03: add x,104: call foo05: check for errorfoo:06: pshx07: pulx08: rts
True Distance should be ….
Verification and Validation, CS Dept, BYU 18
Underestimation Example
main:01: ldx #102: call foo03: add x,104: call foo05: check for errorfoo:06: pshx07: pulx08: rts
errorerror
01
02
03
04
05
main
foo
error
06
07
08
Verification and Validation, CS Dept, BYU 19
main:01: ldx #102: call foo03: add x,104: call foo05: check for errorfoo:06: pshx07: pulx08: rts
True Distance should be ….
errorerror
01
02
03
04
05
main
foo
error
06
07
08
Verification and Validation, CS Dept, BYU 20
main:01: ldx #102: call foo03: add x,104: call foo05: check for errorfoo:06: pshx07: pulx08: rts
True Distance should be ….
errorerror
01
02
03
04
05
main
foo
error
06
07
08
Verification and Validation, CS Dept, BYU 21
8 steps
main:01: ldx #102: call foo03: add x,104: call foo05: check for errorfoo:06: pshx07: pulx08: rts
True Distance should be ….
errorerror
01
02
03
04
05
main
foo
error
06
07
08
Verification and Validation, CS Dept, BYU 22
Solution: Interprocedural CFG
All the nodes in the ICFG that are part of a subroutine will be indexed on two things PC Value Return address to where the subroutine will return
when it encounters a return statement
Verification and Validation, CS Dept, BYU 23
01(init)
main:01: ldx #102: call foo03: add x,104: call foo05: check for errorfoo:06: pshx07: pulx08: rts
Verification and Validation, CS Dept, BYU 24
01(init)
02(init)
main:01: ldx #102: call foo03: add x,104: call foo05: check for errorfoo:06: pshx07: pulx08: rts
Verification and Validation, CS Dept, BYU 25
01(init)
02(init)
06(03)
main:01: ldx #102: call foo03: add x,104: call foo05: check for errorfoo:06: pshx07: pulx08: rts
Verification and Validation, CS Dept, BYU 26
01(init)
02(init)
06(03)
main:01: ldx #102: call foo03: add x,104: call foo05: check for errorfoo:06: pshx07: pulx08: rts
Verification and Validation, CS Dept, BYU 27
01(init)
02(init)
03(init)
06(03)
07(03)
08(03)
main:01: ldx #102: call foo03: add x,104: call foo05: check for errorfoo:06: pshx07: pulx08: rts
Verification and Validation, CS Dept, BYU 28
01(init)
02(init)
03(init)
06(03)
07(03)
08(03)
06(05)
07(05)
08(05)
04(init)
05(init)
main:01: ldx #102: call foo03: add x,104: call foo05: check for errorfoo:06: pshx07: pulx08: rts
Verification and Validation, CS Dept, BYU 29
01(init)
02(init)
03(init)
06(03)
07(03)
08(03)
06(05)
07(05)
08(05)
04(init)
05(init)
main:01: ldx #102: call foo03: add x,104: call foo05: check for errorfoo:06: pshx07: pulx08: rts
Verification and Validation, CS Dept, BYU 30
01(init)
02(init)
03(init)
06(03)
07(03)
08(03)
06(05)
07(05)
08(05)
04(init)
05(init)
main:01: ldx #102: call foo03: add x,104: call foo05: check for errorfoo:06: pshx07: pulx08: rts
8 steps
Verification and Validation, CS Dept, BYU 31
Nested Function Calls x → f → g y → f → g Same problem as before
main:1 call x2 call yerror
f:7 call g8 rts
g:9 xyza rts
x:3 call f4 rts
y:5 call f6 rts
f gx
1:call x(init)
3:call f(2)
7:call g(4)
9(8)
2:call y(init)
5:call g(error)
7:call g(6)
error
mainx f g
y f
a:rts(8)
8:rts(4)
4:rts(2)
8:rts(6)
6:rts(error)
Verification and Validation, CS Dept, BYU 32
Improved ICFG Algorithm
ICFG_Algorithm(state S)sa0 = icfgState(S)<sa1,sa2,….,san> // are abstracted states in the call stack of Sd = 0for i = 0 to n do srtn = rtn(sai) if FSM(sai, error) < FSM(sai,srtn) then d = d + FSM(sai,error) return d d = d + FSM(sai, srtn) + 1return d
Verification and Validation, CS Dept, BYU 33
Improved ICFG Algorithm
ICFG_Algorithm(state S)sa0 = icfgState(S)<sa1,sa2,….,san> // are abstracted states in the call stack of Sd = 0for i = 0 to n do srtn = rtn(sai) if FSM(sai, error) < FSM(sai,srtn) then d = d + FSM(sai,error) return d d = d + FSM(sai, srtn) + 1return d
Verification and Validation, CS Dept, BYU 34
Abstract states from the stack
08
04
02
PC: 09 0a(08)
08(04)
04(02)
02(init)
abstract statesgenerated fromthe stack
sa0
sa1
sa2
sa3
Verification and Validation, CS Dept, BYU 35
Improved ICFG Algorithm
ICFG_Algorithm(state S)sa0 = icfgState(S)<sa1,sa2,….,san> // are abstracted states in the call stack of Sd = 0for i = 0 to n do srtn = rtn(sai) if FSM(sai, error) < FSM(sai,srtn) then d = d + FSM(sai,error) return d d = d + FSM(sai, srtn) + 1return d
Verification and Validation, CS Dept, BYU 36
Improved ICFG Algorithm
ICFG_Algorithm(state S)sa0 = icfgState(S)<sa1,sa2,….,san> // are abstracted states in the call stack of Sd = 0for i = 0 to n do srtn = rtn(sai) if FSM(sai, error) < FSM(sai,srtn) then d = d + FSM(sai,error) return d d = d + FSM(sai, srtn) + 1return d
Verification and Validation, CS Dept, BYU 37
Improved ICFG Algorithm
ICFG_Algorithm(state S)sa0 = icfgState(S)<sa1,sa2,….,san> // are abstracted states in the call stack of Sd = 0for i = 0 to n do srtn = rtn(sai) if FSM(sai, error) < FSM(sai,srtn) then d = d + FSM(sai,error) return d d = d + FSM(sai, srtn) + 1return d
Verification and Validation, CS Dept, BYU 38
Marking returns statically
foo prologue
beq
epilogue return
Verification and Validation, CS Dept, BYU 39
Improved ICFG Algorithm
ICFG_Algorithm(state S)sa0 = icfgState(S)<sa1,sa2,….,san> // are abstracted states in the call stack of Sd = 0for i = 0 to n do srtn = rtn(sai) if FSM(sai, error) < FSM(sai,srtn) then d = d + FSM(sai,error) return d d = d + FSM(sai, srtn) + 1return d
Verification and Validation, CS Dept, BYU 40
Improved ICFG Algorithm
Calculating the Heuristic:D = 0FSM ((a,8),error) = 4
f gx
1:call x(init)
3:call f(2)
7:call g(4)
9(8)
2:call y(init)
5:call g(error)
7:call g(6)
error
mainx f g
y f
a:rts(8)
8:rts(4)
4:rts(2)
8:rts(6)
6:rts(error)
08
04
02
PC: 09
Verification and Validation, CS Dept, BYU 41
Improved ICFG Algorithm
08
04
02
PC: 09
Calculating the Heuristic:D = 0FSM ((a,8),error) = 4FSM ((a,8), (rts,8) = 11 < 4D += 1
x f g
1:call x(init)
3:call f(2)
7:call g(4)
9(8)
2:call y(init)
5:call g(error)
7:call g(6)
error
main
y f
a:rts(8)
8:rts(4)
4:rts(2)
8:rts(6)
6:rts(error)
Verification and Validation, CS Dept, BYU 42
Improved ICFG Algorithm
ICFG_Algorithm(state S)sa0 = icfgState(S)<sa1,sa2,….,san> // are abstracted states in the call stack of Sd = 0for i = 0 to n do srtn = rtn(sai) if FSM(sai, error) < FSM(sai,srtn) then d = d + FSM(sai,error) return d d = d + FSM(sai, srtn) + 1return d
Verification and Validation, CS Dept, BYU 43
Improved ICFG AlgorithmPC: 0a
D = 11
1:call x(init)
3:call f(2)
7:call g(4)
9(8)
2:call y(init)
5:call g(error)
7:call g(6)
error
main
y f
a:rts(8)
8:rts(4)
4:rts(2)
8:rts(6)
6:rts(error)
x f g
Verification and Validation, CS Dept, BYU 44
Results: Number of states generated
BFS DFS FSM Improved ICFG
Hyman’s mutex 4528 7006 3648 1560Naïve dining phil (threads) 47,246 8062 152,196 14,140Moody dining phil (threads) 225,269 44,238 555,609 28,565Lazy dining phil (threads) 317,131 56,685 >2.86 mil 50,984Bulls and cows 27,613 28,014 28,014 28,007
Verification and Validation, CS Dept, BYU 45
Conclusions Small overhead allowed use of more static information The Dynamic call stack with static analysis gave a better
estimate Testing shows an significant improvement in FSM distance The Improved ICFG algorithm can be used on any graph The algorithm is admissible and consistent
Verification and Validation, CS Dept, BYU 46
QUESTIONS