View
76
Download
0
Category
Preview:
DESCRIPTION
Automated Whitebox Fuzz Testing. Patrice Godefroid (Microsoft Research) Michael Y. Levin (Microsoft Center for Software Excellence) David Molnar (UC-Berkeley, visiting MSR). Presented by Yuyan Bao. Acknowledgement. This presentation is extended and modified from - PowerPoint PPT Presentation
Citation preview
Automated Whitebox Fuzz Testing
Patrice Godefroid (Microsoft Research)Michael Y. Levin (Microsoft Center for
Software Excellence)David Molnar (UC-Berkeley, visiting MSR)
Presented by Yuyan Bao
Acknowledgement
• This presentation is extended and modified from• The presentation by David Molnar
Automated Whitebox Fuzz Testing• The presentation by Corina Păsăreanu
(Kestrel) Symbolic Execution for Model Checking and
Testing
Agenda• Fuzz Testing• Symbolic Execution• Dynamic Testing Generation• Generational Search• SAGE Architecture• Conclusion• Contribution• Weakness• Improvement
Fuzz Testing• An effective technique for finding security vulnerabilities in software• Apply invalid, unexpected, or random data to the inputs of a program.• If the program fails(crashing, access violation
exception), the defects can be noted.
Whitebox Fuzzing• This paper present an alternative whitebox fuzz testing
approach inspired by recent advances in symbolic execution and dynamic test generation– Run the code with some initial well-formed input– Collect constraints on input with symbolic
execution– Generate new constraints (by negating constraints
one by one)– Solve constraints with constraint solver– Synthesize new inputs
Symbolic Execution• A method for deriving test cases which satisfy a given path• Performed as a simulation of the computation on the path• Initial path function = Identity function, Initial path condition = true• Each vertex on the path induce a symbolic composition on the path
function and a logical constraint on the path condition:If an assignment was made:
If a conditional decision was made:path condition path condition branch condition
Output: path function and path condition for the given path
White Box Testing and Symbolic Execution 6
f g f
( )X g X
[ ( )]X f X
Slide from “White Box Testing and Symbolic Execution” by Michael Beder
Symbolic Execution
7
Code:
(PC=“path condition”)
- “Simulate” the code using symbolic values instead of program numeric data
Symbolic execution tree: x:X,y:YPC: true
x:X,y:YPC: X>Y
x:X,y:YPC: X<=Y
true false
x:X+Y,y:YPC: X>Y
x:X+Y,y:XPC: X>Y
x:Y,y:XPC: X>Y
x:Y,y:XPC: X>Y Y-X>0
x:Y,y:XPC:X>Y Y-X<=0
true false
FALSE!
Not reachable
int x, y; if (x > y) { x = x + y; y = x - y; x = x - y; if (x – y > 0) assert (false); }
Dynamic Test Generation
• Executing the program starting with some initial inputs
• Performing a dynamic symbolic execution to collect constraints on inputs
• Use a constraint solver to infer variants of the previous inputs in order to steer the next executions
Dynamic Test Generationvoid top(char input[4])
{
int cnt = 0;
if (input[0] == ‘b’) cnt++;
if (input[1] == ‘a’) cnt++;
if (input[2] == ‘d’) cnt++;
if (input[3] == ‘!’) cnt++;
if (cnt >= 3) crash();
}
input = “good”
• Running the program with random values for the 4 input bytes is unlikely to discovery the error: a probability of about
• Traditional fuzz testing is difficult to generate input values that will drive program through all its possible execution paths.
301/ 2
Depth-First Search
void top(char input[4])
{
int cnt = 0;
if (input[0] == ‘b’) cnt++;
if (input[1] == ‘a’) cnt++;
if (input[2] == ‘d’) cnt++;
if (input[3] == ‘!’) cnt++;
if (cnt >= 3) crash();
}
I0 != ‘b’I1 != ‘a’I2 != ‘d’I3 != ‘!’good
I0 != ‘b’
I1 != ‘a’
I2 != ‘d’
I3 != ‘!’
Depth-First Search
goo!good
void top(char input[4])
{
int cnt = 0;
if (input[0] == ‘b’) cnt++;
if (input[1] == ‘a’) cnt++;
if (input[2] == ‘d’) cnt++;
if (input[3] == ‘!’) cnt++;
if (cnt >= 3) crash();
}
I0 != ‘b’I1 != ‘a’I2 != ‘d’I3 == ‘!’
Practical limitation
godd
void top(char input[4])
{
int cnt = 0;
if (input[0] == ‘b’) cnt++;
if (input[1] == ‘a’) cnt++;
if (input[2] == ‘d’) cnt++;
if (input[3] == ‘!’) cnt++;
if (cnt >= 3) crash();
}
I0 != ‘b’I1 != ‘a’I2 == ‘d’I3 != ‘!’good
Every time only one constraint is expanded, low efficiency!
Generational search
• BFS with a heuristic to maximize block coverage• Score returns the number of new blocks covered
14Slide from “Symbolic Execution” by Kevin Wallace, CSE504 2010-04-28
Generational SearchKey Idea: One Trace, Many Tests
goo!
godd
gaod
bood
“Generation 1” test cases
good
void top(char input[4])
{
int cnt = 0;
if (input[0] == ‘b’) cnt++;
if (input[1] == ‘a’) cnt++;
if (input[2] == ‘d’) cnt++;
if (input[3] == ‘!’) cnt++;
if (cnt >= 3) crash();
}
I0 == ‘b’I1 == ‘a’I2 == ‘d’I3 == ‘!’
void top(char input[4])
{
int cnt = 0;
if (input[0] == ‘b’) cnt++;
if (input[1] == ‘a’) cnt++;
if (input[2] == ‘d’) cnt++;
if (input[3] == ‘!’) cnt++;
if (cnt >= 3) crash();
}
0good
1goo!
1godd
2god!
1gaod
2gao!
2gadd
gad!
1bood
2boo!
2bodd
2baod
bod! bad!
badd baod
The Generational Search Space
SAGE Architecture(Scalable Automated Guided Execution)
Task: TesterCheck forCrashes
(AppVerifier)
Task: TracerTrace
Program(Nirvana)
Task: CoverageCollector
Gather Constraints(Truscan)
Task: SymbolicExecutor
SolveConstraints(Disolver)
Input0 Trace File Constraints
Input1
Input2…
InputN
• Since 1st MS internal release in April’07: dozens of new security bugs found (most missed by blackbox fuzzers, static analysis)
• Apps: image processors, media players, file decoders,…
• Many bugs found rated as “security critical, severity 1, priority 1”
• Now used by several teams regularly as part of QA process
Initial Experiences with SAGE
Experiments ANI Parsing - MS07-017
RIFF...ACONLISTB...INFOINAM....3D Blue Alternate v1.1..IART....................1996..anih$...$...................................rate....................seq ....................LIST....framicon......... ..
RIFF...ACONBB...INFOINAM....3D Blue Alternate v1.1..IART....................1996..anih$...$...................................rate....................seq ....................anih....framicon......... ..
Only 1 in 232 chance at random!
Initial input Crashing test case
Initial input : 341 branch constraints, 1,279,939 total instructions. Crash test cases: 7706 test cases, 7hrs 36 mins
Different Crashes From Different Initial Input Files 100
zerobytes
1867196225 X X X X X
2031962117 X X X X X
612334691 X X
1061959981 X X
1212954973 X X
1011628381 X X X
842674295 X
1246509355 X X X
1527393075 X
1277839407 X
1951025690 X
Media 1: 60 machine-hours, 44,598 total tests, 357 crashes, 12 bugs
Bug ID seed1 seed2 seed3 seed4 seed5 seed6 seed7
Conclusion Blackbox vs. Whitebox Fuzzing
• Cost/precision tradeoffs
– Blackbox is lightweight, easy and fast, but poor coverage
– Whitebox is smarter, but complex and slower
– Recent “semi-whitebox” approaches• Less smart but more lightweight: Flayer (taint-flow analysis, may
generate false alarms), Bunny-the-fuzzer (taint-flow, source-based, heuristics to fuzz based on input usage), autodafe, etc.
• Which is more effective at finding bugs? It depends…
– Many apps are so buggy, any form of fuzzing finds bugs!
– Once low-hanging bugs are gone, fuzzing must become smarter: use whitebox and/or user-provided guidance (grammars, etc.)
• Bottom-line: in practice, use both!
Contribution
• A novel search algorithm with a coverage-maximizing heuristic
• It performs symbolic execution of program traces at the x86 binary level
• Tests larger applications than previously done in dynamic test generation
Weakness
• Gathering initial inputs• The class of inputs characterized by each symbolic
execution is determined by the dependence of the program’s control flow on its inputs
• Not a general solution• Only for x86 windows applications• Only focus on file-reading applications
• Nirvana simulates a given processor’s architecture
Improvement
• Portability• Translate collected traces into an
intermediate language• Reproduce bugs on different platforms
without re-simulating the program• Better way to find initial inputs efficiently
Thank you!Questions?
Recommended