Embed Size (px)
Citation preview
Yet Another Heapspray Detector
Danny KovachRaytheon SI
Introduction
• Our main purpose is to detect malware.
Introduction
• Currently we monitor an application in a VM for such behavior as:– Loading drivers– Creating executable files– Network activity
• Heap sprays are very hard to detect.
What is a heapspray?
• Technique used to put executable code onto the heap.
• Consists of– NOP sled– Shellcode
• Goal: direct execution flow to the NOP sled; shellcode.
How to detect a heapspray?
• Nozzle [1]
• BuBBle [3]
• Entropy
Idea!
• Treat byte values on the heap as a random variable and do math!
Assumptions
• Bytes on a normal heap should be randomly distributed (white noise)
• Fourier transform of white noise has constant magnitude.
Visualizing the Heap(normal program operation)
256 512 768
50 000
100 000
150 000
Visualizing the Heap(normal program operation)
256 512 768 1024
50 000
100 000
150 000
Visualizing the Heap(normal program operation)
256 512 768 1024 1280 1536 1792 2048
50 000
100 000
150 000
Fourier Transforms of the Heap(normal program operation)
200 400 600
500 000
1 .0 106
1 .5 106
2 .0 106
2 .5 106
3 .0 106
Fourier Transforms of the Heap(normal program operation)
200 400 600 800 1000
500 000
1 .0 106
1 .5 106
2 .0 106
2 .5 106
3 .0 106
3 .5 106
Fourier Transforms of the Heap(normal program operation)
500 1000 1500 2000
1 106
2 106
3 106
4 106
Visualizing the Heap(heap spray)
256 512 768
20 000
40 000
60 000
80 000
100 000
120 000
Visualizing the Heap(heap spray)
256 512 768 1024
20 000
40 000
60 000
80 000
100 000
120 000
140 000
Visualizing the Heap(heap spray)
256 512 768 1024 1280 1536 1792 2048
20 000
40 000
60 000
80 000
100 000
Fourier Transforms of the Heap(heap spray)
200 400 600
1 106
2 106
3 106
4 106
5 106
Fourier Transforms of the Heap(heap spray)
500 1000 1500 2000
1 106
2 106
3 106
4 106
Fourier Transforms of the Heap(heap spray)
200 400 600 800 1000
2 106
3 106
4 106
Problem:
Low hanging fruit?
More Analysis
• Used the open source tool RapidMiner
• Started by making a decision tree
Results
Results
• 100% accurate for all our test cases.
• Rushed into production (without further testing).
• FAIL!
Next attempt: Statistics
• Assume that distribution of bytes is Gaussian
Statistics for normal heap
• About 40 counts > 1 standard deviation
• Actual measurement: 20 – 30
Statistics for Heapspray
• NOP slide altered distribution
• Typically 2 – 8 > 1 standard deviation
Advantages of a Statistical Approach
• Easy to code
• Friendly to system resources
• More general than hard coded approach
• Theoretically sound
Results
• Out of over 500 files tested, we had 100% success.
• 0 false positives
• 0 false negatives
How to defeat
• Write shellcode so as to minimally alter normal distribution.
• Most likely will leave some signature.
• Invites cat and mouse game.
References
1. http://research.microsoft.com/en-us/projects/nozzle/
2. http://en.wikipedia.org/wiki/Heap_spraying
3. https://lirias.kuleuven.be/bitstream/123456789/265421/1/fulltext.pdf
4. http://www.mathnstuff.com/math/spoken/here/2class/90/normal.htm
