1

Effective and Efficient Malware Detection at the End Host

Clemens Kolbitsch, Paolo Milani Comparetti @ TU ViennaChristopher Kruegel @ UCSB

Engin Kirda @ Institute EurecomXiaoyong Zhou, XiaoFeng Wang @ Indiana Univ. at Bloominton

USENIX Security Symposium ‘09

2

Outline

• Motivation• System Overview• System Details• Evaluation• Limitation• Conclution

3

MOTIVATIONEffectiveness & Efficiency

4

Motivation

• Efficiency– Binary signature based detection– Network-based detection

• Effectiveness– Behavior-based detection• Detection based on malware's behavior• Behavior is hard to obfuscate• Behavior is hard to randomize• Behavior is often stable across various malware version

5

Motivation

• This Paper proposes…– A behavior-based solution with Efficiency– For end hosts

6

SYSTEM OVERVIEWModeling Behaviors and Making detection efficient

7

System Overview

• Malware behaviors– Manifest on system (i.e., survive reboot)

• (Over-) write system executables, dlls, files• Create registry entries• Register as Windows (startup) service

– Conceal from being detected• Restart under some stealthy name (e.g., svchost.exe)• Inject into legitimate processes

– Replicate• Send emails• Copy to Samba shares, USB drives, etc.• Scan and exploit services on LAN or WAN

8

System Overview

• Detection based on execution characteristics– Execute malware in full system emulator (Anubis)– Monitor interaction with the operating system– Perform detailed taint analysis– Generate detection graphs

• Describe sequence of required system calls leading to security relevant system activity

• Include dependencies to related, previous calls (using taint dependencies)

• Detect described behavior on end host– Log system call activity of unknown executable– Match against behavior graph

9

System Overview

• Example: Agent (trojan)• As part of its system manifestation, it– Reads content from binary image– Decrypts binary content

• Proprietary decryption routine• Simple, XOR based algorithm

– Stores binary in system file (C:\Windows\system32\drivers\ip6fw.sys)

– Later, restarts IPv6 firewall• Turns itself into a system service

10

System Overview

11

SYSTEM DETAILS

Generate Behavior Graphs,Match Behavior Graphs

12

System Details

• Behavior graphs– Directed acyclic graph– Node: system calls– Edges: dependencies

• Dependencies– Handle dependencies• Direct value propagation• System provided identifiers• Must be constant

13

System Details

• Data dependencies– Arbitrary data (& control) dependency between

system calls– Might modify values between system calls

14

System Details

• Generate behavior graphs– Analyze executable in Anubis sandbox• Obtain instruction level log• Obtain program flow log• Obtain memory access log• Generate precise taint propagation trees

– Data/control dependencies– Instructions that access/generate tainted data– Link system calls consuming data with all taint generating calls

(sources)

15

System Details

• Generate behavior graphs (cont.)– Scan logs for security relevant behavior• Provided with a list of interesting system calls

• Extract propagation formulas

16

System Details

• Match behavior graphs– Active(inactive) node– Simple(complex) function– Security-relevant system calls or the Buttom– Confirmed(deactivate all)

17

System Details

18

System Details

19

EVALUATION

Effectiveness,Efficiency

20

Evaluation

• Effectiveness

21

Evaluation

22

Evaluation

• False Positive– IE, Firefox, Thunderbird, putty, notepad– 0

23

Evalution

• Efficiency

24

LIMITATION & CONCLUSION

25

Limitation

• Evading signature generation– Detect the virtual environment– Delays, time-triggered behavior

• Modifying the algorithm behavior

26

Conclusion

• Behavior can be detected• Behavior detection is fast enough for end

hosts– Approach intrinsically robust against

polymorphism and metamorphism– To some extent, behavior graphs are usable across

malware variants