指導教授：王國禎博士　　學生：蔡薰儀國立交通大學網路工程研究所行動計算與寬頻網路實驗室

Copyright © 2012, [email protected]

Suspicious Behavior-based Malware DetectionUsing Artificial Neural Network

基於可疑行為及類神經網路之惡意軟體偵測機制指導教授：王國禎博士　　學生：蔡薰儀

國立交通大學網路工程研究所行動計算與寬頻網路實驗室


2

Outline

• Introduction• Related Work• Problem Statement• Background

– Sandboxes• Design Approach

– Suspicious Behaviors– Proposed ANN-MD System– Weight Adjusting– Malicious Degree

• Evaluation– Training Phase– Testing Phase

• Conclusion and Future Work• References


3

Introduction

• In recent years, malware has been severe threats to cyber security– Viruses, Worms, Trojan horses, Botnets, …

• Drawbacks of traditional signature-based malware detection algorithms [1] [2]– Need human and time to approve– Need to update the signatures of malware frequently– Easily bypassed by obfuscation methods– Can not detect zero day malware– Increase false negative rate


4

Introduction (Cont.)

• To conquer the shortcomings of signature-based malware detection algorithms, behavior-based malware detection algorithms were proposed

• Behavior-based malware detection algorithms [3] [4]– Detect unknown malware or variations of known malware– Decrease false negative rate (FNR)

• However, existing behavior-based malware detection algorithms may have a higher false positive rate (FPR)– Benign software may have some behaviors which are alike with

malware


5

Introduction (Cont.)

• We proposed a behavioral artificial neural network (ANN)-based malware detection (ANN-MD) algorithm– Detect unknown malware and variations of known malware– Decrease FNR and FPR


6

Related Work

• MBF [3]– File, process, network, and registry actions– Malicious Behavior Feature (MBF) MBF = <Feature_id, Mal_level, Bool_expression>– Three malicious level: high, warning, and low

• RADUX [4]– Reverse Analysis for Detecting Unsafe eXecution (RADUX)– API function call sequences, e.g. load register: RegOpenKey,

RegCreateKey, RegSetValue, RegCloseKey– Collected 9 common suspicious behaviors– Use Bayes’ theorem to compute suspicious degree (malicious

degree). Combine the appearance probability of each behavior all together not individually


7

Related Work (Cont.)

Approach ANN-MD (proposed) MBF [3] RADUX [4]

Main idea Analyze suspicious behaviors Analyze malicious behavior features (MBF) Analyze API call sequences

Number of suspicious behaviors

13(Only host behaviors)

16(Host behaviors + Network behaviors)

9(Only host behaviors)

Weights of suspicious behaviors

The weights are initialized by function initnw and adjust by

ANN

Malicious level: high, warning, and low

Appearance frequency of each suspicious behavior

Calculating of malicious degree ANN Make the summation of the

mal_level of each MBF Bayes’ theorem

Pros

• Set different weights for different behaviors

• Considered each suspicious behavior

individually

• Considered the malicious level of each malicious

behavior feature

• Set different weights for different behaviors according to their

appearance probabilities

Cons • Not considered network behaviors

• Only used high, warning, and low to represent the malicious level of each

MBF

• Combined the appearance probability of each behavior all together

not individually• Not considered network

behaviors


8

Problem Statement

• Given– Several sandboxes– i known malware M = {M1,M2, …, Mi} for training– j known malware N = {N1, N2, …, Nj} for testing– k benign software O = {O1, O2, …, Ok} for training– l benign software P = {P1, P2, …, Pl} for testing

• Objective– m behaviors B = {B1,B2, …, Bm} – m weights W = {ω1, ω2, …, ωm}– Malicious Degree (MD) expression


9

Problem Statement (Cont.)

• Try to find the optimal MD threshold to make FPR and FNR as small as possible.

Benign Malicious

MD Threshold

Ambiguous

Number of Samples


10

Background – Sandboxes

• A sandbox is a testing environment which can isolate unknown sample from make changes to the operating system

• It can interact with samples and record all the runtime behaviors of samples

• Web-based sandboxes– GFI Sandbox [5]– Norman Sandbox [6]– Anubis Sandbox [7]


11

Design Approach – Suspicious Behaviors

• Choose the behaviors in the intersection of the behaviors these sandboxes investigate

• Choose the behaviors which are not in the intersection but have high appearance frequency, i.e.– Creates Mutex– Creates Hidden File– Starts EXE in System– Checks for Debugger– Starts EXE in Documents– Windows/Run Registry Key Set– Hooks Keyboard– Modifies Files in System– Deletes Original Sample– More than 5 Processes– Opens Physical Memory– Deletes Files in System– Auto Start


12

Design Approach – Suspicious Behaviors (Cont.)

Ulrich Bayer et al. [8]The behaviors we chooseThe behaviors which may cause false positive rate

Observed Behavior Percentage of samplesInstallation of a Windows kernel driver 3.34%

Installation of a Windows service 12.12%Creating a file 70.78%Deleting a file 42.57%Modifying a file 79.87%Installation of an IE BHO 1.72%Installation of an IE Toolbar 0.07%Display a GUI window 33.26%Network Traffic 55.18%Writing to stderr 0.78%Writing to stdout 1.09%Modifying a registry value 74.59%Creating a registry key 64.74%Creating a process 52.19%


13

Design Approach – Proposed ANN-MD System


14

Design Approach – Weight Adjusting

• Using ANN to train weights


15

Design Approach – Weight Adjusting (Cont.)

• Neuron for ANN hidden layer: the first neuron

𝑓 ( 1) (𝑛1 )=𝑒𝑛1 −𝑒−𝑛 1

𝑒𝑛1+𝑒−𝑛1=𝑎1∑

𝑖=1

13

ω𝑖 ,1𝑥 𝑖+𝑏1=𝑛1


16


• Neuron for ANN output layer

∑𝑖=1

10

ω𝑖′ 𝑎𝑖+𝑏′=𝑛′ 𝑓 ( 2) (𝑛′ )=𝑒𝑛 ′ −𝑒−𝑛 ′

𝑒𝑛 ′+𝑒−𝑛 ′


17


• Delta learning process

2)(21 OdE

Expected target value:

}101|'{}101,131|{ , kjiW kji

Mean square error:

Weight set:

,W

oldnew

xE

, : learning factor; x: set of input values

𝑑={ 0 ,𝐵𝑒𝑛𝑖𝑔𝑛1,𝑀𝑎𝑙𝑖𝑐𝑖𝑜𝑢𝑠; output value: O (MD)


18

Design Approach – Malicious Degree

• Malicious Degree Expression– Suspicious behaviors: – Weights: – Bias: – Transfer function: (tangent-sigmoid function)


19

Evaluation (Cont.)

• Use matlab 7.11.0 to implement ANN in our system• Initial weights and bias: chosen by function initnw

to distribute the weight of each neuron in the layer evenly [9] (according to the Nguyen-Widrow initialization algorithm)

• Transfer function: tangent-sigmoid function• Learning factor η () : 0.5


20

Evaluation (Cont.)

• Architecture of ANN (from matlab) :


21

Evaluation (Cont.)

• Malicious sample sources: Blast’s Security [10] and VX Heaven [11] websites

• Benign sample sources: Portable execution files under windows XP SP2

• Training samples and testing samples Samples Malicious Benign TotalTraining 500 500 1000

Testing 500 500 1000


22

Evaluation – Training Phase

• Execution time: 3 seconds (training and testing phase)• MD threshold (according to training samples)

0 0.04 0.08 0.12 0.16 0.2 0.24 0.28 0.32 0.36 0.4 0.44 0.48 0.52 0.56 0.6 0.64 0.68 0.72 0.76 0.8 0.84 0.88 0.92 0.96 10

50

100

150

200

250

300

350

400

Benign Samples Malicious Samples

Malicious Degree

Num

ber o

f Sam

ples 0.19 0.87

Range of MD Threshold


23

Evaluation – Training Phase (Cont.)

• Choose MD threshold

0.19

0.22

0.25

0.28

0.31

0.34

0.37 0.4 0.4

30.4

60.4

90.5

20.5

50.5

80.6

10.6

40.6

7 0.7 0.73

0.760

0000

0000

0001

0.790

0000

0000

0001

0.820

0000

0000

0001

0.850

0000

0000

0001

0

1

2

3

4

5

6

7

8

95

95.5

96

96.5

97

97.5

98

98.5

99

FPR FNR Accuracy Rate

Malicious Degree

Fals

e R

ate

(%)

Acc

urac

y R

ate

(%)

0.5


24

Evaluation – Testing Phase

• Experiment resultsTP: True positive FN: False negativeFP: False positive TN: True negative

TP TN FP FN FPR FNR Accuracy485 496 4 15 0.8% 3% 98.1%


25

Evaluation – Testing Phase (Cont.)

• Distribution of testing samples

0 0.04 0.08 0.12 0.16 0.2 0.24 0.28 0.32 0.36 0.4 0.44 0.48 0.52 0.56 0.6 0.64 0.68 0.72 0.76 0.8 0.84 0.88 0.92 0.96 10

50

100

150

200

250

300

350

400

450

Benign Samples Malicious Samples

Malicious Degree

Num

ber o

f Sam

ples MD Threshold = 0.5


26

Evaluation (Cont.)

ApproachTP / (TP + FN) FN / (TP + FN)

Accuracy rateFP / (FP + TN) TN / (FP + TN)

ANN-MD(proposed)

97% 3%98.1%

0.8% 99.2%

MBF [14]83% 17%

88.7%5.6% 94.4%

RADUX [19]96.6% 3.4%

91.2%14.2% 85.8%


27

Evaluation (Cont.)

Weights

FPR FNR Accuracy rateAdjusting

of weightWeights in

hidden layerWeights in

output layer

With ANN

Chosen by initnw

Chosen by initnw 0.8% 3% 98.1%

Appearance frequency

Chosen by initnw 1.2% 2.8% 98%

Without ANN Appearance frequency 7.8% 4.8% 93.7%


28

Conclusion and Future Work

• Conclusion– Collect 13 common behaviors of malwares– Construct Malicious Degree (MD) expression– FPR and FNR are as small as possible– Has a better effect on detecting unknown malware compared to

the related work [14] [19]• Future work

– Automate the proposed ANN-MD system– Implement PC-based sandboxes– Add more suspicious network behaviors– Classify malwares according to their typical behaviors


29

References

[1] C. Mihai and J. Somesh, “Static analysis of executables to detect malicious patterns,” Proceedings of the 12th conference on USENIX Security Symposium, Vol. 12, pp. 169 - 186, Dec. 10-12, 2006.[2] J. Rabek, R. Khazan, S. Lewandowskia, and R. Cunningham, “Detection of injected, dynamically generated, and obfuscated malicious code,” Proceedings of the 2003 ACM workshop on Rapid malcode, pp. 76 - 82, Oct. 27-30, 2003.[3] W. Liu, P. Ren, K. Liu, and H. X. Duan, “Behavior-based malware analysis and detection,” Proceedings of Complexity and Data Mining (IWCDM), pp. 39 - 42, Sep. 24-28, 2011. [4] C. Wang, J. Pang, R. Zhao, W. Fu, and X. Liu, “Malware detection based on suspicious behavior identification,” Proceedings of Education Technology and Computer Science, Vol. 2, pp. 198 - 202, Mar. 7-8, 2009.[5] GFI Sandbox. http://www.gfi.com/malware-analysis-tool[6] Norman Sandbox. http://www.norman.com/security_center/security_tools[7] Anubis Sandbox. http://anubis.iseclab.org/


30

References (Cont.)

[8] U. Bayer, I. Habibi, D. Balzarotti, E. Krida, and C. Kruege, “A view on current malware behaviors,” Proceedings of the 2nd USENIX Workshop on Large-Scale Exploits and Emergent Threats : botnets, spyware, worms, and more, pp. 1 - 11, Apr. 22-24, 2009.[9] Neural Network Toolbox. http://dali.feld.cvut.cz/ucebna/matlab/toolbox/nnet/initnw.html[10] Blast's Security. http://www.sacour.cn[11] VX heaven. http://vx.netlux.org/vl.php

Documents

指導教授：王國禎 博士 學生： 蔡薰儀 國立交通大學網路工程研究所 行動計算與寬頻網路實驗室

指導教授：王國禎博士　　學生：蔡薰儀國立交通大學網路工程研究所行動計算與寬頻網路實驗室