36
EFFECTIVE FLOWGRAPH-BASED MALWARE VARIANT DETECTION Silvio Cesare, Ph.D. Candidate, Deakin University http://www.foocodechu.com [email protected]

Effective flowgraph-based malware variant detection

Embed Size (px)

DESCRIPTION

AusCERT 2012

Citation preview

Page 1: Effective flowgraph-based malware variant detection

EFFECTIVE FLOWGRAPH-BASED MALWARE VARIANT DETECTIONSilvio Cesare, Ph.D. Candidate, Deakin University

http://www.foocodechu.com

[email protected]

Page 2: Effective flowgraph-based malware variant detection

WHO AM I AND WHERE DID THIS TALK COME FROM?

Ph.D. Student at Deakin University.

Research interests include: Automated vulnerability discovery. Software similarity and classification. Malware detection.

This presentation is based on my malware research.

Page 3: Effective flowgraph-based malware variant detection

OUTLINE

1. Introduction (you might already know this)

2. New approaches to flowgraph-based classification

3. Evaluation

4. Other things we use our system on.

5. Conclusion

Page 4: Effective flowgraph-based malware variant detection

INTRODUCTIONThis is to make sure everyone is up to speed.

If you’ve been to my presentations before, you might have already seen it.

Page 5: Effective flowgraph-based malware variant detection

INTRODUCTION Malware a significant problem.

Static detection of malware a dominant real-time technique.

Detecting unknown variants from known samples very useful.

Klez.aKlez.bKlez.cKlez.d...

Roron.aoRoron.bRoron.dRoron.eRoron.f...

Page 6: Effective flowgraph-based malware variant detection

SIGNATURES AND BIRTHMARKS A birthmark is an invariant property in

related samples.

Birthmark comparison should allow inexact matching.

Page 7: Effective flowgraph-based malware variant detection

LIMITATIONS OF EXISTING BIRTHMARKS Byte-level content can change in every variant.

Comparing birthmarks often exact matching only.

Inefficient for inexact database searching.

Unable to detect unknown variants of known samples.

Program structure a better birthmark.

Page 8: Effective flowgraph-based malware variant detection

THE SOFTWARE SIMILARITY PROBLEM

Page 9: Effective flowgraph-based malware variant detection

THE SOFTWARE SIMILARITY SEARCH Need a dissimilarity or distance metric. “Metric” property allows efficient database

search.

q

Query Malicious

Query Benign

d(p,q)

p

r

Malware

Query

Page 10: Effective flowgraph-based malware variant detection

EXISTING APPROACHES: A CALL GRAPH BIRTHMARK

Inter-procedureal control flow.

Page 11: Effective flowgraph-based malware variant detection

AN OPTIMAL DISSIMILARITY METRIC FOR GRAPHS

Graph edit distance.

Number of operations to transform one graph to another.

Complexity in NP.

Non optimal solutions possible in cubic time.

Page 12: Effective flowgraph-based malware variant detection

OUR APPROACH: A SET OF CONTROL FLOW GRAPHS BIRTHMARK

Intra-procedural control flow. Many procedures.

Page 13: Effective flowgraph-based malware variant detection

TRANSFORMING GRAPH DISSIMILARITY TO A STRING DISSIMILARITY PROBLEM

Decompile control flow graphs to strings. Compare strings using ‘string metrics’.

L_0

L_3

L_6

L_7L_1

L_2 L_4

L_5

true

true

true

true

true

W|IEH}Rproc(){L_0: while (v1 || v2) {L_1: if (v3) {L_2: } else {L_4: }L_5: }L_7: return;}

Page 14: Effective flowgraph-based malware variant detection

NEW APPROACHES TO FLOWGRAPH-BASED CLASSIFICATION

Page 15: Effective flowgraph-based malware variant detection

TRANSFORMING A SET OF STRINGS PROBLEM INTO A STRING PROBLEM

Decompiled CFGs give us a set of strings.

Order and concatenate strings.

Deliminate substrings with ‘Z’.

Order based on metrics. Number of instructions in procedure. Number of basic blocks. etc

R

IEHR

W|}R

SR

W|IEH}R

W|IEH}RZRZW|}RZIEHRZSRZ

R

IEHR

W|}R

SR

W|IEH}R

Page 16: Effective flowgraph-based malware variant detection

WHAT WE TRIED (AND ENDED UP NOT USING)

String metrics:

Edit distance

Normalized Compression Distance

Sequence alignment

All databases indexed using metric trees.

)}(),(max{

)},(),(min{)(),(

yCxC

yCxCxyCyxNCD

A K TKT K | | | | |ATKTT T K

ed(“hello”, “ggello”) = 2

Page 17: Effective flowgraph-based malware variant detection

SEQUENCE ALIGNMENT WITH BLAST A heuristic genome sequence search tool.

Local sequence alignment.

Hugely popular in bioinformatics.

So.. transform our strings into genome sequences.

Then, do a genome search.

Page 18: Effective flowgraph-based malware variant detection

GENOME SEQUENCE EXTRACTION

L_0

L_3

L_6

L_7L_1

L_2 L_4

L_5

true

true

true

true

true

W|IEH}Rproc(){L_0: while (v1 || v2) {L_1: if (v3) {L_2: } else {L_4: }L_5: }L_7: return;}

A = AdelineC = CytosineG = GuanineT = Thyamine...

ACGTRYKMACGTRYKM

L_0

L_3

L_6

L_7L_1

L_2 L_4

L_5

true

true

true

true

true

W|IEH}Rproc(){L_0: while (v1 || v2) {L_1: if (v3) {L_2: } else {L_4: }L_5: }L_7: return;}

Page 19: Effective flowgraph-based malware variant detection

WHY DIDN’T WE USE THOSE APPROACHES? Not optimally effective.

Too slow.

Best speed was using NCD.

Page 20: Effective flowgraph-based malware variant detection

A DISSIMILARITY METRIC FOR SETS OF STRINGS (WHAT WE ENDED UP USING)

Find a mapping between strings to minimize the sum of distances.

qd=ed(p,q)

p

BW|{B}BR

BI{B}BR

BSSR

BSR

BSSSR

BR

BW|{B}BR

BSSR

BR

Page 21: Effective flowgraph-based malware variant detection

COMBINATORIAL OPTIMISATION: THE ASSIGNMENT PROBLEM

Finding a minimum cost mapping is known as the “assignment problem”

Optimal solutions exist in cubic time.

“Greedy” heuristic solutions faster.

Has the properties of a metric.

Page 22: Effective flowgraph-based malware variant detection

EVALUATION

Page 23: Effective flowgraph-based malware variant detection

IMPLEMENTATION Malwise system is 100,000 lines of code of C++.

The modules for this work < 3000 lines of code.

Unpacks malware using an application level emulator (Ruxcon 2010)

Pre-filtering stage to quickly cull non matching variants (Ruxcon 2011)

Page 24: Effective flowgraph-based malware variant detection

EVALUATION - EFFECTIVENESS Calculated similarity between Roron malware

variants.

Compared results to Ruxcon 2010 work.

In tables, highlighted cells indicates a positive match.

The more matches the more effective it is.

Page 25: Effective flowgraph-based malware variant detection

EVALUATION - EFFECTIVENESS

ao b d e g k m q aao 0.86 0.49 0.54 0.50 0.87 0.86 0.86 0.86b 0.87 0.57 0.63 0.62 0.96 1.00 1.00 0.96d 0.61 0.64 0.85 0.91 0.64 0.64 0.64 0.64e 0.64 0.69 0.85 0.90 0.68 0.69 0.69 0.68g 0.62 0.68 0.91 0.91 0.68 0.68 0.68 0.68k 0.88 0.96 0.58 0.62 0.61 0.96 0.96 0.99m 0.87 1.00 0.57 0.63 0.62 0.96 1.00 0.96q 0.87 1.00 0.57 0.63 0.62 0.96 1.00 0.96a 0.87 0.96 0.58 0.62 0.61 0.99 0.96 0.96

ao b d e g k m q aao   0.70 0.28 0.28 0.27 0.75 0.70 0.70 0.75b 0.74 0.31 0.34 0.33 0.82 1.00 1.00 0.87d 0.28 0.29 0.50 0.74 0.29 0.29 0.29 0.29e 0.31 0.34 0.50 0.64 0.32 0.34 0.34 0.33g 0.27 0.33 0.74 0.64 0.29 0.33 0.33 0.30k 0.75 0.82 0.29 0.30 0.29 0.82 0.82 0.96m 0.74 1.00 0.31 0.34 0.33 0.82 1.00 0.87q 0.74 1.00 0.31 0.34 0.33 0.82 1.00 0.87a 0.75 0.87 0.30 0.31 0.30 0.96 0.87 0.87  

Exact Matching (Ruxcon 2010)

Heuristic Approximate Matching (Ruxcon 2010)

ao b d e g k m q aao 0.44 0.28 0.27 0.28 0.55 0.44 0.44 0.47b 0.44 0.27 0.27 0.27 0.51 1.00 1.00 0.58d 0.28 0.27 0.48 0.56 0.27 0.27 0.27 0.27e 0.27 0.27 0.48 0.59 0.27 0.27 0.27 0.27g 0.28 0.27 0.56 0.59 0.27 0.27 0.27 0.27k 0.55 0.51 0.27 0.27 0.27 0.51 0.51 0.75m 0.44 1.00 0.27 0.27 0.27 0.51 1.00 0.58q 0.44 1.00 0.27 0.27 0.27 0.51 1.00 0.58a 0.47 0.58 0.27 0.27 0.27 0.75 0.58 0.58

Assignment problem

Page 26: Effective flowgraph-based malware variant detection

EVALUATION – FALSE POSITIVES Database of 10,000 malware.

Scanned 1,601 benign binaries.

7 false positives. Less than 1%.

Very small binaries have small signatures and cause weak matching.

Page 27: Effective flowgraph-based malware variant detection

EVALUATION - EFFICIENCY Median benign and malware processing time

is 0.06s and 0.84s.% Samples Benign Time(s)

Malware Time(s)

10 0.02 0.16

20 0.02 0.28

30 0.03 0.30

40 0.03 0.36

50 0.06 0.84

60 0.09 0.94

70 0.13 0.97

80 0.25 1.03

90 0.56 1.31

100 8.06 585.16

Page 28: Effective flowgraph-based malware variant detection

BUT THAT’S NOT ALL WE USE THE MALWISE ENGINE FOR..

Page 29: Effective flowgraph-based malware variant detection

SIMSEER – A SOFTWARE SIMILARITY WEB SERVICE

An online service to identify similarity between programs

Based on Malwise.

Renders an evolutionary tree to show program relationships.

Free to use!

http://www.foocodechu.com/?q=simseer-a-software-similarity-web-service

Page 30: Effective flowgraph-based malware variant detection
Page 31: Effective flowgraph-based malware variant detection
Page 32: Effective flowgraph-based malware variant detection

SIMSEER - DEMO http://www.youtube.com/watch?

v=ymo7DKlKCH4

Page 33: Effective flowgraph-based malware variant detection

BUGWISE Automatically detect bugs and vulnerabilities in

Linux executable binaries.

Uses static program analysis from Malwise. Decompilation Data Flow Analysis

Free to use!

http://www.foocodechu.com/?q=bugwise-a-bug-detection-web-service-for-binary-executables

Page 34: Effective flowgraph-based malware variant detection

BUGWISE – SGID GAMES XONIX BUG IN DEBIAN LINUX

memset(score_rec[i].login, 0, 11);

strncpy(score_rec[i].login, pw->pw_name, 10);

memset(score_rec[i].full, 0, 65);

strncpy(score_rec[i].full, fullname, 64);

score_rec[i].tstamp = time(NULL);

free(fullname);

if((high = freopen(PATH_HIGHSCORE, "w",high)) == NULL) {

fprintf(stderr, "xonix: cannot reopen high score file\n");

free(fullname);

gameover_pending = 0;

return;

}

Page 35: Effective flowgraph-based malware variant detection

PUBLICATIONS Book published by Springer. http://www.springer.com/computer/

security+and+cryptology/book/978-1-4471-2908-0

Page 36: Effective flowgraph-based malware variant detection

CONCLUSION Malwise effectively identifies malware variants.

Runs in real-time in expected case.

Large functional code base and years of development time.

Happy to talk to vendors.

http://www.FooCodeChu.com


Malware Analysis Analysis.pdf · Definisi Malware Analysis • What is a malware? • Malware (malicious software) ... Malware Sinkronisasi Token • Pelaku: Zeus Banking Trojan •

Malware Analysis Analysis.pdf · Definisi Malware Analysis • What is a malware? • Malware (malicious software) ... Malware Sinkronisasi Token • Pelaku: Zeus Banking Trojan •

Documents
A Fast Flowgraph Based Classification System for Packed and Polymorphic Malware on the Endhost

A Fast Flowgraph Based Classification System for Packed and Polymorphic Malware on the Endhost

Technology
AMVG: Adaptive Malware Variant Generation Framework ......perturbation applications similarly as in [6]. C. Genetic algorithm for malware variant generations GA has been used widely

AMVG: Adaptive Malware Variant Generation Framework ......perturbation applications similarly as in [6]. C. Genetic algorithm for malware variant generations GA has been used widely

Documents
Exploiting Similarity Between Variants to Defeat Malware · Impact of Variation on Malware Defense Adds layer of complication defense was bad enough before variant flood now malware

Exploiting Similarity Between Variants to Defeat Malware · Impact of Variation on Malware Defense Adds layer of complication defense was bad enough before variant flood now malware

Documents
Malware Fails Best Bugs in Malware Felix Leder [Malware ... · 1 Malware Fails Best Bugs in Malware Felix Leder [Malware Detection Team] Felix.Leder@norman.com 5. desember 2011 malware

Malware Fails Best Bugs in Malware Felix Leder [Malware ... · 1 Malware Fails Best Bugs in Malware Felix Leder [Malware Detection Team] [email protected] 5. desember 2011 malware

Documents
CYBERSEC 2020 臺灣資安大會 Windows Malware Hot 5TrickBot malware now checks screen resolution to evade analysis Agent Tesla New AgentTesla variant steals WiFi credentials Reference

CYBERSEC 2020 臺灣資安大會 Windows Malware Hot 5TrickBot malware now checks screen resolution to evade analysis Agent Tesla New AgentTesla variant steals WiFi credentials Reference

Documents
‘Supervised Automation’ for Malware Variant Generation: Theoretical and Practical Implications

‘Supervised Automation’ for Malware Variant Generation: Theoretical and Practical Implications

Documents
Verint Discovers Smart Variant of Zero-Day Sandwormcyber.verint.com/wp-content/uploads/2016/05/Malware_Zero... · 2019-07-27 · The attacker placed the malware payload at a remote

Verint Discovers Smart Variant of Zero-Day Sandwormcyber.verint.com/wp-content/uploads/2016/05/Malware_Zero... · 2019-07-27 · The attacker placed the malware payload at a remote

Documents
Intrusion Detection and Malware Analysis - Malware collection

Intrusion Detection and Malware Analysis - Malware collection

Documents
THE EVOLUTION OF MALWARE & FUTURE MALWARE MITIGATION

THE EVOLUTION OF MALWARE & FUTURE MALWARE MITIGATION

Documents
Intrusion Detection and Malware Analysis - Malware collection134.2.173.140/lehre/ws10/ids-malware/12-malware-collection.pdf · Intrusion Detection and Malware Analysis Malware collection

Intrusion Detection and Malware Analysis - Malware collection134.2.173.140/lehre/ws10/ids-malware/12-malware-collection.pdf · Intrusion Detection and Malware Analysis Malware collection

Documents
SIM5102 SOFTWARE EVALUATION Measuring Internal Product Attributes :Flowgraph measurement(structure) 1

SIM5102 SOFTWARE EVALUATION Measuring Internal Product Attributes :Flowgraph measurement(structure) 1

Documents
Reversing malware analysis trainingpart9 advanced malware analysis

Reversing malware analysis trainingpart9 advanced malware analysis

Technology
Malware, Malware y más Malware ¿Cómo me puedo proteger? “Dijo el cliente web”

Malware, Malware y más Malware ¿Cómo me puedo proteger? “Dijo el cliente web”

Documents
Faster, More Effective Flowgraph -based Malware Classification

Faster, More Effective Flowgraph -based Malware Classification

Documents
Windows Malware Firewall: Remove Windows Malware Firewall

Windows Malware Firewall: Remove Windows Malware Firewall

Technology
Beyond Whitelisting: Fileless Attacks Against Linux · 2019-03-05 · Fileless Malware - Definition “… a variant of computer related malicious software that exists exclusively

Beyond Whitelisting: Fileless Attacks Against Linux · 2019-03-05 · Fileless Malware - Definition “… a variant of computer related malicious software that exists exclusively

Documents
Malware Analysis and Antivirus Technologies: Kernel Malware & A Look at Malware … · 2011-08-16 · Malware Analysis and Antivirus Technologies: Kernel Malware & A Look at Malware

Malware Analysis and Antivirus Technologies: Kernel Malware & A Look at Malware … · 2011-08-16 · Malware Analysis and Antivirus Technologies: Kernel Malware & A Look at Malware

Documents
A Malware Variant Detection Method Based on …seclab.hdu.edu.cn/static/work/paper/13-08.pdfA Malware Variant Detection Method Based on Byte Randomness Test Shuhui Qi, Ming Xu and

A Malware Variant Detection Method Based on …seclab.hdu.edu.cn/static/work/paper/13-08.pdfA Malware Variant Detection Method Based on Byte Randomness Test Shuhui Qi, Ming Xu and

Documents
Malware and anti-malware (fun with Trojans) - Z. …z.cliffe.schreuders.org/edu/FS/Malware and anti-malware... · 2014-04-02 · Malware and anti-malware (fun with Trojans) ... with

Malware and anti-malware (fun with Trojans) - Z. …z.cliffe.schreuders.org/edu/FS/Malware and anti-malware... · 2014-04-02 · Malware and anti-malware (fun with Trojans) ... with

Documents
Metamorphic Malware 1 Metamorphic Malware Research

Metamorphic Malware 1 Metamorphic Malware Research

Documents
Malware Nilgün Kablan. Malware …………………………………….…….……………………………………………… Geschichte der Malware ……………….…………………………………………………

Malware Nilgün Kablan. Malware …………………………………….…….……………………………………………… Geschichte der Malware ……………….…………………………………………………

Documents
On labeling Android malware signatures using minhashing ... · AV27: a variant of Android/AdDisplay.Startapp.B AV28: Adware/Startapp.A AV58: Adware.AndroidOS.Youmi.Startapp (v) Clearly,

On labeling Android malware signatures using minhashing ... · AV27: a variant of Android/AdDisplay.Startapp.B AV28: Adware/Startapp.A AV58: Adware.AndroidOS.Youmi.Startapp (v) Clearly,

Documents
Cashing in on ATM Malware - Europol · 2017-09-27 · ATM malware is one of the digital threats that have been around for a while now, with the discovery of the first known variant

Cashing in on ATM Malware - Europol · 2017-09-27 · ATM malware is one of the digital threats that have been around for a while now, with the discovery of the first known variant

Documents
Dynamic Accident Sequence Analysis using Dynamic Flowgraph

Dynamic Accident Sequence Analysis using Dynamic Flowgraph

Documents
SAP Assortment Planning for Retail 2.0 … article variant product variant product variant product variant product variant article variant 6 CUSTOMER SAP Assortment Planning for Retail

SAP Assortment Planning for Retail 2.0 … article variant product variant product variant product variant product variant article variant 6 CUSTOMER SAP Assortment Planning for Retail

Documents
ISACA - Evolution of Malware and the Next Generation ... · • Malware professionalization • Use of marketing techniques in spam campaigns • Country/Time based malware variant

ISACA - Evolution of Malware and the Next Generation ... · • Malware professionalization • Use of marketing techniques in spam campaigns • Country/Time based malware variant

Documents
Chapter 8 Malware - FTMS · – Boot virus – Logic Bomb virus – Directory virus – Resident virus. CSCA0101 Computing Basics 8 Malware Types of Malware ... Malware Types of Malware

Chapter 8 Malware - FTMS · – Boot virus – Logic Bomb virus – Directory virus – Resident virus. CSCA0101 Computing Basics 8 Malware Types of Malware ... Malware Types of Malware

Documents
RANCANG BANGUN SISTEM INFORMASI MANAJEMEN …repositori.uin-alauddin.ac.id/3547/1/Maulana Malik.pdf · V.18 Flowchart dan Flowgraph Halaman Login..... 60 V.19Flowchart dan Flowgraph

RANCANG BANGUN SISTEM INFORMASI MANAJEMEN …repositori.uin-alauddin.ac.id/3547/1/Maulana Malik.pdf · V.18 Flowchart dan Flowgraph Halaman Login..... 60 V.19Flowchart dan Flowgraph

Documents
Improving accuracy of malware detection by …...FFRI, Inc. Background – malware and its detection 4 Increasing malware Targeted Attack (Unknown malware) Malware generators Obfuscators

Improving accuracy of malware detection by …...FFRI, Inc. Background – malware and its detection 4 Increasing malware Targeted Attack (Unknown malware) Malware generators Obfuscators

Documents