Anomaly Detection Using Call Stack Information

Security Reading GroupJuly 2, 2004

Henry Feng, Oleg Kolesnikov, Prahlad Fogla, Wenke Lee, Weibo Gong

Presenter: Jonathan McCune

2

Overview

• Introduction• Related work• VtPath• Experiments• Exploits• Comparisons• Conclusion

3

Introduction

• Runtime, training-phase based, anomaly detection system

• Uses call stack and program counter in addition to system call hooks

• Novel features based on virtual paths• Compared experimentally and

analytically with many other approaches

4

Related work

• Static analysis– Wagner, et al. (abstract stack)– Callgraph

• Run-time, training based:– This– Sekar, et al. (FSA) (last week’s srg)– N-gram, Var-gram

5

“Intrusion detection via static analysis.” Wagner, et al.

• Static analysis of program source code• NDFA from control-flow graph

– Cannot predict branches statically• Impossible path problem• Abstract stack model

– Pushdown automaton• Large automata can be resource intensive• Claimed zero false positives

6

“Detecting Manipulated Remote Call Streams.” Giffin, et al.

• Static analysis of binary executables• Tied to platform, not programming

language• Insertion of “null calls” helps impossible

path problem• Unusual performance analysis

7

“A fast automaton-based method for detecting anomalous program

behavior.” Sekar, et al.• Compact deterministic FSA from

system call analysis of running programs

• Suffers from false positives, but not as a result of non-determinism

• Impossible paths not addressed• DLLs not adequately addressed

8

Virtual Path (VtPath)

• Run-time analysis• Learns correct behavior via training

executions • Utilizes return address information• Generates abstract path and compares

with learned behavior• Similar to abstract stack of Wagner, et

al., but avoid pushdown automaton

9

Virtual Path Example

main()

foo1()

foo()

foo2()

foo3()

main()

fooA()

foo()

fooB()

fooC()

10

VtPath Training Phase

• Two hash tables:– RA (return address) table– VP (virtual path) table

• RAs and VPs gradually added during normal program execution

• Use NULL entries at beginning and end of paths

11

VtPath Online Detection Phase

• Stack anomaly – if virtual stack list unavailable (common during buffer overflow attacks)

• Return address anomaly – if virtual stack list {a0, a1, … an} contains ai not in RA table

• System call anomaly – if an does not have the correct system call

• Virtual path anomaly – if virtual path at current system call is not in VP table

12

VtPath – Impossible Path Problem

• Return addresses are part of virtual path information

• Modifying return address saved on the stack will cause the program to return to a different location

• The next system call is likely to trigger a virtual path anomaly

13

VtPath - Implementation Issues

• Non-standard control flows– Signals (sigreturn system call)

• Treat each one like a separate program invocation– setjmp()/longjmp()and function pointers

• Hard to handle statically; handled at runtime if trained properly

• Dynamically linked libraries– Relative loading positions can change, invalidating PC

values from training runs– Use a “block” model during training to capture file name and

block length, ignoring start address• Block anomaly – block lookup fails because attacker

is trying to load a malicious DLL

14

Experiments – VtPath vs FSA

• Convergence times similar, but – FSA generates more transitions than VtPath (less

efficient, less precise)– In practice, multiple levels of DLL functions are

called quite frequently – VtPath more effective• False Positives nearly identical• VtPath executes faster, uses more memory• Common exploits detected by both

15

Exploits

• Authors developed two masked mimicry attacks detectable only by VtPath

• Impossible Path Execution (IPE) Attack 1– Exploits parallel structure of privileged /

unprivileged code within same function• IPE Attack 2:

– F() called twice from within same function with different arguments

– Overflow local variable to change option passed in

16

Attack 1

17

Attack 2

18

Comparison of SysCall-based Anomaly Detection Schemes

1. State-based / Information captured• Data/heap values least useful (transient)• Code segment of some value• Syscalls and callstack most useful• More information = runtime overhead

2. False positives• How well is normal behavior captured?• Only relevant for dynamic systems• Proportional to resolution of program analysis

3. Detection capability• More granularity = better detection capability• SysCalls from invalid points vs. statistical regularity of

training data vs. attacks’ deviation from perceived normal

19

Comparison of SysCall-based Anomaly Detection Schemes

4. Space requirement• System call sequences• Number of NDFA transitions (Wagner)• Transitions in automaton

5. Convergence (training) time• Cover most possible states / transitions• VtPath requires more data than FSA• Static techniques have big advantage here

6. Runtime overheads• SysCall interception• Hash lookup for valid state / return address / virtual path

20

Conclusions

• Using call stack (VtPath) has value• There is no magic bullet

– Everything is “complementary”• Impossible path execution (IPE) is an

important class of attack• Use of FSA to analyze more than two

consecutive SysCalls could improve VtPath

21

Discussion!