In-Band Detection of Virtual Machines

In-Band Detection of Virtual Machines

Estefan Ortiz & Cory Hayes

Computer Science and Engineering

Graduate Operating Systems

December 16, 2011

1

2

Introduction

Malicious programs (malware) need to know if they are in a virtual environment so they can modify their behavior and avoid detection

Related work Red Pill Tests: Examine byte-level behavior of instructions

for physical and emulated CPUs. If any disagreements in output, create one or more “red pills” that can avoid detection

SubVirt: Virtual machine-based rootkit installed underneath host OS that runs OS as a guest to remain nearly undetectable

3

Our Approach

Similar to Red Pill and SubVirt, but client-server based

Idea: Instead of monitoring system call discrepancies, analyze network data sent to/from physical and virtual machines

Goal: Determine if there are sufficient differences in network traffic to detect if a client/server is being run on a virtual machine

4

Goal

Client <-> Native TCP/IP Packet Client <-> Virtual Machine TCP/IP Packet

Byte 0

Byte n

Byte k1

Byte k2

Difference Found

5

General Setup

6

Actual Setup

Host Server(Apache)

Wireshark

Port

Client

Port

Switch

Network output saved for analysis

Functions as the “MITM”

7

Experiment Setup

Using Wireshark, capture and compare the raw info of TCP/IP packets sent back and forth between a client and a physical/virtual server running Apache Bits 1-160: IP Remainder: TCP

Virtual machine OS matches the OS of the host (Ubuntu-Ubuntu, Vista-Vista)

Use a small set of Matlab commands to send regular and malformed packets

Dynex 5-port 10/100/1000 Gigabit Ethernet Switch

8

Sample Captured Wireshark Output

8th Packet sent between Client & VM running Apache

8th Packet sent between Client & Host running Apache

VM

Client

Host

9

Metrics

Bit Difference Comparison: Fractional Hamming distance between two packets

10

Metrics (cont.)

Round trip time: Time from SYN request sent by client to received ACK from server

11

Metrics (cont.)*

Pairwise Packet Length Comparison: Number of concurrent packet pairs that differ in length

12

Experiment #1

Client: Windows Vista (4GB RAM, 2.6GHz) Server: Ubuntu 11.04 32-bit w/ Apache Web Server 2.2 Server: Host OS Ubuntu: VirtualBox w/ Ubuntu running

Apache

On isolated switch network (no other traffic)

13

Exp. #1: Frac. Hamming Distance

14

Exp. #1: Round-trip Timing

15

Example: Packet #9

These bits correspond to the header length & flags in the TCP header

16

Experiment #2

Client: Mac (4GB RAM, 2.4GHz, MacOSX 10.6.8) Server: Windows Vista 32-bit w/ Apache Web Server 2.2 Server: Host OS Windows Vista: VirtualBox w/ Windows

Vista running Apache

On isolated switch network (no other traffic)

17

Exp. #2: Frac. Hamming Distance

18

Exp. #2: Round-trip Timing

19

Example: Packet #4

Destination Address in IP header

Flags in TCP header

20

Experiment #3

Client: Windows Vista (4GB RAM, 2.6GHz) Server: Ubuntu 11.04 32-bit w/ Apache Web Server 2.2 Server: Host OS Ubuntu: VirtualBox w/ Ubuntu running

Apache

Both client and server on CVRL subnet (at ~3:00 am)

21

Exp. #3: Frac. Hamming Distance

22

Exp. #3: Round-trip Timing

23

Example: Packet #3

Destination Address in IP header

24

Experiment #4

Internet

Sprint Mobile Hotspot

Host Server(Apache)

Port

ND/CVRL subnet

Client

PortFIREWALL

25

Experiment #4

Client: Windows Vista (4GB RAM, 2.6GHz) Server: Ubuntu 11.04 32-bit w/ Apache Web Server 2.2 Server: Host OS Ubuntu: VirtualBox w/ Ubuntu running

Apache

Could not monitor packet information; only ping tests Varied number of bytes sent using ping Performed 100 per fixed byte amount Calculated avg. & std. dev

Executed at ~3:30 am

26

Exp. #4: Ping Timing

27

Conclusion

Examined packet information from a high level (packet-length) down to specific bit difference comparisons

Packet length provided no insight

Timing tests didn’t provide conclusive evidence of a connection to a virtual machine

Fractional hamming dist. provided first level of insight

Further analysis of differences at the bit level provided clues where to look for VM traces

28

Future Direction

Experiments 1-3 were conducted under somewhat “ideal” scenarios

More realistic approach would be packet analysis on multi-hop connections with knowledge of which sections of the TCP/IP packets to monitor