Scalability, Fidelity, and Scalability, Fidelity, and Containment in the Potemkin Containment in the Potemkin

Virtual HoneyfarmVirtual Honeyfarm

Michael Vrable, Justin Ma, Jay chen, David Moore, Erik Michael Vrable, Justin Ma, Jay chen, David Moore, Erik Vandekieft, Alex C. Snoeren, Geoffrey M. Voelker, Stefan Vandekieft, Alex C. Snoeren, Geoffrey M. Voelker, Stefan

SavageSavage

Collaborative Center for Internet Epidemiology and DefensesCollaborative Center for Internet Epidemiology and DefensesDepartment of Computer Science and EngineeringDepartment of Computer Science and Engineering

Univsersty of California, San DiegoUnivsersty of California, San Diego

Background InfoBackground Info

Network Telescope TheoryNetwork Telescope Theory

HoneyPots – A system of Intrusion/Threat HoneyPots – A system of Intrusion/Threat Detection where the value lies in that all traffic in Detection where the value lies in that all traffic in system is not legitimatesystem is not legitimate

High Interaction or Low Interaction?High Interaction or Low Interaction? Benefit of Low Interaction is large number of IPs can Benefit of Low Interaction is large number of IPs can

be coveredbe covered Benefit of High Interaction is you can gain better Benefit of High Interaction is you can gain better

insight into the methods used and possible outcomes insight into the methods used and possible outcomes of attacksof attacks

Bottom LineBottom Line

You can have one a system that You can have one a system that represents a larger net so you have better represents a larger net so you have better odds of finding something maliciousodds of finding something malicious

Or, you can have a system that monitors a Or, you can have a system that monitors a smaller set of IPs because there is more smaller set of IPs because there is more overhead in providing kernel and system overhead in providing kernel and system access to the potential threat, and not just access to the potential threat, and not just mimicking network presence.mimicking network presence.

Bottom Line ?Bottom Line ?

So why cant you have your cake and eat it too?So why cant you have your cake and eat it too?

Is it possible to provide a system that will allow Is it possible to provide a system that will allow you to combine the best of both worlds.you to combine the best of both worlds.

Can you provide a Honeyfarm solution that Can you provide a Honeyfarm solution that allows you monitor a large IP set, and provide a allows you monitor a large IP set, and provide a valid system for each threat to incubate so valid system for each threat to incubate so analysis can be in-depth? Can you do it with out analysis can be in-depth? Can you do it with out throwing large amounts of money at it?throwing large amounts of money at it?

Basis of PaperBasis of Paper

This is the aim of this paper.This is the aim of this paper.

Utilize VM technology and custom Utilize VM technology and custom software design to create a system which software design to create a system which has high fidelity, and can scale well to has high fidelity, and can scale well to monitor a large environment if the need monitor a large environment if the need arises.arises.

Don’t break the bank doing it either!Don’t break the bank doing it either!

ProblemsProblems

ResourcesResources MemoryMemory CPUCPU HD SpaceHD Space

RoutingRouting How do we route the packets so Honeyfarm is How do we route the packets so Honeyfarm is

invisible?invisible? How do we route packets so as not to cause an How do we route packets so as not to cause an

outbound attack?outbound attack?

LatencyLatency How do we provide interaction so that the attacker How do we provide interaction so that the attacker

does not know he is in a virtual environment?does not know he is in a virtual environment?

Solutions!Solutions!

Flash CloningFlash Cloning Allow Farm to scale as need arisesAllow Farm to scale as need arises

Delta Virtualization (Copy-On-Write)Delta Virtualization (Copy-On-Write) Addresses timing and resource use of each Addresses timing and resource use of each

cloneclone

Creative RoutingCreative Routing Limits farm to only dealing with IPs that solicit Limits farm to only dealing with IPs that solicit

communication.communication.

Flash CloningFlash Cloning

VM Machine instantiation can have high VM Machine instantiation can have high overhead and latency, especially when VM overhead and latency, especially when VM needs to boot and load devices.needs to boot and load devices.To work around this, provide a “Reference To work around this, provide a “Reference Image”.Image”.An Image of an already loaded O/S is kept An Image of an already loaded O/S is kept frozen and unchanged. When need arises frozen and unchanged. When need arises for a new VM, clone this one. It is already for a new VM, clone this one. It is already to run, just change IPs.to run, just change IPs.

Flash CloningFlash Cloning

BenefitsBenefits Quicker Load timeQuicker Load time New VMs can react to each new outside New VMs can react to each new outside

probe/threatprobe/threat Allows a pristine VM to be examined after Allows a pristine VM to be examined after

compromise. You have a baseline to compare compromise. You have a baseline to compare a compromised VM to.a compromised VM to.

Clone can be created and threat will only Clone can be created and threat will only receive initial delay between first packet and receive initial delay between first packet and response.response.

Flash CloningFlash Cloning

Courtesy of the paper and its authors

Delta VirtualizationDelta Virtualization

Essentially an optimized Copy-on-Write Essentially an optimized Copy-on-Write technique.technique.For each VM Cloned, the entire image For each VM Cloned, the entire image need not be copied.need not be copied.There will always be static parts of the OS There will always be static parts of the OS memory that does not change.memory that does not change.If need for that specific VM to alter If need for that specific VM to alter memory tables arise, then copy memory memory tables arise, then copy memory for that location and change memory table for that location and change memory table for VM to point to new locationfor VM to point to new location

Delta VirtualizationDelta Virtualization

Courtesy of the paper and its authors

Creative RoutingCreative Routing

Each Incoming Packet is Mirrored at Edge Each Incoming Packet is Mirrored at Edge Router to HoneyFarmRouter to HoneyFarm

The farm has it’s own machine dedicated The farm has it’s own machine dedicated to routing packets.to routing packets.

For each packet destined for an IP known For each packet destined for an IP known to be unused, the gateway notifies Cloning to be unused, the gateway notifies Cloning Manager on least busy machine to Manager on least busy machine to allocate new clone with specific IP.allocate new clone with specific IP.

Creative RoutingCreative Routing

After initial lag from cloning, clone is ready and After initial lag from cloning, clone is ready and notifies Clone Manager.notifies Clone Manager.

Clone Manager tells gateway which then flushes Clone Manager tells gateway which then flushes buffer of packets waiting for clone and adds buffer of packets waiting for clone and adds routing rule to push all future communication for routing rule to push all future communication for that IP address to that clone.that IP address to that clone.

To prevent horizontal port scans from To prevent horizontal port scans from overwhelming farm, all future unused attempts overwhelming farm, all future unused attempts from that IP are ignored to keep clone numbers from that IP are ignored to keep clone numbers in check.in check.

Here is where the creativity comes Here is where the creativity comes inin

What about threats that spread like worms? What about threats that spread like worms? Viruses that call home? Rootkits that update Viruses that call home? Rootkits that update themselves?themselves?

Each communication between an outside IP and Each communication between an outside IP and an Internal IP is considered a Universe and the an Internal IP is considered a Universe and the route reflects it. route reflects it.

If compromised clone attempts outside If compromised clone attempts outside communication, the communication is reflected communication, the communication is reflected back toward another clone inside the farm.back toward another clone inside the farm.

Here is where the creativity comes Here is where the creativity comes inin

Thus, the farm can also serve as a ‘incubator’, Thus, the farm can also serve as a ‘incubator’, providing a microcosm for the threat to grow.providing a microcosm for the threat to grow.

Also allows for the possibility of cross Also allows for the possibility of cross contamination. You could setup rules to allow to contamination. You could setup rules to allow to uniquely infected clones to communicate with uniquely infected clones to communicate with each other and create hybrid compromises.each other and create hybrid compromises.

Another unseen benefit is you can provide a Another unseen benefit is you can provide a concrete spread rate of a new threat. Thus, concrete spread rate of a new threat. Thus, providing some reliable scale to rate new threats providing some reliable scale to rate new threats on.on.

The numbers don’t lieThe numbers don’t lie

The largest HoneyFarm known to the The largest HoneyFarm known to the authors was Symantec’s DeepSight using authors was Symantec’s DeepSight using 40 servers with VMware to mimic 2000 IP 40 servers with VMware to mimic 2000 IP addresses.addresses.During Potemkin’s ‘Live Deployment’, the During Potemkin’s ‘Live Deployment’, the max they were able to simulate was 2100 max they were able to simulate was 2100 VMs using one gateway and 9 servers. All VMs using one gateway and 9 servers. All using 2.8 GHZ Xeons’s with 2GB of using 2.8 GHZ Xeons’s with 2GB of memory and a gigabit NIC. Roughly memory and a gigabit NIC. Roughly $10,000 total by current market value.$10,000 total by current market value.

Performance NumbersPerformance NumbersRight hand side represents possible future enhancements by recycling data structures and tables of VMs that were tore down.

Tables Courtesy of the Paper and it’s

Authors.

StrengthsStrengths

Provides some real good ideas to Provides some real good ideas to maximize performance with limited maximize performance with limited hardware.hardware.

Incubator idea is real interesting.Incubator idea is real interesting.

Infection rate idea is real interesting.Infection rate idea is real interesting.

Considered legalities of HoneyFarm Considered legalities of HoneyFarm infecting external IPs and also considered infecting external IPs and also considered Hybrid Infections.Hybrid Infections.

WeaknessesWeaknesses

Live testing did not last longer then 10 Live testing did not last longer then 10 minutes.minutes.A lot of bugs still left to work out before the A lot of bugs still left to work out before the solution could be considered stable solution could be considered stable enough for long term deployment.enough for long term deployment.System can be exploited by attacker to System can be exploited by attacker to exhuast amount of resources in system.exhuast amount of resources in system.Time characteristics can be used against Time characteristics can be used against HoneyFarm to signal virtual environment.HoneyFarm to signal virtual environment.

WeaknessesWeaknesses

Threat could be able to look at limited devices Threat could be able to look at limited devices available and conclude in virtual environment.available and conclude in virtual environment.

Threat could also reference outside IP to Threat could also reference outside IP to determine if in virtual environment.determine if in virtual environment.

Could only be useful in examining malicious Could only be useful in examining malicious programs that are not designed to look for virtual programs that are not designed to look for virtual environments, as an actual attacker worth their environments, as an actual attacker worth their salt could determine it is virtual environment.salt could determine it is virtual environment.

ExtensionsExtensions

Elaborate on the idea of incubation more.Elaborate on the idea of incubation more.

Improve multiple OS support.Improve multiple OS support.

Enable packet analysis at gateway to Enable packet analysis at gateway to determine which OS to clone to provide determine which OS to clone to provide ‘best fit’ for attack.‘best fit’ for attack.

Stabilize system and introduce VM HD Stabilize system and introduce VM HD support so each clone can get access to support so each clone can get access to swap space.swap space.