Hypervisor Security - OpenStack Summit Hong Kong

Robert ClarkLead Security ArchitectHP Cloud

Hypervisor Security

About the Speaker

OpenStack Security Group

• Established 18-24 months ago• Issues OpenStack Security Notes• Consults on OpenStack Security Advisories• Security Initiatives• Nearly 100 members

OpenStack Security Guide

http://docs.openstack.org/security

OpenStack Security Guide

Virtualization Overview

Virtualization Technologies

• Hosted OS Virtualization – VMware Desktop Solutions

• Para Virtualization – The guest needs to know it’s running in a virtualized environment

• Full Virtualization – The guest is un-aware that it is running on a virtualized platform.

Virtualization StackCompute Host

AliceVM

AliceVM

AliceVM

Hardware

Hypervisor

Device Emulation

Simplified KVMCompute Host

AliceVM

AliceVM

AliceVM

Hardware CPU VIRT

Linux Kernel KVM

QEMU

Linux OS

Simplified XenCompute Host

Dom0 AliceVM

Hardware

Xen Hypervisor

AliceVM

QEMU

Generalized Virtualization StackCompute Host

AliceVM

AliceVM

AliceVM

Hardware

Hypervisor / Host OS / Dom0

QEMU

Compute Instances

Device Emulation/ Paravirt

Hardware Interfacing/ Enabling

HardwareMemory, Disk, CPU etc

Attack Vectors

Introducing ‘Mal’

MalVM

Compute Host Attack Vectors

Compute Host [Nova]

AliceVM

AliceVM

Compute Instance Attack Vectors

Compute Host [Nova]

AliceVM

AliceVM

BobVM

BobVM

Compute Instance Attack Vectors

Compute Host [Nova]

AliceVM

AliceVM

BobVM

BobVM

KVM / XEN

QEMU

Compute Instance Attack Vectors

Compute Host [Nova]

AliceVM

AliceVM

BobVM

BobVM

KVM / XEN

QEMU

Dom0

Linux Kernel

Linux OS

Compute Instance Attack Vectors

Compute Host [Nova]

AliceVM

AliceVM

BobVM

BobVM

MalVM KVM / XEN

QEMU

Dom0

Linux Kernel

Linux OS

Compute Instance Attack Vectors

Compute Host [Nova]

AliceVM

AliceVM

BobVM

BobVM

MalVM

QEMU

Basic VM to VM network Attacks

KVM / XEN

QEMU

Dom0

Linux Kernel

Linux OS

Compute Instance Attack Vectors

Compute Host [Nova]

AliceVM

AliceVM

BobVM

BobVM

MalVM KVM / XEN

QEMU

VM to hypervisor attacks

KVM / XEN

QEMU

Dom0

Linux Kernel

Linux OS

Compute Instance Attack Vectors

Compute Host [Nova]

AliceVM

AliceVM

BobVM

BobVM

MalVM KVM / XEN

QEMU

VM to QEMU / Device attacks

KVM / XEN

QEMU

Dom0

Linux Kernel

Linux OS

Dom0

Compute Instance Attack Vectors

Compute Host [Nova]

AliceVM

AliceVM

BobVM

BobVM

MalVM

QEMU

Linux Kernel

Linux OS

VM to QEMU

KVM / XEN

QEMU

Dom0

Compute Instance Attack Vectors

Compute Host [Nova]

AliceVM

AliceVM

BobVM

BobVM

MalVM

QEMU

Linux Kernel

Linux OS

VM to QEMU

KVM / XEN

QEMU

Dom0

Compute Instance Attack Vectors

Compute Host [Nova]

AliceVM

AliceVM

BobVM

BobVM

MalVM

QEMU

Linux Kernel

Linux OS

1.

VM to QEMU

KVM / XEN

QEMU

2.

Dom0

Compute Instance Attack Vectors

Compute Host [Nova]

AliceVM

AliceVM

BobVM

BobVM

MalVM

QEMU

Linux Kernel

Linux OS

1.

VM to QEMU

KVM / XEN

QEMU

2.

Dom0

Compute Instance Attack Vectors

Compute Host [Nova]

AliceVM

AliceVM

BobVM

BobVM

MalVM

QEMU

Linux Kernel

Linux OS

VM to hypervisor attacks

KVM / XEN

QEMU

Dom0

Compute Instance Attack Vectors

Compute Host [Nova]

AliceVM

AliceVM

BobVM

BobVM

MalVM

QEMU

Linux Kernel

Linux OS

VM to hypervisor attacks

KVM / XEN

QEMU

Dom0

Compute Instance Attack Vectors

Compute Host [Nova]

AliceVM

AliceVM

BobVM

BobVM

MalVM

QEMU

Linux Kernel

Linux OS

VM to hypervisor attacks

KVM / XEN

QEMU

Dom0

Compute Instance Attack Vectors

Compute Host [Nova]

AliceVM

AliceVM

BobVM

BobVM

MalVM

QEMU

Linux Kernel

Linux OS

VM to OS / Management / Linux Kernel / Dom0

KVM / XEN

QEMU

Dom0

Compute Instance Attack Vectors

Compute Host [Nova]

AliceVM

AliceVM

BobVM

BobVM

MalVM

QEMU

Linux Kernel

Linux OS

VM to OS / Management / Linux Kernel / Dom0

KVM / XEN

QEMU

Dom0

Compute Instance Attack Vectors

Compute Host [Nova]

AliceVM

AliceVM

BobVM

BobVM

MalVM

QEMU

Linux Kernel

Linux OS

1.

2.

VM to OS / Management / Linux Kernel / Dom0

KVM / XEN

QEMU

Dom0

Compute Instance Attack Vectors

Compute Host [Nova]

AliceVM

AliceVM

BobVM

BobVM

MalVM

QEMU

Linux Kernel

Linux OS

1.

2.

3.

VM to OS / Management / Linux Kernel / Dom0

KVM / XEN

QEMU

Cloud Issues

Compute Host [Nova]

AliceVM

BobVM

Cloud Issues - Scale

Compute Host [Nova]

CherVM

DaveVM

Compute Host [Nova]

AliceVM

BobVM

Cloud Issues - Scale

Compute Host [Nova]

CherVM

DaveVM

Compute Host [Nova]

AliceVM

BobVM

Compute Manager

Block Storage

Network Nodes

Operations Systems

Object Storage

Cloud Issues - Scale

Compute Host [Nova]

CherVM

DaveVM

Compute Host [Nova]

AliceVM

BobVM

Compute Manager

Block Storage

Network Nodes

Operations Systems

Object Storage

MalVM

Cloud Issues – Flat Exploitation

Compute Host [Nova]

CherVM

DaveVM

Compute Host [Nova]

AliceVM

BobVM

Compute Manager

Block Storage

Network Nodes

Operations Systems

Object Storage

MalVM

Cloud Issues – Flat Exploitation

Compute Host [Nova]

CherVM

DaveVM

Compute Host [Nova]

AliceVM

BobVM

Compute Manager

Block Storage

Network Nodes

Operations Systems

Object Storage

MalVM

Cloud Issues – Service Trust

Compute Host [Nova]

CherVM

DaveVM

Compute Host [Nova]

AliceVM

BobVM

Compute Manager

Block Storage

Network Nodes

Operations Systems

Object Storage

MalVM

Cloud Issues – Service Trust

Compute Host [Nova]

CherVM

DaveVM

Compute Host [Nova]

AliceVM

BobVM

Compute Manager

Block Storage

Network Nodes

Operations Systems

Object Storage

MalVM

Cloud Issues – Nova RPC

Compute Host [Nova]

CherVM

DaveVM

Compute Host [Nova]

AliceVM

BobVM

Compute Manager

Block Storage

Network Nodes

Operations Systems

Object Storage

MalVM

What about side channels?

Cross-VM Side Channel Attacks

• Web Servers providing SSL• VOIP providers• Cloud VPN• Chat Applications• Secure File Storage• Virtually any service doing anything useful

Cross-VM Side Channel Attacks

AliceClient

Compute Host [Nova]

BobVM

TLS/SSL

CPU

L1 Cache

• Disrupting or observing system operation

Cross-VM Side Channel Attacks

AliceClient

Compute Host [Nova]

BobVM

TLS/SSL

Stealing the bits!

MalMITM

CPU

L1 Cache

Cross-VM Side Channel Attacks

AliceClient

Compute Host [Nova]

BobVM

MalVM

TLS/SSL

MalMITM

CPU

L1 Cache

Stealing the bits!

Cross-VM Side Channel Attacks

AliceClient

Compute Host [Nova]

BobVM

MalVM

TLS/SSL

MalMITM

CPU

L1 Cache

Stealing the bits!

Cross-VM Side Channel Attacks

AliceClient

Compute Host [Nova]

BobVM

MalVM

TLS/SSL

MalMITM

CPU

L1 Cache

Stealing the bits!

Isn’t this all a bit theoretical?

CloudBurst

• Date: 2008• Type: OS Virtualization - VMWare• Result: Full Breakout• Author: Kostya Kirtchinsky, Immunity Inc

Xen Ownage Trilogy

• Date: 2011• Type: Xen • Result: Full Breakout• Author: Joanna Rutkowska

VirtuNoid

• Date: 2011• Type: Kernel Side Full Virtualization - KVM• Result: Full Breakout• Author: Nelson Elhage• CVE-2011-1751

SYSRET-64

• Date: 2012• Type: Para Virtualization - Xen• Result: Full Breakout• Author: Rafal Wojtczuk• US-CERT #649219

VMDK Has Left The Building

• Date: 2012• Type: ESXi File Handling Logic• Result: Data Leakage / Loss• Author: Friedwart Kuhn

KVM IOAPIC, SET MSR, TIME

• Date: 2013• Type: Full Virtualization - KVM• Result: Denial of Service, Potential Breakout• Author: Andrew Honig• IOAPIC: CVE-2013-1798• TIME: CVE-2013-1797• SET MSR: CVE-2013-1796

Virtualization Security Trends

IBM X-Force 2010 Mid-Term Report

Virtualization Security TrendsAttack Vector Xen KVM

Virtual CPUs 5 (8.5%) 8 (21.1%)

SMP 1 (1.7%) 3 (7.9%)

Software MMU 4 (6.8%) 2 (5.3%)

Interrupt and Timer Mechanisms 2 (3.4%) 4 (10.5%)

I/O and Networking 11 (18.6%) 10 (26.3%)

VM Exits 4 (6.8%) 2 (5.3%)

Hypercalls 2 (3.4%) 1 (2.6%)

VM Management 7 (11.9%) 2 (5.3%)

Remote Management Software 9 (15.3%) 1 (2.6%)

Hypervisor add-ons 5 (8.5%) 0 (0.0%)

TOTAL 59 38

Time to unplug?

Go home cloud, you’re drunk!

Protections – Compiler Hardening

• RELocation Read-Only• Stack Canaries• Never eXecute (NX) / (DEP)• Position Independent Executable• Address Space Layout Randomization• QEMU:

CFLAGS="-arch x86_64 -fstack-protector-all -Wstack-protector --param ssp-buffer-size=4 -pie -fPIE -ftrapv - D_FORTIFY_SOURCE=2 O2 -Wl,-z,relro,-z,now"

Protections – Reduce Attack Surface

• Out of the box you probably support– 3D Graphics– Multiple Network Devices– Sound– Bluetooth!?

• Compile them out!

Protections – Mandatory Access Controls

• Limit the capabilities of a successful exploit• Define and constrain with QEMU should be

doing• Provide isolation for VM processes (KVM)• SELinux • AppArmour

Protections – Mandatory Access Controls

Protection

• Reduce Attack Surface• Harden Compilation• Isolate, detect and alert on exploitation

through MAC• Harden your base OS/Dom0 using the same

techniques• Apply MAC to other OpenStack components

OpenStack Security Guide

• http://docs.openstack.org/sec• Chapter 26 – Securing OpenStack Networking

Services• Chapter 40 – Hypervisor Selection• Chapter 41 – Hardening the Virtualization

Layers• Chapter 43 – Security Services for Instances

Thank You

Please consider contributing to the OpenStack Security Group

References• Directly Referenced / Informed This Talk

– http://www.insinuator.net/2013/05/analysis-of-hypervisor-breakouts/– https://www.ernw.de/download/ERNW_DCVI-HypervisorsToClouds.pdf– https://www.hashdays.ch/downloads/slides/jonathan_sinclair_vm_state.pdf– ftp://public.dhe.ibm.com/linux/pdfs/LXW03004-USEN-00.pdf– http://blog.cryptographyengineering.com/2012/10/attack-of-week-cross-vm-timing-

attacks.html– http://www.vupen.com/blog/

20120904.Advanced_Exploitation_of_Xen_Sysret_VM_Escape_CVE-2012-0217.php– http://www.symantec.com/avcenter/reference/Virtual_Machine_Threats.pdf– http://invisiblethingslab.com/resources/bh08/part1.pdf– http://blogs.gartner.com/neil_macdonald/2011/01/26/yes-hypervisors-are-

vulnerable/– ftp://public.dhe.ibm.com/common/ssi/ecm/en/wgl03003usen/

WGL03003USEN.PDF