Challenges  in  designing  a  type  1  hypervisor  for  ARM  v7  VE

Manish  Jaggi  manish.jaggi@gmail.com  

http://in.linkedin.com/in/manishjaggi

Hypervisor  Layering  in  Software  Stack

• Type  1  HyperOne,  Xen,  PikeOS,  OKL4,  HyperV,  Vmware  ESX  !• Type  2  KVM,  VirtualBox,  Virtual  PC,  Parallels,  BlueStacks  

Hardware

Hypervisor

UserSpace

Kernel

Hardware

OS  Kernel

UserSpace

Kernel

Hypervisor

VM1 VM2

UserSpace

KernelUserSpace

Monolithic  vs  Microkernel  Design

HyperOne  Microkernel  Design..

Hardware

Hypervisor

VM2 Driver  VM

VM1  Root

InterVM  IPC

Virtual  Memory  Scheduling Interrupt  

Routing

Virtualization  Stack

Device  Drivers Device  DriversDevice  Drivers

• Hypervisor  Privilege  Mode  (PL2)  NS  • Different  set  of  CP15  registers  for  hypervisor  • Stage  2  MMU  and  SMMU  • GIC  Virtual  CPU  I/f  • Timer  Virtualization  (Virtual  Timer  Interrupts  for  guest  /  hypervisor)

ARM  v7  Virtualization  Extensions

Guest  Stage  2  MMU  Translation

GICv2

SMMU

Bare  Minimal  Hypervisor

• A  hypervisor  boots  up  like  a  normal  smp  os  – MMU  /  GIC  /  UART  /  Timer  drivers  – Board  specific  /  platform  specific  code  – Needs  a  device  tree  – Has  a  console  (shell)  

• Hypervisor  functions  – Trap  guests  accesses    – Emulate    – Load/run/suspend/resume  guests

vm_tvcpu_t

interrupts

Core_regs

Cp15_regs

VM  State

vGIC  State

virtual  devices  state

IPC  Channels

Memory  Assigned

Hypervisor  keeps  the  state  of  each  VM  and  its  associated  vcpus.  !A  VM  has  a  machine  definition  and  physical  and  virtual  devices  connected  to  it  !State  of  virtual  devices  is  kept  with  hypervisor  like  VGIC,  console,  network  and  other  virtual  devices.  !VCPU  has  cpu_regs  context,  the  cp15  regs  and  the  descriptors  of  interrupts  injected  into  the  vcpu.  !This  state  is  saved  and  restored  as  and  when  the  VM  or  vcpu  is  scheduled  out  /  in.  !There  are  two  types  of  context  saves,  Shallow  and  deep.  

• Hypervisor  handles  exceptions  originated  from  hypervisor  mode  • Hypervisor  also  handles  the  exceptions  originated  from  Guest  

Mode,  these  are  routed  to  hypervisor  as  hypervisor  trap  and  hypervisor  identifies  the  type  of  exception  using  a  syndrome    register.  

• Virtual  Devices’  Memory  mapped  address  space  access  is  trapped  by  hypervisor  and  the  read  write  is  emulated  in  hypervisor  mode  

• Guests  can  also  issue  hyper  calls  to  the  hypervisor.  These  are  mainly  used  by  para  virtualised  drivers.  

• Privileged  instructions  such  as  WFI  and  WFE  trap  into  hypervisor.

Hypervisor  Exception  Handling

• Hypervisor  emulated  GICD  and  injects  interrupts  in  guest  using  GICH  

• Hypervisor  programs  the  Link  registers  

• Guest  access  the  virtual  CPU  I/f  in  the  GIC  itself,  no  emulation  is  required  

• There  are  2  methods  for  hypervisor  to  b  notified  that  guest  has  deactivated  the  interrupt  • Maintenance  Interrupt  • UnderFlow  Interrupt

Guest  Interrupt  HandlingGICD  Driver

Guest

Hypervisor

SoCUART

VCPU  I/F CPU  I/F Hyp  I/f Distributor

Level  Interrupt

GIC

GICD  Emulation

Trap

GICC  Driver

• ARM  provides  a  Physical  timer  register  and  a  Virtual  Timer  register.  Virtual  Timer  is  an  offset  from  physical  timer  

• Hypervisor  Programs  the  CNTVOFF  register  before  guest  is  resumed.  

• ARM  provides  separate  timer  interrupt  for  hypervisor  and  guest.  

• Timer  interrupt  is  injected  in  same  way  as  other  interrupts

Timer  Virtualization

Managing  Virtual  Device  -­‐  Virtio  MMIO

• virtio  is  an  abstraction  layer  over  devices  in  a  paravirtualized  hypervisor.  

• virtio  mmio  does  not  require  a  PCI  device  

15

• Debugging  a  guest  can  be  done  on  hypervisor  console  • for  dumping  guest  state  on  console  • for  setting  software  breakpoints  

• or  gdb  interface  can  be  added  in  hypervisor  to  debug  a  guest  

Guest  Debug  Support

VM  Management  Summary

• Context  Save  and  Restore  • Booting  and  Loading  • Scheduling  • VCPU  Management  • Remote  Guest  Image  loading

18

Remote  Device  Management

A typical use case of Virtual Machines on a device is an enterprise Virtual Machine to provide a secure and separate workspace environment for employees. !Basic management operations include creating/deleting, resuming/suspending and powering on/off of a Virtual Machine

Questions