vNUMA in Xen

Wei Liu & Elena Ufimtseva

Chicago – August 18, 2014

Agenda Motives History and status Design Problems Benchmark Future Work

Agenda

I Motives

I History and status

I Design

I Problems

I Preliminary benchmark results

I Future work

Chicago – August 18, 2014 vNUMA in Xen 2 / 19

Agenda Motives History and status Design Problems Benchmark Future Work

Motives

I Cross NUMA node memory access is expensiveI Need to avoid cross node memory access

I Xen is NUMA awareI NUMA aware schedulingI NUMA aware guest memory placement

I Operating system like Linux is NUMA awareI NUMA aware schedulingI NUMA aware memory allocation / migration

I The missing bitsI Memory layout informationI CPU topology

Chicago – August 18, 2014 vNUMA in Xen 3 / 19

Agenda Motives History and status Design Problems Benchmark Future Work

History and status

I PV vNUMA presented in Xen Summit 2010 by Dulloor Raohttp://slidesha.re/1AXsFbu

I HVM vNUMA patches posted by Andre Przywara circa 2010

I Elena Ufimtseva has been working on upstreamable PVvNUMA since 2013

Chicago – August 18, 2014 vNUMA in Xen 4 / 19

Agenda Motives History and status Design Problems Benchmark Future Work

Design: PV and PVH

I Toolstack puts enlightenment information in hypervisor

I Guest memory allocation in accordance with enlightenmentinformation

I Guest retrieves enlightenment information via hypercall duringboot up

Chicago – August 18, 2014 vNUMA in Xen 5 / 19

Agenda Motives History and status Design Problems Benchmark Future Work

Design: PV and PVH

Enlightenment information structure

struct vnuma_info

{

nr_vnodes;

vdistance[nr_vnodes * nr_vnodes];

vcpu_to_vnode[nr_vnodes];

vnode_to_pnode[nr_vnodes];

vmemrange[nr_vnodes];

}

Chicago – August 18, 2014 vNUMA in Xen 6 / 19

Agenda Motives History and status Design Problems Benchmark Future Work

Design: HVM

I Toolstack puts enlightenment information in hypervisor

I Toolstack arranges ACPI tables

I Guest memory allocation in accordance with enlightenmentinformation

I Guest retrieves layout information via ACPI tables during bootup

Chicago – August 18, 2014 vNUMA in Xen 7 / 19

Agenda Motives History and status Design Problems Benchmark Future Work

Problems: vNUMA and other features

PV PVH HVMBallooning Y Y* NPoD N/A ? N

Chicago – August 18, 2014 vNUMA in Xen 8 / 19

Agenda Motives History and status Design Problems Benchmark Future Work

Problems: CPU topology

Chicago – August 18, 2014 vNUMA in Xen 9 / 19

Agenda Motives History and status Design Problems Benchmark Future Work

Benchmark

I HostI 2 sockets, 12 PCPUs, HT disabledI 36GB RAM, 2 NUMA nodesI NUMA balancing enabled

I GuestI 12 VCPUsI 16GB RAM, 2 virtual NUMA nodesI vnodes mapped to different pnodes, vcpu pinned to pnodeI NUMA balancing enabled

I Benchmarks to runI AutonumaI SPECJBBI STREAM

Chicago – August 18, 2014 vNUMA in Xen 10 / 19

Agenda Motives History and status Design Problems Benchmark Future Work

Benchmark: Autonuma

Chicago – August 18, 2014 vNUMA in Xen 11 / 19

Agenda Motives History and status Design Problems Benchmark Future Work

Benchmark: Autonuma

Chicago – August 18, 2014 vNUMA in Xen 12 / 19

Agenda Motives History and status Design Problems Benchmark Future Work

Benchmark: Autonuma

Chicago – August 18, 2014 vNUMA in Xen 13 / 19

Agenda Motives History and status Design Problems Benchmark Future Work

Benchmark: SPECJBB

Chicago – August 18, 2014 vNUMA in Xen 14 / 19

Agenda Motives History and status Design Problems Benchmark Future Work

Benchmark: SPECJBB

Chicago – August 18, 2014 vNUMA in Xen 15 / 19

Agenda Motives History and status Design Problems Benchmark Future Work

Benchmark: STREAM

Chicago – August 18, 2014 vNUMA in Xen 16 / 19

Agenda Motives History and status Design Problems Benchmark Future Work

Benchmark: Conclusion

I vNUMA improves performance for PV guest

I vNUMA has weird result in SPECJBB for PVH guest, butother two benchmarks have good results

Chicago – August 18, 2014 vNUMA in Xen 17 / 19

Agenda Motives History and status Design Problems Benchmark Future Work

Future Work

I basic vNUMA support for all guest types

I Dom0 vNUMA

I address vNUMA compatibility issues with PoD and ballooningfor HVM guest

I address performance issue for PVH

Chicago – August 18, 2014 vNUMA in Xen 18 / 19

Thank you!

Chicago – August 18, 2014 vNUMA in Xen 19 / 19