Upload
others
View
7
Download
0
Embed Size (px)
Citation preview
KVM Enhancements for OPNFV
Jun Nakajima
Intel Corporation
Contributors: https://wiki.opnfv.org/nfv-kvm
Project: NFV Hypervisors-KVM
1. Minimal Interrupt latency variation for data plane VNFs
2. Inter-VM Communication
3. Fast Live Migration
11/12/2015 KVM Enhancements for NFV
https://wiki.opnfv.org/nfv-kvm
2
Enhancements for NFV Hypervisor
1
1. Exclusive allocation of whole CPU cores to VMs
2
3
3. Inter-VM Communication (direct-memory mapped)
4
2. Direct I/O (e.g. SR-IOV)
4. vSwitch implementation as a high performance VM
General public and enterprise cloud Hypervisor Architecture
NFV Hypervisor Architecture
From ETSI“Network Functions Virtualization (NFV); Infrastructure;Hypervisor Domain”
3
KVM Hypervisor
Hardware
System Configuration
Testing and Perf Tools
NIC
VM
core core
Virtual Switch (vSwitch)
VM
VM
VM
KVM kernel module
2. Inter-VM Communication
1. Minimal Latency Variation
Configuration Information (BIOS, boot,
kernel)Latency Variation
Perf. Tools
Inter-VM Config. Information
Inter-VM Comm. Perf. Tools
Live Migration Perf. Tools
Live Migration Config.
Information3. Fast Live Migration
Scope of the Project
11/12/2015 KVM Enhancements for NFV 4
Minimal Interrupt Latency Variation
11/12/2015 KVM Enhancements for NFV 5
Goals of Minimal Interrupt Latency Variation
• Timer latency => AVG 5 us, MAX 15 us
• Interrupt latency => MAX 20 us (Future target: 5 us)
• Maximum guest vCPU pre-emption period => 10 us (Future target: 2 us)
• Cumulative pre-emption within 1 ms window => X < 20 us (Future target: X < 4 us)
https://etherpad.opnfv.org/p/nfv-kvm
11/12/2015 KVM Enhancements for NFV 6
User-Level
Linux Kernel
HardwareNIC
core
KVM Modules
User-Level
core
NIC
core core
Real-time Guest
Summary Of Solutions
PREEMPT-RT Kernel
Intr_HandlerExcusive/Static Allocation
Soft “Partitioning”, CPU Binding, Huge Pages
SoftwareReal-Time Linux, Code
inspection, testing/measurements
Hardware TechnologiesCache Allocation
Technology, Advanced VT features
11/12/2015 KVM Enhancements for NFV 7
Update on Enhancements: Cyclictest (Initial)
Latency (μs)0
100000
200000
300000
400000
500000
600000
700000
800000
900000
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58Baseline
w/ RT Extensions
Histogram (Total: 1M Samples)
RT Linux Guest on KVM
Latency(μs)11/12/2015 KVM Enhancements for NFV 8
Update on Enhancements: Cyclictest (After)
• Cyclic Test in Guest: Latency (in μs )
– Min: 7– Avg: 9– Max: 16
RTLinuxGuestonKVM
Latency(μs)
Host:LinuxwithRTpatches
000007 000003000008 11757562000009 67812652000010 159222000011 069100000012 011004000013 000379000014 000207000015 000049000016 000005
Latency
99.69%(Total#:79,810,183)
Occurrences
0
10000000
20000000
30000000
40000000
50000000
60000000
70000000
80000000
0 2 4 6 8 10 12 14 16 18 20 22
Histogram
11/12/2015 KVM Enhancements for NFV 9
Update on Enhancements: Cyclictest (Latest)
• Cyclic Test in Guest: Latency (in μs )
– Min: 6 7– Avg: 6 9– Max: 11 (9*) 16
• Host
– 4.1.0 + PREEMPT-RT (upstream)
11/12/2015 KVM Enhancements for NFV 10
*: w/ 1Hz tick disabled
Linux Kernel
Hardware
KVM Modules
Intr_Hander
PCIe Device
Interrupt (MSI)
Device Driver
Periodic (1ms)
Update on Enhancements:Latency from Periodic External Interrupts
• Latency from periodic external interrupt:
– Time delta from interrupt occurrence to invocation of interrupt handler in guest (unit: in μs)
Min: 3.98Avg: 4.42Max: 9.10
11/12/2015 KVM Enhancements for NFV 11
Inter-VM Communication
11/12/2015 KVM Enhancements for NFV 12
Goals of Inter-VM Communication
• Add fast-paths in VMs as optimized inter-VM communication
• Maintain consistent flow table entries in VMs
• Enable protected access to the destination VM or shared memory
– Open the Window when needed– Close it immediately when done
11/12/2015 KVM Enhancements for NFV 13
VM1
virtio-net
In-VM Switch
VM2
TX
RX
virtio-net
Simple Flow Table
NIC
In-VM Switch
VM3
TX
RX
virtio-net
In-VM Switch
Inter-VM data transfer
API Libraries
ProcessProcess
Summary Of Solutions
Flow Table
Entries
Shared Memory
If (In-VM Switch is present)
vSwitch(e.g.OVS-dpdk,SnabbSwitch)
11/12/2015 KVM Enhancements for NFV 14
Update on Enhancements:Upstream Project
• Proposal from Michael S. Tsirkin
– Inter-VM Communication for two VMs– Receiver allows one to access its own memory via existing IOMMU
protection mechanism• Our plan
– Extend Michael’s proposal to support multiple VMs– Work with him for implementation
11/12/2015 KVM Enhancements for NFV
https://wiki.opnfv.org/vm2vm_mst
15
Update on Enhancements:PoC
• Transfer 64B packets from virtio-net to another VM (fast path)
– Memory Copy– VMFUNC– VMFUNC with
Trampoline Code• 65Mpps with 32-
packet batching*:
11/12/2015 KVM Enhancements for NFV *Intel internal estimation 16
Fast Live Migration
11/12/2015 KVM Enhancements for NFV 17
18
Goals of Fast Live Migration
• Migration Time– < 2 sec.
• Down Time– Hypervisor downtime for IO intensive
workload: 10 ms (future 3ms)– Hypervisor downtime for memory
intensive workload: < 500 ms– Network downtime: 25 ms
• Configuration– Typical VM size: 8GB to 16GB– Network throughput: 5Gbps for
64bytes packets
• SR-IOV Migration– Support NIC with kernel Land driver– Support NIC with DPDK driver
11/12/2015 KVM Enhancements for NFV
19
Summary Of Solutions
• Shorten VM Downtime- Clean up operation after completion of
data transmission
- Minimize cpu_synchronize_all_states() calls
• Shorten Total Migration Time- Optimize handling of unused or zero
pages
- Use CPU instructions to optimize zero pages checking
- Use hardware accelerator to compress data before transmitting
11/12/2015 KVM Enhancements for NFV
Update on Enhancements
11/12/2015 KVM Enhancements for NFV
Host CPU: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz RAM: 64GOS: RHEL 7.1, Kernel: 4.2-rc6 QEMU:2.4.0Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+
Test environment
0
500
1000
1500
2000
Idle guest DPDK FW workload
Totalmigrationtime(ms)
Original
Afteroptimization
0
10
20
30
40
50
Idle guest DPDK FW workload
VMdowntime(ms)
Original
Afteroptimization
Performance data
20
Host
Guest
Host
Guest
10Gbps for LM
10GbE used by pktgen
DPDK L2 FW APP
Eth1 Eth0
pktgen
DPDK L2 FW APP
Eth0 Eth1
Eth1 Eth2Eth0
vitbr0vitbr1
Eth1 Eth2 Eth0
vitbr0 vitbr1
Eth1
Q & AJoin us!
11/12/2015 KVM Enhancements for NFV 21