Embed Size (px)
Citation preview
Lars Kurth Community Manger, Xen Project
Chairman, Xen Project Advisory Board
Lead CentOS Virtualization SIG
Director, Open Source Business Office, Citrix lars_kurth
Released on January 15, 2015 (10 months of development)
Resources:Blog: bit.do/xen-4-5-blogDocs: bit.do/xen-4-5-docsDownload: bit.do/xen-4-5-download
Stats:Changesets: 1812 KLOC Added: 81KLOC Removed: 141 (mostly removal of XM)
Contributors:102 individuals Employers: 39 (93 individuals working for them)
0
5
10
15
20
25
30
35
40
45
50
2010 2011 2012 2013 2014 2015
4.0 4.1 4.2 4.3 4.4 4.5
Xen 4.x Number of New Major Features
0
50
100
150
200
250
2010 2011 2012 2013 2014
Developers
Employers
Using GitDM over Git logs using our database of developers and organizations to remove duplicates
across all sub-projects
Reasons for faster Innovation:
More developers and orgs
Fewer forked up-streams (e.g. Linux, BSDs, QEMU, …)
Architecture clean-up(e.g. XM – XL)
Better Development Process
with a twist
HWCPUsMemoryI/O
VM1
Guest OS
Applications
VM0 (or Dom0)
Dom0 Kernel
Drivers
VM2 VMn
Applications
Guest OS
Applications
Guest OS
Toolstack
Scheduler MMU Timers InterruptsConfig
back PV front
Xen 4.5: XEND / XM has been removed
XL now the default interface into Xen
Resources:Docs: bit.do/xen-xlComparison: bit.do/xen-4-5-xm-2-xl-compareMigration Guide: bit.do/xen-4-5-xm-2-xl
Libvirt integration has been vastly improved
Resources:Docs: bit.do/xen-libvirtComplete List: bit.do/xen-4-5-blog
Dom0
Dom0 Kernel
Drivers
Toolstack(s)
LIBXENLIGHT
XL LIBVIRT
XEND
XM
Xen via Libvirt in Openstack:
Great Platform for Production DeploymentsGet into Quality Group A in 2015
Great Platform for DevelopmentGreat DevStack support
Libvirt:
Better Quality, Stability & UsabilityDrivers: OpenStack, CentOS Virt SIG – learning what a distro needs
Resources:Docs: bit.do/xen-openstackPlans: bit.do/xen-openstack-fosdem15Install Video: https://vimeo.com/119572029
XenServer
(XAPI)ESX Hyper-V
Group B
NOVA
LIBVIRT
KVM
Group A
Xen
Group C
Group A
Number 1 priority for the projectVendor funded Test Infrastructure More capacity & coverage Automated performance testing
Vendor funded OpenStack CI loop
Xen Project Rack
OverviewXen 4.5: Real-Time Deferrable Server SchedulerWhat is next?
Resources:
Docs: bit.do/xen-schedulers
HWCPUsMemoryI/O
Dom0
Dom0 Kernel
Drivers
The Xen Project Hypervisor supports several
different schedulers with different properties.
Different schedulers can be assigned to…… an entire host
e.g. Credit2 Scheduler
HWCPUsMemoryI/O
Dom0
Dom0 Kernel
Drivers
The Xen Project Hypervisor supports several
different schedulers with different properties.
Different schedulers can be assigned to…… an entire host
… a pool of physical CPU’s (=CPU Pool) on a host (VMs need to be assigned to a pool or pinned to a CPU)
e.g. RTDS Scheduler e.g. Credit Scheduler
HWCPUsMemoryI/O
Dom0
Dom0 Kernel
Drivers
VM1
Guest OS
RT App
VMn
Guest OS
RT App
e.g. RTDS Scheduler
VMn+1
Apps
Guest OS
VMm
Apps
Guest OS
e.g. Credit Scheduler
Scheduler parameters can be modified per …
Host
CPU Pool
VM
Soft Real-time CPU scheduler (experimental)
Guarantees CPU capacity to guest VMs on SMP hostsBudget: Amount of time assigned to a VMPeriod: Time period in which depleted budgets are replenished
Global:Allow VCPU Migration across CPUs
Partitioned: Pin VCPU to a physical CPUSchedule VMs per CPU
More flexibility & best utilization
Migration Overhead & Cache Penalty
May underutilize CPU
Lower overheads & lower latency
Embedded & automotive
Latency sensitive workloads
Guaranteed QoSCloud based gaming, video,
TV delivery, …
Guaranteed QoS(Price SLAs QoS)
Scheduler Use-cases Xen 4.5 Plans for 4.6+
Credit General Purpose Supported
Default
Supported
Optional
Credit 2 General Purpose
Optimized for lower latency, higher VM density
Experimental Supported
Default
RTDS Soft & Firm Real-time
Multicore
Embedded, Automotive, Graphics & Gaming in
the Cloud, Low Latency Workloads
Experimental Hardening
Optimization
Better XL support
<1μs granularity
Supported
ARINC 653 Hard Real-time
Single core
Avionics, Drones, Medical
Supported
Compile time
No change
Legend:
likely in 4.6
possible in 4.6
Overview
Jan 2015: Intel GVT-g (XenGT) UpdatesWhat is next?
Resources:
News: bit.do/xengt-jan15Docs: bit.do/xengt-jan15-docs
Watch the demo at
https://www.youtube.com/
watch?v=V2i8HCcAnY8
Virtual GPU per VM
Performance critical resources
directly assigned to VM
XenGT support is currently out-of-tree
Q4-2014 refresh by Intel: In use by XenClient 5.5
First patches have been posted for review on xen-devel
Requires some Linux and QEMU patches alsoMotivation: create a common code base for Xen & KVM
Likely complete for Xen 4.6 (or shortly afterwards)Will initially be experimental
Virtualization ModesPVH Dom 0
Other Performance improvements
What is next?
Shortcut Mode With
HVM / Fully Virtualized HVM
HVM + PV drivers HVM PV Drivers
PVHVM HVM PVHVM Drivers
PVH PV pvh=1
PV PV
Old
New
Win
do
ws
Lin
ux, B
SD
s, …
3.0
4.0
4.4/4.5
Xen
Shortcut Mode With
HVM / Fully Virtualized HVM
HVM + PV drivers HVM PV Drivers
PVHVM HVM PVHVM Drivers
PVH PV pvh=1
PV PV
Poor Performance
Scope for Improvement
Optimal Performance
VS VS VS VH
P VS VS VH
P P VS VH
P P P VH
P P P P
P = Paravirtualized
VS = Software Virtualized (QEMU)
VH = Hardware Virtualized
Win
do
ws
Lin
ux, B
SD
s, …
PVH PV P P P VH
PV PV P P P P
ARM PV P P P VH
Simplicity: Less code & fewer Interfaces in Linux/FreeBSD
– Security : smaller TCB and attack surface, fewer possible exploits
– Clean-up : possibility to simplify Linux kernel and reduce maintenance burden
Better Performance & Lower Latency
– Dom0 must be a PV guest
– 64 bit: VM’s run in ring 0 instead of ring 3(fewer expensive TLB flushes)
This is the most complex part
of Xen today!
Feature Complete
Hardware support for AMD x86 chipsAdd support for PCI passthroughMigration of PVH Dom U’s (including systems with PVH Dom 0)
Hardening & Tuning
Add PVH to test suite and make test failures blockingBenchmarking and performance testsCode clean-up
x86
HPET: Better and faster resolution valuesParallel memory scrubbing on boot (large machines)Lower interrupt latency for PCI passthrough (machines > 2 sockets)Soft affinity for non-NUMA machinesMultiple IO-REQ services for guests (remove bottlenecks for HVM guests by allowing multiple QEMU back-ends)
Intel
SandyBridge: VT-d posted interrupts for PVHVM (I/O intensive workloads)
Vulnerabilities published in 2014Evolution of Xen Security Features
Xen 4.5 : Virtual Machine IntrospectionA new Model for Cloud Security
What is next?
Escalation Linux Container KVM + QEMU Xen (PV)
Xen (HVM+Stub)
Privilege
Escalation(guest to host)
7 – 9 3 – 5 0
Denial of Service(by guest of host) 12 5 – 7 3Information Leak(from host to guest) 1 0 1
Assumptionsx86 vulnerabilities from guest to host that hosting/cloud providers worry about
Xen (HVM) without stub domains has slightly more than Xen (PV) due to use of QEMU, less than KVM + QEMU
Have the underlying analysis (but won’t cover it in the talk)
2007 2008 2009 2010 201520142011 2012 2013
Stub Domains : QEMU in separate domains
Flask / Xen Security Modules (Xen’s version of SE Linux)
vTPM (Virtual Trusted Module)
Driver Domains (Network, Disk, … drivers in a separate VM)
TODAY: Mainly used by security apps (XenClient,
Qubes OS, …), Forensic, Military & Embedded
TODAY: In general use
(but has trade-offs at cloud scale)
XenAccess / XenProbes VM Introspection (via LibVMI)
Major
Upgrades
2007 2008 2009 2010 201520142011 2012 2013
XenAccess / XenProbes VM Introspection (via LibVMI)
Exposed lots of existing Xen functionality in LibVMI
Hypervisor can bring paged out guest memory
Mem_access-emulate(-with-no-write)
Many more patches currently under review for Xen 4.6
Watch the demo at
https://www.youtube.com/watc
h?v=ZJPHfpDiN4o
Credit: Tamas K Lengyel
VM3
Guest OS
App
VMn
Guest OS
App
VM2
Guest OS
App
Dom0
Dom0 Kernel
Drivers Agent(s) Agent(s) Agent(s)
Installed in-guest agents, e.g. anti-virus software,
VM disk & memory scanner, network monitor, etc.
Anti virus storm, deployment/maintenance, …
Several
VM3 VMnVM2Dom0
Dom0 Kernel
Drivers
VM3
Guest OS
App
VMn
Guest OS
App
VM2
Guest OS
App
Security
Appliance
VM1
IntrospectionEngine
Protected area
Agent Agent Agent
Hybrid approach: no need to move
everything outside (chose best trade-off)
XSM/Flask
Major re-work of Virtual Machine IntrospectionOptimization, Code cleanup/future-proofing Support for ARM CPUsIntel #VE support
Turn on Xen Security Modules on by default and include in test suiteDisabled today and not automatically tested
Specialist Use General Use!
Reduce TCBQEMU secure mode for HVM without stub domainsMove the instruction emulator into non-privilege modeMove the Xen compatibility layer into a lower privilege ring
Binary Live Patching for the Xen HypervisorDepends on which solution the kernel will standardize on (kpatch / kGraft / ftrace-based)
We want to share tooling
IntroductionXen 4.5: Remus & COLOWhat’s Next?
Resources:Remus: bit.do/xen-remusCOLO: bit.do/xen-coloHA: bit.do/xen-ha
Remus: Non-stop Service Replication
Continually live migrates a copy of a running VM to a backup serverAutomatically activates if the primary server fails
Expensive in terms of overheads and hardware requirements
COLO: A different approach (building on top of Remus)
Relaxes requirement of backup server/VM being an exact replica
If backup server generates the same response to input we are able to fail over without service stop
Eliminates overheads, reduces hardware requirements
Remus
Some “loose ends”, e.g. one fix for PV guests not in upstream kernel
Better tools integration and control (“xl remus” instead of “remus”)
Optimizations for COLO
COLO
Out-of-tree
Integrates with Remus via “xl remus” –works with Xen 4.5
Some known issues
Fix “loose ends”
Include into Xen Hypervisor code base
Switch block replication from blktap2 to qdisk (motivation: performance & alignment)
Hardening
ARM AdditionsWhat is next?
X86
Larger VMs Up to 1TB of guest RAM
Lower virtualization overheadSuper page mappings and faster interrupt EOIs (no maintenance interrupts)
Improved Interrupt handlingSupport for priorities and irq migration (virtual and physical)
Near feature parity with x86Boot via UEFI firmwareQEMU PV backends (disk, console, keyboard, mouse, framebuffer)
Many new IP blocks, firmware interfaces and platforms are supportedE.g. AMD Seattle 64-bit server SoC – see bit.do/xen-4-5-docs
HardeningInclusion of 64 Bit Hardware into test infrastructure
VM Save/Restore and Live MigrationNote: Remus and COLO are architecture independent
PCI PassthroughNote: passthrough of MMIO regions works in 4.6
ACPI and UEFI support for guests
More IP blocks, …Support for more Hardware
Determine the usage of cache by VMs running Monitors the L3 cache (LLC in most server platforms)
$ xl psr-cmt-attach vm-id
$ xl psr-cmt-show cache_occupancy
Identify noisy neighbor VMs and take corrective actionE.g. Migrate VM to a different hostE.g. CPU pinning, CPU pools, schedulers
What’s Next?Intel Cache Allocation TechnologyLonger term: schedulers can use HW utilization information
Release and Roadmap Process
Vinovyn @ Flickr
Release Manager: Wei Liu
Proposal: Tweaked Release Process for Xen 4.6lists.xenproject.org/archives/html/xen-devel/2015-02/msg01214.html
Development start: 6 Jan 2015 Feature freeze: 10 Jul 2015 Release date: 9 Oct 2015 (could release earlier)
Master branch on xen.git
Feature Development
Feature
Freeze
point
Wait period
to clear test pushgate
RC’s
Release
Announcement
RELEASE-4.5.0 branch on xen.git
Master branch on xen.git
Feature Development RC’s
This is when patches for the ongoing release
need to be submitted for review
Wait period
to clear test pushgate
No new features will be accepted, unless there is a Freeze Exception
Bug fixes are allowed, with approval by Maintainers/Release Manager
Release Manager declares that only bug fixes deemed
blockers can be accepted
Release Manager:
Sends first
Xen x.y Development Update
email on xen-devel@
Deferred features from previous
release, Timetable, etc.
Release Manager:
Sends Monthly
Xen x.y Development Update
email on xen-devel@
Release Manager:
RC Announcements, Test Days
Release Manager:
RC Announcement
Contributors:
Expected to reply if they are working on a feature that is not
on the list of tracked features
Expected to provide Status updates on features & bugs on the list
Not engaging with the process may lead to removal or downgrading
Contributors:
Expected to reply if they are working on a feature that is not
on the list of tracked features and tracked bugs
Same as above: can also ask for Freeze Exceptions
Contributors:
Expected to provide Status updates on tracked bugs on the list
Other interesting planned Features
Other sub-projects
Embedded & AutomotiveSound, graphics, and other drivers for Linux and other OS’esLots of other enablers: e.g. security featuresCertification
VMWare Tools supportRun VMWare images unmodified in Xen
More: First 4.6 Development Update lists.xenproject.org/archives/html/xen-devel/2015-02/msg01816.html
Mirage OS
Safer and cleaner TLS stackopenmirage.org/blog/announcing-bitcoin-pinata
Irmin: Git-like distributed, branchable storage
Jitsu: a DNS server that spawns unikernels in response to DNS requests
IPv6, Tooling, etc.
VMn
Language run-time
Application
Cubieboard2 serving
2048 game @ FOSDEM’15
Slides on www.slideshare.net/xen_com_mgr/
