PCI Pass-through - FreeBSD VM on Hyper-V

Dexuan Cui (dexuan@freebsd.org)

Yanmin Qiao (sephe@freebsd.org )

Kylie Liang (kyliel@Microsoft.com)

Microsoft Enterprise Open Source Group

Agenda

• FreeBSD VM on Hyper-V Architecture

• PCI Pass-through

• Demo with FreeBSD VM on Hyper-V

• Next Step

• PCI Pass-through in Azure

Running a Guest OS on Hyper-V

HaveIntegrationServices?

Yes Yes

No No

Emulated Devices• No Hyper-V integration• Limited manageability• Limited functionality• OK CPU, weak disk/net

Integration Services for FreeBSD - High Level Architecture

Windows Server Virtual Machine Management

Services VMMS) (

Virtual Infrastructure Driver VID) (

Virtualization Service Provider (VSP)

VMBusDrivers

Hyper-v

Hardware

Windows Kernel

VMBus

FreeBSD Kernel

ifnetCAM

StorVSC NetVSC

FreeBSD VM

Virtual Machine Worker Process

(VMWP)

Utilities

Daemon User mode

Kernel mode

Changes to FreeBSD

• Driver source code under sys/dev/hyperv/ • include/hyperv.h

• netvsc

• storvsc

• utilities

• vmbus

• Daemon• http://svnweb.freebsd.org/base/head/contrib/hyperv/tools/

Discrete Device Assignment

• Discrete Device Assignment (also known as PCI Passthrough) available as part of the Hyper-V role in Microsoft Windows Server 2016.

• Performance enhancement that allows a specific physical PCIedevice to be directly controlled by a guest VM running on the Hyper-V instance. • GPU

• Network adapter

• NVMe device

System Requirements - Host

• Processor• Intel's Extended Page Table (EPT)

• or AMD's Nested Page Table (NPT).

• Chipset: • Interrupt remapping - Intel's VT-d with the Interrupt Remapping capability (VT-d2) or

any version of AMD I/O Memory Management Unit (I/O MMU).

• DMA remapping - Intel's VT-d with Queued Invalidations or any AMD I/O MMU.

• Access control services (ACS) on PCI Express root ports.

• The firmware tables must expose the I/O MMU to Hyper-V.• Note: This feature might be turned off in the UEFI or BIOS. For instructions, see the

hardware documentation or contact your hardware manufacturer.

System Requirements - Device

• Some devices in your computer don’t mark, or tag, their traffic in way that individually identifies the device, making it impossible for the I/O MMU to redirect this traffic to the memory owned by a specific VM. These devices, mostly older PCI-style logic, can’t be assigned to a guest VM.

• A survey script available to find potential DDA devices on your hosts• Check machine/firmware (iommu: for intel it’s VT-d), host os (WS 2016),

devices (PCIe + no Port IO BAR + MSI/MSI-X + ACS), etc• https://github.com/Microsoft/Virtualization-

Documentation/tree/master/hyperv-samples/benarm-powershell/DDA

Guest on Hyper-V vs. XEN & KVM

• Xen (HVM)/KVM• Full emulation of PCI/PCIe bus

• XEN(PV guest)/Hyper-V• Partial emulation of PCI/PCIe bus + PV hotplug message

Generation 1 vs. 2 VM on Hyper-VGeneration 1 Device Generation 2 Replacement Generation 2 Enhancements

IDE controller Virtual SCSI controllerBoot from .vhdx (64 TB maximum size, and online resize capability)

IDE CD-ROM Virtual SCSI CD-ROM Support for up to 64 SCSI DVD devices per SCSI controller.

Legacy BIOS UEFI firmware Secure Boot

Legacy network adapter Synthetic network adapter Network boot with IPv4 and IPv6

Floppy controller and DMA controller No floppy controller support

Universal asynchronous receiver/transmitter (UART) for COM ports

Optional UART for debugging Faster and more reliable

i8042 keyboard controller Software-based inputUses fewer resources because there is no emulation. Also reduces the attack surface from the guest operating system.

PS/2 keyboard Software-based keyboardUses fewer resources because there is no emulation. Also reduces the attack surface from the guest operating system.

PS/2 mouse Software-based mouseUses fewer resources because there is no emulation. Also reduces the attack surface from the guest operating system.

S3 video Software-based videoUses fewer resources because there is no emulation. Also reduces the attack surface from the guest operating system.

PCI bus No longer required

Programmable interrupt controller (PIC) No longer required

Programmable interval timer (PIT) No longer required

Super I/O device No longer required

Guest and FreeBSD

sys/dev/hyperv/pcib/pcib.c - In code review phase

The patch implements a PCI bridge driver to support PCI pass-through.

• Talks to the host to discover device(s) and presents the device(s) to FreeBSD's pci driver via PCI configuration space

• Allocates resources for the device(s) and initialize the related BARs, when the device driver's attach method is invoked.

• Talks to the host to create MSI/MSI-X interrupt remapping between the guest and the host.

• Supports device hot add/remove.

DemoDismount the device from the hostHot AddConfigure and test the performanceHot RemoveMount the device to the host

Next Step

• Debug: hypervisor’s scheduling can affect the performance greatly

• SR-IOV• Allocates a portion of a NIC to the virtual machine for improved latency and

throughput.

• VF driver in FreeBSD

• Integrate it to Virtual Appliances with ISV vendors

Hybrid

Hyper-scale

Open & flexible

Applications

Clients

Infrastructure

Management

Databases &

Middleware

App Frameworks

& Tools

DevOps

PaaS &

DevOps

Enterprise

proven

AzureOpen & flexible

>80% of Fortune 500 use the Microsoft Cloud

PCI Pass-through in Azure – GPU Access

• Used on N-Series Azure VMs to give the guests access to GPUs.

• VM Size: N-Series instances are enabled with NVIDIA’s cutting-edge GPUs so you can run GPU-accelerated workloads and visualize them.

• Windows and Linux VMs

• Start from South Central US region

• In public preview now.

• Want to learn more – http://gpu.azure.com/ .

PCI Pass-through in Azure – Accelerated Networking• Accelerated Networking enables Single Root I/O Virtualization (SR-IOV)

to a virtual machine (VM), greatly improving its networking performance.

• VM Size: Standard_D15_v2 and Standard_DS15_v2

• Windows Server 2012 R2 and Windows Server 2016 Technical Preview 5. Linux and Windows Server 2012 support will be added soon.

• West Central US, West Europe Azure regions

• Start from South Central US region

• In preview now.

BSD Based Virtual Appliance in Azure Marketplace

FreeBSD 10.3 is available in Azure Marketplace

June: Global AzureSep: Azure China

BSD Meetup in China – First Time

Meet other local geeks and discuss the BSD including FreeBSD, NetBSD, OpenBSD, MacOS X, DragonFlyBSD and any other BSD derived systems. If you want to know about BSD running in the cloud or how BSD is used as networking and security product, you definitely should join us!

BSD的开发者，爱好者们，无论您是在做FreeBSD，NetBSD，OpenBSD，MacOS X, DragonFlyBSD或任何BSD衍生产品的开发，或者您听过并想了解BSD和UNIX，让我们聚集一堂吧！又或者您关心云计算，网络和安全，这样的聚会也必定让您受益匪！

http://www.huodongxing.com/go/bsd_sh 11/21 @ Shanghaihttp://www.huodongxing.com/go/bsd_bj 12/5 @ Beijing

Thank You