PCI Passthrough with Red Hat

OpenStack Platform

Ken Savich

Sr OpenStack Solution Architect, Red Hat2016-12-6

About Me

- Sr OpenStack Solution Architect

- with Red Hat since January 2016

- previously engineering at Rackspace - 9 years

- I love ramen.

пройти через

Ken Savich

Agenda

- Background

- The story

- Technical details

- Scripting

- Results

- Applications

Ken Savich

Red Hat OpenStack Platform 9

- 100% Community based

- Enterprise proven

- Automated updates and upgrades

- Co-engineered performance

ADD NAME (View > Master > Slide master)

Background

Ken Savich

Why PCI Passthrough?

- Performance

- Exclusive access

- No hypervisor support

- Domain isolation

- Security

Ken Savich

device emulation

Ken Savich

device passthrough

ADD NAME (View > Master > Slide master)

The Story

Ken Savich

- Cisco- Nvidia- Petrel- Red Hat

ADD NAME (View > Master > Slide master)

ADD NAME (View > Master > Slide master)

Requirements

- Red Hat OpenStack

- Self-contained infrastructure

- Windows 7 guests w/Petrel software

- Native guest access to Nvidia Tesla m60 grid cards

- Performance!

ADD NAME (View > Master > Slide master)

Technical Details

ADD NAME (View > Master > Slide master)

Requirements

- RHEL 7.3 - kernel < 4.1

- Virtualization Technology - bios & kernel

- vmx (intel)

- svm (amd)

- Red Hat OpenStack Platform 8 or 9

- A PCI card with native drivers for your guest

$ egrep `(vmx|svm)` /proc/cpuinfo

ADD NAME (View > Master > Slide master)

Procedure

- Turn off hyper-threading- Find your pci device- Configure your Compute node(s)- Configure your Controller node(s)- Create a flavor- Launch and instance- Test test test!

ADD NAME (View > Master > Slide master)

PCI Passthrough - first steps

- Get your packages $ sudo yum install pciutils gcc kernel-devel kernel-debug kernel-headers -y

- blacklist any competing drivers/etc/default/grub: GRUB_CMDLINE_LINUX=... rd.driver.blacklist=nouveau

$ sudo echo "blacklist nouveau" > /etc/modprobe.d/nvidia-blacklist.conf

$ sudo echo "blacklist nvidia" >> /etc/modprobe.d/nvidia-blacklist.conf

- turn on IOMMU/etc/default/grub: GRUB_CMDLINE_LINUX=... intel_iommu=on

ADD NAME (View > Master > Slide master)

PCI Passthrough - now we’re feeling it

- Find your device$ sudo lspci |grep -i nvidia

05:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] (rev a1)

05:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1)

- Find the pci id of your device$ sudo lspci -n|grep 05:00.0|awk '{print $3}'

10de:13c2

- Tell grub the PCI stub ID/etc/default/grub: GRUB_CMDLINE_LINUX=... pci-stub.ids=10de:13c2

ADD NAME (View > Master > Slide master)

PCI Passthrough - bounce it

- Update grub$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg

- Configure kernel hugepages# mkdir /hugetlbfs

# echo "2048" > /proc/sys/vm/nr_hugepages

# echo "vm.nr_hugepages=2048" > /etc/modprobe.d/hugepages.conf

# echo "hugetlbfs /hugetlbfs hugetlbfs defaults 0 0" >> /etc/fstab

- REBOOT!!!

ADD NAME (View > Master > Slide master)

PCI Passthrough - Nova - Compute Node

- Find your vendor and product codes# lspci -nn|grep -P '^(?=.*nvidia)(?=.*vga)'

05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)

- Whitelist your device in nova.conf/etc/nova/nova.conf:

pci_passthrough_whitelist = [{"vendor_id": "10de", "product_id": "13c2"}]

- Restart nova service$ sudo systemctl restart openstack-nova-compute.service

ADD NAME (View > Master > Slide master)

PCI Passthrough - Nova - Controller

- Add the filter class to the scheduler/etc/nova/nova.conf:

scheduler_available_filters=nova.scheduler.filters.pci_passthrough_filter.PciPassthroughFilter

- Make sure PciPassthroughFilter is in the default list/etc/nova/nova.conf:

scheduler_default_filters=RamFilter,ComputeFilter,AvailabilityZoneFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,PciPassthroughFilter,NUMATopologyFilter,AggregateInstanceExtraSpecsFilter

ADD NAME (View > Master > Slide master)

PCI Passthrough - Nova - Controller

- Add an alias for your device/etc/nova/nova.conf

pci_alias = {"vendor_id":"10de", "product_id":"13c2", "name":"nvidia"}

- Restart nova services# systemctl restart openstack-nova-api.service openstack-nova-scheduler.service \

openstack-nova-conductor.service

ADD NAME (View > Master > Slide master)

Off to the races...

- Create a flavor$ nova flavor-key m1.large set "pci_passthrough:alias"="nvidia:2"

- Launch an instance$ nova boot --image rhel7 --key_name test --flavor m1.large mypci

- Install native drivers in your guest

ADD NAME (View > Master > Slide master)

Common Problems

- PCIe reset

- Virtio drivers

- Display virtualization

- Live migration

ADD NAME (View > Master > Slide master)

Divider SlideScripting

ADD NAME (View > Master > Slide master)

Scripting

- goo.gl/RsceFv

- Requires RHEL 7.x

- Nvidia specific

- Must be run on compute nodes

ADD NAME (View > Master > Slide master)

Per-node configuration with Tripelo

- Interfaces

OS::TripleO::ControllerExtraConfigPre: Controller node additional configuration

OS::TripleO::ComputeExtraConfigPre: Compute node additional configuration

- for examples, see:

/usr/share/openstack-tripleo-heat-templates/puppet/extraconfig/pre_deploy/per_node.yaml

ADD NAME (View > Master > Slide master)

Results

ADD NAME (View > Master > Slide master)

Benchmarking

- Windows 7 instance- Red Hat virtIO drivers

- Furmark = 60+ fps

ADD NAME (View > Master > Slide master)

Applications

ADD NAME (View > Master > Slide master)

Possibilities!

- HPC applications- Video encoding- Audio pre/post- NFV

- SR-IOV for VNF- Gaming

ADD NAME (View > Master > Slide master)

Try it for yourself, you’ll like it

- Get Red Hat OpenStack 9 here:- https://access.redhat.com/products/red-hat-openstack-platform/evaluation

● Provides a full-featured 60 day entitlement.● Provides self-service support.● Offers three 2-socket server subscriptions - enough to run a small 3-node test

cluster.● Is not intended for production use.

Let me know how it goes: ksavich@redhat.com

ADD NAME (View > Master > Slide master)

References

- Red Hat PCI Passthrough Documentation:

- http://goo.gl/Ea9KLp

- OpenStack PCI Passthrough Documentation:

- https://wiki.openstack.org/wiki/Pci_passthrough

ADD NAME (View > Master > Slide master)

Questions

ADD NAME (View > Master > Slide master)

Divider SlideThank you