Overview of Distributed Virtual Router (DVR) in Openstack/Neutron

Vivekanandan Narasimhan

Architectural Overview of Distributed Virtual Routers in

OpenStack Neutron

Paris OpenStack® Summit

Agenda Introduction

High level architecture and DVR Configuration

East-West Routing OVS Rules North-South Routing

API Changes and DB Extensions

Scheduling

Services

Plans for Kilo

Architectural Overview of Distributed Virtual Routers in OpenStack Neutron Paris OpenStack® Summit

Legacy Routing in Neutron

Network node provides:• IP forwarding

– Inter-subnet (east-west) traffic between VMs– Floating IP (north-south) traffic between

external and VM– Default SNAT (north-south) traffic from VM to

external• Metadata Agent

– access to Nova metadata service

Issues:• Performance bottleneck• Scalability limitations• Single Point of Failure

Architectural Overview of Distributed Virtual Routers in OpenStack Neutron

Compute node

Network node

Compute node

VM1 VM2

VM1

VM2

VirtualPhysical

Router A

Rtr A


Distributed Routing in Neutron

Compute nodes provide:• IP forwarding for local VMs

– Inter-subnet (east-west) traffic between VMs– Floating IP (north-south) traffic between external

and VM• Metadata Agent for local VMs

– access to Nova metadata service

Advantages:• Bypass network node improves performance• Scales with size of compute farm• Limited failure domain (per compute node)Limitations:• Default SNAT function is still centralized


Compute node

Network node

Compute node

VM1 VM2

VM1

VM2

VirtualPhysical

Router A

Rtr ADefaul

tSNAT

Rtr A’’

Rtr A’


High level requirements for DVR

• Provider feature– Tenants should not have to care

• Default router type set by global config knob– Distributed or Legacy

• Distributed and Legacy routers coexist in same cloud• Minimize overhead use of public IP addresses– one public IP per compute node

• Leverage existing code base• Ability to migrate from Legacy to Distributed


DVR High Level Architecture and

Configuration


Network/Service Node(s)

Compute Node(s)

VMVM

VM

l3-agentovs-agent

Controller Node

Neutron controller/plugin

L3 Plugin

Network/Service Node(s)

l3-agentovs-agent

dhcp-agent

qr

qrqr

agent_mode=dvr

router_distributed=True

enable_distributed_routing=Trueagent_mode=dvr_snat

enable_distributed_routing=True

• One FIP namespace supports multiple routers

• Legacy OpenStack Routers (non-distributed) can exist on the network/service node

• Enhanced l3-agent works in different modes “dvr_snat”, “legacy” and “dvr”.

FIP

SNAT


metadata

metadata

qrqr

DVR High level Architecture


Configuring DVR


Plugin Configuration ( Default Router Option)“neutron.conf”router_distributed= True

L3 Agent Configuration and Modes

“l3_agent.ini”agent_mode=dvr_snat ( Network/Service Node)agent_mode=dvr (Compute Node only)agent_mode=legacy ( Network/Service Node)router_namespaces=True ( Enable namespace cleanup)(Optional)

L2/OVS Agent Configuration

“ml2_conf.ini”[ovs]

enable_tunneling = Truelocal_ip = <Local Data Network IP (or) TEP IP> [agents]tunnel_types = vxlanl2_population=Trueenable_distributed_routing = True


DVR Configuration Overview


br-int br-intbr-int

br-tun br-tun br-tunbr-ex

qr

VM1 VM2 VM3 VM4DHCP Agent

L3 Agent

OVS Agent

Q_DVR_MODE=legacy

External Network

Internal Network

Network Node Compute Node Compute Node

OVS Agent

Legacy Deployment


br-int br-intbr-int

br-tun br-tun br-tunbr-ex

qr

VM1 VM2 VM3 VM4DHCP Agent

L3 Agent

OVS Agent

Q_DVR_MODE=dvr_snat Q_DVR_MODE=dvr Q_DVR_MODE=dvr

L3Agent

L3Agent

qr qrqr

snat

Network/Service Node Compute Node Compute Node

DVR Deployment without FIP

br-ex br-ex


br-int br-int

br-tun br-tun

VM1 VM2 VM3 VM4

OVS Agent


L3Agent

L3Agent

qr qr

Enable

FIPEnable

FIP

fip fip

br-ex br-ex

DVR Deployment with FIP

br-int

br-tunbr-ex

qr

DHCP Agent

L3 Agent

qr

snat

Network/Service Node Compute Node Compute Node


East-West Routing


L3 Agent

Rules

------

------

Rules

------

------

vm1 vm2

br-int

br-tun

qr

LMAC

East-West routing is accomplished through a combination of the following in each CN:•qr : a namespace that forwards traffic among routed subnets using route table entries.

•LMAC: a MAC address unique to each CN for use as source address in forwarding routed frames from one CN to another.

•A set of rules in the OVS bridges that prevent frames with src=gateway MAC from egressed tunnel bridge or the IR. These rules also cause peer bridges to substitute/restore gateway MACs with LMACs while routed frames traverse through br-tuns and underlay network.

br-ex

Compute Node entities for enabling the DVR East West

RoutingL3 Agent


vm1

br-int-cn1

br-tun-cn1

qr

vm2

br-int-cn2

br-tun-cn2

qr

Tenant has two VMs - vm1 that is in RED Net & vm2 that is in GREEN NetCN1 and CN2 are two compute nodes. qr represents a dvr owned by Tenant 1

Data Network

CN1 CN2

PING REQUEST from vm1 to vm2

vm1 ipqr red macvm1 mac vm2 ip

vm1 ipqr red mac vm2 ip

vm1 ipvm2 macqr grn mac vm2 ip

vm1 ipvm2 macdvr cn1 mac vm2 ip

vm1 ipvm2 macdvr cn1 mac vm2 ip

1

2

3

4

5

grn vni

grn 2 vlan

vm1 ipvm2 macqr grn mac vm2 ip6

vm1 mac

Flow of DVR routed packets (example)


Table 0: Dispatch from in_portfrom patch-int:goto table 1

default:drop

Table 1: DVR process TableIf packet is from dvr-router-intf-mac, replace SRCMAC with DVR-node-unique-mac and forward to table 2Any other packet, forward as is to table 2

Table 2: Local VLAN to TUN TableIf unicast frame, forward to table 20If broadcast/multicast frame, forward to table 21

Table 20: Unicast to Tunnel Table

Pre-populated L2-pop rules that bind dst-mac in frame to a single output tunnel

Frame out tosingle tunnel

Table 0: Local Switching Table from patch-tun:Is src-mac == DVR-Unique-MAC - goto table 1

default:forward packet to ports - NORMAL action

1

2to dvrrouter

Integration Bridge (br-int)

Tunnel Bridge (br-tun)4

5

6

3from dvr router

7

8


Egress to Cloud


Table 0: Dispatch from in_portfrom vxlan-port: forward to table 4

default: drop

Table 4: VXLAN Tun to Local Vlan Tablematch tun_id: Set local vlan-id, forward to table 9

default: drop

Table 9: DVR Learning blockerIf src_mac == DVR-Unique-MAC frame, don’t learn, just output to patch-int

default:forward to table 10

4

Integration Bridge (br-int)

Tunnel Bridge (br-tun)

1

2

3

Ingress from Cloud

Table 0: Local Switching Table from patch-tun:Is dvr_routed - goto table 1

default:forward packet to ports - NORMAL action

Table 1: DVR to LMAC Table match DVR-Unique-MAC: strip off dvr-unique-mac, add local dvr-router-intf-mac, output direct to vm-port

default:drop

5

vm1


6

North-South Routing


Overview of North-South Routing

External Network

vm1 vm3

br-int

br-tun

vm5

br-ex

• IP Rules

•Routing Entries

• Iptable based NATing rules

• /32 addresses for the floating

IPs

• One Floating IP Namespace

per CN

• Interface on external network

• Proxy ARPs enabled for hosted

floating IPs

qr

FIP

L3 Agent

Data Network



Default SNAT Traffic flow


API Changes and DB Extensions



API ExtensionsAdds ‘distributed’ attributed to ‘router’ object

Router Create:

neutron router-create –distributed=True/False

Router Update:

neutron router-update –distributed=True/False

Can be set by admin user through the above APIGlobal default is set as “router_distributed” in neutron.confThe attribute is only visible to admin tenant in GET.

NOTE: Migrating or Converting a Legacy Router to Distributed is Work in Progress.



DB Changes for DVR

ml2_dvr_port_bindings


Scheduling

Achitectural Overview of Distributed Virtual Routers in OpenStack Neutron Paris OpenStack® Summit

br-int

br-tun

VM3 VM4

OVS Agent

Q_DVR_MODE=dvr_snat Q_DVR_MODE=dvr

L3Agent

qr

br-ex

Scheduling a “qr/snat” in Service/Compute Node

br-int

br-tunbr-ex

qr

DHCP Agent

L3 Agent

qr

snat

Schedule Events

Create a Router

Add one or more subnets with VMs

Set a default Gateway for the Router

Compute NodeService/Network Node


br-int

br-tun

VM3 VM4

OVS Agent

Q_DVR_MODE=dvr_snat Q_DVR_MODE=dvr

L3Agent

qr

Enable

FIPEnable

FIP

fip

br-ex

br-int

br-tunbr-ex

qr

DHCP Agent

L3 Agent

qr

snat

Network/Service Node Compute Node

Scheduling a “fip” in Compute Node

Schedule Events

Make sure you have a router configured with External Network or gateway.

Create a Floating IP

Associate a Floating IP to a VM


Un-Scheduling Routers and FIP Un Schedule Events

FIP Namespace cleanup

Last VM holding the FIP determines the FIP namespace cleanup Delete a FloatingIP Disassociate a Floating IP

Router Namespace cleanup

When no more DVR related ports ( Includes Compute, VIP, DHCP etc.,) are serviced by the routed subnet, the router namespaces are cleaned up.

Both l2 agent and l3 agent takes part in the clean up.

SNAT Namespace cleanup When a gateway is removed from a router the snat namespace will be

cleaned up.

NOTE: Namespace cleanup should be enabled “router_namespaces=True” in l3_agent.ini


Services


Services support with DVR

Services Support in DVR as of Juno

LBaaS

FWaaS• North-South only ( External Traffic)• East-West ( Internal Traffic)

MetadataService

VPNaaS is still supported with Centralized Routers.

VPNaaS support for DVR is Work in Progress.


br-int br-int

br-tun br-tun

VM1 VM2 VM3 VM4

OVS Agent


L3Agent

L3Agent

qr qr

fip fip

br-ex br-ex

Firewall Firewall

LBaaS AgentVIP

br-int

br-tunbr-ex

qr

DHCP Agent

L3 Agent

qr

snat

Network/Service Node

Firewall


Compute Node Compute Node

Firewall

Services deployment with DVR

Metadata

Metadata



DVR Enhancements in kilo

VPNaaS support for DVR Full migration support for DVR routers. HA for Service Node VLAN Support



Liberty Plans for DVR

L3 Agent Refactor Distributed DHCP Performance tuning. Distributed SNAT


Questions?


Technology

Overview of Distributed Virtual Router (DVR) in Openstack/Neutron