BNV : Enabling Scalable Experimentation through Bare-metal ... · BNV provides High-fidelity network experimentation/cloud platform. BNV can scale a network to support arbitrary topologies

BNV : Enabling Scalable Experimentationthrough Bare-metal Network Virtualization

Pravein Govindan Kannan, Ahmad Soltani, Mun Choon Chan Ee-Chien Chang

How to validate ideas over multiple topologies?

Optimizing Topologies

Congestion Control

Resource Allocation

No Single Topology is suitable for all use cases

Forwarding

Forwarding Forwarding

Forwarding

SDN Controller

App1

App2

App n

OpenFlow

Software Tools : Flexible

Mininet, Maxinet: Elegant tools enabling functional testing/ learning.

Container-based virtualization on a single machine or cluster of machines.

Cannot achieve the peak line-rate for performance testing.

Network Testbeds – High Fidelity

DeterLab, CloudLab, GENI, Staging Platforms:

Can provide high fidelity emulation with Bare-Metal provisioning.

Can provide only a fixed topology or subset for Software Defined Networks.

Change in Topology might require a long time (sometimes in order of weeks).

How do we achieve the flexibility of software and fidelity of hardware (bare-metal)?

Topology Scaling : One-to-Many Switch Virtualization

Virtual Topology

Switch1 Switch2 Switch3

Network Hypervisor

OpenFlow

Tenant 2

ONOS Controller

Tenant 1

OpenDaylightController

Tenant Mapping

Tenant 3

POX Controller

FatTree, JellyFish, HyperCube, HyperX, Xpander, Arbitrary

topologies…etc

Tenant n

…

Any SDN Controller

Virtual Topology


BNV

OpenFlow

Flow Translation

Topology Abstraction

Tenant 2

ONOS Controller

Tenant 1


BNV Mapper

Tenant Mapping

Tenant 3

POX Controller


topologies…etc

Tenant n

…

Any SDN Controller

Flow Translation


Idea : Loopback links in substrate topology



Software-Configurable Loopbacks

Topology + Loopback link

5 51 2 3 4 1 2 3 4

H1 H2 H5 H6

pSwitch1 pSwitch2CLink1

LoopLink1 LoopLink2

10G

Backbone link

5 51 2 3 4 1 2 3 4

H1 H2 H5 H6


V1 V312

3 23

1

V2

2 31

vS1

vS2

vS3

Physical Topology

Virtual Topology

vL1

vL2vL31G1G

1G

10G

LoopLink1 LoopLink2

5 51 2 3 4 1 2 3 4

H1 H2 H5 H6

pSwitch1 pSwitch2

V1 V312

3 23

1

V2

2 31

vS1

vS2

vS3

Physical Topology

Virtual Topology

CLink1

vL1

vL2vL21G1G

1G

10G

LoopLink1 LoopLink2

5 51 2 3 4 1 2 3 4

H1 H2 H5 H6

pSwitch1 pSwitch2

V1 V312

3 23

1

V2

2 31

vS1

vS2

vS3

Physical Topology

Virtual Topology

CLink1

vL1 -> CLink1

vL2vL31G1G

1G

10G

LoopLink1 LoopLink2

5 51 2 3 4 1 2 3 4

H1 H2 H5 H6

pSwitch1 pSwitch2

V1 V312

3 23

1

V2

2 31

vS1

vS2

vS3

Physical Topology

Virtual Topology

CLink1

vL1 -> CLink1

vL2vL31G1G

1G

10G

LoopLink1 LoopLink2

51 2 1 2 3 4

H1 H2 H5 H6

pSwitch1 pSwitch2

V1 V31 3 23

1

V2

31

vS1

vS2

vS3

Physical Topology

Virtual Topology

LoopLink1 LoopLink2

5

vL1 -> CLink1

vL2 -> CLink1vL3-> LoopLink11G1G

1G

10GCLink1

3 4

2

2

51 2 1 2 3 4

H1 H2 H5 H6

pSwitch1 pSwitch2

V1 V31 3 23

1

V2

31

vS1

vS2

vS3

Physical Topology

Virtual Topology

LoopLink1 LoopLink2

5

vL1 -> CLink1

vL2 -> CLink1vL3-> LoopLink11G1G

1G

10GCLink1

3 4

2

2

Final Mapping

5 51 2 3 4 1 2 3 4

H1 H2 H5 H6


V1 V312

3 23

1

Tag 3

vL1 : Tag 1

Tag 2

V2

2 31

vS1

vS2

vS3

Physical Topology

Virtual Topology

Link 1 using Packet Tagging

PKT

PKT

PKT

1PKT

LoopLink1 LoopLink2

5 51 2 3 4 1 2 3 4

H1 H2 H5 H6


V1 V312

3 23

1

Tag 3

vL1: Tag 1

vL2: Tag 2

V2

2 31

vS1

vS2

vS3

Physical Topology

Virtual Topology

Link 2PKT

PKT

2PKTPKT

LoopLink1 LoopLink2

5 51 2 3 4 1 2 3 4

H1 H2 H5 H6


H1 H512

3 23

1

vL3: Tag 3

VL1 : Tag 1

H2

2 31

vS1

vS2

vS3

Physical Topology

Virtual Topology

Link3 :Loop Link

PKT

PKT

vL2: Tag 2

LoopLink1 LoopLink2

Consistency & Fidelity of mapping.

How to achieve buffer isolation between multiple virtual switches ?

Burst Traffic

S1 S2 S3

A single virtual switch can fill up the entire shared Buffer !!

Shared Buffer

How to achieve buffer isolation between multiple virtual switches ?

Burst Traffic

S1 S2 S3

Shared Buffer

Metering burst (burst-rate < Approx Buffer Size)

How to achieve link isolation between multiple virtual links?

pSwitch1

S1 S2

pSwitch2

S3 S4

Tenant 1 Tenant 2

Metering Rate Metering Rate

Virtual Topology


BNV

OpenFlow

Flow Translation


Tenant 2

ONOS Controller

Tenant 1


BNV Mapper

Tenant Mapping

Tenant 3

POX Controller


topologies…etc

Tenant n

…

Any SDN Controller

BNV Mapper

BNV Mapper

11

BNVMapper

Request

Mapping1 1

1 1 10

BNV Mapper

• A Virtual Topology Consists of :• A set of hosts V,

• A set of (virtual) switches S, (feature: TCAM capacity, meter capacity, etc)

• A set of (virtual) links L (feature : Bandwidth)

H1 H3 H5 H7H2 H4 H6 H8

Host Link

Core Link

• Solved using ILP

• Variables :

BNV Mapper

𝑿𝒊,𝒗 = ቊ𝟏 ; 𝒊𝒇 𝒊 𝒊𝒔 𝒎𝒂𝒑𝒑𝒆𝒅 𝒕𝒐 𝒗𝟎 ; 𝒐𝒕𝒉𝒆𝒓𝒘𝒊𝒔𝒆

𝒀𝒋,𝒘 = ቊ𝟏 ; 𝒊𝒇 𝒋 𝒊𝒔 𝒎𝒂𝒑𝒑𝒆𝒅 𝒕𝒐 𝒘𝟎 ; 𝒐𝒕𝒉𝒆𝒓𝒘𝒊𝒔𝒆

Core Link::

Host link:

BNV Mapper

Objective is to Minimize the usage of Core-links:

a) Maximizes isolation to the tenant, keeping links dedicated.

b) Provides scope for dynamic expansions. (Adding/upgrading tenants)

𝐦𝒊𝒏 ∶ 𝒊 ∈ 𝑳𝒄𝒗 ∈ 𝒀

𝒙𝒊𝒗𝑏𝒊 + 𝒎∈ 𝑺𝒑,𝒒 ∈𝑹

𝑴𝒎𝒑𝒒

Inter-Switch Links

Big-Switch abstraction

More details in paper!

Implementation

Implemented on top of

Integrated with a Bare-metal provisioning service (Testbed).

Deployed on National Cybersecurity Lab(NCL) Testbed.

Support OpenFlow 1.0/ 1.3

Implementation

Dynamic Loopback Wiring using L2 intermediate Switch

Static loopback wiring 24-ports of SDN Switch

Evaluation

1) faithfully replicate behaviour of topologies?

2) Provide control/data plane isolation to multiple tenants?

3) Scale of complex topologies embedded?

Can BNV :

Evaluation : loopbacks & Big Switch Abstraction


Star (Single Switch) Star (Multi Switch)

Clos (Multi Switch) Clos (Single Switch)


Mapping Complex Topologies

Evaluation : Fidelity of Complex Topologies

Single-Rooted Tree Fat Tree

Jellyfish Star



Jellyfish Star

Single-Rooted TreeSingle-Rooted TreeSingle-Rooted TreeSingle-Rooted TreeSingle-Rooted TreeSingle-Rooted TreeSingle-Rooted TreeSingle-Rooted TreeSingle-Rooted TreeSingle-Rooted TreeSingle-Rooted TreeSingle-Rooted Tree

Fat TreeFat TreeFat TreeFat TreeFat TreeFat TreeFat TreeFat TreeFat TreeFat TreeFat TreeFat Tree

JellyfishJellyfish

JellyfishJellyfish

JellyfishJellyfish

JellyfishJellyfish

JellyfishJellyfish

Jellyfish

Jellyfish

StarStar

StarStar

StarStar

StarStar

StarStar

Star

Star



Jellyfish Star



Jellyfish Star


Fat Tree Jellyfish

28.02 26.32

Jellyfish 7-8% Lower

Low Intra-pod localityHeavy cross rack traffic!!


Fat Tree Jellyfish

19.0617.647

High Intra-pod localityLow cross rack traffic!!

Jellyfish 8% Higher

Take-away: BNV emulates the intrinsic characteristics of the topologies, otherwise observed with heavy wiring & building costs

This observation is backed by Tale of Two Topologies [SIGCOMM ’17]

Evaluation : Isolation of multiple tenants

Fat Tree

Jellyfish

Random (Rest of the nodes)

Take-away: BNV can support multiple arbitrary tenants providing isolated & repeatable environments.

Evaluation : Scale of Topologies



Take-away: Addition of loopback links increases the scalability significantly.Can emulate topologies with 130 switches and 300 links with just 5 ToR switches

Internet Topology Zoo dataset

Summary

BNV provides High-fidelity network experimentation/cloud platform.

BNV can scale a network to support arbitrary topologies using the existing network.

Can emulate networks of more than 100 network devices with just 5 ToR Switches.

Deployed in NCL testbed https://ncl.sg

Code Available : https://github.com/praveingk/bnvirt

https://ncl.sg/

Documents

BNV : Enabling Scalable Experimentation through Bare-metal ... · BNV provides High-fidelity network experimentation/cloud platform. BNV can scale a network to support arbitrary topologies