Sponsored by the National Science Foundation
Building and testing SDN/NFV
applications in a real network
Niky Riga, GENI Project [email protected],
Sponsored by the National Science Foundation 2GRW ASU– March 2016 www.geni.net
• An Introduction to SDN, NFV
• OpenFlow, the 1.0 API and why 1.3 is better
• Build simple SDN and NFV apps
Intro to SDN and NFV
• Debugging techniques for OpenFlowapplications
Extra
Slides
Sponsored by the National Science Foundation 3GRW ASU– March 2016 www.geni.net
• An Introduction to SDN, NFV
Sponsored by the National Science Foundation 4GRW ASU– March 2016 www.geni.net
“The current Internet is at an impasse
because new architecture cannot be
deployed or even adequately evaluated” [PST04]
[PST04]: Overcoming the Internet Impasse through Virtualization, Larry Peterson, Scott Shenker, Jonothan Turner Hotnets 2004
Sponsored by the National Science Foundation 5GRW ASU– March 2016 www.geni.net
SDN basic idea
Configurable Network Device
Brain lives
in device
Sponsored by the National Science Foundation 6GRW ASU– March 2016 www.geni.net
SDN basic idea
Open
API
Controller
Brain lives
anywhereProgrammable Network Device
Sponsored by the National Science Foundation 7GRW ASU– March 2016 www.geni.net
SDN basic idea
MATCH ACTION
dst subnet X output port 48
dst subnet Y output port 47
dst MAC: 00:00:00:00:00:01 output port 2
dst MAC: 00:00:00:00:00:01 output port 5
Forwarding Table
Entries added
by internal logic
Arbitrary
entries by
controller
src subnet Z drop
TCP port 80 output port 10
Sponsored by the National Science Foundation 8GRW ASU– March 2016 www.geni.net
SDN basic idea
Controller
One controller to program many devices
Sponsored by the National Science Foundation 9GRW ASU– March 2016 www.geni.net
Moving the control outside
Not bound to standard algorithms
and protocols
Optimize based on specific
configuration
Dynamic control
Centralized interactions
Abstracts everything to functions
Silicon did not become smarter or
more capable
Non standardized interactions
between functions
Performance, stability
Debugging
Sponsored by the National Science Foundation 10GRW ASU– March 2016 www.geni.net
Abstracts everything to functions …
NAT
firewall
DHCP
DNS
switch
VPN
router
gateway
proxy
access
point
Program the device to be anything …
software
within device’s capabilities
Sponsored by the National Science Foundation 11GRW ASU– March 2016 www.geni.net
NFV: Network Function Virtualization
Slide from: http://docbox.etsi.org/Workshop/2013/201304_FNTWORKSHOP/S07_NFV/BT_REID.pdf
Sponsored by the National Science Foundation 12GRW ASU– March 2016 www.geni.net
SDI: Software Defined Infrastructure
Computation
Storage
Network
Program all resources together to support
any network function
Sponsored by the National Science Foundation 13GRW ASU– March 2016 www.geni.net
Looking ahead
Physical infrastructure
User defined virtual networks with
compute, storage, networking
Modified picture from http://www.slideshare.net/LarryCover/virtualizing-the-network-to-enable-a-software-defined-infrastructure-sdi?related=1
Orchestration Layer (e.g. ONOS)
Everything is virtualized
Highly optimized networks
Dynamic reconfigurations
Network snapshoting
Network engineering ~
~ Software engineering
Sponsored by the National Science Foundation 14GRW ASU– March 2016 www.geni.net
Reality
Performance not there yet
Hybrid environments
Hardware Vendor support is inconsistent
Standards are still in flux
Sponsored by the National Science Foundation 15GRW ASU– March 2016 www.geni.net
• OpenFlow, the 1.0 API and why 1.3 is better
Sponsored by the National Science Foundation 16GRW ASU– March 2016 www.geni.net
OpenFlow an SDN API
Control how packets are forwarded
Implementable on COTS hardware
Exports device’s functionality
Early stages, highly IP based
Good start to start building SDNs
Open
API
Controller
Sponsored by the National Science Foundation 17GRW ASU– March 2016 www.geni.net
OpenFlow versions
(Dec ’09) OpenFlow 1.0.0
Simple & widely supported
(Feb ‘11) OpenFlow 1.1.0
Not implemented by HW vendors(Dec ‘11) OpenFlow 1.2
First ONF standard
(‘12/’13) OpenFlow 1.3.x
Complex &
support in progress
(Oct ‘13) OpenFlow 1.4
(‘11) Open Networking
Foundation (ONF)
formed to shepherd
standards
(Nov‘13) OpenFlow 1.0.2
(Dec’ 14) OpenFlow 1.5
https://www.opennetworking.org/sdn-resources/technical-library
Sponsored by the National Science Foundation 18GRW ASU– March 2016 www.geni.net
OpenFlow
Switch
Data Path (Hardware)
Control Path OpenFlow
Any Host
OpenFlow Controller
OpenFlow Protocol (SSL/TCP)
Modified slide from : http://www.deutsche-telekom-laboratories.de/~robert/GENI-Experimenters-Workshop.ppt
• The controller is responsible for populating forwarding table of the switch
• In a table miss the switch asks the controller
Sponsored by the National Science Foundation 19GRW ASU– March 2016 www.geni.net
OpenFlow in action
Switch
Data Path (Hardware)
Control Path OpenFlow
Any HostOpenFlow Controller
OpenFlow Protocol (SSL/TCP)
Modified slide from : http://www.deutsche-telekom-laboratories.de/~robert/GENI-Experimenters-Workshop.ppt
Host1 sends a packet
If there are no rules about handling this packet
Forward packet to the controller
installs a rule on the forwarding table (flow table)
Subsequent packets do not go through the controller
host1 host2
Sponsored by the National Science Foundation 20GRW ASU– March 2016 www.geni.net
OpenFlow Basics (1.0)
SwitchPort
MACsrc
MACdst
Ethtype
VLANID
IPSrc
IPDst
IPToS
TCPsport
TCPdport
Rule Action Stats
1. Forward packet to port(s)2. Encapsulate and forward to controller3. Drop packet4. Send to normal processing pipeline5. Modify Fields
+ mask what fields to match
Packet + byte counters
slide from : http://www.deutsche-telekom-laboratories.de/~robert/GENI-Experimenters-Workshop.ppt
IPProt
VLANPCP
Sponsored by the National Science Foundation 21GRW ASU– March 2016 www.geni.net
Use Flow Mods
• Going through the controller on every packet is inefficient
• Installing Flows either proactively or reactively is the right thing to do
• A Flow Mod consists of :
– A match on any of the 12 supported fields
– A rule about what to do matched packets
– Timeouts about the rules:
• Hard timeouts
• Idle timeouts
– The packet id in reactive controllers
– Priority of the rule
Sponsored by the National Science Foundation 22GRW ASU– March 2016 www.geni.net
OpenFlow datapaths
Switch
Data Path (Hardware)
Control Path OpenFlow
Any Host
OpenFlow Controller
OpenFlow Protocol
Different OpenFlow modes
– switches in pure OF mode are actingas one datapath
– Hybrid VLAN switches are onedatapath per VLAN
– Hybrid port switches are twodatapaths (one OF and one non-OF)
OpenFlow enabled devices are usually referred to as
datapaths with a unique dpid
Each Datapath can point to only one controller at a time!
It is not necessary that 1 physical device corresponds to 1 dpid
Sponsored by the National Science Foundation 23GRW ASU– March 2016 www.geni.net
Running OpenFlow Experiments
Debugging OpenFlow experiments is hard: – Network configuration debugging requires coordination
– Many networking elements in play
– No console access to the switch
Before deploying your OpenFlow experiment test your
controller.
http://mininet.github.com/http://openvswitch.org/
Sponsored by the National Science Foundation 24GRW ASU– March 2016 www.geni.net
OpenFlow controllers
• Open source controller frameworks
– NoX – C++
– PoX - Python
– OpenDaylight - Java
– FloodLight - Java
– Trema – C / Ruby
– Maestro - Java
– Ryu - Python
• Production controllers
– Mostly customized solutions based on Open Source frameworks
– ProgrammableFlow - NEC
Sponsored by the National Science Foundation 25GRW ASU– March 2016 www.geni.net
• OpenFlow 1.0
+ What you know and love!
• OpenFlow 1.1
+ Multiple tables and group tables
+ Some more matches and actions
• OpenFlow 1.2
+ The OpenFlow Extensible Match (OXM)
• OpenFlow 1.3
+ Meters
+ Table features
Evolution of the OpenFlow Protocol
Sponsored by the National Science Foundation 26GRW ASU– March 2016 www.geni.net
• OpenFlow 1.4– Bundles
– Flow table synchronization
– Flow monitoring
• OpenFlow 1.5– More fine-grained matches and actions
– Egress tables
– Packet type aware pipeline & pipeline registers
– Group/meter table improvements
• …But we struggle to keep up…
Evolution of the OpenFlow Protocol
Sponsored by the National Science Foundation 27GRW ASU– March 2016 www.geni.net
• OF 1.0 primary complaint = too rigid
• OF 1.3 gains Greater match and action support
Instructions add flexibility and capability
Groups facilitate advanced actions
Meters provide advanced counters
Per-table features
Custom table-miss behavior
…and more!
Why OpenFlow 1.3
Sponsored by the National Science Foundation 28GRW ASU– March 2016 www.geni.net
• Build simple SDN and NFV apps
Sponsored by the National Science Foundation 29GRW ASU– March 2016 www.geni.net
OpenFlow 1.0 Intro Exercise
Write simple controllers to control the traffic between the three hosts.
Sponsored by the National Science Foundation 30GRW ASU– March 2016 www.geni.net
OpenFlow 1.3 Exercise
Problem: A single path from point
A to point B leaves a single point of failure in any topology. Upon link failure, end-to-end communication is impossible.
Solution: Use redundant links to
allow end-to-end connections to persist in the event of a link failure.
Sponsored by the National Science Foundation 31GRW ASU– March 2016 www.geni.net
NFV Exercises
Same box as NAT or Firewall
Sponsored by the National Science Foundation
BACKUP SLIDES
www.geni.net
Sponsored by the National Science Foundation 33GRW ASU– March 2016 www.geni.net
• Debugging techniques for OpenFlowapplications
Sponsored by the National Science Foundation 34GRW ASU– March 2016 www.geni.net
Debugging OpenFlow Experiments
• Have two planes to debug!
– Dataplane (MTU, mac learning, etc …)
– Control plane
• Standard networking tools
– tcpdump (dataplane but mostly control plane)
– Wireshark
– dump of the flowtable of the switch
• Discussion on techniques
Sponsored by the National Science Foundation 35GRW ASU– March 2016 www.geni.net
OpenFlow common PitFalls
• Reactive controllers
– Cause additional latency on some packets
– UDP – many packets queued to your controller by time flow is set up
• Performance in hardware switches
– Not all actions are supported in hardware
• No STP to prevent broadcast storms
• Controller is responsible for all traffic, not just your
application!
– ARPs, DHCP, LLDP
Sponsored by the National Science Foundation 36GRW ASU– March 2016 www.geni.net
Remember, You’re In Control of Everything
• LLDP
– Most controllers will handle for you
– Ensure user flows do not conflict
• MTU
– Use of tunnels b/t resources requires lower MTU
• ARP
– Must account for ARP in IPv4 networks
– Proxy ARP for routing b/t OpenFlow subnets
• IP assignment
– Need to permit DHCP packets, or…
– Need to statically assign IPs
Sponsored by the National Science Foundation 37GRW ASU– March 2016 www.geni.net
OpenFlow/non-OpenFlow Hybrid Networks
• GENI experimenter OpenFlow links are over non-OpenFlow infrastructure
– Stitched links might traverse the same physical non-OpenFlow switches but on different VLANs
– Non-OpenFlow switch might drop packets or shut down a VLAN if loop suspected or MAC learning algorithm confused
• VLAN translation
– OpenFlow controller can implement via flows
– Can also be done with a non-OpenFlow switch
– Controller must know how to handle both cases
Sponsored by the National Science Foundation 38GRW ASU– March 2016 www.geni.net
Switch OpenFlow Implementation
• Most OpenFlow specifications are optional
• Many hardware vendors implement a subset of these
– Most matches supported
– Reduced action support
– Limited OpenFlow 1.1+ advanced feature support
Software
OpenFlow
Switches
Hardware
OpenFlow
Switches
Sponsored by the National Science Foundation 39GRW ASU– March 2016 www.geni.net
• Hardware OpenFlow switches
– Are fast
– Have often overlooked limitations
• OpenFlow action/match support
• ARP handling (e.g. Dell S4810 ARP replies must go to controller)
– Are sometimes tricky to configure
• Manually carve flow tables from TCAM (default size 0 in some cases)
• Background MAC learning
• LLDP and VLANs
• Software OpenFlow switches
– Support nearly all OpenFlow operations
– Do not install easily on all flavors of Linux
Switch OpenFlow Implementation
Sponsored by the National Science Foundation 40GRW ASU– March 2016 www.geni.net
• Network switches have physical port
numbers
– e.g. 1, 2, …, 47, 48
• OpenFlow (OF) instances on a physical
OF switch partition the switch into many
logical OF switches (datapaths)
– OF instance exposed to controller as
unique data path ID
– Physical ports set/configured for each OF
instance
OF
Instance 1
DPID=X
OF
Instance 2
DPID=Y
OF
Instance 3
DPID=Z
Controller Controller
Switch Ports and Physical Interfaces
Sponsored by the National Science Foundation 41GRW ASU– March 2016 www.geni.net
OF
Instance 1
DPID=X
OF
Instance 2
DPID=Y
OF
Instance 3
DPID=Z
Controller Controller
• OF instance appears as
the actual, physical
partition of the physical
switch
– Port numbers exposed to
controller as physical ports
themselves
– e.g. 17, 24, 30, 332417 30 33
17,24,30,33
Switch Ports and Physical Interfaces
Sponsored by the National Science Foundation 42GRW ASU– March 2016 www.geni.net
OF
Instance 1
DPID=X
OF
Instance 2
DPID=Y
OF
Instance 3
DPID=Z
Controller Controller
OF instance appears as an
independent switch
– Port numbers exposed to controller
as constant sequence of ports, e.g.
1, 2, 3, 4
– Physical ports mapped to virtual
ports
phys virt
17 <--> 1
24 <--> 2
30 <--> 3
33 <--> 4 2417 30 33
1, 2, 3, 4
Switch Ports and Physical Interfaces
Sponsored by the National Science Foundation 43GRW ASU– March 2016 www.geni.net
• Evening of August 26th Morning of August 27th
– Experiment over AL2S experiment working
• Symptoms
– Port disappeared on Clemson Dell S4810 OpenFlow instance
– All frames arrive in Clemson from Utah
– Utah to Clemson drops
• Ping w/payload over 18 bytes
• TCP packets, even ACKs, with DF (don’t fragment)
• UDP and TCP w/o DF flag truncated by 192 bytes, regardless of payload size
– No errors or drop reports on any switches
• 2 weeks of troubleshooting later, culprit was
– Cisco ASR 100G module at Utah
• Overnight reload of module solved problem
The Problem Is Not Always Your SDN
Sponsored by the National Science Foundation 44GRW ASU– March 2016 www.geni.net
• GENI
– [email protected] : community announcements
– http://groups.geni.net/geni/wiki/GENIExperimenterWelcome
• OVS
• Controllers
Useful links