Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
Scaling networks through softwareMarch 16th 2015 | João Taveira Araújo @jta
Scaling networks through software
network systems @ fastly
Scaling networks through software
Scaling networks through software
scalability
Scaling networks through software
constraints
constraints time
money people
Scaling networks through software
constraints
constraints time
money people
knowledge
Scaling networks through software
constraints
constraints time
money people
knowledgetechnology
Scaling networks through software
constraints
constraints time
money people
knowledgetechnologycomplexity
Scaling networks through software
constraints
constraints time
money people
knowledgetechnologycomplexity
Scaling networks through software
constraints time
money people
Scaling networks through software
constraints time
money people
Scaling networks through software
constraints time
money people
Scaling networks through software
Number of PoPs ……………………….… ~20 BGP announcements …………….……... ~2000 Requests per second .…………………… ~1000000
Becoming a multi terabit network
Scaling networks through software
Number of PoPs ……………………….… ~20 BGP announcements …………….……... ~2000 Requests per second .…………………… ~1000000
Network ops ……..….………………….… 2
Becoming a multi terabit network
Scaling networks through software
Number of PoPs ……………………….… ~20 BGP announcements …………….……... ~2000 Requests per second .…………………… ~1000000
Network ops ……..….………………….… 2
Becoming a multi terabit network
Network software ……..….…………….… me
Scaling networks through software
scalability
Scaling networks through software
scalabilityobservations on network
from a company that used to be a startup
i anything you don’t explicitly control is an implicit liability
Scaling networks through software
the internet
Scaling networks through software
router
A B C D
PoP
the internet
Scaling networks through software
router
A B C D
PoP How do you: ‣ load balance traffic ‣ gracefully failover if a server fails
the internet
Scaling networks through software
router
A B C D
PoPA
25%
25%
25%
25%
B
DC
DNS
Scaling networks through software
router
A B C D
PoPA
33%
33%
33%
0%
B
DC
DNS
Scaling networks through software
router
A B C D
PoPA
33%
33%
33%
0%
B
DC
DNS
Bad idea: ‣ gets hard to manage ‣ do one thing and do it well ‣ you don’t control TTL
Scaling networks through software
load balancer
A B C D
PoP
Scaling networks through software
load balancer
A B C D
PoP
StateC
ost
Scaling networks through software
load balancer
A B C D
PoP
StateC
ost
Exist
ing d
eman
d
StateC
ost
Scaling networks through software
load balancer
A B C D
PoP
StateC
ost
Exist
ing d
eman
d
Proje
cted
dem
and
StateC
ost
Exist
ing d
eman
d
StateC
ost
Scaling networks through software
load balancer
A B C D
PoP
StateC
ost
Exist
ing d
eman
d
Proje
cted
dem
and
StateC
ost
Exist
ing d
eman
d
StateC
ost
Scaling networks through software
load balancer
A B C D
PoPBad idea: ‣ you don’t control demand ‣ you don’t control DDOS
State
Cos
t
Exist
ing d
eman
d
Proje
cted
dem
and
State
Cos
t
Exist
ing d
eman
d
State
Cos
t
Scaling networks through software
ECMP
A B C D
PoP
Scaling networks through software
ECMP
A B C D
PoP
Destination network Next hop
10.0.0.0/24 A
10.0.0.0/24 B
10.0.0.0/24 C
10.0.0.0/24 D
Scaling networks through software
ECMP
A B C D
PoPECMP
A B C D
PoP
Destination network Next hop
10.0.0.0/24 A
10.0.0.0/24 B
10.0.0.0/24 C
10.0.0.0/24 D
Scaling networks through software
ECMP
A B C D
PoPECMP
A B C D
PoP
Destination network Next hop
10.0.0.0/24 B
10.0.0.0/24 C
10.0.0.0/24 D
Bad idea: ‣ connection resets ‣ you don’t control rehashing ‣ you don’t control vendor roadmaps
ii don’t resign to fate just because everything sucks
Scaling networks through software
faild
Scaling networks through software
ECMP
A B C D
PoP
Scaling networks through software
Destination network Next hop
10.0.0.0/24 10.1.A.1
10.0.0.0/24 10.1.A.2
10.0.0.0/24 10.1.A.3
… …
ECMP
A B C D
PoP
Scaling networks through software
Destination network Next hop
10.0.0.0/24 10.1.A.1
10.0.0.0/24 10.1.A.2
10.0.0.0/24 10.1.A.3
… …
IP Address MAC
10.1.A.1 A:A
10.1.A.2 A:A
10.1.A.3 A:A
… …
ECMP
A B C D
PoP
Scaling networks through software
Destination network Next hop
10.0.0.0/24 10.1.A.1
10.0.0.0/24 10.1.A.2
10.0.0.0/24 10.1.A.3
… …
IP Address MAC
10.1.A.1 A:A
10.1.A.2 A:A
10.1.A.3 A:A
… …
ECMP
A B C D
PoP
Scaling networks through software
drain a host
Scaling networks through software
Destination network Next hop
10.0.0.0/24 10.1.A.1
10.0.0.0/24 10.1.A.2
10.0.0.0/24 10.1.A.3
… …
ECMP
A B C D
PoP
Scaling networks through software
Destination network Next hop
10.0.0.0/24 10.1.A.1
10.0.0.0/24 10.1.A.2
10.0.0.0/24 10.1.A.3
… …
IP Address MAC
10.1.A.1 B:A
10.1.A.2 C:A
10.1.A.3 D:A
… …
ECMP
A B C D
PoP
Scaling networks through software
Destination network Next hop
10.0.0.0/24 10.1.A.1
10.0.0.0/24 10.1.A.2
10.0.0.0/24 10.1.A.3
… …
IP Address MAC
10.1.A.1 B:A
10.1.A.2 C:A
10.1.A.3 D:A
… …
ECMP
A B C D
PoP
Scaling networks through software
cut off to failed state
Scaling networks through software
Destination network Next hop
10.0.0.0/24 10.1.A.1
10.0.0.0/24 10.1.A.2
10.0.0.0/24 10.1.A.3
… …
ECMP
A B C D
PoP
Scaling networks through software
Destination network Next hop
10.0.0.0/24 10.1.A.1
10.0.0.0/24 10.1.A.2
10.0.0.0/24 10.1.A.3
… …
IP Address MAC
10.1.A.1 B:B
10.1.A.2 C:C
10.1.A.3 D:D
… …
ECMP
A B C D
PoP
Scaling networks through software
Time
Activ
e flo
ws
ecmp
Scaling networks through software
Time
Activ
e flo
ws
Time
Activ
e flo
ws
ecmp
Scaling networks through software
Time
Activ
e flo
ws
Stat
e
Time
load balancer
Scaling networks through software
Stat
e
Time
Time
Activ
e flo
ws
Activ
e flo
ws
Stat
e
Time
faild
Scaling networks through software
Stat
e
Time
Time
Activ
e flo
ws
Activ
e flo
ws
Stat
e
Time
faild
Scaling networks through software
Stat
e
Time
Time
Activ
e flo
ws
Activ
e flo
ws
Stat
e
Time
faild
iii if it’s expensive you probably don’t need it.
Scaling networks through software
F5 BIG-IP 10350v
Scaling networks through software
$200,000F5 BIG-IP 10350v
Scaling networks through software
$200,000$0
F5 BIG-IP 10350v
Scaling networks through software
load balancer
Scaling networks through software
load balancer balancing
Scaling networks through software
load balancer balancing(a load balancer is just an appliance which provides load balancing)
Scaling networks through software
load balancer balancing(a load balancer is just an appliance which provides load balancing)
distributed
Scaling networks through software
How to build a PoP ‣ buy a router ‣ get BGP table from each provider ‣ install routes to FIB ‣ servers use default gateway A B C D
PoP
Vendor BGP
FIB
router
Scaling networks through software
Juniper MX960 Router
Scaling networks through software
~$500,000
Juniper MX960 Router
Scaling networks through software
router
Scaling networks through software
router routing
Scaling networks through software
router routing(a router is just an appliance which provides routing)
Scaling networks through software
router routing(a router is just an appliance which provides routing)
distributed
Scaling networks through software
Arista DCS-7150S switch family
Scaling networks through software
$29,995 Arista DCS-7150S switch family
Scaling networks through software
PoP
Userspace BGP
FIB
switch
BGP daemon
FIB
server
How to build a Fastly PoP ‣ buy switches ‣ reflect BGP down to servers ‣ inject multipath routes into FIB
Scaling networks through software
PoP
Userspace BGP
FIB
switch
BGP daemon
FIB
server
How to build a Fastly PoP ‣ buy switches ‣ reflect BGP down to servers ‣ inject multipath routes into FIB
Scaling networks through software
PoPd
istribute
d syste
m
Userspace BGP
FIB
switch
BGP daemon
FIB
server
How packets egress Fastly ‣ switches emit nexthop IP and MAC ‣ servers configure p2p link / ARP ‣ send directly to provider nexthop!
Scaling networks through software
PoPd
istribute
d syste
m
Userspace BGP
FIB
switch
BGP daemon
FIB
server
How packets egress Fastly ‣ switches emit nexthop IP and MAC ‣ servers configure p2p link / ARP ‣ send directly to provider nexthop!
Scaling networks through software
Scaling networks through software
Fastly PoPs: engineering perspective
Scaling networks through software
Fastly PoPs: investor perspective
iv It’s easier to make people less busy than hire people.
Scaling networks through software
💿
Scaling networks through software
💿 ☁️
Scaling networks through software
💿 ☁️
Scaling networks through software
💿 ☁️Yes I would
Scaling networks through software
software
Scaling networks through software
softwarenetworking
“you wouldn’t do that to a switch”
Scaling networks through software
“Networking is hard”
Scaling networks through software
“Networking is hard”resource constraints
Scaling networks through software
“Networking is hard”resource constraintsprotocol standards
Scaling networks through software
“Networking is hard”resource constraintsprotocol standardssecurity concerns
Scaling networks through software
network vendors
“Networking is hard”resource constraintsprotocol standardssecurity concerns
Scaling networks through software
where is time spent needlessly?
Scaling networks through software
pinpointing path failures
Scaling networks through software
st-ping: probe all upstreams
Scaling networks through software
changing route preferences
Scaling networks through software
Scaling networks through software
upstream alias
Scaling networks through software
announced prefixes
Scaling networks through software
live BGP info
Scaling networks through software
Scaling networks through software
increase Google localpref
Scaling networks through softwarelocalpref overrides
Scaling networks through software
Scaling networks through software
changing prefix announcements
Scaling networks through software
Scaling networks through software
withdraw all HTTP anycast prefixes
Scaling networks through software
BGP session status
Scaling networks through software
BGP session status
Scaling networks through software
changing global routing policy
Scaling networks through software
Internet
We generate lots of BGP announcements ‣ changing policy manually is hard ‣ changing policy per-device takes long
Scaling networks through software
InternetInternet
route server
We generate lots of BGP announcements ‣ changing policy manually is hard ‣ changing policy per-device takes long
Scaling networks through software
Stage and deploy via Github ‣ generate diff of routing policy and exported routes ‣ peer reviewed, endlessly revertible
Scaling networks through software
Staging lists affected switches and prefixes ‣ human error could withdraw Fastly from the Internet ‣ hard to automate, so make sure people can get it right first
Scaling networks through software
Seems so simple… ‣ reduced time spent needlessly ‣ reduced human error dramatically ‣ allowed us to train netops out of our datacenter team ‣ Arista eAPI allows description changes: instant RESTful orchestration
v existing best practices won’t save you.
Scaling networks through software
Saving money ‣ buy bare essentials ‣ distribute everything ‣ efficiency matters
Saving time ‣ correct architecture helps! ‣ reduce cognitive overhead ‣ solve ops first, automate later
Scaling networks through software
Be wary of: ‣ best practices ‣ cool stuff ‣ perfect
www.fastly.com/about/careers !March 16th 2015 | João Taveira Araújo @jta