March 16th 2015 | João Taveira Araújo @jta - USENIX · Scaling networks through software PoP...

Scaling networks through softwareMarch 16th 2015 | João Taveira Araújo @jta

Scaling networks through software

network systems @ fastly

Scaling networks through software

Scaling networks through software

scalability

Scaling networks through software

constraints

constraints time

money people

Scaling networks through software

constraints

constraints time

money people

knowledge

Scaling networks through software

constraints

constraints time

money people

knowledgetechnology

Scaling networks through software

constraints

constraints time

money people

knowledgetechnologycomplexity

Scaling networks through software

constraints

constraints time

money people

knowledgetechnologycomplexity

Scaling networks through software

constraints time

money people

Scaling networks through software

constraints time

money people

Scaling networks through software

constraints time

money people

Scaling networks through software

Number of PoPs ……………………….… ~20 BGP announcements …………….……... ~2000 Requests per second .…………………… ~1000000

Becoming a multi terabit network

Scaling networks through software

Number of PoPs ……………………….… ~20 BGP announcements …………….……... ~2000 Requests per second .…………………… ~1000000

Network ops ……..….………………….… 2

Becoming a multi terabit network

Scaling networks through software

Number of PoPs ……………………….… ~20 BGP announcements …………….……... ~2000 Requests per second .…………………… ~1000000

Network ops ……..….………………….… 2

Becoming a multi terabit network

Network software ……..….…………….… me

Scaling networks through software

scalability

Scaling networks through software

scalabilityobservations on network

from a company that used to be a startup

i anything you don’t explicitly control is an implicit liability

Scaling networks through software

the internet

Scaling networks through software

router

A B C D

PoP

the internet

Scaling networks through software

router

A B C D

PoP How do you: ‣ load balance traffic ‣ gracefully failover if a server fails

the internet

Scaling networks through software

router

A B C D

PoPA

25%

25%

25%

25%

B

DC

DNS

Scaling networks through software

router

A B C D

PoPA

33%

33%

33%

0%

B

DC

DNS

Scaling networks through software

router

A B C D

PoPA

33%

33%

33%

0%

B

DC

DNS

Bad idea: ‣ gets hard to manage ‣ do one thing and do it well ‣ you don’t control TTL

Scaling networks through software

load balancer

A B C D

PoP

Scaling networks through software

load balancer

A B C D

PoP

StateC

ost

Scaling networks through software

load balancer

A B C D

PoP

StateC

ost

Exist

ing d

eman

d

StateC

ost

Scaling networks through software

load balancer

A B C D

PoP

StateC

ost

Exist

ing d

eman

d

Proje

cted

dem

and

StateC

ost

Exist

ing d

eman

d

StateC

ost

Scaling networks through software

load balancer

A B C D

PoP

StateC

ost

Exist

ing d

eman

d

Proje

cted

dem

and

StateC

ost

Exist

ing d

eman

d

StateC

ost

Scaling networks through software

load balancer

A B C D

PoPBad idea: ‣ you don’t control demand ‣ you don’t control DDOS

State

Cos

t

Exist

ing d

eman

d

Proje

cted

dem

and

State

Cos

t

Exist

ing d

eman

d

State

Cos

t

Scaling networks through software

ECMP

A B C D

PoP

Scaling networks through software

ECMP

A B C D

PoP

Destination network Next hop

10.0.0.0/24 A

10.0.0.0/24 B

10.0.0.0/24 C

10.0.0.0/24 D

Scaling networks through software

ECMP

A B C D

PoPECMP

A B C D

PoP

Destination network Next hop

10.0.0.0/24 A

10.0.0.0/24 B

10.0.0.0/24 C

10.0.0.0/24 D

Scaling networks through software

ECMP

A B C D

PoPECMP

A B C D

PoP

Destination network Next hop

10.0.0.0/24 B

10.0.0.0/24 C

10.0.0.0/24 D

Bad idea: ‣ connection resets ‣ you don’t control rehashing ‣ you don’t control vendor roadmaps

ii don’t resign to fate just because everything sucks

Scaling networks through software

faild

Scaling networks through software

ECMP

A B C D

PoP

Scaling networks through software

Destination network Next hop

10.0.0.0/24 10.1.A.1

10.0.0.0/24 10.1.A.2

10.0.0.0/24 10.1.A.3

… …

ECMP

A B C D

PoP

Scaling networks through software

Destination network Next hop

10.0.0.0/24 10.1.A.1

10.0.0.0/24 10.1.A.2

10.0.0.0/24 10.1.A.3

… …

IP Address MAC

10.1.A.1 A:A

10.1.A.2 A:A

10.1.A.3 A:A

… …

ECMP

A B C D

PoP

Scaling networks through software

Destination network Next hop

10.0.0.0/24 10.1.A.1

10.0.0.0/24 10.1.A.2

10.0.0.0/24 10.1.A.3

… …

IP Address MAC

10.1.A.1 A:A

10.1.A.2 A:A

10.1.A.3 A:A

… …

ECMP

A B C D

PoP

Scaling networks through software

drain a host

Scaling networks through software

Destination network Next hop

10.0.0.0/24 10.1.A.1

10.0.0.0/24 10.1.A.2

10.0.0.0/24 10.1.A.3

… …

ECMP

A B C D

PoP

Scaling networks through software

Destination network Next hop

10.0.0.0/24 10.1.A.1

10.0.0.0/24 10.1.A.2

10.0.0.0/24 10.1.A.3

… …

IP Address MAC

10.1.A.1 B:A

10.1.A.2 C:A

10.1.A.3 D:A

… …

ECMP

A B C D

PoP

Scaling networks through software

Destination network Next hop

10.0.0.0/24 10.1.A.1

10.0.0.0/24 10.1.A.2

10.0.0.0/24 10.1.A.3

… …

IP Address MAC

10.1.A.1 B:A

10.1.A.2 C:A

10.1.A.3 D:A

… …

ECMP

A B C D

PoP

Scaling networks through software

cut off to failed state

Scaling networks through software

Destination network Next hop

10.0.0.0/24 10.1.A.1

10.0.0.0/24 10.1.A.2

10.0.0.0/24 10.1.A.3

… …

ECMP

A B C D

PoP

Scaling networks through software

Destination network Next hop

10.0.0.0/24 10.1.A.1

10.0.0.0/24 10.1.A.2

10.0.0.0/24 10.1.A.3

… …

IP Address MAC

10.1.A.1 B:B

10.1.A.2 C:C

10.1.A.3 D:D

… …

ECMP

A B C D

PoP

Scaling networks through software

Time

Activ

e flo

ws

ecmp

Scaling networks through software

Time

Activ

e flo

ws

Time

Activ

e flo

ws

ecmp

Scaling networks through software

Time

Activ

e flo

ws

Stat

e

Time

load balancer

Scaling networks through software

Stat

e

Time

Time

Activ

e flo

ws

Activ

e flo

ws

Stat

e

Time

faild

Scaling networks through software

Stat

e

Time

Time

Activ

e flo

ws

Activ

e flo

ws

Stat

e

Time

faild

Scaling networks through software

Stat

e

Time

Time

Activ

e flo

ws

Activ

e flo

ws

Stat

e

Time

faild

iii if it’s expensive you probably don’t need it.

Scaling networks through software

F5 BIG-IP 10350v

Scaling networks through software

$200,000F5 BIG-IP 10350v

Scaling networks through software

$200,000$0

F5 BIG-IP 10350v

Scaling networks through software

load balancer

Scaling networks through software

load balancer balancing

Scaling networks through software

load balancer balancing(a load balancer is just an appliance which provides load balancing)

Scaling networks through software

load balancer balancing(a load balancer is just an appliance which provides load balancing)

distributed

Scaling networks through software

How to build a PoP ‣ buy a router ‣ get BGP table from each provider ‣ install routes to FIB ‣ servers use default gateway A B C D

PoP

Vendor BGP

FIB

router

Scaling networks through software

Juniper MX960 Router

Scaling networks through software

~$500,000

Juniper MX960 Router

Scaling networks through software

router

Scaling networks through software

router routing

Scaling networks through software

router routing(a router is just an appliance which provides routing)

Scaling networks through software

router routing(a router is just an appliance which provides routing)

distributed

Scaling networks through software

Arista DCS-7150S switch family

Scaling networks through software

$29,995 Arista DCS-7150S switch family

Scaling networks through software

PoP

Userspace BGP

FIB

switch

BGP daemon

FIB

server

How to build a Fastly PoP ‣ buy switches ‣ reflect BGP down to servers ‣ inject multipath routes into FIB

Scaling networks through software

PoP

Userspace BGP

FIB

switch

BGP daemon

FIB

server

How to build a Fastly PoP ‣ buy switches ‣ reflect BGP down to servers ‣ inject multipath routes into FIB

Scaling networks through software

PoPd

istribute

d syste

m

Userspace BGP

FIB

switch

BGP daemon

FIB

server

How packets egress Fastly ‣ switches emit nexthop IP and MAC ‣ servers configure p2p link / ARP ‣ send directly to provider nexthop!

Scaling networks through software

PoPd

istribute

d syste

m

Userspace BGP

FIB

switch

BGP daemon

FIB

server

How packets egress Fastly ‣ switches emit nexthop IP and MAC ‣ servers configure p2p link / ARP ‣ send directly to provider nexthop!

Scaling networks through software

Scaling networks through software

Fastly PoPs: engineering perspective

Scaling networks through software

Fastly PoPs: investor perspective

iv It’s easier to make people less busy than hire people.

Scaling networks through software

💿

Scaling networks through software

💿 ☁️

Scaling networks through software

💿 ☁️

Scaling networks through software

💿 ☁️Yes I would

Scaling networks through software

software

Scaling networks through software

softwarenetworking

“you wouldn’t do that to a switch”

Scaling networks through software

“Networking is hard”

Scaling networks through software

“Networking is hard”resource constraints

Scaling networks through software

“Networking is hard”resource constraintsprotocol standards

Scaling networks through software

“Networking is hard”resource constraintsprotocol standardssecurity concerns

Scaling networks through software

network vendors

“Networking is hard”resource constraintsprotocol standardssecurity concerns

Scaling networks through software

where is time spent needlessly?

Scaling networks through software

pinpointing path failures

Scaling networks through software

st-ping: probe all upstreams

Scaling networks through software

changing route preferences

Scaling networks through software

Scaling networks through software

upstream alias

Scaling networks through software

announced prefixes

Scaling networks through software

live BGP info

Scaling networks through software

Scaling networks through software

increase Google localpref

Scaling networks through softwarelocalpref overrides

Scaling networks through software

Scaling networks through software

changing prefix announcements

Scaling networks through software

Scaling networks through software

withdraw all HTTP anycast prefixes

Scaling networks through software

BGP session status

Scaling networks through software

BGP session status

Scaling networks through software

changing global routing policy

Scaling networks through software

Internet

We generate lots of BGP announcements ‣ changing policy manually is hard ‣ changing policy per-device takes long

Scaling networks through software

InternetInternet

route server

We generate lots of BGP announcements ‣ changing policy manually is hard ‣ changing policy per-device takes long

Scaling networks through software

Stage and deploy via Github ‣ generate diff of routing policy and exported routes ‣ peer reviewed, endlessly revertible

Scaling networks through software

Staging lists affected switches and prefixes ‣ human error could withdraw Fastly from the Internet ‣ hard to automate, so make sure people can get it right first

Scaling networks through software

Seems so simple… ‣ reduced time spent needlessly ‣ reduced human error dramatically ‣ allowed us to train netops out of our datacenter team ‣ Arista eAPI allows description changes: instant RESTful orchestration

v existing best practices won’t save you.

Scaling networks through software

Saving money ‣ buy bare essentials ‣ distribute everything ‣ efficiency matters

Saving time ‣ correct architecture helps! ‣ reduce cognitive overhead ‣ solve ops first, automate later

Scaling networks through software

Be wary of: ‣ best practices ‣ cool stuff ‣ perfect

www.fastly.com/about/careers !March 16th 2015 | João Taveira Araújo @jta