The Netflix Open Connect Network DKNOG 21 March 2013 1
Slide 2
Today we are in 22 locations across the world... (16 Primary, 6
network extensions) 2
Slide 3
3
Slide 4
How do we decide what locations to go to? Where is Netflix
presently providing service? Our service is limited by the content
licenses that we can buy For example, we are in the Nordic
countries, the UK, and Ireland but not France/Germany Can we
interconnect with networks that are relevant? Going to a peering
point thats all content providers does no good Can we reduce the
cost and improve traffic management for networks who need to
deliver Netflix traffic to their end users? 4
Slide 5
Netflix in the Nordics We provide our streaming service to
Denmark Finland Norway Sweden Iceland and Greenland Currently
co-located at Telecity Stockholm 1 & 2 Stockholm 1 is a network
extension from Stockholm 2 due to space/power constraints Likely to
expand to a second market in the Nordics this year 5
Slide 6
IXes Netflix delivers their Nordic traffic from four different
IXes NetNod NIX AMS-IX LINX The latter two may not seem relevant
here, but many networks have significant capacity built there
6
Slide 7
Standard build - European A European Netflix Open Connect IX
build has: 20 Netflix Open Connect Appliances Two Juniper MX480
routers OpenGear console server and Juniper EX2200s for OOB access
Prewired MTP to LC cabling from Elpeus A network extension Open
Connect build has: Two Juniper EX4550 switches Two MRV 40 channel
passive muxes OpenGear console server for OOB access 7
Slide 8
Stockholm 8
Slide 9
Network Extension POP 9
Slide 10
Standard Build - US A US Netflix Open Connect IX build has: 40
Netflix Open Connect Appliances 60 Netflix Open Connect Flash
Appliances Two Juniper MX960 routers OpenGear console server and
Juniper EX2200s for OOB access Prewired MTP to LC cabling from
Elpeus A network extension Open Connect build has: Two Juniper
EX4550 switches Two MRV 40 channel passive muxes OpenGear console
server for OOB access 10
Slide 11
11
Slide 12
Converting some POPs to Cisco ASR Due to concerns with Juniper
code delivery and the need for higher density platforms, we decided
in early 2013 to integrate some Cisco ASR 9K into our network to
replace Juniper MX In order to deliver our traffic goals (~1T per
router), we need denser cards than the currently shipping Juniper
16 port MPC3s 12
Slide 13
High Density Juniper MX960 w/ MPC3s can deliver 176 ports per
chassis A single fabric failure brings the chassis to 3:2
oversubscription Juniper MX960 w/ MPC4s can deliver 336 ports per
chassis (oversubscribed) A single fabric failure (out of 3) brings
the chassis to 2:1 oversubscription unacceptable Cisco ASR9010 w/
Typhoon-based 36 port cards can deliver 288 ports per chassis A
single fabric failure brings the chassis to 3:2 oversubscription
13
Slide 14
High Density, continued We conservatively need 130 host ports
per router Those 130 ports can deliver 1.13T of traffic Assuming we
deliver 1 bit outbound for every bit inbound from a host, that puts
us at 1.1:1 oversubscribed at peak 3:2 and 2:1 just do not work for
us Juniper MPC4s also require brand new 12.3 code What version of
code are you running? 14
Slide 15
Cabling a new ASR 15
Slide 16
The Netflix Open Connect Appliance (OCA) Developed in response
to ISP requests to help scale Netflix traffic efficiently Reduces
ISP cost by serving Netflix traffic from the local ISP datacenter,
CO or headend, rather than upstream network interconnects Speeds up
internet access for consumers to all third- party internet sites,
because Netflix traffic is no longer a source of middle-mile or
backbone congestion Netflix bears the capital and maintenance
costs, not ISP ISP provides space, power and a network port An OCA
is a component of the Netflix CDN (vs a cache) 16
Slide 17
OCA Hardware Space optimized: 4U high-density storage Power
optimized for low power/cooling requirements ( 500W) Redundant 10GE
optical network interfaces Redundant power supplies (AC or DC)
17
Slide 18
Why Deploy a Netflix OCA? Netflix data is a significant
percentage of ISP traffic in the markets we serve Serve >80% of
Netflix traffic from the local ISP datacenter Remaining traffic can
be served by upstream OCAs, peering Saves on transit, transport and
other upstream scaling costs Provided free of charge to
participating ISPs ISPs with > 3 Gbps of Netflix traffic ISP
provides rack space, power, 10 Gbps optical port(s)
Slide 19
OCA Operation Used exclusively for Netflix content Completely
integrated with the Netflix content delivery system Greater
effectiveness than transparent or proxy caches (>80% with
Netflix OCA, versus 20-50% with other caches) Efficient content
fill mechanisms Outside peak times (ISP selected time and BW per
OCA) OCA offline during fill, staggered fill recommended. Fill can
source from neighboring, peer or transit OCAs ~5TB fill per day
(i.e ~4Gb/s for ~2.5 hrs) OCA must be reachable by end users (port
80)
Slide 20
Directing Clients to OCAs Netflix Control Servers Broadband ISP
1. Client contacts Netflix control servers 2. Netflix control
servers respond with address of local OCA 3. Client connects to
local OCA 4. Local OCA delivers video stream User routing is done
by Netflix control servers, not dependent on client DNS
configuration Request is routed to the nearest available OCA
Working sets of popular content deliver up to >80% network
offload Netflix OCA
Slide 21
Network Positioning OCAs are installed close to aggregations of
end users Netflix works with ISPs to determine proper location and
user-to-OCA mapping ISP Controls Traffic Flow End-user netblocks
are associated with OCA via BGP advertisement Netflix honors AS
path-length and MEDs allowing ISP to configure OCA hierarchy and
failover OCAs only serve content to netblocks provided by ISP ISPs
choose peak throughput each OCA is allowed to generate
Slide 22
The Netflix ISP Speed Index Published monthly Helps consumers
make an intelligent choice about their broadband provider We want
our users to have the best experience possible We also want to help
networks deliver Netflix traffic as efficiently as possible 22
Slide 23
23
Slide 24
A recently enabled network (night of 18/03/13) 24 No marked
increase in bandwidth utilization, but a more stable experience for
users
