Interative Traffic Engineering in Changing Internet Economics - Tom Daly at LISA 2015

Name of Presentation

Iterative Traffic Engineeringin Changing Internet Economics

Tom DalyVP, [email protected]


• Provide a framework to make complex bandwidth purchasing decisions:– Business needs– Traffic throughput and performance– Direct cost management and cost optimization– Staff size and cost

• Interest in:– Internet Economics Megatrends– Disruptive Events– Metrics and Tools

• Creation of a long term bandwidth acquisition strategy

Goals


• Industry experience: 15+ years of datacenter and connectivity acquisition deals

• I work for a CDN: Fastly– We use a lot of bandwidth– Therefore, we acquire much bandwidth– Partnerships with Sales / Deal Desk, Finance, Marketing, and

Engineering• Internet T/E techniques: DNS, BGP, and IP Anycast

– And related measurement techniques: Performance Monitoring, Real User Monitoring (RUM), TCP Flow Analysis

• Education:– BS ECE – Worcester Polytechnic Institute– MBA – Bentley University

Presenter Qualifications


• Pricing:– Don’t ask for pricing at volume; I honor non-disclosure

agreements.– IP Transit costs are normalized to USD$10/Mbps/MRC

for services delivered in carrier neutral datacenter.– IP Peering costs are normalized to USD $5000/10G

Port/MRC.• Net Neutrality:

– I/My employer have a duty to deliver content bits on behalf of our customers in an performance optimal and efficient manner.

• Opinions and views expressed are my own; not the viewpoint of Fastly, Inc.

Assumptions


• IP Transit: Paid access to the full Internet routing table.• Peering: Access to a subset of Internet routes via agreed

upon interconnection.– Settlement Free Peering: Free and balanced.– Paid Peering: Dominant party pays

• Internet Exchange Point (IXP): Peering across a shared Layer 2 switch domain

• Cloud Interconnect Service: Peering across an allocated Layer 2 switch domain

• Private Network Interconnect (PNI): Peering across a physical circuit

Nomenclature


The Challenge: Growth and Scale





$$$


• 10Gb Ethernet is mainstream; 100GbE is accessible

• Net Neutrality rulings in US and EU– Highly varying benefits and downside between US and

EU• The “dumb” pipes problem: Transport and IP

services increasingly commoditized• Resultantly; the global IP Transit market

compresses 16-18% per year, depending on region

• Movements to private datacenter/cloud interconnection

• Massive improvements to measurement and monitoring

Industry Shifting Economic Trends


Capitalize the Open Market Economy


Question: Transit vs. Peering?


Phase 1: Transit


Transit Pricing at Scale


Transit Pricing at Scale

“Yolo…just go peer some traffic”


Phase 2: Transit + Some Offload

IP Transit

PeeringCloud Interconnect


Peering Offload Modeling (20Gbps)




Transit Pricing – Compressing 17% YoY



No consideration for peering staff overhead or

peering externalities



No consideration for peering staff overhead or

peering externalities2x Peering Coordinators3x NOC---------------------------------$425,000/yr + Benefits


Peering Path Connection Complexity






• IXP outages• Congestion through IXP ports• Unbalanced peers won’t upgrade

PNIs• “de-peering” between

interconnected networks• Changes to routing and geo-location

degrade or improve performance

Disruptive Peering Events and Behaviors


Single Month Outage (40Gbps)


Question: Capacity Planning + Cost Models

What we know

The Future!


Regression Analysis - Excel


Regression Analysis – Peering Prediction


Scenario Analysis: Expected Peering Productivity


Question: 10Gb vs. 100Gb Ethernet12x10Gb Ports- 12x 10GBase-LR Optics:

$150 each = $1,800- 12x SMF Cross Connects:

$350/MRC = $50,400/ARC- Total: $52,200/1 YR


100GbE Scenario12x10Gb Ports- 12x 10GBase-LR Optics:


$350/MRC = $50,400/ARC- Total: $52,200/1 YR- Consider line card ports 2x100Gb Ports

- 2x 100GBase-LR4 Optics:$5600 each = $11,200

- 2x SMF Cross Connects:$350/MRC = $8,400/ARC

- Total: $19,600/1 YR- Consider line card ports


Eggs? Why are they in one basket?12x10Gb Ports- 12x 10GBase-LR Optics:


$350/MRC = $50,400/ARC- Total: $52,200/1 YR

2x100Gb Ports- 2x 100GBase-LR4 Optics:


$350/MRC = $8,400/ARC- Total: $19,600/1 YR

Reliability Modeling?- Afford to loose 10G at a time?- Afford to loose 100G at time?


An incomplete story…

Performance Matters...

Volumetric cost improvements…


Latency and Bandwidth Delay Product


ms and BDP: Doubled Packet Loss


ms and BDP: Halved Latency


“You can't manage what you can't measure.”

– Peter F. DruckerSocial Ecologist;

Management Consultant;Professor


Detailed Measurements• Synthetic Agents: Latency, packet loss,

and path validation• RUM: Geolocation, Recursive DNS

divergence and positioning, /latency/• NetFlow / sFlow: Bandwidth by Remote

ASN and Next-Hop Peer• Linux TCPExt: Retransmission and Packet

Loss• ICMP Ping: Position confirmation;

triangulation


RIPE Atlas – https://atlas.ripe.net


Real User Monitoring (RUM)


NetFlow / sFlow Analysis


Linux TCP Extensions


ICMP Ping – “Fastly Style”


Methodology and Take Away(s)


• Early Stage: Just buy transit!– Too small and too inefficient to peer–Work on tooling to help you peer later

by analyzing traffic patterns– Automate automate automate your

routine peering and monitoring workloads

Iterative Traffic Engineering (1)


• Mid Stage: Identify growth plans and potential– Regression analysis across 18-24

months data– Identify and connect strategic / high

value peers– Use cost or performance scenario

analysis– Note: Not all networks are good peers;

see risks



• Ongoing: Constant evolution– Transit providers apprised of growth plan– Contract duration adjusted to long term risk of

market price compression– Peering load adjusted to team side– Measure, measure, measure



Questions?

Tom DalyVP, [email protected]

Technology

Interative Traffic Engineering in Changing Internet Economics - Tom Daly at LISA 2015