ESnet Abilene 3+3 ESnet Abilene 3+3 MeasurementsMeasurements

Presented at the Joint Techs Meeting in Presented at the Joint Techs Meeting in ColumbusColumbus

July 19July 19thth 2004 2004

Joe MetzgerJoe Metzger

ESnet Network EngineerESnet Network Engineer

metzger@es.netmetzger@es.net

CollaboratorsCollaborators

Chin Guok, Bill Johnston & Kevin Oberman@ Chin Guok, Bill Johnston & Kevin Oberman@ ESnetESnet

Chintan Desai & John Moore @ NCSU Chintan Desai & John Moore @ NCSU Darryl Wohlt & Phil Demar @ FERMIDarryl Wohlt & Phil Demar @ FERMI Jeff Boote, Eric Boyd & Guy Almes @ Jeff Boote, Eric Boyd & Guy Almes @

Internet2Internet2 Jin Guojun @ LBL Jin Guojun @ LBL Kevin Walsh @ SDSC Kevin Walsh @ SDSC Prasad Calyam @ OSU / OARnet Prasad Calyam @ OSU / OARnet

3+3 Measurements3+3 Measurements

3 ESnet Sites3 ESnet Sites LBLLBL FERMIFERMI BNLBNL

3 Abilene 3 Abilene ParticipantsParticipants SDSCSDSC NCSUNCSU OSUOSU

Abilene and ESnet have systems in place to measure our portions of the net. We were not measuring performance across our interconnections.

Why?Why?

We want to ensure that the We want to ensure that the ESnet/Abilene cross connects are ESnet/Abilene cross connects are serving the needs of users in the serving the needs of users in the science community who are science community who are accessing DOE facilities and accessing DOE facilities and resources from universities or resources from universities or accessing university facilities from accessing university facilities from DOE labs.DOE labs.

Existing Measurement Existing Measurement Collections were not meeting Collections were not meeting

our needs.our needs. ESnet and Abilene monitor traffic, errors and ESnet and Abilene monitor traffic, errors and

discards on all of our respective links including discards on all of our respective links including interconnection points.interconnection points.

This monitoring shows interconnections are This monitoring shows interconnections are lightly loaded and error free.lightly loaded and error free.

Other measurement systems we looked at do Other measurement systems we looked at do not contain the mix of ESnet and Abilene sites not contain the mix of ESnet and Abilene sites we are looking for.we are looking for.

Why Start with Latency Why Start with Latency Testing?Testing?

Low ImpactLow Impact Sensitive to network eventsSensitive to network events

What is OWAMPWhat is OWAMP

One Way Measurement ProtocolOne Way Measurement Protocol http://www.ietf.org/internet-drafts/draft-ietf-ippm-owdp-http://www.ietf.org/internet-drafts/draft-ietf-ippm-owdp-

08.txt08.txt

A suite of toolsA suite of tools http://e2epi.internet2.edu/owamp/http://e2epi.internet2.edu/owamp/

Data VisualizationData Visualization

What is the best way to display latency What is the best way to display latency data?data? It is difficult to identify trends in numeric It is difficult to identify trends in numeric

tables.tables. What is interesting or meaningful?What is interesting or meaningful?

Mean?Mean? No.No.

Median and 95Median and 95thth percentiles? percentiles? Maybe…Maybe…

Distribution?Distribution? Yes!Yes!

SmokePing by Tobias SmokePing by Tobias OetikerOetiker

http://www.smokeping.orghttp://www.smokeping.org Shows distribution of latency Shows distribution of latency

measurements.measurements.

Our VisualizationsOur Visualizations

Extended SmokePing graphic design to include Extended SmokePing graphic design to include multiple data sets on one graph.multiple data sets on one graph. SmokePing uses 20 shades of gray and plots one data SmokePing uses 20 shades of gray and plots one data

set on a graph.set on a graph. We are using a different color for each data set on a We are using a different color for each data set on a

graph. graph. We are using different saturations of the colors to show We are using different saturations of the colors to show

the distribution of results.the distribution of results. Show NTP error estimates.Show NTP error estimates. Graphs implemented as RRD templates to leverage Graphs implemented as RRD templates to leverage

existing ESnet statistics tools & infrastructure.existing ESnet statistics tools & infrastructure. Do not show loss information at this time.Do not show loss information at this time.

FERMI to SDSC, LBL & NCSUFERMI to SDSC, LBL & NCSU

Red: FERMI to LBLRed: FERMI to LBLBlue: LBL to FERMIBlue: LBL to FERMI

Green NTP Error Estimates

A couple packets experienced queuing delays

Clock Event

Measurement ServersMeasurement Servers

LBL and NCSU were ready in late LBL and NCSU were ready in late AprilApril

FNAL and SDSC in MayFNAL and SDSC in May OSU in JuneOSU in June BNL in JulyBNL in July

Interesting ObservationsInteresting Observations

NTP Error Estimate QualityNTP Error Estimate Quality NCSU Metro DWDM RerouteNCSU Metro DWDM Reroute Queuing caused by bandwidth Queuing caused by bandwidth

testingtesting Asymetric RoutingAsymetric Routing

NTP Error Estimate QualityNTP Error Estimate Quality

NCSU Metro DWDM RerouteNCSU Metro DWDM RerouteAdds about 350 Micro SecondsAdds about 350 Micro Seconds

Fiber Re-Route

Bandwidth Tests Can Cause Bandwidth Tests Can Cause Queuing on Bottleneck LinksQueuing on Bottleneck Links

Test Traffic Rerouted Tuesday Morning

Large Data Transfers

Bottleneck Link Traffic & Bottleneck Link Traffic & DiscardsDiscards

Asymetric Routing betweenAsymetric Routing betweenFERMI and SDSCFERMI and SDSC

(LBL to CENIC link Maintenance)(LBL to CENIC link Maintenance)

Future DirectionFuture Direction

Utilize a generalized, interoperable Utilize a generalized, interoperable measurement collection and measurement collection and archiving system instead of current archiving system instead of current ad-hoc scripts.ad-hoc scripts.

Look carefully at implementing Look carefully at implementing bandwidth testing, perhaps using bandwidth testing, perhaps using Scavenger QOS.Scavenger QOS.

ConclusionsConclusions

The ESnet/Abilene interconnections The ESnet/Abilene interconnections are not the bottlenecks on the tested are not the bottlenecks on the tested paths.paths.

Latency data can show interesting Latency data can show interesting queuing effects that may not be queuing effects that may not be obvious in other measurements.obvious in other measurements.

A single user with a $5K box can A single user with a $5K box can congest many current access links.congest many current access links.

Is 1-2 ms queuing a Is 1-2 ms queuing a problem?problem?

The EndThe End

For more info see For more info see http://measurement.es.nethttp://measurement.es.net