1

Network Measurement SummaryESCC, Feb. 2006

Joe Metzger

ESnet Engineering GroupLawrence Berkeley National Laboratory

2

General Trends 1

• There has been a community shift in the focus of network measurement effortso From supporting the people running the networkso To supporting the people using the networko How we get there

- Backbone providers agree on one set of tools to archive and publish measurement data

– ESnet, Internet2, GEANT, most European NRENs and others are all collaborating on PerfSONAR

– Standardization on collection tools may also happen

- The user community develops analysis and visualization tools that leverage the backbone provider data to meet their needs

– EGEE network analysis tool– PerfsonarUI– MonALISA?

3

General Trends 2• 10 Gigabit networking will present new challenges

o The high costs of 10GE routers is pushing people to deploy different architectures using lower cost switches.

o Many of these lower cost devices provide very limited diagnostic and debugging support.

o Over subscription of circuits and bottlenecks within equipment may cause un-expected packet loss.

o The differences between OC192, 10GE LANPHY, 10GE WANPHY and almost 10GE line-rate equipment may cause buffering problems.

o These issues will lead to problems that will be challenging to identify and resolve.

4

ESnet Deployed Tools & Capabilities ESnet Performance Centers SNMP & Flow Stats Owamp PerfSONAR

Utilization Measurement ArchiveTraceroute Visualization

5

ESnet Performance CentersA web interface that lets ESnet users perform tests to points in

the ESnet core

•Ping•Traceroute•Iperf•Pathload•Pathrate•Pipechar

6

2 SNMP Collection Systems

• One system based on MRTGo Collects interface utilization, error and discard data every

minute from ~1383 physical and logical interfaceso Used for day to day operations.o Portions accessible via http://netinfo.es.net

• Second collection system based on netSNMPo Collects interface utilizationo Monthly reportingo Details back to 1990

- http://www1.es.net/pub/esnet-stats/

• The two systems back each other up

7

Netflow

• Collect at ESnet ingress points

• Trendingo Identify large traffic flowso Observation: About 100 host pairs generate 30% of all

ESnet traffic

• Peering Analysiso ESnet has around 100 peers

- Most of them at multiple locations- Look at “what if” scenarios for traffic engineering

• Diagnosing problemso Identifying endpoints of a flow that saturated a site access

link

8

OWAMP

• Initially deployed to measure & compareo DOE Lab to DOE Lab performanceo DOE Lab to University performanceo Mesh between

- FERMI, LBNL, BNL- OSU, NCS, SDSC

• Very small network changes can generate measurable differences in latency.

• OWAMP monitors deployed in most ESnet hubs

9

PerfSONAR Utilization MA• Provide ESnet users and network researchers

access to ESnet link utilization datao Deployed a PerfSONAR Measurement Archive (MA)

- Uses the “EGEE” tagged release of PerfSonar Java MA- Provides a web services interface using NMWG V2 schema- Data source is RRD files populated by SNMP system

– Copied to MA every 5 minutes

o Provides link capacity and link utilization information for over 800 ESnet interfaces.

• Software that uses the ESnet MAo EGEE network diagnostic toolo CNM (DFN Customer Network Management tool)o PerfSONARUI o Traceroute Visualizer

10

PerfSONAR Traceroute Visualizer

• Displays a web form

• User pastes a traceroute output into the form

• Parses the form data to find IP addresses

• Displays the traceroute with interface utilization graphs for all the interfaces it can find.

• Developed a quick demo to show the value of a standardized measurement infrastructure

• Show demo if time permits

11

Traceroute

ESnet PerfSONAR Traceroute Visualizer

Trace SubmittedTracing route to cache3.bnl.gov 130.199.3.21 over a maximum of 30 hops 1 1 ms 1 ms 1 ms joem-fe-stub.es.net 198.124.224.5 2 21 ms 21 ms 21 ms chi-ameslab.es.net 134.55.208.38 3 21 ms 21 ms 21 ms chicr1-ge0-chirt1.es.net 134.55.209.189 4 41 ms 41 ms 41 ms aoacr1-oc192-chicr1.es.net 134.55.209.58 5 43 ms 43 ms 43 ms bnl-oc48-aoacr1.es.net 134.55.209.130 6 43 ms 43 ms 43 ms bnl-esbnl.es.net 198.124.216.114 Analyzing Trace

12

CNM from DFN 1

13

CNM from DFN 2

14

PerfsonarUI 1

15

PerfsonarUI 2

16

PerfsonarUI 3

17

ESnet Future Plans 1

• Generalo Continue the PerfSONAR collaboration with Internet2,

GEANT, and the European National Research & Education Networks

o Deploy additional components of the PerfSONAR architecture as it is developed

- Lookup service- Topology service- Bandwidth measurement points- Latency measurement points- Authentication service

o Enhance & extend our end to end monitoring efforts- Include more LHC sites- Interoperate with other tools used in LHC community

18

ESnet Future Plans 2

• SNMPo Use PerfSONAR to continue to publish ESnet interface

data

• Latencyo Integrate existing ESnet latency MPs with PerfSONARo Deploy new latency MP’s in new hubs

• Bandwidtho Integrate ESnet Performance Centers with PerfSONAR

- GE capable bandwidth MPs

o Deploy new PerfSONAR bandwidth MPs- 10 GE capable

19

Suggestions for Sites• Deploy test servers at the edge of your network.

o OWAMP for measuring latencyo IPERF/BWCTL for measuring bandwidtho Internet2 is measurement workshops

• Collect interface utilization datao Capture data with MRTG, Cricket, SNAPP or other tools.

• Join the PerfSONAR development team if you are actively developing measurement tools

• Think about data export policies

• Plan to deploy PerfSONAR in the summero After we generate a release…

20

Summary

• ESnet will deploy tools:o To run the backboneo To provide the data necessary for end users to develop

and tune applications

• Sites should deploy tools that will allow them to:o Determine if their applications are working correctlyo Determine if the local network is working correctlyo Generate useful information for diagnosing problems

• Use these tools to:o Continuously document performance so you know when it

changeso Share your network measurements results

21

Pointers

o http://www.perfsonar.neto http://netinfo.es.neto https://performance.es.neto http://e2epi.internet2.edu