Introduction to the PPNCG Networking for the PPARC Community

Astronomy Sysman Meeting 29/30 April 02R. Hughes-Jones Manchester

1

Introduction to the PPNCG Networking for the PPARC Community

Introduction to the PPNCG UK Network TopologiesExternal Connectivity – Europe & USAstronomy & Astrophysics Sites Grid Network Monitoring PingER – World wide MonitoringQoS – a micro Introduction


2

Introduction to the PPNCGMembership includes HEP and Astronomy users

Dave Terrett , Bob Bentley, Ralph Spencer

RemitEnsure the community has the required networking facilities

Monitor end-to-end performance

Investigate new network applications / technologies

Provide advice on kit / facilities

Active Network MonitoringPPNCG ping, ftp and traceping

ICFA monitoring

Report problems to UKERNA

Regular meetings with UKERNA invitedRecognised as a subject group in JNUG and JISCLinks to several Grid Projects


3

SuperJANET4: Backbone and Access links

Worldcom supplied the transmissionUKERNA layer the IP service

Core PoP IP router at Worldcom

Backbone Access Router at MANs

Access Links:Large MAN 2.5 Gbit -> 10-20 Gbit

Medium MAN 622 Mbit -> 2.5 Gbit

4 node DWDM development netDeployment Status:

Backbone Oct 00 Routers Nov 00

All sites Mar 01

Proved to be Stable

Constant growth of traffic

Upgrade Backbone to 10Gbit Jun 02


4

SuperJANET4: ping rtt Core routers Jun 01


5

SuperJANET4: ping rtt Site nodes

Lancaster Glasgow

MAN / LAN Issues

Bristol Cambridge


6

London MAN Upgrade

UDP Throughput

Mbit/s

UDP Packet loss

%

UDPmon Tests

Manchester – London

MAN was 155 Mbit ATM

1st Oct Time interval in Weeks Richard HJ


7

Previous External Connectivity

Europe:TEN-155155Mbit Access link

US:6 * 155 Mbit links

Peer in Hudson St. 622 Mbit to Esnet622 Mbit to Abilene.


8

ICFAMON Plot from RAL to CERN for 19th Oct to 1st Nov 2001

Europe – Access links (1)

UK Access link 155 Mbit ATMSustained rate 130 MbitContract to end of Nov 01Bad news for users !


9

Traceping Oxford to CERN for 31st October 2001

Europe – Access links (2)

loss around ten155-gw.ja.net router

J. Macallister


10

New External Connectivity

6 * 155 Mbit links2.5Gbit line installedIP commodity peer in London Research traffic over 2.5G bitPeer in Hudson St. 622 Mbit to Esnet622 Mbit to Abilene.


11

Connectivity to Europe : GeantStart mid November 2001UKERNA switched off TEN-155 3 Dec 2001


12

ICFAMON Plot from DL to CERN for 18th Feb to 3rd Mar 2002

Connectivity to Europe

UK Dante Access link 2.5 Gbit POS

Remember 19th Oct to 1st Nov 2001Access link over loaded


13

Monitoring: US Traffic UKERNA Traffic data Kbit/s. Blue Traffic from US; Maroon Traffic to US

7 day periods 1 hour averages

Weekend-Before Weekday-After

Weekday-Before

14 Jan 2002 (800Mbit/s)peak 86% of total 930 Mbit

17 Jan 2002Peering altered 22 Jan

22 Jan 2002 Weed day peak 175 Mbit/s


14

Monitoring: US Traffic UKERNA Traffic data Kbit/s. Blue Traffic from US; Maroon Traffic to US7 Dec 2001 (900kbit/s) 29 Jan 2002 (175kbit/s)peak is 88% of total BW 930 Mbit10 minute averages 10 minute averages

Last 7 days 1 hour averages


15

Astronomy & Astrophysics Sites


16

ICFAMON Plot from DL to Anglo-Australian Observatory for 11th Apr to 24th Apr 2002

Connectivity to Australia

Packet loss reasonablertt improves ~420 ms to ~300 msVariations ~100ms


17

ICFAMON Plots for 11th Apr to 24th Apr 2002 DL to NOAO, Arizona DL to Goddard GSFC NASA

Connectivity to US


18

ICFAMON Plot from DL to The Joint Astronomy Centrefor 11th Apr to 24th Apr 2002

Connectivity to Hawaii

Packet loss goodrtt ~210 msVariations – queuing

traceroute:Cross SuperJANET4 to NY OKCross Abilene to Seattle OKEnters uhnetStops after 2-3 routers

No connectivity to La Palmatraceroute ends in iac.es network Tenerife ?


19

Grid Network Monitoring

Several tools in test – plugged into a coherent structure: PingER, RIPE one way times, iperf, UDPmon, rTPL, GridFTP, and

NWS prediction engine continuous tests for last few months to selected sites:

DL Man RL UCL CERN Lyon Bologna SARA NBI SLAC … The aims of monitoring for the Grid:

to inform Grid applications, via the middleware, of the current status of the network – input for resource broker and scheduling

to identify fault conditions in the operation of the Grid to understand the instantaneous, day-to-day, and month-by-month

behaviour of the network – provide advice on configuration etc.

Network information published in LDAP schema Will be used by UK GridPP and e-science centresAstroGrid ?


20

Local NetworkMonitoring

Store & Analysisof Data (Access)

Access to current and historic dataand metrics via the Web, i.e. WP7NM Pages, access to metric forecasts

Backend LDAP script to fetch metricsMonitor process to push metrics

localLDAPServer

Grid Application access viaLDAP Schema to- monitoring metrics; - location of monitoring data.

PingER(RIPE TTB)

IperfERUDPmon

rTPLNWS

etc

LDAPSchema

Grid AppsGridFTP

Network Monitoring Architecture

Robin Tasker


21

Network Monitoring Components

Ping Netmon UDPmon iPerf Ripe

Cronscript

plot

Table

LDAP

raw

control Cronscript

controlCronscript

plot

Table

LDAP

raw plot

Table

LDAP

raw

WEB Display AnalysisGrid BrokerPredictions

Web I/f

Scheduler

Tool

Clients

LDAP

raw

LDAP

raw


22

Ping & UDP throughput MAN-RAL From 20 Oct 01

PingER rtt (ms)

dl – RAL

1000 byte packet

Forecast

UDPmon Zero packet loss!

UDPmon throughput Mbit/s

man – RAL

300 * 1400 byte frames


23

Ping & UDP throughput MAN-CERNFrom 20 Oct 01

PingER rtt (ms)

dl – cern

1000 byte packet

Forecast


man – cern



24

iperf TCP & UDP throughput MAN-SARA From 20 Oct 01

Iperf TCP throughput Mbit/s

ucl – sara

262144 byte buffer

Forecast


man – sara



25

iperf & Pinger UK-Bologna From 20 Oct 01

Iperf throughput

ucl – Bologna

262144 byte buffer

Forecast in green

PingER rtt (ms)

dl – Bologna

1000 byte packet

Forecast


26Geant Enabled Routing Stable

Iperf ThroughputMbit/sUCL – SARA262144 byte

buffer

UDPmon Loss

ThroughputMbit/sMAN – SARA

iperf throughput UCL-SARA From 1 Nov 01 – Geant Operational


27

PingER deploymentLes Cottrell Measurements from

34 monitors in 14 countries Over 600 remote hosts Over 72 countries Over 3300 monitor-remote site pairs Measurements go back to Jan-95 Reports on RTT, loss, reachability, jitter, reorders, duplicates …

Countries monitored Contain 78% of world population 99% of online users of Internet

Lightweight (100bps/host pair) Very useful for inter-regional and poor links, need more intensive for high

performance & Grid sites


28

Losses: World by region, Jan ‘02 Packet loss <1%=good, <2.5%=acceptable, < 5%=poor, >5%=bad

Russia, S

America bad Balkans,

M East, Africa, S Asia, Caucasus poor

Monitored Region \ Monitor Country

BR (1)

CA (2)

DK (1)

DE (1)

HU (1)

IT (3)

JP (2)

RU (2)

CH (1)

UK (3)

US (16) Avg Region

Avg -(H

Avg NA + WEU + JP Pairs

COM 0.2 0.3 0.3 0.2 COM 0.27 23

Canada 1.8 1.6 0.3 0.5 9.0 0.3 1.4 21.7 0.7 0.7 0.5 3.5 Canada 0.74 126

US 0.4 2.6 0.2 0.3 8.0 0.1 1.4 13.8 0.3 1.3 0.9 2.7 US 0.88 2149

C America 0.9 0.9 C America 0.89 19

Australasia 0.8 1.8 1.3 Australasia 1.30 18

E Asia 1.2 3.5 1.0 1.1 9.0 0.9 2.0 5.2 1.5 1.4 1.5 2.6 E Asia 1.61 215

Europe 0.4 5.6 0.3 0.5 5.4 0.4 1.3 15.5 1.1 1.0 1.0 2.9 Europe 1.38 852

NET 1.7 6.2 1.0 1.3 8.0 1.6 3.6 21.9 0.7 0.8 0.9 4.3 NET 2.00 85

FSU- 4.5 0.5 9.8 0.5 1.6 11.2 4.3 1.2 2.0 4.0 FSU- 2.09 48

Balkans 3.8 3.8 Balkans 3.83 109

Mid East 4.6 1.4 3.0 8.5 2.8 3.2 11.8 2.0 2.5 2.1 4.2 Mid East 2.70 57

Africa 5.8 1.5 12.0 1.2 4.2 11.9 2.0 1.9 2.5 4.8 Africa 2.72 45

Baltics 5.3 0.8 2.3 7.7 2.2 3.5 10.8 4.8 2.1 3.9 4.3 Baltics 3.12 67

S Asia 1.6 7.3 0.1 3.1 9.2 3.0 3.9 17.9 1.5 3.1 3.0 4.9 S Asia 3.12 97

Caucasus 3.2 3.2 Caucasus 3.22 19

S America 24.1 11.3 0.6 0.9 6.7 12.9 7.7 23.0 9.3 1.1 6.6 9.5 S America 6.30 203

Russia 35.9 24.1 22.2 13.4 23.8 21.7 13.6 0.7 8.7 24.1 12.7 18.3 Russia 17.57 91

Avg 7.5 6.9 2.8 2.4 9.8 3.7 3.9 13.8 3.1 3.2 2.8 4.4 Avg 3.16

Pairs 64 144 54 67 70 203 190 114 209 192 1990 Pairs


29

Quality improvement seen

from SLAC & NASA

NASA results courtesy of Andy Germain, NASA, GSFC


30

Iperf mem-mem vs file copy disk to disk

0 400

100

Iperf TCP Mbits/s

File copy dis k-to- dis k

Fast Ethernet

OC3

Disklimited

Over 60Mbits/s iperf >> file copy


31

QoS: Terms and Concepts

Identifying frames – marking / setting IP precedence bitsSorting frames into queuesSelecting which frame to sendAction taken when a queue is full

Sort Fail

DiscardTest

Identify & ClassifyPolice

Dequeue

Configurable Queues


32

QoS: What Next?

Dante propose the following services: IP Premium (EF)

(AF) difficult to define in a way that suits most NRNs Best Efforts Scavenger “Less than best efforts”

UKERNA ran a Think Tank to Study QoS requirements in the UK MB-NG Network development project to test MPLS and QoS SuperJANET is expected to offer similar services to Dante

Applications need end to end QoS – so we need to cross: LAN SuperJANET4 Dante Remote NRN Remote LAN


33

More Information Some URLsPPNCG Home page with Stop Press:

http://ppncg.rl.ac.uk/

PPNCG Page for monitoring Astronomy & Astrophysics Sites http://icfamon.dl.ac.uk/ppncg/astronomy.html

and e-mail:[email protected]

DataGrid WP7 Networking: http://www.gridpp.ac.uk/wp7/index.html

IEPM PingER home site:http://www-iepm.slac.stanford.edu/

IEPM-BW site:http://www-iepm.slac.stanford.edu/bw

Documents

Introduction to the PPNCG Networking for the PPARC Community