Jumbo Packets for Application Performance Joint Techs Workshop Vancouver, BC 2005

ApparentNetworks

ApparentNetworks

9k APP Project

ResearchRutherford

ResearchRutherford

Jumbo PacketsJumbo Packets forfor

Application PerformanceApplication Performance

Joint Techs Workshop

Vancouver, BC

2005

Research Rutherford Apparent

Networks

CA*net4 – 9k CA*net4 – 9k AppApp Project Project

HEPnet

WestGrid

Subatomic Physics


Networks

Internet2 9k MTU ProjectInternet2 9k MTU Project BillBill Rutherford (Rutherford Research Rutherford (Rutherford Research/RRX/RRX))

Kevin WalshKevin Walsh (Cal-ITEC/(Cal-ITEC/SDSCSDSC))

John Moore (NC-ITEC/NCSU)John Moore (NC-ITEC/NCSU)

Loki Jorgenson (Apparent Networks/SFU)Loki Jorgenson (Apparent Networks/SFU)

Paul Schopis Paul Schopis (ITEC-Ohio(ITEC-Ohio//OARnet)OARnet)

Jorge Serran Jorge Serran (CUDI2/UNAM)(CUDI2/UNAM)

Dave Hartzell (NASA Ames Research Center)Dave Hartzell (NASA Ames Research Center)

Bill Jones (UBill Jones (University of niversity of TTexasexas-Austin)-Austin)

Woojin Seok (Woojin Seok (Supercomputing Korea/Supercomputing Korea/KISTI)KISTI)


Networks

CA*net4 9k CA*net4 9k AppApp Project Project BillBill Rutherford Rutherford (Rutherford Research (Rutherford Research/RRX – Project Coordinator)/RRX – Project Coordinator)

Loki JorgensonLoki Jorgenson (Apparent Networks/SFU (Apparent Networks/SFU – Project Coordinator– Project Coordinator))

Thomas TamThomas Tam (CANARIE/ (CANARIE/CA*net4 – CANARIE/UCLP CoordinatorCA*net4 – CANARIE/UCLP Coordinator))

Bryan CaronBryan Caron (TRIUMF/ (TRIUMF/UofAlberta – TRIUMF/UCLP CoordinatorUofAlberta – TRIUMF/UCLP Coordinator))

Randy Sobie Randy Sobie (HEPnet/(HEPnet/UVic – HEPnet UVic – HEPnet President/Grid IntegrationPresident/Grid Integration))

Brian CorrieBrian Corrie (WestGrid/IRMACS/SFU - IRMACS (WestGrid/IRMACS/SFU - IRMACS CoordinatorCoordinator))

Rob BallantyneRob Ballantyne (IRMACS/SFU - IRMACS (IRMACS/SFU - IRMACS Network CoordinatorNetwork Coordinator))

Martin SiegertMartin Siegert (WestGrid/SFU – WestGrid/GridStore (WestGrid/SFU – WestGrid/GridStore CoordinatorCoordinator))

Dave BickleDave Bickle ((HEPnet/HEPnet/UVic – HEPnet UVic – HEPnet Coordinator/Grid IntegrationCoordinator/Grid Integration))

Ken HowardKen Howard ((Network Services/Network Services/UVic – Network Coordinator)UVic – Network Coordinator)

Peter van EppPeter van Epp ((Network Services/Network Services/SFU – Network Coordinator)SFU – Network Coordinator)


Networks

9k MTU9k MTU > App > > App > DDSDDS

9k MTU Project

Core R&ERouter Troubleshooting

Large Xfer MeasurementsInternational Links

Internet 2 SponsoredPhysics Participants

Manual LightpathTRIUMF to CERN TestFew 9k Taps onto Core

Main Tap SDSC

3 yrs

9k App Project

Lightpath R&ENon-Routed… UCLP

Build Dedicated 9k LinksZX GBICs/quad gigE/campus MM

CANARIE NETERA BCNET 3 Universities Only 9k Link on Campus

Physics & Biochem ParticipantsHEPnet - WestGrid - TRIUMF

9k Node - HPC - 9k NodeViz HPC 9k TestsNFS HPC 9k Tests

1 yrs

9k DDS Project

Lightpath R&EClone-Tune 9k APP… Handoff

Provide 9k Switch BiochemUVic - SFU - UofA

Prelim 9k SOA FrameworkDrug Discovery System Flow

Tune 9k SOA FrameworkIntegrate 9k SOA Framework

Demo 9k SOA-DDS Based Collab9k SOA-DDS Software Dev Examples

Complete Handoff … ;-)

2 yrs


Networks

What are Jumbo Packets ?What are Jumbo Packets ?PRE

MAC/LLC

IP Header TCP Header Payload Data FCSIFG PREMAC/LLC

IP Header TCP Header Payload Data FCSIFG

OSILayer

Description

7 Application6 Presentation5 Session4 Transport3 Network2 Data Link1 Physical

PREMAC/LLC


MSS(1460bytes)

Maximum Segment Size (MSS)

PREMAC/LLC


Packet (1500 bytes = MTU)

Maximum Transmission Unit (MTU) = Packet

PREMAC/LLC


Frame (1518 bytes)

Frame

Why 9K ?


Networks

Internet2 9K Internet2 9K MTUMTU Project - 2003 Project - 2003GigE 2-way bandwidth vs. MTU

from Kansas City to various universities

0

200

400

600

800

1000

1200

1400

1600

1800

2000

0 2000 4000 6000 8000 10000

MTU size (bytes)

2-w

ay

Ba

nd

wid

th (

Mb

ps

)GigE 2-way bandwidth vs. MTU

from Kansas City to various universities

0

200

400

600

800

1000

1200

1400

1600

1800

2000

0 2000 4000 6000 8000 10000

MTU size (bytes)

2-w

ay

Ba

nd

wid

th (

Mb

ps

)

Standard 1500 M

TU


Networks

CA*net4 9K CA*net4 9K MTUMTU Project - 2004 Project - 2004

CA*net4 - MTU PerformanceVancouver, Ottawa, Dalhousie, Edmonton

0

200

400

600

800

1000

1200

1400

1600

1800

2000

0 2000 4000 6000 8000 10000

MTU Size (bytes)

2-w

ay

Ba

nd

wid

th (

Mb

ps

)CA*net4 - MTU Performance

Vancouver, Ottawa, Dalhousie, Edmonton

0

200

400

600

800

1000

1200

1400

1600

1800

2000

0 2000 4000 6000 8000 10000

MTU Size (bytes)

2-w

ay

Ba

nd

wid

th (

Mb

ps

)

Standard 1500 M

TU


Networks

Approaching Application PerformanceApproaching Application Performance

Metrics for network applicationsMetrics for network applications

E.g. MOS as indication of VoIP performanceE.g. MOS as indication of VoIP performance

Ranges of network behaviorsRanges of network behaviors

Models of dependenceModels of dependence

ITU Y-1541 exemplarITU Y-1541 exemplar


Networks

Application Test BedApplication Test Bed

CANARIE

ONS15454

BCNET

ONS15454

CANARIENETERA

ONS15454

CANARIENETERA

VANCOUVER CALGARY EDMONTON

Enterasys N7

STS-24c gigE

gigE

Cisco ONS 10,000 bytes

10,239 bytes

gigELinux

gigE

Dell L3

9,018 bytes

UofA

Physics

ONS15454

CANARIEBCNET

VICTORIA

gigE gigE

Cisco 6509

UVic

Network Services

Linux

gigE

lightpath.phys.ualberta.caRouted IPv4 = 129.128.241.113Lightpath IPv4 = 172.31.241.113Reference IPv4 = 10.128.241.113

phys02.comp.uvic.caRouted IPv4 = 142.104.21.13Lightpath IPv4 = 172.31.21.13Reference IPv4 = 10.104.21.13

STS-24c

STS-24c

UCLP STS-24c

gigE

SM 22 km

long range ZX GBICsfrom CANARIE

HEPnet

Port_3 ZX

172.31 VLAN

ZX GBICZX GBIC SONET SONET

SONET9,216 bytes

172.31 VLAN

172.31 VLAN

172.31 VLAN

172.31 VLAN

dedicatedgigE port

IBM p650 AIX

Routed IPv4 = 206.12.24.65Lightpath IPv4 = 172.31.24.65Reference IPv4 = 10.12.24.65

gridstore.westgrid.ca

7

0 dual port

TRIUMFgigE

campus router

campus router

dual port

route all 172.31.0.0/16(172.31) to one VLAN of 6 on4 port gigE link aggregator

WestGrid

SFU

Enterasys ER16

64,000 bytes

IRMACS

Cisco 6509

9,216 bytes

172.31VLAN

MM

SGIIRMACS

ONYX 3000

note: possibly use unrouted black hole range10.12.24.0/22 to shadow routed IPv4 fromWestgrid 206.12.24.0/22 for lightpaths ... ?... use 172.31 due to conflicts ...

vizserver.westgrid.caRouted IPv4 = 206.12.24.8Lightpath IPv4 = 172.31.24.8Reference IPv4 = 10.12.24.8

172.31 VLANgigE

MM

phase 2 reconfigure N7 inseries between ER16 andCisco 6509

phase 1 direct path fromER16 to Cisco 6509 (ZX GBICslot not available on 6509)

gigE

MM


Networks

9k 9k APPAPP Project – Lightpaths Project – Lightpaths


Networks

9k 9k APPAPP Project - Short Circuit Routing Project - Short Circuit Routing

NIC-0

NIC-1

NIC-0

NIC-1

Routed

Lightpath Short Circuit

Machine 0 dual port Machine 1 dual port

IPv4-0-1

IPv4-0-0

IPv4-1-1

IPv4-1-0

APP-0 APP-1

always on

on/off

APP-Socket-0-1

VM/OS-Socket-0-1

Handler TCP-Session-0-1

TCP-Socket/Port-0-1

LLC-Socket/Port/ARP-0-1

MAC-0-1

Local map unrouted black holeIPv4 to start ... try routed switchover later... ?

1.5k

9k


Networks

Approaching User ExperienceApproaching User Experience User-driven application performanceUser-driven application performance Examples… Examples…

Interaction with 3D modelsInteraction with 3D models Collaboration with multiple Collaboration with multiple

models/data/voice/videomodels/data/voice/video Massive data set manipulationMassive data set manipulation

Typical applicationsTypical applications 3D Visualization on remote server3D Visualization on remote server Distributed file systemsDistributed file systems


Networks

9k 9k APPAPP – Test Results – Test Results

0

200

400

600

800

1000

1200

1400

1600

1800

2000

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000

0

200

400

600

800

1000

1200

1400

1600

1800

2000

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000

Raw performance

SFU to Devstore UoA to Devstore


Networks

9k 9k APPAPP – File System Results – File System Results

0

5000

10000

15000

20000

25000

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000

Raw performance

SFU to Devstore SFU to Devstore

Bonnie performance

0

200

400

600

800

1000

1200

1400

1600

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000


Networks


0

10000

20000

30000

40000

50000

60000

70000

80000

90000

100000

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000

Raw performance

SFU to Vizserver SFU to Vizserver

Bonnie performance

0

200

400

600

800

1000

1200

1400

1600

1800

2000

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000


Networks


Data Missing

Raw performance

UoA to Vizserver UoA to Vizserver

Bonnie performance

0

500

1000

1500

2000

2500

3000

3500

4000

0 1000 2000 3000 4000 5000 6000

0

200

400

600

800

1000

1200

1400

0 1000 2000 3000 4000 5000 6000 7000 8000 9000


Networks

NEXT PHASENEXT PHASE9k 9k DDSDDS Project – Project – Drug Discovery SystemDrug Discovery System

• Combined Physics, Viz, Grid, Bio

• Preliminary integration (SOA)• Collaborative viz • Distributed files (databases)• Instruments (XRD, NMR)• GTK4 integration• UCLPv2 integration


Networks

9k 9k DDS DDS Project – Network OverviewProject – Network Overview

CANARIE

ONS15454

BCNET

ONS15454

CANARIENETERA

ONS15454

CANARIENETERA

VANCOUVER CALGARY EDMONTON

Enterasys N7

STS-24c

gigE

Cisco ONS 10,000 bytes

10,239 bytes

gigE

ONS15454

CANARIEBCNET

VICTORIA

gigE

STS-24c

STS-24c

note: available bandwidth on "lightpath" e2e is dependenton configuration of ONS15454 and activity of ports

UCLP STS-24c

gigE

SM 22 km

long range ZX GBICsfrom CANARIE

Port_3 ZX

10/8 VLAN

ZX GBICZX GBIC SONET SONET

SONET

10/8 VLAN10/8 VLAN

dedicatedgigE port

IBM p650 AIX

Routed IPv4 = 206.12.24.65Lightpath IPv4 = 10.12.24.65

gridstore.westgrid.ca

7

0

route all 10.0.0.0/8 (10/8)todedicated port of 8 portgigE link aggregator

route all 10.0.0.0/8 (10/8)toone VLAN of 6 on 4 portgigE link aggregator

WestGrid

SFU

Enterasys ER16

64,000 bytes

IRMACS

Cisco 6509

9,216 bytes

10/8VLAN

MM

SGIIRMACS

ONYX 3000

note: possibly use unrouted black hole range10.12.24.0/22 to shadow routed IPv4 fromWestgrid 206.12.24.0/22 for lightpaths ... ?

vizserver.westgrid.caRouted IPv4 = 206.12.24.8Lightpath IPv4 = 10.12.24.8

10/8 VLANgigE

MM

9k APP phase 2 reconfigureN7 in series between ER16and Cisco 6509

9k APP phase 1 direct pathfrom ER16 to Cisco 6509(ZX GBIC slot not availableon 6509)

gigEMM gigE

LinuxgigE

Dell L39,018 bytes

UofA

Physics

lightpath.phys.ualberta.caRouted IPv4 = 129.128.241.113Lightpath IPv4 = 10.128.241.113

10/8 VLAN

dual port

TRIUMFgigE

campus router

gigE

LinuxgigE

9k Switch

UofABiochemistry

lightpath.pence.caRouted IPv4 = 129.128.139.2XXLightpath IPv4 = 10.128.139.2XX

10/8 VLAN

dual port

PENCEgigE

campus router

Supplied by Pence

Supplied and set up byTRIUMF, reimbursedby 9k DDS Project

gigE

gigE

Cisco 6509

UVicNetwork Services

Linux

phys02.comp.uvic.caRouted IPv4 = 142.104.21.13Lightpath IPv4 = 10.104.21.13

HEPnet9,216 bytes

10/8 VLAN

campus router

dual port

Supplied by TVBR

gigE

gigE

UVicBiochemistry & Microbiology

Linux

lightpath.bioc.uvic.caRouted IPv4 = 142.104.33.XXXLightpath IPv4 = 10.104.33.XXX

TVBR

10/8 VLAN

campus router

dual port

9k Switch

Supplied and set up byHEPnet, reimbursedby 9k DDS Project

gigE

SFUMolecular Biology and Biochemistry

10/8 VLAN

9k Switch

Supplied and set up byWestGrid, reimbursedby 9k DDS Project

Linux

lightpath.mbb.sfu.caRouted IPv4 = 142.58.213.XXXLightpath IPv4 = 10.58.213.XXX

MBB

campus router

dual portSupplied by MBB

9k DDS phase 2reconfigure todedicated gigE portfrom ONS


9k DDS phase 1 tapTRIUMF gigE portfrom ONS for setup

9k DDS phase 1 tap HEPnetgigE port from ONS for setup


9k DDS phase 1 tap WestGrid-IRMACS gigE port from ONSfor setup


Networks

End of Presentation

Documents

Jumbo Packets for Application Performance Joint Techs Workshop Vancouver, BC 2005