Atlas Canada Lightpath Data Transfer Trial Corrie Kost, Steve McDonald (TRIUMF) Bryan Caron...

Atlas Canada Lightpath Data Transfer Trial

Corrie Kost, Steve McDonald (TRIUMF)

Bryan Caron (Alberta), Wade Hong (Carleton)

The iGrid2002 Network

TRIUMF Local Loop

• Only one SM fiber pair available between TRIUMF and the CA*net 4 ONS at the BCnet POP• CWDM gear arrived at TRIUMF on Sept 4th• Extreme 10GbE LR blades arrived on the same day

CERN Local Loop

• Extreme BD with LR Blade arrives on Sept 17 within

2 days of shipping• ONS finally installed and operational on Sept 20th

We are live continent to continent!

• e2e lightpath up and running 21:45 CET

traceroute to cern-10g (192.168.2.2), 30 hops max, 38 byte packets 1 cern-10g (192.168.2.2) 161.780 ms 161.760 ms 161.754 ms

The Network Hardware

• At CERN and TRIUMF– Intel 10GbE Server NICs (alpha)– Extreme Networks Black Diamond 6808

• At TRIUMF– MRV Nbase Xyplex Fiber Driver WDM44s to

multiplex 2GbEs to CA*net 4 ONS at BCnet POP

Intel 10GbE Cards

• Intel kindly loaned us 2 of their Pro/10GbE LR server adapters cards despite the end of their Alpha program– based on Intel® 82597EX 10 Gigabit Ethernet

Controller

Extreme Networks

• Extreme Networks generously loaned and shipped to us 2 Black Diamond 6808s with their new10GbE LRi blades

Extreme NetworksTRIUMF CERN

The Endpoint Hosts

• Linux-based PC Servers– SuperMicro P4DL6 (Dual Xeon 2GHz)– ServerWorks GC-LE Chipset– 400 MHz front side bus– 1 GB DDR2100 RAM– Dual Channel Ultra 160 onboard SCSI– SysKonnect 9843 SX GbE– 3 independent PCI buses– PCI-X 64 bit/133 Mhz capable

IDE Disk Arrays

CERN Receive HostTRIUMF Send Host

Operating System

• Redhat 7.3 based Linux kernel 2.4.18-3– Needed to support filesystems > 1TB

• Upgrades and patches– Patched to 2.4.18-10– Intel Pro 10GbE Linux driver (early stable)– SysKonnect 9843 SX Linux driver (latest)– Ported Sylvain Ravot’s tcp tune patches

BBFTP Transfer

Vancouver ONSons-van01(enet_15/1)

Vancouver ONSons-van01(enet_15/2)

BBFTP Transfer

Chicago ONS

GigE Port 1

Chicago ONS

GigE Port 2

Tsunami Transfer

Vancouver ONSons-van01(enet_15/1)

Vancouver ONSons-van01(enet_15/2)

Tsunami Transfer

Chicago ONS

GigE Port 1

Chicago ONS

GigE Port 2

Sunday Nite Summaries

Tuesday Nite Summaries

Vancouver ONSons-van01(enet_15/1)

Vancouver ONSons-van01(enet_15/2)

Comparative ResultsTool Transferred Average Max Avg

wuftp 100 MbE 600 MB 3.4 Mbps

wuftp 10 GbE 6442 MB 71 Mbps

iperf 275 MB 940 Mbps 1136 Mbps

pftp 600MB 532 Mbps

bbftp (10 streams) 1.4 TB 666 Mbps 710 Mbps

Tsunami - disk to disk 0.5 TB 700 Mbps 825 Mbps

Tusnami - disk to memory 12 GB > 1GBps

Lessons Learned

• Linux software RAID faster than most conventional SCSI based systems

• More disk spindles the better• Channel bonding of two GbEs seems to

work very well (on an unshared link)• The larger files the better for thruput

Acknowledgements• Canarie

– Bill St. Arnaud, Rene Hatem, Damir Pobric, Thomas Tam, Jun Jian

• Atlas Canada– Mike Vetterli, Randall Sobie, Jim Pinfold, Pekka Sinervo, Gerald

Oakham, Bob Orr, Michel Lefebrve, Richard Keeler

• HEPnet Canada– Dean Karlen

• TRIUMF– Peter Gumplinger, Fred Jones, Mike Losty, Jack Chakhalian,

Renee Poutissou

• BCNET– Mike Hrybyk, Marilyn Hay, Dennis O’Reilly, Don McWilliams

Acknowledgements

• Extreme Networks– Amyn Pirmohamed, Steven Flowers, John Casselman, Darrell

Clarke, Rob Bazinet, Damaris Soellner

• Intel Corporation– Hugues Morin, Caroline Larson, Peter Molnar, Harrison Li, Layne

Flake, Jesse Brandeburg

Acknowledgements

• Indiana University– Mark Meiss, Stephen Wallace

• Caltech– Sylvain Ravot, Harvey Neuman

• CERN– Olivier Martin, Paolo Moroni, Martin Fluckiger, Stanley Cannon,

J.P Martin-Flatin

• SURFnet/Universiteit van Amsterdam– Pieter de Boer, Dennis Paus, Erik.Radius, Erik-Jan.Bos, Leon

Gommans, Bert Andree, Cees de Laat

Acknowledgements

• Yotta Yotta– Geoff Hayward, Reg Joseph, Ying Xie, E. Siu

• BCIT– Bill Rutherford

• Jalaam– Loki Jorgensen

• Netera– Gary Finley