Atlas Canada Lightpath Data Transfer Trial
Corrie Kost, Steve McDonald (TRIUMF)
Bryan Caron (Alberta), Wade Hong (Carleton)
The iGrid2002 Network
TRIUMF Local Loop
• Only one SM fiber pair available between TRIUMF and the CA*net 4 ONS at the BCnet POP• CWDM gear arrived at TRIUMF on Sept 4th• Extreme 10GbE LR blades arrived on the same day
CERN Local Loop
• Extreme BD with LR Blade arrives on Sept 17 within
2 days of shipping• ONS finally installed and operational on Sept 20th
We are live continent to continent!
• e2e lightpath up and running 21:45 CET
traceroute to cern-10g (192.168.2.2), 30 hops max, 38 byte packets 1 cern-10g (192.168.2.2) 161.780 ms 161.760 ms 161.754 ms
The Network Hardware
• At CERN and TRIUMF– Intel 10GbE Server NICs (alpha)– Extreme Networks Black Diamond 6808
• At TRIUMF– MRV Nbase Xyplex Fiber Driver WDM44s to
multiplex 2GbEs to CA*net 4 ONS at BCnet POP
Intel 10GbE Cards
• Intel kindly loaned us 2 of their Pro/10GbE LR server adapters cards despite the end of their Alpha program– based on Intel® 82597EX 10 Gigabit Ethernet
Controller
Extreme Networks
• Extreme Networks generously loaned and shipped to us 2 Black Diamond 6808s with their new10GbE LRi blades
Extreme NetworksTRIUMF CERN
The Endpoint Hosts
• Linux-based PC Servers– SuperMicro P4DL6 (Dual Xeon 2GHz)– ServerWorks GC-LE Chipset– 400 MHz front side bus– 1 GB DDR2100 RAM– Dual Channel Ultra 160 onboard SCSI– SysKonnect 9843 SX GbE– 3 independent PCI buses– PCI-X 64 bit/133 Mhz capable
IDE Disk Arrays
CERN Receive HostTRIUMF Send Host
Operating System
• Redhat 7.3 based Linux kernel 2.4.18-3– Needed to support filesystems > 1TB
• Upgrades and patches– Patched to 2.4.18-10– Intel Pro 10GbE Linux driver (early stable)– SysKonnect 9843 SX Linux driver (latest)– Ported Sylvain Ravot’s tcp tune patches
BBFTP Transfer
Vancouver ONSons-van01(enet_15/1)
Vancouver ONSons-van01(enet_15/2)
BBFTP Transfer
Chicago ONS
GigE Port 1
Chicago ONS
GigE Port 2
Tsunami Transfer
Vancouver ONSons-van01(enet_15/1)
Vancouver ONSons-van01(enet_15/2)
Tsunami Transfer
Chicago ONS
GigE Port 1
Chicago ONS
GigE Port 2
Sunday Nite Summaries
Tuesday Nite Summaries
Vancouver ONSons-van01(enet_15/1)
Vancouver ONSons-van01(enet_15/2)
Comparative ResultsTool Transferred Average Max Avg
wuftp 100 MbE 600 MB 3.4 Mbps
wuftp 10 GbE 6442 MB 71 Mbps
iperf 275 MB 940 Mbps 1136 Mbps
pftp 600MB 532 Mbps
bbftp (10 streams) 1.4 TB 666 Mbps 710 Mbps
Tsunami - disk to disk 0.5 TB 700 Mbps 825 Mbps
Tusnami - disk to memory 12 GB > 1GBps
Lessons Learned
• Linux software RAID faster than most conventional SCSI based systems
• More disk spindles the better• Channel bonding of two GbEs seems to
work very well (on an unshared link)• The larger files the better for thruput
Acknowledgements• Canarie
– Bill St. Arnaud, Rene Hatem, Damir Pobric, Thomas Tam, Jun Jian
• Atlas Canada– Mike Vetterli, Randall Sobie, Jim Pinfold, Pekka Sinervo, Gerald
Oakham, Bob Orr, Michel Lefebrve, Richard Keeler
• HEPnet Canada– Dean Karlen
• TRIUMF– Peter Gumplinger, Fred Jones, Mike Losty, Jack Chakhalian,
Renee Poutissou
• BCNET– Mike Hrybyk, Marilyn Hay, Dennis O’Reilly, Don McWilliams
Acknowledgements
• Extreme Networks– Amyn Pirmohamed, Steven Flowers, John Casselman, Darrell
Clarke, Rob Bazinet, Damaris Soellner
• Intel Corporation– Hugues Morin, Caroline Larson, Peter Molnar, Harrison Li, Layne
Flake, Jesse Brandeburg
Acknowledgements
• Indiana University– Mark Meiss, Stephen Wallace
• Caltech– Sylvain Ravot, Harvey Neuman
• CERN– Olivier Martin, Paolo Moroni, Martin Fluckiger, Stanley Cannon,
J.P Martin-Flatin
• SURFnet/Universiteit van Amsterdam– Pieter de Boer, Dennis Paus, Erik.Radius, Erik-Jan.Bos, Leon
Gommans, Bert Andree, Cees de Laat
Acknowledgements
• Yotta Yotta– Geoff Hayward, Reg Joseph, Ying Xie, E. Siu
• BCIT– Bill Rutherford
• Jalaam– Loki Jorgensen
• Netera– Gary Finley