Gigabyte Bandwidth Gigabyte Bandwidth Enables Global Co-Enables Global Co-LaboratoriesLaboratories

Gigabyte Bandwidth Gigabyte Bandwidth Enables Global Co-Enables Global Co-LaboratoriesLaboratories

Prof. Harvey Newman, CaltechProf. Harvey Newman, CaltechJim Gray, MicrosoftJim Gray, MicrosoftPresented at Windows Hardware Engineering ConferencePresented at Windows Hardware Engineering ConferenceSeattle, WA, 2 May 2004Seattle, WA, 2 May 2004

Analysis and Analysis and developmentdevelopment

GlobalGlobal

DistributedDistributed

Cooperative Cooperative

Massive amounts Massive amounts of dataof data

~1 Gigabytes per second~1 Gigabytes per second

~20 Petabytes per year~20 Petabytes per year

Gigabyte bandwidth Gigabyte bandwidth requiredrequired

200720075,000+5,000+

PhysicistsPhysicists250+250+ InstitutesInstitutes

60+60+CountriesCountries

TodayToday500+ Physicists500+ Physicists

100+ 100+ InstitutesInstitutes

35+ 35+ CountriesCountries

CMS

Atlas

Challenges Faced by Challenges Faced by Global Science Global Science ExperimentsExperiments

3030 PetPetabytes by 2008abytes by 20081 Exabyte by 20141 Exabyte by 2014

~5 GBps~5 GBpsCERNCERN

Filter

Tier 0 +1Tier 0 +1

Tier 2Tier 2

Tier 3Tier 3

Tier 1Tier 1 ……INP3 RAL INFN FNAL

Tier 2

Institute

Tier 2Tier 2Tier 2Tier 2

Institute Institute Institute

Tier 4Tier 4

ExperimentExperiment ~1 GBps~1 GBps

~PBps~PBps

.1 GBps.1 GBps

Physics Physics data data

cachecache

~1 GBps~1 GBps

~1 GBps~1 GBps

WorkstationsWorkstations

OC192 = 9.9 Gbps

6.8 gbps6.8 gbps~ 850 mBps~ 850 mBps

Send data from CERN to Caltech – FastSend data from CERN to Caltech – FastUse standard TCP/IPUse standard TCP/IPRules set by Internet2 @ Rules set by Internet2 @ http://lsr.internet2.edu/http://lsr.internet2.edu/

OC 192 = 9.9 Gbps

New speed record, multiple parallel streams:New speed record, multiple parallel streams:6.2 gbps ~ 800 mBps6.2 gbps ~ 800 mBps

mb

ps p

er

secon

dm

bp

s p

er

secon

d00

1,0001,000

2,0002,000

3,0003,000

4,0004,000

5,0005,000

6,0006,000

7,0007,000

20002000 20012001 20022002 20032003 20042004 20052005

OC 192 = 9.9 Gbps

4:42 AM4:42 AM

6.8 6.8 GbpsGbps

Windows Disk IO Windows Disk IO Bandwidth Bandwidth

Details at: Details at: http://research.microsoft.com/~peteku/http://research.microsoft.com/~peteku/

WorkstationWorkstation ServerServer

2 x AMD Opteron™2 x AMD Opteron™ 4 x AMD Opteron™4 x AMD Opteron™

Tyan motherboardTyan motherboard Newisys systemNewisys system

SATA + NTFS SATA + NTFS sequentialsequential

SATA + NTFS SATA + NTFS sequentialsequential

20 disks20 disks 48 disks48 disks

1.0+ GBps 1.0+ GBps read/writeread/write

2.0+ GBps2.0+ GBps read/writeread/write

Beyond TCP/IP Beyond TCP/IP

Predictable stabilityPredictable stability

Better error and congestion controlBetter error and congestion control

Hybrid networks with dynamic optical Hybrid networks with dynamic optical pathspaths

Higher speed linksHigher speed links

Truly global collaboration and Truly global collaboration and virtual organizationsvirtual organizations