Provision of networks for the LHC experiments in Germany

- Status 03/2006 -

K. Schauerhammer, K. Ullmann, (DFN)

2. / 3.3.2006

Meeting GridKa TAB

Karlsruhe

Seite 2

Contents

I. Technical options X-WiN and Geant2

II. Architecture LCG - Network (T0 - T1)

III. Performance Measurements

DFNInternet

IV. Connecting T2 centres in Germany

Seite 3

I. Technical options X-WiN and Geant2

Seite 4

X-WiN - status as of 1. 1. 2006

• Technical concept backbone (after Europe wide tender)– backbone consists of (dark) fibre and leased circuits– operational responsibility: DFN– more (than before) bought in (partial) services– much more (than before) DFN PoPs

• Access-Network very similar as in the past• Economic concept

– backbone put together from different service offerings from the market and integrate that under DFN responsibility

– most effective usage of competition on the market

Seite 520.02.2006

X-WiN (optical platform)

GAR

ERL

BAY

MUE

FZJ

AAC BIR

DES

HAM

POTTUB

FZK

GSI

DUI

BRE

HAN

BRA MAGBIE

FRA

HEI

STU

REG

DRE

CHE

ZIB

ILM

KIE

ROS

LEI

JENWEI

ESF

HUB

ADH

KAI

GOEKAS

MAR

GIE

Dark fibre

Leased circuits

GRE

80km toFrankfurt/O(Pionier/PL)

120km to Zürich(Switch/CH, GARR/IT)

50km toKehl/Straßburg

(Renater/FR)

50km toEnschede

(Surfnet/NL)

43 DFN PoPs

Seite 6

Optical platform based services

• Toolbox for the provision of– DFNInternet (2Mbps to 10 GE)– VPN-Services based on optical links– services like DFNVC, DFNPKI, DFN-CERT,

DFNRoaming, DFNNews etc. unchanged

• New cost structures for optical networks enable economic solutions for specialised services (Optical Private Networks / OPNs) i.e. Grids

Seite 7

General targets X-WiN design

• More performanceperformance increase by a factor of 4 since 01/06

more performance available (for the same price) as of 01/07

• More flexibilityno volume charging (and limitation)

Ethernet as additional access technology

Hybrid PoPs enable VPNs

• More availibilityduring design for backbone implicitly taken into

account

Seite 8

Availibility backbone by means of ...

GAR

ERL

BAY

MUE

FZJ

AAC BIR

DES

HAM

POTTUB

FZK

GSI

DUI

BRE

HAN

BRA MAGBIE

FRA

HEI

STU

REG

DRE

CHE

ZIB

ILM

KIE

ROS

LEI

JEN

ESF

HUB

ADH

• strongly protected physical topology – protected fibre paths– meshed fibre ring structure– more mathematical

optimisation will show, how to meaningful complement physical topology

Seite 9

Availibility backbone by means of...

• Fault tolerant logical topology, especially for DFNInternet

• Well-known optimisation for – logical links between PoPs – best mapping to physical topology

• Optimisation targets, especially– minimal delay– minimal loss of connectivity in case of severe

failures– best („smooth“) utilisation in case of failures

Seite 10

X-WiN DFNInternet as of 01/06

Logical IP backbone

Seite 11

worldwide Internet

worldwide Internet

Availibility backbone by means of...

User

X-WiN

At least two accesses to global IP networks, more redundant carriersFull throughput in case of failuresAdditional roughly 100 direct peerings

Redundant global connectivity by means of independant paths and PoPs assures protection against problems with Carriers (e.g. network failures or insolvency)

Redundant global connectivity by means of independant paths and PoPs assures protection against problems with Carriers (e.g. network failures or insolvency)

Redundant global connectivity

Seite 12

Dark Fibre

• Technical definition– ITU-T conform and "WDM-applicable"– some minimal requirements on technical

parameters• for example attenuation

• Service: – Support : 24/7 hotline and debug– provision of colocation space – End-to-end operational responsibility– usual time for reaction– integration into DFN defined processes

Seite 13

Equipment to „light“ Dark Fibre

• Technical definition:– equip with (several optional) digital interfaces– provision of a management system

• Service:– includes installation and maintenance – support: 24/7 hotline and debug– usual time for reaction– integration into DFN defined processes

Seite 14

Operations

• Definition– Monitoring of management data– pursue well defined procedures

• failure identification (devices? - links?)• alarm of responsible support groups• supervision of debug• if necessary escalation

– all procedures DFN defined– regular (i.e. monthly) reporting– open for integration of new devices

Seite 15

PoP A

PoP B

PoP C

Operational model X-WiN platform

dark fiber

Leased circuit

Access to management dataof DFN „owned“ devices

Operations Groupalarms and triggers support(links, devices)

Seite 16

GÉANT2 footprint

LU

RU

EELV

LT

DK

IT

FR

BE

CH

SI

AT

HR

PLDE

CZ

PT ES

IENL

UK

RO

BG

TR

CYIL

MT

GR

HU

SK

2

2

3

3

3

3

2

2

22

2

3

3 2

2

2

222

1 1

11

1

1

Seite 17

GÉANT2 Hybrid POP

Dark Fibre

Dark Fibre

N x 10G N x 10G

10G N x 10G

GÉANT2 POP

to NREN

Leased Service

Seite 18

Gigabit Ethernet p2p Service

GÉANTBorder

GÉANTBorder

Interconnect:N x physical GEth

GÉANT2

Interconnect:N x 10GEth

VLAN/10GEth(LAN or WAN)

Physical GEth

GEth (NREN transport)

GEth (GÉANT2 transport)

NREN A NREN B

Seite 19

II. Architecture LCG - Network (T0 - T1)

Seite 20

Architecture LCG network (1)

• Assumptions, design criteria:– bandwidths 10Gbit/s per T1 resp. multi 10Gbit/s

for T0– continuous stream of data– „keep it simple“ (as possible)– organise transport if possible in layer 2– security already considered during design

• system should be protected from unauthorised access • access from “trusted sources” not limited

Seite 21

Architecture LCG Network (2)

• Optical Private Network, consists of dedicated 10G links from T0 and any T1,

• T1 access in two options:– “Light path T1”– “Routed T1”

• Back-up for T0-T1 is part of the design

• T0 interface very likely 10GE LAN-PHY

Seite 22

Scalability of the architecture

• Options for transfer:– 10 Gbit/s ~ 1014 byte/d or 100 Tbyte/d– eleven 10G links -> more than 1 Pbyte/d or

roughly 0.5 Exabyte/a

• If one 10G link per T1 is not sufficient:– Installation of just another 10G link T0 - T1 if

possible on physically separated fibre path– architecture covers this completely

Seite 23

LCG T0 - T1 OPN

Seite 24

III. Performance Measurements DFNInternet

Seite 25

Issues in setting up a 10Gbit connetion between GridKa

and CERN openlab Forschungszentrum Karlsruhe GmbH

Institute for Scientific ComputingP.O. Box 3640

D-76021 Karlsruhe, Germanyhttp://www.gridka.de

Bruno Hoeft

Taken from:

Measurements DFNInternet

Seite 26

CERNGridFTPserver

GridFTPserver

WAN 2003/4 -- Gigabit GridKa – CERN

Géant10 Gbps

DFN

2.4 Gbps

GridFTP tested over 1 Gbps

Karlsruhe

Frankfurt

2x 1 Gbps

98% of 1Gbit

Geneva

1000

0

Seite 27

10Gigabit WAN SC GridKa – CERN

CERNGridFTPserver

GridFTPserver

Géant10 Gbps

DFN

10 Gbps

10 Gbps 10 Gbps

Karlsruhe

Frankfurt

- Bandwidth evaluation (tcp/udp)- MPLS via France (MPLS - MultiProtocol Label Switching)

- LBE (Least Best Effort)

- GridFTP server pool HD to HD Storage to Storage - SRM

Geneva

Seite 28

Evaluation of max. throughput

700

1400

5600

4900

4200

3500

2800

2100

7000

6300

Mbit/s

18:00 20:00

9 nodes each site-8 * 845 Mbit-1 * 540 Mbithigher speed at one stream is resulting in a packet loss

Seite 29

IV. Connecting T2 centres in Germany

Seite 30

LCG network services in Germany

• T0-T1 problem– 10 Gbit/s over European VPN (GridKa - CERN)– backup via CNAF or SARA

• T1-T2 problem– within DE no problem as most LHC sites are

close to X-WiN PoPs– for T2 in DE rough guess of traffic flows are

needed (source, destination, transferred volume per day, peak requirements?)

Seite 31

DESHAM

ZIB

HAN

LEIDUI

BIR

FRA

ERL

GAR

STU

MUE BIE

ROS

DORKAS

KOE

AAC

BON

FZK

TUM

AUG

REG

BAM

CHE

DREBOC

WUP

HEI

GSI

MAI

FRE

MAN

SIE

BRE

X-WiN-Nodes

LHC User Locations at X-WiN-Nodes

LHC User locations to be connected to X-WiN nodes

LCG Sites DE and X-WiN