US LHCNet Update Dan Nae California Institute of Technology [email protected] LHC OPN Meeting Munich, April 2007

US LHCNet UpdateUS LHCNet Update

Dan NaeDan Nae

California Institute of TechnologyCalifornia Institute of Technology

[email protected]@cern.ch

LHC OPN MeetingLHC OPN MeetingMunich, April 2007Munich, April 2007

LHC OPN Meeting, Munich

What is the US LHCNetWhat is the US LHCNet

A transatlantic network designed to support A transatlantic network designed to support the the LHCLHC and the U.S. HEP community and the U.S. HEP community

Funded by the US Funded by the US DoEDoE and and CERNCERN and and managed by managed by CaltechCaltech in collaboration with in collaboration with CERNCERN

Evolved from a network between US and Evolved from a network between US and CERN which dates back to 1985CERN which dates back to 1985

Our mission is to deliver a reliable network Our mission is to deliver a reliable network service to support the upcoming LHC service to support the upcoming LHC experiments at CERNexperiments at CERN

Designed to support the LHC three-tiered Designed to support the LHC three-tiered model and to deliver data directly to/from the model and to deliver data directly to/from the US Tier 1’sUS Tier 1’s ( (FNALFNAL and and BNLBNL))


Atlantic Ocean

Circuit StatusCircuit Status

NYC 111 8th

Pottington (UK)

VSNL South

NY60 Hudson

HighbridgeVSNL North

AMS-SARAAC-2

Bude

GVA-CERN

FrankfurtVSNL

Wal, NJ

London

Global Crossing

Qwest

Colt

GEANT

NYC-MANLAN

CHI-Starlight

Paris

BellportWhitesands

Unprotected circuits (lower cost) Service availability from provider’s offers:

Colt Target Service Availability is 99.5% Global Crossing guarantees Wave Availability at 98%

Canarie and GEANT: No Service Level Agreement (SLA)

LCG Availability LCG Availability requirement: 99.95%requirement: 99.95%


Circuit Status (cont.)Circuit Status (cont.)

- 6 x 10 Gbps6 x 10 Gbps SONET/10GbE WAN-PHY. SONET/10GbE WAN-PHY.- AmsterdamAmsterdam PoP installed in January. PoP installed in January.- Last circuit (Last circuit (Geneva-AmsterdamGeneva-Amsterdam) )

delivered at the end of last month.delivered at the end of last month.- Not as diverseNot as diverse as one might expect. as one might expect.- We’ve never had an isolated node so far.We’ve never had an isolated node so far.- It happened more than once that the It happened more than once that the

circuits went down in pairs (GC NYC-CHI circuits went down in pairs (GC NYC-CHI and NYC-AMS, Qwest NYC-CHI and CHI-and NYC-AMS, Qwest NYC-CHI and CHI-GVA).GVA).

- Rumors of a new RFP in May-June?Rumors of a new RFP in May-June?


Circuit Availability (March)Circuit Availability (March)

Proposal to produce a monthly reportProposal to produce a monthly report


““Old” Network Map (March ’07)Old” Network Map (March ’07)


Equipment StatusEquipment Status

- Next generation network based on Ciena Next generation network based on Ciena CoreDirector/CI – CoreDirector/CI – VCAT/LCAS/GFP capabilities.VCAT/LCAS/GFP capabilities.

- Four nodesFour nodes, one at each PoP to complement the , one at each PoP to complement the existing existing Force10Force10 boxes. boxes.

- Number of links on each box expected to Number of links on each box expected to increase each year; the upgrade plan has been increase each year; the upgrade plan has been discussed with Ciena up to discussed with Ciena up to ~2010.~2010.

- Deployment has already started (the Deployment has already started (the New YorkNew York and and ChicagoChicago nodes were installed, tested and nodes were installed, tested and commissioned by the commissioned by the end of last monthend of last month).).

- The The GenevaGeneva node is physically installed, will be node is physically installed, will be tested and most likely accepted next week.tested and most likely accepted next week.

- The fourth node (The fourth node (AmsterdamAmsterdam) expected to be ) expected to be operational in May.operational in May.


Past and Present IssuesPast and Present Issues

- At least one At least one Stratum 1 PRSStratum 1 PRS is recommended for normal is recommended for normal operation (still to be installed)operation (still to be installed)

- Second Second DC power feedDC power feed in Chicago is still to be installed in Chicago is still to be installed (today in non-redundant mode)(today in non-redundant mode)

- One ESLM card (10GbE) locked up during the migration of One ESLM card (10GbE) locked up during the migration of the the NYC-CHI GC circuit (fixed by hardware reset)the the NYC-CHI GC circuit (fixed by hardware reset)

- DC rectifier at CERN not fully populated (but enough for DC rectifier at CERN not fully populated (but enough for the moment)the moment)

- CERN commissioning has been CERN commissioning has been delayeddelayed due to due to complicated site access procedurescomplicated site access procedures

- We need more We need more operational experienceoperational experience with the boxes with the boxes- Currently only one circuit (GC NYC-CHI) runs over the new Currently only one circuit (GC NYC-CHI) runs over the new

equipment (migrated on equipment (migrated on March 29March 29thth))


Equipment Status TodayEquipment Status Today

- Three nodes delivered (GVA, CHI, NYC)Three nodes delivered (GVA, CHI, NYC)

- Two nodes installed (NYC, CHI)Two nodes installed (NYC, CHI)

- One circuit migrated (GC NYC-CHI)One circuit migrated (GC NYC-CHI)


Proposed Migration (May)Proposed Migration (May)

Two complementing networks:

- first network is essentially last year’s US LHCNet – stable, reliable, dedicated 10Gbps T0-T1 (and T1-T1)

- the second is based on the new equipment – dynamic or static point-to-point VCs between any of the four nodes

- the “new network” can be reconfigured to carry T0-T1 traffic in case of problems

Equipment diversity (more robust)


GEANT-ESNET Peering over US GEANT-ESNET Peering over US LHCNetLHCNet

The idea came up last year during the meeting at The idea came up last year during the meeting at FNALFNAL

US LHCNet would transparently carry a L1/L2 US LHCNet would transparently carry a L1/L2 Virtual CircuitVirtual Circuit between NYC and Amsterdam between NYC and Amsterdam

GEANT and ESNET peer over this VCGEANT and ESNET peer over this VC The The bandwidthbandwidth of the circuit is still to be of the circuit is still to be

discusseddiscussed An An MOUMOU between between ESNETESNET and and US LHCNetUS LHCNet was was

signed (with DOE’s blessing)signed (with DOE’s blessing) ESNETESNET will install a new 10 GbE connection to will install a new 10 GbE connection to

the US LHCNet CD/CI in the US LHCNet CD/CI in NYCNYC for this purpose for this purpose A new 10 GbE connection in A new 10 GbE connection in AmsterdamAmsterdam

between US LHCNet and between US LHCNet and GEANTGEANT is required is required


Proposed Migration (cont.)Proposed Migration (cont.)

Additional links are needed to make full use of the new infrastructure: Additional links are needed to make full use of the new infrastructure:

-New BNL/ESNET link in NYCNew BNL/ESNET link in NYC

- New FNAL/ESNET link in Chicago (or existing to be moved to the CD/CI)New FNAL/ESNET link in Chicago (or existing to be moved to the CD/CI)

- New LCG – CD/CI link(s) in GenevaNew LCG – CD/CI link(s) in Geneva

- New GEANT & SARA links in Amsterdam (T1-T1 and peerings)New GEANT & SARA links in Amsterdam (T1-T1 and peerings)


Network Forecast - 2007Network Forecast - 2007

Geneva

Chicago

New York

4 x 10GBE4 x 10GBE

4 x 10GBE

1 x 10G SDH

2 x 10G SDH

1 x 10G SDH

ESnet SDN

Amsterdam

1 x 10G SDH

2 x 10GBE

1 x 10G SDH

1 x 10G SDH 1 x 10G SDH

Ports Ports at Each at Each

PoPPoP

10 10 GbEGbE

OC-192OC-192/STM-64/STM-64

AMSAMS 22 22

GVAGVA 44 33

NYCNYC 44 55

CHICHI 44 44

Mostly sunny, scattered Mostly sunny, scattered cloudsclouds



Geneva

Chicago

New York

4 x 10GBE

4 x 10GBE

6 x 10GBE

2 x 10G SDH

1 x 10G SDH

ESnet SDN


2 x 10G SDH Amsterdam

1 x 10G SDH

2 x 10GBE

2 x 10G SDH


I2 NewNet

Ports at Ports at Each PoPEach PoP

10 GbE10 GbE OC-192OC-192/STM-64/STM-64

AMSAMS 22 33

GVAGVA 66 55

NYCNYC 44 77

CHICHI 44 55

Clouds gathering…Clouds gathering…



Geneva

Chicago

New York

4 x 10GBE

4 x 10GBE

8 x 10GBE

3 x 10G SDH

3 x 10G SDH

2 x 10G SDH

ESnet SDN



Amsterdam

2 x 10G S

DH

2 x 10GBE

2 x 10G SDH

1 x 10G S

DH


1 x

10G

SD

H

I2 NewNet

Hybrid Network

Ports at Ports at Each Each PoPPoP

10 10 GbEGbE

OC-192OC-192/STM-64/STM-64

AMSAMS 22 55

GVAGVA 88 77

NYCNYC 44 1212

CHICHI 44 1010

Overcast ;-)Overcast ;-)



Geneva

Chicago

4 x 10GBE

4 x 10GBE

10 x 10GBE

4 x 10G SDH

2 x 10G SDH

ESnet SDN

1 x 10G SDH 1x 10G SDH

3 x 10G SDH

New York

4 x 10G SDH

Amsterdam

3 x 10G S

DH

2 x 10GBE

3 x 10G SDH

1x 10G SDH

1x 1

0G S

DH

1x 10G S

DH

1x 10G SDH

3 x 10G SDH

I2 NewNetHybrid Network

Ports at Ports at Each Each PoPPoP

10 GbE10 GbE OC-192OC-192/STM-64/STM-64

AMSAMS 22 77

GVAGVA 1010 99

NYCNYC 44 1616

CHICHI 44 1212

Severe chances of high-speed data Severe chances of high-speed data transferstransfers


Ciena CD/CICiena CD/CI

- VCAT/LCAS/GFPVCAT/LCAS/GFP support – allows for support – allows for provisioning of dynamic circuitsprovisioning of dynamic circuits

- Difficult to manage with current network tools Difficult to manage with current network tools (no SNMP for example)(no SNMP for example)

- Integrated control plane functionality (still to be Integrated control plane functionality (still to be explored)explored)

- Need to be integrated with our Need to be integrated with our provisioning/monitoring tools (provisioning/monitoring tools (MonALISAMonALISA))

- Also selected by Internet2 for their Also selected by Internet2 for their NewNet NewNet installation (dynamic provisioning with installation (dynamic provisioning with DRAGON software)DRAGON software)

- Inter-domain integration for dynamic circuits is Inter-domain integration for dynamic circuits is desirable in the futuredesirable in the future


Ciena CD/CI (cont.)Ciena CD/CI (cont.)

- Bandwidth granularity of Bandwidth granularity of STS-1STS-1 (51Mbps) (51Mbps)- Bandwidth adjustments Bandwidth adjustments without interrupting the without interrupting the

circuitcircuit (very important for long-lived (very important for long-lived transatlantic data transfers)transatlantic data transfers)

- OSRP + Mesh restorationOSRP + Mesh restoration allows for easy allows for easy provisioning and re-routing of circuits over provisioning and re-routing of circuits over alternate pathsalternate paths

- Fast circuit restoration (Ciena claims 50ms)Fast circuit restoration (Ciena claims 50ms)- Associated circuit priorities (lower priority Associated circuit priorities (lower priority

circuits can be preempted during restoration)circuits can be preempted during restoration)- Need to be integrated with Need to be integrated with perfSONARperfSONAR

monitoring (help needed)monitoring (help needed)


Other Technical IssuesOther Technical Issues

The CD/CI management/monitoring is done only The CD/CI management/monitoring is done only via TL1 or GUI (NodeManager)via TL1 or GUI (NodeManager)

Circuit status is not reflected in the terminating Circuit status is not reflected in the terminating 10 GbE interface status (workaround – UDLD, 10 GbE interface status (workaround – UDLD, FAFD)FAFD)

We need to understand how to interact with the We need to understand how to interact with the CD/CI control planeCD/CI control plane

We need an API to work with for automated We need an API to work with for automated operations; apparently there is a CORBA operations; apparently there is a CORBA interface but we don’t have any specifications interface but we don’t have any specifications for it yet.for it yet.


US LHCNet Network MonitoringUS LHCNet Network Monitoring

Various monitoring tools are deployed Various monitoring tools are deployed - SpectrumSpectrum – used by the CERN operators (first – used by the CERN operators (first

line of support)line of support)- MonALISA MonALISA – used by the US LHCNet team to get – used by the US LHCNet team to get

a global view of all the servicesa global view of all the services- Various open-source toos (Various open-source toos (Nagios, Cricket, Nagios, Cricket,

rancid, RRDrancid, RRD) - used by the US LHCNet NOC for ) - used by the US LHCNet NOC for configuration tracking, logs and alarmsconfiguration tracking, logs and alarms

- perfSONARperfSONAR – used by the E2ECU – used by the E2ECU

Current perfSONAR installation extracts data Current perfSONAR installation extracts data from the Spectrum monitoring service at CERN; from the Spectrum monitoring service at CERN; we have plans to migrate this to a stand-alone we have plans to migrate this to a stand-alone setup (at least for the CD/CIs)setup (at least for the CD/CIs)


US LHCNet TeamUS LHCNet Team

- Harvey NewmanHarvey Newman – Project PI – Project PI- Dan NaeDan Nae – Network Engineer – tech team lead – Network Engineer – tech team lead- Ramiro VoicuRamiro Voicu – Software Engineer – Software Engineer- Two people left – Two people left – Sylvain RavotSylvain Ravot (October, 2006) (October, 2006)

and and Yang XiaYang Xia (March, 2007) (March, 2007) - Artur BarczykArtur Barczyk - new Network Engineer, started - new Network Engineer, started

on March 1on March 1st, st, 2007, based at CERN2007, based at CERN- Tony ChengTony Cheng – new Network Engineer, started – new Network Engineer, started

April 15April 15thth, based in Pasadena, CA, based in Pasadena, CA


Questions?Questions?

Thank You!Thank You!

Documents

US LHCNet Update Dan Nae California Institute of Technology [email protected] LHC OPN Meeting Munich, April 2007