WLCG 1 Service Challenge 4: Preparation, Planning and Outstanding Issues at INFN [email protected] Workshop sul Calcolo e Reti dell'INFN Jun

1

WLCG

Service Challenge 4:

Preparation, Planning and

Outstanding Issues at INFN

[email protected]

Workshop sul Calcolo e Reti dell'INFN Jun 7, 2006

2

WLCGOutline

1. Service Challenge 4: overall schedule

2. SC4 service planning at T1 and T2 sites

3. Results of pre-SC4 testing at INFN

4. Accounting results

5. VO Box

6. Outstanding issues

3

WLCGSC Milestones 2006

January SC3 disk repeat – Nominal rate (200 MB/s) capped at 150MB/s/ February CHEP Workshop; T1-T1 Use Cases,

SC3 disk - tape repeat (50MB/s, 5 drives)March Detailed plan for SC4 service agreed (M/W + DM service

enhancements). gLite 3.0 release beta testing

April SC4 disk – disk (200 MB/s) and disk – tape (75 MB/s) throughput tests.

gLite 3.0 release available for distribution May Installation, configuration and testing of gLite 3.0 release at sitesJune Start of SC4 production tests by experiments of ‘T1 Use Cases’

T2 Workshop: identification of key Use Cases and Milestones for T2s July Tape throughput tests at full nominal rates! August T2 Milestones – debugging of tape results if needed September LHCC review – rerun of tape tests if requiredOctober WLCG Service Officially opened. Capacity continues to build upNovember 1st WLCG ‘conference’. All sites have network / tape h/w

in production(?) December Final service / middleware review leading to early 2007

upgrades for LHC data taking??

4

WLCGINFN sites in SC4

• ALICE– Catania– Torino– (Bari TBD)– (Legnaro TBD)

• ATLAS– Frascati (in)– Napoli– Milano (sj)– Roma

• CMS– Bari (in)– Legnaro– Pisa (sj)– Roma

• LHCb– CNAF

5

WLCG

SC

4 gL

ite

3.0

Ser

vice

Pla

nnin

gCOMPONENT Tier-1 Tier-2 Comments

VOMS server ATLAS, LHCb - ALICE: server instance only needed either at T0 or T1

CMS: server at T0

Myproxy server - - ALICE: server instance only needed either at T0 or T1

CMS/LHCb: server at T0

BDII/GLUE ALICE ALICE CMS/LHCb: server at CERN

FTS server ALL -

Local File Catalogue

ALICE, ATLAS, LHCb

(CMS: backup solution)

ALICE LHCb: in Lyon, RAL e CNAF, a local read-only replica of the central master is required (to be synchronized every 30’)

Lcg-utils ALL ALL

GFAL ATLAS, LHCb ATLAS, LHCb

Resource Broker Glite 3.0

ALICE,

(ATLAS),

(LHCb)

ALICE ATLAS: at T0, but some instances in a few T1 sites for redundancy purposes, would be useful

CMS: server at T0 can suffice

LHCb: at T0, both LCG and glite versions (another instance, in some T1s may be useful for redundancy)

CE: classic

and gLite 3.0

ALL ALL ALICE: CE gLite not necessarily required in every T1 and T2

CMS: CE gLite only at T1

LHCb: CE classic only

RGMA/GridICE ALICE, ATLAS ALICE, ATLAS

SRM 1.1 ALL ALL CMS: using SRM 2.1 (dCache) in US

VOBOX ALICE, ATLAS

CMS, LHCb

ALICE, CMS ALICE: VOBOX + gLite UI

APEL/DGAS ALICE, ATLAS, CMS ALICE, ATLAS, CMS

6

WLCGSC4 Preparation

• Activities: – Mar: Installation and testing of gLite 3.0 in PPS

passed, with the exception of the FTS service (Oracle backend), under finalization

– Apr, May: testing of CERN to CNAF disk-disk throughput target rate (200 MB/s) with Castor2 ongoing

– May: testing of CERN to CNAF disk-tape throughput target rate (75 MB/s) with Castor2 passed

– May 22: start of deployment of gLite 3.0.0 RC5 (released on May 4) at CNAF (all WN, CE gLite, CE LCG still waiting for upgrade, some UIs, FTS)

7

WLCGDisk – Tape Throughput (CERN – CNAF)

• Tue Apr 19 to Thu Apr 27: estimated overall rate to tape 76.94 MB/s• Target: 75 MB/s• Estimated Daily Rate to Tape (MB/s):

Wed 19 5.32 MB/s Thu 20 103.80 MB/s Fri 21 148.00 MB/s Sat 22 25.46 MB/s Sun 23 11.60 MB/s Mon 24 62.50 MB/s Tue 25 62.50 MB/s Wed 26 91.34 MB/s Thu 27 182.00 MB/s

• 6 tape drives • long down times of the WAN transfer sessions from CERN to CNAF

due to LSF Castor2 issues already experienced during the disk-disk throughput phase in April

8

WLCGDisk-disk throughput (CERN – CNAF) 1/3

• Apr: LSF issues caused instability and seriously limited the avg throughout• May 2: Castor2 upgrade to version 2.0.4-0; • May 3: local write tests and remote transfers (few concurrent file transfers). Results show

good local Castor-2-to-LSF interaction. Power problem in the Tier-1 premises in the afternoon;

• May 4: transfers re-activated by gradually increasing the number of concurrent transfers by steps of 10 files. Throughput increases linearly (10 files -> 100 MB/s, 40 files -> 200 MB/s)

• Problems with the name server DB (this service is shared by Castor1 and Castor2). Tests with 50 concurrent files, 2 parallel streams: 1800 Mb/s (200 MB/s)

• Stable run at 200 MB/s for approximately 1 and a half day (until May 6, 8 p.m.)

9


• Average always around 170 MB/s bottleneck exists, still to be understood

• Daily statistics (May 26)

10


• RTT: 11.1 ms, default tx/rx socket buffer: 2 MB (tuning of TCP kernel parameters - net.core.rmem_max, net.core.wmem_max, net.core.rmem_default and net.core.wmem_default - on the 4 file servers does not improve the overall performance)

• From 4 to 5 file servers: no improvement, back to 4• Aggregate local write performance: more than 350 MB/s• LSF: increase of the number of slots per file server (i.e. the

number of transfer jobs that a given file server can receive), no improvement

• Next: direct gridftp sessions to the Castor file servers

11

WLCGSC4 Accounting

• Accounting metrics for SC4 monthly reports: – CPU usage in KSI2K-days (via DGAS)

• kSI2K days = raw_cpu_hours * GlueHostBenchmarkSI00 / (1000*24)

• GlueHostBenchmarkSI00 can be replaced by a weighted average of the computing power of Worker Nodes in the farm

– Wall-clock time in KSI2K-days (via DGAS)– Disk space used in TB– Disk space allocated in TB– Tape space used in TB

• Validation of raw data gathered, by comparison via different tools

12

WLCG

CPU Grid Accounting Results (T2)Jan-Apr 2006

T2 sites: Bari, Catania, Frascati, Legnaro, Napoli, Pisa Roma1-CMS, Roma1-Atlas, Torino

13

WLCG

Disk Space Accounting Results (T2)Jan-Apr 2006

14

WLCGPlanning (1/3)

• June 01-15: – Ongoing deployment of gLite 3.0.0 RC5 at T1

• FTS production server now working, deployment of catch-all channels

– Start of gLite 3.0.0 RC5 deployment in other INFN sites • INFN Grid 3.0 to be released soon• SRM still not deployed in every SC4 site

• June 01: start of service phase (see following slides)• July:

– Testing of concurent data transfer T1 all SC4 INFN sites– FTS: Testing of VO channel shares– Upgrade of the existing 1 GE production link at CNAF to 10 GE

15

WLCGPlanning (2/3)

• ATLAS transfer tests (GOAL: stability of DM infrastructure):– From June 19, for 3 weeks – T0 CNAF (59 MB/s) T2’s (max 20 MB/s, full set of AOD)– raw data to tape at T1 – Network reliability still to be evaluated for some T2s– Read performance of Castor2?– Usage of LFC catalogues at T1 sites– DQ2 to submit, manage and monitor the T1 export:

• DQ2 is based on the concept of dataset subscriptions: a site is subscribed by the T0 management system @ CERN to a dataset that has been reprocessed

• The DQ2 site service running at the site's VObox will then pick up subscriptions, submit and manage the corresponding FTS requests

• T2s subscribing to datasets stored on their associated Tier1also using DQ2

16

WLCGPlanning (3/3)

• CMS: load transfer tests (T1 and T2’s) from May 26:– T1 T2: bursty transfers driven by analysis

• 20 MB/sec aggregate Phedex (FTS) traffic to/from temporary disk at each T1

• Currently still based on srmcp ftscp

• SC3 functionality rerun using gLite infrastructure, trying to rise to 25000 jobs/day aggregated over all sites by end of June

– Bari, Legnaro and Pisa already running jobs

– T2 T1: continuous Monte Carlo transfers• aggregating to 1TB/day to CERN

• Total of 25M events/month

• Test T2 T1 transfers at 20 MB/sec per Tier 2

• Last 2 weeks of July: 150 MB/sec simulated raw data to tape at T1s, 25 MB/s to CNAF

17

WLCGVO Box

• “The term VO Box designates a system provided by the Site for use by one or more designated Virtual Organisations (VOs) for interactive access and running persistent processes.” (from VO-Box Security Recommendations and Questionnaire, CERN-LCG-EDMS-639856)

• installed and maintained by the Site Resource Administrators, as the VO Box is part of the trusted network fabric of the Site

• Interactive access to the VO box, and the capability to run services on the system, MUST be limited to a

• Interactive access and capability to run services limited to specific, named list of individuals within the VO. These individuals MUST have user-level access only.

• Well defined network connectivity requirements (ports in use)

• Vo Box subject to security service challenges.

18

WLCG

VO Box Task Force: Some Conclusions (from GDB Meeting slides)

• Classification of VO services into two types– Class 1 : sites not worried– Class 2 : sites want to see them go away ASAP

• VO boxes with purely class 1 services:– Needed by all VOs, high-level distributed VO services– Not really a problem for sites (given solution of a few

policy and security aspects)

• Class 2 services:– Potential threat to operational stability and security– Try to avoid new ones as much as possible– Class 2 services may need to be put somewhere else

(otherwise entire VO box is class 2)

19

WLCGVO Box Core Services

• Gsissh or other authenticated way of logging in• Grid client tools (UI)• Proxy renewal service• Shared file system (for ALICE packman, class 2,

need replacement ASAP)• Candidates for moving into LCG stack:

– Xrootd• Need some integration / cooperation with SRM products

– Package manager• Incorporate AliEn packman or develop some alternative

– MonaLisa• Monitoring

– Many other Longer Term Issues

20

WLCGOutstanding issues

• Accounting (monthly reports):– CPU usage in KSI2K-days DGAS– Wall-clock time in KSI2K-days DGAS– Disk space used in TB– Disk space allocated in TB– Tape space used in TB– Validation of raw data gathered, by comparison via different tools

• Monitoring of data transfer: GridView or other tools?• Routing in LHC Optical Private Network? Backup connection to

FZK becoming urgent• Implementation of a LHC OPN monitoring infrastructure still in its

infancy• SE Reliability when in unattended mode• Several mw components still require in-depth testing• SE kernel configuration:

– TCP send/receive socket buffer autotuning only for kernels ≥ 2.4.27 and ≥ 2.6.7

• What VO Box?

Documents

WLCG 1 Service Challenge 4: Preparation, Planning and Outstanding Issues at INFN [email protected] Workshop sul Calcolo e Reti dell'INFN Jun