Dmytro Soroka – GRID 2008, Protvino, Russia July 2008

KK

II

PP

TT

Status and structure of Kharkov Institute of Physics and Technology (KIPT) Grid-cluster

(Peculiarities of the architecture of KIPT Grid-cluster)

L. Levchuk, S. Lukyanenko, P. Sorokin, D. Soroka, S. Zub, O. Bunetskyi

Kharkov Institute of Physics & Technology, 61108 Kharkov, Ukraine

http://www.kipt.kharkov.ua/~cmsdsoroka@kipt.kharkov.ua

Dmytro Soroka – GRID 2008, Protvino, Russia July 2008

KK

II

PP

TTPlanPlan

q KIPT CMS cluster (KCC) : status and development in 2008 q Internet connectivity issues q Grid middleware at KCC and

peculiarities of its architectureq Conclusions

Dmytro Soroka – GRID 2008, Protvino, Russia July 2008

KK

II

PP

TT

KIPT CMS cluster: status and development in 2006 - 2008

KIPT CMS cluster: status and development in 2006 - 2008

Performances:Performances:qSeptember 05:

11 nodes (22 CPU11 nodes (22 CPU’’s, 14 s, 14 kSIkSI20002000), ), 2.5 TB HDD2.5 TB HDDqSeptember 06:

30 nodes (60 CPU30 nodes (60 CPU’’s, 71 s, 71 kSIkSI20002000),),41 TB HDD41 TB HDDqThe end of the year 2007:

38 nodes (76 CPU38 nodes (76 CPU’’s, 96 s, 96 kSIkSI20002000), ), 57 TB HDD57 TB HDDThis fulfils our needs in computing resources This fulfils our needs in computing resources that we planned to have by LHC startupthat we planned to have by LHC startup

Dmytro Soroka – GRID 2008, Protvino, Russia July 2008

KK

II

PP

TT

2005: KIPT CMS cluster was registered in WLCG as an LCG/RDIG site

Dmytro Soroka – GRID 2008, Protvino, Russia July 2008

KK

II

PP

TT

KCC structureKCC structure

Dmytro Soroka – GRID 2008, Protvino, Russia July 2008

KK

II

PP

TT

KCC viewKCC view

Dmytro Soroka – GRID 2008, Protvino, Russia July 2008

KK

II

PP

TTInternet Connectivity Internet Connectivity

Creation of the transCreation of the trans--Ukrainian highUkrainian high--speed speed (~10 (~10 GbitGbit/s) network backbone/s) network backbone

Project has been approved by Ukrainian Government and got funding. It has been

completed in 2008.

?

KIPT

2.5Gbit/s (10Gbit/s)

L’vov Kiev Kharkov

GEANT Russia

2.5Gbit/s (10Gbit/s)

Today: 100 Mbit/s

Dmytro Soroka – GRID 2008, Protvino, Russia July 2008

KK

II

PP

TTGrid middleware at KIPT

CMS ClusterGrid middleware at KIPT

CMS Cluster

qPeculiarities of LCG MW deployment and operation:• use of our local repository of LCG MW rpms• combination of WN and UI on a single node• pool-account users authenticated by means of NIS with their

home directories being NFS-shared • use of “fake” IP addresses for all worker nodes

(further details in S.Zub et al., NIM A599 (2005) 35 )• KCC is currently running Glite 3_1_0 • WNs, SE, VOBOX are running SLC 4.5 x86_64• Supported VOs: cms dteam ops rdteam rfusion rgstest

Dmytro Soroka – GRID 2008, Protvino, Russia July 2008

KK

II

PP

TT

Network infrastructureNetwork infrastructure

48- port HP ProCurveSwitch 2848

24- port SwitchCisco Catalyst 2960G

Dmytro Soroka – GRID 2008, Protvino, Russia July 2008

KK

II

PP

TT

Network Infrastructure - 2Network Infrastructure - 2

Dmytro Soroka – GRID 2008, Protvino, Russia July 2008

KK

II

PP

TT

DPM servers configurationDPM servers configuration

DPM DPM mysqlmysql: : SATA II RAID Controller 3Ware 9SATA II RAID Controller 3Ware 9500 (12 500 (12 portsports))

u 2 HDDs 400 GB in RAID1 (mirror)SLC 4.5 x86_64 (2.6.9-55.EL.cernsmp)

u 9 HDDs 400 GB in RAID 5 + 1 hot sparepartition /storage file system ext3

DPM diskDPM disk: : SATA II RAID Controller 3Ware 9550SX SATA II RAID Controller 3Ware 9550SX (1(166 portsports))u 1 System SLC 4.5 x86_64 (2.6.9-55.EL.cernsmp)u 11 disks in RAID 5 + 1 hot spare disk partition /storage, ext3

Dmytro Soroka – GRID 2008, Protvino, Russia July 2008

KK

II

PP

TT

Disk ServerDisk Server

Dmytro Soroka – GRID 2008, Protvino, Russia July 2008

KK

II

PP

TT

Disk server - 2Disk server - 2

6 “hot-swap” disks

Dmytro Soroka – GRID 2008, Protvino, Russia July 2008

KK

II

PP

TTSE_DPM_mysqlSE_DPM_mysql

Dmytro Soroka – GRID 2008, Protvino, Russia July 2008

KK

II

PP

TTCMS clusterCMS cluster maintenance maintenance inin 20020088

♦ SE DPM installing and configuration (D.Soroka)

♦ configuration and testing of the VOBOX/PhEDEx (S.Zub)

♦ remote monitoring of the state of High Performance Battery Backup & Protection UPS’s (APC Smart-UPS RT 5000VA 230V)and the temperature monitoring In the cluster room (O.Bunetskyi)

♦ Cooling system maintenance, power feeder upgrade (NSC KIPT back up power supply system) (?.Gorbenko, V.Popov, I.Barankov)

Dmytro Soroka – GRID 2008, Protvino, Russia July 2008

KK

II

PP

TT

SummarySummary

u The 64-bit upgrade of the KCC : 24×3.2 GHz [Intel XeonEM_64T] + 22×3.4 GHz [Intel Xeon EM_64T] + 6×[AMD Opteron252] = 52 CPU (Glite 3.1 WN’s on SLC 4.5 x86_64 )

u SE DPM (Glite 3.1) has been installed on SLC 4.5 x86_64(2.6.9-55.EL.cernsmp).

u Disk pools /storage ~ 3.6 TB permit to store and process large data files.

u High performance switch Cisco Catalyst 2960G enables separate infrastructure for DPM front-end and disk servers.

u After accomplishing all PHEDEX tests SE DPM will provideGrid access to ~ 45 ?B storage.

u Certification of the KCC as ?2 WLCG RC will give us opportunity to participate in CSA`08.

Dmytro Soroka – GRID 2008, Protvino, Russia July 2008

KK

II

PP

TTSummary-2Summary-2

u The KCC goes from the level of a prototype PC farm to a functioning Resource Center to be used in CMS data analysis

• essential upgrade of the KCC is under way (including internet connectivity, disk servers, etc.); this upgrade substantiates our intention to carry on data processing and analysis according to the CMS computing model

u LCG (currently, Glite_3_1) MW at KCC is running in an “economical” but reliable way

• NSC KIPT RC showed satisfactory functioning over the last 4 months

• When configuring the SE (DPM) we expected to have the single front-end DPM server with host certificate and dpm_disk servers having fake IP-s without host certificates. But it was impossible and we had to obtain 10 IP addresses and 10 host certificates. We have a suggestion to the developers to realize a single entry for DPM server.

Dmytro Soroka – GRID 2008, Protvino, Russia July 2008

KK

II

PP

TTAcknowledgementsAcknowledgements

u To our SINP colleagues: A.P. Kryukov, O.L. Kodolova, V.A. Iliyn, L. Shamardin, N. Kruglov.

u To our JINR colleagues: E. Tikhonenko, V.V.Mitsyn, V.V. Korenkov, S. Belov.

u To our PNPI colleague: A. Kiryanov

u To our colleague KIAE E. Ryabinkin,

and to many other Russian specialists