Perspectives of JINR Grid-infrastructure

Perspectives of JINR Perspectives of JINR Grid-infrastructureGrid-infrastructure

V.V. IvanovV.V. Ivanov, Gh. Adam, V.V. Korenkov, T.A. Strizh, P.V. Zrelov, Gh. Adam, V.V. Korenkov, T.A. Strizh, P.V. Zrelov

Laboratory of Information TechnologiesLaboratory of Information TechnologiesJoint Institute for Nuclear ResearchJoint Institute for Nuclear Research

V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR)

Participating in LHC, RHIC, TEVATRON… In future: FAIR, ILC …

LHC-2008 FAIR-2015 ILC – after 2020 (?)

The JINR participation in the LHC experiments and in other large-scale projects asked for a substantial increase of its networking and information resources as well as the deployment of a large volume of work toward the development of the JINR Grid-segment and its integration in the Russian Grid-infrastructure RDIG (Russian Data IntensiveGrid).


The Laboratory of Information Technologies (LIT) is responsible for the computing infrastructure development in JINR and hosts the Central Information and Computing Complex (CICC) of the Institute.

Further perspective tasks follow from the decision of the JINR Committee of the Plenipotentiary Representatives, concerning the radical improvement of the JINR computer telecommunication links with major partner laboratories in the JINR Member States during the years 2010-2015 and the development of a unified Grid-environment of the JINR Member States. This is a cornerstone task of the JINR seven-year plan on 2010–2016 within the direction ”Networks. Computing. Computational Physics”.


Development of the JINR Development of the JINR Grid-environment – 2010-2016Grid-environment – 2010-2016

Three levels within the JINR Grid-environment can be distinguished: network, resource, and applied.

NETWORK

RESOURCE

APPLIED


CooperationCooperationCollaborators from the JINR Member States (protocols of cooperation with

INRNE (Bulgaria), ArmeSFo (Armenia), FZK Karlsruhe GmbH (Germany), IHEPI TSU (Georgia), NC PHEP BSU (Belarus), KFTI NASU (Ukraine), etc.).

Grants of Plenipotentiaries of the JINR Member States: Poland, Bulgaria, Slovakia, Romania. Agreement on Joint Research Programme “Meshcheryakov-Hulubei” LIT-Romania.

BMBF grant “JINR Network Security, Failover Measures, Network monitoring”.

CERN-JINR Cooperation Agreement on several topics, and JINR in the LCG Project (participation of JINR in the LCG project).

JINR-South Africa Cooperation Agreement - project “Development of Grid segment for the LHC experiments”.

Participation in joint projects: Enabling Grids for E-sciencE (EGEE -a project co-funded by the European Commission through the Sixth Framework Programme, CERN-INTAS project and "SKIF-GRID“ project of the Program of the Belarusian-Russian Union State as a part of joint propositions of NASB and Federal Agency of Science and Innovations of Russian Federation “The development and use of hardware and software in GRID-technologies and advanced supercomputer systems SKIF in 2007-2010 (SKIF-GRID)”.

Joint research grants from RFBR and Ministry of Science and Education with MSU and SINP MSU, Saratov University, IM SB RAS, IMM RAS, TU Gomel (Belorussia).


June 2009 - two important projects completed

Two large-scale very important projects have been completed: -- JINR - Moscow 20Gbps telecommunication channel -- JINR CentralJINR Central Information and Computing Complex major upgrade.Information and Computing Complex major upgrade.

Comprehensive presentation was done at the meeting of the Programme Advisory Committee for Particle Physics, on June 10.

All organizations involved in the project realizations, together with the representatives of Ministry of Communications, Federal Agency for Science and Innovation, attended the Presentation.

Our community colleagues and prospective users - residents of the special economical zone - also took part in the event.


Collaboration with RSCC, RIPN, MSK-IX, JET Infosystems, Nortel

JINR - Moscow 20Gbpstelecommunication channel

In perspective, the used DWDM and 10Gb Ethernet technologies allow the creation of up to 80 channels of 10 Gbps each, resulting in a throughput of up to 800 Gbps.


JINR Central Information and Computing ComplexJINR Central Information and Computing Complex

In 2008In 2008, the total CICC , the total CICC performance was 1400 performance was 1400 kSI2KkSI2K, , the disk storage capacity 100 TB the disk storage capacity 100 TB

At presentAt present, the CICC , the CICC performance equals performance equals 2300 kSI2K2300 kSI2K and the disk storage capacity and the disk storage capacity 500 TB500 TB

20082008 20092009 20102010

CPUCPU(kSI2K)(kSI2K)

12501250 17501750 25002500

DiskDisk (TB)(TB)

400400 800800 12001200

TapesTapes(TB)(TB)

-- 100100 200200

PLAN


JINR Local Area Network Backbone (LAN)

• Comprises 6579 computers and nodes,Users – 3593,IP – 7633,Remote VPN users (Lanpolis, Contact, TelecomMPK) - 1248;• High-speed transport (1Gbps) (Min. 100 Mbps to each PC);• Controlled-access (Cisco PIX-525 firewall) at network entrance;• Partially isolated local traffic (8 divisions have ownsubnetworks with Cisco Catalyst 3550 as gateways);• General network authorization system involves many services (AFS, batch systems, Grid, JINR LAN remote access, etc.)

• Comprises 6579 computers and nodes,Users – 3593,IP – 7633,Remote VPN users (Lanpolis, Contact, TelecomMPK) - 1248;• High-speed transport (1Gbps) (Min. 100 Mbps to each PC);• Controlled-access (Cisco PIX-525 firewall) at network entrance;• Partially isolated local traffic (8 divisions have ownsubnetworks with Cisco Catalyst 3550 as gateways);• General network authorization system involves many services (AFS, batch systems, Grid, JINR LAN remote access, etc.)

Plans: Step-by-step modernization of the

JINR Backbone – transfer to 10 Gbps Development and modernization of

the control system of the JINR highway network


• upgrade of the JINR backbone to a data transfer rate of 10 Gbps and gradual connection to it of various JINR laboratories; • increase of the performance of the LAN central telecommunication node kernel; • increase of the LAN protection at the hardware level;• wireless access to the LAN within the JINR territory; • implementation of new solutions and methods for the traffic control, information security, development of effective tools for control and the quality of service of JINR LAN; • development of the system of protection of the JINR LAN resources from non-authorized access.

The plan for the JINR LAN development for the years 2010-2016 foresees several general purpose tasks as well as specific site-related ones. A selective enumeration points to the:

First stage

JINR Laboratories

JINR Laboratories


JINR Central Information and Computing ComplexJINR Central Information and Computing Complex

12 1212121212

64KW

Super Blades5 x 10

APS

CICCServers

Users servers

dCacheServers

JINR nx 1 GbE x 10 GbE - plan)

Backbo e 2 (2

5

2

22

3

6

6

1

1

4 2 2

A1-1282 TB

B1-8144 TB

B9-16144 TB

RAID

Twins20 X 2

HP20

Т- 20

platforms

Batch Cluster

2

3

4

5

HP Procurve 5412zl-966

HP Procurve 3500yl-24G

Hp Procurve 2810-24 G

Hp Procurve 2810-48 G

Cisco Catalyst 2970

Cisco Catalyst 3750

CICCServers

Т- 20

platforms


JINR JINR CICC structural schemeCICC structural scheme


The protocol between CERN, Russia and JINR on participation in LCG Project was approved in 2003. MoU on Worldwide LHC Computing Grid (WLCG) was signed by JINR in October 2007

The tasks of the JINR in the WLCG:•WLCG-infrastructure support and development at JINR;WLCG-infrastructure support and development at JINR;• participation in WLCG middleware testing/evaluation, participation in WLCG middleware testing/evaluation, • participation in Service/Data Challenges,participation in Service/Data Challenges,• grid monitoring and accounting tools development;grid monitoring and accounting tools development;• FTS-monitoring and testing;FTS-monitoring and testing;• participation in ARDA activities in coordination with experiments; participation in ARDA activities in coordination with experiments; • JINR LCG portal support and development;JINR LCG portal support and development;• HEP applications; HEP applications; • MCDB development; MCDB development; • support of JINR Member States in the WLCG activities;support of JINR Member States in the WLCG activities;• User & Administrator training and education.User & Administrator training and education.

LHC Computing Grid Project


support and development WLCG/EGEE infrastructure;

participation in middleware testing/evaluation,

participation in Data and Service Challenges,

grid monitoring and accounting system development;

FTS-monitoring and testing

MCDB development;

Participation in ARDA activities in coordination with experiments;

HEP applications;

User & Administrator Training and Education

support of JINR member states in the GRID activities.

JINRJINR in the LCG/EGEE in the LCG/EGEE


CICC comprises:65 servers

4 interactive nodes960 computing nodes, Xeon 5150, 8GB RAM (GEthernet)(160 computing nodes, Xeon X5450, 16GB RAM, InfiniBand).

Site name: JINR-LCG2Internal CICC network – 1Gbit/secOperating system -Scientific Linux CERN 4.6 (transfer to Scientific Linux SL release 5.2.) ; Middleware version GLITE-3.1 File Systems – AFS (the Andrew File System) for user Software and home directories is a world-wide distributed file system. AFS permits to share easily files in an heterogeneous distributed environment (UNIXes, NT) with a unique authentication scheme (Kerberos). dCache- for data.User registration system – Kerberos 5 ( AFS use Kerberos 5 for authentication )

JINR WLCG JINR WLCG infrastructureinfrastructure


JINR provides the following services in the WLCG environment:JINR provides the following services in the WLCG environment:

Basic services:Basic services: - Berkley DB Information Index (top level BDII); - Berkley DB Information Index (top level BDII); - site BDII; - site BDII; - 2 x Computing Element (CE); - 2 x Computing Element (CE); - Proxy Server (PX); - Proxy Server (PX); - 2 x Workload Management System (WMS); - 2 x Workload Management System (WMS); - Logging&Bookkeeping Service (LB); - Logging&Bookkeeping Service (LB); - RGMA-based monitoring system collector server (MON-box); - RGMA-based monitoring system collector server (MON-box); - LCG File Catalog (LFC); - LCG File Catalog (LFC); - Storage Element (SE), dCache - Storage Element (SE), dCache 400400 TB, 4 x gridftp door, 14 x pool; TB, 4 x gridftp door, 14 x pool; - 4 x User Interface (UI), installed in AFS. - 4 x User Interface (UI), installed in AFS.

Special Services - Special Services - VO boxes for ALICE and for CMS; ROCMON;VO boxes for ALICE and for CMS; ROCMON;PPS and testing infrastructure - PPS and testing infrastructure - Pre-production gLite version; Pre-production gLite version;

Software for VOs: dCache xrootd door, AliROOT, ROOT, GEANT packages for ALICE; Software for VOs: dCache xrootd door, AliROOT, ROOT, GEANT packages for ALICE; ATLAS packages; CMSSW packages for CMS and DaVinchi, Gauss packages for ATLAS packages; CMSSW packages for CMS and DaVinchi, Gauss packages for LHCb.LHCb.

JINR WLCG JINR WLCG infrastructureinfrastructure


dCache in JINRdCache in JINR

Internet / GRID

lxfs07

lxfs07lxfs07

lxfs07lxpub01

lxfs07lxfs07

lxfs07wn…

lxfs07lxfs07

lxfs07lxfs71

RAID

500 TBAdmin

Doors

local

WNs

JINR Backbone

Protocols gFTP, SRM, XROOT

DCAP PNFS

lxfs07lxfs07

lxfs07rda02

Pools


Frames for Grid cooperation of JINR in 2009Frames for Grid cooperation of JINR in 2009 Worldwide LHC Computing Grid (WLCG); Enabling Grids for E-sciencE (EGEE); RDIG Development (Project of FASI) CERN-RFBR project “GRID Monitoring from VO perspective” BMBF grant “Development of the GRID-infrastructure and tools

to provide joint investigations performed with participation of JINR and German research centers”

“Development of Grid segment for the LHC experiments” was supported in frames of JINR-South Africa cooperation agreement;

NATO project "DREAMS-ASIA“ (Development of gRid EnAbling technology in Medicine&Science for Central ASIA);

JINR-Romania cooperation Hulubei-Meshcheryakov programme Project "SKIF-GRID" (Program of Belarussian-Russian Union

State). Project GridNNN (National Nanotechnological Net) We work in close cooperation and provide support to our

partners in Armenia, Belarus, Bulgaria, Czech Republic, Georgia, Germany, Poland, Romania, South Africa, Ukraine, Uzbekistan.


Development of the JINR Development of the JINR Grid-environment – 2010-2016Grid-environment – 2010-2016

Network level: links between Moscow and Dubna on the basis of state-of-the-art technologies DWDM and 10Gb Ethernet. JINR Local area network :JINR High-speed backbone construction – 10Gbps

Resource level:requirements of the LHC experiments stimulate the development of a global Grid-infrastructure, together with the resource centers of all the cooperating organizations. First of all, this is of primary concern for such large research centers as the JINR. To reach effective processing and analysis of the experimental data, further increase in the JINR CICC performance and disk space is needed.

Now the RDIG infrastructure comprises 15 Resource Centers with > 7000 kSI2K CPU and > 1850 TB of disc storage.

The RDIG infrastructure

• RDIG Resource Centres:– ITEP– JINR-LCG2– Kharkov-KIPT– RRC-KI– RU-Moscow-KIAM– RU-Phys-SPbSU– RU-Protvino-IHEP– RU-SPbSU– Ru-Troitsk-INR– ru-IMPB-LCG2– ru-Moscow-FIAN– ru-Moscow-GCRAS– ru-Moscow-MEPHI– ru-PNPI-LCG2– ru-Moscow-SINP


RDIG monitoring&accounting http://rocmon.jinr.ru:8080

Monitored valuesCPUs - total /working / down/ free / busyJobs - running / waiting Storage space - used / availableNetwork - Available bandwidth Accounting valuesNumber of submitted jobsUsed CPU time

Totally sum in secondsNormalized (with WNs productivity)Average time per job

Waiting timeTotally sum in secondsAverage ratio waiting/used CPU time per job

Physical memoryAverage per job

JINR CICC

• Monitoring – allows to keep an eye on parameters of Grid sites' operation in real time

• Accounting - resources utilization on Grid sites by virtual organizations and single users


Production Normalised CPU time per EGEE Regions (June-August 2009)


RDIG Normalised CPU time per Site and per VO

(June-September 2009)

V. Ivanov (LIT, JINR)V. Ivanov (LIT, JINR) 24

Production Normalised CPU Production Normalised CPU time per EGEE site for VO time per EGEE site for VO LHC (June – August 2009)LHC (June – August 2009)

GRID-site CPU time Num CPU time/N------------------------------------------------------------------------FZK-LCG 6,039,304 8620 700INFN-T1 3,493,208 2862 1220CERN-PROD 3,309,795 6812 485CYFRONET-LCG 3,198,111 2384 1341PIC 3,037,977 1337 2272JINR 2,938,039 960 3060IN2P3-CC 2,668,914 4544 587RAL-LCG2 2,629,889 2532 1038UKI-GLASGOW 2,610,116 1912 1365UKI-LT2-IC-HEP 2,533,230 960 2638


JINR-LCG2 in RDIG

JINR - 40% of total RDIG

CPU time

Normalised CPU time per VO at JINR site


Further participation in the WLCG project

Support and development of the JINR WLCG-segment Support and development of the JINR WLCG-segment

in frames of global WLCG infrastructure;in frames of global WLCG infrastructure; Participation in Service/Data Challenges in coordination Participation in Service/Data Challenges in coordination

with LHC experiments;with LHC experiments; WLCG middleware testing/evaluation;WLCG middleware testing/evaluation; Grid-monitoring the LCG-infrastructure at JINR and Grid-monitoring the LCG-infrastructure at JINR and

other sites of the Russian Tier2 cluster; other sites of the Russian Tier2 cluster; Evaluation of new Grid technologies in context of their Evaluation of new Grid technologies in context of their

usage in WLCG; usage in WLCG; MCDB development (data base structure; basic MCDB development (data base structure; basic

modules; modules; Participation in ARDA activities in coordination with Participation in ARDA activities in coordination with

experiments;experiments; JINR WLCG web-portal improvement; JINR WLCG web-portal improvement; User’s support to stimulate their active usage of WLCG User’s support to stimulate their active usage of WLCG

resources (courses, lectures, trainings, publication of resources (courses, lectures, trainings, publication of user guides in Russian); user guides in Russian);

Support of JINR member states in WLCG activities.Support of JINR member states in WLCG activities.


Applied level developments of the Applied level developments of the JINR Grid-environmentJINR Grid-environment (I)(I)

1. The applied level covers the user applications working in a virtual organization (VO) environment which comprises both users and owners of computing resources.

2. In the existing Grid-systems, a VO defines a collaboration of specialists in some area, who combine their efforts to achieve a common aim.

3. The virtual organization is a flexible structure that can be formed dynamically and may have a limited life-time. VOs working within the WLCG project are the VOs on the LHC VOs working within the WLCG project are the VOs on the LHC

experiments - ATLAS, CMS, Alice, LHCb, the first three being experiments - ATLAS, CMS, Alice, LHCb, the first three being carried out with the noticeable and direct participation of the carried out with the noticeable and direct participation of the JINR. Nowadays, as a Grid-segment of the EGEE/RDIG, the JINR JINR. Nowadays, as a Grid-segment of the EGEE/RDIG, the JINR CICC supports computations of the virtual organizations CICC supports computations of the virtual organizations registered in RDIG ( LHC experiments, BioMed, PHOTON, eEarth, registered in RDIG ( LHC experiments, BioMed, PHOTON, eEarth, Fusion, HONE, Panda.Fusion, HONE, Panda.

In the future, as the interest arises at a large-scale level, VOs In the future, as the interest arises at a large-scale level, VOs can be organized at JINR in the fields of nuclear physics and can be organized at JINR in the fields of nuclear physics and condensed matter physics and, most probably, in the new condensed matter physics and, most probably, in the new promising direction related to the research of the nanostructure promising direction related to the research of the nanostructure properties.properties.


The creation of new VOs gets The creation of new VOs gets possible and necessary under possible and necessary under maturation of the algorithmic maturation of the algorithmic approaches to the problem solution, approaches to the problem solution, the development of corresponding the development of corresponding mathematical methods and tools:mathematical methods and tools:

methods and tools for simulation of methods and tools for simulation of physical processes and analysis of physical processes and analysis of experimental data software and experimental data software and computer complexes for computer complexes for experimental data processing;experimental data processing;

numerical methods, algorithms and numerical methods, algorithms and software for modeling complex software for modeling complex physical systems; physical systems;

methods, algorithms and software of methods, algorithms and software of computer algebra;computer algebra;

new computing paradigmsnew computing paradigms;; adaptation of specialized software for adaptation of specialized software for

solving problems within the Grid-solving problems within the Grid-environment.environment.

Applied level developments of the Applied level developments of the JINR Grid-environmentJINR Grid-environment (II)(II)


Formation of the Grid environment of the Formation of the Grid environment of the JINR-Member StatesJINR-Member States

Creation of the Grid environment of the JINR-participating countries Creation of the Grid environment of the JINR-participating countries will allow to bring out the research underway at JINR in the field of will allow to bring out the research underway at JINR in the field of high energy physics, nuclear physics, solid state and condensed high energy physics, nuclear physics, solid state and condensed matter physics, biophysics, biosensor (nano)technologies, etc., on a matter physics, biophysics, biosensor (nano)technologies, etc., on a qualitatively new level. qualitatively new level.

Basic directionsBasic directions:: Creation of a working group of the representatives of the JINR-Creation of a working group of the representatives of the JINR-

participating countries for the development of the main directions of participating countries for the development of the main directions of the activity toward forming a joint Grid environment.the activity toward forming a joint Grid environment.

Definition, together with the JINR Member States, of the basic Definition, together with the JINR Member States, of the basic directions of activity toward forming the joint Grid environment.directions of activity toward forming the joint Grid environment.

Preparation of corresponding multi-side agreements.Preparation of corresponding multi-side agreements. Development of the network level of the Grid environment of the Development of the network level of the Grid environment of the

JINR Member States.JINR Member States. Development of the resource level of the Grid environment of the Development of the resource level of the Grid environment of the

JINR Member States.JINR Member States. Formation of the applied level of the Grid environment of the JINR Formation of the applied level of the Grid environment of the JINR

Member States.Member States. Preparation and implementation of joint projects.Preparation and implementation of joint projects.

V.Ivanov (LIT, JINR)V.Ivanov (LIT, JINR)

Thank you for your attention !Thank you for your attention !Thank you for your attention !Thank you for your attention !

Documents

Perspectives of JINR Grid-infrastructure