CC-IN2P3 : Status & perspectives

Centre de Calcul de l’Institut National de Physique Nucléaire et de Physique des Particules

CC-IN2P3 : Status & perspectivesJFPPL Computing WS - March 2015

2015-03-10FJPLL Computing workshop

Organization

2

2015-03-10FJPLL Computing workshop 3

} Steering committee

} + 1 internal projects evaluation committee

} Goals : • list every transversal IT projects • ensure they have the necessary

ressources • make some recommandations, if

any, to the Direction } ~1 expert of each IT teams } First results

• 15 projects evaluated • good feedback from project leaders • a better knowledge

Organization


} 3 permanent positions opened in 2014 storage, system administration, administration

} Temporary contracts mainly funded on EU project or various agreements

} Globally : less WFTE (because of some leaves : retirements…)

} Regulation constraints : } strong turnover… } less candidates…

} 2015 forecast : } +1 permanent position (DBA) } 7 contracts end : some can be renewed

Staff

Worked FTE

0,0

20,0

40,0

60,0

80,0

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

2011

2012

2013

2014

Permanent Non-Permanent

78 people : 30 % temporary contracts 1 « chargé de recherche » et 77 technicians and engineers (ITA) :

61 BAP E : IT engineers 8 BAP J : administration 5 BAP G : facility management 3 BAP F : communication / mediation


Budget

5


} From 2011 to 2013 : more or less constant • no capacity growth, only renewal

} In 2014 : + 550 k€ • a 900 k€ increase coming from the CNRS allocation (TGIR/IR) • but lower funding coming from CEA = -200 k€

} Other incomes (ANR, european projects…) : ~ 383 k€

Resources : budget evolution

Bud

get

en k€

0

3000

6000

9000

12000

2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

900110011001100

1364

13641364

1300

17001600

18501745179917002000

545050004587

20001350

4050370536563500422337503750375037503400

TGIR non LHCTGIR LCGCEA

Strong investments corresponding with the LHC startup : impact on capacity maintaining 3 to 5years later

non LCG budget more or less stable :

but running costs in increase (electricity: x3,3 in 8 years)

less investments for others


Budget 2014 : by destination

IN2P3 Labs4 %

Runnning costs4 %

HR5 %

FM26 %

Non LCG36 %

LCG (except op.)25 %

Running costs4 %

HR5 % FM

26 %

IT @ IN2P3 labs13 %

IT @ CC51 %

Maintenance21 %

Services11 %

Network17 %

Storage38 %CPU

12 %

Tertiary5 %Infra

20 %

Maintenance29 %

Fluids46 %

=

Half of the budget is dedicated to IT investments at CC-IN2P3

A quarter of the budget goes into electricity

TertiaryRunning costs

HRFluids

Infrastructure

IT increase

Maintening capacity


LHC Computing Grid

8

FJPLL Computing workshop 2015-03-10 9

WLCG and LCG-France

LCG-France :

MoU between IN2P3 and the Tiers sites for 2013-2016 : to keep the french contribution to WLCG at least at ~10% to keep the 100% of the T1 and its growth (~10 to 20%) maintain the T2-T3s by funding them to up 70% of the needed money

With annual budget forecast : ~1 800 €


CC-IN2P3 within WLCG

107 kHS06 ~18 PB~10 PB

2015 T1 Pledged CPU Capcacity

22 %

11 %

2 %4 %

9 % 3 %4 %6 %

12 %

12 %

10 %4 %

Canada France Germany Italy Netherlands Nordic Korea Russia Spain Taïwan UK USA

2015 T1 Pledged Disk Capacity

20 %

11 %

3 %4 %

9 %1 %5 % 6 %

14 %

14 %

10 %4 %

2015 T1 Pledged Tape Capacity

25 %

12 %

2 %5 %

7 %1 %2 %5 %

15 %

12 %

10 %4 %

0

20,000

40,000

60,000

80,000

100,000

120,000

0

2000

4000

6000

8000

10000

12000

2006 2007 2008 2009 2010 2011 2012 2013 2014

TB

HEP

-SPE

C06

Evolution Ressources LCG-France Sites T2-T3 hors CC-IN2P3

Disk [TB] CPU [HEP-SPEC06]

% of 2014 pledges CPU Disk Tapes

French T1 / ∑ T1 10,15 10,29 10,11

French T1+T2s / ∑ T1+T2s

9,61 10,3 10,11


Network connectivity for WLCG and LCG-France

2013 Renater report status:

LCG is 46 % of total french trafic


Capacities and usages

12


CPU capacity

2011-13 : core number is constant

2014 : 20 000 cores for a global power of 208 866 HEPSpec06 2011 2013

Available power is growing :

- newer servers bought to maintain capacity - OS is more efficient

FJPLL Computing workshop 2015-03-10

Disk storage

Standard performance disk > 12 PB High performance disk (for GPFS) = 1,6 PB

Tapes

Storage used on magnetic tape : 25 PB out of 340 PB nominal capacity

Backup (TSM)

Stored volume : about 5,5 Po

14

Storage capacities and global consumption

2015 Allocation

CPU (HS06.h) 1 947 949 840

MSS (To) 24 957

dCache (To) 8 586

iRods (To) 524

GPFS (To) 1 673

xRootd (To) 2 545


Users : distribution and CPU comsumption (Nov. 2014)

21,5 %0,3 %

6,4 %

0,4 %1,7 %10,4 %

59,3 %

Particle PhysicsHadron PhysicsTheoryLabsIN2P3 Multidisciplinary & openingsNuclear PhysicsAstroparticles + Neutrino

0,5 %

LHCb21,4 %

D05,7 %

CMS22,4 %

ATLAS50,0 %

CTA13 %

dchooz6 %

Virgo1 %

AMS13 %

km3net2 %

GLAST16 %

CRAL2,1 %Planck

12,7 %HESS15,3 %Auger3,3 %Antares15,1 %

Top 10 Astroparticle + Neutrino sharing

Top HEP


Perspectives

16


Electricity cost…

0200400600800

100012001400

012345678

2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

k€ HT Cons. (en 10 GWh) kWh price (€ cents/kWh)

1000

1350

1700

janv. 12févr. 12m

ars 12avr. 12m

ai 12juin 12juil. 12août 12sept. 12oct. 12nov. 12déc. 12janv. 13févr. 13m

ars 13avr. 13m

ai 13juin 13juil. 13août 13sept. 13oct. 13nov. 13déc. 13jan. 14fév. 14m

ars 14m

ai 14juin 14juil. 14août 14sept. 14oct. 14

Average consumed power (kW)

Average consumed power by month is decreasing :

- hardware renewal- energetic optimisation (setpoints

tuning)- VIL 2 : best PUE

But the global bill is more or less constant :

- kWh price increases- threshold effects of the EDF contracts

Estimation


Computing rooms : VIL2

Base infrastructure designed to fit the final configuration (wires, pipes…)

Everything else is modular (transformers, UPS, chillers, etc.) and can be installed later as a function of the needs

Best PUE : ~ 1.47 Capacity : 80 racks

28 installed - 52 left

1 rack = 730 TB or 20,5 kHS06 (2014)

2*250 m2 to use

Investment for 80 racks : from 1.5 to 2 M€ ( Tier II) :

wires, pipes, transformers, chillers without racks and servers


LSST (Large Synoptic Survey Telescope) : - reprocessing of half of the data- CC will host all the processed data

EUCLID : CC-IN2P3 will be one of the 8 “Sciences Data Centers” of this European space mission and should provide 30% of the resources (CPU and storage)

CTA : CC-IN2P3 may be led to operate for it, also by 2030, at least a 25% of 88 kHS06 for CPU, 207 PB for disk and 507 PB for tapes.

LHC : ~+25% per year indicates that at the end of the LHC Run 3 a T1 capacity of

19

Estimated needs of future (and on-going) experiments

En 2024 CPU kHS06

DisquePo

MSSPo

LHC 1 000 80 150

x2014 5 6 6

In 2030 CPUkHS06

DiskPB

MSSPB

LSST 2 400 100 266

EUCLID 67 150 52

CTA 22 52 127

∑ 2 489 302 445

x2014 12 22 18

Cumulated capacities by year for the 4 experiments

PB

0,00

200,00

400,00

600,00

800,00

kHS0

6

0

1 250

2 500

3 750

5 000

2015

2016

2017

2018

2019

2020

2021

2022

2023

2024

2025

2026

2027

2028

2029

2030

CPU kHS06 Disk PB Tapes PB

MOA Signed !

FTP Submitted

Not only a infrastructure challenge. Stronger collaboration with experiments, with similar T1s, new softwares, interfaces between data scientists & computer scientists …


} IN2P3 computing budget 2015 is constant over 2014 : 6 200 k€ • it should permit to face the experiments needs (as far as we know them today) and to

continue infrastructure upgrades

} New projects begin } Indigo-Datacloud, EGI-Engage, EU-T0, CPER

} Human resources

• 2015 should be « calm » year : not much leaves expected (except projects) but some concern for the future

• 2016 : new cycle of retirements, end of the on-going CEA agreement • 2016+ : very strong concerns about CNRS budget

Many things to do …

20

2015 Forecast

Documents

CC-IN2P3 : Status & perspectives