HEP Grid Computing HEP Grid Computing in Chinain China

Gang CHEN@IHEP-CCGang CHEN@IHEP-CC

06-16-200606-16-2006

Workshop on Future PRC-U.S. Cooperation Workshop on Future PRC-U.S. Cooperation

in High Energy Physicsin High Energy Physics

Gang Chen/IHEP-CC 23/4/19 - 2

AgendaAgenda• Computing RequirementsComputing Requirements• Grid DeploymentGrid Deployment• ApplicationsApplications• NetworkingNetworking• ProspectsProspects

Gang Chen/IHEP-CC 23/4/19 - 3

BEPCII/BESIIIBEPCII/BESIII

BEPC: Beijing BEPC: Beijing Electron Positron Electron Positron ColliderCollider• Started in 1989Started in 1989• 2~5GeV/C2~5GeV/C• Being upgraded toBeing upgraded to Dual-Ring, Dual-Ring, (3~10)×10 (3~10)×103232 cm cm-2-2ss-1-1 • 27 27 Universities andUniversities and Institutes from Institutes from Korea, US, UK, Korea, US, UK, Japan, and China.Japan, and China.

Gang Chen/IHEP-CC 23/4/19 - 4

BESIII Computing BESIII Computing RequirementsRequirements

• CPUCPU ：： ~2~2000 P-IV/3GHz CPU for 000 P-IV/3GHz CPU for data data production/reconstruction production/reconstruction　　　 　　　 and physics analysisand physics analysis

• StorageStorage ：： ~5~5PB in 5 years. PB in 5 years. • NetworkNetwork ：： for for

international/domesticinternational/domestic communications communications

Gang Chen/IHEP-CC 23/4/19 - 5

YBJYBJ--ARGO/ASARGO/AS

• International cosmic International cosmic ray observatories in ray observatories in Tibet.Tibet.

• 200TB raw data per 200TB raw data per year.year.

• Data transferred to Data transferred to IHEP and processed IHEP and processed with 400 CPUs.with 400 CPUs.

• Rec. data accessible Rec. data accessible by collaborators. by collaborators.

Gang Chen/IHEP-CC 23/4/19 - 6

LHCLHC

• China is involved in all four LHC China is involved in all four LHC experiments, experiments,

• IHEP in CMS and ATLAS.IHEP in CMS and ATLAS.

Gang Chen/IHEP-CC 23/4/19 - 7

More CollaborationsMore Collaborations

Members of International Members of International Collaborations:Collaborations:Huge computing demands.Huge computing demands.

Gang Chen/IHEP-CC 23/4/19 - 8

Grid VisionGrid VisionThe GRID: networked data The GRID: networked data processing centres and processing centres and ”middleware” software as ”middleware” software as the “glue” of resources.the “glue” of resources.

Researchers perform Researchers perform their activities regardless their activities regardless geographical location, geographical location, interact with colleagues, interact with colleagues, share and access datashare and access data

Scientific instruments and Scientific instruments and experiments (and experiments (and simulations) provide huge simulations) provide huge amount of dataamount of dataFederico.Carminati@cern.ch

Gang Chen/IHEP-CC 23/4/19 - 9

HEP Grid in ChinaHEP Grid in China• Coordinated by IHEPCoordinated by IHEP• Based on EGEE etc.Based on EGEE etc.• To build a Tier-2 center at IHEP for To build a Tier-2 center at IHEP for

LHC, WLCG MoU has been signedLHC, WLCG MoU has been signed• Cooperating among institutes and Cooperating among institutes and

universitiesuniversities

Gang Chen/IHEP-CC 23/4/19 - 10

LCG Deployment in ChinaLCG Deployment in China

UI

CE_torque

SE_classicl

MON WN1

UI

CE_torque

SE_dcache

BDII

PX

RB

LFC_mysql

WN2

SDU VO: ATLASSDU VO: ATLAS

PC farm

UI

CE_torque

SE_classicl

WN1

WN2

PC farm

UI

WN1

WN2

PC farm

PKU VO: CMSPKU VO: CMS

CSTNET\CERNET

CNICCNICIHEPIHEP

Gang Chen/IHEP-CC 23/4/19 - 11

LCG Beijing Site LCG Beijing Site Operation Operation

• UI (User Interface)UI (User Interface) ： ： gLite-3.0.0gLite-3.0.0• RB (Resource Broker)RB (Resource Broker) ： ： LCG-2.7.0LCG-2.7.0• MyProxyMyProxy ： ： gLite-3.0.0gLite-3.0.0• SE (SRM/Dcache)SE (SRM/Dcache) ： ： gLite-3.0.0gLite-3.0.0• CE (OpenPBS->Torque) CE (OpenPBS->Torque) ：： LCG-2.7.0LCG-2.7.0• MON (R-GMA )MON (R-GMA ) ： ： gLite-3.0.0gLite-3.0.0• BDIIBDII ： ： gLite-3.0.0gLite-3.0.0• WNsWNs ： ： gLite-3.0.0gLite-3.0.0• LFCLFC ： ： LHC File CatalogueLHC File Catalogue• CACA ： ： IHEP CAIHEP CA• VOMSVOMS ：： for BES/YBJ-ARGO VOsfor BES/YBJ-ARGO VOs

Gang Chen/IHEP-CC 23/4/19 - 12

CA and BES VOMS CA and BES VOMS OperationOperation

• IHEP CA is Unique CA accredited by IHEP CA is Unique CA accredited by EUGridPMA and APGridMPA, covering EUGridPMA and APGridMPA, covering HEP and other grid applications.HEP and other grid applications.

Gang Chen/IHEP-CC 23/4/19 - 13

Grid Applications in Grid Applications in HEPHEP

• LHCLHC• BES-III: BES-III:

Beijing Electron-Positron Collider/Beijing Beijing Electron-Positron Collider/Beijing Electron Spectrometer: tau/charm physicsElectron Spectrometer: tau/charm physics

• ARGO-YBJ: ARGO-YBJ: China-Italy cosmic ray observatories in TibetChina-Italy cosmic ray observatories in Tibet

• D0D0• ……

Gang Chen/IHEP-CC 23/4/19 - 14

HEP Grid with BES HEP Grid with BES SupportSupport

Gang Chen/IHEP-CC 23/4/19 - 15

CMS PHeDex Data CMS PHeDex Data transfertransfer

• Setup PHeDEX Setup PHeDEX system for system for Dataset Dataset transfer.transfer.

• Successful test Successful test for dataset for dataset subscription, subscription, data transfer, data transfer, error error recovery,etc. recovery,etc.

Gang Chen/IHEP-CC 23/4/19 - 16

YBJ-ARGO Computing on YBJ-ARGO Computing on LCGLCG

• MC, Medea+ and Reconstruction in MC, Medea+ and Reconstruction in /opt/exp_soft/bes area./opt/exp_soft/bes area.

• Submit jobs via BES VO to Beijing Site.Submit jobs via BES VO to Beijing Site.• Results show everything is working.Results show everything is working.

CE R N GE A N T 4 CL H E P B O O S T

E x t e r nal ex t er nal

ar gog med ea++

O ffl ine

ar go

Gang Chen/IHEP-CC 23/4/19 - 17

Data Processing modelData Processing model

• Based on the sharing of the resources Based on the sharing of the resources and the use of synchronized Data and the use of synchronized Data CataloguesCatalogues

• A common ARGO Virtual OrganizationA common ARGO Virtual Organization • A top ARGO-VO BDII and RB A top ARGO-VO BDII and RB • The output files in the “local” Data The output files in the “local” Data

Catalogue Catalogue • Synchronization procedure for the Data Synchronization procedure for the Data

Catalogues to have a copy of the files in Catalogues to have a copy of the files in each siteeach site

• All information in the experiment Data All information in the experiment Data BaseBase

Gang Chen/IHEP-CC 23/4/19 - 18

ARGO ARGO AchievementsAchievements

• Use of Gilda infrastructure to test Use of Gilda infrastructure to test the computing model: GILDA the computing model: GILDA elements installed at IHEP and elements installed at IHEP and INFN ROMA TREINFN ROMA TRE

• Most of the scripts implementing Most of the scripts implementing the data transfer model already the data transfer model already designed and implemented designed and implemented

• Soon extensive testSoon extensive test• Many requirements specifiedMany requirements specified

Gang Chen/IHEP-CC 23/4/19 - 19

Grid VPN between IHEP and Grid VPN between IHEP and USTCUSTC

•MotivationMotivation• International bandwidth bottleneck of International bandwidth bottleneck of

China universitiesChina universities• Academic users in universities have Academic users in universities have

to use commercial service shared to use commercial service shared with millions of public users. with millions of public users.

• USTC d0ustc MC production remote USTC d0ustc MC production remote farm to FNAL SAM Grid running at farm to FNAL SAM Grid running at unstable 50kBps, which is the best unstable 50kBps, which is the best can be provided by China Telecom. can be provided by China Telecom.

• Grid won’t be function this way!Grid won’t be function this way!

Gang Chen/IHEP-CC 23/4/19 - 20

Grid VPN between IHEP and Grid VPN between IHEP and USTCUSTC

•SolutionSolution• Virtual Private Network (VPN) channel between Virtual Private Network (VPN) channel between

CSTNET (USTC) and CSTNET (IHEP). d0ustc gain 120-CSTNET (USTC) and CSTNET (IHEP). d0ustc gain 120-150kBps stable net connection. 150kBps stable net connection.

•ProblemProblem• Grid CONDOR CLOBUS does not recognize a VPN IP Grid CONDOR CLOBUS does not recognize a VPN IP

farm, and there is no standard solution provided by farm, and there is no standard solution provided by Wisconsin Group yet. Wisconsin Group yet.

• IHEP, USTC and FNAL computing experts are working IHEP, USTC and FNAL computing experts are working on it now. If we success, then we can distribute on it now. If we success, then we can distribute university LCG Tiers. university LCG Tiers.

• 120-150kBps is far from acceptable for Grid, a P2P 120-150kBps is far from acceptable for Grid, a P2P link between IHEP to FNAL etc is highly desired.link between IHEP to FNAL etc is highly desired.

Gang Chen/IHEP-CC 23/4/19 - 21

Secure VPNSecure VPN

Gang Chen/IHEP-CC 23/4/19 - 22

d0ustc is on via USTC-IHEP VPN, much faster and stable still have some problem with gridftp

SAM GRID SAM GRID SUMMARYSUMMARY

Gang Chen/IHEP-CC 23/4/19 - 23

• To foster the creation of a To foster the creation of a intercontinental e-Science communityintercontinental e-Science community• Training peopleTraining people• Supporting existing and new applications: Supporting existing and new applications: LHC, LHC,

ARGO, Biology…ARGO, Biology…

• To support interoperable infrastructure To support interoperable infrastructure for grid operations between Europe and for grid operations between Europe and ChinaChina

A EU funded project of Interconnection & A EU funded project of Interconnection & Interoperation of Grid between Europe Interoperation of Grid between Europe and Chinaand China

EUChinaGridEUChinaGrid

Gang Chen/IHEP-CC 23/4/19 - 24

PartnersPartners11 Istituto Nazionale di Fisica Nucleare (IT) Istituto Nazionale di Fisica Nucleare (IT) (coordinator)(coordinator)22 European Organisation for Nuclear Research CERN (CH)European Organisation for Nuclear Research CERN (CH)

33 Dipartimento di Biologia - Università di Roma Tre (IT)Dipartimento di Biologia - Università di Roma Tre (IT)

44 Consortium GARR (IT)Consortium GARR (IT)

55 Greek Research & Technology Network (GR)Greek Research & Technology Network (GR)

66 Jagiellonian Jagiellonian University – Medical College, Cracow (PL)University – Medical College, Cracow (PL)

77 School of Computer Science and Engineering – Beihang School of Computer Science and Engineering – Beihang University Beijing (CN)University Beijing (CN)

88 Computer Network Information Center, Chinese Computer Network Information Center, Chinese Academy of Sciences – Beijing (CN)Academy of Sciences – Beijing (CN)

99 Institute of High Energy Physics, Beijing (CN)Institute of High Energy Physics, Beijing (CN)

1010 Peking University – Beijing (CN)Peking University – Beijing (CN)

Gang Chen/IHEP-CC 23/4/19 - 25

GILDA sitesGILDA sites

Gang Chen/IHEP-CC 23/4/19 - 26

GILDA sitesGILDA sites

Gang Chen/IHEP-CC 23/4/19 - 27

Job StatisticsJob Statistics

Gang Chen/IHEP-CC 23/4/19 - 28

EUChinaGrid Workshop: EUChinaGrid Workshop: 12-14 June, 200612-14 June, 2006

Gang Chen/IHEP-CC 23/4/19 - 29

EUChinaGrid Tutorial:EUChinaGrid Tutorial:15-16 June, 200615-16 June, 2006

Organized by IHEPOrganized by IHEPTutors from IHEP and INFNTutors from IHEP and INFN

Gang Chen/IHEP-CC 23/4/19 - 30

NetworkingNetworking

1010GG

155M

Gang Chen/IHEP-CC 23/4/19 - 31

NetworkingNetworking

11GbpsGbps

Gang Chen/IHEP-CC 23/4/19 - 32

GLORIADGLORIAD• Gloriad is an initiative from China, Gloriad is an initiative from China,

USA and Russia. And now has been USA and Russia. And now has been extended with Korea, Netherlands, extended with Korea, Netherlands, and Canada.and Canada.

• Gloriad provides expanded capacity Gloriad provides expanded capacity for science and education for science and education collaboration (10 Gbps).collaboration (10 Gbps).

• Gloriad is open to more members, Gloriad is open to more members, especially in Asia Pacific region!especially in Asia Pacific region!

Gang Chen/IHEP-CC 23/4/19 - 33

Users of Gloriad Users of Gloriad

IHEPIHEP

Gang Chen/IHEP-CC 23/4/19 - 34

Network Performance Network Performance TestTest

Gridftp perf ormace wi th l arge st ream

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

10 20 30 40 50 60 70 80 90 100 110 120130 140 150 160170 180 190 200

Stream Number

Thou

ghpu

t(KB

/sec

)

f 1f 2f 3f 4f 5

gridftp perf ormace wi th l arge st ream number

0

500

1000

1500

2000

2500

3000

10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180 190 200

St ream Number

Thou

ghpu

t(KB

/sec

)

f 1f 2f 3f 4f 5

IHEP-TaipeiIHEP-Taipei IHEP-CERNIHEP-CERN

Gang Chen/IHEP-CC 23/4/19 - 35

In near futureIn near future• expand computing power and storage expand computing power and storage

capacity.capacity.• Participate in LCG SC4 soon.Participate in LCG SC4 soon.• Promote HEP applications on Grid Promote HEP applications on Grid

platform.platform.• Establish more collaborations Establish more collaborations • Main issues:Main issues:

- Interoperability among other grid - Interoperability among other grid projects: CNGrid, ChinaGrid …projects: CNGrid, ChinaGrid …

- Network bandwidth upgrading.- Network bandwidth upgrading.

Gang Chen/IHEP-CC 23/4/19 - 36

Resource Plan at Resource Plan at IHEPIHEP

ResourcResourceses

Planned ResourcesPlanned Resources

20052005 20062006 20072007 20082008 20092009 20102010

CPU CPU (kSI2K)(kSI2K)

200200 600600 15001500 20002000 30003000 40004000

Disk (TB)Disk (TB) 33 100100 200200 600600 10001000 14001400

Tape (TB)Tape (TB) 1414 500500 10001000 20002000 30003000 44004400

Tape Tape (MB/s)(MB/s)

1515 8080 160160 200200 260260 300300

WAN WAN (Mb/s)(Mb/s)

10001000 10001000 10001000 1000010000 1000010000 1000100000

~50% for BES, ~50% LHC and others:~50% for BES, ~50% LHC and others:

Gang Chen/IHEP-CC 23/4/19 - 37

Thank You!Thank You!