Upload
ryann
View
20
Download
0
Tags:
Embed Size (px)
DESCRIPTION
CCJ Computing Center in Japan for spin physics at RHIC. T. Ichihara , Y. Watanabe, S. Yokkaichi, O. Jinnouchi, N. Saito, H. En’yo, M. Ishihara,Y.Goto (1) , S. Sawada (2) , H. Hamagaki (3) RIKEN , RBRC (1) ,KEK (2) , CNS (3) - PowerPoint PPT Presentation
Citation preview
CCJComputing Center in Japan for spin physics at RHIC
T. Ichihara, Y. Watanabe, S. Yokkaichi, O. Jinnouchi, N. Saito,
H. En’yo, M. Ishihara,Y.Goto(1) , S. Sawada(2), H. Hamagaki(3)
RIKEN, RBRC(1),KEK(2), CNS(3)
Presented on 17th October 2001 at First Joint meeting of the Nuclear Physics Division of APS and JPS (Hawaii 2001)
RIKEN CCJ : Overview Scope
Center for the analysis of RHIC Spin Physics Principal site of computing for PHENIX simulation
PHENIX CC-J is aiming at covering most of the simulation tasks of the whole PHENIX experiments
Regional computing center in Japan Size
Data amount: handing 225 TB /year Disk Storage : ~ 20 TB, Tape Library: ~500 TB CPU performance : 13 k SPECint95 (~300 Pentium 3/4 CPU)
Schedule R&D for the CC-J started in April ‘98 at RBRC in BNL Construction began in April ‘99 over a three years period CCJ started operation in June 2000 CCJ will reach full scale in 2002
Concept of RIKEN CCJ System
STKTapeRobot
HPSS
RCF
CRS
RawDST
BigDisk
SMPServers
CAS
DSTμDST
μDSTPhysics
40TB
Tape drive unitsto duplicate data
Tapes(50GB/volume)
Duplicating Facility
STKTapeRobot
importDST
DST
BigDisk
SMPServers
PC farmsfor ana. &simulation10k Spectnt95
DST μDST
15TB
PHENIX CC -J
μDSTsim.
μDST
Phys.
sim.ExportSim.
HPSSServers
HPSS
APAN/ESNETWAN
Trackreconstruction
20MB/s
HPSSServers
Tape drive unitsto duplicate data
Tapes(50GB/volume)
Duplicating Facility
PHENIX
Annual Data amount• DST 150 TB• micro-DST 45 TB• Simulated Data 30 TB• Total 225 TB
Hierarchical Storage System• Handle data amount of 225 TB/year• Total I/O bandwidth: 112 MB/s
• HPSS system
Disk storage system• 15 TB capacity• All RAID system• I/O bandwidth: 520 MB/s
System Requirement for CCJ
CPU ( SPECint95)
Simulation 8200
Sim. Reconst 1300
Sim. ana. 170
Theor. Mode 800
Data Analysis 2000
Total 12470 SPECint95
( = 120K SPECint2000)
Data Accessibility (exchange 225 TB /year)
Data Duplication Facility (tape cartridge) Wide Area Network (IMnet/APAN/ESnet)
Software EnviromentAFS, Objectivity/DB, Batch Queuing System
Requirements as a Regional Computing Center Software Environment
• Software environment of the CCJ needed be compatible to the PHENIX software environment at the RHIC Computing Facility (RCF) at BNL
• AFS (/afs/rhic) • Remote access over the network is very slow and unstable (AFS sever : BNL)• daily mirroring at CCJ and accessed via NFS from Linux farms
• Objectivity/DB • Remote access over the network is very slow and unstable (AMS server: BNL)• Local AMS server at CCJ (regularly update DB)
• Batch Queuing System: Load Sharing Facility (LSF 4.1) • RedHat Linux 6.1 (Kernel 2.2.19 with nfsv3 enabled), gcc-2.95.3 etc.• Veritas File System (Journaling file system on Solaris): free from fsck for ~TB disk
Data Accessibility• Need to exchange data of 225 TB/year to BNL RCF
• Most part of the data exchange is carried out with SD3 tape cartridges • Data duplicating Facility at both (10 MB/s performance by airbone)
• Some part of the data exchange is carried out over the Wide Area Network (WAN)• CC-J will use Asia-Pacific Advanced Network (APAN) for US-Japan connection
• http://www.apan.net/• APAN has currently ~100 Mbps bandwidth for Japan-US(STAR TAP) connection
• 3 MB/s transfer performance was obtained using bbftp protocol
Plan and current status of RIKEN CCJ
1998 1999 2000 2001 2002
RBRC(BNL)
R&D for CC-J
RIKENWako
Phase 1Phase 2
Phase 31/ 3 scale2/ 3 scale
full scale
Prot ot ype of CPU farmsDat a Duplicat ion facilit y
April April April April
CC-J review at BNL(Dec. 1 99 8 )
HPSS Soft ware/ HardwareInst allat ion ( March 1999)( Supplement ary Budget)
CC-J starts operationat 1 / 3 scale(J une. 2 00 0)
Full scale CC-J(Mar. 2 0 0 2 )
CC-Jconstruction
CC-J frontend at BNL
April
CC-J Working Groupformed (Oct. 1 9 9 8 )
PHENIX Exp. at RHIC
CC-J Operation
Tape Robot/driveupgrade
2003
April
Oct 2001 JFY2002CPU farm (number) 221 288+CPU farm (SPECint95) 9000 13000CPU farm (SPECint2000) 90 k 130kTape Storage size(TB) 115 460Disk Storage size(TB) 21 21+HPSS Tape Drive(model)
4(Redwood)
10(9940C)
HPSS Tape I/O (MB/s) 45 200Data Disk I/O (MB/s) 720 720+SUN SMP Data Server 4 4+HPSS Server unit 5 5+
Current configuration of CCJ
STKTapeRobot
HPSS
SP RouterAscend GRF
Tape Mover
Tape Mover
Disk Mover
HPSS Server
Disk Mover
HIPPI
HIPPISWITCH
288 GBLVD RAID
100BaseT x n
RIKEN LAN
RIKENsupercomputer
4 RedWooddrives
(Info. Blldg. 1F)
2F
32 Pentium III (700 MHz)+
96 Pentium III (850 MHz)+96 Pentium III (1000 MHz) 512 MB Memory /CPU
100 TB
1000BaseSX(9kB MTU)
1000BaseSX
ACSLS
GigabitSwitch #2 (L3)
1000BaseSX x 5 (9kB MTU)
1000BaseSX (9kB MTU)
WAN
IBM SP2 (AIX 4.3.2)
HPSS
Privateaddress
GigabitSwitch#1 (L3)
EPS-1000
Alteon 180(9kB MTU)
Alteon 180 (9KB MTU)
Gigabit SwitchCatalyst 2948G
Alta cluster) * 14 box
Switch
HPSSController
10/100BaseT
SUNACSLS
1000BaseSX
150 GB
RAID
compacDS20
AFS01AFS server
Altaclustercontrol WS
3.2 TBLVD RAID
288 GB
Raid (Work)
4 TB RaidHPSS Cache
ÇP00GB
SUN E450Data Server
5.7 TBFC RAID
6.0 TBFC RAID
6.0 TBFC RAID
ÇP00GB
SUN E450
Data Server
ÇP00GB
SUN E450
Data Server
ÇP00GB
SUN E450
Data Server
ÇP00GB
SUN E450
Data Server
Pentium III
Pentium III
Pentium III
Pentium III
Redhat 6.ÇPLinux
Pentium III
Pentium III
Pentium III
Pentium III
Redhat 6.ÇPLinux
Pentium III
Pentium III
Pentium III
Pentium III
Redhat 6.ÇPLinux
Pentium III
Pentium III
Pentium III
Pentium III
Redhat 6.ÇPLinux
Pentium III
Pentium III
Pentium III
Pentium III
Redhat 6.ÇPLinux
21.2 TB RAID
Pentium III
Pentium III
Pentium III
Pentium III
Redhat 6.ÇPLinux
Components of RIKEN CCJ
STK Tape Robot (100 TB [240 TB] )
HPSS Server (IBM RS-6000/SP)StorageTek Tape Robot (100TB [250 TB])
CPU Farm of 224 CPUsGigabit Network Switch
x
5.7 TBRAID
12TBRAID
Data server(SUN E450)
Data servers(SUN E450)
File Transfer over WAN s RIKEN - (Imnet) - APAN (~100 Mbps) -startap- ESnet - BNL
s Round Trip Time(RTT) for RIKEN-BNL :170 ms• RFC1323 (TCP Extensions for high performance, May 1992) describes the method of using large TCP window-size (> 64 KB)• FTP performance : 290 KB/s (64 KB windowsize), 640 KB/s(512KB window)
• Large TCP-window size is necessary to obtain high-transfer rate
BBFTP (http://doc.in2p3.fr/bbftp/) Software designed to quickly transfer files accross a wide area network. It has been written for the babar experiment in order to transfer big files transferring data through several parallel tcp streams RIKEN-BNL bbftp Performance (10 tcp streams for each session)
1.5 MB/s (1-sessinon) 3 MB/s (3-sessions)
MRTG graph during bbftp transfer between RIEKN and BNL
Analysis for the first PHENIX experiment with CCJ Quark Matter 2001 conference (Jan 2001)
• 21 papers using CCJ / total 33 PHENIX papers Simulation
• 100k event simulation/reconst. (Hayashi)• DC Simulation (Jane)• Muon Moc Data Challenge (MDC) (Satohiro) 100k Au+Au (June 2001)
DST Production • 1 TB of Raw Data (Year-1) was transferred via WAN from BNL to RIKEN• PHENIX official DST (v03) production (Sawada) (Dec. 2000)
- about 40% of DSTs produced at CCJ and transferred to BNL RCF Detector calibration/simulation
• TOF -(Kiyomochi,Chujo, Suzuki) • EMCal - (Torii,Oyama, Matsumoto)• RICH -(Akiba,Shigaki, Sakaguchi)
Physics/simulation• Single electron spectrum - (Akiba, Hachiya)• Hadoron particle ratio/spectrum ( Ohnishi, Chujo)• Photon [π0 spectrum] (Oyama, Sakaguchi)
CCJ Operation
Operation, maintenance and development of CC-J are carried out under the charge of the CCJ Planning and Coordinate Office (PCO).
CCJ Director (Chief Scientist of the Radiation Lab.)H. En’yo
Å@Planning and Coordination OfficeÅ@manager T. Ichihara (RIKEN and RBRC)
Å@technical manager Y. Watanabe (RIKEN and RBRC)
Å@scientific programming coordinator
Å@ H. En'yo (RIKEN and RBRC, PHENIX-EC)
Å@ H. Hamagaki (CNS-U-Tokyo, PHENIX-EC)
Å@PHENIX Liaison N. Saito (RIKEN and RBRC)
computer scientists S. Yokkaichi (RIKEN)
O. Jinnouchi (RIKEN)
Y. Goto (RBRC)
S. Sawada (KEK)
Å@
Technical Management OfficeÅ@Manager, Data duplication Y. Watanabe (RIKEN and RBRC)
Å@System engineer N. Otaki(IBM Japan)
Å@Tape duplication operator (TBD)
Construction of the CCJ started in 1999 The CCJ operation started in June 2000 at 1/3 scale.
• 43 user’s account created. Hardware/upgrade and software improvement :
• 224 Linux CPU(90k Specint2000), 21 TB disk, 100TB HPSS Tape library• Data Duplicating Facility at BNL RCF started operation in Dec 2000.
PHENIX Year -1 experiment : summer in 2000• Analysis of the PHENIX first experiment with CCJ
• Official DST (v03) Product ion • Simulations• Detector calibration/simulation• Physics analysis/simulation• Quark Matter 2001 conference (Jan 2001)
• 21 papers using CCJ / total 33 PHENIX papers
PHENIX Year 2 (2001) Run started in August 2001 Spin experiments at RHIC will start in December 2001 !!
Summary