Program Systems Institute of the Russian Academy of Sciences Supercomputer Projects SKIF and...

Program Systems Institute of the Russian Academy of Sciences

Supercomputer Projects SKIF and SKIF-GRID of Russia and Belorussia

Sergei AbramovWorkshop introducing the AURORA

project4 June 2009 Conference Room FBK,

via Sommarive, 18 - Povo. Trento, Italy

SKIF-AURORA SKIF-AURORA ProjectProject

Program Systems Institute of RAS — Supercomputer Project «SKIF-GRID»

Outline

Pereslavl-Zalessky and Program Systems Institute of the RAS: Short introduction

2008–2010: Series 4 of SKIF supercomputersSeries 4 of SKIF supercomputer == SKIF-

AURORASKIF-AURORA Selected Topics

Management Subsystem 3D-torus Interconnect Combining standard CPUs and FPGA-accelerators

Pereslavl-Zalessky andPereslavl-Zalessky andProgram Systems Institute of the Program Systems Institute of the

RAS:RAS:Short introductionShort introduction

Pereslavl-Zalessky

Beautiful ancient Russian Beautiful ancient Russian town, 860 years oldtown, 860 years old

The center of the The center of the Russian Golden Ring Russian Golden Ring CityCity

Hometown of Great Hometown of Great Dukes of RussiaDukes of Russia

The first building site The first building site Peter The Great navyPeter The Great navy

Ancient capital of Russian Ancient capital of Russian Orthodox churchOrthodox church

Moscow

Pereslavl-Zalessky

PSI RAS, Pereslavl-Zalesski

Foundation of the Institute

The Program Systems Institute was founded in 1984 by a decree of the USSR government. The foundation was aimed at the development of computer science in the country.The first (1984–2003) director of the Institute wasProf. A. Ailamazyan

Program Systems Institute of RAS — Supercomputer Project «SKIF-GRID»2009: Organization of the Institute

Artificial Intelligence Research Center

Medical Informatics Research Center

Research Center for Multiprocessor Systems

System Analysis Research Center

Control Processes Research Center

Program Systems Institute of RAS — Supercomputer Project «SKIF-GRID»Ailamazyan University of Pereslavl

Supercomputer Supercomputer ProjectsProjects

SKIF and SKIF-GRID SKIF and SKIF-GRID ofof

Russia and BelorussiaRussia and Belorussia

Program Systems Institute of RAS — Supercomputer Project «SKIF-GRID»SKIF and SKIF-GRIDSupercomputing Projects

Joint Supercomputing Projects ofRussian Federation and Republic of Belarus

R&D in all directions and levels of supercomputer and grid-technologies: hardware, operating system, parallel programming systems, applications etc.

SKIF: 2000–2004,10 + 10 = 20 organizations

SKIF-GRID: 2007–2010,12 + 23 = 35 organizations

PSI RAS is lead organizationfrom Russian Federation

SKIF-GRID Project organization

Project directions1.Grid technology2.Supercomputer

• SW

• HW3.Security4.Pilot projects —

applications of HPC and grid technology

Program Systems Institute of RAS — Supercomputer Project «SKIF-GRID»Series 1, 2, and 3 of the SKIF supercomputers

Series 1 (2000–2003)2000: SKIF Firstborn 0.02/0.0112001: SKIF ВМ-5100 0.048/0.0262003: SKIF ES1710.03 0.04/.023

Series 2 (2003–2007)2003: SKIF -Forge-32 0.1/0.0742003: SKIF K-500 0.717/0.4172004: SKIF К-1000 2.53/2.03

Series 3 (2007–2008)2007: SKIF Cyberia 12/9.012008: SKIF Ural 15.94/12.2 2008: SKIF MSU 60/47.17

Program Systems Institute of RAS — Supercomputer Project «SKIF-GRID»Flagship of SKIF supercomputers:SKIF MSU (March 2008)

June 2008: #36 in Top500Peak performance 60 Tflops, Linpack: 47

TflopsOriginal blade design, CPU model: 4-cores

Intel XEON E5472 3,0 GHzNodes (dual CPU): 625CPU cores total: 5,000 Interconnect:

Infiniband DDR,Fat Tree

10,000,000

1,000,000

100,000

10,000

Top 1Top10Top100Top200Top300Top400Top500M ade in Russia

LipackGflops

2002 JuneMVS 1000M0.734/1.024 TFlops

2003 NovemberSKIF K-5000.423/0.717 TFlops

2004 NovemberSKIF K-10002.032/2.534 TFlops

2007 FebruarySKIF Cyberia9.013/12.002 TFlops

2008 MaySKIF Ural12.2/15.9 TFlops

2008 майSKIF MSU47.1/60 TFlops

Only six developed in Russia supercomputers were ranked in the Top500… Five of them are SKIFs

Top1Top10Top100Top200Top300Top400Top500TopSKIF

100,000,000

1,000,000

100,000

10,000

LipackGflops

10,000,000

SeriesSeries11

Series Series 22

Series Series 33

Series Series 44

2032 Gflops SKIF K-1000472 Gflops SKIF K-500

57 Gflops Firstborn-M26 Gflops VM510011 Gflops Firstborn

47.17 Tflops SKIF MSU12.2 Tflops SKIF Ural9 Тflops SKIF Syberia

Top1Top10Top100Top200Top300Top400Top500TopSKIF

100,000,000

1,000,000

100,000

10,000

LipackGflops

10,000,000

1Q 2012 SKIF P~5.03Q 2010 SKIF P-1.03Q 2009 SKIF P-0.5

Completed: Series 1–3Nearest plan: Series 4

Linpack

Series 1, 2, 3 and 4 of SKIF supercomputers

2008–2010: 2008–2010: Series Series 44 of ofSKIF supercomputersSKIF supercomputers

SKIF Series 4: Aims of R&D

Highest density of performance(biggest possible number CPU per 1U) Smaller latency Less cables and connectors — better reliability Enlarged emission of heat per 1U

• We need new technology of cooling… How to? Improved Interconnect: we need better

scalability, bandwidth and latency that it’s provided by best available solutions (eg. Infiniband QDR)

New approach to monitoring and management of the supercomputer

Combining standard CPUs and accelerators in computational nodes of the supercomputer

Program Systems Institute of RAS — Supercomputer Project «SKIF-GRID»Spring’2008: SKIF Series 4 — How To?

How to enlarge number of

CPU per 1U?

How to enlarge number of

CPU per 1U?

How to cool supercomputer

nodes?

How to cool supercomputer

nodes?

How to developimproved

interconnect?

How to developimproved

interconnect?

How to combinestandard CPUs and

accelerators?

How to combinestandard CPUs and

accelerators?

How developmanagementsubsystem?

SKIF series 4SKIF series 4is extremelyis extremely

complex project.complex project.We need strongWe need strong

partners!partners!

Program Systems Institute of RAS — Supercomputer Project «SKIF-GRID»Summer’2008: SKIF Series 4 — Know How!

Italian-Russian Cooperation«SKIF Series 4» ==

«SKIF-AURORA Project»Designed by an alliance of

Eurotech, PSI RAS and RSC SKIF with support by Intel

Program SystemsInstitute of RAS

Program Systems Institute of RAS — Supercomputer Project «SKIF-GRID»SKIF-AURORA: Designed by the alliance of Eurotech, PSI RAS and RSC SKIF

PCBs, schematics,

mechanics, power

supply, cooling,

1 and 2 levels of

management system

3 level of management

system, Interconnect

(3D-torus: firmware,

routing, drivers,

MPI-2…), FPGA as

accelerator

SKIF-AURORA:SKIF-AURORA:State of the ProjectState of the Project

Node Card

19 апреля, 2023

PSU Card

Root Card

Chassis

Program Systems Institute of RAS — Supercomputer Project «SKIF-GRID»ISC’09, Hamburg, June 23–25, 2009

RackRack

Program Systems Institute of RAS — Supercomputer Project «SKIF-GRID»SystemPoject SKIF-Aurora 500 Tflops

SKIF-AURORA:SKIF-AURORA:Management SubsystemManagement Subsystem

Program Systems Institute of RAS — Supercomputer Project «SKIF-GRID»Subjects of Management Subsystem

1 Pflops = 42 racks == 10,752 nodes+ 672 DC/DC trays+ 672 root nodes

For scalability we need robust and redundant management subsystem

Comprehensive monitoring and control in all situations

Program Systems Institute of RAS — Supercomputer Project «SKIF-GRID»1st Level of Management Subsystem

Standard solution: IPMI over TCP/IP (Infiniband)

Available when nodes, root card, and IB-network are powered on and work properly

Root cards and DC/DC trays are not covered by monitoring and control

Program Systems Institute of RAS — Supercomputer Project «SKIF-GRID»2nd Level of Management Subsystem

Catalyst module on the root card implements node power control and serial console for the nodes

Available when root card and IB-network are powered on and work properly

Root cards and DC/DC trays are not covered by monitoring and control

Program Systems Institute of RAS — Supercomputer Project «SKIF-GRID»3rd Level of Management Subsystem

SKIF Servnet: independent sensor network

Available always, uses dedicated power network, power consumption: 3W per chassis

Accessible over dedicated network: Ethernet + CANbus + I2C

Monitors temperature, humidity, supply voltages on node cards, root card, DC/DC tray. Transfer this information to 2nd level (to Catalist)

Can turn off DC/DC PSU in case of emergency

Program Systems Institute of RAS — Supercomputer Project «SKIF-GRID»SKIF-AURORA Management Subsystem: Total monitoring and control

3-way redundantDesigned for “dark

datacenter”Robust management

subsystem

SKIF-AURORA Management Subsystem: Total monitoring and control

3D-torus Interconnect3D-torus Interconnect

3D-torus Interconnect

Only QCD specific is implemented by Italian teamRussian teams to upgrade network to general-

purpose interconnect (MPI 2.0)Due to appear fall 2009 Support and improvements in 2010–2012

Program Systems Institute of RAS — Supercomputer Project «SKIF-GRID»3D-torus Interconnect. Current status

Simple rounting implemented on a prototype (SKIFino)

Routing on single-FPGA prototype is working

MPI is based on MPICH2 codebase — prototyped

MPICH2 self-test implemented

Program Systems Institute of RAS — Supercomputer Project «SKIF-GRID»R&D Directions Using FPGA Resources

Collective MPI operations using FPGA FPGA to facilitate support of PGAS-languages

(UPC, Titanium, etc) FPGA+CPU hybrid computing

System Interconnect, 3D-torus

Subsidiary Interconnect, Infiniband

FPGA FPGA FPGA FPGA...

CPU CPU CPU CPUstandard part

non-standard part

ConclusionsConclusions

Conclusions

SKIF-AURORA project Is based on collaboration between international

teams Harnesses shared expertise and results Aimed to develop a family of top-level

supercomputers with innovative techniques: Higher density of CPUs (flops per volume) Efficient water cooling system Efficient power supply system Scalable powerful 3D-Torus Interconnect Most modern standard CPUs for computation and

FPGA for its acceleration Redundant robust management subsystem Etc.

Conclusions

The collaboration between Italian and Russian teams Allows to obtain world class supercomputer

technologies Provides leading positions in supercomputer

industry (at least in the nearest future) for all participants of the collaboration

Makes all results available in reasonable time and by reasonable efforts and resources

Grazie per l’attenzione

Program Systems Institute of the Russian Academy of Sciences Supercomputer Projects SKIF and...

Documents

La escala del tiempo, Alexander y Serguei Abramov

The SKIF K-1000 Supercomputer 2.5 Tflopsskif.pereslavl.ru/psi-info/rcms/rcms-leaflets.eng/skif-k1000-leaflet-engl.pdfThe SKIF K-1000 Supercomputer The SKIF K-1000 Supercomputer was

Matiushkin, E. N., Abramov, V. K., Dunishenko, Y. M ... · Matiushkin, E. N., Abramov, V. K., Dunishenko, Y. M., Nikolaev, I. G., Pikunov, D. G., Salkina, G. P., Smirnov, E. N., and

Alexey Abramov , Christopher Bayer , Claudio Heller

Siberian circular low emittance light source SKIF · 2019. 8. 7. · SKIF 158.76 m х 3= 476.14 m (harmonic number 567) 14 beamlines from wigglers and undulators 8 hard X-ray beamlines

Beam Loss Diagnostics System for SKIF Synchrotron Light Source

kata boys U12 page-0001 - skif-2019KATA BOYS U12 - RESULTS SKIF Russia Ukrainian Shotokan karate-do federation SKIF SKIF Czech Republic 3 Banykin Roman / RUS 154 GAVRYSHKO MAKARIY

Vasily Abramov

Abramov-Hlebnyikov - Kis Gőzgépek Házi Készítese

Abramov L. Cafferty B. - Chess Move by Move

Analysis of Deep Attack Operations. Operation Bagration, Belorussia 22 JUNE-29 AUGUST 1944

Mourão & Abramov | Biofísica Essencial

Руководство системного программистаskif.pereslavl.ru/skif/index.cgi?module=chap&action=getpage&data=... · В предлагаемой программе

Skif security service - Presentation of logo concept

Campamento Continental de Karate SKIF América

RUSSIA & UP FR: SOVETSKAYA BELORUSSIA TO: SOVETSKAYA … · title: russia & up fr: sovetskaya belorussia to: sovetskaya estoniya subject: russia & up fr: sovetskaya belorussia to:

Abramov, Alexander & Abramov, Sergei - Viaje Por Tres Mundos

Ratinglist of the ICCF Wertungszahlenliste der ICCFkszgk.com/wp-content/uploads/2012/12/66.pdf · Abramov, Gennady Aleksandrovich RUS 2185 59 Abramov, Sergey Pavlovich RUS 2370 222

Belorussia obrez 123 200 - CIS-EMO · Кирилл Аверьянов-Минский, Владислав Мальцев Белорусский национализм против

ПРАВИЛА СОРЕВНОВАНИЙ ПО СЁТОКАН КАРАТЭ ДО (SKIF)dzanshin.com/wp-content/uploads/2017/12/skif-pravila.pdf · площадки 8 м (измерение