219
98 #*,+50! 14-325- MPI 714-325- .5/6 2018/4/26 7MPI 1 &$)"' hanawa 8 cc.u-tokyo.ac.jp (%#($

98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

  • Upload
    others

  • View
    1

  • Download
    0

Embed Size (px)

Citation preview

Page 1: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

�98 �#*,+50�!��14-325-���

�MPI�7��14-325-������ ���.5/6

2018/4/26 ���7MPI� 1

��&�$)"��'hanawa8 cc.u-tokyo.ac.jp(%� ��#($�

Page 2: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

%"���

• (��C2018� 4�26�A�B 10C30 - 17C00

• ��C�� �� 2?4@ 4) 413'*�&�(�+1:@1C3)328 �&�B

• %"�9>/=<C %�C�• 10C00 - 10C30 ��• 10C30 - 11C30 7@6830?-$�,5169>/=<-�#A�"B• 11C30 - 12C30 ��9>/=;?/-�A��B

(12C30 - 14C00 ��.)• 14C00 - 15C00 MPI9>/=<�"� A�"B• 15C10 - 16C00 MPI9>/=<�"�A�"B• 16C10 - 17C00 MPI9>/=<�"�A�"B

%"�CMPI! 22018/4/26

Page 3: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

4C5E+39/C2�+�136=H6�0-.AF)*(G

3

FY11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Yayoi: Hitachi SR16000/M1IBM Power-7

54.9 TFLOPS, 11.2 TB

Reedbush, HPEBroadwell + Pascal

1.93 PFLOPS

T2K Tokyo140TF, 31.3TB

Oakforest-PACSFujitsu, Intel KNL25PFLOPS, 919.3TB

BDEC System50+ PFLOPS (?)

Oakleaf-FX: Fujitsu PRIMEHPC FX10, SPARC64 IXfx1.13 PFLOPS, 150 TB

Oakbridge-FX136.2 TFLOPS, 18.4 TB

Reedbush-L HPE

1.43 PFLOPS

Oakbridge-IIIntel/AMD/P9/ARM CPU only

5-10 PFLOPS

Big Data & Extreme Computing

�����3E9E/C:?E5

>8E/,���3E9E/C:?E5F%#"$&#I��D G

7E5��D1<?BE1@C��3E9E/C:?E5

�!2@;���������'��3E9E/C:?E5

2018/4/26 ���IMPI��

Page 4: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

2D,)+3,4E126;&��• Oakleaf-FX (��% PRIMEHPC FX10)

• 1.135 PF, �0@9=B4��, 2012�4�� 2018�3�• Oakbridge-FX (��% PRIMEHPC FX10)

• 136.2 TF, '�( !�D168�(E, 2014�4�� 2018�3�• Reedbush (HPE, Intel BDW + NVIDIA P100 (Pascal))

• 7B4"�A1:=?B1>@ �2B8B0@9=B4• 2016-Jun.2016�7�G2020�6�

• ���3�*GPU�$126;• Reedbush-U: CPU only, 420 nodes, 508 TF (2016�7�)• Reedbush-H: 120 nodes, 2 GPUs/node: 1.42 PF (2017�3�)• Reedbush-L: 64 nodes, 4 GPUs/node: 1.43 PF (2017�10�)

• Oakforest-PACS (OFP) (��%, Intel Xeon Phi (KNL))• JCAHPC (���CCSC��ITC)• 25 PF, ���9� (2017�11�) (���2�)• Omni-Path -B.6/5<, DDN IME (Burst Buffer)

42018/4/26 #��FMPI�

Page 5: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

Oakforest-PACS (OFP)• 2016�12�1�(�:�• 8,208 Intel Xeon/Phi (KNL)=TZL�325PFLOPS

• ��8> -

• TOP 500 9�[ 2�\]HPCG 6�[ 2�\[2017�11�\

•��*�HPC �$�6(JCAHPC: Joint Center for Advanced High Performance Computing)• +#��5,'�%)PXQZ• �������$PXQZ

• �����KVXSOG�������$PXQZF=�";G�2�>��DEBC65@IOZSZMXTWZQNORUJ60?]��*G�4!<�35,�$J -Y9�@IAHG./

• http://jcahpc.jp

52018/4/26 71�^MPI�&

Page 6: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

Oakforest-PACS �'/&• Intel Xeon Phi (Knights Landing)

• 1'/&1#�$%, 68 �

• MCDRAM: �.)$�/!��(.&�*+-16GB

+ DDR4*+- ������

������������

2018/4/26 ��2MPI��

Knights Landing Overview

Chip: 36 Tiles interconnected by 2D Mesh Tile: 2 Cores + 2 VPU/core + 1 MB L2 Memory: MCDRAM: 16 GB on-package; High BW DDR4: 6 channels @ 2400 up to 384GB IO: 36 lanes PCIe Gen3. 4 lanes of DMI for chipset Node: 1-Socket only Fabric: Omni-Path on-package (not shown) Vector Peak Perf: 3+TF DP and 6+TF SP Flops Scalar Perf: ~3x over Knights Corner Streams Triad (GB/s): MCDRAM : 400+; DDR: 90+

TILE

4

2 VPU

Core

2 VPU

Core

1MB L2

CHA

Package

Source Intel: All products, computer systems, dates and figures specified are preliminary based on current expectations, and are subject to change without notice. KNL data are preliminary based on current expectations and are subject to change without notice. 1Binary Compatible with Intel Xeon processors using Haswell Instruction Set (except TSX). 2Bandwidth numbers are based on STREAM-like memory access pattern when MCDRAM used as flat memory. Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Omni-path not shown

EDC EDC PCIe Gen 3

EDC EDC

Tile

DDR MC DDR MC

EDC EDC misc EDC EDC

36 Tiles connected by

2D Mesh Interconnect

MCDRAM MCDRAM MCDRAM MCDRAM

3

DDR4

CHANNELS

3

DDR4

CHANNELS

MCDRAM MCDRAM MCDRAM MCDRAM

DMI

2 x161 x4

X4 DMI

HotChips27

KNL",�&��

Knights Landing: Next Intel® Xeon Phi™ Processor

First self-boot Intel® Xeon Phi™ processor that is binary compatible with main line IA. Boots standard OS.

Significant improvement in scalar and vector performance

Integration of Memory on package: innovative memory architecture for high bandwidth and high capacity

Integration of Fabric on package

Potential future options subject to change without notice. All timeframes, features, products and dates are preliminary forecasts and subject to change without further notification.

Three products

KNL Self-Boot KNL Self-Boot w/ Fabric KNL Card

(Baseline) (Fabric Integrated) (PCIe-Card)

Intel® Many-Core Processor targeted for HPC and Supercomputing

2 VPU 2 VPU

Core Core

1MB

L2

MCDRAM: 490GB/�� 0��1DDR4: 115.2 GB/=(8Byte�2400MHz�6 channel)

6

Page 7: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���������� ������������!Oakforest-PACS�������

2018/4/26 � �!MPI�� 7

Page 8: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��������

•� ����������� : t00871 ������ gt00

•���5/26 17:00����

2018/4/26 � ��MPI�� 8

Page 9: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

������������ �����������

���MPI�� 92018/4/26

Page 10: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

)4-1-3(/.�

•8�� 5Fortran90 ��,$&17Samples-ofp.tar.gz

• tar����8���Fortran90�� *%2'+0��!#"

• C/ : C���• F/ : Fortran90���

•�� ,$&1�����"��/work/gt00/z30105 (/home��� ��6

���7MPI�� 102018/4/26

Page 11: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���Hello5<*87%+=2(:�#�(1/2)1. cd +6=1%��� Lustre4&(:,-.7 ���$

$ cd /work/gt00/t008XX (���"��!ID �$��)2. /work/gt00/z30105 �$ Samples-ofp.tar.gz %��!/';)09 +3>�$$ cp /work/gt00/z30105/Samples-ofp.tar.gz ./

3. Samples-ofp.tar %���$$ tar xvfz Samples-ofp.tar.gz

4. Samples/';)09 �$$ cd Samples

5. C�� ? $ cd CFortran90�� ? $ cd F

6. Hello/';)09 �$$ cd Hello

2018/4/26 ���?MPI� 11

Page 12: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���Hello&+!)'�",# *���(2/2)6. $(�MPI��Makefile (Makefile_pure)����make��$ make -f Makefile_pure

7. ��%� *(hello)��������� ��

$ ls

2018/4/26 ���-MPI� 12

Page 13: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

#!/bin/bash

#PJM -L rscgrp=lecture-flat

#PJM -L node=16

#PJM --mpi proc=1088

#PJM -L elapse=0:01:00

#PJM -g gt00

mpiexec.hydra –n

${PJM_MPI_PROC} ./hello

JOB"�.+& 1+/��3)-%&89743hello-pure.bash, C���Fortran����4

2018/4/26 ���6MPI� 13

.$2"�/2+�6lecture-flat

���/2+�6gt00

MPI!,*�68*16 = 1088 +0#"������

��(2'�MPI+0#"

������65�

Page 14: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

2018/4/26 ���@MPI�� 14

• Intel Xeon Phi (Knights Landing)• 10=/1+&-.����'#Knights Landing Overview

Chip: 36 Tiles interconnected by 2D Mesh Tile: 2 Cores + 2 VPU/core + 1 MB L2 Memory: MCDRAM: 16 GB on-package; High BW DDR4: 6 channels @ 2400 up to 384GB IO: 36 lanes PCIe Gen3. 4 lanes of DMI for chipset Node: 1-Socket only Fabric: Omni-Path on-package (not shown) Vector Peak Perf: 3+TF DP and 6+TF SP Flops Scalar Perf: ~3x over Knights Corner Streams Triad (GB/s): MCDRAM : 400+; DDR: 90+

TILE

4

2 VPU

Core

2 VPU

Core

1MB L2

CHA

Package

Source Intel: All products, computer systems, dates and figures specified are preliminary based on current expectations, and are subject to change without notice. KNL data are preliminary based on current expectations and are subject to change without notice. 1Binary Compatible with Intel Xeon processors using Haswell Instruction Set (except TSX). 2Bandwidth numbers are based on STREAM-like memory access pattern when MCDRAM used as flat memory. Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Omni-path not shown

EDC EDC PCIe Gen 3

EDC EDC

Tile

DDR MC DDR MC

EDC EDC misc EDC EDC

36 Tiles connected by

2D Mesh Interconnect

MCDRAM MCDRAM MCDRAM MCDRAM

3

DDR4

CHANNELS

3

DDR4

CHANNELS

MCDRAM MCDRAM MCDRAM MCDRAM

DMI

2 x161 x4

X4 DMI

HotChips27 KNL)8$/!"

Knights Landing: Next Intel® Xeon Phi™ Processor

First self-boot Intel® Xeon Phi™ processor that is binary compatible with main line IA. Boots standard OS.

Significant improvement in scalar and vector performance

Integration of Memory on package: innovative memory architecture for high bandwidth and high capacity

Integration of Fabric on package

Potential future options subject to change without notice. All timeframes, features, products and dates are preliminary forecasts and subject to change without further notification.

Three products

KNL Self-Boot KNL Self-Boot w/ Fabric KNL Card

(Baseline) (Fabric Integrated) (PCIe-Card)

Intel® Many-Core Processor targeted for HPC and Supercomputing

2 VPU 2 VPU

Core Core1MBL2

MCDRAM: 490GB/��� >��?DDR4: 115.2 GB/�=(8Byte�2400MHz�6 channel)

37#BCA �� >0=/�?

BCA4;*)

�� ,$:(�)

MCDRAM: %<2-&=( �1</56916GB+ DDR4569 ������������������

Page 15: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��Hello-3(0.$� �#�6,/%MPI7•� )4-2 JOB*'1-+!hello-pure.bash���

•�� )4-2�!�&/5��” lecture-flat”�����"�

•$ emacs hello-pure.bash��“lecture-flat” → “tutorial-flat”��������

2018/4/26 ���8MPI�� 15

Page 16: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���HelloIO?MJ:��-6*QFK<MPIR1. HelloG>NC�1��:��.8

$ pjsub hello-pure.bash2. �3��,9/@LH:�".8

$ pjstat3. ��+��.82'��3G;=N+��,98

hello-pure.bash.eTTTTTThello-pure.bash.oTTTTTT QXXXXXX4��R

4. �!3���G;=N3�$: 058$ cat hello-pure.bash.oTTTTTT

5. %Hello parallel world!&+'68IOBA*16EPD=1088��,90)/7� (

2018/4/26 #��SMPI�� 16

Page 17: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

/,+)31��!"�������'48��• /,+)31������"������0$&6���'48��0$&6��)31�� �-%7(.5����# ��

• ����0$&6��)31����������'48��0$&6��)31���'482,*8)����# ��

2018/4/26 ���9MPI� 17

)31�.oXXXXX ---����0$&6)31�.eXXXXX ---��'48��0$&6(XXXXX�)31�� ����#")31�)31ID)

Page 18: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���Hello7:398-"�;C !<

#��=MPI�� 18

#include <stdio.h>#include <mpi.h>

int main(int argc, char* argv[]) {

int myid, numprocs;int ierr, rc;

ierr = MPI_Init(&argc, &argv);ierr = MPI_Comm_rank(MPI_COMM_WORLD, &myid);ierr = MPI_Comm_size(MPI_COMM_WORLD, &numprocs);

printf("Hello parallel world! Myid:%d ¥n", myid);

rc = MPI_Finalize();

exit(0);}

MPI-��

��-ID��2��=�?>+�.�,0

��-7:564 �2��=�?>+�.�*;����+.1088%/)&.16<

MPI-��

'-7:398.%�?>+$(10

2018/4/26

Page 19: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���Hello7:398-"�;Fortran !<

#��=MPI�� 19

program mainuse mpiimplicit noneinteger :: myid, numprocsinteger :: ierr

call MPI_INIT(ierr)call MPI_COMM_RANK(MPI_COMM_WORLD, myid, ierr)call MPI_COMM_SIZE(MPI_COMM_WORLD, numprocs, ierr)

print *, "Hello parallel world! Myid:", myid

call MPI_FINALIZE(ierr)

stopend program main

MPI-����-ID��2��=�?>+�.�,0

��-7:564 �2��=�?>+�.�*;����+.

1088%/)&.16<MPI-��

'-7:398.%�?>+$(10

2018/4/26

Page 20: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��7��,����

��• !�%"& $� ��+-5 1*�…Hello parallel world1. ,�6�3

$ grep Hello hello-pure.bash.oXXXX | wc -l

1088(��#44.OK!

myid2. !��$&�3 )� �.1.1+��#43,'5 1*�

$ grep Hello hello-pure.bash.oXXXX | sort | less

Myid: 0 1/2�Myid: 1087'�54.0�

2018/4/26 ���7MPI�� 20

Page 21: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

����*�.2:8*�"�>36592:8�40=<2:8?• �.2:831;97 go1.sh * )�go2.sh /�"&�• $-)�go2.sh* )�go3.sh/�"&��(��#(��.• ��/�36592:8(,&+40=<2:8)(���• Oakforest-PACS)�!.36592:8*�"�

1. $pjsub --step go1.sh [INFO] PJM 0000 pjsub Job 800967_0 submitted.

2. ��*2:8�800967/��'� ���*��/%.$pjsub --step --sparam jid=800967 go2.sh[INFO] PJM 0000 pjsub Job 800967_1 submitted

3. ���

$pjsub --step --sparam jid=800967 go3.sh[INFO] PJM 0000 pjsub Job 800967_2 submitted

2018/4/26 ���@MPI�� 21

Page 22: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

������������ ��!��� ������� ��� ��

2018/4/26 ���"MPI�� 22

Page 23: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

�+�~$/��• Wqtn|u{mxv|m�7���(bLNKc�/�X

• %" �# 0U

• !�&�UKOHMECD;@;=?;<B>@A?;@U)���><=?�?�=>�U �FGA:9><<8

• Y �c'�Z• I24a15• I24UJRSTSPQE<24cp|uzu{mxw\��•��kzoyrwdU�ai[hg_]5�• 6.c��j�`ls}•��d�-V�f`����3,j�e��^c�7�

6/��MPI�* 232018/4/26

Page 24: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

�)��",��

• `�y~q|z�q�5�s�y}y~q|{g�kMXSVKNhMXSVFHHa��6�

• # �! .

• ���$�^JOGL:<;E7?<>;A=?@A>^JOGL:<>E7DBC:?<>;A=?@A@^'��� =;<@�@�=@�

• b��i%�c• H12^IWY[YQVD;12g03• H12^IWY[YQVD;12i/�is�y}y~q|{d���-�uo�~�v���

• �4+i��n�fpw�• PUVRW\Z7NH",�-8H]T\UV�&9_txr�gm",�-_• ��j*_lf�y~q|z�qn�k� ei�5�

4,��MPI�( 242018/4/26

Page 25: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

�0�

• Okljmh+gH<Z`�,Te��`��b[P

• ��% M'#�&M;$�� 1• "�(�MJLIK=?>H<B?A>DABCC>MJLIK=?AH<GEF=B?A>DABCCEM*��n@>?C�@�@>�M?ED9

• Q �`)�R• kljm`35�[XN��hTfdXV35WbXN

•kljma�_�SgT•kljma]i^�.c[M^Y8V4-[Ug`T•���2M��`6:\�!��M^]

2018/4/26 7/�nMPI�, 25

Page 26: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

�U9

• ���6��F l fa��0V������ l�

• bG,@ >!54 F� RW�D?#C >!E��54 � ªF$« W�gG�B 1"!�54� ªF$« W�)=;� M2!54 � ª*$« W� �J<C©7OY�3 W�

• �§¡L�IX+:7®pnonmnrmqnh­ � �® ws­ £¨�6®ptpd�z|x{®vtulrlqqvlnpsuvlt­ % ®qkvvn� ª=�qkunn�¬Nsijh

• �=9�E/�• y����~�\^�[_• 6��¦�¥�¤��6-����'�_8• =`S��&������;��e�[A�HX��.[A�yy}��¨ �����Z�6�]P�¦�¥�¤��¢¨

• �&��Q¯�Q�(c�����6�]P�$������

2018/4/26 `T®MPI�K 26

Page 27: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���$�%

1. ��)7-7(6.27*$��

2. ��/5'316'$��

3. ����

4. ���#MPI��5. +7*�� �

6. 0&,4" !$��

7. 0&,4-���

���8MPI�� 272018/4/26

Page 28: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

����������� ��

���MPI�� 282018/4/26

Page 29: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

4C5E+39/C2�+�136=H6�0-.AF)*(G

29

FY11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Yayoi: Hitachi SR16000/M1IBM Power-7

54.9 TFLOPS, 11.2 TB

Reedbush, HPEBroadwell + Pascal

1.93 PFLOPS

T2K Tokyo140TF, 31.3TB

Oakforest-PACSFujitsu, Intel KNL25PFLOPS, 919.3TB

BDEC System50+ PFLOPS (?)

Oakleaf-FX: Fujitsu PRIMEHPC FX10, SPARC64 IXfx1.13 PFLOPS, 150 TB

Oakbridge-FX136.2 TFLOPS, 18.4 TB

Reedbush-L HPE

1.43 PFLOPS

Oakbridge-IIIntel/AMD/P9/ARM CPU only

5-10 PFLOPS

Big Data & Extreme Computing

�����3E9E/C:?E5

>8E/,���3E9E/C:?E5F%#"$&#I��D G

7E5��D1<?BE1@C��3E9E/C:?E5

�!2@;���������'��3E9E/C:?E5

2018/4/26 ���IMPI��

Page 30: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

2D,)+3,4E126;&��• Oakleaf-FX (��% PRIMEHPC FX10)

• 1.135 PF, �0@9=B4��, 2012�4�� 2018�3�• Oakbridge-FX (��% PRIMEHPC FX10)

• 136.2 TF, '�( !�D168�(E, 2014�4�� 2018�3�• Reedbush (HPE, Intel BDW + NVIDIA P100 (Pascal))

• 7B4"�A1:=?B1>@ �2B8B0@9=B4• 2016-Jun.2016�7�G2020�6�

• ���3�*GPU�$126;• Reedbush-U: CPU only, 420 nodes, 508 TF (2016�7�)• Reedbush-H: 120 nodes, 2 GPUs/node: 1.42 PF (2017�3�)• Reedbush-L: 64 nodes, 4 GPUs/node: 1.43 PF (2017�10�)

• Oakforest-PACS (OFP) (��%, Intel Xeon Phi (KNL))• JCAHPC (���CCSC��ITC)• 25 PF, ���9� (2017�11�) (���2�)• Omni-Path -B.6/5<, DDN IME (Burst Buffer)

302018/4/26 #��FMPI�

Page 31: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

Reedbush����Reedbush-U2016�7�1� �����2016�9�1� ����

Reedbush-H2017�3�1� �����2017�4�3� ����

Reedbush-L2017�10�2� �����2017�11�1� ����

31

Top500: RB-L 291�@Nov. 2017RB-H 203�@Jun. 2017RB-U 361�@Nov. 2016

Green500: RB-L 11�@Nov. 2017RB-H 11�@Jun. 2017

2018/4/26 ����MPI��

Page 32: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

2018/4/26 ����MPI�� 32

外部接続ルータ 1Gigabit/10Gigabit Ethernet Network

InterConnect( 4x EDR InfiniBand) InterConnect( 4x EDR InfiniBand)

ログインノード群SGI Rackable C1110-GP2

6nodes

NFS Filesystem16TB

Lustre FilesystemDDN SFA14KE x3set

5.04PB

高速キャッシュDDN IME14K x6set

209TB

NAS Storage24TB

E5-2680v4 2.4GHz 14core,256GiB Mem

管理サーバ群SGI Rackable C1110-GP2

9nodes

GbE SW

x6

x6 x2x2(for PBS)

Reedbush-Hx240 (FDRx2/node)

Reebush-Ux420

x36(IME:6x6) x24(OSS(VM):x 12 x2)

x4(MDS:x 2)

x12

高速キャッシュDDN IME240 x8set

153.6 TB

管理用補助サーバ

SGI RackableC1110-GP2 x2

x16(IME:8x2) x2x12

x8 x2x10(Ctrl:8,MDS:2)

Reedbush-Lx128( EDR x2/node) x4

x6 x9x4

x9

x64

x120

x420

x9

Management port

管理コンソールMac Pro

電力管理サーバ

電力計器

Reedbush-USGI Rackable C2112-4GP3420 nodes, 508.03TFLOPS・CPU : E5-2695v4 2.1GHz 18core

Reedbush-HSGI Rackable C1102-GP8120 nodes, 240GPUs, 1.418PFLOPS・CPU : E5-2695v4 2.1GHz 18core ・GPU : NVIDIA Tesla P100 SXM2 x2/node

Reedbush-LSGI Rackable C1102-GP864 nodes, 256GPUs, 1.434PFLOPS・CPU : E5-2695v4 2.1GHz 18core ・GPU : NVIDIA Tesla P100 SXM2 x4/node

E5-2680v4 2.4GHz 14core,128GiB Mem

ライフ/管理ネットワーク 1Gigabit/10Gigabit Ethernet Network

Page 33: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

Reedbush�������

33

Reedbush-U Reedbush-H Reedbush-L

CPU/node Intel Xeon E5-2695v4 (Broadwell-EP, 2.1GHz, 18core) x 2sockets (1.210 TF), 256 GiB (153.6GB/sec)

GPU - NVIDIA Tesla P100 (Pascal, 5.3TF, 720GB/sec, 16GiB)

Infiniband EDR FDR�2ch EDR�2ch

���� 420 120 64

GPU� - 240 (=120�2) 256 (=64�4)

����(TFLOPS) 509 1,417

(145 + 1,272)1,433

(76.8 + 1,358)

�������(TB/sec) 64.5 191.2

(18.4+172.8)194.2

(9.83+184.3)

�� � 2016.07 2017.03 2017.10

2018/4/26 �� MPI��

Page 34: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

Oakforest-PACS [OFP\• 2016�12�1�(�:�• 8,208 Intel Xeon/Phi (KNL)]TZL�325PFLOPS

• ��8> -

• TOP 500 9�[ 2�\]HPCG 6�[ 2�\[2017�11�\

•��*�HPC �$�6(JCAHPC: Joint Center for Advanced High Performance Computing)• +#��5,'�%)PXQZ• �������$PXQZ

• �����KVXSOG�������$PXQZF=�";G�2�>��DEBC65@IOZSZMXTWZQNORUJ60?]��*G�4!<�35,�$J -Y9�@IAHG./

• http://jcahpc.jp

342018/4/26 71�^MPI�&

Page 35: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

Oakforest-PACSの特徴 (1/2)• ��+7)

• 1+7) 68#�;3TFLOPS�8,208+7)= 25 PFLOPS

• 0138MCDRAM8��;16GB9:DDR48��;96GB99

• +7)���• -4, &!$26,6)���

Fat-Tree*'(57!• ��� ���/3"7$26����;�%2.�

• Intel Omni-Path Architecture

352018/4/26 ���<MPI�� 35

Page 36: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

Oakforest-PACS !��36���=MPI��2018/4/26

�/:$�� � 25 PFLOPS-:,� 8,208��-:,

Product �� PRIMERGY CX600 M1 (2U) + CX1640 M1 x 8node

18)*& Intel® Xeon Phi™ 7250;��%:,: Knights Landing<68 %" 1.4 GHz

346 �.9,� 16 GB, MCDRAM, � 490 GB/sec�.9,� 96 GB, DDR4-2400, /:$ 115.2

GB/sec�����

Product Intel® Omni-Path Architecture69$�� 100 Gbps+28( 07.#)$'59.9,�Fat-tree�

36

Page 37: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

Oakforest-PACS +��F2 / 2G• >01CI/O

• ��>01C569?:Lustre 26PB

• >01C2@85A569?FDDN IMEGH1TB/sec/$'.����,�1PB• ��D=84:E7��D���*,"�

• �#%�• Green 500),��6�

(2016.11)• LinpackH 2.72 MW

• 4,986 MFLOPS/WFOFPG• 830 MFLOPS/WF�G

��>01C569?

>01C2@85A569?

B83�(-120<E;+&� ��

372018/4/26 !��HMPI� 37

Page 38: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

Oakforest-PACS �� 1��2��)��/"#%*

Type Lustre File System�� 26.2 PBProduct DataDirect Networks SFA14KE�'0&� 500 GB/sec

��)��/ +$","#%*

Type Burst Buffer, Infinite Memory Engine (by DDN)

�� 940 TB (NVMe SSD,(.%����)Product DataDirect Networks IME14K�'0&� 1,560 GB/sec

����� 4.2MW1�����2�-$!� 102

382018/4/26 ���3MPI� 38

Page 39: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

Oakforest-PACS ��'�• Intel Xeon Phi (Knights Landing)

• 1�'�1����

• MCDRAM: �&!��'��� &��"#%16GB+ DDR4"#%

2018/4/26 ���*MPI�

Knights Landing Overview

Chip: 36 Tiles interconnected by 2D Mesh Tile: 2 Cores + 2 VPU/core + 1 MB L2 Memory: MCDRAM: 16 GB on-package; High BW DDR4: 6 channels @ 2400 up to 384GB IO: 36 lanes PCIe Gen3. 4 lanes of DMI for chipset Node: 1-Socket only Fabric: Omni-Path on-package (not shown) Vector Peak Perf: 3+TF DP and 6+TF SP Flops Scalar Perf: ~3x over Knights Corner Streams Triad (GB/s): MCDRAM : 400+; DDR: 90+

TILE

4

2 VPU

Core

2 VPU

Core

1MB L2

CHA

Package

Source Intel: All products, computer systems, dates and figures specified are preliminary based on current expectations, and are subject to change without notice. KNL data are preliminary based on current expectations and are subject to change without notice. 1Binary Compatible with Intel Xeon processors using Haswell Instruction Set (except TSX). 2Bandwidth numbers are based on STREAM-like memory access pattern when MCDRAM used as flat memory. Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Omni-path not shown

EDC EDC PCIe Gen 3

EDC EDC

Tile

DDR MC DDR MC

EDC EDC misc EDC EDC

36 Tiles connected by

2D Mesh Interconnect

MCDRAM MCDRAM MCDRAM MCDRAM

3

DDR4

CHANNELS

3

DDR4

CHANNELS

MCDRAM MCDRAM MCDRAM MCDRAM

DMI

2 x161 x4

X4 DMI

HotChips27 KNL�$����

Knights Landing: Next Intel® Xeon Phi™ Processor

First self-boot Intel® Xeon Phi™ processor that is binary compatible with main line IA. Boots standard OS.

Significant improvement in scalar and vector performance

Integration of Memory on package: innovative memory architecture for high bandwidth and high capacity

Integration of Fabric on package

Potential future options subject to change without notice. All timeframes, features, products and dates are preliminary forecasts and subject to change without further notification.

Three products

KNL Self-Boot KNL Self-Boot w/ Fabric KNL Card

(Baseline) (Fabric Integrated) (PCIe-Card)

Intel® Many-Core Processor targeted for HPC and Supercomputing

2 VPU 2 VPU

Core Core1MBL2

MCDRAM: 490GB/�� (��)DDR4: 115.2 GB/=(8Byte�2400MHz�6 channel)

�������"#%�*16GB�6+96GB

39

Page 40: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

Oakforest-PACS: Intel Omni-Path Architecture *.0?D>2835BE>E<�Fat-tree�

768 port Director Switch12�(Source by Intel)

48 port Edge Switch362 �

2 2

241 4825 7249

Uplink: 24

Downlink: 24

. . . . . . . . .

47;,%%0&?D>2835BE>E<�1��• 57:A�����*-#$����1�• �!)"�G6B@*�'0��=F<�/�(+�� &#$

402018/4/26 ��GMPI�

��=F<C93*��

Page 41: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

41

http://www.top500.org/

Site Computer/Year Vendor Cores Rmax(TFLOPS)

Rpeak(TFLOPS)

Power(kW)

1 National Supercomputing Center in Wuxi, China

Sunway TaihuLight , Sunway MPP, Sunway SW26010 260C 1.45GHz, 2016 NRCPC

10,649,600 93,015(= 93.0 PF) 125,436 15,371

2 National Supercomputing Center in Tianjin, China

Tianhe-2, Intel Xeon E5-2692, TH Express-2, Xeon Phi, 2013 NUDT 3,120,000 33,863

(= 33.9 PF) 54,902 17,808

3 Swiss Natl. Supercomputer Center, Switzerland

Piz DaintCray XC30/NVIDIA P100, 2013 Cray 361,760 19,590 33,863 2,272

4 JAMSTEC GyoukouZettaScaler-2.2, PEZY-SC2 19,860,000 19,135.8 28,192 1,350

5 Oak Ridge National Laboratory, USA

TitanCray XK7/NVIDIA K20x, 2012 Cray 560,640 17,590 27,113 8,209

6 Lawrence Livermore National Laboratory, USA

SequoiaBlueGene/Q, 2011 IBM 1,572,864 17,173 20,133 7,890

7 DOE/NNSA/LANL/SNL, USA Trinity, Cray XC40, Intel Xeon Phi 7250 68C 1.4GHz, 2017 Cray 979,968 14,137 43,902 3,844

8 DOE/SC/LBNL/NERSCUSA

Cori, Cray XC40, Intel Xeon Phi 7250 68C 1.4GHz, Cray Aries, 2016 Cray

632,400 14,015 27,881 3,939

9Joint Center for Advanced High Performance Computing, Japan

Oakforest-PACS, PRIMERGY CX600 M1, Intel Xeon Phi Processor 7250 68C 1.4GHz, Intel Omni-Path, 2016 Fujitsu

557,056 13,555 24,914 2,719

10 RIKEN AICS, Japan K computer, SPARC64 VIIIfx , 2011 Fujitsu 705,024 10,510 11,280 12,660

50th TOP500 List (Nov., 2017)

Rmax: Performance of Linpack (TFLOPS)Rpeak: Peak Performance (TFLOPS), Power: kW

2018/2/14 ����KNL��

Page 42: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

42

http://www.hpcg-benchmark.org/

HPCG Ranking (Nov, 2017)

Computer Cores HPL Rmax(Pflop/s)

TOP500 Rank

HPCG (Pflop/s) Peak

1 K computer 705,024 10.510 10 0.6027 5.3%2 Tianhe-2 (MilkyWay-2) 3,120,000 33.863 2 0.5801 1.1%3 Trinity 979,072 14.137 7 0.546 1.8%4 Piz Daint 361,760 19.590 3 0.486 1.9%

5 Sunway TaihuLight 10,649,600 93.015 1 0.4808 0.4%

6 Oakforest-PACS 557,056 13.555 9 0.3855 1.5%7 Cori 632,400 13.832 8 0.3554 1.3%8 Sequoia 1,572,864 17.173 6 0.3304 1.6%9 Titan 560,640 17.590 5 0.3223 1.2%

10 TSUBAME3.0 136,080 8.125 13 0.189 1.6%

2018/2/14 ����KNL��

Page 43: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

Green 500 Ranking (SC16, November, 2016)

Site Computer CPU

HPL

Rmax

(Pflop/s)

TOP500

Rank

Power

(MW)GFLOPS/W

1NVIDIA Corporation

DGX SATURNV

NVIDIA DGX-1, Xeon E5-2698v4 20C 2.2GHz, Infiniband EDR, NVIDIA Tesla P100

3.307 28 0.350 9.462

2Swiss National Supercomputing Centre (CSCS)

Piz DaintCray XC50, Xeon E5-2690v3 12C 2.6GHz, Aries interconnect , NVIDIA Tesla P100

9.779 8 1.312 7.454

3 RIKEN ACCS Shoubu ZettaScaler-1.6 etc. 1.001 116 0.150 6.674

4National SC Center in Wuxi

Sunway TaihuLight

Sunway MPP, Sunway SW26010 260C 1.45GHz, Sunway

93.01 1 15.37 6.051

5SFB/TR55 at Fujitsu Tech.Solutions GmbH

QPACE3PRIMERGY CX1640 M1, Intel Xeon Phi 7210 64C 1.3GHz, Intel Omni-Path

0.447 375 0.077 5.806

6 JCAHPCOakforest-PACS

PRIMERGY CX1640 M1, Intel Xeon Phi 7250 68C 1.4GHz, Intel Omni-Path

1.355 6 2.719 4.986

7DOE/SC/ArgonneNational Lab.

ThetaCray XC40, Intel Xeon Phi 7230 64C 1.3GHz, Aries interconnect

5.096 18 1.087 4.688

8Stanford Research Computing Center

XStreamCray CS-Storm, Intel Xeon E5-2680v2 10C 2.8GHz, Infiniband FDR, NvidiaK80

0.781 162 0.190 4.112

9ACCMS, Kyoto University

Camphor 2Cray XC40, Intel Xeon Phi 7250 68C 1.4GHz, Aries interconnect

3.057 33 0.748 4.087

10Jefferson Natl. Accel. Facility

SciPhi XVIKOI Cluster, Intel Xeon Phi 7230 64C 1.3GHz, Intel Omni-Path

0.426 397 0.111 3.837

http://www.top500.org/

2018/2/14 ����KNL�� 43

Page 44: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

44

IO 500 Ranking (Nov., 2017)

Site Computer File system Client nodes IO500 Score BW(GiB/s)

MD�kIOP/s)

1 JCAHPC, Japan

Oakforest-PACS DDN IME 204 101.48 471.25 21.85

2 KAUST, Saudi Shaheen2 Cray DataWarp 300 70.90 151.53 33.17

3 KAUST, Saudi Shaheen2 Lustre 1000 41.00 54.17 31.03

4 JSC, Germany JURON BeeGFS 8 35.77 14.24 89.83

5 DKRZ, Germany Mistral Lustre 100 32.15 22.77 45.39

6 IBM, USA Sonasad Spectrum Scale 10 21.63 4.57 102.38

7 Fraunhofer, Germany Seislab BeeGFS 24 18.75 5.13 68.58

8 PNNL, USAEMSL Cascade

Lustre 126 11.17 4.88 25.57

9 SNL, USA Serrano Spectrum Scale 16 4.25 0.65 27.98

http://www.io500.org/

2018/2/14 ����KNL��

Page 45: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���� �BQDROakforest-PACSARJR>QKNRD@AEM6�*%S2018�4�1�T

• JRCHP>RA• 100,000� U 18IRG( �) 382,��2048IRG82FR<QU2IRG x 24�+ x 360��S1T

• =PRL>RA• 400,000� (�� 480,000�) U 1 8IRGS �T,��2048IRG82FR<QU8IRG x 24�+ x 360�� S1T

• ��7,-FR<Q.2)�• �IRG�827,FR<Q�'��01.0• �IRG�;(/:3,(/1�7,�'��02.054:• ��!6OR?7Reedbush36��FR<Q $9�#

2018/4/26 &"�UMPI � 45

Page 46: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

�����"Q`SaReedbushPaYaM`Z]aSOPT\A�0*b2018�4�1�c

• YaRW_MaP• 150,000 d RB-U: 4XaVb� c3��128XaVC=

RB-H: 1XaVb� 3��BUA2.5�c3�� 32XaVC=RB-L: 1XaVb� 3��BUA4�c3�� 16XaVC=

• L_a[MaP• 300,000d RB-U 1� 4XaVb� c3��128XaVC=3

RB-H: 1XaVb� 3��BUA2.5�c3�� 32XaVC=RB-L: 1XaVb� 3��BUA4�c3�� 16XaVC=

• �� RB-UAD 360,000 d 1� 4XaVb� c3��128XaVC=• �� RB-HAD 216,000 d 1� 1XaVb� c3��32XaVC=• �� RB-LAD 360,000 d 1� 1XaVb� c3��16XaVC=

• ��B34UaK`�5=/�• !:.DXaV�2360�224�1A4UaK`58�7FIH• � XaVC=B3UaK`�,��81.0 (HB;A2.5�, LB4�)• � XaVJ-7H>3-7<B3�,��89F@2�@?H• ��&A^aNBOakforest-PACS>A#�UaK`%)E (• XaV��E6G (Reedbush-U, L)

2018/4/26 +'�dMPI�$ 46

Page 47: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

[fNMgejT��D@;A• ��D�UhWjEReedbush-U/H/L, Oakforest-PACSSTYc>�<L8(�[fNMgejT9=JG8"�[fNMgejT9��>:KH?7• MOZbXP�)

• _jV]gRjTk1q3i!lkRB-U: �16^j\, RB-H: �4^j\6RB-L: �4^j\6OFP: �16^j\l

• QgjaRjTk1q9i!l kRB-U: �128^j\6RB-H: �32^j\6RB-L: �16^j\6OFP: �2048^j\l

• �&�)• _jV]gRjTkmqni!lkRB-U: �16^j\, RB-H: �4^j\6RB-L: �4^j\6OFP: �16^j\l#2.�E2>�56�$(

• QgjaRjT• (�[fNMgejTpkmi!qni!lp(�kRB-U: �128^j\6RB-H: �32^j\6OFP: �2048^j\l

• "�[fNMgejTpkmi!q �4,oi!l6"�k1,3'F(�B�+l

• Tj_jRh`djW�)3%/�$� �E�$>�0k�2���l• �ERjTBID6-�C�)���E��>�0

2.�pMPI�* 472018/4/26

Page 48: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

-606*5146.+-/3&��•��&26,)����" �•����� •����•����•��&$#���%$('!�

https://www.cc.u-tokyo.ac.jp/guide/

���7MPI�� 482018/4/26

Page 49: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

�� ������

����MPI�� 492018/4/26

Page 50: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��U\MYW]MAF�4b• (��"EU\MYX_�"�,T `J/c E#��J�>?/T / c D;I8A0

• �� 3@F!�0• �-F/@6I4B24F/�%�E��_K[NZOX`@ �67 .:95*2• K[NZOX�/��D���@6C1+E��• )�E<GEL^TVQSE��• )��=�5H�,• R^P&'�,

$��aMPI�� 50

T

T / c

2018/4/26

Page 51: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��������• Michael J. Flynn��!����� �"���!#%$$"

•�������� ��!SISD, Single Instruction Single Data Stream"

•������ � ��!SIMD, Single Instruction Multiple Data Stream"

•� ������ ��!MISD, Multiple Instruction Single Data Stream"

•� ���� � ��!MIMD, Multiple Instruction Multiple Data Stream"

���&MPI� 512018/4/26

Page 52: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���� �&'(������

A) &'(!%)# ������-���&'(�!"$#��

1. ��&'(�+SMP: Symmetric Multiprocessor,UMA: Uniform Memory Access,

2. ����&'(�+DSM:Distributed Shared Memory,

��*��&'(�+ccNUMA�Cache Coherent Non-UniformMemory Access,

���-MPI� 522018/4/26

Page 53: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

������$%&������

B) $%&�"'����,���$%&��� ���

3. �$%&�*$! )�#!�(�+

���,MPI 532018/4/26

Page 54: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

7@-<9A-���)�"��

1. 8>2/?35• Pthreads, …

2. 4B1��• OpenMP• C��&)Fortran• PGAS (Partitioned Global Address Space)��: XcalableMP, UPC,

Chapel, X10, Co-array Fortran, …

3. 1/,��• Cilk (Cilk plus), Thread Building Block (TBB), StackThreads,

MassiveThreads, …

4. :30B.��• MPI

���EMPI�� 54

��6B5%("�*��%��*�':;=+��!#�* $C��:;=D�

2018/4/26

Page 55: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��&-"*'."�)%,

•�����&-"*(���MIMD•!,#+$( ������0SIMD���1•�������������

� �/MPI�� 552018/4/26

Page 56: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��.5&3/6&#1,4•��#MIMD�!#��.5&3/6&#1,4

1. SPMD8Single Program Multiple Data9• : #��#.5&30���������"��.5*+'�!���$

• MPI8-7(26:9#1,4

2. Master / Worker8Master / Slave9• : #.5*)8Master9����#.5*)8Worker9%��8� ��9�$�

���;MPI� 562018/4/26

Page 57: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

����������

� ��MPI�� 572018/4/26

Page 58: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

�#%���Y���

• ���• �Z• Z'�<�$�)- ZP8<�$�)• P�07 <92-�� :WidealX(���• P�07 <92-FVNTMBUFPVLBIQ

• �: =-���;>?-JVHBDGF1���4A7-CRIESOIK�1��5@39;>@,(�

• �����• �Z

• +��#• (���<*�• Saturation-.46@/

&"�ZMPI�! 58

)0(/ pPSP STTS £=

ST PTPSP =PSP >

[%])0(100/ pPP EPSE £´=

[

2018/4/26

Page 59: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

Weak ScalingDStrong Scaling����F20BVWZ^&�R�56<O��

• Weak Scaling: >P?PG�-USXH�1=���R/8OØ��G�-USX4`���F��:Ba�56EO

Ø)�GT_\]Y[H/IN�QME03.KK�<O

• Strong Scaling: ��G�-USXR�1=F���R/8OØ�-USX4$"�F��:B�96EO

Ø)�GT_\]Y[H���F�56EO

(#�bMPI 59

Strong ScalingJ*%�;�-&�C.��+

F!�R�O`LN,:0a

Weak Scaling>PIC'7E3A@

&�G�-4'7O

2018/4/26

Page 60: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

MSPVTE��

• '��"�*L K B:K.;E2=-��5A6K��L α B:K.

• 8EB6-�� �F��EI2DCK.

• �%E�4J-<B3�+�E�ERUOQNL�>@HXP→∞Y-�� �F-,/ \[X\ZαY A0K.XMSPVTE��Y• �E90%5��A6<B9@H-�+�E�ERUOQNL?4>@H- \/(\-0.9) = \0 � D94CJC1W

→,�!L(�:K<GDF-�9AH�� �L�7K�#L:K8B5B@H)$A0K

& �]MPI�� 60

)1)1/1(/(1))1(/(/1))1(//(

+-=-+=-+=

PPKPKKSP

aaaaa

2018/4/26

Page 61: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

�$!'%� �����

���-MPI� 61

��������������(*#&" ) ��������(,#&" )

�����(+��)

�����(,��)

9/3=3�

9/2=4.5� ≠ 6�

.88.8%�����

2018/4/26

Page 62: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

Byte/Flop, B/F�1. MWATO�2��-;PQU>@ED6 �

double A[N][N];…A[i][j] = 0.25 * (A[i][j] + A[i][j-1] + A[i][j+1] + A[i-1][j] + A[i+1][j]);

• PQU>@EDY8byte * 5�6WXK + 8byte * 1�6DJ> = 48byte• �� : ��4�#��1� = 5 FLOP B/F = 9.6

2. PQUCDHO(IXF=��B>5�-;��#�2). �

• "�7 0.1��#9*180.5��• B/F=0.56CDHO2�6 �=-;3#96��6 �(2);3+<=#5�,'�'4$ => LX@65%,'&4$

• B/F�6�!=?RGCS5901�%• N@JV�: ���57B/F = 1*:$/0.#�7 0.53'

����MPI�� 62

������������� �����

2018/4/26

Page 63: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���������MPI

2018/4/26 ����MPI�� 63

Page 64: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

MPIK%�• odbw_hd^vX'KrUkrs2�K{F

• odbw_hd^vXKpftG=P• ZvhUrK2�<%�KcjgVWSNrUkrsQ�EMKGLI>x

• ��ops���3,#G���1J�A• �2"3,@/

• {lubd\J?BPops\UaNjRUt\UaK�-Q�)/• lubd\�K�>��^`enyMassively Parallel Processing (MPP)^`enzQ'>P�1J�A• {lubd\�,G0�I�1�9K3,Q<(�9G�&/

• + @��• APIyApplication Programming InterfacezK!$�

• `YwriseT<�/@;>• 7��&Qqw]@46EPCHJOPSt[sanK�8�@/• luXrmvX@:D>y��@;>z

5.�|MPI * 642018/4/26

Page 65: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

MPI-� A(20)B• MPI:6@=<Ahttp://www.mpi-forum.org/B'����

• 1994 5� 1.0�AMPI-1B• 1995 6� 1.1�• 1997 7� 1.2�% &1/ 2.0�AMPI-2B• 2008 5� 1.3�% 2008 6� 2.1�• 2009 9� 2.2�

• ��"� http://www.pccluster.org/ja/mpi.html

• MPI-2 ).%��3�C• ��I/O• C++%Fortran 90�4?9@:5@7• �;>87��/��

• �,%������+*-�$

#!�CMPI�� 652018/4/26

Page 66: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

MPI&��MPI-3.1• MPI-3.0 2012�9�• MPI-3.1 2015�6�• ��&17+#��6.)325-(���

• http://mpi-forum.org/docs/docs.html• http://meetings.mpi-forum.org• http://meetings.mpi-forum.org/mpi31-impl-status-Nov15.pdf

• ��"'!��• /5604,)5*������8MPI_IALLREDUCE %$9

• � �%�����8RMA Remote Memory Access)

• Fortran2008 � %$

���:MPI� 662018/4/26

Page 67: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

MPI4��MPI-4.0���•��4R]C1��\J>UT[I8�$

• http://meetings.mpi-forum.org/MPI_4.0_main_page.php

•��,70)6��•K;OXGJPZ?WS[?54� • MPI:PX@]BV[4��&�^Fault Tolerance, FT_

•)+/*4:;H:8���• Active Messages (TGE]C#�4PZIAY)

• ��2#�4=]LWGP• ��%4�8�).'�#�• �)=]L]QGJ(M;PW;[!"• LGN9X[?3-1(;[FWPIK[JW1�+

• Stream Messaging• �PZN9;Y\;[F]N<]D

��`MPI� 672018/4/26

Page 68: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

MPI8�"• MPICHT?MRJGFU

• ��=OBPH�����,%�• MVAPICH (?MQ<JGF)

• ��@I>@���3%�&MPICH;LSC• InfiniBand�-8�:0�"

• OpenMPI• @SKPDSC

•LPEMPI• �&�85:+,LSC7612*9�: $'�(&FX10�8MPI: Open-MPILSC

Intel MPI: MPICH&MVAPICHLSC• ���:NSA�!� ��,6/:2*9.4,)9

#��VMPI�� 682018/4/26

Page 69: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

MPI5:<%�• &��6&$5�/• &$5� 4��H1. ��6��)$;�6��2. �5�01+<9673.5*<-3. �5�01+<966�(4. �5�01+<966'5. F��=����5$<�6G!#��FA?G

• MPI27H1. ��6!#ID),:8)$;�6!#ID2. BEA���6>CD@3. BEA�4. BEA'5. A?�

"��HMPI � 692018/4/26

Page 70: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

MPI��

• 0136��• �����&"*; �����'%%�(�&#; �����'%%�)"-!; �����"&�$"-!;

• 9�9����• 572.8/�• �����!& ; �����!�,;

• 48572.8/�• �����)!& ; �����(!�,;

• 9������• �������)*

• ������• �����! +�!; �����$$(! +�!; ������(("!(;

• � ��• �����*"%!

���:MPI�� 702018/4/26

Page 71: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

CORLBUH

• MPI_COMM_WORLD7(CORLBUH4;8?>��@��/>��

• CORLBUH7(��@ *�"6MTGJD�@�:>

• ����37(V�Ynumprocs –W�936MTGJD+(W16CORLBUH5=�2<?>• -6��+(&MPI_COMM_WORLD'

• MTGJD�@�.0) (MPI_Comm_split%�@��• PJGUE@(�$6MTGJD�5�#/>4,5��

• “NSIAQFK”3��

!��XMPI�� 712018/4/26

Page 72: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

�#=MPI�#• MPI@,RVPO-(?&�H!/A7+RVPO@X�&@Y,RVPQN-XB63@*MJY>���<�D�;CFA7+

• ��*,MPIRVPO-=�3?@'/?<*44<@PEXProcesser Elements?�Y=�2A7+• 9:6�#=6;,PE-@��@.AD�GF;/A8I+

•TWKXRankY• �,MPIRVPO-?,%�� -?4=+• &�MPI<@*MPI_Comm_rank)�<"�5FE��XNWRURVLTS<@myidY>*[^PE�Z\ ?��1E

• �?�?MPIRVPO�H�E>@*MPI_Comm_size)�H�0+

XNWRURVLTS<@*numprocs >*4?��1EY

$ �]MPI�� 722018/4/26

Page 73: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

����MPI���� �������������

��MPI�� 732018/4/26

Page 74: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

C��!)%*'"*$�Fortran!)%*'"*$���•,��� ����ierr � �

ierr = MPI_Xxxx(….);• Fortran���������ierr���

call MPI_XXXX(…., ierr)•#$&(��������

•,��MPI_Status istatus;

• Fortran��integer istatus(MPI_STATUS_SIZE)

���+MPI�� 742018/4/26

Page 75: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

C��%+(,*&,' Fortran%+(,*&,'"��• MPI!��$�),(�"�

• C��MPI_CHAR ( ��) � MPI_INT (���)�MPI_FLOAT (���)�MPI_DOUBLE(�����)• Fortran��MPI_CHARACTER ( ��) �MPI_INTEGER (���)�MPI_REAL (���)�MPI_REAL8 (=MPI_DOUBLE_PRECISION)(�����) �MPI_COMPLEX(����)

• ��#�.��%+(*&,'����$

���-MPI�� 752018/4/26

Page 76: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��)MPI��―MPI_Recv 91/2:• ierr = MPI_Recv(recvbuf, icount, idatatype, isource,

itag, icomm, istatus);

• recvbuf : ���*����.��&-!

• icount : ���!���*482���.��&-!

• idatatype : ���!���*482*�.��&-!

• MPI_CHAR (���) MPI_INT (���) MPI_FLOAT ( ��) MPI_DOUBLE(��� ��)

• isource : ���!��%'"53180.��&-PE*67/.��&-!

• ��*PE#,��%'"($+ MPI_ANY_SOURCE .��&-!

���;MPI� 762018/4/26

Page 77: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

�!�?MPI-�―MPI_Recv [2/2\• itag : ���0��8;2VRPZO@�2<2GQLA�I��0

• ��AQL�AVRPZOI��8;2>5B/MPI_ANY_TAG I��0• icomm : ���0PE. I&)9G��=1GNUWTMZQ

I��0

• +�=BMPI_COMM_WORLD I��9HCD20• istatus : MPI_Status�[���A,\0����@-9G��4G03?E:��A��%I8;,I �9G6>0

• $"�4MPI_STATUS_SIZEA��,4�%7HG0• ��8;VRPZOA*��AXYK4 istatus[MPI_SOURCE]/QL4 istatus[MPI_TAG] @�7HG0

• C%'] MPI_Status istatus;• Fortran%'] integer istatus(MPI_STATUS_SIZE)

• ierr(�F�) : ���0JXZNZS4G0

(#�]MPI�! 772018/4/26

Page 78: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���+MPI��―MPI_Send• ierr = MPI_Send(sendbuf, icount, idatatype, idest,

itag, icomm);

• sendbuf : �� -�!��2��• icount : ���"�� -=G;���2��• idatatype : ���"�� -=G;-�2��• idest : ���"��')$PE-icomm�*-DF42��• itag : ���"��')$B<:G9,�&.1);5-�2��• icomm : ���"@E:<8G�2��(0�*#0

7AC?6G;2��

• ierr (�/�) : ���"3DG7G>%�0"

���HMPI�� 782018/4/26

Page 79: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

Send�Recv�����������"MPI� 79

PE0 PE� PE PE!

MPI_Send

MPI_Recv

2018/4/26

��������������

��

Page 80: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���-MPI��―MPI_Bcast• ierr = MPI_Bcast(sendbuf, icount, idatatype,

iroot, icomm);

• sendbuf : ��&0/��! .�"��3��*2#• icount : ���#��! .;B9���3��*2#• idatatype : ���#��! .;B9.�3��*2#• iroot : ���#��(+%?:8B7'$2PE.��3

��*2#�PE,)�3��*2��'$2#• icomm : ���#PE 3��*2��,$2

6>@=5B93��*2#

• ierr (�1�) : ���#4AB6B<'�2#

���CMPI�� 802018/4/26

Page 81: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

MPI_Bcast��������

�� MPI�� 81

PE0 PE� PE� PE�

irootMPI_Bcast() MPI_Bcast() MPI_Bcast() MPI_Bcast()

�PE������������!!

2018/4/26

Page 82: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

E?;<CG��

•N��O04,-N��O7� IE?;<CGJ(*5��•�M ����B;@FIP���#J → =:DIK���#J

•E?;<CG��2%!�/��7��/)5•$!���Icollective communication operationJ/�365

•����1�+�1"&.%L�18G>A9H='�)5

��MMPI�� 822018/4/26

Page 83: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

$" !#%��

•���������PE���• MPI_Reduce�• $" !#%������������PE�����

• MPI_Allreduce�• $" !#%����������PE�����

���&MPI� 83

PE0PE0

PE1 PE2���� PE0

PE0

PE1 PE2��

PE0

PE1 PE2

2018/4/26

Page 84: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

�!�@MPI-�―MPI_Reduce• ierr = MPI_Reduce(sendbuf, recvbuf, icount,

idatatype, iop, iroot, icomm);• sendbuf : +�.�A�/��H��9E1• recvbuf : ��.�A�/��H��9E1iroot >��8:PEAC>�5)C4@7FE1+�.�?��.�B0 �>2<=B@D@31

9@G;0�@E,H �8@6=B@D@31

• icount : ���1+�.�AJKI'$�H��9E1• idatatype : ���1+�.�AJKIA�H��9E1

• LFortranMP��N���?�%QH*9�"H��9E��B0MPI_2INTEGER(���)0 MPI_2REAL(#��)0 MPI_2REAL8 (=MPI_2DOUBLE_PRECISION) (�#��) 0H��9E1

(&�OMPI�! 842018/4/26

Page 85: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���6MPI'�―MPI_Reduce• iop : �� +��7�)9��08+

• MPI_SUM ( �)* MPI_PROD (�)* MPI_MAX (��)*MPI_MIN (��)* MPI_MAXLOC (��417�!)*MPI_MINLOC (��417�!) 65+

• iroot : �� +��9�.�8PE7icomm�37CD;9��08+�27icomm�7PE3/�9��08�#-,8+

• icomm : �� +PE(�9$&08�3,8=AB@<E>9��08+

• ierr : �� + :CE=E?-�8+

%"�FMPI�� 852018/4/26

Page 86: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

MPI_Reduce��������� ���MPI� 86

PE0 PE� PE� PE�

iroot�������� ���� ����

iop��������

MPI_Reduce() MPI_Reduce() MPI_Reduce() MPI_Reduce()

2018/4/26

Page 87: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

MPI_Reduce�����������MPI_2DOUBLE_PRECISION �MPI_MAXLOC�

���MPI� 87

PE0 PE� PE� PE�

iroot3.1

MPI_MAXLOC

2.04.15.0

5.99.0

2.613.0

5.99.0

LU���� ����

MPI_Reduce() MPI_Reduce() MPI_Reduce() MPI_Reduce()

2018/4/26

Page 88: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

� �:MPI+�―MPI_Allreduce• ierr = MPI_Allreduce(sendbuf, recvbuf, icount,

idatatype, iop, icomm);• sendbuf : )�,�;�-��A��4?/• recvbuf : ��,�;�-��A��4?/)�,�9��,�<. �8067<:>:1/4:@5.�:?*A��3:27<:>:1/

• icount : ���/)�,�;CDB&#�A��4?/• idatatype : ���/)�,�;CDB;�A��4?/

• ���=���9�$A(4�!A��4?��<.MPI_2INT(���).MPI_2FLOAT ("��).MPI_2DOUBLE(�"��) A��4?/

'%�EMPI� 882018/4/26

Page 89: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���(MPI� ―MPI_Allreduce• iop : � �!��)��+�$*!

• MPI_SUM (��) MPI_PROD (�) MPI_MAX (�) MPI_MIN (��) MPI_MAXLOC (�&��) MPI_MINLOC(��&��) ('!

• icomm : � �!PE��+��$*��%"*.231-5/+�$*!

• ierr : � �! ,45.50#�*!

���6MPI�� 892018/4/26

Page 90: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

MPI_Allreduce��������

���!MPI�� 90

PE0 PE� PE� PE

�������� ���� ���

iop������� �

� �������

MPI_Allreduce() MPI_Allreduce() MPI_Allreduce() MPI_Allreduce()

2018/4/26

Page 91: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

1,)+/3 �

•�$ �!•1,)+/3 �%�55��$�&��

•-2*0.�"���&�"#�4• MPI_Allreduce % MPI_Reduce $�&��

• MPI_Allreduce %�������'�•#'&��MPI_Reduce (���

���6MPI�� 912018/4/26

Page 92: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��$��

• �� �3Block,54��)��(!�(�• �� $���� *�(#%�MPI %�$6�'$��* �(• MPI_Gather��

• MPI_Scatter��

���7MPI� 92

AA TA

abc

a b c

a b c abc

�&(10/2-,+.��PE ��$!���

�&(,+.��PE �� "�!�%7MPI_GatherV��MPI_ScatterV��

2018/4/26

Page 93: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���;MPI$�―MPI_Gather• ierr = MPI_Gather (sendbuf, isendcount, isendtype,

recvbuf, irecvcount, irecvtype, iroot, icomm);• sendbuf : "�&�<�'��D��4A)• isendcount: ���)"�&�<FGE���D��4A)• isendtype : ���)"�&�<FGE<�D��4A)• recvbuf : ��&�<�'��D��4A)iroot 9��25PE<>9�.!>-;1BA)

• ;,�:28("�&�:��&�=( �9*78=;?;+)4;C6(�;A#�D��2;/8=;?;+)

• irecvcount: ���)��&�<FGE���D��4A)• 0<���=(HPE�5@<"�FGE�D��4A0:)• MPI_Gather $�9=�PE9�;A�<FGED%4A0:=9.;+<9( 3�D��4A0:)

��IMPI�� 932018/4/26

Page 94: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���(MPI��―MPI_Gather• irecvtype : �� ���)180 +��

%*

• iroot : �� ��180+#�*PE)icomm �')67-+��%*

•�&)icomm �)PE'�$�+��%*��"!*

• icomm : �� PE��+��%*�'!*/453.80+��%*

• ierr : �� ,68/82"�*

���9MPI�� 942018/4/26

Page 95: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

MPI_Gather������ ������MPI� 95

PE0 PE� PE� PE�

iroot���B���A ���C ���D

���

���A���B���C���D

MPI_Gather() MPI_Gather() MPI_Gather() MPI_Gather()

2018/4/26

Page 96: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���9MPI%�―MPI_Scatter• ierr = MPI_Scatter ( sendbuf, isendcount, isendtype,

recvbuf, irecvcount, irecvtype, iroot, icomm);• sendbuf : #�&�;�'��C��2@)• isendcount: ���)#�&�;EFD!��C��2@)

• /;!��<(GPE�3?:#>A@#�EFD�C��2@/8)• MPI_Scatter %�7< PE7�9@�;EFDC��2@/8<7-9+;7(�1�C��2@/8 )

• isendtype : ���)#�&�;EFD;�C��2@)iroot 7��03PE;=�89@)

• recvbuf : ��&�;�'��C��2@)• 9,�806(#�&�8��&�<(��7*56<9>9+)29B4(�9@$�C��09.6<9>9+)

• irecvcount: ���)��&�;EFD!��C��2@)

" �HMPI�� 962018/4/26

Page 97: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���(MPI��―MPI_Scatter• irecvtype : �� ���)180 +��

%*

• iroot : �� ��180+#�*PE)icomm �')67-+��%*

•�&)icomm �)PE'�$�+��%*��"!*

• icomm : �� PE��+��%*�'!*/453.80+��%*

• ierr : �� ,68/82"�*

���9MPI�� 972018/4/26

Page 98: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

MPI_Scatter������ ������MPI� 98

PE0 PE� PE� PE�

iroot

���

���A���B���C���D

���C ���D���B���A

MPI_Scatter() MPI_Scatter() MPI_Scatter() MPI_Scatter()

2018/4/26

Page 99: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

MPI_Scatter������ ��

����MPI� 99

PE0 PE� PE� PE�

iroot

���

���A���B���C���D

���C ���D���B���A

MPI_Scatter() MPI_Scatter() MPI_Scatter() MPI_Scatter()

2018/4/26

Page 100: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

BE=5F7�?FBE=5F7

1. BE=5F7• ��I��+@=A3��*C=;H9���"1��I���+@=A3�����*46;:G�� ' 0.'�-�#��/)�

• @=A3���+>H<+�� 2��

• MPI_Send, MPI_Bcast)(

2. ?FBE=5F7•��I��+@=A3��+>H<2��&%$!*-�#��0

• @=A3���+>H<+�� 2��&%• �� +��,DH8+��

2018/4/26 ���JMPI�� 100

Page 101: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

462-7.��(��*�"�

• 560/0#��)381,�&'"+�

2018/4/26 ���;MPI� 101

�� send ��

560/0

560/1 �� recv

���

�*� (*���

560/9 �� recv

560/: �� recv

send ��� send ��� …

��

��

��

�*� (*���%

�*� (*���%

�*� (*���%

��$+send(!��*�"���%� #��

Page 102: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

0514.(5*����

• ierr = MPI_Isend(sendbuf, icount, datatype, idest, itag, icomm, irequest);

• sendbuf : ���#����' ��%• icount : ������#/6-���' ��%• datatype : ������#/6-#' ��%• idest : ������ �PE#icomm�!#35)' ��%

• itag : ������ �2.,6+"��$& -*#�' ��%

2018/4/26 ���7MPI�� 102

Page 103: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

<C=B92C3����

• icomm : ����PE��0��$.��'�.5>@;4D80��$.�

•��'*MPI_COMM_WORLD 0��$/+, �

• irequest : MPI_Request�E���)��F���0��#%?97D6(&"-/%��! .�

• ierr : ����1AD5D:!�.�

���GMPI� 1032018/4/26

Page 104: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���4)�

• ierr = MPI_Wait(irequest, istatus);

• irequest : MPI_Request M�� (�N+'�@"�03HFDLC85-=?3&��+

• istatus : MPI_Status M�� (�N+���8)1>��,�>+• " �,MPI_STATUS_SIZE9��(�@�#06��1>+

• �03HFDLC9'��9IJA,istatus[MPI_SOURCE] *EB,istatus[MPI_TAG] 8��/?>+

• '�GLE@��1>K�GLE@$<�18:�2�;.7

2018/4/26 %!�OMPI�� 104

Page 105: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

;?@<>94?5 �/12��

• =>760'��.:A83�+,$2�

2018/4/26 ���DMPI� 105

�� send ��

=>760

=>761 �� recv

�0��-0���

=>76B �� recv

=>76C �� recv

send ���send …

��

��

��

�0��-0���*

�0��-0���*

�0��-0���*

!�)2send/&(2���*�"3;?@<>94?5 �-��

���*3#MPI_Wait-��0 /�%1%/��

Page 106: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��

•��"#�!��������-•MPI_Send�•��!MPI_Wait������$.•MPI_Isend�•��!MPI_Wait������ �.

•�����!),&'+%*(�$.

2018/4/26 � �-MPI�� 106

Page 107: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

�"��

1. MPI��5;,96<,)P.31+- $@�#�&2. ��5;,96<,%*�MPI�)( ��$)���� ����/<0> http://accc.riken.jp/HPC/training/text.html ?

3. Message Passing Interface Forum> http://www.mpi-forum.org/ ?

4. MPI-J7=:<,:.2> http://phase.hpcc.jp/phase/mpi-j/ml/ ?

5. ��-<48=0��)����$)��>ACCB?

'!�DMPI�� 1072018/4/26

Page 108: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���$•.���BG67TcR�18<2*•���E;9AG67TcR���8<2*EDHd1. PEF7�$-(8I�#E?H

• `cUbV\aOaMi ���F����F�@

• %�+�

2. PEF7� Z[]38I�#E?H3. �$E�:/��4I!&?H4. PEF7TcRbJLQPWRca8I50D��E?Hek.���E;=HTcR�1C�>f

•)TcRF���• j��_Y^li g�����6h�����• j�_Y^li X`SL���6NKL]SLe��f���

,'�iMPI�" 1082018/4/26

Page 109: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��(���• -.,�• -.,+��#*!%$��#*�• -.,(��/6��0) �%&*�• -.,�(�5�1��

���5MPI�� 109

÷÷÷

ø

ö

ççç

è

æ

987654321

÷÷÷

ø

ö

ççç

è

æ

123456789

÷÷÷

ø

ö

ççç

è

æ

++++++++++++++++++

1*94*87*72*95*88*73*96*89*71*64*57*42*65*58*43*66*59*41*34*27*12*35*28*13*36*29*1

6

÷÷÷

ø

ö

ççç

è

æ

987654321

÷÷÷

ø

ö

ççç

è

æ

123456789

÷÷÷

ø

ö

ççç

è

æ

++++++++++++++++++

1*94*87*72*95*88*73*96*89*71*64*57*42*65*58*43*66*59*41*34*27*12*35*28*13*36*29*1

6

���

CPU2

CPU3

CPU4

�CPU$��

�'��5�-.,)�&* ��) �

SIMD(���% "

2018/4/26

Page 110: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

16�6� ���• BA?� • BA?J@EDK<��/:,43� �/:)• CIB6��JR��K7�5:+9.;5*)• BA?� 6�Q>GI<�:• ��LQ$�<�:• ��MQ�<�:• ��NQ�<�'-0:• ��OQ$�H�<;2�"8• ��PQ>GIF=<;:

!��QMPI�� 110

��L

��M

��N

��O ��P

(� �

�%

2018/4/26

��&< #

Page 111: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

" ���

���$MPI�� 111

PE=0PE=1PE=2PE=3

•!���) �������•(Block, *) ���

•(���) ���������•(Cyclic, *) ���

•(���)���� ���������•(Cyclic(2), *) ���

N/4�

N/4�N/4�N/4�

N�1�

2�

�����#�$ %����&���

2018/4/26

Page 112: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���MPI�� 112

2���

0 0 1 1 0 0 1 1

0 0 1 1 0 0 1 1

2 2 3 3 2 2 3 3

2 2 3 3 2 2 3 3

0 0 1 1 0 0 1 1

0 0 1 1 0 0 1 1

2 2 3 3 2 2 3 3

2 2 3 3 2 2 3 3

PE=0 PE=1

PE=2

•������������•(Block, Block)���

•����������������•(Cyclic, Cyclic)���

•����������������•(Cyclic(2), Cyclic(2))���

PE=3

0 1 0 1 0 1 0 1

2 3 2 3 2 3 2 3

0 1 0 1 0 1 0 1

2 3 2 3 2 3 2 3

0 1 0 1 0 1 0 1

2 3 2 3 2 3 2 3

0 1 0 1 0 1 0 1

2 3 2 3 2 3 2 3

N/2

N/2

N/2 N/2

2018/4/26

Page 113: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

B=AG2)-5��• ��5��

• ,,1%α6><E%P%N%O 6B=AG• 259)3@H?����18���*�

• /0-%><E α 6�PE1��.;&• B=AG6O(M)5CDF$�*��354�-%><E6O(J)5CDF$�1 ��&→><ECDF$�6���

• �"KO(N/P)• '7:#�+3(

!��KMPI�� 113

yxaz +=

L I

P N Oα

2018/4/26

Page 114: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��#.*-/%�• 2��3#2��3��(�• 2,0+��3#2�3�%&�)"��'��!� �

���1MPI�� 114

.( . 0 . *.+(

( 0 *.+ ( ,*.+* + * +

2��31 ��$��C���

2��31 Fortran���

=… = …

( 0 * +(( 0

.( . 0 . *.+ ( , ) * +

�� ��� �

2018/4/26

Page 115: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��,7568.�

���<MPI�� 115

�PE�+��7568�4�$��75684 MPI_Allgather��4��(!�PE+��)3

PE=0PE=1PE=2PE=3

PE=0PE=1PE=2PE=3

=

�PE�+��-7568�4�$

=

MPI_Reduce��+��4�239�"3;PE-7568)/*% 13:

+ + +

=���. >=������> <���-'����

=������> <7568.��)/*%0(#,&-'

2018/4/26

Page 116: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��/><=?2� ��CMPI � 116

��;MPI_Reduce"�167��;�58

�!><=?; MPI_Allgather"�;��+-%�PE.��,8

PE=0PE=1PE=2PE=3

PE=0PE=1PE=2PE=3

=

PE�.��-><=?�;�(

=

MPI_Reduce"�.��;�58@�&8BPE1><=?,3-)#48A

+ + +

D���2�ED�������E C�$)�*�:90'

D�������E C���1�*����

= + + +

2018/4/26

Page 117: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

MPI���������������������� ��� � �

��� MPI�� 1172018/4/26

Page 118: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

����

���MPI�� 1182018/4/26

Page 119: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

/=6;6<.97&��

• Hello/• ���Hello6<.97• hello-pure.bash, hello-hy??.bash : 0851-:634+,;

• Cpi/• ����6<.97• cpi-pure.bash 0851-:634+,;

• Wa1/• ��� �%()���• wa1-pure.bash 0851-:634+,;

• Wa2/• ���!� �%()���• wa2-pure.bash 0851-:634+,;

• Cpi_m/• ����6<.97%�"��;>2=*��#$'&• cpi_m-pure.bash 0851-:634+,;

���?MPI�� 1192018/4/26

Page 120: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

( */%'Hello+1#.,1. ( */%'9;7� Makefile_hy68 ����

make ���$ make -f Makefile_hy68 clean$ make -f Makefile_hy68

2. ��)� 03hello_omp4��������������

$ ls4. 8:6$"/+&�3hello-hy68.bash4�!-2�������“lecture-flat” → “tutorial-flat”������

$ emacs hello-hy68.bash

� �5MPI�� 1202018/4/26

Page 121: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���HelloJQ?MK;��.7+SG=INDEMPIT1. HelloH>OC�2��;��/9

$ pjsub hello-hy68.bash2. ��4�-:0@LI;�#/9

$ pjstat3. ��,��/93(��4H<=O,��-:9

hello-hy68.bash.eVVVVVVhello-hy68.bash.oVVVVVV SXXXXXX5��T

4. �"�� H<=O4�%;!169$ cat hello-hy68.bash.oVVVVVV

5. &Hello parallel world!',(1JQBA*16FRE*68APDE=1088� �-:1*08��)

2018/4/26 $��UMPI�� 121

Page 122: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

#!/bin/bash

#PJM -L rscgrp=lecture-flat

#PJM -L node=16

#PJM --mpi proc=16

#PJM --omp thread=68

#PJM -L elapse=0:01:00

#PJM -g gt00

mpiexec.hydra –n

${PJM_MPI_PROC} ./hello_omp

JOB$ /-("3-0���5OpenMP+:;9+�,/')65hello-hy68.bash, C���Fortran����6

2018/4/26 ���8MPI�� 122

/&4$!04-�8lecture-flat

� !04-�8gt00

MPI#.,�1*16 = 16 -2%$�����

� *4)�MPI-2%$

*4)�� $1')

�����87�

Page 123: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

2018/4/26 ���AMPI�� 123

• Intel Xeon Phi (Knights Landing)• 10>/1+&-.����'#Knights Landing Overview

Chip: 36 Tiles interconnected by 2D Mesh Tile: 2 Cores + 2 VPU/core + 1 MB L2 Memory: MCDRAM: 16 GB on-package; High BW DDR4: 6 channels @ 2400 up to 384GB IO: 36 lanes PCIe Gen3. 4 lanes of DMI for chipset Node: 1-Socket only Fabric: Omni-Path on-package (not shown) Vector Peak Perf: 3+TF DP and 6+TF SP Flops Scalar Perf: ~3x over Knights Corner Streams Triad (GB/s): MCDRAM : 400+; DDR: 90+

TILE

4

2 VPU

Core

2 VPU

Core

1MB L2

CHA

Package

Source Intel: All products, computer systems, dates and figures specified are preliminary based on current expectations, and are subject to change without notice. KNL data are preliminary based on current expectations and are subject to change without notice. 1Binary Compatible with Intel Xeon processors using Haswell Instruction Set (except TSX). 2Bandwidth numbers are based on STREAM-like memory access pattern when MCDRAM used as flat memory. Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Omni-path not shown

EDC EDC PCIe Gen 3

EDC EDC

Tile

DDR MC DDR MC

EDC EDC misc EDC EDC

36 Tiles connected by

2D Mesh Interconnect

MCDRAM MCDRAM MCDRAM MCDRAM

3

DDR4

CHANNELS

3

DDR4

CHANNELS

MCDRAM MCDRAM MCDRAM MCDRAM

DMI

2 x161 x4

X4 DMI

HotChips27 KNL)8$/!"

Knights Landing: Next Intel® Xeon Phi™ Processor

First self-boot Intel® Xeon Phi™ processor that is binary compatible with main line IA. Boots standard OS.

Significant improvement in scalar and vector performance

Integration of Memory on package: innovative memory architecture for high bandwidth and high capacity

Integration of Fabric on package

Potential future options subject to change without notice. All timeframes, features, products and dates are preliminary forecasts and subject to change without further notification.

Three products

KNL Self-Boot KNL Self-Boot w/ Fabric KNL Card

(Baseline) (Fabric Integrated) (PCIe-Card)

Intel® Many-Core Processor targeted for HPC and Supercomputing

2 VPU 2 VPU

Core Core1MBL2

MCDRAM: 490GB/��� ?��@DDR4: 115.2 GB/�=(8Byte�2400MHz�6 channel)

CDB5<*)

�� ,$:(�)

MCDRAM: %=3-&>( �2=/67916GB+ DDR4679 ������������������

1$49-/CDB �� ?0>/�@

);-/

Page 124: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

#���LOGNMQ+�)*��R

• �LOIH3��6DKPJF-�LOIH:��5-/DLOIHS83"�F��6D��F%1D.

• �<��Q+�)*��R1. Q0�:<4E@R�,>LOIH2BKPJF�6DU2. �,>LOIH2BKPJ3�907BU

1. �6D;2. V&�>KPJW;V�KPJWF��6D;3. Q1023�:<4E@R,>LOIH=V2>��57"�FW*�6

D;4. ��F!�6D;

• �'�>��• �,C;?-Qmyid-SR>IDFA8LOIH• ,C;?-Qmyid+SR>IDFA8LOIH• myid=0>LOIH?-�,C?<0>:-�5<0• myid=p-S>LOIH?-,C?<0>:-*�5<0

($�TMPI � 1242018/4/26

Page 125: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

������ ��������!����

���(MPI�� 125

CPU0 CPU$ CPU% CPU&

# $ % &

#

����

# " $ ) $

$

$ " % ) &

&

& " & ) '

�� �� ��

���

���� ���� ����

2018/4/26

Page 126: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

=�=#���;"� !��%C��<

���>MPI�� 126

void main(int argc, char* argv[]) {MPI_Status istatus;

….dsendbuf = myid;drecvbuf = 0.0;if (myid != 0) {

ierr = MPI_Recv(&drecvbuf, 1, MPI_DOUBLE, myid-1, 0,MPI_COMM_WORLD, &istatus);

}dsendbuf = dsendbuf + drecvbuf;if (myid != nprocs-1) {ierr = MPI_Send(&dsendbuf, 1, MPI_DOUBLE, myid+1, 0,

MPI_COMM_WORLD);}if (myid == nprocs-1) printf ("Total = %4.2lf ¥n", dsendbuf);

….}

���4579$.��

��/1�*�,&ID��;myid-1<'0%double 8:6=*3��(drecvbuf��-��

��/1�*�&ID��;myid+1<-%dsendbuf��-�)+&2double 8:6=*3!�

2018/4/26

Page 127: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

=�=#���;"� !��%Fortran��<

���>MPI�� 127

program main….integer :: istatus(MPI_STATUS_SIZE)….dsendbuf = myiddrecvbuf = 0.0if (myid .ne. 0) then

call MPI_RECV(drecvbuf, 1, MPI_REAL8, &myid-1, 0, MPI_COMM_WORLD, istatus, ierr)

endifdsendbuf = dsendbuf + drecvbufif (myid .ne. numprocs-1) then

call MPI_SEND(dsendbuf, 1, MPI_REAL8, &myid+1, 0, MPI_COMM_WORLD, ierr)

endifif (myid .eq. numprocs-1) then

print *, "Total = ", dsendbufendif….stop

end program main

���4579$.��

��/1�*�,&ID��;myid-1<'0%double 8:6=*3��(drecvbuf��-��

��/1�*�&ID��;myid+1<-%dsendbuf��-�)+&2double 8:6=*3!�

2018/4/26

Page 128: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���),$+*.�� ����/

• �� ����1. k = 0;2. for (i=0; i < log2(nprocs); i++) 3. if ( (myid & k) == k)

• (myid – k)�),&%�!(-'#��2• �� (-'����(-'#���"2

• k = k * 2;4. else

• (myid + k)�),&%��(-'#���"2• ��#���"2

���1MPI� 1282018/4/26

Page 129: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

�����������������

����MPI� 129

0 1 2 3 4 5 6 71�

1 3 5 7��

3 7����og2����

0 1 2 3 4 5 6 7

1 3 5 7

3 7

7

2018/4/26

Page 130: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

'��&SVMUTW��1���X

• �+�F��• ,�[ SVON!�FZ2�*-F��L� =K• %i�E87A5��=KSVONF��G5��B�;K[

myid & k : k C�)• <<B5k = 2^(i-Y) 6• @HJ5SVON!�FZ2�*-B 9Ii!"FRPQ:$?A7KSVON:5/�=K<CE=K

• H>5/�FSVON!�G5��B�;K[myid + k

• @HJ 51�:�$=KSVON!�F34GZ\(i-Y)←��DFB• /�SVONE@7AG5�-F0:�J$@6

.(�[MPI�# 1302018/4/26

Page 131: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

, %+Y\T[Z^�� 8���_

• 7"46��J8���• �O=I;nprocs`b�

• �� 8���J8���• 2)MPJ�

• �#F0SRQ8�K;��I��F0SRQ^8�J1*K'&AH<_

• #�J�J8���GHQ• DLP;log2(nprocs) �

• �.J8���J$5• Y\WXU��>�BG;8���J�^g�0�9_>GEM�?@HQ

• baceY\WV!�FK;bacd� � ba�]

• FM;�CAM�� 8���>N<GK:OH<^8�1*>�'BQ/�_

3-�fMPI�( 1312018/4/26

Page 132: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

������=C��>

���?MPI� 132

double t0, t1, t2, t_w; ..ierr = MPI_Barrier(MPI_COMM_WORLD);t1 = MPI_Wtime();

@&&+��'("7;2980�$A

t2 = MPI_Wtime();

t0 = t2 - t1;ierr = MPI_Reduce(&t0, &t_w, 1,

MPI_DOUBLE,MPI_MAX, 0, MPI_COMM_WORLD);

6:1�� ��0��'��

�7;453<) B0,�-�*/!&,�- �.�".,,�07;4530�#�%�/

2018/4/26

Page 133: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

������=Fortran��>

���@MPI� 133

real(8) :: t0, t1, t2, t_w..call MPI_BARRIER(MPI_COMM_WORLD, ierr)t1 = MPI_WTIME()

A&&+��'("7;2980�$B

t2 = MPI_WTIME()

t0 = t2 - t1call MPI_REDUCE(t0, t_w, 1, MPI_REAL8, &

MPI_MAX, 0, MPI_COMM_WORLD, ierr)

6:1�� ��0��'��

�7;453<) C0,�-�*/!

&,�- �.�".,,�07;453?�#�%�/

2018/4/26

Page 134: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

MPI����' �(�"����• Oakforest-PACS�*#*�)$&*���!%�� MPI��������' �(�"������•�+mpiexec.hydra ./a.out < in.txt > out.txt

��,MPI� 1342018/4/26

Page 135: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��� ��

����MPI�� 1352018/4/26

Page 136: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��1:/"#7

Oakforest• -PACSIntel • VTune AmplifierPAPI (Performance API)•Oakforest• -PACS PA7#078

Web• 2<)9� � -&65;,��� ⇨Oakforest-PACS '(+4����7.1. ./$<3;(�*<9���Oakforest-PACS PA7#078��%#-!���������

2018/4/26 ���=MPI� 136

Page 137: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���&

1. #�!"��5AE;CB: �• Wa1 5AE;CB

2. ���$���5AE;CB: �• Wa25AE;CB

3. �%��AE;CB: �• Cpi_m5AE;CB

4. AE>=�:��-02)<FADAE;CB: �5. HelloAE;CB:)��57+4��

• MPI_Send:�*2)AE>=0,8IKJL5@G?'Hello World!!(:)15�5AE>=4"�/9

• 15�5AE>=36)MPI_Recv3�.2��/9

��HMPI�� 1372018/4/26

Page 138: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

MPI������������������������ ��� � �

2018/4/26 ��� MPI�� 138

Page 139: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

� ���

1. ��-����"&$�#2. ��-�����!��� ������3. �!��� ������4. ���"%#'�����

���'MPI� 1392018/4/26

Page 140: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

�-��5��5�<•���%>�?•��495>B@���2��?•!�#�)���

•���%>�?•/;03$,(>A�"��2��?•A�A!�#�)��•���%>�?)�*�=01,70-�6&:0181*.+('

��CMPI � 1402018/4/26

Page 141: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��-����������������������

� ��MPI� 1412018/4/26

Page 142: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

&�B!

•&�! C = A k BC2Rj]NgD*"�B`jWalPA�KIH9?64

•��or (��B-4>�%A�7@�6>H•��pr�:J@�0>3Hm_iQgb;�4n•��qr ��'*"B��6E8�=4H

1. /�A.4s,#MPUTt63H2. OeXSfA�G�F@4s�)�@ZlVtA�<H�">3H

3. cdh\j[�L15�"mcdhkNjYjS^n@��>3H

2018/4/26 +$�rMPI� 142

Page 143: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

����� ����

l�� �

for (i=0; i<n; i++) for (j=0; j<n; j++)

for (k=0; k<n; k++) C[i][j] += A[i][k] *B[k][j];

2018/4/26 ����MPI�� 143

C A Bi

j

i

k

k

j

Page 144: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��2�

• ���

2 ��3%�2� 8*�7:.'9S

1. LOH���• !�<?CB2�;�)9��/%��-���; �+9R#LOH2$�;��+9

2. GME?PD=KN@Q�• >IEAJ1&9FOD;���+9��/%&9404-,��2"�FOD;%��5<?CB+96(1 �+9

2018/4/26 ���SMPI�� 144

)...,,2,1,(1

njibacn

kkjikij ==å

=

Page 145: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��)�7<��8

• 564���

• ���)263* ��),#'9�564('/for(i=0; i<n; i++) {

for(j=0; j<n; j++) {for(k=0; k<n; k++) {

c[ i ][ j ] = c[ i ][ j ] + a[ i ][ k ] * b[ k][ j ];}

}}

•���)��* �)95641��%&+ ����$0-'"→ :�.)��) �$!/

2018/4/26 ���;MPI�� 145

Page 146: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��)�7Fortran��8• 564���

• ���)263* ��),#'9�564('/do i=1, n

do j=1, ndo k=1, n

c( i , j ) = c( i, j) + a( i , k ) * b( k , j )enddo

enddoenddo

•���)��* �)95641��%&+ ����$0-'"→ :�.)��) �$!/

2018/4/26 ���;MPI�� 146

Page 147: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

����

��• '.&��"#%$)&.-� ����/�������!

����1. (inner-product form) �,.*�"#%$)&.-�1+#(,���2���

���2. (outer-product form) �,.*�"#%$)&.-�1+#(,��2���

�����3. (middle-product form)�������

2018/4/26 ���0MPI� 147

Page 148: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

"=�PU&(Q

• ���� (inner-product formQ• ijk, jikMOL<AD��• for (i=0; i<n; i++) {

for (j=0; j<n; j++) {dc = 0.0;for (k=0; k<n; k++){

dc = dc + A[ i ][ k ] * B[ k ][ j ];}C[ i ][ j ]= dc;

} }

2018/4/26 ) �RMPI � 148

A B

….

-"�:�=FGJI0C→"�N���&(=��9�!��$�

%��RA, B;7B3��E*�5824

��+.��=MOL3B=��=,�9�#�E�@/6:1?�'=HOK>SijkMOLT/

Page 149: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

"=�PFortran&(Q

• ���� (inner-product formQ• ijk, jikMOL<AD��

• do i=1, n

do j=1, n

dc = 0.0d0

do k=1, n

dc = dc + A( i , k ) * B( k , j )

enddo

C( i , j ) = dc

enddo

enddo

2018/4/26 ) �RMPI � 149

A B

….

-"�:�=FGJI0C→"�N���&(=��9�!��$�

%��RA, B;7B3��E*�5824

��+.��=MOL3B=��=,�9�#�E�@/6:1?�'=HOK>SijkMOLT/

Page 150: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��� '*��(

• � �� (outer-product form(• kij, kji$&"�����• for (i=0; i<n; i++) {

for (j=0; j<n; j++) {C[ i ][ j ] = 0.0;

} }for (k=0; k<n; k++) {for (j=0; j<n; j++) {

db = B[ k ][ j ];for (i=0; i<n; i++) {C[ i ][ j ]= C[ i ][ j ]+ A[ i ][ k ]* db;}

}}

2018/4/26 ���)MPI�� 150

A B

�kji$&"������! �#�%→�������'+./0/,-��(

….

Page 151: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��� 'Fortran��(• � �� (outer-product form(

• kij, kji$&"�����• do i=1, n

do j=1, nC( i , j ) = 0.0d0

enddoenddodo k=1, ndo j=1, n

db = B( k , j )do i=1, nC( i , j ) = C( i , j )+ A( i , k ) * db

enddoenddo

enddo

2018/4/26 ���)MPI�� 151

A B

�kji$&"������! �#�%→�������'*-./.+,��(

….

Page 152: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

����(C��)• ���� (middle-product form)

• ikj, jki&'%�� � • for (j=0; j<n; j++) {

for (i=0; i<n; i++) {C[ i ][ j ] = 0.0;

}for (k=0; k<n; k++) {db = B[ k ][ j ];for (i=0; i<n; i++) {C[ i ][ j ] = C[ i ][ j ] + A[ i ][ k ] * db;

}}

}

2018/4/26 ���*MPI�� 152

A B

�jki&'%������!"$#→������������� (+./0/,-��)

.

.

Page 153: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

����(Fortran��)• ���� (middle-product form)

• ikj, jki&'%�� � • do j=1, n

do i=1, nC( i , j ) = 0.0d0

enddodo k=1, ndb = B( k , j )do i=1, n

C( i , j ) = C( i , j ) + A( i , k ) * dbenddo

enddoenddo

2018/4/26 ���*MPI�� 153

A B

�jki&'%������!"$#→������������� (+./0/,-��)

.

.

Page 154: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

����

• ������� ����)����*� )$'"���!�&(�+ (#�%�*

• ����

2018/4/26 ���+MPI�� 154

å=

=n

kkjikij BAC

1

~~~ npn /

pn /

=ijC~

1~iA…

piA~ jB1

~

jpB~

Page 155: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

2�

•�� 8;KD?L1�57>:A1-7(1. HOD<��/&#0��+ )72. �9N=MAJ2��+��/,7

•� �9N=MAJ3'EQC"�2��*6'��2T�1�% �U1. BIP?@FMD<��• A'B2� 2�$8EQC�RCannon29N=MAJS

2. GNP?@FMD<��• A'B2� 2-4.8EQC�RFox29N=MAJS

2018/4/26 !��UMPI�� 155

Page 156: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

�� � ���������-����

���MPI�� 1562018/4/26

Page 157: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��-���#+91718*52#���• C���/Fortran���#��/&'7�

Mat-Mat-ofp.tar.gz•,40-)61./&'7mat-mat.bash �#(3:�%lecture-flat �$ tutorial-flat"� �!�$pjsub �!� ���• lecture-flat : ���#(3:• tutorial-flat :����#(3:

���;MPI�� 1572018/4/26

Page 158: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��-����#+%)%*!('��•���"&+$ ���$ cp /work/gt00/z30105/Mat-Mat-ofp.tar.gz ./$ tar xvfz Mat-Mat-ofp.tar.gz$ cd Mat-Mat

•������� �$ cd C : C�� ���$ cd F : -0121./�� ���

•����$ make$ pjsub mat-mat.bash

•�� ������� ���$ cat mat-mat.bash.oXXXXXX

2018/4/26 ���,MPI� 158

Page 159: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��-���%094748.65%�:>��;��• %(�$�� ��+& �

N = 1088Mat-Mat time = 2.822111 [sec.]912.730515 [MFLOPS]OK!N = 1088Mat-Mat time = 0.567127 [sec.]4541.887430 [MFLOPS]OK!N = 1088Mat-Mat time = 0.302566 [sec.]8513.271503 [MFLOPS]OK!

2018/4/26 ���=MPI� 159

</,%'#�8.5G?@OPS%��

/,%�)�" �!"�*(13-72)

/,%���)�"export I_MPI_PIN_DOMAIN=4

MCDRAM%��export I_MPI_HBW_POLICY=hbw_bind

Page 160: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��-���%094748.65%�:Fortran��;

• ��%(�$�� ��+& �NN = 1088Mat-Mat time[sec.] = 2.93095993995667MFLOPS = 878.833894104587OK!NN = 1088Mat-Mat time[sec.] = 0.556317090988159MFLOPS = 4630.14165702036OK!NN = 1088Mat-Mat time[sec.] = 0.307068109512329MFLOPS = 8388.45473594594OK!

2018/4/26 ���=MPI� 160

</,%'#�8.5G>?OPS%��

/,%�)�" �!"�*(13-72)

/,%���)�"export I_MPI_PIN_DOMAIN=4

MCDRAM%��export I_MPI_HBW_POLICY=hbw_bind

Page 161: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

183637054*�

• #define N 1088*��.��$-'���1/2!��&"+$

• #define DEBUG 0*�9�.�:�)$-'���-���*����! �&"+$�

• MyMatMat�*��•?DEACB�N�N��<'=*���. #(��?DEACB�@�@��>)%*��!�,+$

���;MPI�� 1612018/4/26

Page 162: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

Fortran����"� �!������

•��������$$��������integer, parameter :: NN=1088

�#MPI�� 1622018/4/26

Page 163: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

� ��+-,

• MyMatMat�&����!�����• #define N 1088• #define DEBUG 1#�!�)*('&�!�����

•��/�0�1$��42"���!�� ��+3�3,��!%�"��

� �.MPI�� 1632018/4/26

Page 164: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��� JLK

•>HCFCG=ED.3!��N!O2��:��L/)-!��-���2� :6,��P2���&Q.#9%�4!� ��)-$5*"AB@<1��)-'+($"

•��P2����178!

��� ?;@<FI?H2���&�

1085*"�)-'+($"

���MMPI�� 1642018/4/26

Page 165: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

�� 2>A=

• ��26(0<B;�1,8/&/-5��.,'

• #�%�3��!.,'• ��-?:=@�2��/�+��.�� .)4,'

2018/4/26 "��EMPI�� 165

PE0

PE2

PED

PE3

PE0PEDF C

I G HK/4

N

PE0

PE2

PED

PE3

K/4

N

PE2PE3

�LJ.$ *-�!�9��

����3�074,

Page 166: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

MPI���7��I��$J•SPMD•�&7C?HAG@EBImat-matJ 8(•/937ML4(+2(• �6'-=0��

+;��,�:<)

•��CDF���#!�•�ML8(��6� .0CDF>�13*<)I��CDF485*J

%"�KMPI�� 1662018/4/26

Page 167: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

21* 21-21,21+

MPI������ (���)• �21� �/�$'#&%�����0�"����"!��

2018/4/26 ���.MPI� 167

mat-mat mat-mat mat-mat mat-mat

mpiexec.hydra mat-mat

Page 168: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

• �BA(,�>��� ��*��?"0.$�•

• myid��,�MPI_Init()��7/#!,�1346258 �-0%��(�>�BA �+�?*)&'�.$�

BA9 BA<BA;BA:

MPI��+��7���8

2018/4/26 ���=MPI�� 168

@CNDCND @CNDCND @CNDCND @CNDCND

BA9 BA<BA;BA:

myid = 0 myid = : myid = 2 myid = 3

Page 169: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

�PE�!��!����• �"���!$� ������&������ ���� �%#�

2018/4/26 ���*MPI� 169

PE0

+,/4

N

PE'

+

N

PE2

+

N

PE3

+

N

PE0 PE' PE( PE)

Page 170: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���<?;>=1TIPSmyid• , numprocs2� ��-+

myid• (=��1EC)"'63"numprocs(=�1�1FD��)1��2� ��-+#MyMatVec!�'63main�-"����5��/*0"��-)4+#

• @Fortran-2module MyCalc1�0$74+Amyid• , numprocs1��9�&��($74+

MyMatMat• !�9��+802" myid'63numprocs��9�*/%."��(-)4,:#

��BMPI�� 1702018/4/26

Page 171: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

�������(��-$ #'����C��)• SIMD�'!&"%����(4PE��)

2018/4/26 ���+MPI� 171

for ( j=0; j<n; j++) { ��( j, i ) }

PE0

for ( j=0; j<n/4; j++) { ��( j, i ) }

PE*

for ( j=n/4; j<(n/4)*2; j++) { ��( j, i ) }

PE2

for ( j=(n/4)*2; j<(n/4)*3; j++) { ��( j, i ) }

PE3

for ( j=(n/4)*3; j<n; j++) { ��( j, i ) }

�.-��������

��,

$ #'/

n

n

Page 172: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

�������'��-#�"&���Fortran��(

SIMD• �& %!$����'4PE��(

2018/4/26 ���*MPI� 172

do j=1, n ��( j, i )

enddo

PE0

do j=1, n/4��( j, i )

enddo

PE)

do j=n/4+1, (n/4)*2��( j, i )

enddo

PE2

do j=(n/4)*2+1, (n/4)*3��( j, i )

enddo

PE3

do j=(n/4)*3+1, n��( j, i )

enddo

�-,��������

��+

#�"&.

n

n

Page 173: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���>�+RW%(S

1. PE:$�AFN-N>�45.NGKOX.YFN>�45.��69A0;7D/

2. �PE?.��>!�>@& 7DA1=.OQM>,��;"��F��7D/

• LPIG���:?.��=<D/Rn 3 numprocs : C�ED��Sib = n / numprocs;for ( j=myid*ib; j<(myid+T)*ib; j++) { … }

3. RU>���3�="�68BS�PE:��>JQH*62$�F��6<0A1=��7D/

• �'>OQM?.��>A1=<D/for ( j=0; j<ib; j++) { … }

2018/4/26 )#�VMPI�� 173

Page 174: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���>�+RFortran%(S1. PE:$�AFN-N>�45.NGKOW.XFN>�45.��69A0;7D/

2. �PE?.��>!�>@& 7DA1=.OQM>,��;"��F��7D/LPIG• ���:?.��=<D/Rn 3 numprocs : C�ED��Sib = n / numprocsdo j=myid*ib+T, (myid+T)*ib …. enddoRU>3. ���3�="�68BS�PE:��>JQH*62$�F��6<0A1=��7D/�'• >OQM?.��>A1=<D/do j=T, ib …. enddo

2018/4/26 )#�VMPI�� 174

Page 175: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���$�

• -/,��'*./+-��#�& ����%�(���./)-������0 2 "!1#�������

• for (i=i_start; i<i_end; i++) {……

}

���2MPI� 175

./)-��#�&�

2018/4/26

Page 176: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

MPI��������(�)(�)����������� �����������

2018/4/26 ��MPI�� 176

Page 177: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

����

1. ��-���+-,�"*%(%)!'&��

"*%(%)!'&2. ���

���+-,.3. �����������

���4. �$*#

���.MPI�� 1772018/4/26

Page 178: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

�� � ���������-�������

���MPI�� 1782018/4/26

Page 179: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��-���!)7/5/6(30! �• C���/Fortran���!-$%5�

Mat-Mat-d-ofp.tar.gz•*2.+'4/,-$%5mat-mat-d.bash �!&18�#

lecture-flat �" tutorial-flat �����"qsub �������• lecture-flat: ����!&18• tutorial-flat: ����!&18

���9MPI�� 1792018/4/26

Page 180: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��-���(2)�#+%)%*!('��• ���"&+$ ���

$ cp /work/gt00/z30105/Mat-Mat-d-ofp.tar.gz ./$ tar xvfz Mat-Mat-d-ofp.tar.gz$ cd Mat-Mat-d

• ������� �$ cd C : C�� ���$ cd F : -0121./�� ���

• ����$ make$ pjsub mat-mat-d.bash

• �� ������� ���$ cat mat-mat-d.bash.oXXXXXX

���,MPI� 1802018/4/26

Page 181: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��-���'/50304.21' �7:���8• ��'+�&�� ��,)��Error! in ( 0 , 2 )-th argument in PE 0 Error! in ( 0 , 2 )-th argument in PE 61…N = 2176Mat-Mat time = 0.000896 [sec.]22999044.714267 [MFLOPS]...N = 2176Mat-Mat time = 0.000285 [sec.]72326702.959176 [MFLOPS]...N = 2176Mat-Mat time = 0.000237 [sec.]86952122.772853 [MFLOPS]

2018/4/26 ���9MPI�� 181

�� ��"$�&�'%-26 �*#�%# �!,(�"��%#

Page 182: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��-���&.4/2/3-10& �6Fortran��7• ��&*�%�����+(��Error! in ( 1 , 3 )-th argument in PE 0Error! in ( 1 , 3 )-th argument in PE 1033...NN = 2176Mat-Mat time = 1.497983932495117E-003MFLOPS = 13756232.6624224

...NN = 2176Mat-Mat time = 1.003026962280273E-003MFLOPS = 20544428.2904683

...NN = 2176Mat-Mat time = 1.110076904296875E-003MFLOPS = 18563232.3492268

2018/4/26 ���8MPI�� 182

�����!#�%�&$,15��)"�$"�� +'�!��$"�

Page 183: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

183637054*�

#define N 2176•�• .��$-'���1/2!��&"+$

#define DEBUG • <�;�.�<�)$-'�• ��-���*����! �&"+$�

MyMatMat• �*��AFGCED• �*��>99N/NPROCS:�N��:'?99N�9N/NPROCS:��::*���. #(��AFGCED�*9N/NPROCS)�B��@)�%*��!�,+$�

2018/4/26 ���=MPI�� 183

Page 184: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

Fortran����"� �!������

•��������$$��������integer, parameter :: NN=2176

�#MPI�� 1842018/4/26

Page 185: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

����/10

• MyMatMat��/��0(���!#�" ��•,-+) '#define N 2176$!#�" ��

•��3�4�5&����/,.*��0(���%�!#�" ��

���2MPI�� 1852018/4/26

Page 186: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��<�=�>,����

• ��<�=�>,��-��,0 +*'(�/$�6%&#��-4PE,�)�����-�*1/$�7

• 9 9����"��)$�• ��<�=�>,��,.!+����4352,��"��)$�

2018/4/26 ���:MPI� 186

PE0

PE2

PE9

PE3

PE0; 8

> < =?/NPROCS

N

PE0

PE2

PE9

PE3

?/NPROCS

N

PE9

PE2

PE3

?/NPROCS

N

Page 187: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

�������

����MPI�� 187

"PE0

PE2

PE�

PE3

N /NPROCS

N

PE0

PE2

PE�

PE3

PE0

!N /NPROCS

N

PE�

PE2

PE3

N / NPROCS

N���

���

l����4PE�������� �������

2018/4/26

Page 188: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���+� :>��;

• ��,���*��!/&�-#�

• @?',���+.�)085423+��()%&�-#�

• PE'���7916)��-������+085423��*� "& $!��

2018/4/26 ���<MPI� 188

=[i][j] B[i][j]C[i][j]

0 N-10

N/NPROCS-1

j

i

0

N-1

j

0N/NPROCS-1

i

0

N/NPROCS-1i

0j

N-1

Page 189: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���+� :Fortran��;

• ��,���*��!/&�-#�

• ?>',���+.�)085423+��()%&�-#�

• PE'���7916)��-������+085423��*� "& $!��

2018/4/26 ���<MPI� 189

=( i, j ) B( i, j )

C( i, j )

1 N1

N/NPROCS

j

i

1

N

j

1N/NPROCS

i

1

N/NPROCSi

1j

N

Page 190: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��,=A<

• ���5��#3+-�PE(��*��B,;B8"*�,(���B,;B8+&�'��"��(#�• $)!.���,0 +��#3��"�2/#�• 7:9>E

2018/4/26 ���FMPI�� 190

PE0

PE2

PEE

PE3

PE0G C

J H I

PE0

PE2

PEE

PE3

PEE

PE2

PE3

KDKMNLJO

@B6?*;B85�%' 14$��-�����

KDKMNLJO

Page 191: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���$493

• .106@

2018/4/26 ���BMPI�� 191

PE0

PE2

PE?

PE3

PE0

C =

F D

E

PE0

PE2

PE?

PE3

PE?

PE2

PE3

��$�� �)2:/+&!���(#���);HG>%�HGA#�)<� �-53���

PE3

E

PE0

PE?

PE2

8:,7"2:/+�� '*���-�����

Page 192: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���$493

• .1063

2018/4/26 ���AMPI�� 192

PE0

PE2

PE?

PE3

PE3

B =

C

D

PE0

PE?

PE2

��$�� �)2:/+&!���(#���);GF>%�GF@#�)<� �-53���

PE2

D

PE3

PE0

PE?

8:,7"2:/+�� '*���-�����

EPE0

PE2

PE?

PE3

Page 193: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���H"�

• �$�V\Y/0Q�,>O35��:MPI_SendQ�G&+>OE5?H��D#:�JO6^ 'GI5 7AN5 9F9BAN5>O_• MPI_SendH#�D5��GLN5ZX\R4�:F;FO6• ZX\R4�:);JD�C^W[]TUSY>O_6• =9=5ZX\R4��.9M5!1G)9F76

• <PQ�2>OAK5��H�,Q+86• cb%�:`D�NPOcba• MPI_Send();• MPI_Recv();

• ?P��Hcba• MPI_Recv();• MPI_Send();

2018/4/26 -*�aMPI�( 193

?P@PG��

Page 194: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

#!$(!������% *)• ���,�"!'����&$�����

� �-MPI�� 194

PE3

.

PE0

PE+

PE2

�"!'+-,������0/�#* ���

�"!',-,�������0/�#* ���

2018/4/26

Page 195: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��)MPI��―MPI_Recv (1/2)ierr• = MPI_Recv(recvbuf, icount, idatatype, isource,

itag, icomm, istatus);

recvbuf• : ���*����.��&-!

icount• : ���!���*482���.��&-!

idatatype• : ���!���*482*�.��&-!

MPI_CHAR • (���) MPI_INT (���) MPI_FLOAT ( ��) MPI_DOUBLE(��� ��)

isource• : ���!��%'"53180.��&-PE*67/.��&-!

��• *PE#,��%'"($+ MPI_ANY_SOURCE .��&-!

2018/4/26 ���9MPI� 195

Page 196: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���7MPI(�―MPI_Recv (2/2)• itag : �� +�13-MIGQF8�-4->HC9�@��2>+

• ��9HC�9MIGQF@�13-6/:*MPI_ANY_TAG @��2>+

• icomm : �� +PE)�@"$2>��5,>ELNKDQH@��2>+

• &�5:MPI_COMM_WORLD @��2?;<-+• istatus : MPI_Status R�� 9'S+���8(2>��.�>+

• ��.MPI_STATUS_SIZE9��'.�!0?>+• �13MIGQF9%��9OPB. istatus[MPI_SOURCE]*HC. istatus[MPI_TAG] 8��0?>+

• ierr(�=�) : �� +AOQEQJ.�>+

2018/4/26 #��TMPI�� 196

Page 197: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

�!�=��

•MK(itag)<6'7• MPI_Send(), MPI_Recv()8�GFMKSitagT>%��=

int�=��I��/7C'81&• 34/%0�S0;:TI��1F9%:=$�<��1F*H*D;-;E%"53$�+ HGF*B/GA2J&

• ���LON$�8>%MPI_Send()9 MPI_Recv()=�+%U687,A1&.GDI=MK</3�+%CE��81&

• 39)?%��>� QRP=�iloop9/7%B(��Iiloop+NPROCS91G?%�QRP�8MK+@6*F.9+;-%��81&

#��VMPI�� 1972018/4/26

Page 198: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

�%�&��� )*(�����'�%��� �$ ����

!"� �����#��

���+MPI�� 1982018/4/26

Page 199: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���5GNE

1. ���AIE6#TS��-P� ��2. ��R5DOB<*�;.8#��B[][]4"-;FCI=��B_T[][](��

3. *�0.B_T[][] <#MO>L3��-���2�&.8#B[][]7@HO-;$

4. MO>L3��-���<-; 5#��JMC?5���QJMC?�*myid$LOK�4JMC?�/*�9,1%)(#N<!'.:04�+3)16%*3%$

2018/4/26 ��QMPI�� 199

Page 200: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

����"& (�����,��)• �������'!����

2018/4/26 ���*MPI� 200

ib = n/numprocs;

for (iloop=0; iloop<NPROCS; iloop++ ) {

%'�$� �- � C = A * B;

if (iloop != (numprocs-1) ) {

if (myid % 2 == 0 ) {

MPI_Send(B, ib*n, MPI_DOUBLE, isendPE,

iloop, MPI_COMM_WORLD);

MPI_Recv(B_T, ib*n, MPI_DOUBLE, irecvPE,

iloop+numprocs, MPI_COMM_WORLD, &istatus);

} else {

MPI_Recv(B_T, ib*n, MPI_DOUBLE, irecvPE,

iloop, MPI_COMM_WORLD, &istatus);

MPI_Send(B, ib*n, MPI_DOUBLE, isendPE,

iloop+numprocs, MPI_COMM_WORLD);

}

B[ ][ ] � B_T[ ][ ] ��#'��+}

}

Page 201: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���� #�%�����(��&

• "$�!� �- ����������$�����

2018/4/26 ���'MPI� 201

jstart=ib*( (myid+iloop)%NPROCS );for (i=0; i<ib; i++) {

for(j=0; j<ib; j++) {for(k=0; k<n; k++) {

C[ i ][ jstart + j ] += A[ i ][ k ] * B[ k ][ j ];}

}}

Page 202: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

����!%�'����)Fortran��(• �������& ����

2018/4/26 ���*MPI� 202

ib = n/numprocs

do iloop=0, NPROCS-1

$&�#� �- � C = A * B

if (iloop .ne. (numprocs-1) ) then

if (mod(myid, 2) .eq. 0 ) then

call MPI_SEND(B, ib*n, MPI_REAL8, isendPE, &

iloop, MPI_COMM_WORLD, ierr)

call MPI_RECV(B_T, ib*n, MPI_REAL8, irecvPE, &

iloop+numprocs, MPI_COMM_WORLD, istatus, ierr)

else

call MPI_RECV(B_T, ib*n, MPI_REAL8, irecvPE, &

iloop, MPI_COMM_WORLD, istatus, ierr)

call MPI_SEND(B, ib*n, MPI_REAL8, isendPE, &

iloop+numprocs, MPI_COMM_WORLD, ierr)

endif

B � B_T ��"&��endif

enddo

Page 203: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

���� #�%����'Fortran��&• "$�!� �- ����������$�����

2018/4/26 ���(MPI� 203

imod = mod( (myid+iloop), NPROCS ) jstart = ib* imoddo i=1, ib

do j=1, ibdo k=1, n

C( i , jstart + j ) = C( i , jstart + j ) + A( i , k ) * B( k , j )enddo

enddoenddo

Page 204: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

�� ������

����MPI�� 2042018/4/26

Page 205: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

����������!� ��� ������������

���"MPI� 2052018/4/26

Page 206: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

Cannon-3@5?7>•;C9����-��

•=A8:6B4?:<. �����

• PE�# 2-/$��&"+1,!

•PE. ��A B C-��(0���2 D)*)��

•��A B-���+' $%-����2��

2018/4/26 ���EMPI�� 206

)0,0(P

)1,0( -pP

)0,1( -pP

…)1,1( -- ppP

Page 207: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��,�:��)��

•��• =��>)-�;',654732;',@?+�0%)( 0• MPI_Send� �MPI_Recv� (��($0��,%)• ;�;��).!"

•��• =��>)-���654732� ,@?+8��+9��&0%)( 0

• MPI_Bcast� (��($0��,%)• ;���).!"• ��,��*��)�#/10

2018/4/26 ���<MPI�� 207

Page 208: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

Cannon��,�+"*• �,�+"*���• ��!%$)

• ��!%$)

2018/4/26 ���2MPI� 208

A B

A B

20�����

21�����

0�����

20�����

21�����0�����

1�����

���'#/.�0���� (&�

���'#/.�0���� (&�

1�����

-/�,���-����

-/�,���-����

Page 209: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

Cannon09J=I?F3-5•Q��• >EA��R04,���

O�O��M� ��N• 04,���

���• /� PE/(#*.$)+!.!C@AKL<M�"2���G@>HN�%

����• $DLB:;9,,%7C@AKL<86*���,1 �&.7'-6

2018/4/26 ���PMPI� 209

Page 210: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

Fox��%�$�#

•�%�$�#��

• ����"

• ����"

2018/4/26 ���'MPI�� 210

A B

A B��PE�&������! �

&�����

Page 211: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

Fox2;P@OCK

•6/7•V�� �SJPD>MBFTW(�•���1"�PE1*',0(+-%0%HEFQR?S�&4���LEANT.�(�%S ���(��T

•�� �(IRG<=;..)9HEFQR?:8,���.3$Cannon2;P@OCK1�5#!/09

2018/4/26 ���UMPI�� 211

Page 212: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��6�+*�-2���

•��1. <C:����H��A!B!CH���=B;7����DBlockGFE

2. ?@A1��0��%"4'.H��(5*��B6�PE1�� -&4'.

•/#3+,!��B6 )4$I•����2��6>B9;8� ��

2018/4/26 ���HMPI�� 212

abc

a b c! !…

abc

a b c

Page 213: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

��/�$#�&(���

��•�����• B(����! ,..*�����)��

��• B(����! ,.%�-(&�#' *�����0132(+&���!�&"-4����(��!��5

2018/4/26 ���6MPI� 213

Page 214: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

SUMMA�PUMMA•����#���$4(3+11. SUMMA6Scalable Universal Matrix

Multiplication Algorithm7Ø R.Van de Geijin ��1997Ø �� �604,&2*.7�!��

2. PUMMA6Parallel Universal MatrixMultiplication Algorithms7

Ø Choi ��1994Ø ���/5-')%'3-'����"��Fox�$4(3+1

2018/4/26 ���8MPI�� 214

Page 215: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

SUMMA• �� ������• ������

• ������

2018/4/26 ��MPI�� 215

A B

A B

Page 216: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

SUMMA•��• ���"9CI?;J<$�L�NMPI_BcastM/��.80'����*�+17����3�218• SUMMA2),8EH>;F=A4'&��#�3O O#�L�NMPI_IsendM/��.8-0/'#�0 �3:KBG?DL #�%� M��•�3P=@?D956��2

2018/4/26 !��QMPI� 216

A B A B

�P=@?D�/�(#�9:KBG?D

Page 217: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

PUMMA•��•���9=72412;72�����)Fox0<3;5:• ScaLAPACK����9=72412;72��/��"&�- '�,��!.$•�?

2018/4/26 ���?MPI� 217

A A@�#PEA� �"&�-8>6%�,� �8>6/*'+&@���PEA(��(�-

Page 218: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

Strassen?HQKPLO•"�='��U ?� ; ?

• Strassen?HQKPLO:@ ?�

•HIMHU V��#��W•'�G�'�>��69/* G��

•�.?�%•����B'�?JNR2�)•"�=�(�CD,4=E5;20E

•��?�8�D/�����-=<?��G7FA/SX2�31!�:T�?&1�(2�%

2018/4/26 +$�UMPI � 218

3n 3)1( -n7logn

Page 219: 98 #*,+50 ! 14-325- · 2018. 4. 24. · 2D,)+3,4E126;& • Oakleaf-FX ( %PRIMEHPC FX10) • 1.135 PF, 0@9=B4 4 2018 , 20123 • Oakbridge-FX ( %PRIMEHPC FX10) • 136.2 TF, ' (D168

StrassenAHOINJK•��A��•HOINJKG$@��LMN ��)#�@�(9D=-�5��

•�&5<?3

• PE�A'�!GStrassen<'31PE0GSUMMA?><�(9D=���?�(5�&•=6F5-�/B1HOINJKA�+4C1-�A'�-'�!HOINJK@�8;��9D26A�+G��8;1,�1StrassenG�3:-��.HOINJK5�"7E;3D2

2018/4/26 *%�PMPI� 219