MPI Introduction

MPI

Rohit Banga

Prakher Anand

K Swagat

Manoj Gupta

Advanced Computer Architecture

Spring, 2010

ORGANIZATION

Basics of MPI Point to Point Communication Collective Communication Demo

GOALS

Explain basics of MPI Start coding today! Keep It Short and Simple

MESSAGE PASSING INTERFACE

A message passing library specification Extended message-passing model Not a language or compiler specification Not a specific implementation, several

implementations (like pthread) standard for distributed memory, message

passing, parallel computing Distributed Memory – Shared Nothing

approach! Some interconnection technology – TCP,

INFINIBAND (on our cluster)

GOALS OF MPI SPECIFICATION

Provide source code portability Allow efficient implementations Flexible to port different algorithms on

different hardware environments Support for heterogeneous architectures –

processors not identical

REASONS FOR USING MPI

Standardization – virtually all HPC platforms Portability – same code runs on another

platform Performance – vendor implementations

should exploit native hardware features Functionality – 115 routines Availability – a variety of implementations

available

BASIC MODEL

Communicators and Groups Group

ordered set of processes each process is associated with a unique integer

rank rank from 0 to (N-1) for N processes an object in system memory accessed by handle MPI_GROUP_EMPTY MPI_GROUP_NULL

BASIC MODEL (CONTD.)

Communicator Group of processes that may communicate with

each other MPI messages must specify a communicator An object in memory Handle to access the object

There is a default communicator (automatically defined):

MPI_COMM_WORLD identify the group of all processes

COMMUNICATORS

Intra-Communicator – All processes from the same group

Inter-Communicator – Processes picked up from several groups

COMMUNICATOR AND GROUPS

For a programmer, group and communicator are one

Allow you to organize tasks, based upon function, into task groups

Enable Collective Communications (later) operations across a subset of related tasks

safe communications Many Communicators at the same time Dynamic – can be created and destroyed at run

time Process may be in more than one

group/communicator – unique rank in every group/communicator

implementing user defined virtual topologies

VIRTUAL TOPOLOGIES coord (0,0):

rank 0 coord (0,1):

rank 1 coord (1,0):

rank 2 coord (1,1):

rank 3

Attach graph topology information to an existing communicator

SEMANTICS

Header file #include <mpi.h> (C) include mpif.h (fortran) Java, Python etc.

Format: rc = MPI_Xxxxx(parameter, ... )

Example:

rc = MPI_Bsend(&buf,count,type,dest,tag,comm)

Error code:

Returned as "rc". MPI_SUCCESS if successful

MPI PROGRAM STRUCTURE

MPI FUNCTIONS – MINIMAL SUBSET

MPI_Init – Initialize MPI MPI_Comm_size – size of group associated

with the communicator MPI_Comm_rank – identify the process MPI_Send MPI_Recv MPI_Finalize

We will discuss simple ones first

CLASSIFICATION OF MPI ROUTINES

Environment Management MPI_Init, MPI_Finalize

Point-to-Point Communication MPI_Send, MPI_Recv

Collective Communication MPI_Reduce, MPI_Bcast

Information on the Processes MPI_Comm_rank, MPI_Get_processor_name

MPI_INIT

All MPI programs call this before using other MPI functions int MPI_Init(int *pargc, char ***pargv);

Must be called in every MPI program Must be called only once and before any other MPI

functions are called Pass command line arguments to all processes

int main(int argc, char **argv){

MPI_Init(&argc, &argv);…

}

MPI_COMM_SIZE Number of processes in the group associated with a

communicator int MPI_Comm_size(MPI_Comm comm, int *psize);

Find out number of processes being used by your application


MPI_Init(&argc, &argv);int p;MPI_Comm_size(MPI_COMM_WORLD, &p);…

}

MPI_COMM_RANK Rank of the calling process within the communicator Unique Rank between 0 and (p-1) Can be called task ID

int MPI_Comm_rank(MPI_Comm comm, int *rank); Unique rank for a process in each communicator it belongs to Used to identify work for the processor


MPI_Init(&argc, &argv);int p;MPI_Comm_size(MPI_COMM_WORLD, &p);int rank;MPI_Comm_rank(MPI_COMM_WORLD, &rank);…

}

MPI_FINALIZE

Terminates the MPI execution environment Last MPI routine to be called in any MPI program

int MPI_Finalize(void);


MPI_Init(&argc, &argv);int p;MPI_Comm_size(MPI_COMM_WORLD, &p);int rank;MPI_Comm_rank(MPI_COMM_WORLD, &rank);printf(“no. of processors: %d\n rank: %d”, p,

rank);MPI_Finalize();

}

HOW TO COMPILE THIS

Open MPI implementation on our Cluster mpicc -o test_1 test_1.c Like gcc only mpicc not a special compiler

$mpicc: gcc: no input files Mpi implemented just as any other library Just a wrapper around gcc that includes required

command line parameters

HOW TO RUN THIS

mpirun -np X test_1 Will run X copies of program in your current

run time environment np option specifies number of copies of

program

MPIRUN

Only rank 0 process can receive standard input. mpirun redirects standard input of all others to

/dev/null Open MPI redirects standard input of mpirun to

standard input of rank 0 process Node which invoked mpirun need not be the

same as the node for the MPI_COMM_WORLD rank 0 process

mpirun directs standard output and error of remote nodes to the node that invoked mpirun

SIGTERM, SIGKILL kill all processes in the communicator

SIGUSR1, SIGUSR2 propagated to all processes All other signals ignored

A NOTE ON IMPLEMENTATION

I want to implement my own version of MPI Evidence

MPI_Init

MPI Thread

MPI_Init

MPI Thread

SOME MORE FUNCTIONS

int MPI_Init (&flag) Check if MPI_Initialized has been called Why?

int MPI_Wtime() Returns elapsed wall clock time in seconds

(double precision) on the calling processor int MPI_Wtick()

Returns the resolution in seconds (double precision) of MPI_Wtime()

Message Passing Functionality That is what MPI is meant for!

POINT TO POINT COMMUNICATION

POINT-TO-POINT COMMUNICATION

Communication between 2 and only 2 processes

One sending and one receiving Types

• Synchronous send• Blocking send / blocking receive• Non-blocking send / non-blocking receive• Buffered send• Combined send/receive• "Ready" send


Processes can be collected into groups Each message is sent in a context, and must be received in the same context A group and context together form a

Communicator A process is identified by its rank in the

group associated with a communicator Messages are sent with an accompanying

user defined integer tag, to assist the receiving process in identifying the message

MPI_ANY_TAG


How is “data” described? How are processes identified? How does the receiver recognize messages? What does it mean for these operations to

complete?

BLOCKING SEND/RECEIVE

int MPI_Send(void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm communicator)

buf: pointer - data to send count: number of elements in buffer . Datatype : which kind of data types in

buffer ?




buffer ?




buffer ?




buffer ? dest: the receiver tag: the label of the message communicator: set of processors involved

(MPI_COMM_WORLD)





(MPI_COMM_WORLD)





(MPI_COMM_WORLD)

BLOCKING SEND/RECEIVE (CONTD.)

Processor 1

Process 1

Application Send

System Buffer

Processor 2

Process 2

Application Send

System Buffer

Data

A WORD ABOUT SPECIFICATION

The user does not know if MPI implementation: copies BUFFER in an internal buffer, start

communication, and returns control before all the data are transferred. (BUFFERING)

create links between processors, send data and return control when all the data are sent (but NOT received)

uses a combination of the above methods

BLOCKING SEND/RECEIVE (CONTD.) "return" after it is safe to modify the application

buffer Safe

modifications will not affect the data intended for the receive task

does not imply that the data was actually received Blocking send can be synchronous which means

there is handshaking occurring with the receive task to confirm a safe send

A blocking send can be asynchronous if a system buffer is used to hold the data for eventual delivery to the receive

A blocking receive only "returns" after the data has arrived and is ready for use by the program

NON-BLOCKING SEND/RECEIVE

return almost immediately simply "request" the MPI library to perform

the operation when it is able Cannot predict when that will happen request a send/receive and start doing other

work! unsafe to modify the application buffer (your

variable space) until you know that the non-blocking operation has been completed

MPI_Isend (&buf,count,datatype,dest,tag,comm,&request)

MPI_Irecv (&buf,count,datatype,source,tag,comm,&request)

NON-BLOCKING SEND/RECEIVE (CONTD.)

Processor 1

Process 1

Application Send

System Buffer

Processor 2

Process 2

Application Send

System Buffer

Data

To check if the send/receive operations have completed

int MPI_Irecv (void *buf, int count, MPI_Datatype type, int dest, int tag, MPI_Comm comm, MPI_Request *req);

int MPI_Wait(MPI_Request *req, MPI_Status *status); A call to this subroutine cause the code to wait until

the communication pointed by req is complete input/output, identifier associated to a

communications event (initiated by MPI_ISEND or MPI_IRECV). input/output, identifier associated to a

communications event (initiated by MPI_ISEND or MPI_IRECV).


int MPI_Test(MPI_Request *req, int *flag, MPI_Status *status); A call to this subroutine sets flag to true if the

communication pointed by req is complete, sets flag to false otherwise.


STANDARD MODE• Returns when Sender is free to access and overwrite

the send buffer.

• Might be copied directly into the matching receive buffer, or might be copied into a temporary system buffer.

• Message buffering decouples the send and receive operations.

• Message buffering can be expensive.

• It is up to MPI to decide whether outgoing messages will be buffered

• The standard mode send is non-local.

SYNCHRONOUS MODE

Send can be started whether or not a matching receive was posted.

Send completes successfully only if a corresponding receive was already posted and has already started to receive the message sent.

Blocking send & Blocking receive in synchronous mode.

Simulate a synchronous communication. Synchronous Send is non-local.

BUFFERED MODE

Send operation can be started whether or not a matching receive has been posted.

It may complete before a matching receive is posted.

Operation is local. MPI must buffer the outgoing message. Error will occur if there is insufficient buffer

space. The amount of available buffer space is

controlled by the user.

BUFFER MANAGEMENT

int MPI_Buffer_attach( void* buffer, int size) Provides to MPI a buffer in the user's memory

to be used for buffering outgoing messages. int MPI_Buffer_detach( void* buffer_addr, int*

size) Detach the buffer currently associated with

MPI.MPI_Buffer_attach( malloc(BUFFSIZE), BUFFSIZE); /* a buffer of BUFFSIZE bytes can now be used by MPI_Bsend */

MPI_Buffer_detach( &buff, &size); /* Buffer size reduced to zero */

MPI_Buffer_attach( buff, size); /* Buffer of BUFFSIZE bytes available again */

READY MODE

A send may be started only if the matching receive is already posted.

The user must be sure of this. If the receive is not already posted, the

operation is erroneous and its outcome is undefined.

Completion of the send operation does not depend on the status of a matching receive.

Merely indicates that the send buffer can be reused.

Ready-send could be replaced by a standard-send with no effect on the behavior of the program other than performance.

ORDER AND FAIRNESS

• Order: • MPI Messages are non-overtaking. • When a receive matches 2 messages.• When a sent message matches 2 receive

statements.• Message-passing code is deterministic, unless

the processes are multi-threaded or the wild-card MPI_ANY_SOURCE is used in a receive statement.

• Fairness: – MPI does not guarantee fairness– Example: task 0 sends a message to task 2.

However, task 1 sends a competing message that matches task 2's receive. Only one of the sends will complete.

EXAMPLE OF NON-OVERTAKING MESSAGES.

CALL MPI_COMM_RANK(comm, rank, ierr)

IF (rank.EQ.0) THEN

CALL MPI_BSEND(buf1, count, MPI_REAL, 1, tag, comm, ierr)

CALL MPI_BSEND(buf2, count, MPI_REAL, 1, tag, comm, ierr)

ELSE ! rank.EQ.1

CALL MPI_RECV(buf1, count, MPI_REAL, 0, MPI_ANY_TAG, comm, status, ierr)

CALL MPI_RECV(buf2, count, MPI_REAL, 0, tag, comm, status, ierr)

END IF

EXAMPLE OF INTERTWINGLED MESSAGES.


IF (rank.EQ.0) THEN

CALL MPI_BSEND(buf1, count, MPI_REAL, 1, tag1, comm, ierr)

CALL MPI_SSEND(buf2, count, MPI_REAL, 1, tag2, comm, ierr)

ELSE ! rank.EQ.1

CALL MPI_RECV(buf1, count, MPI_REAL, 0, tag2, comm, status, ierr)

CALL MPI_RECV(buf2, count, MPI_REAL, 0, tag1, comm, status, ierr)

END IF

DEADLOCK EXAMPLE


IF (rank.EQ.0) THEN

CALL MPI_RECV(recvbuf, count, MPI_REAL, 1, tag, comm, status, ierr) CALL MPI_SEND(sendbuf, count, MPI_REAL, 1, tag, comm, ierr)

ELSE ! rank.EQ.1

CALL MPI_RECV(recvbuf, count, MPI_REAL, 0, tag, comm, status, ierr) CALL MPI_SEND(sendbuf, count, MPI_REAL, 0, tag, comm, ierr)

END IF

EXAMPLE OF BUFFERING


IF (rank.EQ.0) THEN

CALL MPI_SEND(buf1, count, MPI_REAL, 1, tag, comm, ierr)

CALL MPI_RECV (recvbuf, count, MPI_REAL, 1, tag, comm, status, ierr)

ELSE ! rank.EQ.1

CALL MPI_SEND(sendbuf, count, MPI_REAL, 0, tag, comm, ierr)

CALL MPI_RECV(buf2, count, MPI_REAL, 0, tag, comm, status, ierr)

END IF

COLLECTIVE COMMUNICATIONS

COLLECTIVE ROUTINES Collective routines provide a higher-level way to

organize a parallel program. Each process executes the same communication

operations. Communications involving group of processes in

a communicator. Groups and communicators can be constructed

“by hand” or using topology routines. Tags are not used; different communicators

deliver similar functionality. No non-blocking collective operations. Three classes of operations: synchronization,

data movement, collective computation.

COLLECTIVE ROUTINES (CONTD.) int MPI_Barrier(MPI_Comm comm) Stop processes until all processes within a

communicator reach the barrier Occasionally useful in measuring performance

COLLECTIVE ROUTINES (CONTD.)

int MPI_Bcast(void *buf, int count, MPI_Datatype datatype, int root, MPI_Comm comm)

Broadcast One-to-all communication: same data sent

from root process to all others in the communicator


Reduction The reduction operation allow to:

Collect data from each process Reduce the data to a single value Store the result on the root processes Store the result on all processes

Reduction function works with arrays other operation: product, min, max, and, …. Internally is usually implemented with a

binary tree


int MPI_Reduce/MPI_Allreduce(void * snd_buf, void * rcv_buf, int count, MPI_Datatype type, MPI_Op op, int root, MPI_Comm comm)

snd_buf: input array rcv_buf output array count: number of element of snd_buf and

rcv_buf type: MPI type of snd_buf and rcv_buf op: parallel operation to be performed root: MPI id of the process storing the result comm: communicator of processes involved

in the operation

MPI OPERATIONSMPI_OP operator

MPI_MIN Minimum

MPI_SUM Sum

MPI_PROD product

MPI_MAX maximum

MPI_LAND Logical and

MPI_BAND Bitwise and

MPI_LOR Logical or

MPI_BOR Bitwise or

MPI_LXOR Logical xor

MPI_BXOR Bit-wise xor

MPI_MAXLOC Max value and location

MPI_MINLOC Min value and location


Learn by Examples

Parallel Trapezoidal Rule

Output: Estimate of the integral from a to b of f(x) using the trapezoidal rule and n trapezoids. Algorithm: 1. Each process calculates "its" interval of integration. 2. Each process estimates the integral of f(x) over its interval using the trapezoidal rule. 3a. Each process != 0 sends its integral to 0. 3b. Process 0 sums the calculations received from the individual processes and prints the result.Notes: 1. f(x), a, b, and n are all hardwired.2. The number of processes (p) should evenly divide the number of trapezoids (n = 1024)

Parallelizing the Trapezoidal Rule

#include <stdio.h>#include "mpi.h"main(int argc, char** argv) { int my_rank; /* My process rank */ int p; /* The number of processes */ double a = 0.0; /* Left endpoint */ double b = 1.0; /* Right endpoint */ int n = 1024; /* Number of trapezoids */ double h; /* Trapezoid base length */ double local_a; /* Left endpoint my process */ double local_b; /* Right endpoint my process */ int local_n; /* Number of trapezoids for */ /* my calculation */ double integral; /* Integral over my interval */ double total; /* Total integral */ int source; /* Process sending integral */ int dest = 0; /* All messages go to 0 */ int tag = 0; MPI_Status status;

Continued…

double Trap(double local_a, double local_b, int local_n,double h);

/* Calculate local integral */ MPI_Init (&argc, &argv);

MPI_Barrier(MPI_COMM_WORLD); double elapsed_time = -MPI_Wtime();

MPI_Comm_rank(MPI_COMM_WORLD, &my_rank); MPI_Comm_size(MPI_COMM_WORLD, &p);

h = (b-a)/n; /* h is the same for all processes */ local_n = n/p; /* So is the number of trapezoids */ /* Length of each process' interval of integration = local_n*h.

So my interval starts at: */ local_a = a + my_rank*local_n*h; local_b = local_a + local_n*h; integral = Trap(local_a, local_b, local_n, h);

Continued…

/* Add up the integrals calculated by each process */ if (my_rank == 0) { total = integral; for (source = 1; source < p; source++) { MPI_Recv(&integral, 1, MPI_DOUBLE, source, tag,

MPI_COMM_WORLD, &status); total = total + integral; }//End for } else MPI_Send(&integral, 1, MPI_DOUBLE, dest, tag,

MPI_COMM_WORLD); MPI_Barrier(MPI_COMM_WORLD); elapsed_time += MPI_Wtime(); /* Print the result */ if (my_rank == 0) { printf("With n = %d trapezoids, our estimate\n",n); printf("of the integral from %lf to %lf = %lf\n",a, b, total); printf("time taken: %lf\n", elapsed_time); }

Continued…

/* Shut down MPI */ MPI_Finalize();} /* main */

double Trap( double local_a , double local_b, int local_n, double h) { double integral; /* Store result in integral */ double x; int i; double f(double x); /* function we're integrating */ integral = (f(local_a) + f(local_b))/2.0; x = local_a; for (i = 1; i <= local_n-1; i++) { x = x + h; integral = integral + f(x); } integral = integral*h; return integral;} /* Trap */

Continued…

double f(double x) { double return_val; /* Calculate f(x). */ /* Store calculation in return_val. */ return_val = 4 / (1+x*x); return return_val;} /* f */

Program 2

Process other than root generates the random value less than 1 and sends to root. Root sums up and displays sum.

#include <stdio.h>#include <mpi.h>#include<stdlib.h>#include <string.h>#include<time.h>

int main(int argc, char **argv){ int myrank, p; int tag =0, dest=0; int i; double randIn,randOut; int source;

MPI_Status status;

MPI_Init(&argc,&argv);

MPI_Comm_rank(MPI_COMM_WORLD,&myrank);

MPI_Comm_size(MPI_COMM_WORLD, &p);

if(myrank==0)//I am the root{ double total=0,average=0; for(source=1;source<p;source++) { MPI_Recv(&randIn,1, MPI_DOUBLE, source, MPI_ANY_TAG, MPI_COMM_WORLD, &status);

printf("Message from root: From %d received number %f\n",source ,randIn);

total+=randIn; }//End for average=total/(p-1); }//End if

else//I am other than root

{

srand48((long int) myrank);

randOut=drand48();

printf("randout=%f, myrank=%d\n",randOut,myrank);

MPI_Send(&randOut,1,MPI_DOUBLE,dest,tag,MPI_COMM_WORLD);

}//End If-Else

MPI_Finalize();

return 0;

}

MPI References The Standard itself:

at http://www.mpi-forum.org All MPI official releases, in both postscript and HTML

Books: Using MPI: Portable Parallel Programming with the Message-

Passing Interface, 2nd Edition, by Gropp, Lusk, and Skjellum, MIT Press, 1999. Also Using MPI-2, w. R. Thakur

MPI: The Complete Reference, 2 vols, MIT Press, 1999. Designing and Building Parallel Programs, by Ian Foster,

Addison-Wesley, 1995. Parallel Programming with MPI, by Peter Pacheco, Morgan-

Kaufmann, 1997. Other information on Web:

at http://www.mcs.anl.gov/mpi For man pages of open MPI on the web

: http://www.open-mpi.org/doc/v1.4/ apropos mpi

THANK YOU

Education

MPI Introduction