mpiP Evaluation Report

Hans Sherburne,Adam LekoUPC Group

HCS Research LaboratoryUniversity of Florida

2

Basic Information Name:

mpiP Developer:

Jeffrey Vetter (ORNL), Chris Chambreau (LLNL) Current versions:

mpiP v2.8 Website:

http://www.llnl.gov/CASC/mpip/ Contacts:

Jeffrey Vetter: vetterjs@ornl.gov Chris Chambreau: chcham@llnl.gov

3

mpiP: Lightweight, Scalable MPI Profiling mpiP is a simple lightweight tool for profiling Gathers information through the MPI profiling layer

Probably not good candidate to be extended for UPC or SHMEM Supports many platforms running Linux, Tru64, AIX, UNICOS, IBM BG/L Very simple to use, and output file is very easy to understand

Provides statistics for the top twenty MPI calls based on time spent in call, and total size of messages sent, also provides statistics for MPI I/O

Callsite traceback depth is variable to allow user to differentiate between and examine the behavior of routines that are wrappers for MPI calls

A mpip viewer, Mpipview, is available as part of Tool Gear Some of its functionality is extended to developers through an API:

stackwalking, address-to-source translation, symbol demangling, timing routines, accessing the name of the executable These functions might be useful is source-code correlation is to be

included in a UPC or SHMEM tool

4

What is mpiP Useful For? The data collect by mpiP is useful for analyzing the scalability of parallel applications. By examining the aggregate time and rank correlation of the time spent in each MPI call versus the total

time spent in MPI calls while increasing the number of tasks, one can locate flaws in load balancing and algorithm design.

This technique is described in [1] “Statistical Scalability Analysis of Communication Operations in Distributed Applications” –Vetter, J. & McCracken, M.

The following are courtesy of [1]:

5

The Downside… mpiP does provide the measurements of aggregate callsite time, and total

MPI call time necessary for computing the rank correlation coefficient mpiP does NOT automate the process of computing the rank correlation,

which must utilize data from multiple experiments

Equations for calculation of coefficients of correlation (linear and rank), care of [1]:

6

Partial Sample of mpiP Output

7

Information Provided by mpiP Information displayed in terms of task assignments, and callsites, which correspond

to machines and MPI calls in source code arranged in the following sections: Time per task

(AppTime, MPITime, MPI%) Location of callsite in source code

(callsite, line#, parent function, MPI call) Aggregate time per callsite (top twenty)

(time, app%, MPI%, variance) Aggregate sent message size per callsite (top twenty)

(count, total, avg. MPI%) Time statistics per callsite per task (all)

(max, min, mean, app%, MPI%) Sent message size statistics per callsite per task (all)

(count, max, min, mean, sum) I/O statistics per callsite per task (all)

(count, max, min, mean, sum)

8

mpiP Overhead

mpiP Profiling Overhead

5%

3%

2%

3%

1%

1%

0%

7%

5%

0%

1%

0%

0% 1% 2% 3% 4% 5% 6% 7% 8%

CAMEL

NAS LU (8p, W)

NAS LU (32p, B)

PP: Big message

PP: Diffuse procedure

PP: Hot procedure

PP: Intensive server

PP: Ping pong

PP: Random barrier

PP: Small messages

PP: System time

PP: Wrong way

Ben

chm

ark

Overhead (instrumented/uninstrumented)

9

Source Code Correlation in Mpipview

10

Bottleneck Identification Test Suite Testing metric: what did profile data tell us? CAMEL: TOSS-UP

Profile showed that MPI time is a small percentage of overall application time Profile reveals some imbalance in the amount of time spent in certain calls, but

doesn’t help the user understand the cause Profile does not provide information about what occurs when execution is not in

MPI calls. Difficult to grasp overall program behavior from profiling information alone

NAS LU: TOSS-UP Profile reveals that a MPI function calls consume a significant portion of

application time Profile reveals some imbalance in the amount of time spent in certain calls, but

doesn’t help the user understand the cause Profile does not provide information about what occurs when execution is not in

MPI calls. Difficult to grasp overall program behavior from profiling information alone

11

Bottleneck Identification Test Suite (2) Big message: PASSED

Profile clearly shows that Send and Recv dominate the application time

Profiles shows a large number of bytes transfered

Diffuse procedure: FAIL Profile showed large amount of time spent in

barrier Time is diffused across processes Profile does not show that in each barrier a

single process is always delaying completion Hot procedure: FAIL

No profile output, due to no MPI calls (other than setup and breakdown

Intensive server: PASSED Profile showed one process spent very little

time in MPI calls, while the remaining processes spent nearly all their time in Recvs

Profile showed one process sent an order of magnitude more data than the others, and spent far more time in Send

Ping pong: PASSED Profile showed time spent in MPI function

calls dominated the total application time Profile showed excessive number of Sends

and Recvs with little load imbalance Random barrier: PASS

Profile shows that the majority of execution time is spent in Barrier called by processes not holding “potato”

Small messages: PASS Profile clearly shows single process spends

almost all of the total application time in Recv, and recvs an excessive amount of messages sent by all the other processes

System time: FAIL No profile output, due to no MPI calls (other

than setup and breakdown Wrong way: TOSS-UP

One process spends most of the execution time in sends the other spends most of the execution time in receives

Profile does not reveal the improperly ordered communication pattern

12

Evaluation (1) Available metrics: 1/5

Only provides a handful statistical information about time, message size, and frequency of MPI calls

No hardware counter support Cost: free 5/5 Documentation quality: 4/5

Though brief (a single webpage), documentation adequately covers installation and available functionality

Extensibility: 2/5 mpiP is designed to use the MPI profiling layer so they would not be readily adapted to UPC

or SHMEM and so it would be of little use The source code correlation functions work well

Filtering and aggregation: 2/5 mpiP was designed to be lightweight, and presents statistics for the top twenty callsites Output size grows with number of tasks (machines)

Hardware support: 5/5 64-bit Linux (Itanium and Opteron), IBM SP (AIX), AlphaServer (Tru64), Cray X1, Cray XD1,

SGI Altix, IBM BlueGene/L Heterogeneity support: 0/5 (not supported)

13

Evaluation (2) Installation: 5/5

About as easy as you could expect Interoperability: 1/5

mpiP has it’s own output format Learning curve: 4/5

Easy to use Simple statistics are easily understood

Manual overhead: 1/5 All MPI calls automatically instrumented for you when linking against mpiP library No way to turn on/off tracing in places without relinking

Measurement accuracy: 4/5 CAMEL overhead: ~5% Correctness of programs is not affected Overhead is low (less than 7% for all test suite programs)

14

Evaluation (3) Multiple executions: 0/5 (not supported) Multiple analyses & views: 2/5

Statistics regarding MPI calls are displayed in output file Source code location to callsite correlation provided by Mpipview

Performance bottleneck identification: 2.5/5 No automatic methods supported Some bottlenecks could be deduced by examining gathered statistics Lack of trace information makes some bottlenecks impossible to detect

Profiling/tracing support: 2/5 Only supports profiling Profiling can be enabled for various regions of code by editing source code Turning on/off profiling requires recompilation

(a runtime environment variable for deactivating profiling is given in documentation, and acknowledged in the profile output file when set, but profiling is not disabled)

15

Evaluation (4) Response time: 3/5

No results until after run Quickly assembles report at the end of experimentation run

Searching: 0/5 (not supported) Software support: 3/5

Supports C, C++, Fortran Supports a large number of compilers Tied closely to MPI applications

Source code correlation: 4/5 Line numbers of source code provided for each MPI callsites in output file Automatic source code correlation provided by Mpipview

System stability: 5/5 mpiP and Mpipview work very reliably

Technical support: 5/5 Co-author, Chris Chambreau, responded quickly, and provided good information

allowing us to correct a problem with one of our benchmark apps