Supporting Efficient Execution in Heterogeneous Distributed Computing Environments with Cactus and Globus

Gabrielle Allen, Thomas Dramlitsch, Ian Foster, Nick Karonis, Matei Ripeanu, Ed Seidel, Brian Toonen

Proceedings of Supercomputing 2001 (Winning Paper for Gordon Bell Prize - Special Category)

Presenter

Imran Patel

Outline

Introduction Computational Grids Grid-enabled Cactus Toolkit Experimental Results Ghostzones and Compression Adaptive Strategies Conclusion

Introduction

Widespread use of numerical simulation techniques has led to a high demand for traditional high-performance computing resources (supercomputers).

Low-end computers are becoming increasingly powerful and are connected with high-speed networks.

“Computational Grids” aim to tie these scattered resources into an integrated infrastructure.

Applications for the grid include large-scale simulations which require high resources for increased throughput.

Introduction: The Problem

Heterogeneous and dynamically behaving resources makes development of grid-enabled applications extremely difficult.

One approach is to develop computational frameworks which hide this complexity from the programmer.

Cactus-G: a simulation framework which uses grid-aware components and message passing library (MPICH-G2)

Computational Grids

Computational Grids differ from other parallel computing environments:

A grid may have nodes with different processor speeds, memory, etc

Grids may have widely different network interconnects and topologies.

Resource availability varies in a grid. Nodes in a grid may have different software

configurations.

Computational Grids – Programming Techniques

To overcome these problems, some generic techniques have been devised:

Irregular Data Distributions: use application/network/node information.

Grid-aware Communication schedules: overlapping/grouping, dedicated nodes

Redundant Computation: at the expense of reduced communication.

Protocol Tuning: TCP tweaks,compression

Cactus-G Cactus is a modular and parallel simulation

environment used by scientists and engineers in the fields of numerical relativity, astrophysics, climate modeling, etc.

Cactus design consists of a core (flesh) which connects to application modules (thorns).

Various thorns exist for services such as Globus Toolkit, PETSc library, visualization, etc.

Cactus is highly portable and parallel since it uses abstraction APIs which themselves are implemented as thorns.

MPICH-G2 exploits Globus services and provides faster communication and QOS.

Cactus-G: Architecture Applications thorns

need not be grid-aware.

Example of a grid-aware Cactus thorn is PUGH, which provides MPI-based parallelism.

The DUROC library handles process management.

Experimental Results: Setup

An application in Fortran for solving numerical relativity problems

57 3-d variables, 780 flops per gridpoint per iteration.

N x N x 6 x ghostzone_size x 8 variables need to be sync’ed at each processor.

Total 1500 CPUs organized in a 5 x 12 x 25 3-d mesh

Experimental Results: Setup

4 supercomputers at SDSC and NCSA 1024-CPU IBM Power-SP (306

MFlops/s). One 256 CPU and two 128-CPU SGI

Origin2000 systems. (168 MFlops/s). Intramachine: 200 MB/s Intermachine: 100 MB/s SDSC<->NCSA: 3 MB/s on 622 Mb/s

Communication Optimizations

Communication/Computation Overlap: Processors across WANs were given few grid points so that they could overlap their communication with computations.

Compression: A cactus thorn which exploits the regularity of data for compression using the libz library.

Ghostzones: Larger ghostzones were used for efficient communication at the expense of redundant computations.

Performance Metrics

Flop/s rate and efficiency are used as metrics. Total execution time (ttot): Measured using

MPI_Wtime() Expected Communication Time (tcomp): Ideal time

calculated from single node. Flop Count F: Calculated using hardware counters

780 Flops per gridpoint per iteration. Flop/s Rate =

F * num_gridpts * num_iterations / ttot

E = tcomp/ttot

Performance Figures

4 supercomputers: 42 GFlop/s, 14% Compression + 10 Ghostzones: 249

GFlop/s, 63.3% Smaller run on 120+1120=1140

processors: 292 GFlop/s, 88%

Ghostzones

Increasing ghostzone size can reduce latency overhead by transferring fewer messages with same amount of total data.

Increasing the ghostzone size beyond a certain point does not give any benefits and wastes memory.

Compression

For increased throughput across WANs.

Compression found to be highly useful since data is regular/smooth.

Since smoothness of data changes over time, compression effects can change. So, we need adaptive compression.

Adaptive Strategies - Compression

Predicting optimal values of ghostzone/compression parameters might be good.

Don’t want to use detailed network characteristics.

For ex: change the compression state based on efficiency, which is averaged over N iterations.

Adaptive Ghostzone Sizes

Adapting Ghostzone size is challenging: Many ghostsize values possible, memory re-allocations, ripple effects, need to get extra data from neighbors.

Start with a size of 1 and increase/decrease in accordance with the efficiency.

Adaptive Ghostzone Sizes

Further Information

The Cactus Framework and Toolkit: Design and Applications Tom Goodale, et al.

Vector and Parallel Processing - VECPAR'2002

Grid Aware Parallelizing Algorithms Thomas Dramlitsch, Gabrielle Allen, Edward Seidel

Journal of Parallel and Distributed Computing