CuMAPz: A Tool to Analyze Memory Access Patterns in CUDA

Yooseong Kim and Aviral ShrivastavaCompiler and Microarchitecture Laboratory, Arizona State University

DAC2011

Outline

Introduction Preliminaries Motivating examples CuMAPz approach Experimental results and Conclusions

Introduction

Currently, the computational power of Graphics Processing Units (GPUs) has reached teraFLOP scale.

NVIDIA CUDA and OpenCL make GPGPU (General Purpose computation on GPUs) programming more easier.

The performance will be heavily affected by memory performance for the sake of large data size.

Introduction (cont.)

Shared memory is as fast as registers, and is the only fast memory where both reads and writes are enabled.

Many factors affect performance: data reuse, global memory access coalescing, shared memory, bank conflict, channel skew.

Develops CuMAPz (CUDA Memory Access Pattern analyZer) to analyze the memory performance of CUDA program.

Preliminaries

NVIDIA GPU architecture. Comparisons between CPU and GPU. CUDA programming. Memory coalescing. Execution of GPU thread

Architecture of Nvidia GTX280

A collection of 30 multiprocessors, with 8 streaming processors each.

The 30 multiprocessors share one off-chip global memory. Access time: about 300 clock cycles

Each multiprocessor has a on-chip memory shared by that 8 streaming processors. Access time: 2 clock cycles

Architecture diagram

About some differences between GPU and CPU

GPU (NVIDIA GeForce 8800 GTX)

CPU (Intel Pentium 4)

Cores and clock rate

128 / 575MHz (core clock), 1.35GHz (shader clock)

1 / 3.0GHz

flops 345.6G ~12G

Memory bandwidth

86.4GB/s (900MHz memory clock, 384 bit interface, 2 issues)

6.4GB/s (800MHz memory clock, 32 bit interface, 2 issues)

Access time of global memory

Slow (about 500 memory clock cycles)

Fast (about 5 memory clock cycles)

Abstract comparisons of memory between GPU and CPU (cont.)

CPU (Intel Pentium 4)GPU (NVIDIA GeForce 8800 GTX)

Register Register

Cache Texture cache or Constant cache

Main memory Shared memory

Hard disk Global memory, Texture memory, Constant memory

Memory coalescing

Several memory transactions can be coalesced into one transaction when consecutive threads access consecutive memory locations.

Due to access time of global memory is relatively large, it is important to achieve this.

CUDA programming

Compute Unified Device Architecture The CPU code does the sequential

part. Highly parallelized part usually

implement in the GPU code, called kernel.

Calling GPU function in CPU code is called kernel launch.

Execution of GPU thread

Threads are grouped into thread blocks. Each thread block is assigned to a streamin

g multiprocessors (SMs), which contains multiple scalar processors (SPs), to be executed.

The actual execution of threads on SPs is done in groups of 32 threads, called warps.

SPs execute one warp at a time.

Motivating examples

What to fetch into shared memory? How to access shared memory? How to access global memory?

What to fetch into shared memory?

A simple program that does not use shared memory.

What to fetch into shared memory? (cont.)

If we fetch row*MAX+col+1 to the shared memory…

What to fetch into shared memory? (cont.)

Generally, higher data reuse should imply better performance. => But may not be true here.

This counter-intuitive result is mainly caused by global memory access coalescing.

How to access shared memory?

In Figure 2, data is accessed in a column-wise manner, as shown at Line 4, 9, 11, and 16.

What if we change into row-wise manner (i.e. s_in[tIdx.y][tIdx.x]) or skewing the access pattern (i.e. __shared__ float s_in[BLKDIM][BLKDIM+1])?

How to access shared memory? (cont.)

Shared memory bank conflicts occur if there are multiple requests to different addresses in the same bank. In this case, the requests are serialized.

How to access global memory? A programmer might have designed the global memory

write reference at Line 18 in Figure 2 to be in a column-wise manner as in out[col*MAX+row].

This unexpected slowdown is caused by channel skew. Channel skew is the ratio of the number of concurrent accesses to the most used channel to theleast used channel.

Previous works

[20] modeled the amount of parallelism employed in a program and the efficiency of a single kernel execution in a thread. Did not consider memory performance and their analysis

is only for compute intensive benchmarks. [8] includes the effect of parallelism to hide global

memory access latency. Does not take into account branch divergence.

[14][15][16][17][18] automate optimization of GPGPU applications. None of the above work comes up with a comprehensive

performance metric to estimate the efficiency of memory access pattern.

CuMAPz overview

Data Reuse Profit Estimation

CuMAPz maintains a counter to count the number of times shared memory buffers are accessed. The degree of data reuse is represented in a term, data reuse, as follows:

Coalesced Access Profit Estimation

Due to coalescing, the actual transfer size that will consume bus width can be different from the size of data requested from threads. CuMAPz calculates the bandwidth utilization as the following:

Channel Skew Cost Estimation

Channel skew refers to the case where the concurrent memory accesses are not evenly distributed to all the channels but focused on only a few channels.

When a kernel is launched, threads blocks are assigned to SMs in a sequential order so that adjacent blocks are executed on adjacent SMs. Then, it becomes unpredictable after the first round of schedule since the order in which thread blocks finish the execution cannot be determined [13].

Channel Skew Cost Estimation (cont.)

The impact of channel skew can be stated in figures as the skewness of mapping to channels which can be calculated as follows:

Bank Conflict Cost Estimation

Similarly to global memory channels, shared memory space is divided into multiple banks. Each bank can serve one address at a time.

Efficiency of shared memory access is modeled as follows:

Branch Divergence Cost Estimation

Branches are introduced when there is uncovered region that is not buffered into shared memory, as shown at Line 6 and 13 in Figure 2.

When threads in a warp take different execution paths, then all paths are serialized.

We simply model the impact of branch divergence as follows:

Overall Memory Performance Estimation

Memory performance estimation is calculated by the following formula.

Experimental results

Environments Using C language. CUDA driver version 3.2 on NVIDIA Tesla

C1060. Benchmark are from benchmark suites in

[6], and CUDA SDK.

Two experiments

Validation: studying the correlation between our memory performance estimation and the performance of the benchmarks for different ways.

Performance Optimization: trying to find the best way to accesses shared and global memory using CuMAPz and the previous technique [8].

Validation

Performance optimization

Runtime Considerations

The timing complexity of the CuMAPz analysis is O(|W|*|R|*|B|), where W, R, and B are the set of all warps, global memory references, and shared memory buffers respectively.

Limitations

Compile-time analysis Cannot handle any information that can

only be determined during run-time. Assume adequate occupancy

A measure of how many thread blocks can be scheduled on one SM so that the hardware is kept busy.

Conclusions

GPU is a new platform for high-performance computing.

Develops CuMAPz to analyze memory performance of CUDA.

Considering many aspects like channel skew, etc.

Experimental results show very high correlation between the actual execution times and CuMAPz estimation.