Scientific discovery, analysis and prediction made possible through
high performance computing.
An Introduction to GPGPU Programming
Bob TorgersonArctic Region Supercomputing Center
March 14th, 2014
Introduction
Contents• What is GPU computing?• Brief History of GPGPU• Introduction to CUDA• CUDA API Basics• Advanced CUDA Concepts
What is GPU computing?• GPU computing is the use of the GPU together
with the CPU to accelerate general purpose applications– GPGPU (General Purpose Computing on GPUs)
• Offload the most computationally intense work to the GPU
• As a tag-team, the CPU and GPU work well together– CPU: Optimized for serial processes – SISD (MIMD)– GPU: Optimized for parallel processes – SIMD
AMD Opteron 2435 Specs (CPU)• 6 processor cores
– 12 virtual cores (hyperthreading)• 904 million transistors• ~100 GFLOPs• 768 KB L1 Cache• 3 MB L2 Cache• 6 MB L3 Cache
Nvidia Tesla x2090 Specs(GPU)
• 512 CUDA cores• 3 billion transistors• 1.33 TFLOPs (SP floating point)• 665 GFLOPs (DP floating point)• 6 GB on-board memory
– 177 GB/s memory bandwidth
Brief History of GPGPU• On October 11th, 1999, Nvidia creates the first
ever GPU– Offloaded the task of transformation & lighting
• In the early 2000’s, many started to notice the power of the GPU– Researchers started writing code in OpenGL and Cg
• Limited accessibility to general programmers and industry• Seeing a need, Nvidia made their GPUs fully
programmable– Offered the CUDA parallel programming model
• Works in a variety of languages, most notably C, C++ and Fortran
Introduction to CUDA• Compute Unified Device Architecture (CUDA)• With CUDA, an Nvidia GPU can be used for general
purpose processing– Only Nvidia GPUs are able to be used with CUDA
• Different versions of CUDA result in different API calls being available
• CUDA will work on all Nvidia GPUs from the G8x series onwards– Nvidia Tesla GPUs available on ARSC’s supercomputer Fish
compute nodes• CUDA works on all major operating systems
– Microsoft Windows, Mac OSX, and many variants of Linux
Introduction to CUDA (cont.)• To run CUDA at home, you can visit: https
://developer.nvidia.com/cuda-downloads– Download the CUDA release for your operating system
• Follow the instructions in the provided “Getting Started” guide
• Once you have installed the CUDA toolkit, you will have all of the necessary tools to compile and run CUDA on your system
• An important tool is “nvcc” which does the work of compiling your CUDA source code into a binary– CUDA source code is normally contained in a file with the
suffix .cu
CUDA API Basics• In the following section, I will be going through
some of the basic API calls available in CUDA• For those familiar with C/C++, these will seem
fairly natural to the language– With a few caveats, such as << >>
• Each new API call will give information about the API call and a small piece of example code to show how it could be used.
DISCLAIMER: While there are Fortran examples online for use with CUDA, I have neither tested nor tried any. All of the following works with C.
cudaMalloc• Similar to the malloc command for allocation of
memory on a server– Allocates a chunk of memory in the GPU’s available
memory• Can use a pointer to indicate the start of
available memory– float * or void*
• Called with two arguments:– A reference to a memory location (i.e. &var1)– Size of memory to allocate to the memory location
• Made easier with a function called sizeof()
cudaMallocconst int N = 20;size_t size = 30 * sizeof(float);float* d_A, d_B;void* d_C;
cudaMalloc(&d_A, (10 * sizeof(float)));cudaMalloc(&d_B, (N * sizeof(float)));cudaMalloc(&d_C, size);
cudaFree• cudaFree releases the memory that has been
allocated on the device– Identical to free() for C/C++ malloc()
• cudaFree and cudaMalloc behave differently depending on where they were executed– cudaFree run on the device cannot free device memory that
was allocated by the host– cudaMalloc run on the device will only be able to allocate
space up to the “cudaLimitMallocHeapSize”• Called with a single argument:
– Pointer to memory location on device
cudaFreefloat* d_A;cudaMalloc(&d_A, 30 * sizeof(float));...cudaFree(d_A);
cudaMemcpy• This function copies data between the host system and the
GPU device– It is required that the memory copy has a pre-allocated amount of space
available for the data to be copied• The function is used for copying data to and from the device
and also to copy on the device– cudaMemcpyHostToDevice– cudaMemcpyDeviceToHost– cudaMemcpyDeviceToDevice
• Called with 4 arguments:– A pointer to the memory that is being copied to– A pointer to the memory that is being copied from– The size of the data being transferred from the second arg. to the first
arg.– The direction to copy the data (host to device, device to host)
cudaMemcpycudaMemcpy(d_A, h_A, 30 * sizeof(float), cudaMemcpyHostToDevice);...cudaMemcpy(h_A, d_A, 30 * sizeof(float), cudaMemcpyDeviceToHost);
const int N = 50;size_t size = N * sizeof(float);cudaMemcpy(d_B, h_B, size, cudaMemcpyHostToDevice);...cudaMemcpy(h_B, d_B, size, cudaMemcpyDeviceToHost);
Example Code• What does this code do?• What would you expect the result to be from this
running on the GPU?
Kernels• When I think of kernels, I think of two things…
CUDA Kernels• A kernel is a function callable from the host system to the CUDA-
enabled device for being run on many threads in parallel– This allows for work to be performed on data that has been loaded onto the
memory of the GPU• CUDA expands the C language with a set of its own directives
for controlling the flow of execution• Kernels are defined using one of three prefixes:
_ _ host _ _ : Runs only on the host, can only be executed from the host_ _ device _ _ : Runs only on the device, can only be executed from device_ _ global _ _ : Runs only on the device, can only be executed from the host
• A limitation of CUDA kernels is that they can not be recursive (i.e. call themselves) and cannot have a variable number of arguments
CUDA Kernel Examples__device__ void increment_values(...) { ... }
__global__ float gpu_main(...) { ... }
__host__ int main(...) { ... }
CUDA Kernel Examples__host__ void incrementOnHost(float *host_a, int N) { for (int i = 0; i < N; i++) { host_a[i] = host_a[i] + 1.f; }}__global__ void incrementOnDevice(float *device_a, int N){ int idx = blockIdx.x * blockDim.x + threadIdx.x; if (idx < N) { device_a[idx] = device_a[idx] + 1.f; }}
CUDA Thread Indexing• You may have noticed the undefined syntax in the
previous example– i.e. threadIdx.x, blockDim.x, blockIdx.x
• CUDA has these built-in variables for the blocks of threads that are run against a kernel– Rather than performing a loop, we use the parallel nature of
the threads to perform the same work• For a better understanding of this concept, take a
look at the picture in the following slide
CUDA Thread Indexing• The first thing to understand is that for every kernel, a
“grid” is created when executing that kernel– A grid is a 3-D array of blocks
• A block is a 3-D array of threads – All of the threads within a block are able to communicate and
synchronize– Threads within a block share memory
• A thread is a single instance of a parallel process – To gain the true power of the GPU, hundreds of threads must be
executing in parallel– Due to hardware restrictions, the most threads possible per block is
512
CUDA Thread Indexing• Every CUDA thread has its own unique ID
– This can be determined in a straightforward way using its blockDim, blockIdx & threadIdx variables
• For a 1D block:int idx = blockIdx.x * blockDim.x + threadIdx.x;
• For a 2D block:int idx = blockIdx.x * blockDim.x * blockDim.y + threadIdx.y * blockDim.x + threadIdx.x;
• For a 3D block:int idx = blockIdx.x * blockDim.x * blockDim.y * blockDim.z + threadIdx.z * blockDim.x * blockDim.y + threadIdx.y * blockDim.x + threadIdx.x;
CUDA Thread Indexing• Modeling the block after the problem can result in
easier thread indexing– For example, a matrix can be indexed like this:int idx = blockIdx.x * blockDim.x + threadIdx.x;int idy = blockIdx.y * blockDim.y + threadIdx.y;
• These built-in variables can be used in a variety of ways to index your data
• The data is accessible by all threads– The programmer decides how best to access and
manipulate the data
CUDA Dim3 Variables• Another CUDA addition to the C language is the
dim3 type for a variable– dim3 provides a way of defining dimensions that a grid of
blocks or a block of threads can have• These provide the ability for unsigned integers to
be used as the limits to these dimensions– As their name implies, they are capable of being 3-D
definitions to match with thread structure within a block• dim3 variables are defined using parentheses
to indicate the dimensions– dim3 <varname>(<dim1>,<dim2>,<dim3>);
CUDA Dim3 Exampleint M = 4;int N = 8;dim3 blocks_per_grid(M,M);dim3 threads_per_block(N,N);
Running a CUDA Kernel• To run a CUDA kernel, an extension to the C
language has been addedfunction_name<<<dimGrid, dimBlock>>>(args);
• CUDA kernels run asynchronously– You can continue running sequential code on the CPU while
the parallel work is being done on the GPU• Calls to the function cudaMemcpy() block the
execution of the next lines of code– All threads running on the GPU are synchronized before
they are returned by cudaMemcpy
Example Code• What does this code do?• What would you expect the result to be from this
running on the GPU?
The More You Know…• Now you know everything you need to make a
working CUDA program– “Know enough to be dangerous…”
• Basics are easy just like in every parallel programming extension– Doing things right takes practice
• May not be obvious changes, requires optimization
• Understanding when a code should be written for the GPU– Lots of data to compute over (same data is even better)– Using the same commands regardless of input– Limited branching (or branching in an expected way)
CUDA Warps• A warp may sound like something out of Star Trek
– A weaving term used to describe threads arranged lengthwise on a loom and crossed by the “woof”
• In CUDA, the hardware is designed to execute in groups of 32 threads– This is known as a warp
• The smallest amount of threads that can be executed
• Naturally, 32 threads of parallel work on data is hardly working the GPU to its fullest– The GPU takes the input of blocks and breaks them down into warps to be
executed on the GPU• Can be run on old, new, or future Nvidia GPUs due to the abstraction in code for the SMs
• Conditional branching done based on warps can be much more efficient– Conditionals can have a profound effect on the runtime of kernels
Nvidia GPU Architecture
CUDA Memory• Threads within the same block CAN communicate with
each other during execution– This is due to a shared 48 KB memory block inside a streaming
multiprocessor• Threads outside of the same block must write back their
results before they are accessible by other blocks– Makes memory management more complicated
• All of the threads in a block are guaranteed to run on the same SM– Uses the same shared memory block
• Similar to cache in CPUs, this shared memory block is much faster than reading from the GPU memory– Very nearly as fast as reading / writing to a GPU register
CUDA Memory
CUDA Limitations• An SM has 32,768 registers shared amongst all threads
– All blocks using that SM are limited by this value• Choice of SM is done by the hardware, not by the programmer
• The number of active blocks on an SM can not exceed 8• The number of active warps on an SM can not exceed
24– Meaning only 1,024 threads per SM maximum
• 16,384 threads can be executing on all SMs at a time
• Optimizing a CUDA program– Finding a balance between number of blocks and their size
• A block of 768 threads would be very inefficient since only one block could be running on an SM (1024 – 768 = 256 threads idle)
• Nvidia recommends running between 128 and 256 threads per block
CUDA Shared Memory• The shared memory available to all threads in a
block is managed by the programmer– The CUDA software does not make use of this memory
unless requested• Efficient use of the shared memory in blocks
contributes to faster execution of code– Reads / writes to global device memory can be 100-150
times slower than shared memory accesses• Takes 4 clock cycles to read from shared memory• Takes 400 clock cycles to read from global memory!
• Registers >(=) Shared > Global
CUDA Shared Memory• For a kernel to use shared memory, it must first
declare an amount of shared memory to allocate– A third optional argument to the CUDA kernel execution
function_name<<<numBlocks, numDims, sharedMemSize>>>(args)
• To use the shared memory, it is easiest to let the memory be dynamically allocatedextern __shared__ float* shared_data;
• This will allocate the full size of the shared memory to this variable
• To have more than one array of data allocated in shared memoryextern __shared__ float* shared;float* a = &shared[0];float* b = &shared[count_a];
CUDA Shared Memory Example__global__ void testfunc(int count_a) { extern __shared__ float* shared; float* a = &shared[0]; float* b = &shared[count_a]; ...}
int problemSize = 256 * 2048;int numThreadsPerBlock = 256;int numBlocks = problemSize / numThreadsPerBlock;int sharedMemSize = numThreadsPerBlock * sizeof(float);int count_a = 64;testfunc<<<numBlocks,numThreadsPerBlock,sharedMemSize>>>(count_a);
Synchronize Threads• Before attempting to write out data to global
memory, you must synchronize the threads– You run the risk of trying to pull from memory for data that
has not been written yet• Race conditions…
__syncthreads();• This is run from inside a kernel to block until all threads in a block reach
this point
• For blocking until all of the threads in a grid have finishedcudaThreadSynchronize();
• This must be run from the host
Example Code• What does this code do?• What would you expect the result to be from this
running on the GPU?
CUDA Errors• Detecting and handling errors is essential to creating
robust and usable software– No one wants to use code that fails with no way to determine why
• CUDA provides error codes specific to particular problems encountered– These error codes can be converted into a string of characters to be
displayed• CUDA error codes have their own type: cudaError_t
char* cudaGetErrorString(cudaError_t code);• Provides a human-readable description of the error code
• A convenient command to get the most recent CUDA error cudaGetLastError();
• Useful if done after a blocking call since it will get the latest error at the end of a kernel execution for example
CUDA Error Examplevoid checkCUDAError(const char *msg) { cudaError_t err = cudaGetLastError(); if ( cudaSuccess != err) { fprintf(stderr, “CUDA Error: %s: %s.\n”, msg, cudaGetErrorString(err)); exit(EXIT_FAILURE); }}
Example Code• What does this code do?• What would you expect the result to be from this
running on the GPU?
Conclusion• CUDA is only one example of how to write code for
the GPU– OpenCL, Microsoft’s DirectCompute, and C++ AMP
• CUDA attempts to make programming for the GPU “easy” by providing a familiar code structure– Easier than passing textures in OpenGL
• Further work is being done to make running code on the GPU truly easy– OpenACC directives or compilation by LLVM
• Give GPU programming a try and have fun!