Upload
mikayla-valenzuela
View
164
Download
11
Tags:
Embed Size (px)
DESCRIPTION
Introduction to OpenMP. OpenMP Introduction. Credits: www.sdsc.edu/~allans/cs260/lectures/OpenMP.ppt www.mgnet.org/~douglas/Classes/cs521-s02/... openmp /MPI- OpenMP . ppt OpenMP Homepage: http://www.openmp.org/. Module Objectives. Introduction to the OpenMP standard - PowerPoint PPT Presentation
Citation preview
Introduction to OpenMPIntroduction to OpenMP
OpenMP IntroductionOpenMP Introduction
Credits:www.sdsc.edu/~allans/cs260/lectures/OpenMP.ppt
www.mgnet.org/~douglas/Classes/cs521-s02/...openmp/MPI-OpenMP.ppt
OpenMP Homepage: http://www.openmp.org/
Module ObjectivesModule Objectives
• Introduction to the OpenMP standard
• After completion, users should be equipped to implement OpenMP constructs in their applications and realize performance improvements on shared memory machines
DefinitionDefinition
Parallel Computing:
• Computing multiple things simultaneously.
• Usually means computing different parts of the same problem simultaneously.
• In scientific computing, it often means decomposing a domain into more than one sub-domain and computing a solution on each sub-domain separately and simultaneously (or almost separately and simultaneously).
• Perfect (a.k.a Embarrassing, Trivial) Parallelism • Monte-Carlo Methods• Cellular Automata
• Data Parallelism• Domain Decomposition• Dense Matrix Multiplication
• Task Parallelism• Pipelining• Monte-Carlo?• Cellular Automata?
Types of ParallelismTypes of Parallelism
Performance MeasuresPerformance Measures
• Peak Performance: Theoretical upper bound on performance.
• Sustained Performance: Highest consistently achievable speed.
• MHz: Million cycles per second. • MIPS: Million instructions per second.• Mflops: Million floating point operations per second. • Speedup: Sequential run time divided by parallel run time.
Parallelism IssuesParallelism Issues
• Programming notation
• Algorithms and Data Structures
• Load Balancing
• Problem Size
• Communication
• Portability
• Scalability
Getting your feet wetGetting your feet wet
Memory TypesMemory Types
CPU
Memory
CPU
Memory
CPU
Memory
CPU
Memory Memory
CPU CPU
CPUCPU
Distributed
Shared
Clustered SMPsClustered SMPssymmetric multiprocessors
Cluster Interconnect Network
Memory Memory Memory
Multi-socket and/or Multi-core
Distributed vs. Shared MemoryDistributed vs. Shared Memory
• Shared - all processors share a global pool of memory– simpler to program– bus contention leads to poor scalability
• Distributed - each processor physically has its own (private) memory– scales well– memory management is more difficult
What is OpenMP?What is OpenMP?
• OpenMP is a portable, multiprocessing API for shared memory computers
• OpenMP is not a “language”
• Instead, OpenMP specifies a notation as part of an existing language (FORTRAN, C) for parallel programming on a shared memory machine
• Portable across different architectures
• Scalable as more processors are added
• Easy to convert sequential code to parallel
Why should I use OpenMP?Why should I use OpenMP?
Where should I use OpenMP?Where should I use OpenMP?
OpenMP SpecificationOpenMP Specification
OpenMp consists of three main parts:
• Compiler directives used by the programmer to communicate with the compiler• Runtime library which enables the setting and querying of parallel parameters
• Environmental variables that can be used to define a limited number of runtime parameters
OpenMP Example UsageOpenMP Example Usage (1 of 2)
OpenMPCompiler
AnnotatedSource
SequentialProgram
ParallelProgram
compiler switch
OpenMP Example UsageOpenMP Example Usage (2 of 2)
• If you give sequential switch,– comments and pragmas are ignored.
• If you give parallel switch,– comments and/or pragmas are read, and– cause translation into parallel program.
• Ideally, one source for both sequential and parallel program (big maintenance plus).
Simple OpenMP ProgramSimple OpenMP Program
• Most OpenMP constructs are compiler directives or pragmas• The focus of OpenMP is to parallelize loops• OpenMP offers an incremental approach to parallelism
OpenMP Programming ModelOpenMP Programming Model
OpenMP is a shared memory model.• Workload is distributed among threads
– Variables can be• shared among all threads• duplicated and private to each thread
– Threads communicate by sharing variables• Unintended sharing of data can lead to race conditions:
– race condition: when the program’s outcome changes as the threads are scheduled differently.• To control race conditions:
– Use synchronization (Chapter Four) to protect data conflicts.• Careless use of synchronization can lead to deadlocks (Chapter Four)
OpenMP Execution ModelOpenMP Execution Model
• Fork-join model of parallel execution
• Begin execution as a single process (master thread)• Start a parallel construct:
Master thread creates a team of threads
• Complete a parallel construct:Threads in the team wait until all team work has been completed
• Only master thread continues execution
The Basic IdeaThe Basic Idea
OpenMP directive format in COpenMP directive format in C#pragma directives, defined by C standard as a mechanism to do
compiler-specific tasks e.g. ignore errors, generate special code#pragma must be ignored if not understood; thus, SOME OpenMP
programs can be compiled for sequential OR parallel executionTypically, OpenMP directives can be enabled by compiler option
• OpenMP pragma Usage:#pragma omp directive_name [ clause [ clause ] ... ] CR• Conditional compilation#ifdef _OPENMP
printf(“%d avail.processors\n”,omp_get_num_procs());
#endif• case sensitive• Include file for library routines:
#ifdef _OPENMP#include <omp.h>#endif
Microsoft Visual Studio OpenMP OptionMicrosoft Visual Studio OpenMP Option
Other CompilersOther Compilers
• Intel (icc, ifort, icpc)– -openmp
• PGI (pgcc, pgf90, …)– -mp
• GNU (gcc, gfortran, g++)– -fopenmp– need version 4.2 or later
OpenMP parallel region OpenMP parallel region constructconstruct
Block of code to be executed by multiple threads in parallel
Each thread executes the same code redundantly!• C/C++:#pragma omp parallel [ clause [ clause ] ... ] CR{
structured-block}• clause can be either or both of the following: private(comma-separated identifier-list) shared(comma-separated identifier-list)• If no private/shared list, shared is assumed for all
variables
OpenMP parallel region OpenMP parallel region constructconstruct
Communicating Among ThreadsCommunicating Among Threads
• Shared Memory Model– threads read and write shared variables
• no need for explicit message passing
– change storage attribute to private to minimize synchronization and improve cache reuse because private variables are duplicated in every team member
Storage Model – Data ScopingStorage Model – Data Scoping
• Shared memory programming model: variables are shared by default
• Global variables are SHARED among threads– C: file scope variables, static
• Private Variables:– exist only within the scope of each thread, i.e. they are
uninitialized and undefined outside the data scope– loop index variables– Stack variables in sub-programs called from parallel regions
OpenMP -- exampleOpenMP -- example
#include <stdio.h>
int main() {
// Do this part in parallel printf( "Hello, World!\n" );
return 0; }
OpenMP -- exampleOpenMP -- example
#include <stdio.h>#include <omp.h>
int main() {
omp_set_num_threads(16);
// Do this part in parallel #pragma omp parallel { printf( "Hello, World!\n" ); }
return 0; }
OpenMP environment variablesOpenMP environment variables
OMP_NUM_THREADS– sets the number of threads to use during execution– when dynamic adjustment of the number of threads is enabled,the value of this environment variable is the maximum numberof threads to use
setenv OMP_NUM_THREADS 16 [csh, tcsh] export OMP_NUM_THREADS=16 [sh, ksh, bash]
At runtime, omp_set_num_threads(6)
OpenMP runtime libraryOpenMP runtime library
omp_get_num_threads FunctionReturns the number of threads currently in the team executing the parallel region
from which it is called– C/C++:int omp_get_num_threads(void);
omp_get_thread_num FunctionReturns the thread number, within the team, that lies between 0 and
omp_get_num_threads()-1, inclusive. The master thread of the team is thread 0
– C/C++:int omp_get_thread_num(void);
Hello…WorldWorldWorldWorld!
Programming Model - Fork/JoinProgramming Model - Fork/Join
int main() {
// serial region
printf(“Hello…”);
// serial again
printf(“!”);
}
Fork
Join
// parallel region
#pragma omp parallel
{
printf(“World”);
}
Programming Model – Thread Programming Model – Thread IdentificationIdentification
Master Thread• Thread with ID=0• Only thread that exists in
sequential regions• Depending on
implementation, may have special purpose inside parallel regions
• Some special directives affect only the master thread (like master)
• Other threads in a team have ids 1..N-1
Fork
Join
0
0 1 2 3 4 5 6 7
0
Run-time Library: TimingRun-time Library: Timing
• There are 2 portable timing routines• omp_get_wtime
– portable wall clock timer returns a double precision value that is number of elapsed seconds from some point in the past
– gives time per thread - possibly not globally consistent– difference 2 times to get elapsed time in code
• omp_get_wtick– time between ticks in seconds
Loop ConstructsLoop Constructs
• Because the use of parallel followed by a loop construct is so common, this shorthand notation is often used (note: directive should be followed immediately by the loop)– #pragma parallel for [ clause [ clause ] ... ] CR
for ( ; ; ) { }
• Subsets of iterations are assigned to each thread in the team
Programming Model – Programming Model – Concurrent LoopsConcurrent Loops
• OpenMP easily parallelizes loops– No data dependencies between
iterations!
• Preprocessor calculates loop bounds for each thread directly from serial source ?
?
for( i=0; i < 25; i++ ) {
printf(“Foo”);
}
#pragma omp parallel for
Sequential Matrix MultiplySequential Matrix Multiply
for( i=0; i<n; i++ )for( j=0; j<n; j++ ) {
c[i][j] = 0.0;for( k=0; k<n; k++ )c[i][j] += a[i][k]*b[k][j];
}
OpenMP Matrix MultiplyOpenMP Matrix Multiply
#pragma omp parallel forfor( i=0; i<n; i++ )
for( j=0; j<n; j++ ) {c[i][j] = 0.0;for( k=0; k<n; k++ )
c[i][j] += a[i][k]*b[k][j];}
OpenMP parallel for directiveOpenMP parallel for directiveclause can be one of the following: private( list ) shared( list ) default( none | shared | private ) if (Boolean expression) reduction( operator: list) schedule( type [ , chunk ] ) nowait num_threads(N)
• Implicit barrier at the end of for unless nowait is specified• If nowait is specified, threads do not synchronize at the end of the parallel loop
schedule clause specifies how iterations of the loop are divided among the threads of the team.– Default is implementation dependent
OpenMP parallel/for directiveOpenMP parallel/for directive
#pragma omp parallel private(f)
{
f=7;
#pragma omp for
for (i=0; i<20; i++)
a[i] = b[i] + f * (i+1);
} /* omp end parallel */
// i is private
// a, b are shared
Default ClauseDefault Clause
• Note that the default storage attribute is DEFAULT (SHARED)
• To change default: DEFAULT(PRIVATE)– each variable in static extent of the parallel region is
made private as if specified by a private clause
– mostly saves typing
• DEFAULT(none): no default; must list storage attribute for each variable
USE THIS!
If ClauseIf Clause
• if (Boolean expression)• executes (in parallel) normally if the
expression is true, otherwise it executes the parallel region serially
• Used to test if there is sufficient work to justify the overhead in creating and terminating a parallel region
Conditional Parallelism: Conditional Parallelism: ExampleExample
for( i=0; i<n; i++ )
#pragma omp parallel for if( n-i > 100 )
for( j=i+1; j<n; j++ )
for( k=i+1; k<n; k++ )
a[j][k] = a[j][k] - a[i][k]*a[i][j] / a[j][j]
Data modelData model
• Private and shared variables
•Variables in the global data space are accessed by all parallel threads (shared variables).
• Variables in a thread’s private space can only be accessed by the thread (private variables)
• several variations, depending on the initial values and whether the results are copied outside the region.
#pragma omp parallel for private( privIndx, privDbl ) for ( i = 0; i < arraySize; i++ ) { for ( privIndx = 0; privIndx < 16; privIndx++ ) { privDbl = ( (double) privIndx ) / 16; y[i] = sin( exp( cos( - exp( sin(x[i]) ) ) ) ) +
cos( privDbl ); } }
Parallel for loop index isPrivate by default.
Reduction VariablesReduction Variables
#pragma omp parallel for reduction( op:list )
• op is one of +, *, -, &, ^, |, &&, or ||
• The variables in list must be used with this operator in the loop.
• The variables are automatically initialized to sensible values.
The reduction clauseThe reduction clause
sum = 0.0;#pragma parallel for default(none) shared (n, x) reduction(+ : sum) for (int I=0; I<n; I++) sum = sum + x(I);
– A private instance of sum is allocated to each thread– Performs a local sum in each thread– Before terminating, each thread adds its local sum to the global
sum variable
Programming Model – Loop Programming Model – Loop SchedulingScheduling
• schedule clause determines how loop iterations are divided among the thread team– static([chunk]) divides iterations statically between
threads• Each thread receives [chunk] iterations, rounding as necessary to
account for all iterations
• Default [chunk] is ceil( # iterations / # threads )
– dynamic([chunk]) allocates [chunk] iterations per thread, allocating an additional [chunk] iterations when a thread finishes
• Forms a logical work queue, consisting of all loop iterations
• Default [chunk] is 1
– guided([chunk]) allocates dynamically, but [chunk] is exponentially reduced with each allocation
Loop schedulingLoop scheduling
Programming Model – Loop Programming Model – Loop SchedulingScheduling
for( i=0; i<16; i++ )
{
doIteration(i);
}
// Static Scheduling
int chunk = 16/T;
int base = tid * chunk;
int bound = (tid+1)*chunk;
for( i=base; i<bound; i++ )
{
doIteration(i);
}
#pragma omp parallel for \
schedule(static)
Programming Model – Loop Programming Model – Loop SchedulingScheduling
for( i=0; i<16; i++ )
{
doIteration(i);
}
// Dynamic Scheduling
int current_i;
while( workLeftToDo() )
{
current_i = getNextIter();
doIteration(i);
}
#pragma omp parallel for \
schedule(dynamic)
OpenMP sections directiveOpenMP sections directive
Several blocks are executed in parallel
• C/C++:
#pragma omp sections [ clause [ clause ] ... ] new-line
{
[#pragma omp section new-line ]
structured-block1
[#pragma omp section new-line
structured-block2 ]
...
}
OpenMP sections directiveOpenMP sections directive#pragma omp parallel{#pragma omp sections
{{ a=...;b=...; }
#pragma omp section{ c=...;d=...; }
#pragma omp section{ e=...;f=...; }
#pragma omp section{ g=...;h=...; }
} /*omp end sections*/} /*omp end parallel*/
The omp sections clause - exampleThe omp sections clause - example
Threadprivate
• Private variables are private on a parallel region basis.
• Threadprivate variables are global variables that are private throughout the execution of the program.
ThreadprivateThreadprivate
#pragma omp threadprivate( list )
Example: #pragma omp threadprivate( x)
• Requires program change in POSIX threads.
• Requires an array of size p.
• Access as x[pthread_self()].
• Costly if accessed frequently.
• Not cheap in OpenMP either.
ThreadprivateThreadprivate
• Makes global data private to each thread– C: file scope and static variables
• Different from making them PRIVATE– with PRIVATE global scope is lost– THREADPRIVATE preserves global scope for
each thread
• Threadprivate variables can be initialized using COPYIN clause
Master structured blockMaster structured block• Only the master (0) thread executes the block • Rest of the team skips the section and continues execution
from the end of the master• No barrier at the end (or start) of the master section• The worksharing construct, OMP single is similar in
behavior but has an implied barrier at the end. Single is performed by any one thread.
• Syntax:– #pragma omp master
{……}
Ordered Structured BlockOrdered Structured Block
• Enclosed code is executed in the same order as would occur in sequential execution of the loop
• Directives:– #pragma omp ordered
{…..}
OpenMP synchronizationOpenMP synchronization
Implicit Barrier - all threads in a team wait for all threads to complete up to the barrier point
– beginning and end of parallel constructs– end of all other control constructs– barrier can be removed with nowait clause
• Explicit critical - only one thread at a time may execute a critical region
OpenMP critical directiveOpenMP critical directiveEnclosed code– executed by all threads, but– restricted to only one thread at a time• C/C++:#pragma omp critical [ ( name ) ] new-linestructured-block
• A thread waits at the beginning of a critical region until no otherthread in the team is executing a critical region with the same name.All unnamed critical directives map to the same unspecifiedname.
OpenMP criticalOpenMP critical
C / C++: cnt = 0;f=7;#pragma omp parallel{#pragma omp for
for (i=0; i<20; i++) {if (b[i] == 0)
#pragma omp critical {
cnt ++;} /* endif */
a[i] = b[i] + f * (i+1);} /* end for */
} /*omp end parallel */
Clauses by Directive TableClauses by Directive Table
https://computing.llnl.gov/tutorials/openMP