Parallel Computing with Matlab UVACSE Short Course Computing with Matlab UVACSE Short Course ... use the spmd keyword to mark the parallel blocks of code and the ... (MPICH2) labSendReceive,

Parallel Computing with MatlabUVACSE Short Course

E Hall1

1University of Virginia Advanced Computing Services and [email protected]

October 6, 2014

(UVACSE) October 6, 2014 1 / 1

Outline


NX Client for Remote Linux Desktop

Starting and Configuring NX Client

Once logged into fir.itc.virginia.edu through NX

Open a terminal from Applications/Accessories/Terminal menuSelect and right-click on Terminal to add to launcher

Create a Matlab directory with mkdir commandStart web browser from icon at top of desktop


NX Client for Remote Linux Desktop

Download Short Course Examples

Download the short-course materials fromhttp://www.uvacse.virginia.edu/software/Matlab-at-uva/

Follow the links,→High Performance Computing→Parallel Computing Toolbox→Parallel Computing Short Courseand download 3 files to Matlab directory you create with mkdircommand

ClassExamples_Fa14.zipmatlab_parallel_Fa14.pdf


http://www.uvacse.virginia.edu/software/Matlab-at-uva/

Matlab Parallel Computing Toolbox

Solving Big Technical Problems

Computationally intensive, long-running codes

Run tasks in parallel on many processorsTask parallel

Large Data sets

Load data across multiple machines that work in parallelData parallel



Parallel Computing Toolbox Features

Support for data-parallel and task-parallel application development

Ability to annotate code segments

parfor (parallel for-loops) for task-parallel algorithmsspmd (single program multiple data) for data-parallel algorithms

These high-level programming constructs convert serial Matlabcode to run in parallel on several workers

The parallel programming constructs function even in the absence ofworkers so that a single version of code can run for both serial andparallel execution



Programming Parallel Applications in Matlab

Simplifies parallel code development by abstracting away complexity ofmanaging coordination and distribution of computations and databetween a Matlab client and workers.

Run 12 workers locally on a multicore desktop

Integration with Matlab Distributed Computing Server forcluster-based applications that can use up to 256 workers



Matlab Parallel Workflow

The toolbox enables application prototyping on the desktop with up toeight local workers (left), and with Matlab Distributed ComputingServer (right), applications can be scaled to multiple computers on acluster.



UseParallel for Optimization Algorithms

The Optimization Toolbox solvers fmincon, fgoalattain, andfminimax can automatically distribute the numerical estimation ofgradients of objective functions and nonlinear constraint functions tomultiple processors.

Parallel computing is enabled with matlabpool, a ParallelComputing Toolbox function

The option UseParallel is set to ’Always’. The default valueof this option is ’Never’.




Optimization Toolboxfmincon

fminimax

fgoalattain

Genetic Algorithm and Direct Search Toolbox

ga

gamultiobj

patternsearch




Caveats:

The built-in parallel support in Optimization Toolbox is beneficialfor problems that have objective/constraint functions withexecution times greater than network overhead associatedwith distributing computations across multiple workers

However, parallelizing the objective/constraint function itselfcan be a better approach if it is the most expensive step in theoptimization problem and can be accelerated by parallelizing theobjective function

Documentation:

Improving Optimization Performance with Parallel ComputingMinimizing an Expensive Optimization Problem Using ParallelComputing Toolbox


http://www.mathworks.com/company/newsletters/digest/2009/mar/parallel-optimization.html?s_cid=HP_NL

http://www.mathworks.com/products/optimization/demos.html?file=/products/demos/shipping/optim/optimparfor.html

http://www.mathworks.com/products/optimization/demos.html?file=/products/demos/shipping/optim/optimparfor.html


Example Code: Parallel Optimization with fmincon



Task-parallelism using parfor

Parallel for-Loops - parfor

Code Annotation

parfor i = 1 : n% do something with i

end

Mix task-parallel and serial code in the same function

Run loops on a pool of Matlab resources

Iterations must be order-independent










Parallel for-loops (parfor) automatically distribute a set ofindependent tasks over a set of workers.

Work distribution across worker is dynamic for load balancing.

The matlabpool command sets up the interactive executionenvironment for parallel constructs such as parfor or spmd

The running code automatically detects the presence of workers andreverts to serial behavior if none are present.

Using the batch command allows parallel code to run in batch modeacross the compute nodes of a cluster.

Applications: Monte Carlo simulations, Parameter sweeps



Setting Up the Matlab Workers with matlabpool

The matlabpool command allocates a set of dedicatedcomputational resources by reserving a number of Matlab workers forexecuting parallel Matlab code.

Parallel Computing Toolbox provides the ability to use up to 12 localworkers on a multicore or multiprocessor computer using a singletoolbox license.

Within this environment, parfor and spmd constructs can set up dataand Matlab code exchange between a Matlab client.



spmd for Data-Parallel Processing

For Matlab algorithms that require large data set processing, theParallel Computing Toolbox provides,

distributed arrays, parallel functionsthe spmd keyword to annotate sections of your code for parallelexecution on several workers.

These parallel constructs handle the inter-worker communication andcoordinate parallel computations behind the scenes.




Using distributed arrays, you can allocate matrices of any data typeacross all workers participating in the parallel computation.

Parallel functions let you perform mathematical operations such asindexing, matrix multiplication, decomposition, and transforms directlyon distributed arrays.

The toolbox also provides more than150 parallel functions fordistributed arrays, including linear algebra routines based onScaLAPACK.




Single Program Multiple Data - spmd

Code Annotation

spmd% data parallel operation

end

Single programRuns simultaneously across all workersEnables easy writing and debugging

Multiple DataData spread across workers

Runs serially if no workers are available




Distributed Arrays

Distributed arrays are special arrays that store segments of dataon Matlab workers that are participating in a parallel computation.

can handle larger data sets than you can on a single Matlabsession.

You can construct distributed arrays in several ways:Using constructor functions such as rand, ones, and zerosConcatenating arrays with same name but different data ondifferent labsDistributing a large matrix



spmd for Data-Parallel Processing: Example Code

Solving a Large Linear System



spmd for Data-Parallel Processing: Example

Numerical Estimation of Pi Using Message Passing

Use the fact that∫ 1

0

41 + x2 dx = 4(atan(1)− atan(0)) = π

to approximate pi by approximating the integral on the left

use the spmd keyword to mark the parallel blocks of code and theMatlab worker pool performs the calculations in parallel



spmd for Data-Parallel Processing: Example

Divide the work between four labs by having each lab calculate theintegral of the function over a subinterval of [0, 1]

Define the variables a and b on each lab using the labindex so thatthe intervals [a, b] correspond to the subintervals above.



spmd for Data-Parallel Processing: Example Code

Parallel Esimation of Pi:



Matlab MPI for Message Passing

Use when a high degree of control over parallel algorithm is required

High-level abstractions of MPI message-passing routines basedon the MPI standard (MPICH2)

labSendReceive, labBroadcast, and others

Send, receive, and broadcast any data type in Matlab includingstructures and cell arrays, without any special setup.

Use any MPI implementation that is binary-compatible withMPICH-2



Message Passing for Data Parallel Algorithms

Message passing functions can be used with spmd statements



pmode for Interactive Parallel Computing

Parallel Command Window

The Parallel Command Window provides an extension to theMatlab command window for executing data-parallel Matlab codedirectly on workers participating in the interactive parallel sessionCommands issued at the pmode prompt are executed immediatelyon all the labs and results are returned immediately to the clientsession.You can use distributed arrays, associated parallel functions, aswell as message-passing functions in this mode.This tool facilitates the debugging process, as it allows you towatch the results and the interaction between labs at each step.




Interactive Prototyping / Development Using Parallel CommandWindow

Parallel Command Window similar to regular command window

Execute commands and observe behavior on each worker of thecluster

Can run up to 8 workers on Matlab desktop

Workers are processes, not tied to cores or processors as such




Example Code

>>pmode startP>>a=1P>>labindexP>>numlabsP>> A=rand(2000, 2000, codistributor());P>>whosP>>size(localPart(A))





Parallel Computing on the Linux Cluster

Scaling Up from the Desktop

Parallel Computing Toolbox provides the ability to use up to 12 localworkers on a multicore or multiprocessor computer using a singletoolbox license.

When used together with MATLAB Distributed Computing Server, youcan scale up your application to use any number of workers running onany number of computers.

ITS Linux cluster has MDCS licenses for 256 workers.

Alternatively, you can run up to 12 workers on a single multi-corecompute node of the cluster.



Using Parallel Configurations with PBS Pro

Parallel Configurations - Where and How the Code is Executed

Maintain named configurations

Predefine cluster information and environment-specificparametersNo code changes requiredSet once, use many times

Useful for both interactive and batch workflows

Toolbox provides GUI to manage configurations



Scaling Up from the Desktop

Example: Running a gene regulation model on a cluster usingMATLAB Distributed Computing Server.



Parallel Matlab on ITC Linux Clusters

ITC Linux Clusters

Distriubted Computing Server Licenses for 256 workers. Usersencouraged not to use more than 16 at one time.

Matlab Configurations interface to PBS Pro for submitting jobs tothe cluster

Distributed Computing Server Licenses available on any Linuxcluster that mounts /common for ITC servers


http://www.uvacse.virginia.edu/itc-clusters/


Interactive Parallel Matlab Jobs on Cluster

The matlabpool command can be used with the cluster configurationfile to launch an interactive job to the cluster from within the Desktopinterface on the cluster front-end.



Interactive Parallel Matlab Jobs on Cluster

After execution of the matlabpool command with the clusterconfiguration file, the PBS command qstat command shows thecompute nodes serving as workers have been allocated.

After the matlabpool close command, the job is exiting.


Additional Computational Resources

Additional Computational Resources

The Cross Campus Grid Projecthttp://www.uvacse.virginia.edu/cross-campus-grid-xcg/

XSEDE: Extreme Science and Engineering Discovery Environmenthttps://www.xsede.org/

MATLAB GPU Computing with NVIDIA CUDA-Enabled GPUsA Tesla GPU card is available for testing Matlab GPU code in the CSdepartment.


http://www.uvacse.virginia.edu/cross-campus-grid-xcg/

https://www.xsede.org/

http://www.mathworks.com/discovery/matlab-gpu.html

References

References

1 Mathworks Parallel Computing Toolbox Documentationhttp://www.mathworks.com/products/parallel-computing/

2 Mathworks Parallel Computing Toolbox Demos and Webinarshttp://www.mathworks.com/products/parallel-

computing/demos.html

3 Parallel Matlab for Multicore and Multinode Computers, by JeremyKepner, SIAM Press.

Need further help? Email [email protected].


http://www.mathworks.com/products/parallel-computing/

http://www.mathworks.com/products/parallel-computing/demos.html



Documents

Parallel Computing with Matlab UVACSE Short Course Computing with Matlab UVACSE Short Course ... use the spmd keyword to mark the parallel blocks of code and the ... (MPICH2) labSendReceive,