CNL2 WP Architecture

8/12/2019 CNL2 WP Architecture

1/20

Whitepaper: Distributed Numerical Computing

with Microsoft Codename Cloud NumericsMicrosoft Corporation

Published: January 2012

This content is preliminary content. It might be incomplete and is subject to change.

AbstractThis white paper introduces Microsoft Codename Cloud Numerics lab (referred to as Cloud Numerics

in the following content), a numerical and data analytics library. It provides guidelines for data scientists

and others who write C# applications to enable their applications to be scaled out, deployed, and run on

Windows Azure.


2/20

Copyright Information

This document supports a preliminary release of a software product that may be changed substantially

prior to final commercial release, and is the confidential and proprietary information of Microsoft

Corporation. It is disclosed pursuant to a non-disclosure agreement between the recipient and Microsoft.

This document is provided for informational purposes only and Microsoft makes no warranties, either

express or implied, in this document. Information in this document, including URL and other Internet Web

site references, is subject to change without notice. The entire risk of the use or the results from the use

of this document remains with the user. Unless otherwise noted, the companies, organizations, products,

domain names, e-mail addresses, logos, people, places, and events depicted in examples herein are

fictitious. No association with any real company, organization, product, domain name, e-mail address,

logo, person, place, or event is intended or should be inferred. Complying with all applicable copyright

laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document

may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any

means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the

express written permission of Microsoft Corporation.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property

rights covering subject matter in this document. Except as expressly provided in any written license

agreement from Microsoft, the furnishing of this document does not give you any license to these patents,

trademarks, copyrights, or other intellectual property.

2011-2012 Microsoft Corporation. All rights reserved.

Microsoft, Windows, Windows HPC Server, Visual Studio, and Windows Azure are trademarks of theMicrosoft group of companies.

All other trademarks are property of their respective owners.


3/20

Contents

1 Introduction ......................................................................................................................................... 4

1.1 A Hello, World example in Cloud Numerics............................................................................. 5

1.2 What Cloud Numerics is not ....................................................................................................... 5

1.3 What this document does and does not cover .............................................................................. 6

1.4 High-level overview ....................................................................................................................... 6

2 The Cloud Numerics Programming Model.................................................................................... 8

2.1 Arrays in Cloud Numerics ........................................................................................................... 9

2.1.1 Array element types .............................................................................................................. 9

2.1.2 Indexing and iteration ............................................................................................................ 9

2.1.3 Array broadcasting ................................................................................................................ 9

2.2 Local arrays in Cloud Numerics................................................................................................ 10

2.2.1 Underlying storage .............................................................................................................. 102.2.2 Array factory type ................................................................................................................ 10

2.2.3 Interoperability with .NET arrays ......................................................................................... 10

2.2.4 Indexing and iteration with local arrays ............................................................................... 11

2.2.5 Other operations.................................................................................................................. 11

2.3 Distributed Arrays ........................................................................................................................ 12

2.3.1 Partitioning of arrays ........................................................................................................... 12

2.3.2 Propagation of distributions ................................................................................................. 12

2.4 Distributed array types ................................................................................................................ 13

2.5 Working with distributed arrays ................................................................................................... 13

2.5.1 Conversion between local and distributed arrays ............................................................... 13

2.5.2 Indexing distributed arrays .................................................................................................. 13

2.5.3 Other distributed array operations ....................................................................................... 14

2.6 Expressing parallelism in Cloud Numerics............................................................................... 14

3 The Cloud Numerics Runtime Execution Model........................................................................ 15

3.1 Application development, debugging and deployment ............................................................... 15

3.2 Process model............................................................................................................................. 15

4 Conclusions ....................................................................................................................................... 19

5 References ......................................................................................................................................... 20


4/20

1 Introduction

Cloud Numericsis a new .NET programming framework tailored towards performing numerically-

intensive computations on large distributed data sets. It consists of

a programming model that exposes the notion of apartitionedor distributedarray to the user

an execution framework or runtime that efficiently maps operations on distributed arrays to a

collection of nodes in a cluster

an extensive library of pre-existing operations on distributed arrays and tools that simplify the

deployment and execution of a Cloud Numericsapplication on the Windows Azureplatform

Programming frameworks such as Map/Reduce [2, 3](and its open-source counterpart, Hadoop [9]) have

evolved to greatly simplify the processing of large datasets. These frameworks expose a very simple end-

user programming model and the underlying execution engine abstracts away most of the details of

running applications in a highly scalable manner on large commodity clusters. These simplified modelsare adequate for performing relational operations and implementing clustering and machine learning

algorithms on data that is too large to fit into the main memory of all the nodes in a cluster. However, they

are often not optimal for cases when the data does fit into the main memory of the cluster nodes.

Additionally, algorithms that are inherently iterative in nature or that are most conveniently expressed in

terms of computations on arrays are difficult to express using these simplified programming models.

Finally, while the Hadoop ecosystem is extremely vibrant and has developed multiple libraries such as

Mahout [7], Pegasus [5]and HAMA for data analysis and machine learning, these currently do not

leverage existing, mature scalable linear algebra libraries such as PBLAS and ScaLAPACK that have

been developed and refined over several years.

In contrast, libraries such as the Message Passing Interface or MPI [7]are ideally suited for efficiently

processing in-memory data on large clusters, but are incredibly difficult to program correctly andefficiently. A user of the library has to carefully orchestrate the movement of data between the various

parallel processes. Unless this is done with a lot of care, the resulting high performanceapplication will

exhibit extremely poor scalability or worse, crash or hang in a non-deterministic manner.

The programming abstractions provided by Cloud Numericsdo not expose any low-level parallel-

processing constructs. Instead, parallelism is implicit and automatically derived from operations on data

types that are provided by the framework such as distributed matrices. These parallel operations in turn

map to efficient implementations that in turn utilize existing high-performance libraries such as the PBLAS

and ScaLAPACK libraries mentioned above.

The rest of this document provides an overview of the programming and runtime execution models in

Cloud Numericsand is intended to complement the quick start guide and the library API reference.


5/20

1.1 A Hello, World example in Cloud Numerics

To briefly illustrate the parallel programming model in Cloud Numerics, the following C# sample code

first loads a distributed matrix, computes its singular values in parallel and prints out its two-norm and

condition number with respect to the two-norm:

varA = Distributed.IO.Loader.LoadData(csvReader);varS = Decompositions.SvdValues(A);vars0 = ArrayMath.Max(S);vars1 = ArrayMath.Min(S);

Console.WriteLine("Norm: {0}, Condition Number: {1}", s0, s0 / s1);

1.2 What Cloud Numericsis not

While the rest of this document provides more details on the programming and execution model in Cloud

Numerics, it is important to briefly emphasize what it is not.

First, the Cloud Numericsprogramming model is primarily based around distributed array operations(c.f. data parallelor SIMD-style of programming). Certain relational operations such as selects with user-

defined functions or complex joins are simpler to express on top of languages such as Pig, Hive and

SCOPE. Similarly, while Cloud Numericsis designed to deal with large data sets, it is currently

constrained to operate on arrays that can fit in the main memory of a cluster. On the other hand, data on

disk can be pre-processed via existing bigdata processing tools and ingested into a Cloud Numerics

application for further processing.

Second, Cloud Numericsis not just a convenient C# wrapper around message-passing libraries such as

MPI, for example MPI.NET [3]; all aspects of parallelism are expressed via operations on distributed

arrays and the Cloud Numericsruntime transparently handles the efficient execution of these high-level

array operations on a cluster.

A key aspect that distinguishes Cloud Numericsfrom parallelization techniques such as PLINQ and

DryadLINQ [10], that are based on implementing a custom LINQ provider, is that parallelization in Cloud

Numericsoccurs purely at runtime and does not involve any code generation from (say) LINQ expression

trees; a usersapplication can be developed as a regular .NET application by referencing the Cloud

Numericsruntime and library DLLs and executed on the cluster in Azure.

Finally, the underlying communication layer in Cloud Numericsis built on top of the message passing

interface (MPI) and does inherit some of the limitations in the underlying implementation such as:

1. The process model is currently inelastic; once a Cloud Numericsapplication has been launched

on (say)

cores in a cluster, it is not possible to dynamically grow or shrink the resources as the

application is running.

2. The implementation is not resilient against hardware failure. Unlike frameworks like Hadoop that

are designed explicitly to operate on unreliable hardware, if one or more nodes in a cluster fails, it

is not possible for a Cloud Numericsapplication to automatically recover and continue

executing.

On the other hand, having MPI as the underlying communication layer in the Cloud Numericsruntime

does endow it with certain advantages. For instance, Cloud Numericsapplications can automatically

take advantage of high-speed interconnects such as Infiniband between nodes in a cluster and


6/20

optimizations such as zero-copy memory transfers and shared-memory-aware collectives within a single

multi-core node. More importantly, array operators in Cloud Numericscan leverage the vast ecosystem

of high-performance distributed memory numerical libraries such as ScaLAPACK built on top of MPI.

1.3 What this document does and does not cover

The intent of this document is to provide an overview of the programming model in Cloud Numerics

along with a high-level description of how a Cloud Numericsapplication runs in parallel on a cluster of

machines. It does not for example cover the extensive library of numerical functions in Cloud Numerics

for operating on local and distributed arrays, the programming interfaces provided to import and export

data in parallel and the details of how a Cloud Numericsapplication is deployed on a cluster on

Windows Azure from the client and executed.

1.4 High-level overview

Figure 1 illustrates the main components in a distributed Cloud Numericsapplication. The end-user or

clientapplication is itself written in a CLI (Common Language Infrastructure) language such as C#,

VisualBasic.NET or F# against a set of .NET APIs provided by the Cloud Numericslibraries. Theseinterfaces primarily consist of a set of localand distributedhigh-performance array data types along with

a large collection of array operations; there are no additional interfaces for parallel programming such as

task creation and synchronization, resource management, etc. Once an application is running on a cluster

of machines, the runtime automatically partitions the data between the nodes in the cluster and executes

operations on partitioned data in parallel.

Operations on distributed or partitioned arrays in turn depend upon a set of runtime services for handling

distributed function execution, error handling and input/output operations. In addition, the Cloud

Numericssystem provides a set of tools to simplify development, deployment and execution of a

distributed application on Azure from a local client machine.

Under the covers, the programming interfaces are implemented as thin wrappers around a native runtimeand library infrastructure. These consist of lower-level local and distributed array objects along with

interfaces for performing operations on them. Several operations on distributed arrays leverage existing

high-performance numerical libraries such as BLAS and LAPACK on a single CPU core and PBLAS,

BLACS and ScaLAPACK across CPU cores. At the lowest level are inter-process communication libraries

such as MPI and C and C++ runtime libraries for memory management and exception handling.


7/20

Figure 1. The components in a Cloud Numericsapplication.

CloudNumerics Application

CloudNumerics.NET Runtime

(Local and

distributed types,remote function

invocation, I/O)

CloudNumerics.NET Library

(User-facing

mathematical,

statistical and

signal processing

functions)

CloudNumerics

Native Runtime

CloudNumerics

Native Libraries

System Libraries

(Process and Thread Management, Message Passing,

Memory Management, Exception Handling, )

Lower-level

Numerical

Libraries(BLAS, BLACS,

PBLAS, LAPACK,

ScaLAPACK, FFTW)

CloudNumerics

Azure Deployment

Tools


8/20

2 The Cloud NumericsProgramming Model

As mentioned previously, Cloud Numericsis ideally suited for algorithms that can be expressed in terms

of high-level operations on multi-dimensional arrays.

For instance, consider the power iterationalgorithm for computing the largest eigenvector of a matrix.

Power iteration is at the heart of several fundamental graph-theoretic computations such as thePageRank for a collection of web pages. If the collection of pages is represented as a matrix where is non-zero when page is connected to page, the principal eigenvector can be computed roughly asfollows:

Arrays such as the web-connectivity matrix described above are fundamental types in Cloud Numerics.

Further, vector operations such as and array operations such as can beexpressed succinctly using the Cloud Numericslibrary and executed efficiently even when the vectors

and matrices are very large and are therefore partitioned between several nodes in a cluster.

On the flip side, the Cloud Numericsprogramming model may not be a good fit for algorithms that

cannot be easily expressed in array notation (such as relational joins between two data sets) or for

applications such as parsing and simplifying petabytes of text data where numerical computation is not

the main bottleneck. However, once the data has been boiled-down and transformed to its core

numerical structure, it can be further analyzed using Cloud Numerics.

, , , T /* Start with an initial guess */

/* Normalize it */

/* Compute the next iteration */

While > do /* Repeat the process until it converges */


9/20

2.1 Arrays in Cloud Numerics

All array types in Cloud Numerics, be they dense or sparse, local or distributed implement the

Microsoft.Numerics.IArrayinterface, where the generic parameter Trefers to the type of

elements in the array. The IArrayinterface provides fundamental properties such as Shape,

NumberOfDimensionsand NumberOfElements, indexers and basic methods such as Transpose,

Reshapeand Flattenthat are supported by all array sub-types. Note that arrays in Cloud Numericsare

always multi-dimensional (i.e., there is no notion of distinct Matrix or Vector types).

2.1.1 Array element types

Both local and distributed arrays support the following element types: boolean, one-byte, two-byte, four-

byte and eight-byte signed and unsigned integers, single and double-precision real floating point numbers

and single and double-precision complex floating point numbers. The corresponding .NET types are listed

below:

Array element type Description Range of values

System.Boolean Single-byte logical values true/false

System.UInt8 Single-byte unsigned integers ,System.Int8 Singled-byte signed integers ,System.UInt16 Two-byte unsigned integers , System.Int16 Two-byte signed integers , System.UInt32 Four-byte unsigned integers , System.Int32 Four-byte signed integers , System.UInt64 Eight-byte unsigned integers , System.Int64 Eight-byte signed integers , System.Single Single-precision IEEE 754 real floating

point values,

System.Double Double-precision IEEE 754 real floatingpoint values

,Microsoft.Numerics.Complex64 Single-precision complex valuesMicrosoft.Numerics.Complex128 Double-precision complex values

2.1.2 Indexing and iteration

An important interface that IArrayin turn implements is the .NET enumerable interface

IEnumerable. Therefore, methods in .NET, for instance LINQ operators that accept IEnumerable

can be passed instances of Cloud Numericsarrays.

2.1.3 Array broadcasting

In NumPy [2], broadcasting refers to two related concepts: (a) the application of a scalar function to every

element of an array producing an array of the same shape and (b) the set of rules governing how two or

more input arrays of different shapes are combined under an element-wise operation. In environmentssuch as NumPy and Cloud Numericsthat support a rich set of operations on multi-dimensional arrays,

broadcasting is often used as a convenient mechanism for implementing certain forms of iteration.

Several element-wise operators and functions in Cloud Numericssupport broadcasting using

conventions similar to NumPy. For example, given a local or distributed array Aand a scalar s,

Microsoft.Numerics.BasicMath.Sin(A)applies the sin function and A + sadds sto each element of


10/20

A. Similarly, for a vector vcontaining the same number of rows of A, A vsubtracts vfrom every column

of A (in other words, Cloud Numericsbroadcaststhe vector valong the columnsof A)1.

2.2 Local arrays in Cloud Numerics

Before describing the programming model for distributed arrays, it is instructive to first understand the

model for interacting with local arrays in Cloud Numericssince distributed arrays support practically the

same semantics as local arrays except with different performance characteristics.

The most important local array type in Cloud Numericsis

Microsoft.Numerics.Local.NumericDenseArray which represents a dense multi-dimensional

array with numerical values. As mentioned previously, dense arrays can have an arbitrary number of

dimensions (so it is possible to create and operate on say, a 10-dimensional array just as easily as a two-

or three-dimensional array).

2.2.1 Underlying storage

Dense local arrays in Cloud Numericsare not backed by arrays allocated in the .NET/CLR heap. This is

primarily because arrays allocated in the CLR are currently constrained to be smaller than two Gigabyteswhereas the ones allocated on the native heap do not have this limitation. Additionally, the underlying

native array types in the Cloud Numericsruntime are sufficiently flexible that they can be easily wrapped

in environments other than .NET (say Python or R) with very little additional effort.

Unlike arrays in .NET that are internally stored in row-major order, multi-dimensional arrays in Cloud

Numericsare represented in column-major order that allows for efficient interoperability with high-

performance numerical libraries such as BLAS and LAPACK.

2.2.2 Array factory type

The local NumericDenseArrayclass does not provide any public constructors; users are expected to

create instances of local arrays using a factory class, NumericDenseArrayFactorythat provides anumber of static array creation methods. For example, to create a local array of double-precision values filled with zero, one would use:

vardoubleArray = NumericDenseArrayFactory.Zeros(4, 10, 12);

2.2.3 Interoperability with .NET arrays

As mentioned above, arrays in Cloud Numericsare not backed by .NET arrays; instead, the

NumericDenseArrayFactoryprovides a method to create a new Cloud Numericsarray from a .NET

array. For example, to create a new local array from a .NET array, one would use:varaDotNetArray = newdouble[,]{{1.0, 2.0}, {3.0, 4.0}};varcnArray = NumericDenseArrayFactory .CreateFromSystemArray(aDotNetArray);

1In the first version Cloud Numerics only supports broadcasting for scalars, but this deficiency will be

addressed for the next release.


11/20

Conversely, the NumericDenseArrayclass provides two methods that return .NET arrays: one for

returning a multi-dimensional .NET array of the same shape and with the same elements as the

NumericDenseArrayinstance and other for returning a one-dimensional .NET array that contains the

elements flattened in row-majororder. These are shown in the code snippets below:

double[,] anotherDotNetArray = (double[,]) cnArray.ToSystemArray();

double[] flatDotNetArray = cnArray.ToFlatSystemArray();

2.2.4 Indexing and iteration with local arrays

A local array in Cloud Numericscan be indexed by a scalar just like any other multi-dimensional array in

.NET. Currently, indices are constrained to be scalars and the number of indices must be the same as the

number of dimensions of the array as shown in the following examples:

doubletwoDValue = twoDArray[3, 4];

doublethreeDValue = threeDArray[2, 3, 4];

Similarly, local arrays support iteration over the elements in flattened column-major order. For instance,

one could compute the Frobenius norm of a matrix defined as: ( )

using the following LINQ expression:

varfroNorm = Math.Sqrt(doubleMatrix.Select((value) => value * value).Sum());

2.2.5 Other operations

In addition to the primitive operations shown in this section, local arrays in Cloud Numericsalso supportoperations such as element-wise arithmetic and logical operators, reshaping an existing array to a

different shape while preserving the order of elements in memory, transposing an array and filling the

elements of an array using a generator function. For example, given a general matrix, one coulddecompose it into a sum of a symmetric and a skew-symmetric matrix:

using:

varsym = 0.5 * (matrix + matrix.Transpose());varskewSym = 0.5 * (matrix - matrix.Transpose());


12/20

2.3 Distributed Arrays

In Cloud Numerics, the end-user creates and interacts with a single instance of a distributed array.

However, under the covers, the runtime creates several arrays on a number of locales2and partitions the

global distributed array between them. Every aggregate operation on a global distributed array (say

computing the sum of its columns or permuting its elements) is transparently transformed by the Cloud

Numericsruntime into one or more parallel operations on these array partitions.

2.3.1 Partitioning of arrays

This section briefly summarizes the partitioning scheme for distributed arrays in Cloud Numerics. Note

however that the Cloud Numericsruntime automatically handles the partitioning of arrays between

several locales and the user normally does not need to specify any aspects of data distribution.

A distributed array is constrained to be partitioned across the locales only along one of its dimensions.

The dimension along with an array is partitioned is referred to as its distributeddimension. Each locale

then gets a contiguous set of slices of the partitioned array along the distributed dimension; the set of

indices along the distributed dimension corresponding to the slices assigned to a particular locale

constitute the spanof the array on that locale. Finally, each locale always gets complete slices of thedistributed array and the slices between two different locales never overlap.

The partitioning scheme described above is illustrated inFigure 2 for a matrix that is distributed across

three locales, first by rows and second by columns.

By default, a multidimensional array is always distributed along its last non-singleton dimension. For

instance, a matrix is distributed by columns whereas a vector is distributed byrows.

Figure 2. Row and column partitioned arrays.

2.3.2 Propagation of distributions

Generally, methods and static functions in Cloud Numericsthat return arrays produce distributed

outputs given distributed inputs (the ToLocalArraymethod and functions that return scalars such as SUM

are important exceptions to this rule). The precise manner in which the distribution propagates is as

2A locale is defined in the section on the execution model, but it can be thought of as a process

containing a portion of a distributed array.

Span 0

Span 1

Span 2

Span

0

Span

1

Span

2


13/20

follows: if one or more input arguments to a function are arrays, the distributed dimension of the output is

the maximum of the distributed dimension of all the inputs. If this distributed dimension is greater than the

last non-singleton dimension of the output, the output is distributed along its last non-singleton dimension.

2.4 Distributed array types

All distributed arrays in Cloud Numericsimplement the

Microsoft.Numerics.Distributed.IArrayinterface where Tis the type of elements in the

distributed array. Besides the properties and methods that the base Microsoft.Numerics.IArray

interface provides, the distributed array interface provides two important properties:

DistributedDimensionand Spanas well as a Redistributemethod that redistributes the array along

a different dimension.

2.5 Working with distributed arrays

The fundamental distributed array type in Cloud Numericsis

Microsoft.Numerics.Distributed.NumericDenseArray. Unlike their local counterparts,

distributed arrays provide explicit constructors. For example to create a new distributed double-precision

dense array of shape one would use:vardArray = newDistributed.NumericDenseArray(100, 100, 500);

2.5.1 Conversion between local and distributed arrays

Local arrays are implicitly converted to distributed arrays by assignment. As a consequence any function

that only takes a distributed array as an input can be passed a local array and the runtime will

transparently promote the local array to be distributed.

varlArray = Local.NumericDenseArrayFactory .Create(400, 400);

Distributed.NumericDenseArray dArray = lArray;

On the other hand, a distributed array cannot be implicitly converted to a local array since they are

assumed to be much larger than what can fit into the memory of a local machine. Therefore, converting a

distributed array to a local array requires the user to explicitly call the

NumericDenseArray.ToLocalArray()method. In a similar fashion, a distributed array can be

converted to a .NET array via the NumericDenseArray.ToSystemArraymethod.

vardArray = newDistributed.NumericDenseArray(100, 100, 500);

varlArray = dArray.ToLocalArray();

varcsArray = dArray.ToSystemArray();

2.5.2 Indexing distributed arrays

Distributed arrays can be indexed by scalars just like their local counterparts. For example, the following

snippet shows indexing into two- and three-dimensional arrays:

doubletwoDValue = twoDDistArray[3, 4];

doublethreeDValue = threeDDistArray[2, 3, 4];


14/20

Unlike local arrays, distributed arrays do not support iterating over the elements. This is primarily because

there is no built-in facility in .NET for iterating over partitioned collections3in parallel.

2.5.3 Other distributed array operations

Distributed arrays support almost exactly the same set of operations that are supported by local arrays.

For instance, the earlier example of decomposing an array into a symmetric and skew-symmetric matrix:

varsym = 0.5 * (matrix + matrix.Transpose());

varskewSym = 0.5 * (matrix - matrix.Transpose());

works unmodified when the argument is a distributed matrix.

2.6 Expressing parallelism in Cloud Numerics

As mentioned several times in this document, the primary mechanism in Cloud Numericsfor expressing

parallelism is via aggregate (data-parallel) operations on distributed arrays; Cloud Numericscurrently

does not support other modes of parallelism such as task or control parallelism.

As an example, the following code segment illustrates an alternate implementation of decomposing amatrix into its symmetric and skew-symmetric components:

vardims = matrix.Shape.ToArray();varsym = newDistributed.NumericDenseArray(dims);varskewSym = newDistributed.NumericDenseArray(dims);for(longj = 0; j < dims[1]; j++)

{

for(longi = 0; i < dims[0]; i++){

sym[i, j] = 0.5 * (matrix[i, j] + matrix[j, i]);sym[i, j] = 0.5 * (matrix[i, j] - matrix[j, i]);

}

}

While this code is syntactically correct and will produce the correct answer for a distributed array, it will

not execute in parallel due to its use of the C# (sequential) loop construct. Further, the scalar indexing

operations (matrix[i, j], sym[i, j]and sym[i, j]) are highly sub-optimal for distributed arrays

since each scalar access generates communication steps where is the number of locales. Theapproach based on array operators such as 0.5 * (matrix + matrix.Transpose())besides being

more concise is also more efficient, possibly by several orders of magnitude.

It is therefore always advantageous in Cloud Numericsapplications to try and translate fine-grained

array accesses into coarse-grained aggregate array operations using mechanisms such as slicing and

broadcasting.

3This could potentially be done by implementing a custom LINQ provider.


15/20

3 The Cloud NumericsRuntime Execution Model

Having described the programming model in Cloud Numericsbased around operations on local and

distributed arrays, this section briefly explains how a Cloud Numericsapplication is developed and

executed on a cluster. The intent of this section is to once again only provide a high-level overview of the

execution model; the Cloud NumericsQuick Start Guide provides a detailed walkthrough of developing

an application and deploying on to a cluster in Azure.

3.1 Application development, debugging and deployment

A Cloud Numericsapplication is initially developed and debugged on a single workstation using the

Cloud Numerics.NET API. The user can then choose to run the application locally on the workstation

either in sequential or parallel modes. When an application is run in the sequential mode, it behaves as

any other .NET application and for instance, can be debugged and profiled inside the Visual Studio IDE.

Once an application is developed and debugged on a single workstation, it can be deployed and

executed on a Windows HPC (High-Performance Computing) cluster on Windows Azure using the Cloud

Numericsdeployment wizard.

3.2 Process model

Figure 3 illustrates the process model in a Cloud Numericsapplication when executed on a cluster on

Azure. Briefly, when a Cloud Numericsapplication is, it starts up a given number, say copies of theCloud Numericsruntime. These runtime instances are referred to as locales. InFigure 3,each locale is

shown on a different physical node, but in reality, the mapping of locales to actual hardware is somewhat

arbitrary. For example, on a multi-core node or virtual machine, there can be as many locales as cores in

the node. Normally, a user only specifies the number of nodes on which the application must be

executed; the HPC Scheduler takes of allocating as many physical cores as required.

The notion of a locale in Cloud Numericsis intimately tied to the notion of an operating systemprocess;each locale in fact runs on a separate operating system process even within a single physical node. For

this reason, the terms locale and process are interchangeably in the rest of this document.

Of the locales created, one is designated to be the masterand the other are designated to beworkerlocales. The users application only runs on the master process (in other words, a parallel

application in Cloud Numericshas a single thread of control) and the remaining processes wait inan infinite loop for notification from the master to do work on their portions of a distributed array. A single

thread of control from the users perspective is an important aspect that makes parallel programming

convenient in Cloud Numerics; the user does not have to explicitly write or manage the execution of any

parallel constructs (such as launching tasks or passing messages between instances) and therefore it is

not possible to programmatically run into pitfalls such as starvation or deadlock that are present in other

parallel programming models.


16/20

Figure 3. A distributed Cloud Numericsapplication executing on a master and three worker

processes. The processes are interconnected in a topology referred to as a hypercube.

Worker 1

User Application,

CloudNumerics Runtime

CloudNumerics Libraries

System Libraries

Master

User Application,



System Libraries

Worker 2

User Application,



System Libraries

Worker 3

User Application,



System Libraries

Interprocess Comm

Interprocess Comm


17/20

Figures 4 through 6 illustrate the steps involved in performing a distributed operation such as computing

the element-wise sum of two partitioned arrays and.First, the users application dispatches the operation to the master process via the overloaded addition

operator.

Figure 4. First step: the user invokes a method on a distributed array on the master process.

Then, the master process broadcasts the operation to all workers. The payload consists of both the

operation to be performed as well as references to the distributed input arguments.

Figure 5. Second step: The command is broadcast from the master to all workers.

A+B

Master

Master

C=Add(A,B)

C=Add(A,B)

Worker 0

Worker 1Worker 2

C=Add(A,B)


18/20

Next, the master and workers perform a part of the underlying computation in parallel. Often, but not

always this involves communicating with other locales to exchange data. For example, in the case of

element-wise addition, if the two arguments are partitioned such that each locale has all the data

necessary to compute its portion of the output, the operation is done element-wise without any

communication. On the other hand if say the first argument is partitioned by rows and the second is

partitioned by columns, the master and workers first collectively exchange pieces of the first matrix in

order to compute the result.

Figure 6. The master and workers work on a part of the computation in parallel. In is particularexample, the computation can be performed without requiring any communication.

Once each worker is finished with its portion of the computation, it sends back a successful completion

flag to the master. On the other hand, if it fails (say because of an exception), it sends back a specific

failure code along with a serialized exception object describing the failure.

Figure 7. Once the master and workers are done with the computation, the master collects anyerrors during computation. In this particular case, all work is completed successfully.

Worker 0

Worker 1Worker 2

Master

C = Add(A,B)C = Add(A,B)

C = Add(A,B)C = Add(A,B)

Master Worker 0

Worker 1Worker 2

OK

OK

OK


19/20

The master process waits for an acknowledgement from each of the workers. If all processes (including it)

completed successfully, the master returns a reference to the result back to the users application. On the

other hand if one or more workers encountered an error, the master process rethrows the exception in the

payload into the application where it can be handled like any other exception in .NET.

Figure 8. The master then returns the result of the computation to the user as a distributed result.

4 Conclusions

The programming model in Cloud Numerics, based around operations on distributed arrays, simplifiesthe implementation of algorithms that are most naturally expressed in matrix notation and introduces no

additional parallel programming or synchronization constructs. Further, the model, based on a single-

thread of execution, greatly simplifies application development and debugging and reduces the possibility

of conditions such as races and deadlocks due to bugs in the usersapplication. The Cloud Numerics

distributed runtime transparently executes operations on distributed arrays in parallel on nodes in a

cluster, leveraging existing mature high-performance libraries when possible.

Although the general principles behind the design and implementation of Cloud Numericsare based on

well-established fundamentals in high-performance scientific computing, it is still a new and rapidly-

evolving framework. It is anticipated that future versions of Cloud Numericswill support:

interoperability with Map/Reduce execution frameworks

a richer set of functionality on distributed arrays

operations on distributed sparse matrices

better performance on existing functions

more user-friendly application deployment wizard

and most importantly, additional features based on end-user feedback.

Master

C


20/20

5 References

1. Blackford, L.S, Choi, J., Cleary, A., et al. ScaLAPACK UsersGuide, SIAM, Philadelphia, PA, 1997.

2. Dean, J. and Ghemawat, S., MapReduce: Simplified data processing on large clusters, Symposium on

Operating System Design and Implementation (OSDI), 2004.

3. Gregor, D. and Lumsdaine, A. Design and implementation of a high-performance MPI for C# and the

Common Language Infrastructure,ACM Principles and Practice of Parallel Programming, pages 133-142,February 2008.

4. Lin, J. and Dyer, C., Data-intensive Text Processing with MapReduce, Morgan-Claypool, 2010.

5. Kang, U., Tsourakakis, C. E. and Faloutsos, C. PEGASUS: A Peta-Scale Graph Mining System -

Implementation and Observations, IEEE Conference on Data Mining, 2009.

6. Oliphant, T., A Guide to NumPy, Trelgol Publishing, 2005.

7. Owen, S., Anil, R. and Dunning, T. Mahout in Action, Manning Press, 2011.

8. Snir, M. and Gropp, W., MPI The complete reference, MIT Press, Cambridge, MA, 1998.

9. White, T., Hadoop: The Definitive Guide, 2e, OReilly, Sebastopol, CA, 2008.

10. Yu, Y., Isard M., Fetterly, D., et al. DryadLINQ: A System for General-Purpose Distributed Data-Parallel

Computing Using a High-Level Language. Symposium on Operating System Design and Implementation

(OSDI), San Diego, CA, 2008.

Documents

CNL2 WP Architecture