Node Level Primitives for Parallel Exact Inference∗

Yinglong XiaComputer Science Department

University of Southern CaliforniaLos Angeles, CA 90089, U.S.A.

yinglonx@usc.edu

Viktor K. PrasannaMing Hsieh Department of Electrical Engineering

University of Southern CaliforniaLos Angeles, CA 90089, U.S.A.

prasanna@usc.edu

Abstract

We present node level primitives for parallel exact in-ference on an arbitrary Bayesian network. We explore theprobability representation on each node of Bayesian net-works and each clique of junction trees. We study the oper-ations with respect to these probability representations andcategorize the operations into four node level primitives:table extension, table multiplication, table division, and ta-ble marginalization. Exact inference on Bayesian networkscan be implemented based on these node level primitives.We develop parallel algorithms for the above and achieveparallel computational complexity of O(w2r(w+1)N/p),O(Nrw) space complexity and scalability up to O(rw),where N is the number of cliques in the junction tree, r isthe number of states of a random variable, w is the maximalsize of the cliques, and p is the number of processors. Ex-perimental results illustrate the scalability of our parallelalgorithms for each of these primitives.

1. Introduction

A full joint probability distribution for any real-worldsystems can be used for inference. However, such a dis-tribution grows intractably large as the number of variablesused to model the system grows. Bayesian networks[9] areused to compactly represent joint probability distributionsby exploiting conditional independence relationships. Theyhave found applications in a number of domains, includingmedical diagnosis, credit assessment, data mining, imageanalysis and genetics [4][10][11][13].

Inference on a Bayesian network is the computation ofthe conditional probability of the query variables, given aset of evidence variables as knowledge. Such knowledge

∗This research was partially supported by the National Science Foun-dation under grant number CNS-0613376 and utilized the DataStar at theSan Diego Supercomputer Center. NSF equipment grant CRI-0454407 isgratefully acknowledged.

is also known as belief. Inference on Bayesian networkscan be exact or approximate. Exact inference is NP hard[9]. The most popular exact inference algorithm converts agiven Bayesian network into a junction tree, and then per-forms exact inference in the junction tree [6].

Early work on parallel algorithms for exact inference ap-pears in [5][9] and [12], which formed the basis of scalableparallel implementations discussed in [7] and [8]. However,unlike [7], which focuses on the conversion of Bayesiannetwork to junction tree and [8], which presents the par-allelization of exact inference using pointer jumping, we donot focus on the network structure at all. Note that structurelevel parallelism can not offer large speedups when the sizeof the cliques of the junction tree or the number of statesof the variables increase, making node-level operations thedominating part of the problem. We focus on parallelizingthe table operations only and start off with a junction treeconverted from an arbitrary Bayesian network.

In this paper, we study parallelism in exact inferencefrom the node level perspective. Node level means theprobability representation and operations on the nodes ofBayesian networks and the cliques of junction trees. Wepropose four node level primitives and achieve a scalabil-ity of 1 ≤ p ≤ rw, compared to 1 ≤ p ≤ n of moststructure level parallel methods. We implement parallelnode level primitives using MPI on state-of-the-art plat-forms. We also propose an exact inference algorithm basedon those parallel node level primitives, which achieves com-putational complexity ofO(w2r(w+1)N/p) and space com-plexity O(Nrw), 1 ≤ p ≤ rw, where N is the number ofcliques in the junction tree, r is the number of states of arandom variable, w is the maximal size of the cliques, andp is the number of processors.

The paper is organized as follows: Section 2 gives a briefbackground on Bayesian networks and junction trees. Sec-tion 3 addresses the node level primitives for exact infer-ence. Section 4 implements the node level primitives andthe parallel exact inference algorithm based on these prim-itives. Experiments are shown in Section 5 and Section 6

19th International Symposium on Computer Architecture and High Performance Computing

1550-6533/07 $25.00 © 2007 IEEEDOI 10.1109/SBAC-PAD.2007.18

221

19th International Symposium on Computer Architecture and High Performance Computing

1550-6533/07 $25.00 © 2007 IEEEDOI 10.1109/SBAC-PAD.2007.18

221

concludes the paper.

2. Background

Consider a set of n random variables, W ={A1, A2, · · · , An}. The probability that random variableAj takes the value a is P (Aj = a) where a ∈ {0, 1, · · · , r−1} and r is the number of states of variable Aj . A joint dis-tribution, P (W) = P (A1, A2, · · · , An), assigns a proba-bility to every possible combination of states of these ran-dom variables.

A Bayesian network exploits conditional independenceto represent a joint distribution more compactly: The vari-ables in the network with directed edges to a certain vari-ableAj are called parents ofAj , denoted by pa(Aj). Giventhe values of pa(Aj), Aj is conditionally independent of allother preceding variables. The joint probability distributionthus can be rewritten using less variables as:

P (W) = P (A1, · · · , An) =n∏

j=1

P (Aj |pa(Aj)) (1)

For each variable Aj , we use a conditional prob-ability table to describe the conditional probabilitiesP (Aj |pa(Aj)). The details of the conditional probabilitytable will be addressed in Section 3.1.

Traditional exact inference using Bayes’s rule fails fornetworks with undirected cycle [9]. Therefore, most infer-ence methods convert a network to a cycle-free hypergraphcalled a junction tree. Each node of the junction tree iscalled a clique. A potential table is used for each cliqueto express the joint probability of all variables representedby a clique. The potential table of clique Ci is denoted asψ(Ci). The details of potential tables are given in Section3.2.

The evidence in a Bayesian network is a group of ob-served variables e.g. E = {Ae1 = ae1 , · · · , Aec

= aec}

where ek ∈ {1, 2, . . . , n}. Given these evidences, we caninquire the distribution of other variables. The variables tobe inquired are called query variables. Exact inference isthe propagation of the evidence throughout the junction treefollowed by computation of the updated probability of thequery variables. Mathematically, evidence propagation canbe represented as:

ψ∗(R) =∑

Y \R

ψ∗(Y ), ψ∗(X) =ψ(X)ψ(R)

ψ∗(R) (2)

whereX is a clique andR is a separator, which is the inter-section of X and its neighbor clique Y . Y \R is defined as{r|r ∈ Y, r �∈ R}; ψ∗(·) is the updated potential function.

P(A|B, C, D) 000 001 · · · 1110 0.4321 0.8275 · · · 0.52731 0.5679 0.1725 · · · 0.4727

Table 1. An example segment of conditionalprobability distribution in table representa-tion.

3. Parallelism in Exact Inference

3.1. Probability Representation onBayesian Network Nodes

Each node of a Bayesian network denotes a random vari-able. The probability distribution of a random variable canbe expressed as a list of non-negative floating point numbersthat sum to 1, where each number indicates the probabilityin a certain state. Assuming a node A has k parents, r0states and its i-th parent has ri states, the conditional prob-ability distribution of A, P (A|pa(A)), can be expressed asa 2-dimensional table of size

∏ki=0 ri.

Table 1 is an example of a conditional probability tablewhere each random variable is binary. In Table 1, each col-umn gives the probability distribution of the variable on agiven condition set; each row indicates the probability of thevariable taking a given state under various condition sets.

3.2. Probability Representation on Junc-tion Tree Cliques

Each node of a junction tree denotes a clique, which isa set of random variables. For each clique C, there is apotential function ψ(C) which is proportional to the jointdistribution of the random variables in that clique [6]. Thediscrete form of the potential function is called a potentialtable. A potential table is a list of non-negative real num-bers where each number corresponds to a probability of thejoint distribution. Assuming a clique contains w randomvariables and the i-th variable has ri states, the length of thepotential table is

∏wi=1 ri.

The potential table of a clique is constructed from someconditional probability tables of the Bayesian network intwo steps: First, a potential table is initialized: ψ(C) =∏

i P (Ai|pa(Ai)), where Ai ∪ pa(Ai) ⊆ C. Second, theinitial potential table is updated from the root to the leavesand from the leaves to the root [6]. The second step is quitesimilar to belief propagation which will be discussed in thenext section.

A potential table can be expressed as a 2-dimensionalmatrix, as shown in Table 2. However, unlike conditionalprobability tables, there is no conditional set in a potentialtable. We use row-major ordering to index the table entries,

222222

ψ(A, B, C, D, E, F) 000 001 · · · 111000 0.0931 0.3917 · · · 0.6821001 0.2957 0.1725 · · · 0.0942· · · · · ·111 0.3402 0.2013 · · · 0.0994

Table 2. An example segment of clique poten-tial function in table representation.

e.g. in Table 2, T (9) = ψ(A = 0, B = 0, C = 1,D =0, E = 0, F = 1) = 0.1725.

3.3. Belief Propagation among NeighborNodes

3.3.1 Deriving the Potential Table of Separators

A separator is defined as the set of shared variables of thetwo cliques.The table representation of a separator can beobtained from the potential tables of adjacent cliques inthe junction tree. For example, assuming A and B aretwo adjacent cliques in a junction tree, the separator be-tween the two cliques is given by S = A ∩ B. Giventhe potential table of a clique, obtaining the potential ofits subset is termed marginalization. Marginalization is il-lustrated in Figure 1, where ψ({B,C}) is a separator po-tential table, ψ({A,B,C,D,E}) and ψ({B,C, F,G}) arethe potential tables of two adjacent cliques. For example,ψ({B = 0, C = 0}) (the gray entry in ψ({B,C})) can beobtained by either summing up the values of all gray entriesin ψ({A,B,C,D,E}) or summing up the values of all graycells in ψ({B,C, F,G}). Note that the states of B and Cin all these gray entries are 0.

Figure 1. Obtaining separator potential tablefrom clique potential tables. In each entry,the states of the binary random variables ina clique (or separator) are given.

3.3.2 Belief Propagation Between Adjacent Cliques

Assume two cliques A and B are adjacent and A has beenupdated. We want to use the updated potential table of Ato renew the potential table of B. Denote S as the separa-tor between A and B. First, we obtain the potential tableof S denoted ψ(S∗) by marginalizing the potential table ofA. Then, we obtain the potential table of S denoted ψ(S)by marginalizing the potential table of B, which is . Fi-nally, the potential table of B can be updated by dividingthe product of the entries in ψ(B) and ψ(S∗) by the en-try in ψ(S), that is, ψ(B)ψ(S∗)/ψ(S). For each entry ofψ(B), the above process can be implemented in parallel.

3.4. Node Level Primitives

In the previous section, we analyzed the node level prob-ability representation and belief propagation. Here, we in-troduce four node level primitives on these tables: tablemarginalization, table multiplication, table division, and ta-ble extension.

Table marginalization is used to obtain the potential ta-ble of a separator from the potential table of a clique. Theinput to table marginalization is a potential table ψ(A), andthe separator betweenA and one ofA’s neighbors. The out-put is the potential table of the separator. Table marginaliza-tion can be implemented in parallel, where each processorhandles a block of the clique potential table and obtains theseparator potential table by combining the results from allthe processors.

Table multiplication and table division are used in be-lief propagation among neighbor cliques. They convertmultiplication and division between two tables to multipli-cation and division between corresponding entries. The in-dependence of operations on different entries leads to a par-allel implementation.

Table extension is used to simplify multiplication anddivision of tables. It equalizes the size of two tables in-volved in multiplication or division. Multiplying entries inthe same locations of the two tables (shown in Figure 2), weobtain the result of table multiplication. Similarly, dividingtwo entries in the same locations of the two tables, if thedivisor is not zero, we obtain the result of the table division.

Table extension can benefit the parallelization of ta-ble operations. Once the tables are extended, they canbe divided into smaller chunks. For example, thereis a random vertical surface in Figure 2 which cutsboth Extended ψ({B,C}) and ψ({A,B,C,D,E}) intotwo chunks. Multiplication between ψ({B,C}) andψ({A,B,C,D,E}) can be implemented by multiplyingthe two left chunks and multiplying the two right chunksin parallel.

223223

Figure 2. Illustration of table extension. Af-ter extension, each entry of one table corre-sponds to the entry in the same location ofthe other table.

4. Parallel Algorithms for Node Level Primi-tives

4.1. Table Extension

Consider the multiplication of tables T1 and T2. Letthe random variables involved in T1 be arranged in lexico-graphic order to form a vector V1 (likewise with T2 and V2).Such a vector is called variable vector of a given table. Forexample, the variable vector of Table 2 is (a, b, c, d, e, f).We define a mapping vector MV1

V2as the corresponding rela-

tionship from V1 to V2. Specifically, if the ith element of V1

is the jth element of V2, then MV1V2

(i) = j. So, vector MV1V2

has the same length as V1. Note that each element of a vari-able vector Vi is a random variable. Suppose the kth randomvariable in Vi has (Rk +1) possible states: 0, 1, ...Rk. In anobservation, assuming the kth random variable in Vi is in thex

(i)k -th state where x(i)

k = 0, 1, · · · , Rk, we define the value

string as X(i) = (x(i)1 , ..., x

(i)|Vi|). Denote TBig (TSmall) as

one of T1 or T2 that has the larger (smaller) number of en-tries. The variable vectors are named as VBig and VSmall,respectively. Using these definitions, we formulate the al-gorithm for table extension shown in Algorithm 1.

In this algorithm, we convert scalar i, the index of TBig,into a value string X(i), so that we can find its correspond-ing element in TSmall. To convert value string Y (i) into ascalar j is given by the following equation:

j =∑

k

Y (i)(k)(Rk + 1)k (3)

Algorithm 1 Table-Extension: TExt(TSmall, VSmall, VBig)

Input: TSmall, variable vectors VSmall and VBig

Output: TBig

if VSmall == VBig thenTBig = TSmall

returnend ifif MVSmall

VBighas not been computed then

Find mapping vector MVSmall

VBigfrom VSmall and VBig

end ifInitialize TBig

for i = 1 to |TBig| in parallel doConvert i into value string X(i)

Create value string Y (i) from X(i) using MVSmall

VBig

Convert Y (i) into index jTBig(i) = TSmall(j)

end for

We analyze the complexity of this algorithm using thewell known concurrent read exclusive write parallel randomaccess machine (CREW PRAM) model [9]. This modelassumes a shared, global memory that processors can readfrom, but not write to, simultaneously.

In Algorithm 1, finding mapping vector MVSmall

VBigcosts

O(|VSmall| log(|VBig|)), where |Vi| gives the number of el-ements in Vi. With an allocated memory block, initializinga table costs constant time. Converting a scalar to a stringtakes O(|VBig|

∑iRi). Creating Y (i) and filling TBig

take O(|VBig|) time. Assuming there are p available pro-cessors, the computational complexity of this algorithm isO(|VSmall| log(|VBig|)+(|TBig| · |VBig|

∑iRi)/p), where

|Ti| is the number of entries in table Ti. Assuming there areat mostw random variables involved in a potential table andthe maximum number of states of those random variables isr, |Vi| is bounded by w and |Ti| is bounded by rw. AsO(w logw) can be ignored compared to O(rw) when r andw are large, the computational complexity can be simpli-fied to O(rw · |VBig|

∑iRi/p) where 1 ≤ p ≤ |TBig|. As

|VBig| ≤ w and∑

iRi ≤ w · r, the complexity is boundedby O(w2r(w+1)/p), 1 ≤ p ≤ |TBig|.

In Algorithm 1, each processor needs TSmall, VSmall

and VBig . We only consider VSmall ≤ VBig − 1. In thiscase, TSmall is bounded by rw−1. Each processor handlesa segment of TBig, assuming there are p processors. Thetotal length of the segment is |TBig|. Therefore, the spacecomplexity for a processor is bounded by O(rw).

4.2. Table Multiplication and Division

After we obtain the extended table, the multiplication oftwo tables is straightforward and easy to parallelize. Wefirst combine the variable vectors of T1 and T2 to form a

224224

union vector V = V1 ∪ V2. Table T1 and T2 are then ex-tended according to V . After the extension, each entry in T1

corresponds to a unique entry in T2, and vice versa. Thus,the table multiplication becomes entry-wise multiplication.The process is summarized in Algorithm 2.

Algorithm 2 Multiplication: TMul(T1, T2, V1, V2, R)Input: Potential tables T1 and T2, variable vectors V1, V2,

variable range ROutput: Table T as the multiplication of T1, T2

Create union variable vector V = V1 ∪ V2

Initialize T according to V , where |T | =∏

iRV (i)

T ′1 = TExt(T1, V1, V )T ′

2 = TExt(T2, V2, V )for i = 1 to |T | in parallel doT (i) = T ′

1(i) ∗ T ′2(i);

end for

When representing V1 and V2 as bit strings, creating theunion variable vector costs O(|V1| + |V2|). By allocatingmemory of size

∏iRV (i), initialization of T can be com-

pleted in constant time. As we have analyzed in Algo-rithm 1, extending T1 and T2 costs O(|V | log(|V |) + |T | ·|V |∑iRi/p), since |V | > |V1| and |V | > |V2|. The entry-wise multiplication requires O(|T |) time. Note that aftertable extension, both T1 and T2 have the same size |T |.The total computational complexity is O(|V | log(|V |) +(|T | · |V |∑iRi + |T |)/p) and the scalability range is1 ≤ p ≤ |T |. As |T | is also bounded by rw, when r andw are large, the computational complexity can be approxi-mated by O(w2r(w+1)/p), 1 ≤ p ≤ |T |.

Algorithm 3 Division: TDiv(T1, T2, V1, V2, R)Input: Potential table T1 and T2, variable vector V1, V2,

variable range ROutput: Table T as the quotient of T1/T2

Create union variable vector V = V1 ∪ V2

Initialize T according to V , where |T | =∏

iRV (i)

T ′1 = TExt(T1, V1, V )T ′

2 = TExt(T2, V2, V )for i = 1 to |T | in parallel do

if T ′2(i) == 0 thenT (i) = 0;

elseT (i) = T ′

1(i)/T′2(i);

end ifend for

In table multiplication, the total length of table assignedto each processor is T , bounded by O(rw). The lengthsof V1, V2 and R are all bounded by O(w), which can beignored compared to O(rw). So, the space complexity forAlgorithm 2 is O(rw).

The algorithm for table division is very similar to thatof table multiplication. In case the denominator equals 0,we simply let the result be 0, according to the definition ofconditional probability. The computational complexity andspace complexity of table division are the same as those oftable multiplication.

4.3. Table Marginalization

The input to table marginalization is a big table TBig

with variable vector VBig , and a smaller variable vectorVSmall ⊆ VBig. The output is a small table TSmall andvariable vector VSmall.

Algorithm 4 Marginalization: TMarg(TBig, VBig, VSmall)

Input: Potential table TBig , variable vector VBig andVSmall

Output: Result table TSmall whose variable vector isVSmall

if MVSmall

VBighas not been computed then

Find mapping vector MVSmall

VBigfrom VSmall to VBig

end ifInitialize TSmall by letting all entries be 0for i = 1 to |TBig| in parallel do

Convert i into value string X(i)

Create value string Y (i) from X(i) using MVSmall

VBig

Convert Y (i) into index jTSmall(j) = TSmall(j) + TBig(i)

end for

In Algorithm 4, each processor receives VBig , VSmall

and a segment of TBig of length �TBig/p. Each pro-cessor produces a segment of TSmall from the segment ofTBig . The computational complexity of marginalization isO(|VSmall| log(|VBig|) + |TBig| · |VBig|

∑iRi/p), which

can be bounded by O(w2r(w+1)/p), 1 ≤ p ≤ |TBig|.As TBig , TSmall, VBig, VSmall,M

VSmall

VBigare stored in the

memory and TBig is dominant, the space complexity isgiven by O(rw).

4.4. Exact Inference with Node Level Prim-itives

The process of exact inference with node level primitivesin junction trees is given in Algorithm 5. The input to thisalgorithm includes a junction tree with the potential tablesfor its cliques, the evidence variables and the query vari-ables. The outputs are the probability tables for the queryvariables.

In Algorithm 5, pa(Ai) is the parent ofAi in JT . T (Aj)denotes the potential table of clique Aj . V (Aj) is the vari-able vector ofAj . In arrayA, the cliques of JT are arranged

225225

by the Breadth First Search (BFS) order. E == e judgesif E is consistent with evidence e. δ(expr) is 1 if expr istrue; otherwise it is 0. Thus, T (Ai) · δ(E == e) makes allentries in T (Ai) inconsistent with e to be 0. The next twofor-loops perform evidence collection to propagate evidencefrom leaf cliques to the root, and evidence distribution topropagate evidence from the root to leaf cliques. Evidencecollection and distribution ensure the evidence is absorbedby all cliques [6]. Finally, we obtain the probability tablesof query variables by marginalizing clique potential tables.V (Q) is a variable vector consisting of query variables.

Assume the number of cliques in JT is N . The com-plexity of BFS is O(N), since JT has N cliques and N −1edges. In evidence collection and distribution, evidencepropagation from one clique to another requires two tablemarginalization, two table extensions, one table multiplica-tion and one table division. There are at most O(N) nodelevel operations. If we compute the BFS order of cliques apriori, the scalability of Algorithm 5 depends on the nodelevel primitives. Therefore, the computational complexityis O(w2r(w+1)N/p), 1 ≤ p ≤ rw.

The dominant part of Algorithm 5 consists of four for-loops. Each iteration of these for-loops only utilizes a con-stant number of primitives. As the space complexity foreach primitive is bounded by O(rw), the space complexityfor each iteration is also bounded by O(rw). Therefore, thespace complexity for Algorithm 5 is O(Nrw).

5. Implementation and Experimental Results

5.1. Computing Facilities

We implement the node level primitives using MessagePassing Interface (MPI) on the DataStar Cluster at the SanDiego Supercomputer Center (SDSC)[2] and on the clustersat the USC Center for High-Performance Computing andCommunications (HPCC)[3].

The DataStar Cluster at SDSC employs IBM P655 nodesrunning at 1.5 GHz with 2 GB of memory per processor.This machine uses a Federation interconnect, and has a the-oretical peak performance of 15 Tera-FLOPS. The DataStarCluster runs Unix with MPICH. IBM Loadleveler was usedto submit the jobs to the batch queue.

The USC HPCC is a Sun Microsystems & Dell Linuxcluster. A 2-Gigabit/second low-latency Myrinet networkconnects most of the nodes. The HPCC nodes are Dual IntelXeon 3.2 GHz with 2 GB memory, running USCLinux withMPICH and Portable Batch System (PBS).

5.2. Data Distribution

Note that we use the row-major ordering to in-dex table entries. The potential table entries are dis-

tributed among the processors by the following way: As-suming the potential table is T = (T0, · · · , T|T |−1),the k-th processor is in charge of |T |/p entries:(Tk·�|T |/p�, · · · , Tmin((k+1)·�|T |/p�,|T |−1)). V1, V2 (orVSmall, VBig), R and the potential table of the separator(T2 in Algorithm 2 and 3) are broadcast to all processors.

Algorithm 5 Exact Inference on Junction TreeInput: Junction tree JT , evidence e for variable E, query

variables Q, variable range ROutput: Probability table of QA = Order cliques of JT by BFSfor i=1 to |A| do

if E ∈ Ai thenT (Ai) = T (Ai) · δ(E == e)

end ifend forfor i = |A| to 2 by -1 doVsepset = V (pa(Ai)) ∩ V (Ai)Tsepset=TMarg(T (pa(Ai)), V (pa(Ai)), Vsepset)T ∗

sepset=TMarg(T (Ai)), V (Ai), Vsepset)Tdivsep=TDiv(T ∗

sepset, Tsepset, Vsepset, Vsepset, R)T ∗=TMul(T (pa(Ai)), Tdivsep, V (pa(Ai)), Vsepset,R)Update potential table by T (pa(Ai)) = T ∗

end forfor i = 2 to |A| doVsepset=V (pa(Ai)) ∩ V (Ai)T ∗

sepset=TMarg(T (pa(Ai)), V (pa(Ai)), Vsepset)Tsepset=TMarg(T (Ai), V (Ai), Vsepset)Tdivsep=TDiv(T ∗

sepset, Tsepset, Vsepset, Vsepset, R)T ∗=TMul(T (Ai), Tdivsep, V (Ai), Vsepset, R)Update potential table by T (Ai) = T ∗

end forfor i=1 to |A| do

if Q ∈ Ai thenTquery=TMarg(T (Ai), V (Ai), V (Q))

end ifend for

5.3. Experimental Results

To test the performance of individual node level prim-itives, we randomly generate two potential tables givenw = 15 and r = 2, 3. We use double precision andthe memory needed by a clique potential table is 256KB(r = 2) and 110MB (r = 3). The separator set we used has5 random variables, so the small table is 0.25KB (r = 2)and 2KB (r = 3). The memory taken by V1, V2 and R isignored compared to the size of the clique potential table.We ran the program with 1, 2, 4, 8, 16, 32, 64 and 128 pro-cessors. The execution times are shown in Figures 3.

226226

Figure 3. Execution time of node level primitives on DataStar and HPCC.

227227

The random Bayesian networks used for exact infer-ence were generated by the Bayesian Network Toolbox forMatlab[1]. We generated three networks with 1024 nodes.The number of states r was set as 2 and 3. As the poten-tial table grew exponentially with w, we controlled the sizeof the cliques by limiting the degree of the nodes in theBayesian networks. The maximal clique size w are 20, 14and 11 respectively when we converted these Bayesian net-works to junction trees. The number of cliques are 244,318 and 427 respectively. We performed the exact infer-ence with 1, 2, 4, 8, 16, 32, 64 and 128 processors. Theresults are shown in Figures 4.

The experimental results on two clusters show scalableperformance of our implementations of all the primitivesand of exact inference with various parameters, where theexecution time decreases as the number of processors in-creases. Such results illustrate the efficiency and scalabilityof the proposed primitives. Note that we are not concernedwith the individual speedups, which can be improved fur-ther, but the general scalability of our results.

6. Conclusions

In this paper, we categorized table-based operations inexact inference into four node level primitives: table exten-sion, table multiplication, table division and table marginal-ization. We developed scalable parallel algorithms for theseprimitives, as well as exact inference using them. As part ofour future work, we plan to study the analysis of the exactinference algorithm using a more realistic system model totake into account the message passing costs. Additionally,we will also explore the load balancing issues that mightarise in the situation when r and w are different across dif-ferent cliques, leading to different sized tables.

References

[1] The Bayesian Network Toolbox for Matlab (BNT).http://bnt.sourceforge.net/.

[2] Datastar Cluster at the San Diego Supercomputer Center(SDSC). http://www.sdsc.edu/us/resources/datastar/.

[3] USC center for High-Performance Computing and Commu-nications (HPCC). http://www.usc.edu/hpcc/.

[4] D. Heckerman. Bayesian networks for data mining. In InData Mining and Knowledge Discovery, 1997.

[5] A. V. Kozlov and J. P. Singh. A parallel lauritzen-spiegelhalter algorithm for probabilistic inference. In Su-percomputing, pages 320–329, 1994.

[6] S. L. Lauritzen and D. J. Spiegelhalter. Local computationwith probabilities and graphical structures and their appli-cation to expert systems. J. Royal Statistical Society B,50:157–224, 1988.

[7] V. K. Namasivayam, A. Pathak, and V. K. Prasanna. Scal-able parallel implementation of Bayesian network to junc-tion tree conversion for exact inference. In Proceedings of

the 18th International Symposium on Computer Architectureand High Performance Computing, pages 167–176, 2006.

[8] V. K. Namasivayam and V. K. Prasanna. Scalable parallelimplementation of exact inference in bayesian networks. InProceedings of the 12th International Conference on Paral-lel and Distributed Systems, pages 143–150, 2006.

[9] D. Pennock. Logarithmic time parallel Bayesian inference.In Proceedings of the 14th Annual Conference on Uncer-tainty in Artificial Intelligence, pages 431–438, 1998.

[10] S. J. Russell and P. Norvig. Artificial Intelligence: A ModernApproach (2nd Edition). Prentice Hall, December 2002.

[11] E. Segal, B. Taskar, A. Gasch, N. Friedman, and D. Koller.Rich probabilistic models for gene expression. In 9th In-ternational Conference on Intelligent Systems for MolecularBiology, pages 243–252, 2001.

[12] R. D. Shachter, S. K. Andersen, and P. Szolovits. Globalconditioning for probabilistic inference in belief networks.In In Proceedings of the Tenth Conference on Uncertainty inArtifficial Intelligence, pages 514–522, 1994.

[13] L. Yin, C.-H. Huang, and S. Rajasekaran. Parallel data min-ing of bayesian networks from gene expression data. InPoster Book of the 8th International Conference on Researchin Computational Molecular Biology, pages 122–123, 2004.

Figure 4. Execution time of exact inferenceon DataStar and HPCC.

228228