Square Root Graphical Models: Multivariate Generalizations ofUnivariate Exponential Families that Permit Positive Dependencies

David I. Inouye DINOUYE@CS.UTEXAS.EDUPradeep Ravikumar PRADEEPR@CS.UTEXAS.EDUInderjit S. Dhillon INDERJIT@CS.UTEXAS.EDU

Dept. of Computer Science, University of Texas, Austin, TX 78712, USA

AbstractWe develop Square Root Graphical Models(SQR), a novel class of parametric graphicalmodels that provides multivariate generalizationsof univariate exponential family distributions.Previous multivariate graphical models (Yanget al., 2015) did not allow positive dependenciesfor the exponential and Poisson generalizations.However, in many real-world datasets, variablesclearly have positive dependencies. For example,the airport delay time in New York—modeled asan exponential distribution—is positively relatedto the delay time in Boston. With this motivation,we give an example of our model class derivedfrom the univariate exponential distribution thatallows for almost arbitrary positive and negativedependencies with only a mild condition on theparameter matrix—a condition akin to the posi-tive definiteness of the Gaussian covariance ma-trix. Our Poisson generalization allows for bothpositive and negative dependencies without anyconstraints on the parameter values. We also de-velop parameter estimation methods using node-wise regressions with `1 regularization and like-lihood approximation methods using sampling.Finally, we demonstrate our exponential gener-alization on a synthetic dataset and a real-worlddataset of airport delay times.

1. IntroductionGaussian, binary and discrete undirected graphicalmodels—or Markov Random Fields (MRF)—have becomepopular for compactly modeling and studying the struc-tural dependencies between high-dimensional continuous,

Proceedings of the 33 rd International Conference on MachineLearning, New York, NY, USA, 2016. JMLR: W&CP volume48. Copyright 2016 by the author(s).

binary and categorical data respectively (Friedman et al.,2008; Hsieh et al., 2014; Banerjee et al., 2008; Raviku-mar et al., 2010; Jalali et al., 2010). However, real-worlddata does not often fit the assumption that variables comefrom Gaussian or discrete distributions. For example, wordcounts in documents are nonnegative integers with manyzero values and hence are more appropriately modeled bythe Poisson distribution. Yet, an independent Poisson dis-tribution would be insufficient because words are often ei-ther positively or negatively related to other words—e.g.the words “machine” and “learning” would often co-occurtogether in ICML papers (positive dependency) whereasthe words “deep” and “kernel” would rarely co-occur sincethey usually refer to different topics (negative dependency).Thus, a Poisson-like model that allows for dependenciesbetween words is desirable. As another example, the delaytimes at airports are nonnegative continuous values that aremore closely modeled by an exponential distribution thana Gaussian distribution but an independent exponential dis-tribution is insufficient because delays at different airportsare often related (and sometimes causally related)—e.g. ifa flight from Los Angeles, CA (LAX) to San Francisco,CA (SFO) is delayed then it is likely that the return flightof the same airplane will also be delayed. Other exam-ples of non-Gaussian and non-discrete data include high-throughput gene sequencing count data, crime statistics,website visits, survival times, call times and delay times.

Though univariate distributions for these types of data havebeen studied quite extensively, multivariate generalizationshave only been given limited attention. One basic approachto forming dependent multivariate distributions is to as-sume that the marginal distributions are exponentially dis-tributed (Marshall & Olkin, 1967; Embrechts et al., 2003)or Poisson distributed (Karlis, 2003). This idea is relatedto copula-based models (Bickel et al., 2009) in which aprobability distribution is decomposed into the univariatemarginal distributions and a copula distribution on the unithypercube that models the dependency between variables.However, the exponential model in (Marshall & Olkin,

Square Root Graphical Models

1967; Embrechts et al., 2003) gives rise to a distributionthat is composed of a continuous distribution and a singulardistribution, which seems unusual and unlikely for generalreal-world situations. The multivariate Poisson distribution(Karlis, 2003) is based on the sum of independent Pois-son variables and can only model positive dependencies.The copula versions of the multivariate Poisson distributionhave significant issues related to non-identifiability becausethe Poisson distribution has a discrete domain (Genest &Neslehova, 2007). There has also been some recent workon semi-parametric graphical models (Liu et al., 2009) thatuse Gaussian copulas to relax the assumption of Gaussian-ity but these models are not parametric and only considercontinuous real-valued data.

Another line of work assumes that the node conditionaldistributions—i.e. one variable given the values of all theother variables—are univariate exponential families1 anddetermines under what conditions a joint distribution existsthat is consistent with these node conditional distributions.Besag (1974) developed this multivariate distribution forpairwise dependencies, and Yang et al. (2015) extended thismodel to n-wise dependencies. Yang et al. (2015) also de-veloped and analyzed an M-estimator based on `1 regular-ized node-wise regressions to recover the graphical modelstructure with high probability. Unfortunately, these mod-els only allowed negative dependencies in the case of theexponential and Poisson distributions. Yang et al. (2013)proposed three modifications to the original Poisson modelto allow positive dependencies but these modifications al-ter the Poisson base distribution or require the specifica-tion of unintuitive hyperparameters. Allen & Liu (2013)allowed positive dependencies by only requiring the LocalMarkov property rather than a consistent joint distributionthat would have Global Markov properties. In a differentapproach, Inouye et al. (2015) altered the Poisson gener-alization by assuming the length of the vector is fixed orknown similar to the multinomial distribution in which thenumber of trials is known. This allows a joint distributionthat is decomposed into the marginal distribution of vectorlength and the distribution of the vector direction given thelength. While the model in (Inouye et al., 2015) allowed forboth positive and negative dependencies, the joint distribu-tion needed to be modified by an ad hoc scalar weightingfunction to avoid very low likelihood values for vectors oflong length—i.e. documents with many words.

Therefore, we develop a novel parametric generalizationof univariate exponential family distributions with non-negative sufficient statistics—e.g. Gaussian, Poisson andexponential—that allows for both positive and negative de-pendencies. We call this novel class of multivariate dis-

1See (Wainwright & Jordan, 2008) for an introduction to ex-ponential families.

tributions Square Root Graphical Models (SQR) becausethe square root function is fundamentally important as willbe described in future sections. SQR models have a sim-ple parametric form without needing to specify any hy-perparameters and can be fit using `1-regularized node-wise regressions similar to previous work (Yang et al.,2015). The independent model—e.g. independent Poissonor exponential—is merely a special case of this class unlikein (Yang et al., 2013). We show that the normalizability ofthe distribution puts little to no restriction on the values ofthe parameters, and thus SQR models give a very flexiblemultivariate generalization of well-known univariate distri-butions.

Notation Let p and n denote the number of dimensionsand number data instances respectively. We will generallyuse uppercase letters for matrices (e.g. Φ, X), boldfacelowercase letters for vectors (i.e x,φ) and lowercase let-ters for scalar values (i.e. x, φ). Let R+ denote the set ofnonnegative real numbers and Z+ denote the set of nonneg-ative integers.

2. BackgroundTo motivate the form of our model class, we present abrief background on the graphical model class as in (Be-sag, 1974; Yang et al., 2015; 2013). Let T(x) and B(x)be the sufficient statistics and log base measure respec-tively of the base univariate exponential family and letD ⊆ Rp+ be the domain of the random vector. We willdenote T(x) : Rp → Rp to be the entry-wise application ofthe sufficient statistic function to each entry in the vector x.With this notation, the previous class of graphical modelscan be defined as (Yang et al., 2015):

Pr(x|θ,Φ) = exp(θTT(x) + T(x)TΦT(x)

+∑ps=1 B(xs)− A(θ,Φ)

) (1)

A(θ,Φ) =∫D exp

(θTT(x) + T(x)TΦT(x)

+∑ps=1 B(xs)

)dµ(x) ,

(2)

where A(θ,Φ) is the log partition function (i.e. log nor-malization constant) which is required for probability nor-malization, Φ ∈ Rp×p is symmetric with zeros along thediagonal and µ is either the standard Lebesgue measure orthe counting measure depending on whether the domainD is continuous or discrete. The only difference from afully independent model is the quadratic interaction termT(x)TΦT(x)—i.e. O(T(x)2)—which is why the exponen-tial and Poisson cases do not admit positive dependenciesas will be described in later sections.

We will review the exponential instantiation of this pre-vious graphical model in which the domain D ∈ Rp+,

Square Root Graphical Models

T(x) = x and B(x) = 0. Suppose there is even one pos-itive entry in Φ denoted φst. Then as x → ∞, the posi-tive quadratic term xsφstxt will dominate the linear termθTx and thus the log partition function will diverge (i.e.A(θ,Φ) → ∞). Thus, Φst ≤ 0 is required for a consis-tent joint distribution. Similarly, in the case of the Poissondistribution where the domain D ∈ Zp+, T(x) = x andB(x) = −ln(x!), suppose there is even one positive entryφst. The quadratic term xsφstxt will dominate the linearterm and the log base measure term which is O(xln(x));thus, Φst ≤ 0 is also required for the Poisson distribution.

In an attempt to allow positive dependencies for the Pois-son distribution, Yang et al. (2013) developed three variantsof the Poisson graphical model defined above. First, theydeveloped a Truncated Poisson Graphical Model (TPGM)that kept the same parametric form but merely truncatedthe usual infinite domain to the finite domain DTPGM ={x ∈ Zp+ : xs ≤ R}. However, a user must a priori spec-ify the truncation value R and thus TPGM is unnatural fornormal count data that could be infinite. In addition, be-cause of the quadratic term, even though the domain is fi-nite, the quadratic term can dominate and push most of themass near the boundary of the domain (Yang et al., 2013).The second proposal was to change the base measure fromln(x!) to x2. This proposal, however, gives the distributionGaussian-like quadratic tails rather than the thicker tails ofthe Poisson distribution. Finally, the last proposal modi-fied the sufficient statistic T(x) to decrease from linear toconstant as x increases. Similar to TPGM, this third pro-posal requires the a priori specification of two cutoff pa-rameters (R1, R2) and behaves similarly to TPGM after thesecond cutoff point because the base measure of −ln(x!)will quickly make the probability approach 0 once the suf-ficient statistics become constant.

In a somewhat different direction, Inouye et al. (2015)proposed a variant called Fixed-Length Poisson MRF(LPMRF) that modifies the domain of the distribution as-suming the length of the vector L = ‖x‖1 is fixed, i.e.D = {x ∈ Zp+ : ‖x‖1 = L}. Because the domain is finiteas in TPGM, the distribution is normalizable even with pos-itive dependencies. However, as with TPGM, the quadraticterm in the parametric form dominates the distribution if Lis large, and thus Inouye et al. (2015) modify the distribu-tion by introducing a weighting function that decreases thequadratic term as L increases. All of these variants of thePoisson graphical model attempt to deal with the quadraticinteraction term in different ways but all of them signifi-cantly change the distribution/domain and often require thespecification of new unintuitive hyperparameters to allowfor positive dependencies. Also, according to the authors’best knowledge, no variants of the exponential graphicalmodel have been proposed to allow for positive dependen-cies. Therefore, we propose a novel graphical model class

that alleviates the problem with the quadratic interactionterm and provides both exponential and Poisson graphicalmodels that allow positive and negative dependencies.

3. Square Root Graphical ModelThe amazingly simple yet helpful change from the previousgraphical model class in Eqn. 1 is that we take the squareroot of the sufficient statistics in the interaction term. Es-sentially, this makes the interaction term linear in the suffi-cient statistics O(T(x)) rather than quadratic O(T(x)2) asin Eqn. 1. This change avoids the problem of the quadraticterm overcoming the other terms while allowing both pos-itive and negative dependencies. More formally, given anyunivariate exponential family with nonnegative sufficientstatistics T(x) ≥ 0, we can define the Square Root Graphi-cal Model (SQR) class as follows:

Pr(x |θ,Φ)=exp(θT√

T(x)+√

T(x)TΦ√

T(x)

+∑sB(xs)− A(Φ)

) (3)

A(θ,Φ)=∫D exp

(θT√

T(x)+√

T(x)TΦ√

T(x)

+∑sB(xs)

)dµ(x) ,

(4)

where√

T(x) is an entry-wise square root except whenT(x) = x2 in which case

√T(x) ≡ x.2 Figure 1 shows ex-

amples of the exponential and Poisson SQR distributionsfor no dependency, positive dependency and negative de-pendency. If θ = 0 and Φ is a diagonal matrix, then werecover an independent joint distribution so the SQR classof models can be seen as a direct relaxation of the indepen-dence assumption, similar to previous graphical models. Inthe next sections, we analyze some of the properties of SQRmodels including their conditional distributions.

3.1. SQR Conditional Distributions

We analyze two types of univariate conditional distribu-tions of the SQR graphical models. The first is the standardnode conditional distribution, i.e. the conditional distribu-tion of one variable given the values for all other variables(see Fig. 2). The second is what we will call the radial con-ditional distribution in which the unit direction is fixed butthe length of the vector is unknown (see Fig. 2). The nodeconditional distribution is helpful for parameter estimationas described more fully in Sec. 3.3. The radial conditionaldistribution is important for understanding the form of theSQR distribution as well as providing a means to succinctlyprove that the normalization constant is finite (i.e. the dis-tribution is valid) as described in Sec. 3.2.

2This nuance is important for the Gaussian SQR in Sec. 4.

Square Root Graphical Models

0

0.10 0.2

0.10.2

Independent Exponentials

0.30.3

0

0.10 0.2

0.10.2

Positive Exp. SQR

0.30.3

0

0.10 0.2

0.10.2

Negative Exp. SQR

0.30.3

Figure 1. These examples of 2D exponential SQR and Poisson SQR distributions with no dependency (i.e. independent), positivedependency and negative dependency show the amazing flexibility of the SQR model class that can intuitively model positive andnegative dependencies while having a simple parametric form. The approximate 1D marginals are shown along the edges of the plots.

Node Conditional Distributions

Radial Conditional Distributions

Figure 2. Node conditional distributions (left) are univariate prob-ability distributions of one variable assuming the other variablesare given while radial conditional distributions are univariateprobability distributions of vector scaling assuming the vector di-rection is given. Both conditional distributions are helpful in un-derstanding SQR graphical models.

Node Conditional Distribution The probability distri-bution of one variable xs given all other variables x−s =[x1, x2, . . . , xs−1, xs+1, . . . , xp] is as follows:

Pr(xs |x−s,θ,Φ) ∝exp{φssT(xs) +

(θs+2φT−s

√T(x−s)

)√T(xs)+B(xs)

},

where φ−s ∈ Rp−1 is the s-th column of Φ with the s-thentry removed. This conditional distribution can be refor-mulated as a new two parameter exponential family:

Pr(xs |x−s,θ,Φ) =

exp(η1T1(xs) + η2T2(xs) + B(xs)− Anode(η)

) (5)

Anode(η) =∫D exp

(η1T1(xs) + η2T2(xs) + B(xs)

)dµ(xs) ,

(6)

where η1 = φss, η2 = θs+2φT−s√

T(x−s), T1(x) = T(x),and T2(x) =

√T(x). Note that this reduces to the base ex-

ponential family only if η2 = 0 unlike the model in Eqn. 1which, by construction, has node conditionals in the baseexponential family. Examples of node conditional distribu-tions for the exponential and Poisson SQR can be seen inFig. 3. While these node conditionals are different from thebase exponential family and hence slightly more difficult touse for parameter estimation as described later in Sec. 3.3,the benefit of almost arbitrary positive and negative depen-dencies significantly outweighs the cost of using SQR overprevious graphical models.

Radial Conditional Distribution For simplicity, let usassume w.l.o.g. that T(x) = x.3 Suppose we condition onthe unit direction v = x

‖x‖1 of the sufficient statistics butthe scaling of this unit direction z = ‖x‖1 is unknown. Wecall this the radial conditional distribution:

Pr(x = zv |v,θ,Φ)

∝ exp(θT√zv +

√zv

TΦ√zv +

∑s B(zvs)

)∝ exp

((θT√v)√z +

(√vT

Φ√v)z +

∑s B(zvs)

).

3If T is not linear than we can merely reparameterize the dis-tribution so that this is the case.

Square Root Graphical Models

0 1 2 30

0.5

1

1.5

2

22=0.00

Positive 22

Negative 22

0 5 100

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

22=0.00

Positive 22

Negative 22

Figure 3. Examples of the node conditional distributions of expo-nential (left) and Poisson (right) SQR models for η2 = 0, η2 > 0and η2 < 0.

The radial conditional distribution can be rewritten as a uni-variate exponential family:

Pr(z |v,θ,Φ) = exp(η1z + η2

√z︸ ︷︷ ︸

O(z)

+ Bv(z)︸ ︷︷ ︸O(B(z))

−Arad(η))

(7)

Arad(η) =

∫D

exp(η1z + η2

√z︸ ︷︷ ︸

O(z)

+ Bv(z)︸ ︷︷ ︸O(B(z))

)dµ(z) , (8)

where η1 =√vT

Φ√v, η2 = θT

√v and Bv(z) =∑

s B(zvs). Note that if the log base measure of the baseexponential family is zero B(x) = 0, then the radial con-ditional is the same as the node conditional distribution be-cause the modified base measure is also zero Bv(z) = 0.If both θ = 0 and B(x) = 0, this actually reduces tothe base exponential family. For example, the exponen-tial distribution has B(x) = 0, and thus if we set θ = 0,the radial conditional of an exponential SQR is merelythe exponential distribution. Other examples with a logbase measure of zero include the Beta distribution and thegamma distribution with a known shape. For distributionsin which the log base measure is not zero, the distributionwill deviate from the node conditional distribution basedon the relative difference between B(x) and Bv(x). How-ever, the important point even for distributions with non-zero log base measures is that the terms in the exponentgrow at the same rate as the base exponential family—i.e.O(z) + O(B(z)). This helps to ensure that the radial con-ditional distribution is normalizable even as z → ∞ sincethe base exponential family was normalizable. As an ex-ample, the Poisson distribution has the log base measureB(x) = −ln(x!) and thus Bv(x) is O(−xlnx) whereas theother terms η1z + η2

√z are only O(z). This provides the

intuition of why the Poisson SQR radial distribution is nor-malizable as will be explained in Sec. 4.2.

3.2. Normalization

Normalization of the distribution was the reason for thenegative-only parameter restrictions of the exponential andPoisson distributions in the previous graphical models (Be-sag, 1974; Yang et al., 2015) as defined in Eqn. 1. However,we show that in the case of SQR models, normalizationis much simpler to achieve and generally puts little to norestriction on the value of the parameters—thus allowingboth positive and negative dependencies. For our deriva-tions, let V = {v : ‖v‖1 = 1,v ∈ Rp+} be the set of unitvectors in the positive orthant. The SQR log partition func-tion A(Φ) can be decomposed into nested integrals over theunit direction and the one dimensional integral over scal-ing, denoted z:

A(θ,Φ) = ln

∫V

∫Z(v)

exp(θT√zv +

√zv

TΦ√zv (9)

+∑s

B(zvs))

dµ(z) dv

= ln

∫V

∫Z(v)

exp(η1(v)z+η2(v)√z+∑s

B(zvs))dµ(z) dv,

(10)

where Z(v) = {z ∈ R+ : zv ∈ D}, and µ and D are de-fined as in Eqn. 2. Because V is bounded, we merely needthat the radial conditional distribution is normalizable (i.e.Arad(η) < ∞ from Eqn. 8) for the joint distribution to benormalizable. As suggested in Sec. 3.1, the radial condi-tional distribution is similar to the base exponential familyand thus likely only has similar restrictions on parametervalues as the base exponential family. In Sec. 4, we giveexamples for the exponential SQR and Poisson SQR dis-tributions showing that this condition can be achieved withlittle or no restriction on the parameter values.

3.3. Parameter Estimation

For estimating the parameters Φ and θ, we follow the ba-sic approach of (Ravikumar et al., 2010; Yang et al., 2015;2013) and fit p `1-regularized node-wise regressions us-ing the node conditional distributions described in Sec. 3.1.Thus, given a data matrix X ∈ Rp×n we attempt to opti-mize the following convex function:

arg minΦ

− 1n

∑s

∑i

(η1sixsi + η2si

√xsi

+B(xsi)− Anode(η1si, η2si))

+ λ‖Φ‖1,off ,

(11)

where η1si = φs,s, η2si = θ+2φT−s√

T(x−si), ‖Φ‖1,off =∑s 6=t |φst| is the `1-norm on the off diagonal elements and

λ is a regularization parameter. Note that this can be triv-ially parallelized into p independent sub problems which

Square Root Graphical Models

allows for significantly faster computation as in (Inouyeet al., 2015). Unlike previous graphical models (Yang et al.,2015) that were known to have closed-form solutions to thenode conditional log partition function, the main difficultyfor SQR graphical models is that the node conditional logpartition function Anode(η) is not known to have a closedform in general.

For the particular case of exponential SQR models, thereis a closed-form solution for Anode using the error functionas will be seen in Sec. 4.1 on exponential SQR models.More generally, because Anode is merely a one dimensionalsummation or integral, standard numerical approximationssuch as Gaussian quadrature could be used. Similarly, thegradient of∇Anode could be numerically approximated by:

∇Anode = 1ε

[(A(η1 + ε, η2)− A(η1, η2)

),(

A(η1, η2 + ε)− A(η1, η2))],

(12)

where ε is a small step such as 0.001. Notice that tocompute the function value and the gradient, only three1D numerical integrations are needed. Another significantspeedup that could be explored in future work would beto use a Newton-like method as in (Hsieh et al., 2014; In-ouye et al., 2015), which optimize a quadratic approxima-tion around the current iterate. Because these Newton-likemethods only need a small number of Newton iterations toconverge, the number of numerical integrations could bereduced significantly compared to gradient descent whichoften require thousands of iterations to converge.

3.4. Likelihood Approximation

We use Annealed Importance Sampling (AIS) (Neal, 2001)similar to the sampling used in (Inouye et al., 2015) forlikelihood approximation. In particular, we need to approx-imate the SQR log partition function A(θ,Φ) as in Eqn. 4.First, we derive a slice sample for the node conditionals inwhich the bounds for the slice can be computed in closedform. Second, we use the slice sampler to develop a Gibbssampler for SQR models. Finally, we derive an annealedimportance sampler (Neal, 2001) using the Gibbs sampleras the intermediate sampler by linearly combining the off-diagonal part of the parameter matrix Φoff with the diag-onal part Φdiag—i.e. Φ = γΦoff + Φdiag. We also mod-ify θ = γθ similarly. For each successive distribution,we linearly change γ from 0 to 1. Thus, we start by sam-pling from the base exponential family independent distri-bution

∏ps=1 Pr(x | η1s = φss, η2s = 0) and slowly move

towards the final SQR distribution Pr(x |θ,Φ). We main-tain the sample weights as defined in (Neal, 2001) and fromthese weights, we can compute an approximation to the logpartition function (Neal, 2001).

4. Examples from Various ExponentialFamilies

We give several examples of SQR graphical models in thefollowing sections (however, it should be noted that wehave been developing a class of graphical models for anyunivariate exponential family with nonnegative sufficientstatistics). The main analysis for each case is determiningwhat conditions on the parameter matrix Φ allow the jointdistribution to be normalized. As described in Sec. 3.2, forSQR models, this merely reduces to determining when theradial conditional distribution is normalizable. We analyzethe exponential and Poisson cases in later sections but firstwe give examples of the discrete and Gaussian SQR graph-ical models.

The discrete SQR graphical model—including the binaryIsing model—is equivalent to the standard discrete graph-ical model because the sufficient statistics are indicatorfunctions Ts(x) = I(x = s),∀s 6= p and the square rootof an indicator function is merely the indicator function.Thus, in the discrete case, the discrete graphical model in(Ravikumar et al., 2010; Yang et al., 2015) is equivalent tothe discrete SQR graphical model. For the Gaussian dis-tribution, we can use the nonnegative Gaussian sufficientstatistic T(x) = x2. Thus, the Gaussian SQR graphicalmodel is merely Pr(x|Φ) ∝ exp(θTx + xTΦx), whichby inspection is clearly the standard Gaussian distributionwhere θ = Σ−1µ and Φ = − 1

2Σ−1 is required to be neg-ative definite.4 Thus, the Gaussian graphical model can beseen as a special case of SQR graphical models.

4.1. Exponential SQR Graphical Model

We consider what are the required conditions on the param-eters θ and Φ for the exponential SQR graphical model. Ifη1 is positive, the log partition function will diverge be-cause even the end point limz→∞ exp(η1z) → ∞. On theother hand, if η1 is negative, then the radial conditional dis-tribution is similar in form to the exponential distributionand thus the log partition function will be finite becausethe negative linear term η1z dominates in the exponent asz →∞.5 See appendix for proof. Thus, the basic conditionon Φ is:

ΦExp ∈ {Φ :√vT

Φ√v < 0,∀v ∈ V} . (13)

Note that this allows both positive and negative dependen-cies. A sufficient condition is that Φ be negative definite—

4This is by the slightly nuanced definition of the square rootoperator in Eqn. 3 and 4 such that

√x2 ≡ x rather than |x|.

5On the edge case when η1 = 0, the log partition functionwill diverge if η2 ≥ 0 and will converge if η1 < 0 by simplearguments. The normalizability condition when η2 = 0 couldslightly loosen the condition on Φ in Eqn. 13 but for simplicitywe did not include this edge case.

Square Root Graphical Models

as is the case for Gaussian graphical models. However,negative definiteness is far from necessary because we onlyneed negativity of the interaction term for vectors in thepositive orthant. It may even be possible for Φ to positivedefinite but Eqn. 13 be satisfied; however, we have not ex-plored this idea.

For fitting the SQR model, the node conditional log parti-tion function AExp(η) has a closed-form solution:

AExp(η) =ln

(√πη1 exp(−η22

4η1

)(1−erf

( −η22√−η1

))−2(−η1)

32

− 1

η1

),

where erf(·) is the error function. The erf function showsup because of an initial substitution of u =

√x to transform

the exponent into a quadratic form. Note that η1 < 0 bythe condition on ΦExp in Eqn. 13 above. The derivativesof AExp can also be computed in closed form for use in theparameter estimation algorithm.

4.2. Poisson SQR Graphical Model

The normalization analysis for Poisson SQR graphicalmodel is also relatively simple but requires a more care-ful analysis than the exponential SQR graphical model.Let us consider the form of the Poisson radial conditional:Prrad(z |v) ∝ exp(η1z + η2

√z −

∑s ln((zvs)!)). Note

that the domain of z, denoted Dz = {z ∈ Z+ : zv ∈ Zp+},is discrete. We can simplify the analysis by taking a largerdomain Dz = {z ∈ Z+} of all non-negative integers andchanging the log factorial to the smooth gamma function,i.e.∑s ln((zvs)!))→

∑s ln(Γ(zvs+1)). Thus, the radial

conditional log partition function is upper bounded by:∑z∈Z+

exp(η1z + η2

√z︸ ︷︷ ︸

O(z)

−∑sln(Γ(zvs + 1))︸ ︷︷ ︸

O(zlnz)

)<∞. (14)

The basic intuition is that the exponent has a linear O(z)term minus an O(zlnz) term, which will eventually over-come the linear term and hence the summation will con-verge. Note that we did not assume any restrictions on Φexcept that all the entries are finite. Thus, for the Pois-son distribution, Φ can have arbitrary positive and negativedependencies. A formal proof for Eqn. 14 is given in theappendix.

5. Experiments and Results5.1. Synthetic Experiment

In order to show that our parameter estimation algorithmhas the ability to find the correct dependencies, we de-velop a synthetic experiment on chain-like graphs. We con-struct Φ to be a k-dependent circular chain-like graph byfirst setting the diagonal of Φ to be 1. Then, we add an

edge between each node and its k neighbors with a valueof 0.9

2∗k , i.e. the s-th node is connected to the (s + 1)-th, (s + 2)-th, . . . , (s + k)-th nodes where the indicesare modulo p (e.g. k = 1 is the standard chain graph).This ensures that Φ is negative definite by the Gershgorindisc theorem. We generate samples using Gibbs samplingwith 1000 Gibbs iterations per sample and 10 slice sam-ples for each node conditional sample. For this experi-ment, we set p = 30, λ = 10−5, k ∈ {1, 2, 3, 4}, andn ∈ {100, 200, 400, 800, 1600}. We calculate the edgeprecision for the fitted model by computing the precisionfor the top kp edges—i.e. the number of true edges in thetop kp estimated edges over the total number of true edges.The results in Fig. 4 demonstrate that our parameter esti-mation algorithm is able to easily find the edges for small kand is even able to identify the edges for large k, though theproblem becomes more difficult when k is large (becausethere are more parameters, which are also smaller), andthus more samples are needed. With 1,600 samples, ourparameter estimation algorithm is able to recover at least95% of the edges even when k = 4.

5.2. Airport Delay Times Experiment

In order to demonstrate that the SQR graphical model classis more suitable for real-world data than the graphical mod-els in (Yang et al., 2015) (which can only model nega-tive dependencies), we fit an exponential SQR model to adataset of airport delay times at the top 30 commercial USAairports—also known as Large Hub airports. We gatheredflight data from the US Department of Transportation pub-lic “On-Time: On-Time Performance” database6 for theyear 2014. We calculated the average delay time per dayat each of the top 30 airports (excluding cancellations).

For our implementation, we set λ ∈ {0.05, 0.005, 0.0005}and set a maximum of 5000 iterations for our proxi-mal gradient descent algorithm. For approximating thelog partition function using the AIS sampling defined inSec. 3.4, we sampled 1000 AIS samples with 100 anneal-ing distributions—i.e. γ took 100 values between 0 and1—, 10 Gibbs steps per annealed distribution and 10 slicesamples for every node conditional sampling. Generally,our algorithm with these parameter settings took roughly35 seconds to train the model and about 25 seconds to com-pute the likelihood (i.e. AIS sampling) using MATLABprototype code on the TACC Maverick cluster.7

We computed the geometric mean of the relative log likeli-hood compared to the independent exponential model, i.e.exp((LSQR−LInd)/n), whereL is the log likelihood. Thesevalues can be seen in Fig. 4 (higher is better). Clearly, the

6http://www.transtats.bts.gov/DL SelectFields.asp?Table ID=236&DB Short Name=On-Time

7https://portal.tacc.utexas.edu/user-guides/maverick

Square Root Graphical Models

exponential SQR model provides a major improvement inrelative likelihood over the independent model suggestingthat the delay times of airports are clearly related to oneanother. In Fig. 5, we visualize the non-zeros of Φ—whichcorrespond to the edges in the graphical model—to showthat our model is capturing intuitive positive dependencies.

0

5

10

15

20

25

30

35

40

Ind. Exp. SQR (0.05)

SQR (0.005)

SQR (0.0005)

Me

an L

ike

liho

od

Im

pro

vem

en

t

0

0.2

0.4

0.6

0.8

1

1.2

1 2 3 4

Number of Edges per Node (k)

Edge Precision for k-Chain n=100 n=200 n=400 n=800 n=1600

Figure 4. (Left) The fitted exponential SQR model improves sig-nificantly over the independent exponential model in terms of rel-ative likelihood suggesting that a model with positive dependen-cies is more appropriate. (Right) The edge precision for the cir-cular chain graph described in Sec. 5.1 demonstrate that our pa-rameter estimation algorithm is able to effectively identify edgesfor small k, and if given enough samples, can also identify edgesfor larger k.

Figure 5. Visualizing the top 50 edges between airports shows thatSQR models can capture interesting and intuitive positive depen-dencies even though previous exponential graphical models (Yanget al., 2015) were restricted to negative dependencies. The delaysat the Chicago airports seem to affect other airports as would beexpected because of Chicago weather delays. Other dependenciesare likely related to weather or geography. (For this visualization,we set λ = 0.0005. Width of lines is proportional to the value ofthe edge weight, i.e. a non-zero in Φ, and the size of airport ab-breviation is proportional to the average number of passengers.)

First, it should be noted that all the dependencies are pos-itive yet positive dependencies were not allowed by previ-ous graphical models (Yang et al., 2015)! Second, as wouldbe expected because of weather delays, the airports in theChicago area seems to affect the delays of many other air-ports. Similarly, a weather effect seems to be evident forthe airports near New York City. Third, as would be ex-

pected, some dependencies seem to be geographic in natureas seen by the west coast dependencies, Texas dependency(i.e. DFW-IAH), and east coast dependencies. Note thatthe geographic dependencies were found even though nolocation data was given to the algorithm. Fourth, the busi-est airport in Atlanta, GA (ATL) is not strongly dependenton other airports. This seems reasonable because Atlantararely has snow and there are few major airports geograph-ically close to Atlanta. These qualitative results suggestthat the exponential SQR model is able to capture multipleinteresting and intuitive dependencies.

6. DiscussionAs full probability models, SQR graphical models couldbe used in any situation where a multivariate distributionis required. For example, SQR models could be usedin Bayesian classification by modeling the probability ofeach class distribution instead of the classical Naive Bayesassumption of independence. As another example, SQRmodels could be used as the base distribution in mix-tures or admixture composite distributions as in (Inouyeet al., 2014; Inouye et al.)—similar to multivariate Gaus-sian mixture models. Another extension would be to con-sider mixed SQR graphical models in which the joint distri-bution has variables using different exponential families asbase distributions as explored for previous graphical mod-els in (Yang et al., 2014; Tansey et al., 2015).

7. ConclusionWe introduce a novel class of graphical models that createsmultivariate generalizations for any univariate exponen-tial family with nonnegative sufficient statistics—includingGaussian, discrete, exponential and Poisson distributions.We show that SQR graphical models generally have few re-strictions on the parameters and thus can model both posi-tive and negative dependencies unlike previous generalizedgraphical models as represented by (Yang et al., 2015). Inparticular, for the exponential SQR model, the parametermatrix Φ can have both positive and negative dependen-cies and is only constrained by a mild condition—akin tothe positive-definiteness condition on Gaussian covariancematrices. For the Poisson distribution, there are no restric-tions on the parameter values, and thus the Poisson SQRmodel allows for arbitrary positive and negative dependen-cies. We develop parameter estimation and likelihood ap-proximation methods and demonstrate that the SQR modelindeed captures interesting and intuitive dependencies bymodeling both synthetic datasets and a real-world datasetof airport delays. The general SQR class of distributionsopens the way for graphical models to be effectively usedwith non-Gaussian and non-discrete data without the unin-tuitive restriction to negative dependencies.

Square Root Graphical Models

AcknowledgementsThis work was supported by NSF (DGE-1110007,IIS-1149803, IIS-1447574, IIS-1546459, DMS-1264033,CCF-1320746) and ARO (W911NF-12-1-0390).

ReferencesAllen, G. I. and Liu, Z. A local Poisson graphical model for

inferring networks from sequencing data. IEEE Trans.on Nanobioscience, 12(3):189–198, 2013.

Banerjee, O., El Ghaoui, L., and D’Aspremont, A. Modelselection through sparse maximum likelihood estimationfor multivariate Gaussian or binary data. JMLR, 9:485–516, 2008.

Besag, J. Spatial interaction and the statistical analysis oflattice systems. Journal of the Royal Statistical Society.Series B (Methodological), 36(2):192–236, 1974.

Bickel, P., Diggle, P., Fienberg, S., Gather, U., and Olkin,I. Copula Theory and Its Applications. Springer, 2009.

Embrechts, P., Lindskog, F., and Mcneil, A. Modellingdependence with copulas and applications to risk man-agement. In Handbook of Heavy Tailed Distributions inFinance, pp. 329–384. Elsevier, 2003.

Friedman, J., Hastie, T., and Tibshirani, R. Sparse inversecovariance estimation with the graphical lasso. Biostatis-tics, 9(3):432–441, jul 2008.

Genest, C. and Neslehova, J. A primer on copulas for countdata. ASTIN Bulletin, 37(2):475–515, 2007.

Hsieh, C-J., Sustik, M. A., Dhillon, I. S., and Ravikumar,P. QUIC: Quadratic approximation for sparse inversecovariance estimation. JMLR, 15:2911–2947, 2014.

Inouye, D. I., Ravikumar, P., and Dhillon, I. S. Captur-ing semantically meaningful word dependencies with anadmixture of Poisson MRFs. In NIPS, pp. 3158–3166.

Inouye, D. I., Ravikumar, P., and Dhillon, I. S. Admixtureof Poisson MRFs: A topic model with word dependen-cies. In ICML, 2014.

Inouye, D. I., Ravikumar, P., and Dhillon, I. S. Fixed-lengthPoisson MRF: Adding dependencies to the multinomial.In NIPS, pp. 3195–3203, 2015.

Jalali, A., Ravikumar, P., Vasuki, V., and Sanghavi, S. Onlearning discrete graphical models using group-sparseregularization. In AISTATS, pp. 378–387, 2010.

Karlis, D. An EM algorithm for multivariate Poisson distri-bution and related models. Journal of Applied Statistics,30(1):63–77, 2003.

Liu, H., Lafferty, J., and Wasserman, L. The nonpara-normal: Semiparametric estimation of high dimensionalundirected graphs. JMLR, 10:2295–2328, 2009.

Marshall, A. W. and Olkin, I. A multivariate exponentialdistribution. Journal of the American Statistical Associ-ation, 62(317):30–44, 1967.

Neal, R. M. Annealed importance sampling. Statistics andComputing, 11(2):125–139, 2001.

Ravikumar, P., Wainwright, M., and Lafferty, J. High-dimensional Ising model selection using L1-regularizedlogistic regression. The Annals of Statistics, 38(3):1287–1319, jun 2010.

Tansey, W., Padilla, O. H. M., Suggala, A. S., and Raviku-mar, P. Vector-space Markov random fields via exponen-tial families. In ICML, 2015.

Wainwright, M. J. and Jordan, M. I. Graphical models,exponential families, and variational inference. Foun-dations and Trends in Machine Learning, 1(12):1–305,2008.

Yang, E., Ravikumar, P., Allen, G. I., and Liu, Z. On Pois-son graphical models. In NIPS, pp. 1718–1726, 2013.

Yang, E., Baker, Y., Ravikumar, P., Allen, G. I., and Liu,Z. Mixed graphical models via exponential families. InAISTATS, number 2012, pp. 1042–1050, 2014.

Yang, E., Ravikumar, P., Allen, G. I., and Liu, Z. On graph-ical models via univariate exponential family distribu-tions. JMLR, 16:3813–3847, 2015.