On an Additive Semigraphoid Model for Statistical Networks ...stat.snu.ac.kr/mvstat/Download/2016_09-12/On an Additive Semigraphoid...Networks With Application to Pathway Analysis

On an Additive Semigraphoid Model for StatisticalNetworks With Application to Pathway Analysis - Bing

Li, Hyunho Chun & Hongyu Zhao

Kim Youngrae

SNU Stat. Multivariate Lab

Nov 25, 2016

Kim Youngrae (SNU Stat. Multivariate Lab) On an Additive Semigraphoid Model for Statistical Networks With Application to Pathway Analysis - Bing Li, Hyunho Chun & Hongyu ZhaoNov 25, 2016 1 / 38

Outline

1 IntroductionGGM, GCGM

2 Additive conditional independentGraphoid, semigraphoidConditional independent, additively conditional independentRelation with previous models

3 EstimationACCO, APOCalculationTuning parameter

4 Simultation results


1. Introduction

Additive semigraphoid model

Gaussian copula graphical model

Additively conditional independent

Additive conditional covariance Operator

Additive precision operator


1.1 GGM, GCGM

Gaussian graphical model

Let X = (X1, ...,Xp)T , Suppose X ∼ N(µ,Σ), and Σ is postive definite.

Let Θ = Σ−1, Θij be the (i, j)th elements of Θ.

Then X follows a GGM with respect to the graph G = Γ,E iff

Xi ⊥Xj ∣X−(i,j), (i, j) ∉ E

Multivariate normal assumption.

In Gaussian graphical model, Xi ⊥Xj ∣X(i,j) iff Θij = 0.


1.1 GGM, GCGM (Cont.)

Copula

Copula is multivariate probability distribution for which the marginalprobability of each variable is uniform.

It can deal with the variables with dependency.

C ∶ [0,1]d → [0,1], is a d−dimensional copula if C is cumulative distributionfunction of a d−dimensional random vector on the unit cube


1.1 GGM, GCGM (Cont.)

Gaussian copula graphical model

Let X = (X1, ...,Xp)T , Suppose the existence of unknown injections f1, ..., fpsuch that (f1(X1), ..., fp(Xp))T is multivariate gaussian.

(f1(X1), ..., fp(Xp))T ∼ N(µ,Σ)

Each Xi shouldn’t be a Gaussian.

We can use all of GGM procedures for non-gaussian Xis.

But the Gaussian copula assumpotion can be violated even for commonlyused interactions.

ex. marginally normal doesn’t imply jointly normal.


2.1 Graphoid, semigraphoid

Graphoid Axioms

A three-way relation R is called a graphoid if it satisfies the followingconditions.

Symmetry: (A,C,B) ∈R⇒ (B,C,A) ∈RDecomposition: (A,C,B ∪D) ∈R⇒ (A,C,B) ∈RWeak Union: (A,C,B ∪D) ∈R⇒ (A,C ∪B,D) ∈RContraction: (A,C ∪B,D) ∈R, (A,C,B) ∈R⇒ (A,C,B ∪D) ∈RIntersection: (A,C ∪D,B) ∈R, (A,C ∪B,D)⇒ (A,C,B ∪D) ∈R

Conditional indenpendent satisfies this axioms.


2.1 Graphoid, semigraphoid (Cont.)

Semigraphoid Axioms

A three-way relation R is called a semigraphoid if it satisfies the conditions ofGraphoid axioms except Intersection condition.

Symmetry: (A,C,B) ∈R⇒ (B,C,A) ∈RDecomposition: (A,C,B ∪D) ∈R⇒ (A,C,B) ∈RWeak Union: (A,C,B ∪D) ∈R⇒ (A,C ∪B,D) ∈RContraction: (A,C ∪B,D) ∈R, (A,C,B) ∈R⇒ (A,C,B ∪D) ∈R

Using this semigraphoid relation in place of probabilistic conditionalindependence can free us from some awkward restrictions.


2.2 Conditional independent, additively conditionalindependent

Notations

Let X(X1, ...,Xp)T be a random vector.

For subvector U = (U1, ..., Up)T of X, let L2(PU) be the class of functions ofU such that Ef(U) = 0,Ef2(U) <∞.

For each U i, let AUi denote a subset of L2(PUi).

Let AU denote the addtive family

AU1 + ... +AUr = f1 + ... + fr ∶ f1 ∈ AU1 , ..., fr ∈ AUr

In this paper, inner product < ., . > represents the L2(PX)−inner product.

< f, g >= ∫ f(x)g(x)dµ(x)


2.2 Conditional independent, addtively conditionalindependent (Cont.)

Additively conditional independent

We say that U and V are additively conditional independent (ACI) give W iff

(AU +AW ) ∩A⊥W ⊥ (AV +AW ) ∩A⊥WAnd we write this relation as U ⊥A V ∣W .

Additively conditional independent is three-way relation.

U ⊥A V ∣W ⇔ (AU ,AW ,AV )


2.2 Conditional independent, addtively conditionalindependent (Cont.)

Theorem

The additive conditional independece relation is a semigraphoid.

Additive semigraphoid model (ASG model)

A random vector X follows and ASG model with respect to a graphG = (Γ,E) iff

Xi ⊥A Xj ∣X−(i,j)⇔ (i, j) ∉ E.We write this condition as X ∼ ASG(G).


2.3 Relation with previous models

Now we investigate the relation between new-concept, Additive conditionalindependence and conditional indepdence.

Theorem

Suppose

(a) X has a Gaussian copula distribution with copula functions f1, ..., fp

(b) U,V, and W are subvectors of X

(c) AXi = spanfi, i = 1, ..., p

Then, U ⊥A V ∣W iff U ⊥ V ∣W .

Under Gaussian copula assumption with AXi = spanfi, addtive conditionalindependence is equivalent to conditional independence.


2.3 Relation with previous models (Cont.)

Theorem

Suppose

(a) X has a Gaussian copula distribution with copula functions f1, ..., fp

(b) U,V, and W are subvectors of X

(c) AXi = L2(PXi), i = 1, ..., p

Then, U ⊥A V ∣W implies U ⊥ V ∣W .

In this case, they are not equivalent.However, equivalence holds apploximately.



The following proposition suggests implication U ⊥ V ∣W to U ⊥A V ∣W holdsapproximately.

Proposition

Suppose Gaussian copula assumption, and WLOG,E[fi(Xi)] = 0, V ar[fi(Xi)] = 1.Let U = fi(Xi), V = fj(Xj),W = fk(Xk) ∶ k ≠ i, j, andRVW = cov(V,W ),RWW = V ar(W ),RWU = cov(W,U), ρUV = cor(U,V ).Then,

ρUV ∣W ≤max∣ραUV −R⊙αVW (R⊙αWW )−1R⊙αWU ∣ ∶ α = 1,2, ...

where A⊙α is the α−fold Hadamard product.



τUV ∣W is upper bound of ρUV ∣W in previous proposition.

These are calculated under assumption that is in the proposition plusU ⊥ V ∣W .

The small numbers of τUV ∣W in the table indicate ρUV ∣W is approximately 0.

So, U ⊥ V ∣W implies U ⊥A V ∣W approximately.


3.1 ACCO, APO

In the GGM, Xi ⊥Xj ∣X−(i,j) iff (i, j) ∉ E, and with precision matrix Θ, wecan cut the edge easily.

Building the graph in ASG, we introduce two linear operator.

Additive conditional covariance operatorThe ACCO is operator from AV to AU such that

ΣUV ∣W = ΣUV −ΣUWΣWV

. where ΣUV ∶ AV → AU such that < f,ΣUV g >= E[f(U)g(V )].

We call the operator the ACCO from V to U given W


3.1 ACCO, APO (Cont.)

With this operator, there are two important results.

LemmaLet PAW

∶ AX → AW be the projection on to AW .Then PAW

∣AU = ΣWU .

TheoremU ⊥A V ∣W iff ΣUV ∣W = 0.



Calculating ACCO is difficult because if W is changed, you should calculateagain.

Similar to GGM, we want to use precision matrix instead of conditionalcovariance.

For defining such operator, we will use different inner product.

Let A⊕X be the Hilbert space consisting of the same set of functions as AX ,

but with the inner product

< f1 + ... + fp, g1 + ... + gp >A⊕X=

p

∑i=1

< fi, gi >AXi



Additive precision operator

The operator Υ ∶ A⊕X → A⊕X defined by the matrix of operator

ΣXiXj ∶ i ∈ Γ, j ∈ Γ is called additive covariance operator. If it is invertiblethen its inverse Θ is called the APO.

Υ = ΣXiXj ∶ i ∈ Γ, j ∈ Γ meansWhen Υfi = h1 + ... + hp, ΣXjXi(fi) = hj .

We can easily check for any f, g ∈ A⊕X ,

< f,Υg >A⊕X= cov[f(X), g(X)] =< f, g >



Question. Is APO well defined?

PropositionSuppose Υ is invertible and, for each i ≠ j,Υij is compact. Then, Υ−1 isbounded.

LemmaSuppose that T ∈ B(A

⊕X ,A⊕X) is a self adjoint, positive definite operator.Then the operatorsTUU , TV V , TUU − TUV T −1V V TV U , TV V − TV UT −1UUTUVare bounded, self adjoint, and positive definite, and the following identitieshold

(T −1)UU = (TUU − TUV T −1V V TV U)−1 (1)

(T −1)V V = (TV V − TV UT −1UUTUV )−1 (2)

(T −1)UV = −(T −1)UUTUV T −1V V = −T −1UUTUV (T −1)V V (3)

(T −1)V U = −(T −1)V V TV UT −1UU = −T −1V V TV U(T −1)UU (4)



The above lemma shows that if Υ has bounded inverse, Θ is well defined.

And finally we derive this equivalence.

TheoremSuppose (U,V,W ) is a partition of X and Υ is invertible.Then, U ⊥A V ∣W iff ΘUV = 0.

CorollaryA random vector X follows an ASG(G) model if and only if ΘXiXj = 0whenever (i, j) ∉ E.


3.2 Calculation

First, how we build a graph with ACCO.

If sample estimate of ∥ΣXiXj ∣X−(i,j)∥ is small enough, we determine there is

no edge between Xi and Xj .

We should derive the finite-sample matrix representation of the operator.


3.2 Calculation (Cont.)

NotationsFor each i = 1, ..., p, let κi ∶ ΩXi ×ΩXi → R be a positive kernel. Let

AXi = spanκ1(.,Xi1) −Enκi(Xi,Xi

1), ..., κi(.,Xin) −Enκi(Xi,Xi

n)

where Enκi(Xi,Xik) = n−1∑

nl=1 κi(Xi

l ,Xik).

Q = In − 1n1Tn /n ; Projection matrix

Let κi be the vector valued function

xi → (κi(xi,Xi1) −Enκi(Xi,Xi

1), ..., κi(xi,Xin) −Enκi(Xi,Xi

n))

With this κi, for any f ∈ AXi can be expressed as f = κi[f].



With these notations, we can get the followings

(f(Xi1), ..., f(Xi

n))T = QKi[f] (5)

< f, g >AXi= n−1(QKi[f])T (QKi[g]) = n−1[f]TKiQKi[g] (6)

where Ki = κi(Xik,X

il ) ∶ k, l = 1, ..., n, f, g ∈ AXi .

K−(i,j) denoted n(p − 2) × n matrix obtained by removing ith and jth block of

(K1, ...,Kp)T .

Lemma

KiQKi[ΣXiXj ] =KiQKj (7)

KiQKi[ΣXiX−(i,j)] =KiQKT−(i,j) (8)

K−(i,j)QKT(−i,j)[ΣX−(i,j)Xi] =K−(i,j)QKi (9)



KiQKi, K−(i,j)QK−(i,j) are singular. We can solve it by Tychonoffregularization.

[ΣXiXj ] = (KiQKi + ε1In)−1KiQKj (10)

[ΣXiX−(i,j)] = (KiQKi + ε1In)−1KiQKT−(i,j) (11)

[ΣX−(i,j)Xi] = (K−(i,j)QKT(−i,j) + ε2In(p−2))−1K−(i,j)QKi (12)

We can represent

[ΣXjXi∣X−(i,j)] = [ΣXjXi] − [ΣXjX−(i,j)][ΣX−(i,j)Xi]



So, we can get

[ΣXjXi∣X−(i,j)] = [ΣXjXi] − [ΣXjX−(i,j)][ΣX−(i,j)Xi]= (KjQKj + ε1In)−1KjQKi − (KjQKj + ε1In)−1

×KjQKT−(i,j)(K−(i,j)QKT

(−i,j) + ε2In(p−2))−1K−(i,j)QKi

= (KjQKj + ε1In)−1

×KjQ(In −KT−(i,j)(K−(i,j)QKT

(−i,j) + ε2In(p−2))−1

×K−(i,j))QKi



By definition, ∥ΣXjXi∣X−(i,j)∥2

is the maximum of

< ΣXjXi∣X−(i,j)f,ΣXjXi∣X−(i,j)f >AXj

= n−1[f]T [ΣXjXi∣X−(i,j)]TKjQKj[ΣXjXi∣X−(i,j)][f]

subject to < f, f >AXj = 1.

Let v = (KiQKi)1/2[f], then [f] = (KiQKi + ε1In)−1/2v.

The right hand side of above is

vT (KiQKi + ε1In)−1/2[ΣXjXi∣X−(i,j)]T

×KjQKj[ΣXjXi∣X−(i,j)](KiQKi + ε1In)−1/2v

subject to vT v = 1,which implies that ∥ΣXjXi∣X−(i,j)∥2

is the largest eigenvalueof ΛXiXj (Cont.)



ΛXiXj =(KiQKi + ε1In)−1/2[ΣXjXi∣X−(i,j)]T

×KjQKj[ΣXjXi∣X−(i,j)](KiQKi + ε1In)−1/2

We plug in [ΣXjXi∣X−(i,j)] , then we can get

ΛXiXj = ∆XiXj∆TXiXj

where

∆XiXj =(KiQKi + ε1In)−1/2

×KjQ(In −KT−(i,j)(K−(i,j)QKT

(−i,j) + ε2In(p−2))−1

×K−(i,j))QKiKj(KjQKj + ε1In)−1/2

So, we should get the largest singular value of ∆XiXj .



We can see the large matrix (K−(i,j)QKT(−i,j) + ε2In(p−2)) and we have to

get inverse of it.But, following proposition shows actual amount computation is not large.

Proposition V ∈ Rs×t with s > t, which has SVD

V = (L1, L0)diag(D,0)(R1,R0)T

where D ∈ Ru×u Then,

(V V T + εIs)−1 = V R1D−1((D2 + εIu)−1 − ε−1Iu)D−1RT1 V T + ε−1Is

V T (V V T εIs)−1V = R1D(D2 + εIs)−1RT1



Second, we calculate ∥ΘXiXj∥.

Similarly, if sample estimatre of ∥ΘXiXj∥ is small enough, we determinethere is no edge between Xi and Xj .

TheoremLet K1∶p = (K1, ...,Kp)T , then matrix representation of Υ satisfies thefollowing equation,

diag(K1QK1, ...,KpQKp)[Υ] =K1∶pQKT1∶p

So, we can get [Θ] as

[Θ] = (K1∶pQKT1∶p + ε3Inp)−1diag(K1QK1, ...,KpQKp)



To find ∥ΘXjXi∥, we maximize

< ΘXjXifi,ΘXjXifi >AXj

= n−1[fi]T [ΘXjXi]TKjQKj[ΘXjXi][fi]

subject to < fi, fi >AXi= 1.

Similarly, ∥ΘXjXi∥2 is the largest eigenvalue of

(KiQKi + ε1In)−1/2[ΘXjXi]TKjQKj[ΘXjXi](KiQKi + ε1In)−1/2

And the largest singular value of

(KiQKi + ε1In)−1/2[ΘXjXi]T (KjQKj + ε1In)1/2


3.3 Tuning parameter

Because of Tychonoff regularization, we used three tuning parameters ε1, ε2, ε3getting inverse of KiQKi,K−(i,j)QK

T−(i,j),K1∶pQK

T1∶p.

In this paper, the cross validation procedure is proposed to select theseparameters.

NotationsLet U and V be random vectors and W = (UT , V T )T .Let F = f0, f1, ..., fr be a set of functions of U .Similarly, G = g0, g1, ..., gs be a set of functions of V .f0, g0 represents the constant function 1.For iid sample W1, ...,Wn, let A = a1, ..., am ⊆ 1,2, ..., n,B = Ac.Let Wa ∶ a ∈ A be a training set, Wb ∶ b ∈ B be a testing set.(Cont.)


3.3 Tuning parameter (Cont.)

(Cont.)

Suppose E(ε)

V ∣U∶ G → F is a training-set-based operator, such that for each

g ∈ G, E(ε)

V ∣Ug is the best prediction of g(V ) based on functions in F .

Then, total error is evaluated as

∑b∈B

s

∑µ=0

[gµ(Vb) − (E(ε)V ∣U

gµ)(Ub)]2

Let LU be the matrix with ith row is fi+1(Ua) ∶ a ∈ A, and let LV bewith gi+1(Va) ∶ a ∈ AAnd let lU , lV be a vector valued functions (f0, ..., fr)T , (g0, ..., gs)T .Then we can express total error as

∑b∈B

∥lV (Vb) −LV LTU(LULTU + εIr+1)−1lU(Ub)∥2



If you let GU be the bth column is lU(Ub), GV be the bth column is lV (Vb),xthen total error can be further simplified as

∥GV −LV LTU(LULTU + εIr+1)−1GU∥2

Now, we apply this cross validation procedure for choosing our parameters.

CVν1(ε1) =p

∑i=1

∑b∈B

∥l−i(X−ib ) −L−iLTi (LiLTi + ε1Im+1)−1li(Xib)∥

2

=p

∑i=1

∥G−i −L−iLTi (LiLTi + ε1Im+1)−1Gi∥2

CV1(ε1) =k

∑ν=1

CV ν1 (ε1)

where ν is k index of k−fold CV.Choose ε1 minimizing CV1(ε1).



CVν2(ε2) =∑i>j

∥G(i,j) −L(i,j)LT−(i,j)(L−(i,j)LT−(i,j) + ε2Im(p−2)+1)−1G−(i,j)∥2

CV2(ε2) =k

∑ν=1

CV ν2 (ε2)

Choose ε2 minimizing CV2(ε2).

And for ε3, in this paper, they recommend to take ε3 to be same as ε2.Because K−(i,j)QK

T−(i,j) and K1∶pQK

T1∶p have similar forms and dimensions.


4. Simulation results

They suggest some comparisons to assess their method. We will see some ofthem.

Comparison 1When the Gaussian copula assumption is violated.


4. Simulation results (Cont.)

Comparison 2When the Gaussian assumption and Gaussian copula assumption is satisfied.


4. Simulation results (Cont.)

Comparison 3When the true model is non additive model


Documents

On an Additive Semigraphoid Model for Statistical Networks ...stat.snu.ac.kr/mvstat/Download/2016_09-12/On an Additive Semigraphoid...Networks With Application to Pathway Analysis