Communication Complexity Lower Bound Techniques for …homes.sice.indiana.edu/qzhangcs/papers/Multi-CC-distribute.pdf2-1 The multiparty number-in-hand (NIH) comm. x 1 = 010011 x 2

1-1

Lower Bound Techniques for MultipartyCommunication Complexity

Qin Zhang

Indiana University Bloomington

Based on works with Jeff Phillips, Elad Verbin and David Woodruff

2-1

The multiparty number-in-hand (NIH) comm.

x1 = 010011 x2 = 111011

x3 = 111111

xk = 100011

They want to jointly compute f(x1, x2, . . . , xk)

Goal: minimize total bits of communication

– A model for proving lower bounds

2-2


x1 = 010011 x2 = 111011

x3 = 111111

xk = 100011


Message-Passing (MP):If x1 talks to x2, otherscannot hear.

Blackboard (BB): Onespeaks, everyone hears.



2-3


x1 = 010011 x2 = 111011

x3 = 111111

xk = 100011


Message-Passing (MP):If x1 talks to x2, otherscannot hear.

Blackboard (BB): Onespeaks, everyone hears.

· · ·S1 S2 S3 Sk

C=Coordinator



3-1

Previously, for multiparty NIH comm. model

Well studied? YES and NO.

3-2



Quite a few papers in BB model:

[Alon et al. ’96], [Bar-Yossef et al. ’04], . . .

3-3




But, Almost nothing in MP model before 2012.


3-4





• Too easy?


3-5





• Too easy?

• Lack of motivations?


4-1

Too easy?

Could be more difficult than BB, since in MP each playerobserves only a piece of protocol transcript.

4-2

Too easy?


We do not want to analyze a k-dimensional matrix directly.

0 1

0

1

0 0

01

AB A communication matrix of the

2-party AND function

4-3

Too easy?


We do not want to analyze a k-dimensional matrix directly.

0 1

0

1

0 0

01

AB A communication matrix of the

2-party AND function

Even for the 2-party Gap-Hamming problem, where Alicehas a n-bit vector and Bob has a n-bit vector, and they wantto compute if the hamming distance of these two vectorsis bigger than n/2 +

√n or less than n/2 −

√n, it tooks

researchers in this area about 10 years and dozens of papers(IW03, Woodruff04, . . ., Sherstov12) to fully understand its2-dim communication matrix!

5-1

Lack of motivations?

(Continous) Distributed Monitoringcommunication → energy, bandwidth, . . .

Data in the cloud

Sensor networksNetwork routers

6-1

Lack of motivations? (cont.)

Data Streamscommunication → space

Memory

CPU

6-2


Data Streamscommunication → space

Memory

CPUP1 P2 Pk

7-1

The BSP model.E.g., Apache Giraph, Pregel.

Input

MapShuffle

Reduce

Output

The MapReduce model.E.g., Hadoop, Google MapReduce.

Distributed Computation for Big Datacommunication → bandwidth, . . .


8-1

Interesting problems

• Frequency moments Fp

F0: #distinct elements

F2: size of self-join

• Heavy hitters

• Quantile

• Entropy

• . . .

Statisticalproblems

8-2





• Heavy hitters

• Quantile

• Entropy

• . . .

Statisticalproblems Graph problems

• Connectivity

• Bipartiteness

• Counting triangles

• Matching

• Minimum spanning tree

• . . .

8-3





• Heavy hitters

• Quantile

• Entropy

• . . .


• Connectivity

• Bipartiteness


• Matching


• . . .

Numericallinear algebra

• Lp regression

• Low-rank

approximation

• . . .

8-4





• Heavy hitters

• Quantile

• Entropy

• . . .


• Connectivity

• Bipartiteness


• Matching


• . . .

Numericallinear algebra

• Lp regression

• Low-rank

approximation

• . . .

DB queries

• Conjuntive

queries

Distributedlearning

• Classifiers

. . .

Geometryproblems

9-1

This talk

Introduce two general techniques thatwork in both BB model and MP modelfor proving lower bounds, with examples.

Conclude with several future directions.

We will talk lower bounds.Most lower bounds match the upper bounds.

10-1

Our new ideas and techniques

Ideas

1. Reduce a k-player problem to a 2-player problem.

2. Decompose a complicated problem to several simpler problems

10-2


Ideas



In fact, we are

exploring the underlying patterns of communication matrices

10-3


Ideas



In fact, we are


SymmetrizationWith Phillips and Verbin, 2012

CompositionWith Woodruff, 2012

1.

2.

Concrete techniques

10-4


Ideas



In fact, we are


SymmetrizationWith Phillips and Verbin, 2012

CompositionWith Woodruff, 2012

1.

2.

Concrete techniques

Example: Graph Connecitivy

Example: Distinct Elements

11-1

Communication complexity

I choose not to introduce formal definitions/concepts ofcommunication complexity. Just want to mentionYao’s lemma:

We will prove lower bounds for randomized protocols. Todo that we will usually pick some input distribution, andthen prove lower bounds for any deterministic protocol thatsucceeds w.pr 0.99 under that input distribution.

12-1

1st Technique: Symmetrization

13-1

Example: graph connectivity

k sites each holds a set of edges of a graph

Goal: compute whether the graph is connected.

A trivial UB: O(nk log n) bits.

We prove that this is tight up to a log factor.

14-1

Set disjointness (2-DISJ)

X ⊆ 1, . . . , n Y ⊆ 1, . . . , n

X ∩ Y = ∅?

14-2


X ⊆ 1, . . . , n Y ⊆ 1, . . . , n

X ∩ Y = ∅?

Exists a hard distribution µβ , under which

|X ∩ Y | = 1 (YES instance) w.p. β and

|X ∩ Y | = 0 (NO instance) w.p. 1− β.

Lemma: (Generalization of [Razborov ’90, BJKS ’04])

E[CCβ/100µβ(2-DISJ)] = Ω(n)

15-1

2-DISJ ⇒ k-CONNx1

x2

x3

x4

x5Alice and Bob want to solve the 2-DISJ(X,Y ) ∼ µβ . (set β = 1/k2)⇒ running a protocol for k-CONN on:

Alice

Bob

Alice plays a random site SI with input XI = X.Bob plays other k − 1 sites, and constructs inputs for them using Y : itchoose Xi ∼ µβ |Y (i 6= I).Note: X1, . . . , Xk are i.i.d. conditioned on Y .

Vertices u1, . . . , uk, v1, . . . , vn. An edge (ui, vj) exists iff Xi,j = 1.Let ν be the distribution of the graph.

16-1

2-DISJ ⇒ k-CONN (cont.)

vj |j ∈ [r]\Y vj |j ∈ Y

vj|Y |+1vj|Y |+2

vj|Y |+3vjn vj1 vj|Y |vj2

u1 u3u2 uk

(ui, vj) exists if and only if Xi,j = 1

XI ∩ Y = ∅?

17-1

2-DISJ ⇒ k-CONNx1

x2

x3

x4


Alice

Bob


Correctness: Solving k-CONN w.pr. 1 − β1.5 under ν also solves2-DISJ w.pr. 1− β/100 under µβ .


17-2

2-DISJ ⇒ k-CONNx1

x2

x3

x4


Alice

Bob




E[CCβ/100µβ (2-DISJ)] = O

(1k

)· CCβ

1.5

ν (k-CONN)

17-3

2-DISJ ⇒ k-CONNx1

x2

x3

x4


Alice

Bob





(1k

)· CCβ

1.5

ν (k-CONN)

Ω(n)

17-4

2-DISJ ⇒ k-CONNx1

x2

x3

x4


Alice

Bob





(1k

)· CCβ

1.5

ν (k-CONN)

Ω(n) Ω(nk)

18-1

A useful message

We can show trivial UBs are tight for many statisticaland graph problems in the MP model (e.g., #distinctelements, bipartiteness, cycle/triangle-freeness, . . .), whenexact answers are wanted. (Woodruff, Z, 2013a)

Approximation is often necessary if we want comm.efficient protocols.

18-2

A useful message

We can show trivial UBs are tight for many statisticaland graph problems in the MP model (e.g., #distinctelements, bipartiteness, cycle/triangle-freeness, . . .), whenexact answers are wanted. (Woodruff, Z, 2013a)

Approximation is often necessary if we want comm.efficient protocols.

Example:

Computing the diameter of a graph exactly needs Ω(kn2) bits(when m = n2) in the MP model.

However, there is a protocol that computes the diameter up to

an additive error of 2 using O(kn3/2) bits.

19-1

Other problems using symmetrization

• k-bitwise-MAJ

• k-bitwise-OR/AND/XOR

With applications in

– Distinct elements reporting

– ε-kernels (geometry)

– Heavy-hitters

• Testing bipartiteness,

Testing cycle

Testing triangle-freeness,

. . .

19-2

Other problems using symmetrization

• k-bitwise-MAJ

• k-bitwise-OR/AND/XOR

With applications in

– Distinct elements reporting

– ε-kernels (geometry)

– Heavy-hitters

• Testing bipartiteness,

Testing cycle

Testing triangle-freeness,

. . .

Key:

Find the right 2-playerproblem for reduction

20-1

2nd Technique: Composition

21-1

Example – the F0 (distinct elements) problem

k sites each holds a set Xi (i ∈ 1, 2, . . . , k).

Goal: compute #distinct elements(∪ki=1Xi) up to a (1 + ε)-approx.

21-2




Current best UB: O(k/ε2)[Cormode, Muth, Yi. 2008]

Holds in MP model

21-3





Holds in MP model

Previous LB: Ω(k)

And Ω(1/ε2)Direct reduction from Gap-Hamming

21-4





Holds in MP model

Previous LB: Ω(k)

And Ω(1/ε2)Direct reduction from Gap-Hamming

New LB: Ω(k/ε2). [Woodruff, Z. 2012, 2013b]. Holdsin MP model. Better UB in the BB model

22-1

The proof framework

Step 1: Find two modular problems k-GAP-MAJ and 2-DISJof simpler structures s.t.

F0 can be thought as a composition of them.

Step 2: Analyze the complexities of k-GAP-MAJ and 2-DISJ.

Step 3: Compose two modular problems so that:

Complexity(F0) = Complexity(k-GAP-MAJ)

× Complexity(2-DISJ).

22-2

The proof framework

Step 1: Find two modular problems k-GAP-MAJ and 2-DISJof simpler structures s.t.

F0 can be thought as a composition of them.

Step 2: Analyze the complexities of k-GAP-MAJ and 2-DISJ.

Step 3: Compose two modular problems so that:

Complexity(F0) = Complexity(k-GAP-MAJ)

× Complexity(2-DISJ).

The proof presented here is for a special case when k = 2/ε2.

23-1

k-GAP-MAJ

k sites each holds a bit Zi chosen uniform at random from 0, 1

Goal: compute

k-GAP-MAJ(Z1, Z2, . . . , Zk) =

0, if

∑i∈[k] Zi ≤ k/2−

√k,

1, if∑i∈[k] Zi ≥ k/2 +

√k,

don’t care, otherwise,

23-2

k-GAP-MAJ

k sites each holds a bit Zi chosen uniform at random from 0, 1

Goal: compute

k-GAP-MAJ(Z1, Z2, . . . , Zk) =

0, if

∑i∈[k] Zi ≤ k/2−

√k,

1, if∑i∈[k] Zi ≥ k/2 +

√k,

don’t care, otherwise,

Lemma: Any protocol Π that computes k-GAP-MAJ correctlyw.p. 0.9999 has to learn Ω(k) Zi’s well, that is,

H(Zi | Π) ≤ Hb(1/100) for Ω(k) of i.

In other words: I(Π;Z1, . . . , Zk) = Ω(k).

24-1


X ⊆ 1, . . . , n Y ⊆ 1, . . . , n

X ∩ Y = ∅?

24-2


X ⊆ 1, . . . , n Y ⊆ 1, . . . , n

X ∩ Y = ∅?

Exists a hard distribution µβ , under which

|X ∩ Y | = 1 (YES instance) w.p. β and

|X ∩ Y | = 0 (NO instance) w.p. 1− β.

Lemma: (Generalization of [Razborov ’90, BJKS ’04])

E[CCβ/100µβ(2-DISJ)] = Ω(n)

25-1

The proof sketch

· · ·S1 S2 S3 Sk

Ccoordinator

sites

Y

X1 X2 X3 Xk

2-DISJ

(Xi, Y ) ∼ µ1/2 (hard input dist. for DISJ) for each i ∈ [k]

25-2

The proof sketch

· · ·S1 S2 S3 Sk

Ccoordinator

sites

Y

X1 X2 X3 Xk

2-DISJ


Zi = |Xi ∩ Y |

1 w.p. 1/20 w.p. 1/2

25-3

The proof sketch

· · ·S1 S2 S3 Sk

Ccoordinator

sites

Y

X1 X2 X3 Xk

2-DISJ

F0(X1, X2, . . . , Xk) ⇐⇒ k-GAP-MAJ(Z1, Z2, . . . , Zk)

(Zi = |Xi ∩ Y |)=⇒ learn Ω(k) Zi’s well

=⇒ need Ω(nk) bits

(using an embedding argument)


Zi = |Xi ∩ Y |

1 w.p. 1/20 w.p. 1/2

25-4

The proof sketch

· · ·S1 S2 S3 Sk

Ccoordinator

sites

Y

X1 X2 X3 Xk

Q.E.D.

For the reduction setuniverse n = Θ(1/ε2)

we get Ω(k/ε2) LB.

2-DISJ

F0(X1, X2, . . . , Xk) ⇐⇒ k-GAP-MAJ(Z1, Z2, . . . , Zk)

(Zi = |Xi ∩ Y |)=⇒ learn Ω(k) Zi’s well

=⇒ need Ω(nk) bits

(using an embedding argument)


Zi = |Xi ∩ Y |

1 w.p. 1/20 w.p. 1/2

26-1

Other problems using composition

• Frequency moments Fp (p > 1)

• Entropy

• `p

• Heavy-hitters

• Quantile

26-2

Other problems using composition

• Frequency moments Fp (p > 1)

• Entropy

• `p

• Heavy-hitters

• Quantile

Key:

Find the right modular problems

27-1

Furture directions

28-1

Future directions – Problem space

The current understandings

Exactcomputation

Approximation

statistics graph linearalgebra

DBqueries

well

well

medium

limited

limited

limitedlimited

28-2

Future directions – Problem space

Concretely, e.g.,

What is the CC of computing a const approx of the size ofthe matching?

The current understandings

Exactcomputation

Approximation

statistics graph linearalgebra

DBqueries

well

well

medium

limited

limited

limitedlimited

29-1

Future directions – Problem space (cont.)

Communication complexity of distributed property testing.

Communication complexity of distributed property testing canalso be used for proving LBs for traditional property testing,by extending a framework by Blais et al. 2011.

29-2




Communication complexity of relations(instead of functions, useful for database queries)

29-3




Communication complexity of relations(instead of functions, useful for database queries)

Need to understand more primitive problems (building blocks),e.g., breadth-first search (BFS), pointer-chasing, etc.

So far building blocks are quite limited: mainly 2-DISJ, 2-GAP-HAMMING, k-GAP-MAJ, k-OR and their variants.

30-1

Future directions – Model/Technique space

Can we relax or generalize the symmetrization technique?

E.g., do not require “totally symmetric distributions”

30-2

Future directions – Model/Technique space

Can we relax or generalize the symmetrization technique?

E.g., do not require “totally symmetric distributions”

Consider round complexity (practical needs).

Consider players with limited space (practical needs).

31-1

The end

T HANK YOU

Questions?

Documents

Communication Complexity Lower Bound Techniques for …homes.sice.indiana.edu/qzhangcs/papers/Multi-CC-distribute.pdf2-1 The multiparty number-in-hand (NIH) comm. x 1 = 010011 x 2