Failures & Redundancy - Zoo | Yale Universityzoo.cs.yale.edu/classes/cs426/2014/lec/l-failures.pdf · Failures & Redundancy Ennan Zhai ... •Byzantine failure Failure Models

Failures & Redundancy

Ennan ZhaiComputer Science at Yale University

[email protected]

About our labs, final project and midterm

• Building a simple and elegant Peerster• Making git log clear• Midterm may be kind of hard

• Lab1 grading



• Lab1 grading



• Lab1 grading


• Building a simple and elegant Peerster• Making git log clear• Midterm

• Lab1 grading


• Building a simple and elegant Peerster• Making git log clear• Midterm

• Lab1 grading

• Failures

• Correlated failures in decentralized systems

Lecture Outline

• Failures


Lecture Outline

• What is the failure?• Why?• Real evidences?

• Partial failure

Failures


• Partial failure

Failures


• Partial failure

Failures

Evidence

Category) Failure)types) Diagnosis)&)Repair)

%)

So#ware(21%( Link(layer(loop( Find(and(fix(bugs(

19%(Imbalance(!(overload( 2%(

Hardware(18%( FCS(error( Replace(cable( 13%(Unstable(power( Repair(power( 5%(

Unknown(23%( Switch(stops(forwarding( N/A( 9%(Imbalance(!(overload( 7%(Lost(configuraNon( 5%(High(CPU(uNlizaNon( 2%(

ConfiguraNon(38%(

Errors(on(mulNple(switches(

Update(configuraNon(

32%(

Errors(on(one(switch( 6%(

Evidence


%)





ConfiguraNon(38%(



32%(


Evidence


%)





ConfiguraNon(38%(



32%(


Evidence


%)





ConfiguraNon(38%(



32%(


Evidence

Top10 Failure Events in Clouds

Failure Models

• Crash failure• Timing failure• Response failure

• Byzantine failure

Failure Models



Failure Models



Failure Models



Failure Models

Solutions

Solutions

• Overcoming failures after the outage occurs:- Diagnosis system- Accountability system- Fault tolerance system

Solutions


Solutions


Solutions


• Overcoming failures before the outage occurs:- Redundancy

• Failures


Lecture Outline

• Failures


Lecture Outline

Example

Email App

Peer A Peer B

Example

Email App

Peer A Peer B

Third-party infrastructure components

Example

Email App

ISP BISP A ISP C


Peer A Peer B

Example

Email App

ISP BISP A ISP C

Power Source


Peer A Peer B

Example

Email App

ISP BISP A ISP C

Power Source


Peer A Peer B

Example

Email App

ISP BISP A ISP C

Power Source


Peer A Peer B

Become unavailable !

Example

Example

• Service providers allocate or tolerate failures via: - diagnosis systems, e.g., Sherlock.- fault-tolerant systems, e.g., F10, Skute.

Existing Efforts


• Solving the problem after the outage occurs

• There is no any effort before the problem occur

Existing Efforts



• We want to prevent the problem before the outage occurs

Existing Efforts



• We want to prevent the problem before the outage occurs

• Recommending truly independent redundancy services when deploying applications

Existing Efforts

What kind of system we want to build?

Consumer

Node A Node B Node C

Select two nodes for redundancy


Consumer

A and B ?


Consumer

B and C ?


Consumer

A and C ?


Consumer

Recommender

Select two nodes for redundancy: A&B? B&C? or A&C?


Consumer

Recommender


Consumer

Recommender

Assessing independence by the # of overlapping components between nodes


Consumer

Recommender


Consumer

Recommender

ISP A Power BPower A


Consumer

Recommender


ISP APower APower B


Consumer

Recommender


ISP APower APower B


Consumer

Recommender

ISP APower APower B

ISP A Power B ISP BPower A


Consumer

Recommender

ISP APower APower B


ISP BPower APower B


Consumer

Recommender

ISP APower APower B


ISP BPower APower B


Consumer

Recommender

ISP APower APower B

ISP BPower APower B

ISP A Power B ISP B Power CPower A


Consumer

Recommender

ISP BPower APower B


ISP BPower C


ISP APower APower B

Consumer

Recommender

Cloud A, C 0 Cloud B, C 1 Cloud A, B 2

Deployment | |

ISP BPower APower B


ISP BPower C


ISP APower APower B

Consumer

Recommender



Deployment | |

ISP BPower APower B

ISP BPower C


ISP APower APower B

Consumer

Recommender

ISP APower APower B

ISP BPower APower B


ISP BPower C


Deployment | |

=2


Consumer

Recommender

ISP APower APower B

ISP BPower APower B


ISP BPower C


Deployment | |

=2

Deployment | |


Node A, C 0 Node B, C 1 Node A, B 2

Consumer

Recommender

ISP APower APower B

ISP BPower APower B


ISP BPower C


Deployment | |Deployment | |



Consumer

Recommender

ISP APower APower B

ISP BPower APower B


ISP BPower C


Deployment | |Deployment | |

=1Node A Node B Node C


Consumer

Recommender

ISP APower APower B

ISP BPower APower B


ISP BPower C

=1

Deployment | |



Consumer

Recommender

ISP APower APower B


ISP BPower C

Deployment | |



Consumer

Recommender

ISP APower APower B


ISP BPower C

Deployment | |

=0



Consumer

Recommender

ISP APower APower B


ISP BPower C

Deployment | |

=0



Consumer

Recommender


Deployment | |


Consumer

Recommender1. Node A, C 02. Node B, C 13. Node A, B 2

| |Deployment

Ranking List


Consumer

But, it is not so easy .

Recommender

Node A

Node B

Node C

Solution 1

Consumer

Recommender

App Provider

Node A

Node B

Node C

Privacy Concern!

Solution 1

Trusted Third Party

Node A

Node B

Node C

Solution 2

Consumer

Trusted Third Party

App Provider

Node A

Node B

Node C

Hard to find!

Solution 2

Secure Multiparty Computation

Node A

Node B

Node C

Solution 3

Consumer

Secure Multiparty Computation

App Provider

Node A

Node B

Node C

SMPC is difficult to scale!

Solution 3

Intersection cardinality does help

Example

Our solution - iRec

• The first independence recommender sys: - achieving our goal- preserving privacy of each node- practical

Our solution - iRec

• The first independence recommender sys: - achieving our goal- preserving privacy of each node- practical

Preliminary background: P-SOP

Our solution - iRec

• Allows k parties to compute the intersection, union cardinality and Jaccard similarity, without learning other information.

S1 S2 ... ... Sn

S1 S2 ... ... Sn

J(S1, S2, ..., Sn) =

P-SOP: Private Jaccard Similarity

| |

||


S1 S2 ... ... Sn

S1 S2 ... ... Sn

J(S1, S2, ..., Sn) =


| |

||

11

3

10

1

5

20

3

7

3

P-SOP



11

3

10

1

5

20

3

7

3


One overlapping element



P-SOP


11

3

10

1

5

20

3

7

3

Protocol


But I do not know which element is overlapping



P-SOP


11

3

10

1

5

20

3

7

3

Protocol


7 elements in union 7 elements in union

7 elements in union

P-SOP


11

3

10

1

5

20

3

7

3

Protocol


But I do not know which elements are in union



P-SOP


11

3

10

1

5

20

3

7

3

Protocol


P-SOP


S1 S2 ... ... Sn

S1 S2 ... ... Sn

J(S1, S2, ..., Sn) = | |

||

11

3

10

1

5

20

3

7

3

Protocol


Jaccard Jaccard

Jaccard

P-SOP


S1 S2 ... ... Sn

S1 S2 ... ... Sn

J(S1, S2, ..., Sn) = | |

||

37

15

203

113

10

Each party maintains a commutative encryption key

David Eve

Frank

37

15

203Kd

Kf

Ke

113

10

Each party maintains a commutative encryption key

David Eve

Frank

37

15

203Kd

Kf

Ke

113

10

Each party maintains a commutative encryption keyCommutative encryption holds: Kx(Ky(E)) = Ky(Kx(E))

David Eve

Frank

Kf(3)Kf(7)

Ke(1)Ke(5)

Ke(20)Ke(3)Kd

Kf

Ke

Kd(11)Kd(3)

Kd(10)

Each party encrypts each item of elements in its dataset through the key

David Eve

Frank

Kf(7)Kf(3)

Ke(5)Ke(3)Ke(1)

Ke(20)Kd

Kf

Ke

Kd(10)Kd(11)Kd(3)

Each party shuffles the encrypted elements in its own dataset

David Eve

Frank

Ke(5)Ke(3)Ke(1)

Ke(20)Kd

Kf

Ke

Kd(10)Kd(11)Kd(3)

Kf(7)Kf(3)

Each party sends its own encrypted dataset to its successor party.

David Eve

Frank

Ke(5)Ke(3)Ke(1)

Ke(20)

Kd

Kf

Ke

Kd(10)Kd(11)Kd(3)

Kf(7)Kf(3)

Each party sends its own encrypted dataset to its successor party.

David Eve

Frank

Kf(Ke(5))Kf(Ke(3))Kf(Ke(1))

Kf(Ke(20))

Kd

Kf

Ke

Ke(Kd(10))Ke(Kd(11))Ke(Kd(3))

Kd(Kf(7))Kd(Kf(3))

Each party encrypts each item of elements in the received dataset using its own key

David Eve

Frank

Kd(Kf(3))Kd(Kf(7))

Kf(Ke(1))Kf(Ke(20))Kf(Ke(5))Kf(Ke(3))

Kd

Kf

Ke


Shuffle too.

David Eve

Frank

Kd(Kf(3))Kd(Kf(7))

Kf(Ke(1))Kf(Ke(20))Kf(Ke(5))Kf(Ke(3))

Kd

Kf

Ke


Each party sends its current encrypted dataset to its successor party.

David Eve

Frank

Kd(Kf(3))Kd(Kf(7))

Kf(Ke(1))Kf(Ke(20))Kf(Ke(5))Kf(Ke(3)) Kd

Kf

Ke



David Eve

Frank

Ke(Kd(Kf(3)))Ke(Kd(Kf(7)))

Kd(Kf(Ke(1)))Kd(Kf(Ke(20)))Kd(Kf(Ke(5)))Kd(Kf(Ke(3))) Kd

Kf

Ke

Kf(Ke(Kd(11)))Kf(Ke(Kd(10)))Kf(Ke(Kd(3)))

Each party encrypts each item of elements in the received dataset using its own key

David Eve

Frank


Kd(Kf(Ke(3)))Kd(Kf(Ke(5)))Kd(Kf(Ke(1)))

Kd(Kf(Ke(20))) Kd

Kf

Ke


Shuffle.

David Eve

Frank



Kd(Kf(Ke(20))) Kd

Kf

Ke



David Eve

Frank



Kd(Kf(Ke(20)))Kd

Kf

Ke


OK. Now, each party has received its own original dataset which has been encrypted by all the parties.

David Eve

Frank



Kd(Kf(Ke(20)))Kd

Kf

Ke


Kx(Ky(E)) = Ky(Kx(E))

David Eve

Frank



Kd(Kf(Ke(20)))Kd

Kf

Ke


Kx(Ky(E)) = Ky(Kx(E))

David Eve

Frank



Kd(Kf(Ke(20)))Kd

Kf

Ke


I know the # of intersection is 1, and union is 7



David Eve

Frank



Kd(Kf(Ke(20)))Kd

Kf

Ke


Jaccard = 1/7

Jaccard = 1/7

Jaccard = 1/7

David Eve

Frank

Consumer iRec



iRec

iRec


Select two nodes for redundancy: A&B? B&C? or A&C?


iRec

Consumer

iRec



iRec: Step 1

Consumer

iRec



iRec: Step 2

Consumer


iRec

ISP APower APower B

ISP BPower APower B

ISP BPower C


iRec: Step 3

Consumer


iRec

ISP APower APower B

ISP BPower APower B

ISP BPower C


iRec: Step 3

P-SOP

Consumer


iRec

ISP APower APower B

ISP BPower APower B

ISP BPower C


iRec: Step 3

=2

Consumer


iRec

ISP APower APower B

ISP BPower APower B

ISP BPower C


iRec: Step 3

=4=2

Consumer


iRec

ISP APower APower B

ISP BPower APower B

ISP BPower C


iRec: Step 3

=4=2

J = 2/4

Consumer


iRec

ISP APower APower B

ISP BPower APower B

ISP BPower C


iRec: Step 3

=4=2

J = 2/4


Deployment | | Cloud A, C 0 Cloud B, C 0.25 Node A, B 0.5

Deployment J

Consumer


iRec

ISP APower APower B

ISP BPower APower B

ISP BPower C


P-SOP

iRec: Step 3 Cloud A, C 0 Cloud B, C 1 Cloud A, B 2


Deployment J

Consumer


iRec

ISP APower APower B

ISP BPower APower B

ISP BPower C


=1



Deployment J

Consumer


iRec

ISP APower APower B

ISP BPower APower B

ISP BPower C


=1=4



Deployment J

Consumer


iRec

ISP APower APower B

ISP BPower APower B

ISP BPower C


=1=4

J = 1/4



Deployment J

Consumer


iRec

ISP APower APower B

ISP BPower APower B

ISP BPower C


=1=4

J = 1/4


Deployment | | Cloud A, C 0 Node B, C 0.25 Node A, B 0.5

Deployment J

Consumer


iRec

ISP APower APower B

ISP BPower C




Deployment J

P-SOP

Consumer


iRec

ISP APower APower B

ISP BPower C




Deployment J

=0,

Consumer


iRec

ISP APower APower B

ISP BPower C




Deployment J

=0 =5 ,,

Consumer


iRec

ISP APower APower B

ISP BPower C




Deployment J

=0 =5 J = 0/5,,

Consumer


iRec

ISP APower APower B

ISP BPower C



Deployment | | Node A, C 0 Node B, C 0.25 Node A, B 0.5

Deployment J

=0 =5 J = 0/5,,

Consumer

iRec





Deployment J

Consumer


iRec




Deployment J

Consumer

Ranking List


iRec


iRec: Step 6



Deployment JConsumer

Failure is a very important topic

It is very hard to solve in decentralized system

Thanks!

Questions?

Documents

Failures & Redundancy - Zoo | Yale Universityzoo.cs.yale.edu/classes/cs426/2014/lec/l-failures.pdf · Failures & Redundancy Ennan Zhai ... •Byzantine failure Failure Models