ADAPT IST-2001-37126

Role of Group Communication in BS Architecture

(or: Which platform are we going to use ?)

Alberto BartoliUniversity of trieste

Bologna

300 Kms

ADAPT

Group Communication

Group Communication (GC)Suite of communication and membership primitivesVery useful for implementing replication algorithmsIn particular, in the presence of failures

(host, network)

BS will certainly use some form of replication

BS will certainly use some form of GC

ADAPT

Options

JavaGroupsUsed in JBoss clustering extensionsImplemented in Java (stack of layers)

SpreadUsed in a variety of environmentsImplemented in C (Java interface

available)

JBoraUsed in my lab only…Thin Java layer on top of Spread

Reliable Broadcast

Uniform Broadcast(much more powerful)

Novel idea (?)“Primary Uniform” Broadcast

(much simpler to use)

ADAPT

Replication

Action A executed upon receiving a multicastExecute a method on a local objectUpdate the serialized state of a local beanCommit a transaction...

m m m

“Frequent” (informal) requirement:

If a process executes A and then crashes,

Then A must be executed also by all processes that do not crash(“actions must not be lost”)

ADAPT

Reliable broadcast ( JavaGroups)

Either all correct processes deliver a message or none of them does

No guarantee on processes that are not “correct” !!!

Failure

Membershipchange

Executing A might not be safe !

ADAPT

NO NO NO

Uniform broadcast ( Spread)

If a process delivers a message,then all correct processes deliver that message

Executing A is always safe !

ADAPT

In practice“Cross-the-fingers” reliability

(don’t know or don’t care)“Real” reliability

Uniform broadcastReliable broadcast + Additional measures

Replicated Databases: if a replica commits a transaction not committed by other replicas, undo the transaction later (Bettina)

Replicated Services: whenever a replica crashes, surviving replicas fetch from all clients the last reply they have received (Karamanolis, Magee — IEEE TSE)

Replicated Data: wait for an explicit responsefrom every available replica(myself, Ozalp — JPDC) (JBoss SFSB clustering)

ADAPT

“Real reliability”Reliable broadcast + Additional measures: ( JavaGroups)Each additional measure is ad-hocEach additional measure is complex

(many failure patterns to consider and to cope with)

Lot of work above the GC layer

Uniform broadcast: ( Spread)No additional measuresSystematic approach, complexity within the GC layer

Very little work above the GC layer

ADAPT

Uniform broadcast:Hhmmm….

If the network can partition,you need additional measures

again !!!

Again the same problem !

“The view is about to split;GC can’t tell whether the processes that are about to leave

have received the messages that follow”

All correct processeswill receive this

In-doubt message:can’t tell who will receive this!

ADAPT

JBora

1 2 3 4 5 6 7 8

1 2 3 4 5 6 7 8

1 2 3 4 5 6 7 8

1 2 3 4 5 6 7 8

1 2 3 4 5 6 7 8

Primary View

Non-Primary View

9 10 11

9 10 11

9 10 11

Very simple reasoning for the “common case”Processes that leave the primary view deliver a prefix

of the sequence of messages in the primary view

Executing A is always safe !No need for additional

measures !

ADAPT

So what ?

JavaGroupsSpreadJBora

ADAPT

Scenario 1

My suggestion:Forget about uniform multicast (Spread, JBora)Stick with JavaGroups

We want to rely (almost) completely on

(some snapshot of) JBoss clustering

Preliminary WP1 Meeting (Bologna, Trieste)Encapsulating Spread within JavaGroupsEncapsulating JBora within JavaGroups…too complex, dubious advantages(see meeting slides for details)

ADAPT

Scenario 2

My suggestion:We are not interested in uniform multicast Use

JavaGroupsWe are interested in uniform multicast Use

JBora

We don’t want to run behind JBoss clustering

(write our own clustering features)

ADAPT

My opinion

If we use JavaGroups I will ask to restructure WP1

(Task 1.3 “Support for group communication”)Month 18: Dear reviewer, Trieste has led Task 1.3. We did almost

nothing.

If we decide to write our own clustering features I don’t see any single reason why we should eliminate

uniform multicast from the beginning

ADAPT

Experiments:“Throughput under stress”

Each sender injects 1000 msg/sec (bursty) All details available in a separate document

(4 PIII 800 MHz, Windows 2000, Ethernet 100 MB)

Important findings about JavaGroups(configured as in JBoss clustering):

Processes may start missing messages and this occurs silently(no failure notification whatsoever)

You cannot start / recover multiple processes simultaneously (they do not discover each other)

Does not seem very “reliable” (at least, when stressed)

ADAPT

A few numbers...

Recall:Spread, JBora: Message throughput Operation

throughoutJavaGroups: Message throughput < Operation

throughput(N responses for each multicast)

Spread JBoraJavaGroups

1 sender (500 Byte) 640 576 5612 senders 1254 871 4961 sender (5 KBytes) 323 323 1652 senders 592 359 275

TotalUniform

FIFOReliable

Failed !

150 !

Verypreliminary...

ADAPT

Appendix

ADAPT

Uniform broadcast:How is it implemented ?

In practice, many, many optimizations:The white messages are not separated

messages,but fields of other messages required anyway

Costly, but not as much as it seems

Messages within the GC Layersfor one uniform broadcast

Uniform broadcast delivered only upon the second broadcast

m

ack

deliver m

ADAPT

JBoss Clustering (I)

Messages from the applicationlayer for one operation

m

done

My belief:Less efficient than uniform multicastThe application injects N one-to-one

messages into the system

ADAPT

JBoss Clustering (II)

I know from our JPDC work that coping with view changes here is VERY complex

Does JBoss really handle all cases correctly ?

m

done

Devising all possible failure patterns and coping with them correctly is very, very complex

Difficult to achieve full confidence in the algorithm and its implementation

ADAPT

Transitional Views:Why cannot be avoided ?

Suppose a network failure during the protocol

GCLayer may end up with one side of the partitionthat does not know whether the other side hasreceived the message

Two approaches:GCLayer waits until the partition recovers

(not feasible)GCLayer notifies the application of the new view after a

warning