Ordered Communication

Ordered Communication

Define guarantees about the order of deliveries inside group of processes

Type of ordering: Deliveries respect the FIFO ordering of the corresponding sendingsDeliveries respect the Causal ordering of the corresponding sendingsDelivery respects a total ordering of deliveries (atomic communication)

Advantages of ordered communication

Orthogonality wrt reliable communication. Reliable broadcast does not have any property on ordering deliveries of messagesThis can cause anomalies in many applicative contexts

“Reliable ordered communication” are obtained adding one or more ordering properties to reliable communication

Example: flight booking system. Consider the message pattern depicted in the figure. The server cansel a reservation that has never been done!

tclient

server

“reserve” “cancel”

“Prices 15% off”

FIFO Broadcast\ specificationMessages sent by the a process has to be delivered in the sending order.

FIFO Reliable broadcast specification is given by properties reliable (regular) broadcast plus an additional property of SafetySafety that captures the notion of order (example):

FIFO Order: if a process sends a broadcast message m before m’, then no correct process delivers m’ if it has not already delivered m.

FIFO order can be uniform/non uniform

FIFO Broadcast = Reliable Broadcast + FIFO Order

Each process q holds: S p

a count of messages broadcast by p Rp the sequence number of the latest message sent by p and delivered by q

For p to FO-multicast a message to g, it piggybacks S p

on the message, rbBroadcasts it and increments S p

by 1

On receipt of a message from q sent by p with sequence number S, p checks whether

S = Rp + 1. If so, q FO-delivers it

if S > Rp + 1 then q places message

in hold-back queue until intervening messages have been delivered. (note that rbBroadcast does eventually deliver messages unless the sender crashes)

Messageprocessing

Delivery queueHold-back

queue

deliver

Incomingmessages

When delivery guarantees aremet

FIFO Broadcast\algorithm

Advantages of Ordered Communication (2)

Prof.

Student 2

m1: “Fri exam cancelled”

Student 1m2:

“let’s party on Thu night”

m3: “but we have an exam on Fri!”

Causal Order FIFO Order, But FIFO OrderCausal Orderthus, Causal Order = FIFO Order + ?

Causal Broadcast\specification

Causal Reliable broadcast specification is given by properties reliable (regular) broadcast plus an additional property of SafetySafety that captures the notion of order (example):

Causal Order: if the sending of a message m causally precedes the sending of a message m’, then every correct process has to deliver m before delivering m’.

Causal Broadcast = Reliable Broadcast+Causal Order

Causal Broadcast\specification

Causal Order = FIFO Order + Local Order.

Local Order: if a process delivers a message m before sending a msg m’, then no correct process deliver m’ if it has not already delivered m.

Example:

p

q

r

tm

m’

The delivery of m’ is delaied till the arrival and the delivery of m

Causal Broadcast\implementations

Two implementations blocking algorthm using vector clocks (already discussed)non-blocking algorithm using piggybacking of causal past

p1

p2

p3

COBcast(m1)

CObcast(m2)

COBcast(m3)

m1

m2

m1,m2 , m3

COdelv(m1)

COdelv(m3)m2 già COdelivered!

COdelv(m1)

COdelv(m1)COdelv(m2)

COdelv(m2) COdelv(m3)

COdelv(m3)

COdelv(m2)Filter out

Advantages of Ordered Communication (3)

Causal Order is not enough strong to avoid anomaliesEs. banking. Bank account replicated on two sites

R1

R2

A:£100

A:£100

Deposit £20

Add 10% interest

A:£120

A:£110

A:£132

A:£130

Despite the fact that replicas initially share the same state, the state reaches a different value in the two sites at the end of the exexution as shown in Figure. Note that the computation is Causally Ordered

To guarantee values of replicas be the same, one has to ensure that the order of delivery be the same at each process. In the above example R1 delivers m1 before m2 and R2 delivers m2 before m1.

Note that ensuring the same delivery order at each replicas does not look at the sending order of messages

m1

m2

Atomic Broadcast\specification

Atomic (Total) Reliable broadcast specification is given by properties reliable (regular) broadcast plus an additional property of SafetySafety that captures the notion of total order (example):

Total Order: if two correct processes p and q deliver m and m’, then p delivers m before m’ if, and only if, q delivers m before m’

Total order is orthogonal with respect to FIFO and Causal Order. Total order would accept indeed a computation in which a process sends n messages to a group, and each of the processes of the group delivers such messages in the reverse order of their sendings. The computation is totally ordered but it is not FIFO.

Causal Atomic broadcast

Hierarchy of Broadcast Specifications

Reliable broadcast

FIFO broadcast

Causal broadcast

FIFO Atomic broadcast

Atomic broadcast

Total Order

Causal Order

Total Order

Total Order

FIFO Order FIFO Order

Local OrderCausal OrderLocal Order

System model

Static set of processes Π = {p1 … pn}

Message passing over perfect channels (message exchanging between correct processes is reliable)AsynchronousCrash fault model for processesWe characterize the system in terms of its possible runs R

Rp1

p2

pn

TOcast(m)

m

m

m

TOdeliver(m)

crash

r

A few notation

Property P: predicate on the system, identifying a set of runs RP R

P P’ iff RP RP’

Specification S(P1,…,Pm): logical and of m properties, identifying a set of runs RS=RP1

∩ … ∩ RPm R

S → S’ iff RS RS’

RP RP’

RS RS’

RP1RPnRS

R

R

R

TO specifications

Total order specifications are usually composed by four properties, namely Validity, Integrity,Agreement, and Order. A Validity property guarantees that messages sent by correct processes will eventually be delivered at least by correct processes; An Integrity property guarantees that no spurious or duplicate messages are delivered; An Agreement property ensures that (at least correct) processes deliver the same set of messages; An Order property constrains (at least correct) processes delivering the same messages to deliver them in the same order.

TO specifications

Total Order Broadcast = S(V,I,A,O)V = ValidityI = IntegrityA = AgreementO = Order

Distinct specifications arise from distinct formulations of each property

uniform vs non-uniformA uniform property imposes restrictions on the behavior of (at least) correct processes on the basis of events occurred in some process

NUVUI

TO(A,O)

TO Specifications

Crash failure + Perfect channels NUV. if a correct process TOCAST a message m then some correct process will eventually deliver mUI. For any message m, every process p delivers m at most once and only if m was previously tocast by some (correct or not) process.

The Agreement property

(Uniform Agreement, UA) If a process (correct or not) todelivers a message m, then all correct processes will eventually todeliver m;(Non-uniform Agreement, NUA) If a correct process todelivers a message m, then all correct processes will eventually todeliver m

The Agreement property

Constrains the set of delivered messages

Correct processes always deliver the same set of messages MEach faulty process p delivers a set Mp

UA: Mp M

NUA: Mp can be s.t. Mp - M ≠

m2

m4

p1

p2

p3

m2m4

m1

m1

m3

m3

m3

m4

m1

m2

UAm4

p1

p2

p3

m2m4

m1

m1

m3

m3

m3

m4

m1

m2

m5

NUA

The Order property

Constrains the order of message deliveries and possibly the set of delivered messages

SUTO: if p delivers m<m’, q delivers m’ only after m

same ordersame prefix of the set of delivered messagesafter an omission, disjoint sets of delivered messages

WUTO: if p,q deliver m,m’, they get the same order

no restrictions on the set of delivered messages

p1

p2

p3

m2

m2

m2

m1

m1

m1m4

m3

m3

m7

m6

m5

p1

p2

p3

m2

m2m1

m1

m1m4

m3

m3

m7

m6

m5

SUTO WUTO

The Order property (2)SUTO and WUTO are uniformThey both have a non-uniform counterparts: SNUTO and WNUTO (Strong Non-uniform Total Order, SNUTO). If some correct process todelivers some message m before message m', then a correct process todelivers m‘ only after it has todelivered m.

(Weak Non-uniform Total Order, WNUTO) If correct processes p and q both todeliver messages m and m', then p todelivers m before m' if and only if q todelivers m before m‘

The Order property (2)

SUTO WUTOSNUTO WNUTO

p1

p2

p3

m1

m2

m2

m1

m1

m2m4

m3

m3

m7

m6

m5

SNUTO

p1

p2

p3

m1

m2m1

m1

m2m4

m3

m3

m7

m6

m5

WNUTOm2

TO specifications

TO(UA,SUTO)The strongest TO spec.

p1

p2

p3

m2

m2

m2

m1

m1

m1

m3

m3

m6

m6

p1

p2

p3

m2

m2

m2

m1

m1

m1m4

m3

m3

m6

m6

m5

TO(NUA,SUTO) TO(UA,SUTO)(Strongest total order)

TO(NUA,SUTO)

TO specifications (2)

TO(UA,WUTO)

m3

p1

p2

p3

m2

m2

m1

m1

m1

m3

m3

m4

m4

m4

m3

p1

p2

p3

m2m1

m1

m1

m3

m4

m4

m2m3 m4

m5

m6

m6

m6

m2

m2

TO(NUA,WUTO)

TO(UA,WUTO)

TO(UA,SUTO)(Strongest total order)

TO(NUA,SUTO)

TO(NUA,WUTO)

TO specifications (3)

TO(UA,WNUTO)

m4

p1

p2

p3

m2

m2

m1

m1

m1

m3

m3

m3

m4

m4

m2

TO(NUA,WNUTO)

m4

p1

p2

p3

m2m1

m1

m1

m3

m3

m4

m2m3 m4

m5

m6

m6

m6m2 TO(NUA,WNUTO)

TO(UA,WNUTO)

TO(UA,SUTO)(Strongest total order)

TO(NUA,SUTO)TO(UA,WUTO)

TO(NUA,WUTO)