Embed Size (px)
Citation preview
Consistency and Replication
Chapter 6
6.1 Concepts
• Reasons for Replication• Reliability
• Earthquake, flood• Misoperation
• Performance• Place copies of data close to the processes using them
• Web cache
• Difficulty• Keep all replicas up-to-date
Object Replication
• Distributed remote objects• Simultaneous access
• Protected by itself• Protected by the server it resides
• Solve inconsistency problem• replication-aware objects• system’s responsibility
Object Replication (1)
Organization of a distributed remote object shared by two different clients.
Object Replication (2)
a) A remote object capable of handling concurrent invocations on its own.b) A remote object for which an object adapter is required to handle concurrent
invocations
Object Replication (3)
a) A distributed system for replication-aware distributed objects.b) A distributed system responsible for replica management
Object Replication
• Replication?• normally improve performance
• but not always, it depends on the access-to-update ratio
• P accesses a local replica N times per sec;
• The replica is updated M times per sec;
• If N << M, then many updated versions of the local replica will never be accessed by P;
• Rendering a lot of useless network communication.
6.2 Data-centric Consistency Models
• Terminology• Data store: data that is physically
distributed across multiple machines.
• Consistency model: a contract between processes and the data store.
• If processes agree to obey certain rules, the store promises to work correctly.
Data-Centric Consistency Models
The general organization of a logical data store, physically distributed and replicated across multiple processes.
Strict Consistency
Any read on a data item x returns a value corresponding to the result of the most recent write on x.
Notation:• Wi(x)a: a write by process Pi to data item x with the value a;
• Ri(x)b: a read by process Pi to data item x with the value b;
Strict Consistency
Behavior of two processes, operating on the same data item.• A strictly consistent store.• A store that is not strictly consistent.
Strict Consistency
• Counterexample• x is a data item on machine B
• a process on machine A reads x at T1 by sending a message to B to get the value of x
• a process on machine B writes x at T2
• T2 - T1 = 1 ns
• distance(A, B)= 3 meters
Strict Consistency
• Summary:• Ideal, but impossible to be implemented in
a DS.
• All writes are instantaneously visible to all processes.
• An absolute global time order is maintained.
Sequential Consistency
The result of any execution is the same as if the (read and write) operations by all processes on the data store were executed in some sequential order and the operations of each individual process appear in this sequence in the order specified by its program.
Sequential Consistency
• Explanation• Any valid interleaving of read and write
operations is acceptable behavior, but all processes see the same interleaving of operations.
• Nothing about time and no reference to the “most recent” word.
Sequential Consistency (1)
a) A sequentially consistent data store.b) A data store that is not sequentially consistent.
R(x)a R(x)b
(c)
c) A sequentially consistent data store.
Linearizable Consistency
The result of any execution is the same as if the (read and write) operations by all processes on the data store were executed in some sequential order and the operations of each individual process appear in this sequence in the order specified by its program. In addition, if tsop1(x) < tsop2(y), then operation OP1(x) should precede OP2(y) in this sequence.
• tsop(x): a timestamp assigned to operation OP that is performed
on data item x.
Linearizability and Sequential Consistency
Three concurrently executing processes.
Process P1 Process P2 Process P3
x = 1;
print ( y, z);
y = 1;
print (x, z);
z = 1;
print (x, y);
Trivial question : how many valid interleaved execution sequences based on possible program order?
Linearizability and Sequential Consistency
Four valid execution sequences for the processes of the previous slide. The vertical axis is time.
x = 1;
print (y, z);
y = 1;
print (x, z);
z = 1;
print (x, y);
Prints: 001011
Signature: 001011
(a)
x = 1;
y = 1;
print (x,z);
print(y, z);
z = 1;
print (x, y);
Prints: 101011
Signature: 101011
(b)
y = 1;
z = 1;
print (x, y);
print (x, z);
x = 1;
print (y, z);
Prints: 010111
Signature:
110101
(c)
y = 1;
x = 1;
z = 1;
print (x, z);
print (y, z);
print (x, y);
Prints: 111111
Signature:
111111
(d)
Is signature 001001 allowed?
Casual Consistency
• Causality• If event B is caused or influenced by an
earlier event A, then the causality requires that everyone else first see A, then see B.
• Example: • Process P1 writes a variable x, then process P2
reads x and writes y. It’s possible that the computation of y may have depended on the value of x read by P2 (i.e., the value written by P1).
Casual Consistency (1)
Writes that are potentially casually related must be seen by all processes in the same order. Concurrent writes may be seen in a different order on different machines.
Casual Consistency (2)
This sequence is allowed with a casually-consistent store, but not with sequentially or strictly consistent store.
Casual Consistency (3)
a) A violation of a casually-consistent store.b) A correct sequence of events in a casually-consistent store.
R(x)a R(x)b
(c)
FIFO Consistency (1)
Writes done by a single process are seen by all other processes in the order in which they were issued, but writes from different processes may be seen in a different order by different processes.
FIFO Consistency (2)
A valid sequence of events of FIFO consistency
FIFO Consistency (3)
Statement execution as seen by the three processes from the previous slide. The statements in bold are the ones that generate the output shown.
x = 1;
print (y, z);
y = 1;
print(x, z);
z = 1;
print (x, y);
Prints: 00
(a)
x = 1;
y = 1;
print(x, z);
print ( y, z);
z = 1;
print (x, y);
Prints: 10
(b)
y = 1;
print (x, z);
z = 1;
print (x, y);
x = 1;
print (y, z);
Prints: 01
(c)
Signature: 001001 which is not allowed in sequential consistency
FIFO Consistency (4)
Two concurrent processes.
Process P1 Process P2
x = 1;
if (y == 0) kill (P2);
y = 1;
if (x == 0) kill (P1);
P1 is killedP2 is killed
Neither is killedBoth are killed
?
Weak Consistency
• Synchronization Variable (S)• A process P performs operations only on
its locally available copy of the data store.
• When the data store is synchronized, all local writes by process P are propagated to the other copies, whereas writes by other processes are brought in to P’s copy.
• synchronize(S) synchronizes all local copies of the data store.
Weak Consistency (1)
• Accesses to synchronization variables associated with a data store are sequentially consistent
• No operation on a synchronization variable is allowed to be performed until all previous writes have been completed everywhere
• No read or write operation on data items are allowed to be performed until all previous operations to synchronization variables have been performed.
Weak Consistency (2)
A program fragment in which some variables may be kept in registers.
int a, b, c, d, e, x, y; /* variables */int *p, *q; /* pointers */int f( int *p, int *q); /* function prototype */
a = x * x; /* a stored in register */b = y * y; /* b as well */c = a*a*a + b*b + a * b; /* used later */d = a * a * c; /* used later */p = &a; /* p gets address of a */q = &b /* q gets address of b */e = f(p, q) /* function call */
Weak Consistency (3)
a) A valid sequence of events for weak consistency.b) An invalid sequence for weak consistency.
Release Consistency (1)
A valid event sequence for release consistency.
Release Consistency (2)
• Before a read or write operation on shared data is performed, all previous acquires done by the process must have completed successfully.
• Before a release is allowed to be performed, all previous reads and writes by the process must have completed
• Accesses to synchronization variables are FIFO consistent (sequential consistency is not required).
Entry Consistency (1)
• An acquire access of a synchronization variable is not allowed to perform with respect to a process until all updates to the guarded shared data have been performed with respect to that process.
• Before an exclusive mode access to a synchronization variable by a process is allowed to perform with respect to that process, no other process may hold the synchronization variable, not even in nonexclusive mode.
• After an exclusive mode access to a synchronization variable has been performed, any other process's next nonexclusive mode access to that synchronization variable may not be performed until it has performed with respect to that variable's owner.
Entry Consistency (2)
A valid event sequence for entry consistency.
Summary of Consistency Models
a) Consistency models not using synchronization operations.
b) Models with synchronization operations.
Consistency Description
Strict Absolute time ordering of all shared accesses matters.
LinearizabilityAll processes must see all shared accesses in the same order. Accesses are furthermore ordered according to a (nonunique) global timestamp
SequentialAll processes see all shared accesses in the same order. Accesses are not ordered in time
Causal All processes see causally-related shared accesses in the same order.
FIFOAll processes see writes from each other in the order they were used. Writes from different processes may not always be seen in that order
(a)
Consistency Description
Weak Shared data can be counted on to be consistent only after a synchronization is done
Release Shared data are made consistent when a critical region is exited
Entry Shared data pertaining to a critical region are made consistent when a critical region is entered.
(b)
6.3 Client-centric Consistency Models
• Client-centric consistency provides guarantees for a single client concerning the consistency of accesses to a data store by that client.
• No guarantees are given concerning concurrent accesses by different clients.
Eventual Consistency
• If no updates take place for a long time, all replicas will gradually become consistent.
• Example: WWW– web pages are updated by a single one– no write-write conflicts– out-of-date local cache – but, acceptable inconsistency
• Eventual consistent data store work fine as long as clients always access the same replica.
Problem
The principle of a mobile user accessing different replicas of a distributed database.
Notations
• xi[t]: the version of date item x at local copy Li at time t.
• WS(xi[t]): a series of write operations to x at Li
that took place since initialization, till time the version xi[t].
• WS(xi[t1]; xj[t2]): If operations in WS(xi[t1]) have also been performed at local copy Lj at a later time t2.
• The time index will be omitted if the timing is clear from the context.
(1) Monotonic Reads
• A data store is said to provide monotonic-read consistency if the following condition holds:
If a process reads the value of a data item x, any successive read operation on x by that process will always return that same value or a more recent value.
If a process has seen a value of x at time t, it will never see an older version of x at a later time
(1) Monotonic Reads
The read operations performed by a single process P at two different local copies of the same data store.a) A monotonic-read consistent data storeb) A data store that does not provide monotonic reads.
(2) Monotonic Writes
• In a monotonic-write consistency store, the following condition holds:
A write operation by a process on a data item x is completed before any successive write operation on x by the same process.
A write operation on a copy of data item x is performed only if that copy has been brought up to date by means of any preceding write operation, which may have taken place on other copies of x.
(2) Monotonic Writes
The write operations performed by a single process P at two different local copies of the same data storea) A monotonic-write consistent data store.b) A data store that does not provide monotonic-write consistency.
(3) Read Your Writes
• A data store is said to provide read-your-writes consistency, if the following condition holds:
The effect of a write operation by a process on data item x will always be seen by a successive read operation on x by the same process.
A write operation is always completed before a successive read operation by the same process, no matter where that read operation takes place.
(3) Read Your Writes
a) A data store that provides read-your-writes consistency.b) A data store that does not.
(4) Writes Follow Reads
• A data store is said to provide writes-follow-reads consistency, if the following condition holds:
A write operation by a process on a data item x following a previous read operation on x by the same process, is guaranteed to take place on the same or a more recent value of x that was read. Any successive write operation by a process on a data item x will be performed on a copy of x that is up to date with the value most recently read by that process.
(4) Writes Follow Reads
a) A writes-follow-reads consistent data storeb) A data store that does not provide writes-follow-reads consistency
(4) Writes Follow Reads
• Example of a network newsgroup– A user first reads an article A;– Then he reacts by posting a response B;– B will be written to any copy of the newsgroup
only after A has been written as well.
The writes-follows-reads consistency assures that reactions to articles are stored at a local copy only if the original is stored there as well.
6.4 Distribution Protocols
• Replica Placement– Permanent Replicas
• Mirroring sites
– Server-Initiated Replicas• push caches• Some servers dynamically replicate files to
other servers close to demanding clients.
– Client-initiated Replicas• Client caches
Replica Placement
The logical organization of different kinds of copies of a data store into three concentric rings.
Server-Initiated Replicas
Specific files on a server can be migrated or replicated to servers placed in the proximity of clients that issue many requests for those files.
Server-Initiated Replicas
Counting access requests from different clients.
cntQ(P, F): access count from P for the file F at server Q, where P is the same “closest” server for C1 and C2.
Server-Initiated Replicas
• Each server keeps track of access counts per file, and where access requests come from.
• When the number of requests for a specific file F at server S drops below a deletion threshold del(S, F), that file can be removed from S.
• A replication threshold rep(S, F) indicates that the number of requests for a specific file is so high that it may be worthwhile replicating it on another server.
• If for some server P, cntQ(P, F) exceeds more than half of the total requests for F at Q, server Q will attempt to migrate F to P.
Client-Initiated Replicas
• A cache could be located on the client’s machine, or on a separate machine in the same LAN as the client.
• Caches work fine as long as the fetched data have not been modified in the meantime.– Data are generally kept in a cache for a limited amount
time
Update Propagation
• What is actually to be propagated?– Propagate only a notification of an update
• Invalidation protocols
• Small read-to-write ratio
• Little network bandwidth
– Transfer data from one copy to another• High read-to-write ratio
– Propagate the update operation to other copies
• Little network bandwidth
Pull versus Push Protocols
• Push-based approach(server-based approach)– Updates are propagated to other replicas
without those replicas even asking for the updates.
• Pull-based approach(client-based approach)– A server or client requests another server to
send it any updates it has at that moment.
Pull versus Push Protocols
A comparison between push-based and pull-based protocols in the case of multiple client, single server systems.
Issue Push-based Pull-based
State of server List of client replicas and caches None
Messages sent Update (and possibly fetch update later) Poll and update
Response time at client
Immediate (or fetch-update time) Fetch-update time
Consistency degree
high low
Read-to-update ratio
high low
Epidemic Protocols
• Update Propagation Models– Terminology:
• infective: holds an update and is willing to spread it
• susceptible: has not been updated
• removed: is not willing or able to spread its update
– Anti-entropy algorithm• A server P picks another server Q at random and
subsequently exchanges updates with Q.– P only pushes its own updates to Q
– P only pulls in new updates from Q
– P and Q send updates to each other (a push-pull approach)
Epidemic Protocols
– Gossiping algorithm• If server P has just been updated for data item x, it
contacts an arbitrary other server Q and tries to push the update to Q.
• If Q was already updated by another server, P may lose interest in spreading the update any further. That is, it then becomes removed.
6.5 Consistency Protocols
• A consistency protocol describes an implementation of a specific consistency model– Primary-based Protocols
• Remote-write Protocols
• Local-write Protocols
– Replicated-write Protocols• Active Replication
• Quorum-based Protocols
– Cache-coherence Protocols (omitted)
Remote-Write Protocols (1)
Primary-based remote-write protocol with a fixed server to which all read and write operations are forwarded.
Remote-Write Protocols (2)
The principle of primary-backup protocol.
Primary-based Protocols
Primary-backup protocols provides a straightforward implementation of sequential consistency, as the primary can order all incoming writes.
Local-Write Protocols (1)
Primary-based local-write protocol in which a single copy is migrated between processes.
Local-Write Protocols (2)
Primary-backup protocol in which the primary migrates to the process wanting to perform an update.
Active Replication
• In replicated-write protocols, write operations can be carried out at multiple replicas instead of only one, as in the case of primary-based replicas.
• sequencer: a central coordinator.– totally-ordered multicasting: first forward each
operation to the sequencer, which assign it a unique sequence number and subsequently forwards the operation to all replicas.
Active Replication (1)
The problem of replicated invocations.
Active Replication (2)
a) Forwarding an invocation request from a replicated object.b) Returning a reply to a replicated object.
Quorum-Based Protocols
• Suppose a file is replicated on N servers– Algorithm
• Whenever a client want to update a file, it needs to get permissions from at least (N/2 + 1) servers. If ok, then the client change the file and associate a new version number to the file.
• To read a file, a client get the file as well as the version number from at least (N/2 + 1) servers. If all version numbers agree, then it gets the most recent copy of the file.
Quorum-Based Protocols
• Suppose a file is replicated on N servers– Gifford’s Algorithm
• To read the file, a client needs to assemble a read quorum, an arbitrary collection of any NR servers, or more.
• To update the file, a client needs to assemble a write quorum, an arbitrary collection of at least Nw servers.
• Constraints:– NR + Nw > N => prevent read-write conflicts
– Nw > N/2 => prevent write-write conflicts
Quorum-Based Protocols
Three examples of the voting algorithm:a) A correct choice of read and write setb) A choice that may lead to write-write conflictsc) A correct choice, known as ROWA (read one, write all)
Homework #3 due 12/24, 2pm
• Chapter 5 question 18, 7:
Systems that use locking for concurrency control usually distinguish read locks from write locks. What should happen if a process has already acquired a read lock and now wants to change it into a write lock? What about changing a write lock into a read lock?
• In Fig. 5-12 (or page 31 of this slide) we have two ELECTION messages circulating simultaneously. While it does no harm to have two of them, it would be more elegant if one could be killed off. Devise an (see the next page)
Homework #3 due 12/24, 2pm
algorithm for doing this without affecting the operation of the basic election algorithm.
• Chapter 6 question 9, 10:
• In Fig. 6-7, is 000000 a legal output for a distributed shared memory that is only FIFO consistent? Explain your answer.
• In Fig. 6-8, is 001110 a legal output for a sequentially consistent memory? Explain your answer.
