View
231
Download
0
Category
Tags:
Preview:
Citation preview
EEC 688/788EEC 688/788Secure and Dependable ComputingSecure and Dependable Computing
Lecture 13Lecture 13
Wenbing ZhaoWenbing ZhaoDepartment of Electrical and Computer EngineeringDepartment of Electrical and Computer Engineering
Cleveland State UniversityCleveland State University
wenbing@ieee.orgwenbing@ieee.org
22
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
OutlineOutline• Fault tolerance and replication• Event ordering• Group communication systems– Ordered multicast– Techniques to implement ordered multicast
• Reference: – Reliable distributed systems, by K. P. Birman, Springer;
Chapter 14-16
33
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
RedundancyRedundancy
• To tolerant fault, some form of redundancy must be used– Replication in time (transaction processing)– Replication in space– Redundancy in software design (n-version
programming)
• The three types of replication techniques are complimentary to each other
44
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Replication in TimeReplication in Time
• Replication in time– From the present state, apply one or more operations – If the system fails before completion, abort and rollback
to the original state– Start all over again
• Example: transaction processing– Essence: roll-backward recovery
55
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Replication in SpaceReplication in Space
• Replication in space: – Run multiple instances (replicas) of the systems so that
if one fails, one or more replicas can take over– Must assume independent faults– Must ensure consistency among the replicas
66
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
N-version ProgrammingN-version Programming
• N-version programming (redundancy in software design)– The system (or component) has n-different designs and
implementations– In case of permanent software bugs, a different version
is used for each replica, or for repeated executions
77
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Replication is not a Trivial TaskReplication is not a Trivial Task
• Suppose we want to replicate a server using the most popular (an inexpensive) approach– We run two servers on separate computers– The primary sends a log (its state, and/or
logged incoming messages) to the backup – If primary crashes, the backup soon catches
up and can take over
88
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Split Brain SyndromeSplit Brain Syndrome
primary
backup
Clients initially connected to primary, which keeps thebackup up to date. Backup collects the log and updates its state
log
99
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Split Brain SyndromeSplit Brain Syndrome
Transient problem causes some links to break but not all.Backup thinks it is now primary, primary thinks backup is down
primary
backup
1010
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Split Brain SyndromeSplit Brain Syndrome
Some clients still connected to primary, but one has switchedto backup and one is completely disconnected from both
primary
backup
1111
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Replica ConsistencyReplica Consistency
• Replicas must be coordinated appropriately so that we can achieve strong replica consistency:– At the end of each execution step, all replicas must
have the same state– The outputs from every replica must be consistent all
the time
1212
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Replica ConsistencyReplica Consistency
• Techniques to achieve strong replica consistency– As a prerequisite, we need to have an agreement on
the membership of the replica group– Ensure that the same set of inputs is delivered to all
replicas and the inputs must be in the same order– Ensure deterministic execution due to each input at
every replica (applicable to active and semi-active replication)
1313
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
One Copy ImageOne Copy Image• When a component or system is replicated, we
must ensure that the replicated unit appears as a single copy to external components/systems that interact with it
• Need a gateway in between the replicated and non-replicated units
• Alternatively, the gateway module can be integrated with the non-replicated unit
• Main tasks of the gateway– Multicast requests/replies to the replicas– Detect and suppress duplicated replies/requests
1414
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Replication StylesReplication Styles• Active replication
– Every input (request) is executed by every replica– Every replica generates the outputs (replies)– Voting is needed to cope with non-fail-stop faults
• Passive replication– One of the replicas is designated as the primary replica– Only the primary replica executes requests– The state of the primary replica is transferred to the backups
periodically or after every request processing
• Semi-active replication– One of the replicas is designated as the leader (or primary)– The leader determines the order of execution– Every input is executed by every replica per the leader’s instruction
1515
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
DuplicateInvocationSuppressed
DuplicateResponsesSuppressed
Active ReplicationActive ReplicationActively Replicated
Client Object AActively Replicated
Server Object B
RM RM RM RM RM
1616
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Active Replication with VotingActive Replication with Voting
Question: to cope with f number of faults (non-malicious), how many replicas are needed?
1717
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
State Transfer
State State
Response
Invocation
Passive ReplicationPassive ReplicationPassively Replicated
Client Object APassively Replicated
Server Object B
PrimaryReplica
PrimaryReplica
RMRM RM RMRM
Question: can passive replication tolerate non-fail-stop faults?
1818
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Ordering info
Ordering info Ordering info
Response
Invocation
Semi-Active ReplicationSemi-Active ReplicationSemi-Actively Replicated
Client Object ASemi-Actively Replicated
Server Object B
PrimaryReplica
PrimaryReplica
RMRM RM RMRM
1919
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Ensuring Strong Replica ConsistencyEnsuring Strong Replica Consistency
• Possible strategies– For active replication, use a group
communication system that guarantees total ordering of all messages (plus deterministic processing in each replica)
– Passive replication with systematic checkpointing
– Semi-active replication– Use two-phase commit
2020
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Total Ordering of MessagesTotal Ordering of Messages• What is total ordering of messages?
– All replicas receive the same set of messages in the same order– Atomic multicast – If a message is delivered to one replica, it is also
delivered to all correct replicas
• With replication, we need to ensure total ordering of messages sent by a group of replicas to another group of replicas– FIFO ordering between one sender and a group is not sufficient
m1
m2m1
m1m1
m1
m2m2
m1
2121
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Potential Sources of Non-determinismsPotential Sources of Non-determinisms
• Multithreading– The order of accesses of shared data by different threads might
not be the same at different replicas
• System calls/library calls– A call at one replica might succeed while the same call might fail
at another replica. E.g., memory allocation, file access
• Host/process specific information– Host name, process id, etc.– Local clocks - gettimeofday()
• Interrupts– Delivered and handled asynchronously – big problem
2222
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Event OrderingEvent Ordering• “Time, Clocks, and the Ordering of Events in
a Distributed System”, by Leslie Lamport, Communications of the ACM, July 1978, Volume 21, Number 7, pp.558-565– What usually matters is not that all processes agree
on exactly what time it is, but rather, that they agree on the order in which events occur
2323
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Happens-Before RelationHappens-Before Relation
• Assumptions:– The system is composed of a collection of processes,
each process consists of a sequence of events– The events of a process form a sequence, where a
occurs before b in this sequence if a happens before b– The sending or receiving of a message is an event in a
process
2424
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Happens-Before RelationHappens-Before Relation• The happens-before relation “→” on the set of
events of a system is the relation satisfying the following three conditions:– If a and b are events in the same process, and a
comes before b, then a → b– If a is the sending of a message by one process and
b is the receipt of the same message by another process, then a → b
– If a → b and b → c, then a → c
2525
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Partial OrderingPartial Ordering• Not all events have the happens-before relationship• Two distinct events a and b are said to be
concurrent if a → b and b → a– Neither event can causally affect the other– This introduces a partial ordering of events in a system
with concurrently operating processes
• “a happens before b” means that information can flow from a to b
• “a is concurrent with b” means that there is no information flow between a and b
2626
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
How to Capture the Partial Ordering?How to Capture the Partial Ordering?
• Use logical clocks to capture the partial ordering– Define a clock Ci for each process Pi. Assign a
number Ci(a) to any event a in that process
– The entire system of clocks is represented by the function C which assigns to any event b the number C(b), where C(b) =Cj(b) if b is an event in process Pj
– The clocks Ci are logical clocks rather than physical clocks
2727
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Lamport ClockLamport Clock
• A Lamport logical clock is a monotonically increasing software counter
• Each process Pi keeps its own logical clock Ci to apply Lamport timestamps to events
• To capture the happens-before relation →, processes must do the following:– Before each event at Pi: Ci := Ci+1
– When Pi sends a message m, it piggybacks t = Ci
– When Pj receives (m,t): Cj := max(Cj,t) + 1
e → e’ C(e) < C(e’)
2828
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Lamport Clock: An ExampleLamport Clock: An Example
a b
c d
e f
m1
m2
21
3 4
51
p1
p2
p3
Physical time
2929
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Group Communication SystemGroup Communication System
• Services provided by the GCS– Membership service: who is up and who is down
• Deals with failure detection and more
– Reliable, ordered, multicast service• FIFO, causal, total
– Virtual synchrony service• Virtual synchrony synchronizes membership change with
multicasts
• GCS is often used to build fault tolerant systems
3030
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Reliable MulticastReliable Multicast• Reliable multicast – the message is targeted to
multiple receivers, and all receivers receive the message reliably– Positive or negative acknowledgement– Need to avoid ack/nack implosion
• Distinguish receiving from delivery!
Application
Middleware
Receiving
Delivering
3131
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Ordered Reliable MulticastOrdered Reliable Multicast• Ordered reliable multicast – if many messages are
multicast by many senders, in what order the messages are delivered at the receivers?– First in first out (FIFO)– Causal – the causal relationship among msgs preserved– Total – all msgs are delivered at all receivers in the same
order
3232
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
FIFO Ordered MulticastFIFO Ordered Multicast
• FIFO or sender ordered multicast:Messages are delivered in the order they were sent (by any single sender)
p
q
r
s
a
b c d
e
delivery of c to p is delayed until after b is delivered
3333
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Causally Ordered MulticastCausally Ordered Multicast
• Causal or happens-before ordering:If send(a) send(b) then deliver(a) occurs before deliver(b) at common destinations
p
q
r
s
a
b
3434
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Causally Ordered MulticastCausally Ordered Multicast
• Causal or happens-before ordering:If send(a) send(b) then deliver(a) occurs before deliver(b) at common destinations
p
q
r
s
a
b cdelivery of c to p is delayed until after b is delivered
3535
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Causally Ordered MulticastCausally Ordered Multicast
• Causal or happens-before ordering:If send(a) send(b) then deliver(a) occurs before deliver(b) at common destinations
p
q
r
s
a
b c
e
delivery of c to p is delayed until after b is deliverede is sent (causally) after b
3636
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Causally Ordered MulticastCausally Ordered Multicast
• Causal or happens-before ordering:If send(a) send(b) then deliver(a) occurs before deliver(b) at common destinations
p
q
r
s
a
b c d
e
delivery of c to p is delayed until after b is delivereddelivery of e to r is delayed until after b&c are delivered
3737
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Totally Ordered MulticastTotally Ordered Multicast
• Total ordering:Messages are delivered in same order to all recipients (including the sender)
p
q
r
s
a
b c d
e
all deliver a, b, c, d, then e
3838
04/21/2304/21/23EEC688/788: Secure & Dependable EEC688/788: Secure & Dependable
ComputingComputing Wenbing ZhaoWenbing Zhao
Implementing Total OrderingImplementing Total Ordering
• Use a token that moves around– Token has a sequence number– When you hold the token you can send the next burst
of multicasts
• Use a sequencer to order all multicast– Message is first multicast to all, including the
sequencer; then the sequencer determines the order for the message and informs all
– Or send to the sequencer and the sequencer multicast with total order information
– Each sender can take turn to serve as the sequencer
Recommended