Upload
reid-lanes
View
253
Download
0
Tags:
Embed Size (px)
Citation preview
Fault Tolerance (I)
Topics
Basic concepts Physical Redundancy Information Redundancy Timing Redundancy RAID
Readings
Tannenbaum: 7.1,7.2
Introduction
A characteristic feature of distributed systems that distinguishes them from single-machine systems is the notion of partial failure
A partial failure may happen when one component in a distributed system fails.
This failure may affect the proper operation of other components, while at the same time leaving other components unaffected.
Introduction
An important goal in design is to construct the system in such a way that it can automatically recover from partial failures without seriously affecting the overall performance.
The distributed system should continue to operate in an acceptable way while repairs are being made.
By the way…. Computing systems are not very
reliable OS crashes frequently (Windows), buggy
software, unreliable hardware, software/hardware incompatibilities
Until recently: computer users were “tech savvy”
• Could depend on users to reboot, troubleshoot problems
By the way…. Computing systems are not very
reliable (cont) Growing popularity of Internet/World Wide
Web• “Novice” users • Need to build more reliable/dependable systems
Example: what if your TV (or car) broke down every day?
• Users don’t want to “restart” TV or fix it (by opening it up)
Need to make computing systems more reliable
Characterizing Dependable Systems
Dependable systems are characterized by Availability
• This refers to the percentage of time system may be used immediately
Reliability• Mean time to failure (MTTF) I.e., mean time
between failures. Safety
• How serious is the impact of a failure Maintainability
• How long does it take to repair the system Security
Characterizing Dependable Systems
Availability and reliability are not the same thing.
If a system goes down for a millisecond every hour, it has an availability of over 99.9999 percent, but it is still highly unreliable.
A system that never crashes but is shut down for two weeks every August has high reliability but only 96 percent availability.
Definitions
A system fails when it does not perform according to its specification.
An error is part of a system state that may lead to a failure.
A fault is the cause of an error.
Definitions
Types of Faults Transient
• Occur once and then disappear.• If the operation is repeated, the fault goes away.• Example: Bird flying through the beam of a
microwave transmitter may cause lost bits on some network (not to mention a roasted bird).
Definitions
Types of Faults (continued) Intermittent
• Occurs and then vanishes of its own accord, then reappears, etc;
• A loose connector will often cause a intermittent fault.
Permanent• Continues to exist until the faulty component is
repaired.• Burnt-out chips, software bugs, and disk head
crashes.
A fault tolerant system does not fail in the presence of faults.
Server Failure Models
Type of failure Description
Crash failure A server halts, but is working correctly until it halts
Omission failure Receive omission Send omission
A server fails to respond to incoming requestsA server fails to receive incoming messagesA server fails to send messages
Timing failure A server's response lies outside the specified time interval
Response failure Value failure State transition failure
The server's response is incorrectThe value of the response is wrongThe server deviates from the correct flow of control
Arbitrary failure A server may produce arbitrary responses at arbitrary times
Server Failure Models
Crash Failure (fail-stop) A server halts, but is working correctly until
it halts. Example: An OS that comes to a grinding
halt and for which there is only one solution: reboot
Server Failure Models
Omission Failure This occurs when a server fails to respond to
incoming requests or fails to receive incoming messages or fails to send messages.
There are many reasons for an omission failure including:
• The connection between a client and a server has been correctly established, but there was no thread to listen to incoming requests.
• A send buffer overflows; The server may need to be prepared that the client will reissue its previous request.
• An infinite loop where each iteration causes a forked process.
Server Failure Models
Timing Failures A server’s response lies outside the
specified time interval. An e-commerce site may state that the
response to a user should be no more than 5 seconds (actually this is too long).
In a video-on-demand application, the client is to receive frames at 25 frames per second give or take 2 frames.
Timing failures are very difficult to deal with.
Server Failure Models
Response Failure A server’s response is incorrect: a wrong
reply to a request is returned or when a server reacts unexpectedly to an incoming request.
Example: A search engine that systematically returns web pages not related to any of the used search terms.
Example: A server receives a message that it cannot recognize.
Server Failure Models
Arbitrary (Byzantine) Failures Arbitrary failures occur Server is producing output it should never
have produced, but which cannot be detected as being incorrect.
A faulty server may even be maliciously working together with other servers to produce intentionally wrong answers.
Server Failure Models
Ideally, we want fail-stop processes. A fail-stop process will simply stop
producing output in such a way that its halting can be detected by other processes.
The server may be so friendly to announce it is about to crash.
The reality is that processes are not that friendly.
We rely on other processes to detect the failure.
Server Failure Models
Problem: How to tell the difference between a process that has halted and a process that is just slow. Timeouts are great but theoretically you
cannot place an exact time on when to expect a response.
If most of the time, the timeout interval is too high then you are delaying the system from reacting to the failure.
Failure Masking by Redundancy
If a system is to be fault tolerant, the best it can do is to try to hide the occurrence of failures from other processes.
Key technique: Use redundancy Types of redundancy
Information redundancy Physical redundancy Time redundancy
Physical Redundancy
Extra equipment or processes are added to make it possible for the system as a whole to tolerate the loss or malfunctioning of some components.
Physical redundancy can thus be done in either hardware or in software.
Examples in hardware: Aircraft: 747’s have 4 engines but fly on 3. Space shuttle: Has 5 computers Electronic circuits
Physical Redundancy
Triple modular redundancy.
Physical Redundancy
For electronic circuits, each device is replicated three times.
Following each stage in the circuit is a triplicated voter.
Each voter is a circuit that has three inputs and one output.
If two or three of the inputs are the same, the output is equal to that input.
If all three inputs are different, the output is undefined.
This kind of design is known as TMR (Triple Modular Redundancy).
Physical Redundancy
TMR can be applied to any hardware unit.
The TMR can completely mask the failure of one hardware unit.
No explicit actions need to be performed for error detection, recovery, etc;
Particularly suitable for transient failures if we assume the basic TMR scheme (one voter, three replicas).
Physical Redundancy
This scheme can’t handle the failure of two units.
Once an unit fails, it is essential that both units should continue to work correctly.
The TMR scheme depends critically on the voting element. The voting element is typically a simple circuit and highly reliable circuits of this complexity can be built.
The failure of a single voter cannot be tolerated.
Physical Redundancy
The TMR approach can be generalized to replicating N units. This is called the NMR approach.
The larger N is then the higher the number of faults that can be completely masked.
Physical Redundancy
The basic TMR/NMR scheme is often complemented with sparing.
Sparing is often referred to as stand-by redundancy since the redundant or spare units usually are not operating online.
The restoring organ for sparing is a switch.
An error detector is also required to determine when the on-line unit has failed.
Failed units may be replaced by a spare.
Physical Redundancy
Some reliability results: Overall reliability decreases when the
degree of redundancy is increased above a certain amount.
TMR provides the least potential for reliability improvement.
NMR systems with spares provide the highest reliability.
Information Redundancy
Coding is often used in information redundancy.
Coding has been extensively used for improving the reliability of communication.
The basic idea is to add check bits to the information bits such that errors in some bits can be detected, and if possible corrected.
The process of adding check bits to information bits is called encoding.
The reverse process of extracting information from the encoded data is called decoding.
Information Redundancy
Detectability/Correctability of a Code A code defines a set of words that are
possible for that code. The Hamming distance of a code is the
minimum number of bit positions in which any two words in the code differ.
If d is the Hamming distance, D is the number of bit errors that it can detect and C is the number of bit errors it can correct, then the following relation is always true:
d = C +D +1 with D C
Information RedundancyDetectability/Correctability of a Code Let’s say that you have a code that looks like this:
000001010011100101110111
Hamming distance is one. You can’t detect an error. Why? Let’s say that a fault transforms 001 to 011.
How do you know this is a fault vs the possibility that 011 is correct?
Information Redundancy
Detectability/Correctability of a Code On the otherhand, let’s say that you have
the following code of 3 codewords:0000 0011 1100If a fault changes one bit in a correct word it will
result in a word that is not in the above list. This is not true if two bits are changed. Hence, the above code can only tolerate one fault.
You can’t correct. Let’s say that 0000 changes to 0010. You know there is an error, but how do you know that this should go to 0000 and not 0011.
Information Redundancy
Simple Parity Bits Simple parity bits have been in common use
in computer systems for many years. The parity bit is selected so that the total
number of 1’s in the codeword is odd (even) for an odd-parity (even-parity) code.
This means that the Hamming distance is 2. The parity bit can only detect single bit
errors.
Information Redundancy
Simple Parity Bits Example (assume odd-parity):
Codeword is 000; the parity bit is 1 Codeword is 001; the parity bit is 0 Codeword is 010; the parity bit is 0 Let’s say that 000 is transferred as 0001.
The parity bit is set as 1 which results in a odd number of ones (remember we are only interested in an odd number of ones).
Information Redundancy
Simple Parity Bits All errors involving an odd number of
bits can be detected because such errors will produce an incorrect parity.
Information Redundancy
Hamming Codes Multiple parity bits are added such that
each parity bit is a parity of a subset of information bits. The code can detect and also correct errors.
Widely used in semiconductor memory and in disk arrays.
Information Redundancy
Hamming Codes Parity bits occupy the bit positions
1,2,4,…. (power of 2) in the encoding. The remaining are the data positions.
Let k be the number of parity bits. Let m be the number of data bits. The word length of the encoded word is
m+k.
Information Redundancy
Hamming Codes – Example Let k = 3 and m = 4 Bits in positions 1,2,4 are the parity bits. Label
these as c1,c2 and c3. Bits in positions 3,5,6,7 are the data bits.
Label these as d1,d2,d3 and d4. The value of parity bits is defined by the
following relations:c1 = d1d2d4 c2 = d1d3d4c3 = d2d3d4
0 000 4 100
1 001 d1 5 101 d2
2 010 6 110 d3
3 011 7 111 d4
Information Redundancy
Hamming Codes – Example Let the word to be transmitted be 1011. 001 010 011 100 101 110 111
c1 c2 d1 c3 d2 d3 d4
0 1 1 0 0 1 1
c1 = d1d2d4 c2 = d1d3d4c3 = d2d3d4
Information Redundancy
Hamming Codes – Example How do we come up with these relations? A Hamming code generator computes the
check bits according to the following scheme. • The binary representation of the position number
j is jk-1 ... j1 j0.
• The value of a check bit is chosen to give odd (or even) parity over all bit positions j such that ji = 1.
• Thus each bit of the data word participates in several different check bits.
Information Redundancy
Hamming Codes – Example Assume the word transferred is 1111.
c1 c2 d1 c3 d2 d3 d4
0 1 1 0 1 1 1
001 010 011 100 101 110 111
Transmitted improperly; wasoriginally a zero.
Information Redundancy
Hamming Codes – Example Location of bits in error
The check bits obtained from the relationship give above are XORed with the actual check bits obtained from the code.
c1’ = d1d2d4 = 1c2’ = d1d3d4 = 1c3’ = d2d3d4 = 1e1 = c1c1’ = 0 1 = 1 e2 = c2c2’ = 1 1 = 0e3 = c3c3’ = 0 1 = 1
Correction is done by simply complementing the bit.
If each error bit is 0, no error; else the error location bits specify the location of the bit in error
d2 is common to c1’ and c3’ as well as c1 and c3.
Information Redundancy
Hamming Codes – Example The use of Hamming codes becomes
more efficient, in terms of numbers of bits needed relative to the number of data bits, as the word size increases.
If the data word length is 8 bits, the number of check bits will be 4. This overhead is 50%.
If the word length is 84 bits, the number of check bits will be 7 giving an overhead of 9 percent.
Information Redundancy
Cyclic Redundancy Code (CRC) These codes are applied to a block of
data, rather than independent words. CRCs are commonly used in detecting
errors in data communication. A sequence of bits is represented as a
polynomial (generator polynomial).
Information Redundancy
Cyclic Redundancy Code (CRC) If the kth bit is 1, then the polynomial contains
xk. Example:1100101101
x9 + x8 + x5 + x3 + x2 + 1
Encoding To the data bit sequence, add (k+1) bits in the end. The extended data sequence is divided (modula 2)
by the generator polynomial. The final remainder is added to the data sequence to
form the encoded data.
Information Redundancy
Cyclic Redundancy Code (CRC) Decoding
The extra (k+1) bits are just discarded to obtain the original data bits.
Error checking: The data bits are again divided by the generator polynomial and the final remainder is checked with last (k+1) bits of the received data.
If there is a difference, an error has occurred.
Information Redundancy
Cyclic Redundancy Code (CRC) Through proper selection of the generating polynomial
CRC codes will: Detect all single bit errors in the data stream Detect all double bit errors in the data stream Detect any odd number of errors in the data stream Detect any burst error for which the length of the
burst is less than the length of the generating polynomial
Detect most all larger burst errors
Time Redundancy
An action is performed and if the need arises, it is performed again.
Example: If a transaction aborts, it can be redone with no harm.
This is especially useful when the faults are transient or intermittent.
Case Study
Let’s look at RAID(Redundant Array of Inexpensive Disks).
Motivation Improve disk access time by using arrays of
disks Disks are getting inexpensive. Lower cost disks:
• Less capacity.• But cheaper, smaller, and lower power.
Disk Organization 1
Interleaving disks. Supercomputing applications. Transfer of large blocks of data at high
rates.
...
Grouped read: single read spread over multiple disks
Disk Organization 1
What is interleaving? Assume you have 4 disks. Byte interleaving means that byte N is on
disk (N mod 4). Block interleaving means that block N is on
disk (N mod 4). All reads and writes involve all disks, which
is great for large transfers
Disk Organization 2
Independent disks. Transaction processing applications. Database partitioned across disks. Concurrent access to independent items.
...
Read Write
Problem: Reliability
Disk unreliability causes frequent backups.
Fault tolerance is needed, otherwise disk arrays are too unreliable to be useful.
RAID: Use of extra disks containing redundant information. Similar to redundant transmission of data.
RAID Levels
Different levels provide different reliability, cost, and performance.
The mean time to failure (MTTF) is a function of total number of disks, number of data disks in a group (G), number of check disks per group (C), and number of groups.
The number C is determined by RAID level.
First RAID Level
Mirrors Most expensive approach. All disks duplicated (G=1 and C=1). Every write to data disk results in write to
check disk. Reads can be from either disk. Double cost and half capacity.
Second RAID Level
Data is split at the bit level and spread over data and redundancy (check) disks.
Redundant bits are computed using Hamming code and placed in the redundancy disk.
Interleave data across disks in a group. Add enough check disks to detect/correct
error. Single parity disk detects single error. Makes sense for large data transfers. Small transfers mean all disks must be
accessed (to check if data is correct).
Third and Fourth RAID Level
The third RAID level is similar to the second RAID level except that splitting of data is at the byte level. There is one parity disk.
The fourth RAID level is similar to the third RAID level except that splitting of data is at the block level. There is one parity disk
The fifth RAID level is similar to the fourth RAID level except that check bits are distributed across multiple disks.
There are 8 RAID levels.
Process Resilience
The key approach to tolerating a faulty process is to organize several identical processes in a group.
Design issues include the following: When a message is sent to the group itself,
all members of the group receive it. Dealing with process groups
Problems of Agreement A set of processes need to agree on a value
(decision), after one or more processes have proposed what that value (decision) should be
Examples: mutual exclusion, election, transactions
Processes may be correct, crashed, or they may exhibit arbitrary (Byzantine) failures
Messages are exchanged on an one-to-one basis, and they are not signed
Problems of Agreement
The general goal of distributed agreement algorithms is to have all the nonfaulty processes reach consensus on some issue and to establish that consensus within a finite number of steps.
What if processes exhibit Byzantine failures.
This is often compared to armies in the Byzantine Empire in which there many conspiracies, intrigue and untruthfulness were alleged to be common in ruling circles.
The Two-Army Problem How can two perfect processes reach
agreement about 1 bit of information ? … over an unreliable communication channel
Red army: 5000 troops Blue army #1, #2: 3000 troops each
How can the blue armies reach agreement on when to attack ?
Their only means of communication is by sending messengers
… that may be captured by the enemy ! No solution!
The Two-Army Problem Proof by contradiction: Assume there is a
solution with a minimum number of messages Suppose commander of blue army 1 is General
Alexander and the command of the blue army 2 is General Bonaparte.
General Alexander sends a message to General Bonaparte reading “I have a plan; let’s attack at dawn tomorrow”.
The messenger gets through and Bonaparte sends him back a message with a note saying “Splendid idea, Alex. See you at dawn tomorrow.”
The messenger gets back.
The Two-Army Problem Proof by contradiction (cont)
Alexander wants to make sure that Bonaparte does know that the messenger got back safely so that Bonaparte is confident that Alexander will attack.
Alexander tells the messenger to go tell Bonaparte that his message arrived and the battle is set.
The messenger gets through, but now Bonaparte worries that Alexander does not know if the acknowledgement got through.
Bonaparte acknowledges the acknowledgement. Etc etc etc
History Lesson: The Byzantine Empire
Time: 330-1453 AD. Place: Balkans and Modern Turkey. Endless conspiracies, intrigue, and
untruthfullness were alleged to be common practice in the ruling circles of the day.
That is: it was typical for intentionally wrong and malicious activity to occur among the ruling group. A similar occurance can surface in a DS, and is known as ‘byzantine failure’.
Question: how do we deal with such malicious group members within a distributed system?
Byzantine Generals Problem
Now assume that the communication is perfect but the processes are not.
This problem also occurs in military settings and is called the Byzantine Generals Problem.
We still have the red army, but n blue generals. Communication is done pairwise by phone; it is
instantaneous and perfect. m of the generals are traitors (faulty) and are
actively trying to prevent the loyal generals from reaching agreement by feeding them incorrect and contradictory information.
Is agreement still possible?
Byzantine Generals Problem We will illustrate by example where there are
4 generals, where one is a traitor (analogous to a faulty process).
Step 1: Every general sends a (reliable) message to every
other general announcing his troop strength. Loyal generals tell the truth. Traitors tell every other general a different lie. Example: general 1 reports 1K troops, general 2
reports 2K troops, general 3 lies to everyone (giving x, y, z respectively) and general 4 reports 4K troops.
Byzantine Generals Problem
Byzantine Generals Problem
Step 2: The results of the announcements of step 1
are collected together in the form of vectors.
Byzantine Generals Problem
Byzantine Generals Problem
Step 3 Consists of every general passing his vector
from the previous step to every other general.
Each general gets three vectors from each other general.
General 3 hasn’t stopped lying. He invents 12 new values: a through l.
Byzantine Generals Problem
Byzantine Generals Problem
Step 4 Each general examines the ith element of
each of the newly received vectors. If any value has a majority, that value is put
into the result vector. If no value has a majority, the
corresponding element of the result vector is marked UNKNOWN.
Byzantine Generals Problem
The same as in previous example, except now with 2 loyal generals and one traitor.
Byzantine Generals Problem
With m faulty processes, agreement is possible only if 2m+1 processes function correctly
The total is 3m+1. If messages cannot be guaranteed to be
delivered within a known, finite time, no agreement is possible if even one process is faulty.
Why? Slow processes are indistinguishable from crashed ones.
Byzantine Generals Problem
Let f be the number of faults to be tolerated.
The algorithm needs f+1 rounds. In each round, a process sends to all the
other processes the values that it received in the previous round. The number of message sent is on the order of:O(Nf+1) where N is the number of generals.
If you do not assume Byzantine faults then you need a lot less infrastructure.