CS555: Distributed Systems [Fall 2019] Dept. Of Computer

SLIDES CREATED BY: SHRIDEEP PALLICKARA L2.1

CS555: Distributed Systems [Fall 2019]Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

Dept. Of Computer Science , Colorado State University

CS 555: DISTRIBUTED SYSTEMS

[TIME]

Shrideep PallickaraComputer Science

Colorado State University

August 29, 2019 L2.1 CS555: Distributed Systems [Fall 2019]


L2.2

Professor: SHRIDEEP PALLICKARA

Frequently asked questions from the previous class survey

August 29, 2019

¨ Quizzes/Exams?¤ Frequency, etc.



L2.3Professor: SHRIDEEP PALLICKARA

Topics covered in this lecture

August 29, 2019

¨ Physical Clocks¨ How time is actually measured

¨ Clock synchronization algorithms¤ Network Time Protocol



SYNCHRONIZATION

August 29, 2019 L2.4




Synchronization

August 29, 2019

¨ Important that processes do not access a shared resource simultaneously¤ Cooperate in granting each other temporary, exclusive access

¨ Multiple processes must sometimes need to agree on ordering of events¤ Was m1 from process P sent out before or after m2 from process Q ?



L2.6


What we will look at:

August 29, 2019

¨ Synchronization based on actual time

¨ Synchronization in which only relative ordering matters¤ Rather than ordering in absolute time






Let’s look at a make example

August 29, 2019

¨ Large programs split up into multiple source files

¨ make examines times at which all source and object files were modified

¨ input.c (@ 2151) and input.o (@ 2150)?¤ Input.c has changed since input.o was created

¨ output.c (@2144) and output.o (@2145)?¤ No recompilation required



L2.8


What could happen in a distributed system with make?

August 29, 2019

¨ output.o (@2145)

¨ output.c modified right after that ¤ But assigned (@2144) by the machine

¨ No recompilation required¤ Incorrect!¤ Executable is a mix of object files from new and old sources



PHYSICAL CLOCKS



L2.10


Nearly all computers have a circuitry for tracking time

August 29, 2019

¨ Timer is a more precise term than clock

¨ Quartz crystals are kept under tension¤ Oscillates at a well-defined frequency¤ Frequency depends on:

n Type and cut of crystal

n The amount of tension



L2.11


Each crystal has two registers associated with it

August 29, 2019

¨ {Counter, Holding Register}

¨ Each oscillation decrements the counter

¨ When counter reduces to zero¤ Interrupt is generated: clock tick

¤ Counter reloaded from the holding register

¨ We can program the timer to generate interrupts N times per second



L2.12


Measuring time

August 29, 2019

¨ Time is stored as number of ticks¤ After some known starting date and time

¨ Stored in special battery-backed CMOS RAM





L2.13


How the OS uses the timer to report actual time

August 29, 2019

¨ The OS reads the node’s hardware timer, Hi(t), scales it and adds an offset§ Ci(t) = αHi(t) + β¤ This is the software clock

¨ In general, the clock is not completely accurate§ Ci(t) will always differ from t



L2.14


On a single machine it is ok if the clock is off by a small amount

August 29, 2019

¨ All processes use the same clock

¨ Processes are internally consistent

¨ Only relative times matter in some cases



L2.15


Things change when we add multiple clocks

August 29, 2019

¨ Frequency at which the quartz oscillator runs is fairly stable

¨ Impossible to guarantee that all oscillations are at the same frequency



L2.16


This causes the (software) clocks to slowly get out of synch

August 29, 2019

¨ You get different values during read outs¤ Clock skew

¨ Time is not independent of the machine on which it was reported




Physical and software clocks

August 29, 2019

¨ Multiple physical clocks are desirable¤ Efficiency

¤ Redundancy

¨ How do we synchronize software clocks with:¤ Real-world clocks

¤ Each other



L2.18


Clock skew and drift

August 29, 2019

¨ The instantaneous difference between the readings of any two clocks is called their skew

¨ All clocks (including crystal-based ones) are subject to clock drift¤ This means that they count time at different rates





L2.19


Clock drift rates

August 29, 2019

¨ The drift rate is the change in the offset between the clock and a nominal perfect reference clock per unit of time¤ Offset: Difference in reading

¨ Some typical drift rates:¤ Regular Clocks based on quartz crystal

n 10-6 seconds/secondn i.e., 1 second every 1,000,000 seconds or 11.6 days

¤ High precision quartz clocks: 10-7 seconds/second

¤ Atomic clocks: 10-13 seconds/second



HOW TIME IS ACTUALLY MEASURED

August 29, 2019 L2.20



L2.21


Since the invention of mechanical clocks, time has been measured astronomically

August 29, 2019

¨ Sun’s highest apparent position in the sky¤ Transit of the sun

¤ At about noon each day

¨ Interval between consecutive transits of the sun¤ Solar day

¨ Solar second?¤ Solar day/86400



L2.22


But the earth’s rotation is not constant

August 29, 2019

¨ Proved in the 1940s

¨ Earth is slowing down¤ Tidal friction¤ Atmospheric drag

¨ Based on growth patterns on ancient corals¤ 300 million years ago, we had 400 days per year




The length of the year is not believed to have changed

August 29, 2019

¨ Days have simply gotten longer¤ Long-term trend

¨ Short term variations¤ Due to turbulence in the earth’s core

¨ Astronomers measure a large number of days ¤ Take their average

¤ And then divide by 86400



L2.24


Physicists have taken over the job of timekeeping from the astronomers

August 29, 2019

¨ Atomic clock invented in 1948

¨ Measures time much more accurately¤ Independent of the wiggling and wobbling of earth

¨ Count energy transitions of the cesium 133 atom





L2.25


Atomic seconds

August 29, 2019

¨ 1 second is the time for a cesium 133 atom to make 9,192,631,770transitions¤ Transitions are between two hyperfine levels of the ground state of cesium-

133 (Cs133)

¨ Why this number?¤ Atomic second = mean solar second in the year of its introduction



L2.26


Several labs around the world have cesium 133 atomic clocks

August 29, 2019

¨ Periodically, each lab tells the Bureau International de L’Heure (BIH) in Paris about their ticks

¨ BIH averages this to produce TAI¤ International Atomic Time

¨ TAI is mean number of ticks of cesium-133¤ Since Jan 1, 1958 divided by 9,192,631,770




TAI is highly stable, but …

August 29, 2019

¨ 86400 TAI seconds is now slower than the mean solar day¤ Days are getting longer

¨ Using TAI exclusively means that over the years noon would get earlier and earlier …



L2.28


People notice these kind of things ...

August 29, 2019

¨ In 1582, Pope Gregory XIII decreed 10 days be removed from the calendar

¨ Led to riots!¤ Landlords demanded full month’s rent¤ Bankers wanted full month’s interest

¤ But employers refused to pay for work that was not done

¨ Protestant countries did not accept the Gregorian calendar for 170 years



L2.29


BIH solves this problem by introducing leap seconds

August 29, 2019

¨ Whenever the discrepancy grows to 800 milliseconds¤ Total number of leap seconds introduced so far? 30

¨ Time based on TAI seconds; but in phase with apparent motion of the sun¤ This is the Universal Coordinated Time (UTC)

¨ UTC has replaced the Greenwich Mean Time



L2.30


UTC uses and availability

August 29, 2019

¨ Power companies synchronize timing of their 50/60Hz clocks to UTC

¨ NIST operates a shortwave radio station in Ft Collins¤ Call letters WWV (accuracy of WWV is ± 1 msec)

¤ Accuracy is ± 10 msec due to atmospheric fluctuations

¨ Radio receivers for WWV and other UTC sources





GLOBAL POSITING SYSTEM



L2.32


GPS

¨ Uses 30 satellites each orbiting at a height of approximately 20,000 km

¨ Each satellite has up to four atomic clocks¤ Regularly calibrated from special stations on Earth

August 29, 2019




Broadcasting

¨ Each satellite continuously broadcasts its position ¤ Timestamps each message with its local time

¨ Broadcasting allows every receiver on Earth to accurately compute its own position

August 29, 2019 CS555: Distributed Systems [Fall 2019]


L2.34


Calculating the receiver’s position [1/4]

¨ Let Δr denote the deviation of the receiver’s clock from actual time

¨ When a message is received from satellite i with timestamp Ti

¤ Measured delay Δi by the receiver consists of two components• Δi = (Tnow - Ti) + Δr

August 29, 2019





¨ Signals travel at the speed of light, c

¨ Measured distance of the satellite is cΔi¨ If the receiver’s position is (xr, yr, zr)

¤ The real distance is also

€

di = (xi − xr)2 + (yi − yr)

2 + (zi − zr)2



L2.36



¨ When we have 4 satellites¤ We have 4 equations in 4 unknowns

¤ Allows us to solve the coordinates (xr, yr, zr) and Δr

August 29, 2019







¨ GPS does not take leap seconds into account¤ Such an error can easily be compensated for in hardware

¨ Other sources of error:¤ Atomic clocks in satellites are not in perfect sync

¤ Receiver’s clock has finite accuracy

¤ Signal propagation speed is not constantn Signals slow down when entering e.g., the ionosphere



L2.38


Accuracy of GPS receivers

¨ Commodity GPS receivers can be precise within a range of 1-5 meters

¨ Professional receivers have a claimed error of less than 20-35 nanoseconds

August 29, 2019



CLOCK SYNCHRONIZATION ALGORITHMS



L2.40


Goals of Synchronization

¨ If 1 machine has a WWV receiver¤ Keep all machines synchronized to it

¨ If no machines have a WWV receiver¤ Keep all machines in synch as well as possible

August 29, 2019



L2.41


Modeling the setting [1/2]

¨ Timer causes an interrupt H times per second

¨ When timer goes off, interrupt handler adds 1 to software clock¤ Tracks ticks from some agreed-upon time

¤ Value of this clock, C

¨ UTC time, t



L2.42


Modeling the setting [2/2]

¨ When UTC time is t¨ Value of clock on machine p is Cp(t)

¨ In a perfect world¤ Cp(t) = t for all p

€

Cp' (t) =

dCdt

=1

Frequency of p’s clock at time t

€

Cp' (t)

August 29, 2019






Skews and Offsets

¨ Offset relative to a specific time t is Cp(t) – t

¨ Skew indicates extent to which frequency differs from that of a perfect clock

€

Cp' (t) −1



L2.44


Timer specifications

August 29, 2019

¨ Real timers do not interrupt exactly H times per second¨ With H = 60; a timer should generate 216000 ticks per hour

¨ Relative error in modern timer chips is about 10-5

¤ In our example of H=60; about ±2

¤ Between 215,998 to 216,002 ticks per hour

¤ Manufactures usually specify a drift rate



L2.45


Clocks ticking at different rates

Cloc

k Ti

me,

C

UTC, t

€

dCdt

=1

Perfe

ct Clock

Slow Clock

€

dCdt

<1Fast

Cloc

k

€

dCdt

>1

August 29, 2019CS555: Distributed Systems [Fall 2019]


NETWORK TIME PROTOCOL

August 29, 2019 L2.46




Network Time Protocol

¨ Clients connect to a time server

¨ Time server provides accurate time¤ Has WWV receiver¤ Accurate clock

¨ CHALLENGE

¤ Network latencies¤ Time is outdated by the time you get it



L2.48


Getting time from a time server

B

A

TIME SERVER

CLIENT

T2 T3

T1 T4

Offset at A = T3 – T4 + time the message was in transit

Offset at A = T3 – T4 + (T2 – T1) + (T4 – T3) 2

Delay = (T4 – T1) + (T3 – T2)

2

August 29, 2019





L2.49


Let’s look at an example

B

A

TIME SERVER

CLIENTT1=10 T4=250

100 T2=115 T3=120

Offset at A = T3 – T4 + (T2 – T1) + (T4 – T3) 2

= 120 – 25 + (115 – 10) + (25 – 120) 2

= 100



L2.50


In the NTP protocol, both the offsets and delays are used

August 29, 2019

¨ Eight pairs of {offset, delay} tuples are buffered

¨ Minimum value of delay is chosen¤ The corresponding offset is taken as the most reliable one




Time adjustment rules

¨ Time is not allowed to run backward¤ Even though the client’s clock is fast

¨ Changes are introduced gradually¤ If timer is set to generate 100 interrupts/second

¤ Each interrupt adds 10 milliseconds to the clockn To slow down we would add only 9 millisecondsn To speed up we would add 11 milliseconds



L2.52


It is possible to apply NTP symmetrically

August 29, 2019

¨ If time server is known to be more accurate¤ No point adjusting time with the client

¨ NTP divides servers into strata

¨ Server with a WWV receiver or atomic clock¤ Stratum-1 server (clock is stratum-0)




Communication and changing stratums

¨ A contacts B for offset adjustments¤ Only if A’s stratum is higher than B¤ E.g. A may be stratum 5, and B is stratum-1

¨ After offset adjustments¤ If B was a stratum-k, A becomes stratum-(k+1)¤ E.g. A is stratum-5, B is stratum-1

n Then A becomes stratum-2



L2.54


The contents of this slide-set are based on the following references

August 29, 2019

¨ Distributed Systems: Principles and Paradigms. Andrew S. Tanenbaum and Maarten Van der Steen. 2nd Edition. Prentice Hall. ISBN: 0132392275/978-0132392273.

¨ Wikipedia: Atomic Clocks http://en.wikipedia.org/wiki/Atomic_clock

http://en.wikipedia.org/wiki/Atomic_clock

Documents

CS555: Distributed Systems [Fall 2019] Dept. Of Computer