1/41

Clock Synchronization

Pedro F. Souto (pfs@fe.up.pt)

April 28, 2021

2/41

Roadmap

Synchronization Models

Clock Synchronization

Centralized Clock Synchronization

NTP

Applications

Further Reading

3/41

Roadmap

Synchronization Models

Clock Synchronization

Centralized Clock Synchronization

NTP

Applications

Further Reading

4/41

Distributed System ModelI A set of sequential processes that execute the steps of a

distributed algorithmI DS are inherently concurrent, with real parallelism

I Processes communicate andsynchronize by exchangingmessagesI The communication is not

instantaneous, but suffers delays

I Processes may have access to alocal clockI But local clocks may drift wrt real

time

p1 p2

m

mn

H(t)

t

I DS may have partial failure modesI Some components may fail while others may continue to operate

correctly

5/41

Fundamental Models

Synchronism characterizes the system according to the temporalbehavior of its components:I processesI local clocksI communication channels

Failure characterizes the system according to the types of failures itscomponents may exhibit

6/41

Models of Synchronism

Synchronous iff:1. there are known bounds on the time a process takes to

execute a step2. there are known bounds on the time drift of the local clocks3. there are known bounds on message delays

Asynchronous No assumptions are made regarding the temporalbehavior of a distributed system

I These 2 models are the extremes of a range of models ofsynchronism

DilemmaI It is relatively simple to solve problems in the synchronous model,

but these systems are very hard to build

7/41

Roadmap

Synchronization Models

Clock Synchronization

Centralized Clock Synchronization

NTP

Applications

Further Reading

8/41

Clock Synchronization

ObservationI In our every-day lives we use time to coordinate our activities

I Distributed applications can do the same, as long as eachprocess has access to a "sufficiently" synchronized clock

Synchronized ClocksI Are essential for implementing:

some basic services:Synchronous distributed algorithms i.e. roundsCommunication protocols e.g. TDMA

some distributed applications:Data collection and event correlationControl systemsGPS

I Enable to reduce communication and therefore improveperformance (Liskov 93)

9/41

Applications (according to David Mills )

I Distributed database transaction journaling and loggingI Stock market buy and sell ordersI Secure document timestampsI Aviation traffic control and position reportingI Radio and TV programming launchI Real-time teleconferencingI Network monitoring, measurement and controlI Distributed network gaming and training

I Take it with a grain of saltsrc: David Mills, Network Time Protocol (NTP) General Overview

10/41

Local Clocks

I Computers have hardware clocks basedon quartz crystal oscillators, which providea local measure of time, H(t). Often:I These clocks generate periodic interruptsI The OS uses these interrupts to keep the

local time

H(t)

t

I These clocks have a drift(

dH(t)dt − 1

)wrt a perfect clock

I Quartz-based oscillators have drifts in the order of 10−6, i.e. theymay run faster/slower a few µs per second

I The clock drift of a quartz-based oscilator depends on theenvironmental conditions, especially temperature

I Even if this drift is bounded, unless clocks synchronize, theirvalues may diverge forever, i.e. their offset/skew (Hi(t)− Hj(t)),may grow unbounded

11/41

What is Clock Synchronization?

External Clock Synchronization (CS)I.e. synchronization wrt an external time reference (R)

|Ci(t)− R(t)| < α, ∀i

where α is known as the CS accuracy

Internal Clock SynchronizationI.e. synchronization among the local clocks in the system – needs notbe related to real time

|Ci(t)− Cj(t)| < π,∀i , j

where π is known as the CS precisionNote that external clock synchronization implies internal clocksynchronization.I What’s the precision?

12/41

Time Scales/StandardsInternational Atomic Time (TAI) This is a weighted average of the

time kept by around 300 atomic clocks in over 50 national labsI More stable clocks are given larger weightsI It uses GPS satellites for clock comparisons that provide the

data for the calculation of TAICoordinated Universal Time (UTC) It is derived from the TAI, by

adding leap seconds to compensate for variations in Earth’srotation speed

0

0

1

1

2

2

3

3

4

4

5

5

6

6

7

7

8

8

9

9

10 11

11

12

12

13

13

14

14

15

15

16

16

17

17

18

18

19

19

20 21

21

22

22

23

23

24

24

25

25

TAI

Solarseconds

Leap seconds introduced into UTC toget it in synch with TAI

GPS provides time with an accuracy better than 100 nanosecondsI Each satellite has its own atomic clocks (3 or 4), which are kept

synchronized with the Master Clock at US Naval ObservatoryI The MC@USNO is a set of more than 40 atomic clocks that

produces UTC(USNO), which is used for computing the UTC

13/41

Roadmap

Synchronization Models

Clock Synchronization

Centralized Clock Synchronization

NTP

Applications

Further Reading

14/41

Clock Synchronization: Centralized Algorithm

Assumptions

Each process has a local clockMaster/server clock provides the time referenceSlave/client clock synchronize with the master/server clock

Local clock drifts are bounded

(1 + ρ)−1 ≤ dCi(t)/dt ≤ 1 + ρ, for all correct processes i

System is synchronousI There are known bounds (both lower and upper) for

communication delaysI The time a process requires to take a step is negligible with

respect to the communication delays (and thereforebounded)

15/41

Clock Synchronization: Specification

Accuracy

|Ci(t)− Cm(t)| ≤ α, for all correct processes iwhere Cm(t) is the master’s clock

16/41

Centralized Clock Synchronization: Idea

MasterI Periodically

I read the local timeI broadcast it in a SYNC message

SlaveI Upon reception of a SYNC message,

I update the local clock

17/41

Master Time EstimationIssue There is a delay between the reading of the time @ the master,

and its processing @ the slaves. By the time the slave gets themessage, the time @ the master will be:

Cs(t) = Cm(t) + delayCm(t) Cs(t)

dey

Cm(t1) = T1 SYNC(T1)

Cm(t) Cs(t)

m

mn

Problem What is the value of delay?Response Do not know, but can be

estimated:

min < delay < max

max is known because the system issynchronous

I To minimize the error, should use [Cm(t) + min,Cm(t) + max ]midpoint:

Cs(t) = Cm(t) +min + max

2

18/41

SYNC Message Period

To ensure accuracy, it must be;

|Cm(t)− Cs(t)| < α

Because the clock drifts are bounded:∣∣∣∣dCs(t)dt

− dCm(t)dt

∣∣∣∣ < 2ρ

Hence,max −min

2+ 2ρT < α

Observation The clock skew lower bound is determined by the delayjitter ((max −min))I If T is large, the second term may dominate

I NTP sometimes uses a T of 15 days, and ρ ∼ 10−6

19/41

Message Delay Estimation in Practical Systems (1/2)

Issue Often, there is no known upper bound for the communicationsdelay

Approach (Christian) Estimate it based on the round-trip-delay, round

Let min be the lower delay boundThen, when the slave receives the master’sclock reading, time at the master will be inthe interval:

[t + min, t + round −min]

By choosing the midpoint in this interval, i.e.by assuming that the delay is round

2 , webound the error to:

round2−min

rond

mn

t

t

mn

20/41

Message Delay Estimation In Practical Systems (2/2)

Insight By measuring the time the server takes to reply, we canreduce the errorI This can be implemented by timestamping with the local

clocks the sending/receiving of messages

T1

T2

T4

T3

s

m

round

(T2, T3)

Let Cs(t) = Cm(t) + O(t)Then,T2 = T1 −O(t1) + d1

T4 = T3 + O(t3) + d2

Note: Upper-case T’s are clock times, lower case are real-timesAssume

Synchronized rates O = O(t1) = O(t3)Symmetry del = (d1 + d2)/2 = ((T4 − T 1)− (T 3− T2))/2

21/41

Clock Offset Estimation Using Timestamps

T1

T2

T4

T3

s

m

round

(T2, T3)

Let Cs(t) = Cm(t) + O(t)Then,T2 = T1 −O(t1) + d1T4 = T3 + O(t3) + d2

Note: Upper-case T’s are clock times, lower case are real-timesAssume O = O(t1) = O(t3)Then O = O + (d1 − d2)/2Where O = ((T4 − T3) + (T1 − T2))/2Remember: del = (d1 + d2)/2 = ((T4 − T 1)− (T3− T 2))/2Because d1,d2 ≥ 0, then d1 − d2 ≤ d1 + d2Therefore: O − del ≤ O ≤ O + delO can be seen as an estimate of the offsetdel can be seen as an estimate of O’s error bound

22/41

Clock Offset and Delay Time Estimation Precision

Precision depends on:Rate synchronization i.e. O(t1) = O(t3)Symmetry assumption i.e. d1 = d2, only in the case of delay

estimationTimestamp accuracy

I The lower in the protocol stack they are generated the betterI In some cases, the timestamps are sent in later messages,

PTP calls them FOLLOW-UP messages.

23/41

Precision Time Protocol (PTP - IEEE 1588)

Description Protocol for clock synchronization with high precision onpacket-based networks

GoalsI Precision of at least one microsecondI Minimum hardware resource requirementsI Minimum administration on single subnet systemsI Applicable, but not limited, to Ethernet

Application domainsI Test and measurementI Industrial automationI Power industryI TelecomsI Aerospace, navigation and positioningI Aujdio and video networks

24/41

Precision Time Protocol: SyntonizationGoal To ensure that master and slaves have the same clock rate

I Two alternative ways to send the T ki

timestampSync message: requires the ability

to insert the timestamp into themessage on the flyI This is an issue if the

timestamps are generated in thelower layers (see below)

Follow_up message: requires anadditional message

I Slave clock rate is adjusted so that:T k+1

1 − T k1 = T k+1

2 − T k2

T k+12 − T k+1

1 = T k2 − T k

1

I Sync messages are sent periodically with a period of a fewsecondsI To correct variations due to changing environmental conditions,

e.g. temperature

25/41

Precision Time Protocol: Offset and Delay Estimation

Sync message (possibly with Follow_up)is used to obtain timestamps T1 and T2

Delay_Req and Delay_Resp messagesare used to obtain timestamps T3 andT4

I Slave clock rate is adjusted so that:T k+1

1 − T k1 = T k+1

2 − T k2

T k+12 − T k+1

1 = T k2 − T k

1

I Sync messages are sent periodically with a period of a fewsecondsI To correct variations due to changing environmental conditions,

e.g. temperature

26/41

Precision Time Protocol

I By including the appropriate timestamps in the SYNC messagesO and del may be updated

T1

T2

T4

T3

s

m

round

(T2, T3)

T5

T6

T7

T8

SY NC(T2, T5) SY NC(T2, T7)

T

I O can be used later to estimate the value of the master clock atthe time of reception of a SYNC message with the master’s time:

Time(SYNC)− O

27/41

Local Clock CorrectionInstantaneous correct offset only at every synchronization point

I May lead to non-monotonic clocksAmortized over the synchronization period, i.e. adjust both a and b

Cs(t) = at + bI So as to ensure that:

I Local time is continuousI Minimize the error at the end of the synchronization period,

possibly subject to the constraint that the change in the ratemay be bounded

C(t)

t

C(t)

tIMP. Each clock between resynchronization points can be

seen as a different clock. Therefore, clock correction canbe seen as jumping from one clock to the next.

28/41

Roadmap

Synchronization Models

Clock Synchronization

Centralized Clock Synchronization

NTP

Applications

Further Reading

29/41

Network Time Protocol (NTP)

I Clock synchronization protocol designed for the InternetI tolerates communication delay jitterI it is robust against loss of connectivityI scales to a large number of clientsI it is robust against interferences, whether malicious or accidental

Mills, Internet time synchronization: the Network Time Pro tocol, IEEETrans. on Communications, October 1991, 1482-93.Mills, Improved Algorithms for Synchronizing Computer NetworkClocks, IEEE Trans. on Networks, June 1995, pp. 245-54.

30/41

NTP: ArchitectureI NTP servers are organized hierarchically in a synchronization

subnet1

22

3 33

1

22

3 33

I Servers at the top (primary) are connected to a UTC sourceI Secondary servers synchronize with the primary and so onI The leaves of the trees are clientsI Each level of the hierarchy is known as a stratum, the lowest

stratum comprises the primary serversI The synchronization subnet may reconfigure in the sequence of

failures. E.g.:I If a UTC source becomes unreachable, a primary server may

become secondaryI If a primary server becomes unreachable, a secondary server may

switch to another primary server

31/41

NTP: Synchronization Modes

2

3

multicast

1

Cristian´s Alg.

Multicast a server multicasts periodically its time to clientsI The precision is relatively low, but is OK for LANs (why?)

Request-reply similar to Cristian’s algorithm.Symmetrical in which servers swap their times. Used by servers at

the lowest strata. (Ensures the highest precision.)I I.e. at this level, NTP uses a decentralized algorithm

I Besides that, NTP uses lots of statistics to compensate for thethe jitter in communication delayI It is this variability that limits the precisionI The OS itself also introduces some jitter

I NTP uses UDP always

32/41

Roadmap

Synchronization Models

Clock Synchronization

Centralized Clock Synchronization

NTP

Applications

Further Reading

33/41

Synchronized Clocks: Practical Usage

I On the Internet, it is possible to have synchronized clocks with:I A small enough skew (δ)I A high enough probability

I We can take advantage of this to reduce:I communicationI state (stored in stable storage)

and therefore improve performanceI What if clocks get ouf of sync? Depends:

Compensate at a higher levelDo nothing there are different reasons:

1. Correctness is not affected2. Domination of other failures3. Engineering trade-off: performance benefits outweight failure

costs

34/41

Synchronized Clocks: At-most-once Messaging (1/3)

Simplistic solutionI Remember all the messages receivedI Discard duplicated messages

Issue: Cannot remember ALL messages receivedSolution Forget far away past

Session-basedI Node has to execute some handshake before sending the

first message after a whileI In response, the receiver generates a conn(ection) id;I The sender must tag subsequent messages with the conn idI The handshake may be too high overhead

Synchronized clocksI All nodes have a synchronized clocks with accuracy α

35/41

Synchronized Clocks: At-most-once Messaging (2/3)Each message has:

A connection idI selected by the sender - no need for handshakeI must be unique (also among all senders)

A timestampReceivers keep:

A connection table (CT) with, for each connection:I The timestamp, ts, of the last message receivedI If not replaced by that of more recent message, it is kept at least

for the life-time, λ, of a message in the network, i.e. as long asts > time − λ− α (Why subtracting α?)

An upper bound, upper , on the value of all timestamps removedfrom the connection table.

Upon reception of a message it is discarded if its timestamp is:I either smaller than that of the entry of the connection table with

the corresponding connection idI or smaller than upper (if there is no entry in the CT)

36/41

Synchronized Clocks: At-most-once Messaging (3/3)

Issue: What if the receiver recovers after a crash?I Must ensure that messages delivered before the crash are not

delivered againSolution

Naïve store in stable storage the largest timestamp of allmessages received, and discard any message with a smaller ...

Efficient Periodicallly save in stable storage an upper-bound,latest = time + β, of the time-stamp of all delivered timestamps

1. Upon reception of a message with a timestamp larger thanlatest either delay its delivery or discard it

2. Upon recovery from a crash:I Set upper to latest , thus discarding messages whose

timestamps are older than the saved upper-bound.

Trade-offI Synchronized clocks improve performanceI At the cost of occasionally rejecting a message

37/41

Synchronized Clocks: Leases

Originally were just like a timed locksI Locks are difficult to manage in an environment where lock

owners may crashI It is even possible to have read/write leases

Generally They can be used to ensure some property during a limitedtime, i.e. the lease duration

Lease duration may beAbsolute requires synchronized clocksRelative requires synchronized rates

Renewals A lease is valid for its durationI Unless its owner renews the lease

Question How long should the lease duration be?

38/41

Synchronized Clocks: Preventing Replay-Attacks

Kerberos uses Needham and Schroeder’s shared key authenticationprotocolI But instead of using their solution to prevent replay attacks,

they use synchronized clocksI And, more generally, to prevent the use of keys for too long, i.e.

for key expirationI Check the details in Liskov’s paper

39/41

Synchronized Clocks vs. Synchronized (Clock) RatesI In most applications, synchronized rates are enough

I In computer networking, retransmission mechanisms usuallyassume clock rates synchronized with real time rates

I TCP also assumes that to prevent segments sent in the scope of aconnection to be delivered in the scope of another connection(when a connection is shutdown abruptly)

I This is a reasonable assumption for computer systems withquartz clocksI Nowadays, these clocks have a drift rate lower than 10−6

I Whenever possible synchronized rates should be used ratherthan synchronized clocksI Synchronized clocks depend on synchronized rates, and on

occasional communicationI But the use of synchronized rates also requires communication, so

that the instant of the relevant event can be boundedI Synchronized clocks are more powerful than synchronized rates

I They are able to support other algorithms for which synchronizedrates are not enough

I They can provide warnings when the clocks get out of synch

40/41

Roadmap

Synchronization Models

Clock Synchronization

Centralized Clock Synchronization

NTP

Applications

Further Reading

41/41

Further Reading

I Tanenbaum and van Steen, Section 6.1 of Distributed Systems,2nd Ed.

I NTP Project PageI Barbara Liskov, Practical Uses of Synchronized Clocks, in

Distributed Computing 6(4): 211-219 (1993)I Time at the Bureau International des Poids e Mesure, keeper of

the TAII Time and Frequency Division of the NIST Physical Measurement

LabI Precise time @ US Naval Observatory, owner of the Master

Clock used in GPS