MARZULLO ‘s agreement algorithm& DTSS intersection algorithm in Golang

Romain Jacotinromain.jacotin@orange.fr

Agenda

• MARZULLO’ agreement algorithm

– Description

– Golang source code

• DTSS intersection algorithm (NTP)

– Description

– Golang source code

• References

MARZULLO's agreement algorithm

B

D

C

A

10 30 38 46 60 1229678

Offset

Marzullo's algorithm, invented by Keith Marzullo for his Ph.D. dissertation in 1984, is an agreement algorithm used to select sources for estimating accurate time from a number of noisy time sources.

The Marzullo's algorithm is also used to compute the relaxed intersection of n boxes (or more generally n subsets of Rn), as required by several robust set estimation methods.

Marzullo's algorithm is efficient in terms of time for producing an optimal value from a set of estimates with confidence intervals where the actual value may be outside the confidence interval for some sources. In this case the best estimate is taken to be the smallest interval consistent with the largest number of sources.

http://cseweb.ucsd.edu/~marzullo/

Keith MARZULLO’s algorithmMarzullo's algorithm begins by preparing a table of the sources, sorting it and then searching (efficiently) for the intersections of intervals.For each source there is a range [c−r,c+r] defined by c ± r.For each range the table will have two tuples of the form {offset,type} :• One tuple will represent the beginning of the range, marked with type = −1 as {c−r,−1}• and the other will represent the end with type = +1 as {c+r,+1}

The description of the algorithm uses the following variables: best (largest number of overlapping intervals found), cnt (current number of overlapping intervals), beststart and bestend (the beginning and end of best interval found so far), i (an index), and the table m of tuples.

1. Build m the table of tuples.2. Sort the table m by the offset. (If two tuples with the same offset but opposite types exist, indicating that one interval

ends just as another begins, then a method of deciding which comes first is necessary. Such an occurrence can be considered an overlap with no duration, which can be found by the algorithm by putting type −1 before type +1. If such pathological overlaps are considered objectionable they can be avoided by putting type +1 before −1 in this case.)

3. [initialize] best=0 cnt=04. [loop] go through each tuple in the table m in ascending order

• [current number of overlapping intervals] cnt=cnt−type[i]• if cnt>best then best=cnt beststart=offset[i] bestend=offset[i+1]

5. [end loop] return [beststart,bestend] as optimal interval. The number of false sources (ones which do not overlap the optimal interval returned) is the number of sources minus the value of best.

commentary: the next tuple, at [i+1], will either be an end of an interval (type=+1) in which case it ends this best interval, or it will be a beginning of an interval (type=−1) and in the next step will replace best.

ambiguity: unspecified is what to do if best=cnt. This is a condition of a tie for greatest overlap. The decision can either be made to take the smaller of bestend−beststart or offset[i+1]−offset[i] or just take an arbitrary one of the two equally good entries.

Confidence intervals

B

D

C

A

10 30 38 46 60 1229678

Offset

-1

-1

-1

-1+1

+1

+1

+1

Golang algorithm

type tuple struct {

segname string // optional : ID of the interval

offset int // the offset value of the start or end interval

tp int // is -1 for a start interval and +1 for an end interval

}

type tuples []tuple

// functions to give sort capability to tuples with the “sort” package

func (ts tuples) Len() int {

return len(ts)

}

func (ts tuples) Swap(i, j int) {

ts[i], ts[j] = ts[j], ts[i]

}

func (ts tuples) Less(i, j int) bool {

return ts[i].offset < ts[j].offset

}

m := tuples{

{"A", 10, -1},

{"A", 30, 1},

{"B", 38, -1},

{"B", 78, 1},

{"C", 46, -1},

{"C", 96, 1},

{"D", 60, -1},

{"D", 122, 1},

}

B

D

C

A

10 30 38 46 60 1229678

Offset

-1

-1

-1

-1+1

+1

+1

+1

Golang algorithm

// 0. Build the table of tuples

m := tuples{

{"A", 10, -1},

{"A", 30, 1},

{"B", 38, -1},

{"B", 78, 1},

{"C", 46, -1},

{"C", 96, 1},

{"D", 60, -1},

{"D", 122, 1},

}

// 1. Sort the table by the offset.

sort.Sort(m)

// 2. [initialize] best=0 cnt=0

best := 0

cnt := 0

Golang algorithm

// 3. [loop] go through each tuple in the table in ascending order

// 4. [current number of overlapping intervals] cnt=cnt−type[i]

// 5. if cnt>best then best=cnt beststart=offset[i] bestend=offset[i+1]

var beststart int

var bestend int

for i, t := range m {

cnt = cnt - t.tp

if cnt > best {

best = cnt

beststart = t.offset

bestend = m[i+1].offset

}

}

// 6. [end loop] return [beststart,bestend] as optimal interval.

// The number of false sources (ones which do not overlap the optimal interval

// returned) is the number of sources minus the value of best.

fmt.Println(”Best source quantity = ", best)

fmt.Println("False source quantity = ", len(m)/2 - best)

fmt.Println("Beststart = ", beststart)

fmt.Println("Bestend = ", bestend)

Golang algorithm

Agenda

• MARZULLO’ agreement algorithm

– Description

– Golang source code

• DTSS intersection algorithm (NTP)

– Description

– Golang source code

• References

DTSS intersection algorithm (NTP)

B

D

C

A

10 30 38 46 60 1229678

Offset

The intersection algorithm is an agreement algorithm used to select sources for estimating accurate time from a number of noisy time sources, it forms part of the modern Network Time Protocol (NTP).It is a modified form of Marzullo's algorithm.

While Marzullo's algorithm will return the smallest interval consistent with the largest number of sources, the returned interval does not necessarily include the center point (calculated offset) of all the sources in the intersection.

The Intersection algorithm returns an interval that includes that returned by Marzullo's algorithm but may be larger since it will include the center points. This larger interval allows using additional statistical data to select a point within the interval, reducing the jitter in repeated execution.

2058

70 91

NTP

DTSS

DTSS intersection algorithm (NTP)

Given M intervals of the form c ± r (which means [c−r,c+r]), the algorithm seeks to find an interval with M−fsources. The value f is referred to as the number of falsetickers, those sources which are in error (the actual value is outside the confidence band). The best estimate is that which assumes the least number of falsetickers, f. The results will be considered valid if f < M/2, otherwise the algorithm will return failure instead of an interval.

The intersection algorithm begins by creating a table of tuples {offset,type}. For each interval there are three entries: the lower endpoint, the midpoint and the upper endpoint, labelled with types −1, 0 and +1 respectively. Thus the interval c ± r results in the entries {c−r,−1}, {c,0} and {c+r,+1}. These entries are then sorted by offset.

Variables: This algorithm uses f as number of false tickers, endcount and midcount are integers. lower and upperare values of offsets, i and j (an index), and the table m of tuples.

1. The algorithm begins by initializing a value f and counters endcount and midcount to zero.2. Then, starting from the lowest offset of the sorted table of tuples, for each tuple {offset, type} the value

of type is subtracted from the counter endcount, which is the number of intersections. If type is zero, increment the value of midcount, which is the number of falsetickers. If endcount >= M-f for some entry, offset of that entry becomes the lower offset of the intersection; otherwise, f is increased by one and the above procedure is repeated.

3. Without resetting f or midcount, a similar procedure is used to find the upper offset, except that the value of type is added to the midcounter counter.

4. If after both offsets have been determined midcount <= f, the procedure continues having found M-ftruechimers; otherwise, f is increased by one and the entire procedure is repeated.

Confidence intervals

B

D

C

A

10 20 30 38 4660

58 70 91 1229678

Offset

-1

-1

-1

-1+1

+1

+1

+1

0

0

0

0

Golang algorithm

type tuple struct {

segname string // optional : ID of the interval

offset int // the offset value of the start or end interval

tp int // lower, midpoint, upper endpoint are types −1, 0, +1

}

type tuples []tuple

// functions to give sort capability to tuples with the “sort” package

func (ts tuples) Len() int {

return len(ts)

}

func (ts tuples) Swap(i, j int) {

ts[i], ts[j] = ts[j], ts[i]

}

func (ts tuples) Less(i, j int) bool {

return ts[i].offset < ts[j].offset

}

B

D

C

A

10 20 30 38 4660

58 70 91 1229678

m := tuples{

{"A", 10, -1},

{"A", 20, 0},

{"A", 30, 1},

{"B", 38, -1},

{"B", 58, 0},

{"B", 78, 1},

{"C", 46, -1},

{"C", 70, 0},

{"C", 96, 1},

{"D", 60, -1},

{"D", 91, 0},

{"D", 122, 1},

}

Golang algorithm

Offset

-1

-1

-1

-1+1

+1

+1

+1

0

0

0

0

// Build the table of tuples and Sort the table by the offset

var endcount, midcount, lower, upper, f, M int

m := tuples{

{"A", 10, -1},

{"A", 20, 0},

{"A", 30, 1},

{"B", 38, -1},

{"B", 58, 0},

{"B", 78, 1},

{"C", 46, -1},

{"C", 70, 0},

{"C", 96, 1},

{"D", 60, -1},

{"D", 91, 0},

{"D", 122, 1},

}

sort.Sort(m)

M = len(m)/3

Golang algorithm

// calling all truechimers

for f = 0; f < M/2; f++ {

endcount = 0

midcount = 0

// find low endpoint

for _, t := range m {

endcount -= t.tp

lower = t.offset

if endcount >= (M - f) { break }

if t.tp == 0 { midcount++ }

}

endcount = 0

// find high endpoint

for j := len(m)-1; j >= 0; j-- {

endcount += m[j].tp

upper = m[j].offset

if endcount >= (M-f) { break }

if m[j].tp == 0 { midcount++ }

}

// continue until all falsetickers found

if midcount <= f { break }

}

Golang algorithm

/ do we found an intersection ?

if lower <= upper {

fmt.Println("Best source quantity = ", M-f)

fmt.Println("False source quantity = ", f)

fmt.Println("Beststart = ", lower)

fmt.Println("Bestend = ", upper)

} else {

fmt.Println("Failed because false source quantity = ", f)

}

Golang algorithm

Agenda

• MARZULLO’ agreement algorithm

– Description

– Golang source code

• DTSS intersection algorithm (NTP)

– Description

– Golang source code

• References

References

Keith MARZULLOhttp://cseweb.ucsd.edu/~marzullo/

Maintaining the time in a distributed sytem (MARZULLO/OWICKI) http://infolab.stanford.edu/pub/cstr/reports/csl/tr/83/247/CSL-TR-83-247.pdf

Marzullo’s algorithmhttp://en.wikipedia.org/wiki/Marzullo's_algorithm

DTSS intersection algorithm (Digital Time Synchronization Service)http://en.wikipedia.org/wiki/Intersection_algorithmhttp://www.eecis.udel.edu/~mills/ntp/html/select.html

Network Time Protocol (NTP)http://tools.ietf.org/pdf/rfc5905.pdfhttp://www.ntp.org/

The Go Programming Languagehttp://www.golang.org/