NPLTime® disseminated through the UBS infrastructure

D Hicks (NPL), P Whibberley (NPL), E Laier English (NPL), L Lobo (NPL)

T Lee (UBS), A Austin (UBS)

A trial of the NPLTime® commercial service was conducted at the UBS facility in LD4 Slough, where the signal was consumed and assessed within the UBS infrastructure to validate the source, internal distribution architecture and systems, and timestamping solutions. This white paper outlines the implementation and results of the trial from the NPLTime® Distribution hub located at Telehouse, Docklands, and delivered to Slough on TMX Atrium connectivity.

The trial was carried out over a leap second implementation and included a period of fibre outage, demonstrating the resiliency of the service and engineered redundancy solutions. The time offset of the signal from UTC (Coordinated Universal Time) was measured by transportable disciplined caesium clocks from the National Physical Laboratory (NPL).

The NPLTime® commercial service provides a time signal to the end user over optical fibre that is independent of GPS and certified by NPL as being traceable to UTC(NPL) at the end user. A trial of the service was conducted at the UBS facility in the LD4 datacentre in Slough, where the signal was consumed and assessed within the UBS infrastructure to validate the source, internal distribution architecture and systems, and timestamping solutions. The solution was proven to be capable of achieving approximately 100 nanosecond synchronisation to UTC(NPL).

Introduction Precise and accurate timing plays a critical role in financial markets, underpinning the time stamping of trades, synchronisation of systems and the measurement of network latency for process optimisation. The rapid expansion of computer-based trading has increased the need for synchronisation of trading systems and traceability to UTC (Coordinated Universal Time), in order to help prevent trading irregularities and to aid forensic investigations.

The global financial sector has widely adopted and driven the development of electronic trading platforms in order to maximise efficiencies and market liquidity whilst striving to minimise volatility. As the frequency of trading increases beyond the several million transactions per second, the need to understand the absolute time of every

transaction is becoming critical from the point of view of clarity and a consolidated audit trail.

As the United Kingdom’s National Measurement Institute, the National Physical Laboratory (NPL) in Teddington, south-west London, is the only precision timing centre in the UK and fulfils a vital role by maintaining the national time scale, known as UTC(NPL).

UBS Investment Bank provides corporate, institutional and wealth management clients with expert advice, innovative solutions, execution and comprehensive access to the world’s capital markets. It offers advisory services and access to international capital markets, and provides comprehensive cross-asset research, along with access to equities, foreign exchange, precious metals and selected rates and credit markets, through our business units, Corporate Client Solutions and Investor Client Services. The Investment Bank

is an active participant in capital markets flow activities, including sales, trading and market-making across a range of securities.

Timestamping A typical enterprise time dissemination solution consists of a time source, an internal distribution and a timestamping engine.

The time source consists of the distribution chain from the UTC institute all the way to the ingress point on the internal distribution. The internal distribution consists of the networked delivery of the time using time transfer protocols to the point of timestamping. Finally, the timestamping engine consists of the last leg via the network interface card (NIC) to the point of timestamping.

It is not enough to have a time source feeding systems; it is critical to understand and manage how time is delivered, distributed and consumed. At each step in the process, knowledge of the traceability of the time signal to UTC is essential to ensure regulatory compliance at the timestamp.

NPLTime® source and distribution architecture

RedundantHydrogen MASERs

OK

ESC

Time Service

Ref. Time

Network

Alarm

Terminal

USB

F1

F2LANTIME M600

NTP: Offs. PPS: 5us Stratum: 1GPS: NORMAL OPERATION Satellites: 8 /8PTP2: ok INITIALIZING GM: 00...........Press F1 for help or F2 for setup overview

07:23:25UTC

Thu, 30.12.2010OK

ESC

Time Service

Ref. Time

Network

Alarm

Terminal

USB

F1

F2LANTIME M600

NTP: Offs. PPS: 5us Stratum: 1GPS: NORMAL OPERATION Satellites: 8 /8PTP2: ok INITIALIZING GM: 00...........Press F1 for help or F2 for setup overview

07:23:25UTC

Thu, 30.12.2010

of redundantPTPv2 Clocks

PTP over Dedicated Dark Fibre Link

Primary Secondary

Primary GrandMaster Backup GrandMaster

1 PPS Signal

Fibre Interface Switch

1 PPS monitoring

Figure 1: Resilient UTC reference architecture at NPL. The 1 PPS difference signal from Primary and Backup Grandmasters is monitored and the secondary Hydrogen maser is steered to within 100 ns of the Primary Hydrogen maser.

A resilient reference is provided from two independent UTC contributing hydrogen maser atomic clocks located at NPL Teddington in geographically separated and separately powered facilities, via resilient Grandmaster clocks and disseminated over a dedicated dark fibre link to distribution sites using PTPv2 (Precision Time Protocol version 2, also known as IEEE1588-2008).

OK

ESC

Time Service

Ref. Time

Network

Alarm

Terminal

USB

F1

F2LANTIME M600

NTP: Of fs. PPS: 5us Stratum: 1GPS: NORMAL OPERATION Satellit es: 8/8PTP2: ok INITIALIZING GM: 00...........Press F1 f or help or F2 for set up overview

07:23:25UTC

Thu, 30.12.2010OK

ESC

Time Service

Ref. Time

Network

Alarm

Terminal

USB

F1

F2LANTIME M600

NTP: Of fs. PPS: 5us Stratum: 1GPS: NORMAL OPERATION Satellit es: 8/8PTP2: ok INITIALIZING GM: 00...........Press F1 f or help or F2 for set up overview

07:23:25UTC

Thu, 30.12.2010

Figure 2: Distribution architecture at Telehouse datacentre. The caesium clock and disciplining unit provide a holdover solution.

The distribution sites are equipped with dual resilient PTP distribution clocks referenced to UTC(NPL) by PTP. In the event of loss of the incoming fibre delivery, the service is maintained by a caesium atomic clock, which is disciplined by the UTC(NPL) PTP signal when that signal is available. The caesium clock is able to hold the NPLTime® delivery within the specified service level agreement (SLA) for many days.

Trial configuration

Figure 3: Trial architecture. Shows the NPLTime® PTP traffic from the distribution switch at Telehouse to the UBS server at LD4, Slough. The Meinberg M1000 clock at LD4 replaces a fibre-copper media converter (used in initial trial configuration). The service was provided to UBS using a dedicated WDM wavelength from TMX Atrium equipment (a PTP-enabled switch connected to the NPLTime® distribution hub) located at Telehouse and a distribution switch located in LD4 Slough. The service was then distributed to a PTP clock and via a fibre cross-connect to the UBS data racks, also located in LD4 Slough, and delivered to a UBS-owned server equipped with a PTP-capable network interface card.

Once the trial infrastructure had been installed and was operational, NPL staff carried out a series of calibrations local to the NPLTime® redistribution clocks at commencement, during and at completion of the trial period over a period of three months; graphs of measured PTP offset at the NPLTime® redistribution clock logged on the NPLTime® management platform are displayed here (Figure 5 and Table 1).

Initial results The initial configuration at LD4 Slough consisted of an Arista DC7150S-24CL PTP-enabled datacentre switch (TMX-owned) configured for transparent-mode PTP connecting via copper interfaces to a Meinberg M600 network clock (NPL-owned) slaved to the NPLTime® distribution clocks located in Telehouse North.

Initial measurements of clock offset against UTC(NPL) at the NPL-owned PTP clock proved satisfactory and well within the service’s advertised SLA with mean offset better than 120 ns referenced to UTC(NPL) and overall peak offset deviation less than 50 ns, approximately an order of magnitude better than the service SLA. In addition to the on-site calibration measurements, continuous measurements of the clock PTP offset were polled by the NPLTime® management platform for the duration of the trial for comparison with both calibration measurements and those taken within the UBS estate.

Measurements at the UBS server, however, were substantially outside expected values. NPL and UBS teams jointly established that the combination of copper-fibre media convertors (to allow the use of a fibre cross-connect within LD4, the Meinberg M600 being equipped with copper 100 Mbit interfaces only) and speed conversions within UBS’ network switches was adding variable delays (jitter) of several microseconds, with a seriously deleterious effect on the UBS PTP network interfaces’ clock stability. The graph showing the stability of the PTP steering correction (Figure 4) during the initial phase of the trial illustrates the degraded service as measured at the UBS server.

In order to resolve the issue of variable delay, the fibre-copper media and speed converter was removed and replaced with a Meinberg M1000 unit equipped with a 1 Gbit optical interface. This was connected direct via the existing optical cross-connect to the UBS estate. An immediate improvement resulted with accuracy and stability at the UBS server within the NPLTime® SLA of 1 µs relative to UTC, with 99.9% availability.

Figure 4: Initial results: The black trace denotes the stability of the NPLTime® local reference clock vs distributed PTP reference as measured in the NPL Meinberg M600 and displayed in Figure 5; the red trace illustrates the difference between the NPLTime® reference presented through media and speed conversions and the UBS Solarflare PTP interface’s internal clock: this media and speed conversion resulted in variable delays up to 5 µs.

Results after configuration changes In addition to a stable reference (the NPLTime® PTP clock), the distribution architecture plays a vital part in the establishment of a serviceable time dissemination within a corporate network;

the effects of network path asymmetry, speed conversions and intervening hardware must be thoroughly considered when employing PTP timing within the estate for a satisfactory result.

LD4 measured offsets The service endpoint time offset from UTC(NPL) was measured using a transportable caesium clock at the LD4 grandmaster on five

occasions during the trial period, with a standard uncertainty of +/-5 ns. The results are given in the following table:

Date  MJD  Offset from UTC(NPL) 20 May 2015 57162.7554 +105.4 ns 15 June 2015 57188.7910 +110.2 ns 22 July 2015 57225.8513 ‐7.8 ns 26 August 2015 57260.8105 +9.5 ns 29 September 2015 57294.8331 ‐7.9ns

Table 1: Measured offsets of NPLTime® from UTC(NPL) at LD4 Grandmaster, Slough. Each measurement was obtained using a caesium clock calibrated at NPL, then transported under power to Slough for comparison with UBS endpoint Meinberg, and transported back to NPL for a final measurement.

Figure 5: Full duration of the trial (16 June – 22 Sept): UBS endpoint stability vs NPLTime® PTP distribution. A: Deviation due to planned switch-over of PTP Grandmasters prior to the leap second (see Figure 6). B, C: Misread SNMP data due to management issue - verified as misread by comparison with endpoint internal logs. D: Deviation due to reacquisition of reference Grandmaster on repair of NPL – Telehouse fibre (see Figure 7)

Leap second adjustment

During the trial a leap second was added at 23:59:60 on 30 June. The leap second addition was propagated via a PTPv2 Announce message through the NPLTime® infrastructure and no deviation was observed in practice.

Prior to the leap-second event, the leap second announcement and PTPv2 Announce process were tested in the NPLTime® test facility and no deviation of time synchronisation was observed when the PPS (pulse-per-second) outputs of a cascade of PTP clocks were measured against UTC(NPL); all clocks synchronously adjusted the UTC time by correctly inserting the leap second.

Figure 6: Stability of Grandmaster at LD4 around 30 June 2015 leap second showing deviation and settling due to planned switch-over of NPL Grandmaster clocks at Teddington. A: Deviation due to planned switch-over; B: Leap second adjustment, no deviation observed.

Resiliency to fibre breakage

Due to failure of the NPL Teddington – Telehouse dark fibre at approximately 08:09 UTC on 11 September the service was maintained for approximately 12 hours without the UTC(NPL) PTP reference using an NPL-owned caesium clock installed at Telehouse

North to provide holdover during such an event. Reacquisition of the PTP reference on fibre repair resulted in an overshoot with a rapid convergence to the UTC(NPL) PTP reference. The service time remained within the SLA for the duration of the fibre time-to-repair.

Figure 7: LD4 Grandmaster stability around a period of carrier fibre failure. A: Fibre failure NPL Teddington – Telehouse hub 08:03 UTC B: Reacquisition of reference on restoration of fibre path 20:40 UTC C: Stabilisation to reference 21:39 UTC

Conclusions For the duration of the trial and beyond, PTP timing provided by the NPLTime® end-point (a Meinberg PTP clock) remained within the SLA

(maximum 1 s deviation from UTC, 99.9% availability) as measured remotely by the NPLTime® management platform and by calibration at intervals throughout the trial. Repeated calibration measurements using a portable caesium clock showed a measured maximum offset of approximately 100 ns from UTC.

Initial deviations from the specification seen at the UBS trial server were found to be due to sub-optimal distribution from the NPLTime® end-point through site cross-connects to the UBS server. Deviations were eliminated by the addition of an extra Meinberg end-point

equipped with 1 Gbit interfaces and the removal of an intermediate fibre-copper media convertor. On reconfiguration, the trial NPLTime® service complied with the advertised and agreed SLA parameters.

This resilient UTC dissemination solution allows market participants to implement a ‘trusted time’ that is certified and can be utilised as a reference to aid infrastructural upgrade and configuration and enable high precision measurement of timing distributions. The complete monitoring, logging and audit capability of the solution is supported by a complete traceability chain to UTC, offering a documented route to demonstrate compliance to regulatory requirements.

Contact Details Further Information

National Physical Laboratory Hampton Road Teddington Middlesex United Kingdom TW11 0LW

Switchboard: 020 8977 3222 Website: www.npl.co.uk/npltime

Dr. Leon LoboStrategic Business Development

Time & Frequency

leon.lobo@npl.co.uk Phone: +44 20 8943 6383 Mobile: +44 77 1819 5448

NPLTime® is a certified precise time signal, directly traceable to Coordinated Universal Time (UTC) and independent of GPS. The solution provides the user with timing capability for traceable timestamping, latency monitoring and synchronisation. The signal is fully compliant with the MiFID II RTS 25 timing traceability requirement. The resilient service completely eliminates reliance on GPS and removes susceptibility to jamming, spoofing, urban canyon effects and solar storms. It reduces the costs associated with managing a complex assortment of timing devices and the need to access roof space to locate GPS antennas. Key benefits of NPLTime®

Risk mitigation o Not affected by solar storms, jamming, spoofing o Resilient close controlled solution

Simplicity of implementation o Time is consumed by the user, managed by the UTC lab o Leap seconds implemented as part of the solution

Maximises confidence in data timing o Ease of audit o Ensure accurate data release timing o Enhanced forensic and playback capability, algo optimisation o Continuous monitoring of traceability to UTC

Maximises benefit realisation of localised PTP infrastructure upgrades o Provides both absolute time and sync across implementations o Eases the requirement internally to achieve compliancy o Lower cost of infrastructure upgrade

Inherent synchronisation of multi-location implementations o NPLTime® SLA everywhere, managed by NPL

Eases measurement capability o Network optimisation and latency metrics o Simplistic tΒ-tΑ latency measurements

Key features of NPLTime®

Certified by the National Physical Laboratory Compliant with MiFID II RTS 25 Precision timing distribution solution No reliance on GPS or internet time Directly traceable and certified to UTC at the point of provision, not the source Eliminates susceptibility to GPS jamming, spoofing, urban canyon effects and solar storms Uses fibre optic links, ensuring maximum resilience and security Provides enhanced synchronisation No roof access required Built-in redundancy Continuous monitoring and audit of traceability to UTC

SLA – 1 s to UTC, 99.9% availability The views expressed above are of NPL and do not necessarily reflect the official views of UBS.