Time and clock synchronization with AFCK for CBM

M. Guminskia W. Zabo lotny a G. Kasprowicz a, K. Pozniak a, R. Romaniuk a

aWarsaw University of Technology, Nowowiejska 15/19, Warsaw, Poland;

ABSTRACT

The AMC FMC Carrier Kintex (AFCK) board is a prototype of Data Processing Board (DPB) for CBMexperiment. AFCK is open hardware and was designed to be a verstaile solution, applicable in multiple systems.In CBM experiment AFCK will serve as a data hub and communication interconnection. One of DPB’s functionsis synchronization of Front End Electronics (FEE). To provide this functionality, it is necessary to receive thereference clock and timing signals from the Timing and Flow Control system. The received clock after the jittercleaning will be used as a reference clock for GBT-FPGA based 4.8 Gbps links used for communication withfront end electronics. This article will briefly describe AFCK board funcion in CBM experiment. Afterwards itwill focus on jitter cleaning technique based on White Rabbit solution that can be used on AFCK board.

Keywords: CBM, AFCK, DBP, synchronization

1. INTRODUCTION

The Compressed Baryonic Matter experiment at FAIR will conduct a systematic research program to explorethe phase diagram of strongly interacting matter at highest net baryon densities and moderate temperatures.1

The experiment will produce very high data rate and require unconventional readout method. Unlike tradi-tional multi level triggering system, all data acquired form detector will be sent to the software event selector.High performance computer farm will sort acquired data into time slices and perform event detection and recon-struction. Interesting data will be stored for further analysis.

1.1 READOUT CHAIN

The elements of the CBM experiment readout chain elements will be located in three different locations. Blockdiagram is shown in Figure 1.

Front End Electronics - STS-XYTER2 will be located inside the STS detector. It will communicate withReadout Boards (ROB) - GBTx3 over 320 Mbps bidirectional link. Due to proximity of the detector both ofthose elements must be resistant to radiation.

Data from ROBs will be send to the CBM electronics area, outside of irradiation zone, via 4.8 Gbps links.Multiple ROB outputs will be concentrated by Data Processing Boards (DPB) - AFCK4 and send via 10 Gbpslink to High Performance Computing Cluster located in another building. Number of DBP inputs is not fixed,it will probably vary depending on readout setup and data processing done on the DPB.

Beside data processing, DPBs will also be an interface of Timing and Fast Control (TFC) and ExperimentControl System (ECS) systems. Gigabit Ethernet link with IPbus protocol will be used for ECS. It will enableconfiguration, calibration and other non-time critical transmission with FEE or ROBs. Utilization of IPbus islooked at as a simple way of debugging and first experimental data readouts.

Timing and Fast Control system is not yet fully defined, but main functions are known. All the electronicsmust work synchronously, so single main clock have to be distributed throughout the system. Time criticalcommands must be delivered in deterministic time.

Previously mentioned Computer Cluster will be used for Event Selection. It will acquire time sorted datafrom multiple 10 Gbps ROB links. FLES will combine this data from multiple ROBs in order to create TimeSlices. Particle detection and tracking will be done in parallel in multiple slices. Since tracking will be done infour dimensions (4th is time), TSs must be overlapping.

CBM Online WS -- Walter F.J. Müller,

FAIR

FEB FLIB FLES

Data

Cu

optical

on/near Detector CBM ‘Bunker’ 'Green Cube' 15-30m ~700m

Data&

Control

clock

GBTx VL DPB

Clock &

Data &

Control

ST

S,T

RD

,MU

CH

,…

TFC

DCS

TFC

DCS

TFC

DCS

Control

Network TFC

Network

Figure 1. CBM experiment readout chain. Front End Boards are used to read data from the detector. GBTx is anASIC designed at CERN as standard, radiation hard readout chain component. Data Processing Board is responsible forconcentration of readout data, timing and control. First Level Event Selector collects readout data through First LevelInterface Board.

AFCK top

28-Jun-15 by Marek Gumiński 1

FMC 2

FMC 1

Kintex-7 325T FFG900 FPGA

2GB DDR3

SATA AM

C

RTM

AFCK bottom

28-Jun-15 by Marek Gumiński 2

CPU LPC1764

clock mux ADN4604

multiple clock

sources and jitter

cleaning

Figure 2. DPB prototype board - AFCK. Main components and connectors are shown.

1.2 Hardware platform for DPB

AMC FMC Carrier Kintex (AFCK) is a versatile open hardware board, used as a prototype solution for CBMexperiment. The most important resources available on AFCK board are:

1. FPGA: Xilinx Kintex-7 325T FFG900 - used to realize most of DPB functionalities

2. CPU: LPC1764FBD100 - IPMI, might be used for clocking and power management

3. External memory: 2 GB DDR3 SDRAM (32-bit interface), 800Mhz

4. SPI Flash for user data storage

5. EEPROM with MAC and unique ID

AFCK may work in a standalone mode, but it is also compatible with MicroTCA for Physics (MTCA.4)crate. Regardless of working mode multiple connectivity options are available. Probably the most interesting aretwo High Pin Count (HPC) FPGA Mezzanine Card (FMC) connectors, Recently there are multiple commercialFMC extenders available in the market. In the CBM application two FMCs with 4 SFP cages are going to beused by default, but they can be replaced at any time. Other then that, AFCK has two SATA and two mini-USBports.

2. SYNCHRONIZATION

Accurate synchronization is essential for any Particle Detector. Since it is impossible to read all the data fromdetector with single electronics device, modern readout systems consists of multiple devices, acquiring datafrom different areas of the detector. Enormous data rates generated by the detector require precise selection ofinteresting event. In order to find such events data from multiple contiguous areas (read by different readoutchain branches) must be gathered in one place. This data is processed by event selector, that analyses possibleparticles trajectories and searches for interesting patterns.

Reconstruction of data, acquired by multiple devices, not only requires information about mutual position ofread areas, but also about time of data capture. All devices in readout chain must have local timers, based onprecise oscillators, because, since readout chain devices might be separated by a few kilometers, it is virtuallyimpossible to directly propagate high quality clock signal to all of them. Unfortunately there are no idealoscillators in the real world, each one has individual frequency comprised into certain range. Despite oscillatorsfrequency accuracy is generally very high, phase drift will occur, and it will raise with time. Phase drift over timemy be periodically corrected by software synchronization algorithm, but this solution does not prevent phasedrift, it only corrects an error that already occurred.

Applications that require wary high precision, such as High Energy Physics, can not accept such solution.

2.1 Clock distribution

Single main frequency distribution among all readout electronics devices is the only solution suitable for precisemeasurements done in CBM. As it was said above, it is not possible to propagate this frequency directly, so itwill be recovered from data send in serial links.

Clock recovery is based on locking Phase Locked Loop on raw data stream. PLL is usually fed with clocksignal so that it could tune its oscillator to reproduce input frequency. It is possible to create such PLL, that lackof some input clock edges will not cause PLL to change its oscillator frequency, or loose synchronization. SuchPLL may be fed with serial link data instead of clock. Locking of the PLL will occur as long as ceratin numberof edges over certain time will be provided. Adequate number of edges is assured by encoding standard. Forexample 8b10b encoding, used in Gigabit Ethernet, guarantees that there will never be more then 5 consecutivebits of the same value.

E-mail: m.guminski@stud.elka.pw.edu.pl

TFC GTX Phase

Detector

Loop

controller

DAC VCXO GTX DPB

system part AFCK FPGA AFCK board

Figure 3. PLL available on AFCK board block design. Block types are marked with different colors. TFC - Timingand Fast Control - timing source in CBM experiment. GTX - type of Xilinx multigigabit transceiver. VCXO - VoltageControlled Crystal Oscillator. DAC - Digital to Analog Converter.

Clock recovery from serial data has one serious disadvantage. Since PLL must be very tolerant on low qualityinput signal, it is impossible to produce high quality output clock. Recovered clock has the same mean-over-timefrequency as the reference, but depending on recent data pattern, instantaneous period might be altered a little.In other words clock is jittery.

3. JITTER CLEANING PLL

One of DPB functions is propagation of clock acquired from TFC system to ROB boards. Clock recoveredfrom TFC link must be used as reference for ROB boards, so that ROBs will be able to recover the same clockfrom DPB link. This task is not as trivial as it sounds because recovered clock quality is too low for fast serialtransmission reference. Clock refinement must be done by PLL with low noise oscillator. This PLL does nothave to be as resistant to jittery input as clock recovery PLL, so the high quality clock, with the same frequencyas TFC one, may be produced.

3.1 Mixed PLL

Figure 3 illustrates block diagram of Phase Locked Loop used in AFCK board. Unlike previously mentioned PLLdesigns, this one is not a single chip solution. Significant part of this design is implemented inside the FPGA,only Voltage Controlled Crystal Oscillator (VCXO) and Data to Analogue Converter (DAC), used to producetuning voltage, are separate integrated circuits. This design has one significant advantage over single chip PLLs:it is possible to compensate for the phase delay caused by wires, traces, transceivers and receivers. Recoveredclock (and cleaned recovered clock) have phase difference from phase of the main clock. To compensate this errorsynchronization protocol such as White Rabbit must be used.5

Reference clock of the design presented in Figure 3 is recovered from TFC serial link by Xilinx GTXtransceiver, with dedicated circuitry. As it was said before, this clock is very noisy and Xilinx does not al-low using it as reference clock for another GTX. It is fed to the phase detector implemented inside the FPGA.The other phase detector input is connected to hight quality clock generated by external VCXO. Depending ondetected phase difference VCXO frequency is tuned to match recovered one. When the PLL becomes locked,VCXO frequency will be stable and equal to TFC reference, it will be used as reference clock for ROB link.

The Digital Dual Mixer Time Difference (DDMTD) is accurate Phase Detector design, that is easily imple-mentable in the FPGA logic. Figure 4 illustrates two input clocks (recovered and clean) sampled by helper clockof different frequency. Sampling output generates two low frequency signals that differ in phase proportionallyto the difference of two input clocks. This new phase difference is big enough to be measured accurately. In thereal solution edges of sampling output contain multiple glitches, but it is possible to find middle of the glitchestime width and use it as edge transition.6

Figure 4. DMTD phase detector operation. Reference and recovered clock (CLK A, CLK B) are sampled with helperclock of different frequency. Sampling output generates two low frequency clocks, that may be easily compared. For moreinformation see.6

GPS WR node

AFCK

WR Grand master switch

Delay measurement

WR PPS

10MHz

Figure 5. Synchronization test setup.

Loop controller may be implemented as Proportional-Integral (PI) Controller. Derivative controller partmight be also used, but its influence on system that should lock to constant frequency would be small. The loopcontroller has to tune VCXO frequency using voltage generated by external DAC.

3.2 Phase compensation

Unlike standard PLL, that will aim for minimal phase offset, usage of separate PD and loop controller enableuser to add arbitrary phase offset between reference and clean clock.

4. WHITE RABBIT NODE

In order to show that AFCK is capable of realizing clock distribution functions White Rabbit node, available onwww.ohwr.org,5 was ported to this board. Calibration and performance measurements were done.

4.1 Accuracy measurements

Synchronization test setup was assembled as it is shown in Figure 5. White Rabbit switch, connected to GPSreceiver (Meinberg Lantime M600) is set up in the Grand Master mode (equivalent of TFC in WR network).WR node, implemented on AFCK, is connected to GM switch. PPS of both devices is fed to Pendulum CNT-91counter, set to measure delay between rising edges.

In order to verify synchronization accuracy (mean delay of node PPS to master PPS) five tests were done,after each test the optical fibre was reconnected. In order to verify synchronization precision (deviation of PPS

Figure 6. PPS delay histogram.

from one second to another, after the connection was locked) measurements of 120 delay values were taken,stdDev, min and max values were determined.

Test results are show in table 1. Tests confirm, that WR node implemented of AFCK is comatibile withofficial WR specification. Measured accuracy was better then 250 ps, standard deviation was lower then 40 ps.

All measuremetns were done with a single WR switch, but with different SFP ports.

Table 1. White Rabbit node synchronization test results. All values are given in ps.

NR. Mean StdDev Min Max

1 -110 35 -210 -50

2 103 39 0 190

3 237 36 140 340

4 206 35 120 280

5 196 38 100 280

5. CBM IMPLEMENTATION

White Rabbit protocol may be used for synchronization in CBM experiment but some modifications are required.GBT-FPGA core that is used as interface for GBTx boards on DPB require 120 MHz reference clock, while WRuses 125 MHz. It is possible to divide WR clock by 25 to acquire 5 MHz, and multiply this clock by the factorof 20. It should also be possible to modify WR protocol to work with 120 MHz clock.

At the moment custom PLL is under development, it is based on block diagram shown in Figure 3, it is using120 MHz clock recovered from TFC link and produces the same frequency for GBT-FPGA. This PLL will beindependent of WR solution, but multiple components will be reused.

REFERENCES

[1] Heuser, J. M., “Status of the cbm experiment,” EPJ Web of Conferences 95 (2015).

[2] Kleczek, R., Szczygiel, R., Grybos, P., Otfinowski, P., and Kasinski, K., “Time and energy measuring front-end electronics for long silicon strip detectors readout,” in [Nuclear Science Symposium and Medical ImagingConference (NSS/MIC), 2013 IEEE ], 1–4 (Oct 2013).

[3] Moreira, P., “The gbt project,” (2009).

[4] Zabo lotny, W. M. and Kasprowicz, G., “Data processing boards design for cbm experiment,” (2014).

[5] “White rabbit project,”

[6] Moreira, P., Alvarez, P., Serrano, J., Darwezeh, I., and Wlostowski, T., “Digital dual mixer time differencefor sub-nanosecond time synchronization in ethernet,” in [Frequency Control Symposium (FCS), 2010 IEEEInternational ], 449–453 (June 2010).