Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
When every nanosecond counts
www.sevensols.com
White Rabbit training
Seven Solutions
White Rabbit technology
White Rabbit technology
White Rabbit (WR) is a technology born at CERN which achieves sub-nanosecond
accuracy in Ethernet based networks. It allows easy deployments of scalable and
reliable networks with high accuracy synchronization requirements.
White Rabbit technology
• +10 years of expertise synchronizing large scientific facilities:
– CERN, GSI, Fermilab, …
• Validated by National Metrology Institutes: NIST, NPL, PTB,
OP, VSL, ROA, VTT, RISE, …
• New PTP High Accuracy profile to be released is based on the
pre-standard approach White Rabbit.
White Rabbit technology
Deterministic and highly accurate This allows saving engineering and equipment costs to achieve a global target time budget.
Cost-effective It compensates dynamically link asymmetries and temperature changes. Easy to deploy, pre-calibration.
Scalable to long distances & high numberof nodes. It supports tree topologies and daisy-chain configurations
DependableIt reduces vulnerabilities to spoofing or GPS jamming. Up to 100 km links without on site calibration.
Facilitates new services Positioning, High Frequency Trading, Time as a Service
Easy to integrate within existing infrastructures (Ethernet, PTPv2).
White Rabbit technology: Syntonization
▪ To reach subnanosecond synchronization, distributing the same clock through thenetwork is needed.
▪ Syntonization: local clock tuning based on a measure of the error between two clocks. InWhite Rabbit, the external clock and the internal reference are compared.
125.00 MHz
124.98 MHZ
125.03 MHz
Free running clocks with the same nominal frequency are not accurate enough for accurate time transfer
Not SyntonizedSyntonizingSyntonized!
L1 syntonization is used as a customized version of Sync-E to transmit the clock over the optical links. It useslocal VCXO to syntonize the local clock to the recovered clock from the link (Slave role).
White Rabbit requires specific clock circuitry to perform the syntonization and ensure accuratetime synchronization.
125.00 MHz
125.00 MHZ
125.00 MHz
White Rabbit technology: Syntonization
White Rabbit measures the offset between devices, taking into account the link asymmetry anddynamic variations because of weather conditions using picosecond level accurate timestamps.
WR slave
WR master
Timestamp generation based on frequencymixing techniques: Clock phasemeasurements
Synchronization: L1 syntonization & PTP (IEEE-1588v2)
The timestamps are capable of measuring time differences between two digital clock signals with very fine resolution(picosecond) and they are used to adjust the received and generated clock offsets.
White Rabbit technology: Ultra-accurate timestamp
White Rabbit technology: Synchronization
White Rabbit uses the information collected by the exchange of timestamped packets for correcting theconstant offset between nodes (bA≠bB)
The information from the calibration is also important for compensating the static offset between nodes.
White Rabbit technology: Topology
PPS Slaves
Time Reference
10 MHz & PPS
WR-ZEN TP WR-ZEN TPWR-SWITCH
WR
PTP Slave
PTP
PTP Switch
PTP
PTP
WR-SLAVES
WR
PPS
WR-ZEN TP-32BNC
X ns from ref.± 50 ns from ref.
< 1 ns from ref.
PTP Slaves
± 100 ns from ref.
WR-SWITCH
White Rabbit technology: Time reference
Atomic clocks count time relying on the resonance frequency of atoms excited by microwave, optical or ultravioletradiation.
• They are one of the most stable time references available.
• The most widespread atomic clocks are based on cesium or rubidium.
Any GPS receiver gets an accurate time, traceable to the atomic clocks present inside GPS satellites, as it is necessary tocalculate the receiver position.
• GPS-synchronized time receivers just ignore the position information and adjust a local tunable oscillator accordingto the data obtained from the satellites.
White Rabbit technology: Network
• The use of fiber makes encoding easier, as the optical medium does not sufferfrom crosstalk or electric interference.
• 1.25 Gbps serial datastream comes out of the serializer.
White Rabbit technology: SyncE
• In a regular Ethernet network, every node uses its own free running oscillator. Small differences of frequency between tx and rx circuits are compensated by asynchronous packetbuffers.
• Sync-E defines a hierarchycal structure where the master at the top is connected to a primaryclock.
• STM syntonizes its oscillator to the primary clock and uses this frequency to encode the data.• At the receiver end, the same frequency is recovered using PLLs and is used with lower nodes
in the hierarchy and back to the master.
White Rabbit technology: PTP
• PTP synchronizes the slave clock with a master clock
• Link delay measured with timestamped frames• Supports HW timestamps
White Rabbit technology: Phase measurements
• Ethernet packets can be received at a device at any moment.• The 125 MHz transmission clock only provides 8 ns granularity.• Phase offset measurements improve the accuracy of PTP timestamp
exchanges.
White Rabbit technology: Assimetry compensation
White Rabbit calculates link assimetries using pre-calibration:• Internal delays• SFP delays• Fiber assimetry
Δ = Δtxm + Δrxm + Δtxs + Δrxs
delayMM = Δ + δms + δsm
White Rabbit technology: Timing daemon
Link up
Syntonize
Calibrate PHYs
Measure coarse delay
Measure phase
Extend timestamps and obtain fine delay
Determine link asymmetry
Compute one-way delays and clock offset
Initial offset correction
Measure phase
Compensate for delay changes
White Rabbit technology: Synchronization
Once syntonized, we still have to synchronize both nodes. This can be divided into two tasks:
• Coarse delay measurement, based on a PTP exchange.• The coarse delay is measured using a PTPv2 two-step packet exchange.
• Precise delay measurement that combines the coarse delay with a DDMTD phase measurement
The packets timestamps are hardware-generated:
• This measurement produces timestamp values t1, t2, t3, t4. The precision of the timestamps is extended toinclude the phase measurements: t2p, t4p.
• Due to the possibility of jitter-related problems, t2 and t4 timestamps are generated for both rising and falling edge.
• Precise round-trip delay can be calculated as delayMM = ( t4p – t1 ) – ( t3 – t2p )
White Rabbit technology: Synchronization
• The asymmetry cannot be directly measured. It can only be estimated from delayMM and knowledge of the mediumand the transmission circuits.
• All these asymmetry sources are taken into account:• Propagation delays of eletronic components and PCB traces• Optical transceivers delay asymmetry• Fiber Rx/Tx different diffraction index.• Internals of the chips structure.
• Fiber asymmetry can be variable depending on operating conditions and link length. All the rest are consideredconstant per device.
White Rabbit technology: Synchronization
• dTX and dRX fixed delays inside the hardware up to the optical port
• δMS and δSM are the one-way fiber delays from master to slave and back
White Rabbit technology: Synchronization
• BiDi SFP (1310/1490 nm)
• Single-mode fiber G652
• Relative Delay Coefficient of the OF (α) • Describes the asymmetry of the fiber• The speed of propagation in the fiber depends on the wavelength
• BiDi SFPs uses λtx and λrx→Slight difference in propagation of speed
α =𝛿𝑀𝑆𝛿𝑆𝑀
− 1
White Rabbit technology: Synchronization
Δ = Δtxm + Δrxm + Δtxs + Δrxs
delayMM = Δ + δms + δsm
α = (δms / δsm) - 1
δms= (1+α)(delayMM-Δ) / (2+α)
delayms= (1+α)(delayMM-Δ) / (2+α) + Δtxm + Δrxs)
offsetms = t1 – t2p - delayms
delayMM = ( t4p – t1 ) – ( t3 – t2p )
White Rabbit technology: Links
• White Rabbit users
https://www.ohwr.org/project/white-rabbit/wikis/WRUsers
• White Rabbit documentation
https://www.ohwr.org/project/white-rabbit/wikis/home
When every nanosecond counts
www.sevensols.com
Overview of products
White Rabbit technology: WR-Switch
It is the main element of the White Rabbit Technology. It has 18 SFP 1GbE ports in a 1U form factor that can be configured to work as master or slave to deploy an operational White Rabbit network.
White Rabbit technology: WR-LEN
It is the standalone WR device capable of supporting daisy chain configurations. It allows a cost-effective solution to distribute PPS/10MHz signals or IRIG-B protocol to your equipment.
White Rabbit technology: WR-ZEN
It is the interoperability element of the White Rabbit Technology. It has 2 SFP 1GbE ports that can be configured to work as master or slave to deploy daisy-chain configurations or to provide a redundant White Rabbit connection. The WR-ZEN TP devices have a redundant dual power supply. This device can obtain its time reference from an external source using both 10MHz and 1PPS signals, from another White Rabbit device through its optical ports or can work as a free-running device. It can also configure a NTP connection to a server to get the time information on its management ports.
White Rabbit technology: WR-ZEN
When every nanosecond counts
www.sevensols.com
White Rabbit synchronization market use cases
5G Telecom Networks
• Synchronization of RRHs and BBUs
• Cross validation of time references in telecom core networks
(monitoring)
• WR over WDM
5G synchronization flow
- Centralizes Baseband Units (BBU) in a server
- BBUs all co-located, simplifying synchronization for eICIC and CoMP
Cross validation of time references in telecom core networks (monitoring)
• Synchronizing (cross-validating) Primary Time References over the core
network
Site 2
Site 3
Site 1
Core NetworkGM
WR-ZEN TP
GM
WR-ZEN TP
GM
WR-ZEN TP
WR over WDM
Fron
thau
l/backh
aul
device
sFr
on
thau
l/b
ackh
aul
dev
ice
s
………..………..………..
………..………..………..
………..………..………..
………..………..………..
Visibility network
• Distribute timing
from the time
reference to the
entire datacenter
• Deliver accurate and
precise PPS to
every capturing
device
• All capturing devices
synchronized
10 MHz & PPSWRPPS
Cabinet 3Cabinet 2Cabinet 1
WR-SWITCH
Time Reference
WR-ZEN TP-32BNC WR-ZEN TP-32BNC WR-ZEN TP-32BNC
Capturing device
Capturing device
Capturing device
Capturing device
Capturing device
Capturing device
Capturing device
Capturing device
Capturing device
Visibility network
• Tap every link
• Timestamp every packet
• A detailed live map of
the network can be
obtained for analysis
• Validated in the finance
sector by the German
stock exchange
(Deutsche Börse press
note).
Level 1
Level 2
Level 3
Level 4
GMGPS
WR-Switch
Timestamper
Timestamper
Timestamper
Timestamper
WR10 MHz
1 PPS
WR-ZEN TP-32BNC
1PPS
Jamming and spoofing protection
• A single local GNSS antenna represents an important risk.
• White Rabbit can be used to diversify the time reception.
• Deploy two or more GNSS antennas and interconnect them through a WR link.
• Jamming/spoofing a two antennas in different locations is much harder.
• Possibility to connect to a National Metrology Institute for UTC reference
alternatively to GNSS.
UTC referenceWR-ZEN TPWR-ZEN TP
10MHz
PPSUTC referenceWR
10MHz
PPS Time referenceTime reference
Saving time budget on inter metro links
• Send a local reference to a different metro location with sub-nanosecond accuracy and
precision.
• A 10 MHz & PPS or PTP reference must be provided to a WR device in location A.
• Possibility to interconnect to a National Metrology Institute for an alternate source of UTC.
• Up to 120 Km in one hop. Several hops maintaining sub-nanosecond accuracy.
• The timing can be recovered in multiple formats:
– 10 MHz & PPS
– IEE 1588
– ToD
– IRIG-B
Location A Location B
WR
Cabinet 2Cabinet 1
WR-ZEN TP NIC
WR-ZEN TP NIC
WR-ZEN TP NIC
WR-ZEN TP NIC
NIC
WR-SWITCH
Time Reference
WR-ZEN TP-32BNC
NIC NIC NIC NIC NIC
NIC NIC NIC NIC NIC NIC
Server synchronization datacenter wide
• Distribute timing from
the time reference to
the entire datacenter
• All WR network with
sub-nanosecond
synchronization
• Only one final
PPS/PTP hop
• Save PTP link
calibration
10 MHz & PPSWRPTPPPS
Inter-datacenter synchronization solutions
Timing master datacenter
Time Transfer using optical network (<80 Km)
>120 Km time Transfer using GNSSreceivers as primary time reference(APTS: network as backup)
>120 Km time Transfer using fiber asprimary time reference (GNSS forasymmetry calibration and backup)
Inter-datacenter
• Transfer of a time reference for distances up to 120 Km
• Performance is dependent of the fiber in place:
– Dedicated fiber: sub-nanosecond (< 1ns)
– DWDM fiber:
• Accuracy depending on the system: Measurable and calibrated
• Sub-nanosecond precision (< 1ns)
Inter-datacenter. Redundant timing
< 120 Km
Datacenter A Datacenter B< 120 Km
Time Reference
10 MHz & PPS
WR-ZEN TP
WR
Time Reference
WR-ZEN TP
WR-SWITCH
WR
WR
10 MHz and/or PPS
Datacenter A Datacenter B
Time distrib.to the cabinetsWR-SWITCH
Time distrib.to the cabinets
Resilient timing to GNSS jamming
Switch over in case of failure of local GNSS
Scientific facilities
• High Energy Physics. Particle accelerators
• Telescopes and sensor arrays
Scientific facilities
• Synchronizing distributed instrumentation over a scientific
infrastructure
– Particle accelerators
– Distributed arrays of radio-astronomy telescopes
– Distributed sensor networks
Scientific facilities
High Energy Physics. Particle accelerators.
▪ CERN (Switzerland)
▪ GSI (Germany)
▪ LHAASO (China)
▪ Sirius (Brazil)
▪ IFMIF-EVEDA / ENS
Telescope and sensors array
▪ KM3Net (Underwater Neutrinos detector)
▪ CTA (Cherenkov Telescope Array)
▪ SKA (Square Kilometer Array)
Smart grid
• Power distribution over smart grid: Synchronizing core elements in the
Smart Grid as Phase Measurement Units (PMUs).
– Time stamping all registered data over smart grid for better detection of failures
in a forensic analysis after a black out.
Defense, Aerospace, Broadcasting…
• Ground base satellite control synchronization.
• Distributed radars needs accurate synchronization between
antennas.
• Broadcast equipment support 10 MHz and 1PPS inputs for
synchronization.
Intra-datacenter time transfer and synchronization
Intra-datacenter
PPS Slaves
Time Reference
10 MHz & PPS
WR-ZEN TP WR-ZEN TPWR-SWITCH
WR
PTP Slave
PTP
PTP Switch
PTP
PTP
WR-SLAVES
WR
PPS
WR-ZEN TP-32BNC
X ns from ref.± 50 ns from ref.
< 1 ns from ref.
PTP Slaves
± 100 ns from ref.
WR-SWITCH
Intra-datacenter
• Sub-nanosecond distribution to each cabinet
• Expend < 1ns of time error budget to each cabinet
• Choice of interoperability for last hop:
– 1 last PTP hop (±50 ns from the reference)
– 2 last PTP hops (±100 ns from the reference)
– 1 PPS hop (sync accuracy depending on clock performance of the slave)
– WR until the end node (< 1ns from the reference)
• Vendors tested: Metamako, SolarFlare, Napatech, Endace, Meinberg, …
Intra-datacenter Time Transfer and Synchronization – 1 and 2 PTP Hops
Intra-datacenter time distribution
▪ PTP interoperability using WR-ZEN TP devices.
▪ Setup 1: Direct connection to PTP Slave (± 50 ns accuracy)
▪ Setup 2: Multiple PTP slave connection using a PTP
switch (± 100 ns accuracy)
▪ Tested devices: Solarflare, Metamako, Oregano,
Endace, Napatech.
Solarflare (SF7322) Test
-25
-20
-15
-10
-5
0
5
10
15
20
25
0 300 600 900 1200 1500 1800 2100 2400 2700 3000 3300 3600
Off
set
(ns)
Samples
PPS Offset between SolarFlare devices
Intra-datacenter Time Transfer and Synchronization – 1 and 2 PTP Hops
Arista Test:
Reference
10 MHz
1PPS
WR PTP PTP
WR switch
WR ZEN TP Arista 7150S
SF7322
100 ns
Intra-datacenter Time Transfer and Synchronization – 1 and 2 PTP Hops
Intra-datacenter
Intra-datacenter PPS Distribution:
▪ Setup 3: Up to 32 PPS slaves connection
or 16 PPS & 10 MHz slaves connection.
▪ Tested devices: Solarflare, Metamako, Meinberg…
Intra-datacenter
MetaMako tests (PPS sync)
WR-LEN
Reference
PPS
10 MHz
WR-ZEN TP-32BNC
Fiber PPS
Metamux 32
20 ns
Inter-datacenter time transfer and synchronization
Key ideas
• PTPv2 does not provide deterministic time transfer → WR does,
– It guarantees high precision PPS distribution through all the network.
– Temperature compensated. Scenarios. Second order drift (<100ps) can be modelled and adjust in most applications if required.
• The asymmetry problem: offset on the PPS signals → Different wavelengths (different group velocities), differences on optical fibers lengths, asymmetries on the amplifiers / DCMs / DWDMs / transponders, etc...
– The asymmetries can be measured and compensated on most circumstances.
– For blind situations, a calibrated GNSS receiver is a simple and cost effective solution.
– Alternatively, we can modify the network adding bidirectional amplifiers (costlyR&D approach) or characterize all devices on the network (not always a realistic approach).
Inter-datacenter time transfer and synchronization
Timing master datacenter
Time Transfer using optical network (<100 Km)
>120 Km time Transfer using GNSSreceivers as primary time reference(APTS: network as backup)
>120 Km time Transfer using fiber asprimary time reference (GNSS forasymmetry calibration and backup)
Inter-datacenter time transfer and synchronization
• Sub-nanosecond
synchronization using
bidirectional optical fiber
links.
• Distances up to 100 Km.
• Dedicated or DWDM
optical networks channelswithout amplification.
• Dedicated or data-shared
timing channels.
Inter-datacenter time transfer and synchronization
< 120 Km
Datacenter A Datacenter B< 120 Km
Time Reference
10 MHz & PPS
WR-ZEN TP
WR
Time Reference
WR-ZEN TP
WR-SWITCH
WR
WR
10 MHz and/or PPS
Datacenter A Datacenter B
Time distrib.to the cabinetsWR-SWITCH
Time distrib.to the cabinets
Inter-datacenter time transfer and synchronizationTim
ing m
aster
de
viceTi
min
g m
aste
r
dev
ice
• Support for unidirectional WDM networks.
• Asymmetry calibration is provided based on GNSS receivers.
Inter-datacenter time transfer and synchronization
WR-ZEN TP WR-ZEN TPWR(20 Km)
Keysight 53230Frequency Counter/Timer
350 MHz – 20 ps resolution
PPS PPS
• Lab test• 24 hours measure• Sub-nanosecond accuracy and precision
Inter-datacenter time transfer and synchronization
Metro links
GPS-DO
PPS
10 MHz
20 Km
20 Km
20 Km
Counter20 ps
resolution
PPS
PPS
HA profileCompatible Device
HA profileCompatible Device
HA profileCompatible Device
HA profileCompatible Device
16 hours
Metro links
Metro links
18 hours
GPS-DO
PPS
10 MHz
50 Km
70 Km
Counter20 ps
resolution
PPS
PPS
HA profileCompatible Device
HA profileCompatible Device
HA profileCompatible Device
Metro links
-100
-75
-50
-25
0
25
50
75
100
0 3000 6000 9000 12000 15000 18000 21000 24000 27000 30000 33000 36000 39000 42000 45000 48000 51000 54000
Tim
e er
ror
(ps)
Seconds
Time Error on 120 Km, 2 hops
GNSS backup bi-fiber
• Unidirectional WDM connection through 50 km fiber
GNSS backup bi-fiber
23,06
23,08
23,10
23,12
23,14
23,16
23,18
23,20
23,22
23,24
23,26
0 2000 4000 6000 8000 10000 12000 14000
Tim
e er
ror
(ns)
Time (s)
Time error on 50 km long DWDM link
Results provided by the University of Granada with 7Sols equipment.Further results Will be presented soon.
Long Haul WR Calibration Assisted by GPS
▪ In-situ calibration impossible due to the great separation between WR nodes.
▪ GNSS techniques can be used to easily compare Master-Slave offsets
▪ It provides a check to ensure the correct functioning of terrestrial networks
▪ Suitable for network with passive or active optical equipment (fixed asymmetry)
> 40 Km
Long Haul WR Calibration Fiber Swap Method
▪ In-situ calibration impossible due to the great separation between WR nodes
▪ Different lengths in the paths
▪ Different wavelength dispersion
“Long range time transfer using optical fiber links and cross comparison with satellite based methods” Namneet Kaur
Long Haul WR Calibration Fiber Swap Method
▪ Both fibers transmitted the same wavelength
▪ Any deviation is caused only by the fiber
▪ Slave PPS is compared with a stable PPS reference
Long Haul over WR BiDi DWDM
▪ Fixed latency of WR equipment calculated using WR Model
▪ Resolves calibration problems of tech and “complex” OF networks:
▪ Uncertainty of the wavelength long distance BiDi SFPs
▪ Uncertainty of chararterisitics among OF vendor
▪ α in networks with mixed version of OF (G652b/d)
▪ Reduces impact of αDWDM ~10-6 < αBiDi ~10-4
Long Haul over WR BiDi DWDM
▪ Resolves calibration in mixed FO networks G652b and G652d
Device SFP Fiber Optic α Offset (ps)
WRS BiDi 50 km G652b 2.59E-04 225
WRS BiDi 50 km G652d 2.59E-04 2860
WRS BiDi 20 km G652b 2.36E-04 1158
WRS DWDM 50 km G652b 3.80E-06 11
WRS DWDM 50 km G652d 3.80E-06 8
WRS DWDM 20 km G652b 3.80E-06 52
WRS DWDM 100 km G652b&d 3.80E-06 46
When every nanosecond counts
www.sevensols.com
White Rabbit topologies
White Rabbit network topology
White Rabbit network topology
• Time server: It is the device that the White Rabbit network uses as the time reference. This time reference will be distributed through the White Rabbit network with sub-nanosecond accuracy. Using a time server is optional, if no time server is connected to the White Rabbit network the top White Rabbit device will work as a free-running mode. The time server must provide 1PPS and 10 MHz signals to the first White Rabbit node, which will lock to the incoming frequency and will determine the second beginning with the 1PPS input. Additionally, a NTP server can be configure to give a notion of time to the first White Rabbit node. Standard PTP could be used in specific conditions to substitute the 1PPS/10 MHz and NTP from the time server.
• First White Rabbit device: This device must be configured as Grandmaster. In this mode it will lock to the incoming 10 MHz signal and will determine the second beginning with the 1PPS input. All its optical interfaces will be automatically configured as master, so this device can distribute the obtained time reference to the subsequent White Rabbit devices.
White Rabbit network topology
• Intermediate White Rabbit devices: These devices receive the time reference using the White Rabbit protocol on one of their optical interfaces. They can daisy-chain this White Rabbit reference on other optical port, acting as master of the lower level. In the case of WR-Switches, they can deploy a tree topology using one of their optical ports to receive the White Rabbit reference from the first White Rabbit device and the other 17 optical ports to distribute it to other White Rabbit compliant devices.
• Last White Rabbit device: This device receives the time reference from a previous White Rabbit device and it is responsible of propagating this time reference to any other device which will be synchronized. There are several interoperability options depending on the White Rabbit device that is used on this last hop:
• IEEE 1588, Standard PTP v2. • 1PPS synchronization. • White Rabbit integration in third party devices. • NTP. • IRIG-B, NMEA, ToD…
White Rabbit network topology
White Rabbit network topology
White Rabbit network topology: Local scenarios
White Rabbit network topology: Local scenarios
White Rabbit network topology: Local scenarios
White Rabbit network topology: Local scenarios
White Rabbit network topology: Remote scenarios
White Rabbit network topology: Remote scenarios
White Rabbit network topology: Remote scenarios
White Rabbit network topology: Remote scenarios
White Rabbit network topology: Remote scenarios