Webinar – October 14, 2020

Impact of Timing on Synchronization in Data Centers and 5G Networks

Gary Giust, PhD

Sr. Mgr, Technical Mktg

Nazariy Tshchynskyy

Sr. Mgr, Customer Engineering

Jeff Gao

Sr. Dir, Marketing

2

Agenda – Impact of Timing on Synchronization in Data Centers and 5G

1. How to Model Oscillator Performance

2. Impact of Temperature Sensitivity

3. Impact of Aging and Wander on Holdover

4. Case Studies

5. Conclusions

Modeling Oscillator Performance in 5G Networks and Data Centers

4

Oscillators in IEEE 1588PTP with Physical Layer Frequency Support

5G Networks

PTP without Physical Layer Frequency Support

Data Centers

5

Modeling Servo Impact to PTP Time Error

-

Loop Filter

Local Time

Local Time

Local Clock

Frequency correctionTimestamp

processing

Network time stamps

Networktime

Local Oscillator

(LO)

Network noise Servo Loop LPF

+Servo Loop HPF for LO

Time Error – combination of Network Performance and

Oscillator Noise

Time_error (s)

time (s)

Filtered network time

LO noiseFiltered LO noise

6

Modeling Oscillator-noise Impact on Time Holdover

Time_error (s)

time (s)

Time error (t)

1.5us

-1.5us

f (ppb)

time (s)

OSC frequency in time

∫0

tΔf(t) Δφ(t) /(2πfc)

f (ppb)

T (°C)

f (ppb)

time (s)

f (ppb)

time (s)

Frequency vs Temperature 1 day aging

Temperature ProfileT (°C)

time (s)

T1 T2

T1

T2

Frequency change in time (due to temperature)

Dwell time = 15 minRamp rate = 0.5 °C/min

+

f (ppb)

time (s)

Frequency wander

Time_error (s)

time (s)

Time error (t)

1.5us

-1.5us

f (ppb)

time (s)

TCXO frequency in time

∫0

tΔf(t) Δφ(t) /(2πfc)

PLL responsemodel

f

H(f)

f (ppb)

T (°C)

f (ppb)

time (s)

f (ppb)

time (s)

Frequency vs Temperature 1 day aging

Temperature ProfileT (°C)

time (s)

T1 T2

T1

T2

Frequency change in time (due to temperature)

Dwell time = 15 minRamp rate = 0.5 °C/min

+

f (ppb)

time (s)

Frequency fluctuations at constant temperature

Impact of Oscillator Temperature Sensitivity on PTP Time Error

5G Networks and Data Centers

8

PTP Time Error for 1 ppb/°C and 10 ppb/°C TCXO (time constant = 1 min)

9

PTP Time Error for 1 ppb/°C and 10 ppb/°C TCXO (time constant = 10 min)

10

Frequency-over-Temperature Slope, dF/dT

• Frequency-over-temperature slope quantifies frequency change per change in temperature, in units of ppb/°C

Measured MEMS TCXO Frequency Stability

-100

-50

0

50

100

-50 -10 30 70 110

Fre

qu

en

cy S

tab

ility

(p

pb

)

Temperature (C)

ΔT

ΔF

Slope = ΔF/ΔT [ppb/°C]

11

Which Part is Better for PTP Applications: 50 or 100 ppb?

-100

-50

0

50

100

-50 -10 30 70 110

Fre

qu

en

cy S

tab

ility

(p

pb

)

Temperature (C)

± 50 ppb

± 100 ppb

What matters is low sensitivity to temperature changes (dF/dT), not lifetime peak-peak stability

12

Frequency Stability Video – Impact of dF/dT on Performance

https://www.youtube.com/watch?v=Bttd1f1wo6g&feature=youtu.be

>200 ppb jump

Elite is stable>20 ppb jump with fan on

50C

±50 ppb Quartz TCXO ±100 ppb MEMS TCXO

https://www.youtube.com/watch?v=Bttd1f1wo6g&feature=youtu.be

13

MEMS Architecture – Optimized for Low PTP Time Error

• Excellent thermal coupling between 2 MEMS

resonators in same die

• Digital low-noise, high-bandwidth TDC

• Enables compensation of fast temperature changes

• Higher-order compensation enables smallest dF/dT

residue

• Limited thermal coupling between

Quartz and ASIC

• Analog noisy temp sensor

• Limited compensation of slow temperature

changes only

• Higher-order large dF/dT residues

SiTime DualMEMS™ Temperature SensorQuartz TCXO

Quartz Crystal

But temp

sensor is here!

ASIC

Need to measure

temperature here

TempFlat™ Resonator

Temp Sensing Resonator

DualMEMS™

Resonator Die

SiTime ASIC

MEMS

TempFlatTM

ResonatorTemp Sensing

Resonator

14

Elite-platform TCXO Architecture – Optimized for PTP Applications

TempFlatResonator

Temp Sensor

DualMEMSThermally Coupled

Mixed-Signal CMOS IC

CLK

I2C

OSC-Temp

OSC-Timing

TempSense(TDC)

Frac-N PLLFrequency Synthesis

TempComp

VCXO ADC I/O

Dividers& Driver

On-ChipRegulators

OEVctrl

TempFlat™ MEMS Resonator

• No aging

• No activity dips

• 30x better vibration immunity

Low Noise CMOS Enabling Frequency Agility

• 1 to 220 MHz, steps in mHz

• 0.2 ps/mv PSNR, 5x better

• In-system programmability

DualMEMS™ Temp Sensing 100% Thermal Coupling

• 30 µK, 10x more accurate

• 350 Hz tracking, 40x faster

• Airflow, temp ramp resistant

15

MEMS vs Quartz TCXO dF/dT Performance

1 ppb

- 1 ppb

Better than 1 ppb/°C

Impact of Oscillator Aging and Wander on Holdover

5G Networks

17

Model of Time Holdover

Time_error (s)

time (s)

Time error (t)

1.5us

-1.5us

f (ppb)

time (s)

OSC frequency in time

∫0

tΔf(t) Δφ(t) /(2πfc)

f (ppb)

T (°C)

f (ppb)

time (s)

f (ppb)

time (s)

Frequency vs Temperature 1 day aging

Temperature ProfileT (°C)

time (s)

T1 T2

T1

T2

Frequency change in time (due to temperature)

Dwell time = 15 minRamp rate = 0.5 °C/min

+

f (ppb)

time (s)

Frequency wander

18

Holdover Definition

• “Holdover begins when a clock output no longer reflects the influence of a connected external reference” [1]

• Example: GPS is used to synchronize timing across base stations in telecom networks. Holdover is used to maintain connections when user moves from one cell site to another and GPS is unavailable.

• Two common usages of “holdover”

1. Frequency holdover is the maximum frequency deviation from the average frequency (in ppm) over some time interval (tau), after the system loses lock to its reference.

2. Phase (or time) holdover is the accumulated time error with respect to a reference clock after the system loses lock to its reference.

[1] https://en.wikipedia.org/wiki/Holdover_in_synchronization_applications

https://en.wikipedia.org/wiki/Holdover_in_synchronization_applications

19

Phase, Frequency, and Time Error

• Relationship between phase and frequency

𝜙 𝑡 = 0𝑡𝜔 𝑡 𝑑𝑡, where

𝜙 𝑡 is the phase of a signal in radians

𝜔 𝑡 is the frequency of the signal in radians/sec

• Hence time error of a clock signal is the integration of its frequency error

𝑇𝑒𝑟𝑟 𝑡 = 0𝑡∆𝑦 𝑡 𝑑𝑡, where

𝑇𝑒𝑟𝑟 𝑡 is the time error

∆y 𝑡 =∆𝑓 𝑡

𝑓is a fractional frequency error

20

Components of Time Holdover

• Useful to separate components impacting oscillator holdover

- For example, to analyze impact of temperature profiles on holdover

• Main oscillator holdover contributors

1. Temperature Variation. Oscillators have frequency sensitivity to temperature changes that can be expressed as ppb/°C (frequency change per 1 °C of temperature change). Therefore frequency of the oscillator is modulated by environment temperature changes.

2. Short term aging quantifies how much frequency of the oscillator changes per day due to aging, and can usually be approximated with a straight line.

3. Time Deviation (TDEV) quantifies wander (random phase fluctuations) in the oscillator.

• If daily aging is a straight line, Time Error (∆𝑇) may be predicted as (per NIST publication 1065):

∆𝑇 = 𝑇0 +∆𝑓

𝑓∙ 𝑡 +

1

2𝐷 ∙ 𝑡2 + 𝜎𝑥 𝑡 , where

• 𝑇0 is the initial synchronization error (seconds)

•∆𝑓

𝑓is the fractional frequency that is the sum of average environment induced frequency error

• 𝐷 is daily aging rate (fractional frequency per second)

• 𝜎𝑥 is the rms noise induced time deviation, or wander (seconds)

21

dF/dT = 5 ppt/°C

aging = 0.2 ppb/day

T variation = 25°C to 35°C

Combined Holdover

• Add all 3 components to compute system holdover

• The following plot shows an example time error computation

22

Example OCXO Holdover at 45±15 C, 0.5 C/min Temperature Ramp

6.58 Hours @ 1.6 µs

23

Aging Compensation Improves Stability

• Daily aging can be predicted with some accuracy

• Aging observed over time can be used to predict/remove frequency drift

1. Fit a line to first 24h of data (solid black line)

2. Extrapolate this line to the next

12h of data (dotted black line)3. Subtract extrapolated

line from the dataset

24

Aging Compensation Improves Holdover

Without aging compensation With aging compensation

25

Hadamard Deviation (HDEV)

• A measure of clock-frequency stability, similar to Allan deviation (ADEV)

• Hadamard variance removes linear frequency drift from ADEV

𝐻𝜎𝑦2 𝜏 =

1

6(𝑀−2)σ𝑖=1𝑀−2[𝑦𝑖+2 − 2𝑦𝑖+1 − 𝑦𝑖]

2,

where 𝑦𝑖 , 𝑖 = 1. .𝑀 is a set of fractional frequency values measured back to back and each averaged over

the time interval 𝜏

• Why remove linear drift from frequency stability?- Better models networks whose time sources are compensated for daily aging

- Common in time-synchronized networks requiring holdover (5G, telecom, etc.)

26

HDEV is Similar to ADEV with Linear Drift Removed

After removing linear drift, ADEV looks like HDEV

Original ADEV

ADEV with linear drift removed

HDEV

Linear drift (aging)

27

Holdover Measurement

• Holdover can be measured at component or system level

• Component level measurement requires an instrument that can…

- measure frequency of the clock continuously with high resolution and without gaps for extended periods of time (multiple days), for example Keysight 53230a frequency counter, or

- measure time of edges (timestamp edges) continuously every 1s or more often with high resolution for extended periods of time (multiple days), for example GuideTechGT667 CTIA

• System level measurement…

- monitors the time difference between 1 PPS signal from a time master and 1 PPS signal from a slave under test (e.g. master to slave time error). It can be measured with a frequency counter, like Keysight 53230a, digital oscilloscope, or any other instrument that can accurately measure time between signal edges on 2 independent inputs.

28

Component-level Holdover Setup

Rubidium

Reference

Time Interval Analyzer

Guide Tech GT667

or

Frequency Counter

Keysight 53230a

Temperature Chamber

TestEquity 115

Test Board

50 Ohm

coax cable

OCXOCLK

29

System-level Holdover Setup

Rubidium

Reference

Frequency Counter

Keysight 53230a

Temperature Chamber

TestEquity 115

Slave

1 pps OCXO

Master (Time Server)

1 pps or

10 MHz 1 pps

1588 link

Case Studies

31

eCPRI

FronthaulSwitch

RRU

Elite TCXO

IEEE 1588Processing

MEMS Clocks for IEEE 1588/eCPRI

RF

BBU Server

CascadeNetwork

SynchronizereCPRI/SyncE

MEMS Timing Engineered for Robust 5G RRU Outdoor Deployments

Problems that MEMS Timing Solves

• Ensure

32

FPGAAccelerator

CascadeNetwork

SynchronizereCPRI/SyncE

Elite TCXO

MEMS Clocks for Accelerator Card

EthernetPCIe

IEEE 1588

IEEE 1588Fronthaul

Switch

BBU Server

RRU

eCPRI

MEMS Timing Enables Zero Downtime in Fronthaul Switches, ORAN HUB

Problems that MEMS Timing Solves

• Enhance robustness: flexible hitless switching and input monitoring for fail-safe

• Ensure

33

MEMS Clocks

MEMS Timing in Smart NIC for IEEE 1588 and BBU/DU

NICSOC

Elite TCXO

MEMS Clock for NIC

eCPRI

FronthaulSwitch

vBBUServer

RRU

Problems that MEMS Timing Solves

• Enabling IEEE 1588 for PCIe NIC: 1 mm think profile

• Ensure

34

Conclusions

• Time error attributed to 3 noise sources in an oscillator:

1. Temperature sensitivity

2. Aging

3. Wander

• Can’t judge PTP performance by frequency-over-temperature number alone

- Frequency-slope over temperature (dF/dT, ppb/°C) and temperature profile influence max time error

• TCXOs with same frequency-over-temperature stability can have significantly different dF/dT performance, and thus PTP time error

- DualMEMS TCXO architecture is optimized for synchronization applications

- DC-TCXOs enable tuning servo loops with ±5 ppt resolution

• 5G networks additionally require holdover, impacted by aging and wander

• Extend holdover by correcting daily aging, but ultimately limited by wander