Network Synchronization: IEEE1588’s Future in Computing and the Data Center Dan Biederman ASIC Architect/Designer March 2, 2017

Legal Notices and Disclaimers Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer.

No computer system can be absolutely secure.

Some results have been estimated, and provided to you for informational purposes. Any differences in your system hardware, software or configuration may affect your actual performance.

Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit http://www.intel.com/performance.

Intel, the Intel logo and others are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others.

© 2017 Intel Corporation.

Agenda •  Introduction •  History of Ethernet Synchronization •  Basics of IEEE1588, and Sync-E •  Other Synchronization technologies •  Common current use cases for Synchronization •  Futuristic Data Center usages.

3

Introduction – Who is Dan Biederman •  Worked on IEEE1588 at Cisco*, Violin Memory*, Ericsson*, and now Intel •  Several IEEE1588 related patents and pending patents •  First product was Cisco’s IE3000*

•  Development started between IEEE1588v1 and IEEE1588v2 •  IEEE1588v2 on 10/100/1000 Ethernet links •  Worked on most stages of the project

•  Architecture, FPGA design, lab testing and sales/customer support. •  Collaborated with Cisco’s* internal community on IEEE1588

•  Industrial, Data Center, Financial, Metro Ethernet, Energy/Smart Grid, etc

•  Performed Synchronization related work at Violin Memory* and Ericsson* •  Expecting great things to happen at Intel.

4

Ethernet Synchronization History

5

NTP - ~1980 IEN173/RFC778 Internet Clock Service

Source: ntp.org, ITU, IEEE, etc. Anything else for TSN to be added?

IEEE1588-2002 Industrial Automation

IEEE1588-2008 Telecom

IEEE1588-20xx

NTPv1 - 1988 RFC1059

NTPv2 - 1992 RFC1119/1305

NTPv3 - 1994 RFC1589

SNTPv4 - 1994 RFC2030

NTPv4 API - 1999 RFC2783

Sync-E ITU-T G.8261 Architecture

ITU-T G.8262 Clocking 2007-2010

ITU–T G.8264 ESMC

IEEE802.3 A/V Streaming Moved to 802.1AS (AVB – 802.11)

AVNU Alliance* 2009

AVB now TSN 2012

Computer

BASICs Of IEEE1588 and Sync-E

6

Accuracy & Precision

7

Not accurate and not precise Accurate and not precise

Precise, but not accurate Accurate and precise

Accuracy & Precision

8

Four graphs show samples of measurements of a slave time compared to a masters time. The center line is the ideal masters time. Black Line is the Master’s time Blue Squares are samples of the Slave’s time.

Not Accurate or Precise More Accurate,

Not Precise Precise but not Accurate Accurate

and Precise

Basics of IEEE1588 Precise Time Alignment • Match slave time to the master’s time. Hardware Timestamps • Time as close to the wire as possible Time Arrival and Departure times • Use by slave to align to the master • Example:

•  Link Delay to Slave is T2-T1 •  Link Delay to Master is T4-T3 •  Link Delay is [(T2-T1) + (T4-T3)]/2

One Step sends T1 with Sync packet All future sync packets can use that delay to track the master clock

9

Sync

Follow-up(T1)

Delay_Request

Delay_Response(T4)

T1

T2

T3

T4

Master Slave

Sync

Follow-up(Tn)

Tn

Tn+delay

Basics of Synchronous Ethernet Match frequencies across the network • Ideally 0 Parts Per Million (PPM) Allows Ethernet Network to interact with SONET and SDN (TDM) networks • Ethernet PPM = +/-100 ppm • SDH PPM = +/- 4.6 ppm Frequency Alignment • Recover the clock from one port • Use that clock locally • Pass the clock to neighbors • All device in a SyncE network are using the same exact clock frequency.

10

Master

Recover Use Internal

Transmit

Recover Use Internal

Transmit

Other Synchronization technologies

11

Many synchronization Technologies

12

Satellite/GPS •  Telecom Networks (SDN ->Ethernet) •  Mobile Backhaul •  Military Applications

•  GPS controlled by US Navy

CERN White Rabbit* •  Sub-ns accuracy •  SYNC-E •  IEEE1588-20xx (v3)

Quantum/Atomic Clocks •  Very Precise •  Very Expensive

Ytterbium Clock: Photo from NIST* – N. Phillips

Image from CERN*

Many Synchronization Technologies Synchronization over WIFI •  802.11 RevMC (802.1AS) Precision Time Management •  Precision Time Over PCIe Sync over USB Latency and Congestion estimation •  ITU Y.1731 •  TWAMP (OAM) Many Proprietary Synchronization Technologies •  NextNav's MBS timing service for GPS-challenged areas - indoor

and urban canyon*

13

Common Current USE CASES for Synchronization

14

IEEE1588 – Industrial Ethernet & Energy The original purpose of IEEE1588 was for Industrial Automation • Machines operating precisely based on the same time reference

Conveyer Belt Example: • At Time T1, Operation A occurs • At Time T2, Operation B occurs • At Time T3, Operation C occurs • At Time T4, Operation D occurs Smart Grid / Energy: • At a precise time, energy sources can start and stop.

15

T1 T2 T3 T4

A B C D

IEEE1588 + Sync-E •  Synchronous Ethernet was added to allow the

frequency to be shared between devices. •  Telecom/Metro Ethernet •  Allowed legacy SDN and Sonet networks to

be merged with Ethernet network •  Mobile Backhaul (Cellular Network Equipment) •  Used GPS for synchronization •  US Navy controls GPS •  With both IEEE1588 and Sync-E:

•  Accurate networks can be created •  Accurate and precise time can be created.

16

Other Markets Financial - Trading times - Timing latency to the trading floor First to sell/first to buy get the best price. -  MiFID II requirement: 100us with respect to UTC.

17

Data Center Map traffic •  Identify congestion area •  Create Heat Maps •  Reroute traffic for lowest latency

Futuristic Data Center usages

18

Synchronization for Software Defined Networking (SDN) SDN is "The physical separation of the network control plane from the forwarding plane, and where a control plane controls several devices."(https://www.opennetworking.org/sdn-resources/sdn-definition)

The control plane that controls several devices can be implemented with an Intel Server or group of servers.

The operation of the SDN controller could be Time-Division Multiplexed (TDM) where operations occur at precise times and for precise durations synchronized across multiple devices.

Controlled device can switch their tables at precise and accurate times.

Access can be permitted at precise and accurate times

Precise analytics across the SDN controller and the controlled devices.

19

Servers

Synchronization and Network Function Virtualization (NFV) Network Function Virtualization, NFV, is an effort to make network services virtualized. These network services can then be implemented on Intel servers. These servers could then implement a specific function as processes or threads on their CPUs, or use Virtualization(Virtual Machines) that can be spawned as needed. Precisely and accurately start/stop/operate different virtualized network functions Accurate and Precise analytics across the Network Functions. These analytics can be used to modify when the network functions are run.

20

VM

VM

VM

VM

WAN

IEEE1588 – Industrial Ethernet and CPUs The original purpose of IEEE1588 was for Industrial Automation • machines operating precisely based on the same time reference

Just-in-Time Processing. • JIT Manufacturing is where the manufacture receives the parts just in time to manufacture a device (like a car).

• Just-in-Time/Time-Based Scheduling allow the data to arrive at the appropriate time for optimal processing.

• Precisely and Accurately place the data where it is required, just before it is needed.

21

T1 T2 T3 T4

D4 D3 D2 D1

I4 I3 I2 I1

T4 T3 T2 T1 Processing Complex

A B C D

Synchronization and AI Artificial Intelligence, Machine Learning and Neural Networks Artificial intelligence, Machine Learning, and Neural Networks are hot topics in the Data Center

Two of the main tools of AI •  Inference Engine (rules and deduction) • Knowledge Base (An understanding of the problem).

Neural Networks need to be trained to learn

22

D4 D3 D2 D1

T4 T3 T2 T1 Processing Complex

D4 D3 D2

T7 T6 T5

Using Precise and Accurate timing, the processing complex can be loaded with the images and associate data Just-In-Time for processing

When the processing starts, the data is available

In the example below, times are know when each image will be processed, so it can be loaded prior to processing

If Precise and Accurate time is used, portions of the image could be loaded at a rate optimized for a given algorithm to allow for Just-In-Time processing.

Synchronization and the Internet of Things

An autonomous car is trying to get route information for a trip from the house to the school There are two routes that can be taken at Intersection B.

The car needs to determine if it should go straight or turn It can query the internet of things on their status • When will the light be green for the path interested •  Is there a person in the crosswalk •  Is there a train approaching From this information, and the knowledge of the rate of speed of the car, decisions can be made to give the car the fastest path Late information is useless. Precise and accurate time between the devices (car, stop lights, cross walk, etc…) can help determine how quickly to respond with the latest information

23

B

North

East

Synchronization and the Internet of Things The street lights change their light patterns to match the cars and speed rates of traffic This could allow most or all of the cars to get to their destination in optimal time. The crosswalk decision could be delayed to allow the optimal amount of traffic to pass. Traffic engineers may make lights more red for speeders than “good’ drivers. Likewise, if you know a car is speeding towards a red light, it can warn other car.

24

Synchronization and Virtual Reality •  In Virtual Reality, many media streams

need to be synchronized. •  Audio •  Video •  Metadata/Captions

25

Example: In sports, camera’s might have a 360 degree view from the player. The videos and audios can be precisely and accurately timed and then stored. The media can be stitched back together based on time, and played back to a fan.

Synchronization and FPGAs in the Data Center

•  FPGAs have been doing Precise and Accurate timing for many years.

•  Some of the first IEEE1588 design used an FPGA to intercept the signals between the Ethernet MAC and PHY.

•  Currently many data center providers are adding FPGAs to their computing systems

•  Such FPGAs can be optimized for Artificial Intelligence, Neural Networks, and Machine Learning.

26

Add FPGA Picture.

Ethernet+1588 Driver w/ 1588 PTP API

IEEE 1588 hardware IP and software solution

Source of the IP

TSE/ 1G/10/25/40/100Gb Ethernet MAC & PHY + 1588 Time Stamp

PTP Packet Parser and Filter

Intel’s FPGA 1588 Hardware IP

Customer / Intel Partner / Intel

TCP/IP/UDP Stack (Optional)

Customer / Intel Partner / OS Provider

1588 ToD Clock Generator, Freq Synthesizer, PLL

Timing Servo Control Software

1588 Precision Time Protocol (PTP) stack

1588 ToD Clk & Freq Synth Driver

Software

Hardware IP

Intel FPGA provide high accuracy sub-5ns IEEE 1588-v2 time stamping in Ethernet IP.

Data Plane Development Kit - DPDK DPDK 2.2 added a sample application for an IEEE1588 PTP client. What does it do? •  DPDK IEEE1588 API communicates with a PTP master clock •  Synchronize the time on the NIC •  Optionally, synchronize time on the Linux system. Why? •  It shows how time on the NIC and on the Linux kernel can be

synchronized with a master clock.

27

Summary •  Reviewed 2 Types of Network Synchronization •  IEEE1588 – Align a devices time to a masters time •  Synchronous Ethernet – Match frequencies across the network •  Together IEEE1588 and Synchronous Ethernet can give a very

precise and accurate time. •  We reviewed original use cases like Industrial Ethernet •  We reviewed other current and future use cases •  One final comment: “Precise and accurate time is a platform for many great things.”

28

References References:

Intel NICs that support Synchronization/IEEE1588:

https://communities.intel.com/thread/87751

http://www.intel.ph/content/dam/doc/product-brief/ethernet-i340-server-adapter-brief.pdf

http://www.intel.com/content/dam/www/public/us/en/documents/datasheets/xl710-10-40-controller-datasheet.pdf

Q: Is there an Intel chip/demo board, where they can play around with Synchronization?

A: It’s possible, on Linux, to use github/OpenAvnu/gPTPd (which uses PHC driver / sys_ptp_offset_precise to measure the relationship between PTP and CLOCK_REALTIME, for example) on SkyLake client systems (i3/i5/i7), using the integrated Ethernet.

29

Q&A

30