Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
© 2017 Ethernet Alliancewww.ethernetalliance.org
April 16, 2020
INTEROPERABILITY – THE FOUNDATION OF ETHERNET SUCCESS
Presenters:Shawn Nicholl, Xilinx – Enabling Interoperability Today for Next-Generation Ethernet SolutionsJeff Twombly, Credo – Interoperability – 400G Beyond – The Case for Active Cables Paul Brooks, Viavi – Driving Interoperability Through Test & MeasurementRay Nering, Cisco – Interoperability – The Foundation of Ethernet Success – A Systems Vendor Perspective
Moderator: David J. Rodgers, Teledyne LeCroy
© 2017 Ethernet Alliance
Presentation Disclaimer
The views being presented in this educational material on the respective IEEE 802.3 standards under
consideration are the views of the author(s), and do NOT represent a formal position or interpretation of the
respective standard by The Ethernet Alliance.
2
© 2017 Ethernet Alliance
The Ethernet AllianceA Global Community of End Users, System Vendors,
Component Suppliers and Academia
3
§ Our Mission• To promote industry awareness, acceptance and advancement
of technology and products based on, or dependent upon, both existing and emerging IEEE 802 Ethernet standards and their management.
• To accelerate industry adoption and remove barriers to market entry by providing a cohesive, market-responsive, industry voice.
• Provide resources to establish and demonstrate multi-vendor interoperability.
© 2017 Ethernet Alliance
Ethernet Alliance Strategy
§ Global Outreacho Cultivate diverse, worldwide membership
§ Technology Advancemento Support and advocacy of standards
developmento PoE Certification Program
§ Thought Leadershipo Annual Ethernet Roadmapo Technology Exploration Forums
4
§ Collaborationo Facilitate interoperability testing eventso Industry Plug Fests supporting member and
technology initiativeso Partnerships with peer organizationso Involvement with SIGs and MSAs
§ Promotion and Marketing of Etherneto Industry media and analyst engagement o Tradeshows and eventso Blogs, white papers, industry articles
Expanding the Ethernet Ecosystem and Supporting Ethernet Development
© 2017 Ethernet Alliance
• Plugfests in the last 12 months– High Speed Networking
• Past Trade Shows – OFC– SC– ECOC– Other
• PoE Certification Program– Gen 1 (.3af / .3at)– Gen 2 (.3bt)
5
The Ethernet Alliance - Investment in Multi-vendor Interoperability
© 2017 Ethernet Alliancewww.ethernetalliance.org
April 16, 2020
ETHERNET INTEROPERABILITY –TEST AND MEASUREMENT CONSIDERATIONS
David J. Rodgers – Teledyne LeCroy PSG
© 2017 Ethernet Alliance
Test and Measurement considerations for High-Speed Ethernet applications
§ Ethernet Standards Evolving at Breakneck Pace• 25GbE to 100GbE, now PAM4 50GbE, 200&400GbE• Soon, 100GbE to 800GbE• Automotive, Industrial, Commercial
§ Ethernet Fabrics Fueling Storage Explosiono Speed and Optimization meeting QOS Expectations
• NVM-oF, iSCSI, FCoE, NFS, IBXoE, FCIP, iSER, iWARP, RoCE, Routable RoCE (v2), and so on, and so on…
§ Standards beget Interoperability?o Interpretation and implementation differences abound
7
© 2017 Ethernet Alliance
Effective Observation
8
Fabric Management Utility/HypervisorTraffic Tap and DPI
(Wireshark®)Line RateAnalysis
© 2017 Ethernet Alliance
“You Can’t Test What You Can’t Measure!”
9
• Hardware Test tools have to change to keep up with market and technology demands
• New Speeds adding new complexities– NRZ to PAM4 signaling
• There is a “protocol” to the Phy– Auto-negotiation– Link Training– FEC
• No two vendors implementations are identical
© 2017 Ethernet Alliance
Test and Measurement – The Bell Weather§ T&M vendors; We’re On the Leading Edge!
• Partners in Pain of “being first”• Traditional – Signal Integrity Tools• New to the Scene – Protocol Specific Tools
§ Purpose Built Protocol Tools!o Compliment to, not replacement for Traditional Tools
• Optimized for the Fabric/Device under testo Becoming essential in all HW/SW test environments
§ The Goal; Testing must be “standardized” and repeatableo Interop PlugFests, 3rd party testing services
10
© 2017 Ethernet Alliance
Key Interoperability Challenges
11
• Identifying Participants– Characterizing Functionality of All Ecosystem Players
• Determining Root Cause– Eliminate “finger pointing”
• Crafting the Solution• Remediation Validation– Test the fix
• Timely Resolution!
© 2017 Ethernet Alliance
• Ubiquitous deployment requires vendor interoperability• Establishing the “link” has become more
complicated than ever• It’s imperative to know what’s “on the wire”– Testing no longer “ends” at the connector
• Integrating legacy and new Ethernet technologies creating new challenges
4/16/2012
Test & Measurement – Navigating the #NextEthernetEra
© 2017 Ethernet Alliance
Test & Measurement – ReCap
13
• Ethernet is a Juggernaut• Content Delivery and Storage Demands are High• Consistent and Predictable Interoperation
is Mandatory• Speed adds Exponential Influences on the
Eco-System• Testing, Testing, Testing!– Tool Sets and Methodologies Must Evolve
© 2017 Ethernet Alliancewww.ethernetalliance.org
April 16, 2020
ENABLING INTEROPERABILITY TODAY FOR NEXT-GENERATION ETHERNET SOLUTIONS
Shawn Nicholl - Xilinx
© 2017 Ethernet Alliance
Programmability for Early Adopters• Early developers of Ethernet solutions are
constantly challenged• Specs are in flux• Proof of Concept development• Interoperability testing
• Programmable platforms (FPGA and ACAP) are the solution• Early implementations without "locking in" the final
solution• Test & Measurement equipment
• Programmability enables compliance testing through evolving standards• Vendors may need to tweak implementations
15
© 2017 Ethernet Alliance
Programmability and Continuous Development
• Recent trend to in-fill Ethernet rates• 25GE, 50GE, 200GE• NBASE-T (2.5GE, 5GE)
• Continuous evolution of products• Evolution of supported Ethernet rates for a port• Evolution of connectivity, form factors (QSFP-DD/OSFP) and break-out• Evolution of applications requiring wire-speed support
• Programmability enables continuous integration• Agile development for products
16
Source: QSFP-DD MSA( http://www.qsfp-dd.com/ )
Source: OSFP MSA( https://osfpmsa.org )
© 2017 Ethernet Alliance
Building Blocks for Interoperability• All network traffic passes over electrical interfaces of
silicon devices• Cutting edge transceivers operate at 56G PAM4 and 112G PAM4
serial signaling rates• Ethernet applications are emerging to take advantage of those
rates, while also expanding to leverage lower rates
• Applications need flexible Serdes• PAM4 and NRZ signaling• Wide range of rates
• Forward Error Correction• PAM4 Bit Error Ratio (BER) demands use of FEC
17
© 2017 Ethernet Alliance
Creating Interoperable Solutions
• Judicious hardening of functions • 16nm devices have 100 Gbps FEC in the Serdes• 7nm devices have 600 Gbps protocol blocks• Area and power efficient
• Platform for interoperability• Develop new Ethernet products• Ensure legacy applications continue to work
• Adaptable engines • Future-proof for emerging protocols• Programmability for applications evolution
18
© 2017 Ethernet Alliance
5G Wireless Application – Radio over Ethernet (RoE)
19
IEEE 1914.3 RoE CPRI
CPRI
eCPRI
eCPRI
IEEE 802.3brFrame Preemption
IEEE 802.1CMTSN
Ethernet Backhaul
© 2017 Ethernet Alliance
Protocol Offload and NFV Acceleration
20
Unencrypted 100GE
Encrypted 100GE
Unencrypted 50GE
Encrypted 25GE
Encrypted 400GE
PCIe to Host
Distributed Network Functions
SecureEthernet
© 2017 Ethernet Alliance
Ethernet Aggregation Application – FEC Variety
21
100G CAUI-4RS(528,514) FEC
200GAUI-4RS(544,514) FEC
25GAUIRS(528,514) FEC
SFICL74 KR FEC
400GAUI-8RS(544,514) FEC
© 2017 Ethernet Alliance
Ethernet-Centric World
• World is becoming Ethernet-dominated• Countless new signaling and port rates• Longevity of legacy rates
• Desire a scalable paradigm• Supportable by everyone - from developers
to testers• Supportable over a range of markets• Ethernet link interoperability and reliability
should be a given for developers
• Testing Automation • Need automation at all levels
22
© 2017 Ethernet Alliance
Test Automation with Programmable Devices
• Pre-hardware simulations• Powerful regressions • Constrained random test• Shared vectors ensure interoperability
• Hardware-based Proof of Concept• Enables early interop and in-system testing• In-house or canned platform
• Ethernet test equipment automation hooks• Script-based controls• Library re-use over multiple generations
23
© 2017 Ethernet Alliance
Standards Development and Interoperability
• Industry must unite to develop interoperable networking standards• Standard and industry bodies like IEEE, OIF, EA, and ETC foster a culture of
openness• Create an improved solution without re-inventing the wheel• Leverage existing solutions for newer market requirements like 800GE
• Strong, well-written standards lead to interoperability• Whole industry benefits from hard work done at the ground floor• Faster bring-ups and less cost wasted in field debug• Companies should become contributors to standards
• Commercial benefits• Common goal and common development focus• Volume improves cost (economies of scale)
24
© 2017 Ethernet Alliance
Tradeshows and Interop events
• Multi-vendor interoperability testing is critical • Customers want the plug-and-play nature of Ethernet
• Ethernet Alliance has taken a leadership role• Live demonstrations at tradeshows • Hot-plug events
• Interoperability events bring many vendors together • Ensure that their equipment talks with each other
25
© 2017 Ethernet Alliance
Summary
• Developers of interoperable Ethernet technologies benefit from programmable platforms• Programmability enables continuous evolution of products• FPGA and ACAP devices provide the building blocks
• In an Ethernet-dominated world, interoperability is supported through:• Efficient test automation• Strong contribution to standards and industry bodies • Live tradeshow demonstrations and hot plug events
• Get involved!
26
© 2017 Ethernet Alliancewww.ethernetalliance.org
April 16, 2020
INTEROPERABILITY –400G BEYOND – THE CASE FOR ACTIVE CABLES
Jeff Twombly – CREDO
© 2017 Ethernet Alliance
Direct Attached Cables – Old Trusty Work Horse
• DAC has been the industry’s favorite– Low cost– Draws no power– Availability
• But, as our industry moves beyond 25Gb NRZ, DACs have become problematic– Performance, interoperability, gauge density, routing challenges,…
28
© 2017 Ethernet Alliance
400G and Beyond - Problems with DACs
• DAC places burden of equalization on the switches– What works on one port may not on the other
• There are established training protocols; however, – “No two vendors implementations are identical”
• Routing challenges for emerging DDC routers and switches– 100’s of fabric connections required
• As transfer rates increase to 800G– Testing and interoperability issues become more problematic
29
© 2017 Ethernet Alliance
400Gb and 800Gb Ethernet Link Budgets & Cu Gauge
30www.ethernetalliance.org
800Gb Ethernet LR Link Budget
400Gb Ethernet LR Link Budget OD (mm)
Scale Cross
Section
Bend Radius (mm)
400G DAC 1m – 30 AWG 2 x 6.7 33.5
400G DAC 2m – 28 AWG 2 x 7.9 39.5
400G DAC 2.5m – 26 AWG 11.4 55.0
OD (mm)
Scale Cross
Section
Bend Radius (mm)
800G DAC 1m – 28 AWG 2 x 6.7 33.5
800G DAC 2m – 26 AWG 11.4 55.0
© 2017 Ethernet Alliance
100G NRZ
400G PAM4 time
Adoption of 400Gb In the Data Center
31
2020 DDC 400G Applications• DR4 Optics for Spine Interconnects• 400G CLOS AEC for Fabric Interconnects
31
64xDD Fabric64xDD Fabric64xDD Fabric64xDD Fabric
32 Ports
32 Ports
32 Ports
32 Ports
32 Ports
32 Ports
32 Ports
32 Ports
128xDDFiber
128xDDFiber
256 –312 Cable CLO
S Fabric
19” Rack
Fibe
r
© 2017 Ethernet Alliance
100Tb Ethernet DDC – Cable Management
32
OD (mm)
Scale Cross
Section
Bend Radius (mm)
400G DAC 1m – 30 AWG 2 x 6.7 33.5
400G DAC 2m – 28 AWG 2 x 7.9 39.5
400G DAC 2.5m – 26 AWG 11.4 55.0
HiWire AEC – 30 AWG 2 x 6.5 32.5
HiWire CLOS <1.5m – 34 AWG 6 24
HiWire CLOS 1.5m–3m – 32 AWG 8 32
• HiWire CLOS AECs consume up to 70% less volume than DACs• HiWire CLOS can be routed with 15cm Cable Managers• DACs require >50cm cable managers
• Can break cages due to weight and stress• Mixed DACs and Optics have shown signal integrity issues
DACs are too thick and bend radius is too largefor 400G the Distributed Disaggregated Chassis Use
64xDD Fabric64xDD Fabric64xDD Fabric64xDD Fabric
32 Ports
32 Ports
32 Ports
32 Ports
32 Ports
32 Ports
32 Ports
32 Ports
128xDDFiber
128xDDFiber
Ingr
ess/
Egre
ss F
iber
64xDD Fabric64xDD Fabric64xDD Fabric64xDD Fabric
32 Ports
32 Ports
32 Ports
32 Ports
32 Ports
32 Ports
32 Ports
32 Ports
128xDDFiber
128xDDFiber
Cable Fabric
128
VerticalCable
Manager19” Rack #1
Fiber/Cu divider
19” Rack #2
Switch Racks stacked next to each other at the T2Cross section of 128 cable fabric going up/down
(256 Total Fabric Cables Required)
Even NIC-TOR – 400G DACs don’t fit well and won’tsee mainstream adoption for 3-4 more years
© 2017 Ethernet Alliance
At 400Gb and 800Gb we need Active Cables• At 400Gb, an active cable (such as a HiWire CLOS AEC):
– Consumes 70% less volume than a passive DAC (1x6mm vs. 1x11.4mm)– Consumes 70% less power than an AOC– Enables routing densities to >500 cables per rack
• At 800Gb, the choice is more stark– Retimed front panel (Transceivers, AOC, AEC)– Switches that support the DAC LR reach will consume 15-20% more power
• For what purpose? Opportunity for ASICs to move to optimized C2M reach & power…– Improbable to route 800Gb DACs at even TOR-NIC densities in a data center
• The time has come to move away from the complexities of managing DAC– Move to fully deterministic and persistent connections– Better TTM and reduced OpEX
33
© 2017 Ethernet Alliance
Active Cable – Testing and Interoperability
• Leverage Ethernet Alliance’s plugfest to ensure interoperability among active cable, switch, server, and test equipment suppliers.
• Leverage industry consortium to ensure simplified application use cases and guarantee plug & play performance.
34
© 2017 Ethernet Alliancewww.ethernetalliance.org
April 16, 2020
DRIVING INTEROPERABILITY THROUGH TEST & MEASUREMENT
Paul Brooks – VIAVI Solutions
© 2017 Ethernet Alliance
Ethernet – Success through interop and flexibility• Ethernet is the dominant solution for high speed interconnect
of communication and computing devices.• With a choice of PMDs to go a few cms over a backplane or
10’s km over fiber• Pluggable optical modules have allowed Ethernet to address
dynamic needs in a flexible ‘pay and adopt’ as you grow approach
• Ensuring the validation, functionality of pluggable optical modules across an open ecosystem has been a key element.
36
© 2017 Ethernet Alliance
Client Pluggable Optics Today
10GECPRI up to 9SFP+
25GECPRI10 SFP28
50GE SFP56
100GE SFP56-DD
100GECFP
100GECFP2
SFP – ‘single lane’
100GECFP4
100GE QSFP28
200GE QSFP56
400GE QSFP56-DD
40GE QSFP+
QSFPx– ‘multi-lane’
• Client optical interfaces are deployed in volumes of millions as pluggable, multi vendor, commoditized products based on robust and clear standards to rates of 400G today.
• The vast majority of the photonic interfaces use direct detect optical links using simple NRZ modulation. From 400G there has been a move to PAM-4 based signalling for both the electrical and optical links.
Standard mechanical form factors and module management and host interfaces allow interchange of modules and ‘pay as you grow’ equipment.
400GE & 800GE OSFP
© 2017 Ethernet Alliance
Inside a 400G QSFP-DD
Double stacked connectorSI issues at 25Gbd PAM-4Mechanical robustness
DSP – host at PAM-4 25Gbd, line side 100G/lambda or ZR coherent
ROSA, TOSA BW, linearity, power, performance
Mechanical integration
Complex module management MIS – real time software to manage events like LOS. Multiple complex parameters like OSNR and PMD to be read-off and checked against alarm levels. Pre-FEC BER to be tracked.
Photonic inter-op & margin
Host AUI-8 inter-op & margin. Equalizer set-up
Thermal – module & system
© 2017 Ethernet Alliance
Critical Areas for Inter-op
Electrical - Host sideOptical - Line side
Data interfaceFEC, eCPRI, Ethernet
Electrical interfacePhy, skew
Management interfaceMIS, I^2C, peek/poke
DSP, controller,
F/W & S/W etc
Photonics
Pluggable module
Mechanical & thermal environment
Optical i/fPMD, optics
© 2017 Ethernet Alliance
Succesful inter-op means an Integrated Environment
The elements required for successful inter-op must be validated together. The close interaction of host (power/thermal/mechanical, data & phy/electrical and the module management are key to the open ecosystem. Also critical is the inter-operability of the optical side. With the emergence of pluggable coherent modules, especially the expected wide scale take up of ZR modules at 400G will cause addition challenges.
PowerCoolingPHY layerEthernet & PCS MISTriggersFECPhotonic
© 2017 Ethernet Alliance
Legacy or State of the art?• A modern host has to consistently
manage and support a wide range of potential clients.
• Take a QSFP-DD slot in a network element– QSFP+ 40G– QSFP28 – 100G– QSFP56 – 200G– QSFP-DD – 400G – Optical or DAC– SFF or CMIS– I^2C speed & termination– Breakout or ?
• A modern host must be able to manage this in a dynamic and consistent manner
• Consistent behaviour under fault conditions like LOS?
41
• What type of module?
• SFF or CMIS• Supported modes• AUI set-up• LOS• Traffic• power
© 2017 Ethernet Alliance
Summary - Ethernet Interop • Pluggable optics give Ethernet the powerful ability to scale and
grow with the right reach (PMD) for each application– From copper DAC to ZR optics for 80km+
• Pluggable optics have evolved significantly from the simple E/O and O/E converters of past generations to highly integrated and complex systems with demanding management and control as well as high speed electrical & photonic interfaces
• All of these need to be orchestrated together to continue the open ‘plug, play & grow’ ecosystem that has made Ethernet the success it is.
42
© 2017 Ethernet Alliancewww.ethernetalliance.org
April 16, 2020
INTEROPERABILITY – THE FOUNDATION OF ETHERNET SUCCESS, A SYSTEMS VENDOR PERSPECTIVE
Ray Nering – Cisco Systems
© 2017 Ethernet Alliance
Industry Relies on Ethernet Interoperability
• Cisco has relied on Ethernet requirements since it first product in 1986
• Today, Cisco’s product line is broad, diverse and still relies on Ethernet requirements
• Any product must be able to interoperate with other vendors to be a relevant player in the market
• Ethernet standards, industry standards organizations (IEEE, etc), MSAs and industry collaboration are the key to maintaining an ecosystem that promotes interoperability
• At 100GE and below Interop wasn’t trivial• PAM4 technology, everything is new: Optics, SERDES, CMIS, form factors, standards
© 2017 Ethernet Alliance
Optical Signaling (module <-> module)Electrical Signaling (module<-> line-card High Speed)
Control Plane (module<-> linecard Low Speed)
Successful implementation on 1-100 GbE client optics, i.e. Switch to Switch, Switch to Router interfacesGenerally within same buiding, campus or data center (<10 km)
Standards (IEEE, MSA, ) try to guarantee interop and uniformity
Pluggable Optics are GREAT
© 2017 Ethernet Alliance
100GbE Pluggable Optic Interoperation • Industry has been releasing platforms w/ 100GbE ports since 2010 timeframe• OEMs have been producing platforms w/ 100GbE ports since 2010
• They have a range of 100G port form factors and all must interoperate optically for all time• CFP, CFP2, CPAK, CFP4, QSFP28 and QSFP-DD…
• A CFP 100BASE-LR4 shipped in 2011 must be able to interoperate with all other 100GBASE-LR4 modules no matter the vendor or the form factor
• Customer may buy a 100G-LR4 and it should work in any platform w/ a compatible port
© 2017 Ethernet Alliance
1, 10, 40 GbE
100 GbE
Equalization on electrical channel
Fixed or adaptive Equalization
Low power management
CDR
Laser turn on
Diagnostic monitoring
Optical and environmental monitoring
Provisioning based on state machine
100GbE was the beginning of more complex host/module interactions
25Gbit/s I/O 3.5 W power
CDR Strong Equal. @ Rx
Complex interactions
with Host
© 2017 Ethernet Alliance
What happens if……Vendor A does not test full compliance and has little margin on equalization settings
On this unit, no errors
On this unit, CRC errors for no apparent reason
Error free area guardband
© 2017 Ethernet Alliance
What happens if……Vendor does test full compiance and has plenty of margin
On this unit, no errors
On this unit, no errors
Error free area guardband
© 2017 Ethernet Alliance
What happens if……
Vendor A has limited acceptance for fast electrical signals from linecard
Error free only in this condition: performance depended on port used
Error free wide range: perfomance consistent across all ports
Vendor B has wide acceptance for fast electrical signals from linecard
© 2017 Ethernet Alliance
Impact on the Transition to PAM4 • Industry turned to PAM4 to increase data rate beyond 25Gb/s• PAM4 is much more sensitive to:
• Noise• Reflections• Non-linearities• Baseline wander
• Receiver design is much more complicated• IEEE has done a great job to embrace PAM4 with standards• 802.3bs/cd/ck/cm/cn/ct/cw/cp/cu….• There will be improvements as we learn more
© 2017 Ethernet Alliance
400 GbE Increases Complexity Further 100 GbE (LR4) 400 GbE (FR4)
IDEAL
MEASURED
POST DSP EQUALIZER
• 400G PAM4 requires DSP and strong equalizations to enable client optics.
• DSPs are not as powerful (and power-hungry) as the ones used on line-side optics.
• Yet interop between different DSPs is a critical item
© 2017 Ethernet Alliance
Tx A Tx B Tx C Tx A Tx B Tx CTx A Tx B Tx C
Rx A Rx B Rx C
400G DSP interop• Interop testing over three 400G vendors• Vendor A and C use same DSP• Vendor B uses different DSP • Tests identified significant issues with
interoperating with anything but itself
© 2017 Ethernet Alliance
Best practice: module level testingOptical
Interoperability
Electrical
Control
• TX• RX
• TX• RX
• Alarms• Monitoring (DOM)
Across full range of
Temperature Voltage DC noise
© 2017 Ethernet Alliance
Best practice: system level testingSoftware Interactions
Module & Fiber Plug /Remove
Power Cycle
Shut/No Shut
Across full range of
Temperature Voltage Platforms
© 2017 Ethernet Alliance
Tier 1 optics supplier
Tier 2 optics supplier
Bareboneoptics
supplier
Basic optical compliance
Full optical compliance
Test over worst case fibers
Full electrical compliance
Basic Monitoring and control compl.
Full Monitoring and control compliance
Full testing/regressionover MCN
Interoperability with other optics
System level testing
Test across all T/V corners
Coverage
Key takeaways
Critical features for seamless interop between modules
on different platform
Faster speeds = More complexity
More sensistive linecards/module interactionsBest
Practice
© 2017 Ethernet Alliance
Conclusion• Ethernet interoperability is literally the foundation of
networking today• Lower speeds (>100GbE, NRZ) interop was straight forward• Higher speeds (50G+) w/ PAM4 signaling interop has become
much more complicated• Interactions between optics, platforms and SW are critical • Corner testing not only at the optical level but also across
platforms & SW are critical for robust interoperability
57
© 2017 Ethernet Alliance
If you have any questions or comments, please email [email protected]
Visit the Ethernet Alliance on Facebook
Ethernet Alliance: visit www.ethernetalliance.orgJoin the Ethernet Alliance LinkedIn groupFollow @EthernetAllianc on Twitter
www.ethernetalliance.org 58