View
10
Download
2
Category
Tags:
Preview:
DESCRIPTION
Ethernet technology has emerged as a cost-effective, mature, robust, high-speed, & versatile choice for MAN/WAN networking of critical defense establishments and military installations – for e.g., army, navy, & air force bases, mission commands, remote war centers, the Pentagon, and other security agencies. Intelligent Ethernet helps to achieve IP-centric service requirements, while...
Citation preview
11
Understanding Intelligent Military-Grade Optical Ethernet Networks:
A Versatile Solution for Achieving DoD’s Net-Centric Operations Strategy
Vishal Sharma, Ph.D.Principal Technologist &
ConsultantMetanoia, Inc.
vsharma@metanoia-inc.com650-641-0082 (p)/650-641-0086 (f)
Shahram Davari, MASc.Associate Technical Director,
Network SwitchingBroadcom, Inc.
davari@broadcom.com408-972-7436 (p)
Understanding Understanding Intelligent Military-Intelligent Military-Grade Optical Ethernet NetworksGrade Optical Ethernet Networks::
A Versatile Solution for Achieving DoDA Versatile Solution for Achieving DoD’’s Net-s Net-Centric Operations StrategyCentric Operations Strategy
© Copyright 2010All Rights Reserved
Shahram Davari, MASc.Associate Technical Director, Network SwitchingBroadcom, Inc.davari@broadcom.com408-972-7436 (p)
Metanoia, Inc.Critical Systems Thinking™
Vishal Sharma, Ph.D.Principal Technologist & ConsultantMetanoia, Inc. vsharma@metanoia-inc.com650-641-0082 (p)/650-641-0086 (f)
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 3
What We Will Discuss in This Tutorial
Elements of DoD’s Net-Centric Data Strategy – key attributes and goals
Requirements and Attributes of Military-Grade Networks
Implications of the Above for Underlying Technology System Architecture and Features Network Architecture and Design
Why Discuss Ethernet? Its Benefits and Applications
Optical Ethernet
3 Roles of Ethernet – Service, Transport, and PHY Carrier Ethernet and Optical Ethernet
Macro-Architectural Options for Building MAN/WAN Interconnects & Key Operational Principles
Key Developments Valuable for Military Adoption of Optical Ethernet
How Optical Ethernet Technology meets the Initial Requirements
Attributes and Goals of DoDAttributes and Goals of DoD’’s s Net-Centric Data StrategyNet-Centric Data Strategy
Metanoia, Inc.Critical Systems Thinking™
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 5
Core Elements of DoDs Net-Centric Operations/Data Strategy (NCDS)
Proactively Collect User-Feedback for Improvements
Key Attributes of DoDs Net-Centric Data
Strategy
Handle Info. only Once for Efficiency Visibility to a Wide Audience
Rich, Descriptive Meta-data for Understandability
Shared-spaces for Posting and Efficient Access
Post-and-Process in Parallel
Rapid & Precise Discovery of Data
Facilitate Repurposing – Separate Data from Applications
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 6
Strategic Goals of DoD’s NCDS
- Establish procedures & policies for effective data sharing
- Embed data-sharing precepts in the organization
Communities of Interest
Strategic Goals of the Net-
Centric Data Strategy
InstitutionalizedVisible
Understandable
Trusted
Interoperable
- Data is trustworthy
- Data integrity & quality is assured by backing of a reliable organization/authority
- Make meaning & purpose of data clear via use of meta-data
- Discoverable
- Facilitate interaction with data for analysis and decision-making insight
- De-centralize data management to dynamically formed user groups
- Allow prioritization/collaboration on data, based on immediate operational needs
- Furnish infrastructure for self-synchronization
- Shareability of data, while preserving accuracy, integrity, usability
- Understandability via semantic and structural meta-data
Responsive
- React to fulfill user needs
- Satisfy needs relative to performance, content coverage & quality
Accessible
- Ease of reaching data location
- # of users who can consume data
Military-Grade Networks: Military-Grade Networks: Requirements & AttributesRequirements & Attributes
Metanoia, Inc.Critical Systems Thinking™
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 8
Key Requirements of Military-Grade Networks
- Accommodate multiple access technologies/media – copper, fiber, coax, TDM, satellite, wireless
- Uniformly aggregate traffic onto the metro/core network
Simultaneous Support of Legacy & Advanced Services
Military-Grade Networks:
Requirements
Rugged
ReliableManageable
Highly Available
- Controllable delay, jitter, and loss
- Flexible bandwidth distribution across diverse users/applications
- Resilient to failures
- Ability to recover automatically in min. time
- Hardened for harsh environments – extreme weather, demanding conditions
- Need to operate in constrained spaces
- OAM capability
- Ability to control network elements & resources
Diverse Last-Mile Access
- Uptime: 99.9999% or more
- Fast error detection and recovery
Secure
- Reliable, uncorrupted data
- Tamper-resistant, high-integrity data
Hard QoS & Determinism
- Support legacy voice, POTS, low-speed satellite backhaul links
- In parallel, allow for rich, multi-media traffic, video commn, sensor data
Fast Connection Setup
- For dynamic and quick call setup
- Setting up commun. over underlying infrastructure
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 9
Key Requirements of Military-Grade Networks
- Accommodate multiple access technologies/media – copper, fiber, coax, TDM, satellite, wireless
- Uniformly aggregate traffic onto the metro/core network
Simultaneous Support of Legacy & Advanced Services
Military-Grade Networks:
Requirements
Rugged
ReliableManageable
Highly Available
- Controllable delay, jitter, and loss
- Flexible bandwidth distribution across diverse users/applications
- Resilient to failures
- Ability to recover automatically in min. time
- Hardened for harsh environments – extreme weather, demanding conditions
- Need to operate in constrained spaces
- OAM capability
- Ability to control network elements & resources
Diverse Last-Mile Access
- Uptime: 99.9999% or more
- Fast error detection and recovery
Secure
- Reliable, uncorrupted data
- Tamper-resistant, high-integrity data
Hard QoS & Determinism
- Support legacy voice, POTS, low-speed satellite backhaul links
- In parallel, allow for rich, multi-media traffic, video commn, sensor data
Fast Connection Setup
- For dynamic and quick call setup
- Setting up commun. over underlying infrastructure
Implications for Technology, and Implications for Technology, and System & Network ArchitecturesSystem & Network Architectures
Metanoia, Inc.Critical Systems Thinking™
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 11
Implications of NCDS Requirements (1)
Implications for: Technology System Design Network Architecture Property
1 Scalability (# locations, # users)
- Large address space to support many end-nodes - Capability to create hierarchy - Control Plane for discovery & topology learning
- Large memory/processing for address & routing tables - Capacity for large # of tunnels
- Accommodate many end-nodes - Hierarchical design & traffic engineering - Support wide geographic reach, seamless across access, metro, core
2 Security (data integrity, trust)
- E2e, segment, and/or Link layer (local) security - Isolate different users or user classes - Enable detection of breaches
- Support encryption, authentication, ACLs - DPI on line cards - User data isolation - Intelligent memory partitioning across users/functions - Provision against DoS/security attacks
- Admission control - Authentication - Architecture that integrates firewalls, appliances with DPI
3 Manageability (of network and data)
- Provide robust OAM tools - Management interface & protocols (e.g. ELMI)
- Support OAM tools/mechanisms - Permit remote access & mgt.
- Allow for Out-of-band (OOB) control - Support a data communication network (DCN)
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 12
Implications of NCDS Requirements (2)
Implications for: Technology System Design Network Architecture Property
4
Dynamic setup and control of communications (within & across COIs)
- Signaling - Dynamic/static tunnel setup - NMS configuration features
- Discovery - Signaling, CP features - Dynamic joining of mcast groups (e.g. IGMP)
- OOB network for signaling (if needed)
5
Native mp2mp, p2mp communication (for many-to-many xchanges, multicast)
- Native broadcast, multicast capability - Mcast signaling & QoS
- System-level brdcast, mcast with intelligent replication - Multicast signaling support - mcast group creation/deletion
- Strategic placement of servers (close to consumers) - Redundancy of data (servers) - Support redundant & disjoint network paths
6
High-Speed at low cost (rapid communication)
- Support multiple i/f speeds - Allow link bundling to enable higher speeds - Have standards for evolving speeds - Backward compatibility with earlier i/fs
- Large fabrics - Versatile, dense line cards - High-rate processing - Low power consumption
- High-speed links -- fiber - Support WDM - Enable link aggregation
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 13
Implications of Military-Grade Network Requirements (1)
Implications for: Technology System Design Network Architecture Property
1 Rugged
- Ubiquitous, with wide reach and minimal constraints - Delivarable over robust media, e.g. fiber
- Robust conduction cooling - Intelligent use of CPUs - Off-load complex processing -- security, protocols -- to central entity or add-on
- Built with robust media E.g. fiber -- inert, free from EMI/EFI
2 Secure
- Standards for encryption, security that are widely accepted/realizable, available - Tunnel user data in real/virtual tunnels to effect isolation - Raise alarm/signal when data is tampered with
- Data plane and control plane robust to DDoS - Apply hardware-based encryption - Isolate users via memory partitioning, queue mgt., tunnels to minimize data impact
- Network and overlay mgt. architecture must resist hacking/tampering - Have rapid alarm propagation
3 Reliable
- Stds for signaling -- for restoration - Setup & control multiple paths via signaling/NMS - Detect/react to faults, mis-routed data - OAM capabilities such as: connectivity check, loopback, link trace
- Hardware/software redundancy - e.g. LCs, fabrics, power supplies - Software redundancy - NSF, NSR, hitless upgrades - Ability to detect/react to failures
- Support topologies supporting redundancy in data routing - Dual-homing, link aggregation (e.g. LAG), multipath (e.g. ECMP) support - 1+1, 1:1, 1:N, ring, mesh protection
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 14
Implications of Military-Grade Network Requirements (2)
Implications for: Technology System Design Network Architecture
Property
4 Hard QoS + Determinism
- Support virtualization of network b/w (e.g. via tunnels, VLANs) - Ability (in technology, e.g. pkt hdrs) to mark, seggregate, prioritize, aggregate traffic - Support perf. measurement OAM
- Traffic isolation via queues, scheduling - Separate tables/memories to segregate traffic of different priorities, classes, apps. - Signal tunnels, and control/ manage tunnels
- Support provisioning and dimensioning - CAC to regulate traffic vols. - Traffic engineering to support traffic placement
5 Manageable
- Management constructs for config, monitoring - Measure loss, delay - Have loopback, link trace, continuity check (e.g. Y1731)
- Control access to/sharing of system resources between different user types - Create/config policy - Gather stats, diagnose problems
- Support remote config. & monitoring - OOB or in-band DCN - Hierarchical design
6 Available
- Error detection & config of multiple alarms - Multipath routing/switching - Rapid switchover on failure
- Fast error detection at L1/L2/L3 - Detect h/w, s/w errors - L1/L2/L3 integration for fault alarming - Hardware/software features to aid redundancy
- Support alternate routes/paths - Architecture to enable rapid recovery from failures (meshy-architecture) - Support intelligent/flexible multi-layer protection
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 15
Implications of Military-Grade Network Requirements (3)
Implications for: Technology System Design Network Architecture Property
7 Diverse Last-Mile Access
- High-speed, cheap, easily upgradable - Simple management or unmanaged - Support aggregation of traffic, while keeping different traffic types/classes seggregated
- Multi-service capable to support variety of interfaces -- TDM, ATM, FR, IP, EPON/GPON -- and protocols - Support vast range of data rates - Ability to aggregate traffic (Appropriate processing in h/w and s/w Ability to queue & route data appropriately)
- Intelligent interworking (type, #, placement of devices) - Provide for aggregation points/on-ramps for termination of diverse traffic and transfer to a common (Ethernet, IP/MPLS) core - Enable clock distribution
8 Support Legacy & Advanced Services
- Advanced security mechanisms - Smart OAM - Virtual partitioning of network resources (for communities) - Scalable multicasting - Sophisticated security mechs.
- Support VPNs to facilitate COIs - Support ckt emulation, clock distribution - Advanced protection/restoration - Ablility to introduce new services by minimal system upgrades (e.g. just add/modify one LC)
- Architect for incremental introduction of advanced services
9 Low Cost
- Uses technologies with mass adoption in non-military setting (e.g. Ethernet, IP, MPLS) - Benefit from operational experience, cost reductions - Use technologies with accumulated deployment experience
- Leverage COTS - Use std. building blocks/sub-systems, components to benefit from adoption of vol. components (or hardened variants)
- Versatile arch. -- uses technologies optimized per segment - Intelligent policy enforcement (via policy servers) - Plug and play operation - Powerful signaling and control
Why Discuss Ethernet?Why Discuss Ethernet?ItIt’’s Benefits and Applicationss Benefits and Applications
Metanoia, Inc.Critical Systems Thinking™
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 17
Why Ethernet? Some Key Benefits … Mature technology
3 decades of operational experience, ~300M+ ports sold in 2008 alone!
Low-cost Mass usage lowers cost, so
compelling to use wherever possible
High-bit rates & Range of speeds 10 Mbps to 10 Gbps! (40-100 Gbps
underway) – 3 orders of magnitude
Versatile Usable as service, transport, PHY More discussion of this ahead ...
Provides consistent technology from edge-to-core Extends reach from LANMAN WAN
Solves both networking & real-time interconnect needs in military environ.
Native support of IP Imp. for GIG and net-centric warfare Simple IP address management
Self-replacement capability Largely backward compatible Easy upgrades, integration of legacy
systems
Widely available COTS ecosystem Easy to adapt commercial h/w & s/w
for military use
Ethernet expertise widely available Network design, planning, architecture Network engineering, troubleshooting
Practically unlimited interoperability
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 18
Representative Applications of Ethernet in the Military Switched Ethernet operates as:
Networking infrastructure for MAN/WAN Real-time fabric interconnect in military systems, warfare systems, & military installments Critical building block for military devices
1-10 Gb/s Ethernet used as “fat-pipe” between sub-systems
Intelligent Ethernet transport adopted for: Support of IP-centric service requirements Evolution of wireless & fixed-line infrastructures Explicitly defined native Ethernet connections w/ reserved resources, dedicated protection
Multi-layer Ethernet switches employed in support of DoD plans to leverage IPv6
Ethernet technology facilitates delivery of: Real-time imaging, sensor data, video Secure mission-critical defense communication
Utilized for furnishing precision timing & sub-microsecond synchronization
USS Ronald Reagan
AH-64 ApacheAH-64 Apache
Optical Ethernet Explained: Optical Ethernet Explained: Three Roles and Its Three Roles and Its
CharacteristicsCharacteristics
Metanoia, Inc.Critical Systems Thinking™
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 20
Versatile Packet Networking with Ethernet
Ethernet technology can play one of three roles in a data network
NetworkComponent
MEF
Standards Organization
Involved
Transport
Service
PHY
Technology/Standard
Carrier Ethernet
IETF
IEEE
ITU-T
MPLS-TP
PBB, PBB-TE
OTN-transport part
IEEE
ITU-T
1GE/10GE/100GE
OTN-PHY part
Ethernet Service – offered to end-customer, runs e2e, where traffic flow into/out of customer systems comprises Ethernet frames
Ethernet Transport - Ability to switch/route Ethernet frames of an Ethernet service, b/ween network nodes by setting up connection-oriented, traffic engineered paths in the network with deterministic perf.
Ethernet PHY – framing and timing of actual bits of the Ethernet frame, and their TX over the physical medium to connect switches at the physical layer
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 21
A Word on Connection-Oriented Ethernet (COE)
Ethernet transport enables the realization of COE
COE – set of control-plane protocols & data-plane settings that create a connection-oriented capability to transfer Eth frames
Ethernet transport could involve:
L2 transport -- Switching/routing traffic (data frames) by
Enhancing Ethernet technology – e.g. PBB-TE (802.1aq)
Using a different technology – e.g. MPLS, MPLS-TP
L1 transport – switching/routing traffic at the physical layer (e.g. optical channel data unit (ODU) level) by
Embedding in a transport networking layer, such as ITU-T’s G.709 OTN
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 22
Optical Ethernet Network Defined Network spanning a MAN/WAN that offers a carrier-grade Ethernet service,
running on a COE transport infrastructure over an optical PHY Optical PHY: OTN’s optical channel or an Ethernet PHY over optics Can be muxed onto fiber using CWDM/DWDM
L2 Transport
Service
L0 PHY
Carrier Ethernet (E-line, E-LAN, E-Tree)
Packet Transport (PBB-TE, MPLS-TP)
SONET/SDH, OTN transport
OTN-PHY partIEEE-Ethernet PHY)
L1 Transport
“Optical Ethernet” Layers
Technology Examples
For p2p services
For p2p or mp2mp services
Relationship of the Layers andtheir corresponding entities
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 23
OK, So What is Carrier Ethernet?
Carrier Ethernet is therefore the service component of optical Ethernet networks
Courtesy: Metro Ethernet Forum
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 24
How Optical Ethernet Relates to Carrier Ethernet
Carrier Ethernet: defined by MEF in 2004-05 as “Ubiquitous carrier-grade Ethernet service with 5 attributes”: Standardized Services (better thought of as building blocks)
Uniformly defined core services, building blocks for applications
E-line, E-LAN, E-Tree (illustrated ahead)
Scalability Span local, access, national, global range, with millions of MACs & UNIs
Reliability Detect & recover from errors/faults, without impacting customers
Hard QoS E2e performance for loss, delay, jitter, and b/w matching requirements of
voice, video, data traffic over heterogeneous networks
Service Management Robust, standards-based, vendor-independent OAM to monitor, diagnose,
manage networks offering Carrier Ethernet service
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 25
MEF’s Service Definitions or Building Blocks
MEF building blocks defined in terms of Ethernet Virtual Connections (EVCs)
EVC
Association of two or more User Network Interfaces (UNIs) at the edge of metro Ethernet network (MEN) cloud
Exchange of Ethernet frames limited to the UNI’s in the EVC
Three building blocks specified
E-Line – p2p EVC
E-LAN – mp2mp EVC
E-Tree – p2mp EVC
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 26
MEF’s Building Blocks Illustrated
26
EVC1
EVC2
Root
Leaf
Leaf
Point-to-Point EVC (E-Line) Multipoint-to-Multipoint EVC (E-LAN)
Rooted-Multipoint EVC (E-Tree)
EVC1
EVC2
Root
Leaf
Leaf
EVC1
EVC2
Root
Leaf
Leaf
EVC1
EVC2
Root
Leaf
Leaf
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 27
Putting it Together: Optical Ethernet Network Components in Operation
Service
Transport
PHY
E-LAN Service
Ethernet Service (end-to-end; what the
user perceives)
Ethernet Transport (what the cloud delivers; the
“pipe” and its routing)
PHY (how the bits are transported
between systems)
Framing, timing, and optical muxing
Ethernet Service
PHY Layer(physical link, fiber)
Switching/Routing
Optical (WDM) transport
Macro-Architectural Options for Macro-Architectural Options for Building MAN/WAN Inter-Building MAN/WAN Inter-
connects & Design Principles connects & Design Principles InvolvedInvolved
Metanoia, Inc.Critical Systems Thinking™
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 29
A Word on Network Architecture
Ultimate goal of a network: to provide end-to-end connectivity between two entities
E.g. client-server, user-to-user, …
Path between entities has many segments, comprising
Access, aggregation, metro/edge, core
Different technologies can be used in each segment, depending on that segment’s requirements
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 30
Applicability of Ethernet to Network Segments
Access
Cost Very cheap
High-speed, vast range (10 Mbps – 1 Gbps)Speed
ManageabilityLittle or no mgt. needed
(plug-and-play)Supports ELMI
Relatively cheap
Aggregation Core
Sophisticated systems increase cost
High speeds/feeds, 1 Gb/s – 10 Gb/s, link agg.
High speeds, 1 Gb/s – 100 Gb/s, LAG
Comprehensive OAM portfolio
Fault & Performance Mgt. OAM
LAG and Dual Homing (IEEE Work-in-Progress)
Via RSTP, MSTP, ring protection (G.8032)
Linear protection (G.8031), Traffic
engineering
Supports 4K services/access link
Allows hierarchy (MAC-in-MAC), Upto 16M
services
Via hierarchy, with inter-operability with IP/MPLS (PBB-VPLS interworking)
Redundancy
Scalability
Works over diverse access media (E.g. fiber,
Cu, wireless, coax, ...)
Multiple logical rings, mesh natively supported,
native multicast
Supports TE, routing extensions (e.g. PLSB)Notable Features
Network Segment
Parameters
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 31
Flexibility with Ethernet
Ethernet has features that make it suitable for the 3 key segments – depending on the operator’s need
Adaptability of Ethernet implies
Ethernet is not always needed end-to-end
Usable in segments where it makes sense
Incrementally extendable to other segments
Interoperability of Ethernet can inter-work with other technologies for optimum realization of services
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 32
Network Architecture Options with Optical Ethernet
In the following, we
Discuss key architectural options using Ethernet & optical Ethernet
Show how Ethernet migrates from the access (it’s forte) to the metro and core
Present the merits & assessment of each architecture
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 33
Ethernet in Access: Operation & Protocol Stack
X X
N-PE N-PE
IP/MPLS
CECE
Q-in-Q
MPLS/PW MPLS/PWMPLS/PW
C-DA
C-SA
S/C-Tag
Payload
C-DA
C-SA
S/C-Tag
Payload Payload
LSP-Label
VC-Label
C-DA
C-SA
S/C-Tag
C-DA
C-SA
S/C-Tag
Payload
C-DA
C-SA
S/C-Tag
Payload
C-DA
C-SA
S/C-Tag
Payload
C-DA
C-SA
S/C-Tag
Payload
LSP-Label
VC-Label
LSP-Label
VC-Label
LSP-Label
VC-Label
LSP-Label
VC-Label
Spoke PWs per VPLS instance
IB-BEBMPLSMPLS
LSRU-PE LSR U-PE
AccessAccess
Metro MetroCore
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 34
Ethernet in Access: Evaluation
Doable today! and allows gradual “upgrade” to Ethernet in metro and/or core
Cheap, flexible, convenient – uses familiar Ethernet tech. in access
Supports up to 2M services (due to 20b MPLS label) – not scalable
Needs PWs/tunnels e2e, u-PE to u-PE – potentially millions – which could become unmanageable
Metro & core networks can be anything, but are typically IP/MPLS
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 35
Ethernet in Access & Metro: Operation & Protocol Stack
B-Tag
X X
N-PE N-PE
IP/MPLSPBB PBB
B-BEBU-PE B-BEB U-PECECE
Ethernet EthernetMPLS/PW
B-DA
B-SA
B-Tag
I-Tag
C-DA
C-SA
S/C-Tag
Payload
C-DA
C-SA
S/C-Tag
Payload Payload
B-DA
B-SA
I-Tag
C-DA
C-SA
S/C-Tag
B-DA
B-SA
I-Tag
C-DA
C-SA
S/C-Tag
Payload
B-DA
B-SA
I-Tag
C-DA
C-SA
S/C-Tag
Payload
B-DA
B-SA
B-Tag
I-Tag
C-DA
C-SA
S/C-Tag
Payload
C-DA
C-SA
S/C-Tag
Payload
Must support B-BEBand VPLS capability
B-VID locally significant in PBB, not sent over core
B-BEB B-BEB
Internal B-VID, enables I-SID
bundlingVD-Label
LSP-LabelB-BEB removes
PBB-specificB-Tag
PBB
AccessAccess
Metro MetroCore
PBB
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 36
Ethernet in Access & Metro: Evaluation
Implementable today, with selected hardware/software
Allows gradual “upgrade” to Ethernet in core, if needed
Cheaper, easier, lower cost & maintenance than previous option (Ethernet in access only)
Metro PBB network enables scaling of services, while reducing processing/memory burden on metro/core devices
Core network can be anything, but is typically IP/MPLS
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 37
Ethernet Everywhere: Protocol Stack
CE
Access (802.1ad)
Metro/Aggregation (802.1ah)
Core (802.1Qay)
Metro/Aggregation (802.1ah)
Access(802.1ad)
PB
IB-BEB
BCB
BCB
PE
PE
B-BEB
B-BEB
BCB
B-BEB B-BEB
BCB
IB-BEB
PB
PE
PE
B-BEB
802.1ad/Q-in-Qencapsulation
802.1ahencapsulation
802.1ahdecapsulation
802.1ad/Q-in-Qdecapsulation
C-DA
C-SA
C-Tag
Payload
C-SA
S-Tag
C-Tag
Payload
C-DA
C-SA
S-Tag
C-Tag
Payload
C-DA
B-DA
B-SA
B-Tag
I-Tag
C-SA
S-Tag
C-Tag
Payload
C-DA
B-DA
B-SA
B-Tag
I-Tag
C-SA
S-Tag
C-Tag
Payload
C-DA
B-DA
B-SA
B-Tag
I-Tag
C-SA
S-Tag
C-Tag
Payload
C-DA
C-DA
C-SA
C-Tag
Payload
CEIB-BEB
IB-BEB
CE
CE
B-BEB
BCB
B-BEB
Provider Bridging (PBB) Provider Backbone Bridging (PBB)
Provider Backbone Bridging (PBB)
Provider Bridging (PBB)
PBB – Traffic Engineered (PBB-TE)
Switching based on pre-configured fwding tables
- Pinned paths- Based only on B-DA, B-SA, B-Tag - No STP- No MAC learning
Last MileLast Mile
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 38
Ethernet Everywhere: Evaluation
Uses proven, uniform technology throughout
Ability to transport Ethernet & IP services (since Ethernet supports IP)
Benefits Easy procurement, management, troubleshooting
Cost efficiencies (opex) from understanding, managing a single technology in the network
No technology interworking required!
Supports link, segment, and e2e (service) OAM with one OAM technology
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 39
Ethernet in Mobile Backhaul
Mobile backhaul architectures derive from the previous basic types
We examine them separately due to their unique needs:
Interface with the core network
Timing and synchronization requirements
Evolution requirements – from TDM or ATM to IP/MPLS and/or Ethernet
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 40
Evolution of Cellular Technology and Backhaul Types
Network Speed Interface
GSM/GPRS
EDGE
3G (UMTS/WCDMA) R3, R4
3G, R5 (HSDPA), R6 (HSUPA)
LTE R8 (20 Mhz)
CDMA1X-RTT
CDMA EV-DORev A/B
WiMAX (10 Mhz)
56-114 Kbps TDM
236 – 473 Kbps
384 Kbps Uplink14.4 Kbps Downlink
500 Mbps Uplink>100 Mbps Downlink
100 Kbps
1.8 Mbps Uplink1.8 to 5 Mbps Downlink
384 Kbps Uplink384 Kbps Downlink
TDM
ATM
IP/Ethernet
IP/Ethernet
IP/Ethernet
TDM
IP/Ethernet50 Mbps
Backhaul Types
2G
2.5G
3G
4G
Legend
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 41
Mobile Backhaul Components Backhaul network – defined as the network that connects
Base Transceiver Station (BTS, or Base Station) to Base Station Controller (BSC) in 3GPP2 – GSM-based cellular networks
Node-B to Radio Network Controller (RNC) in 3GPP – CDMA-based cellular networks
Traditional backhaul networks have used ... E1/T1 leased lines SONET/SDH TDM channels (for higher rate aggregation)
Mobile transport infrastructure has hitherto been ... Microwave links Optical fiber with SDH/SONET
Evolution to packet-based wireless services creates a push for the transport itself to be packet-based: Ethernet or IP/MPLS or a combination
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 42
Traditional Backhaul Evolution
SDH/SONET Network
BSC
RNC
TDMTI/EI Cellsite
Gateway
ATM
nxE1
T1/E1/STM
E1
ATM
3G BTS
2G BTS
ATM Switch
SONET/SDH XConnect
SONET/SDH XConnect
Separate transmission facilities for different technologies (TDM and packets)
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 43
Evolved Backhaul Network
Carrier EthernetNetwork
BSC
TDMTI/EI Cellsite
Gateway
ATM
nxE110/100/1GE
Ethernet
1/10GE Ethernet
3G BTS
2G BTS
IP/Ethernet Switch/Router
IP/Ethernet Switch/Router
Ethernet
3G/4G BTS
PE PECE
To Wireless Core
Common transmission infrastructure for different technologies (TDM and packets)
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 44
A Quick Primer on PseudoWires
PE1 PE2
VSI VSI
ACs AC1
ACn
Tunnel LSP
1
VPN_ID = AVPN_ID = A
2
3 Discovery
Targeted LDP4
VPN_ID = ALabel = 2004
5
Label Mapping VPN_ID = A
Label = 10045
Label Mapping
VC_LSP (2004)
VC_LSP (1004)
PW established
6
4. Targeted LDP session established
5. Mapping of label for the VC LSP (unidirectional virtual circuit (VC)) exchanged between end nodes
6. PW established, data transfer enabled
1. Bind attachment circuit to Virtual Switching Interface inside PE router
2. Assign each PE node a VPN id.
3. Nodes discover each other
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 45
Pseudowires (PW) for Legacy Transport
Carrier EthernetNetwork
BSC
TDMTI/EI
Cellsite Gateway
ATM
nxE1
10/100/1GE Ethernet
1/10GE Ethernet
3G BTS
2G BTS
Ethernet
3G/4G BTS
PE PECE
To Wireless CoreAC
AC
PSN Tunnel
PW
AC: Attachment CktPE: Provider Edge
CE : Customer Edge (BTS)BSC: Base Station Controller
Encapsulation
Structure-Agnostic TDM-over-IP (SAToIP) (RFC 4553)
Structure-Aware TDM Circuit Emulation (CESoPSN) (RFC 5086)
ATMoPSN (RFC 4717)
PSN Tunnels May be IP/MPLS, T-MPLS/MPLS-TP, or
PB/PBB/PBB-TE based
PW Signaling
IEEE 1588-based timing distribution supported
SyncE (Synchronous Ethernet)
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 46
MEF Services for Mobile Backhaul
Metro EthernetEVC EVC
RNC
BSC
Service Multiplexing
BTS BTS
RNC
BSC
BTSBTS
BTS
mp2mp EVC
Metro Ethernet
Services muxed at RNC UNI Needed when inter-BS communication is permitted like in LTE/802.16m (WiMAX)
EVPL Service for Backhaul using Metro Ethernet Networks
EVP-LAN Service for Backhaul using Metro Ethernet Networks
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 47
MEF Services for Mobile Backhaul
Metro Ethernet
EVC EVC
RNC
BSC
Service Multiplexing
BS/BTS
BS/BTS
BS/BTS
EVP-Tree Service for Backhaul using Metro Ethernet Networks
Key Developments Valuable for Key Developments Valuable for Military Adoption of Optical Military Adoption of Optical
EthernetEthernet
Metanoia, Inc.Critical Systems Thinking™
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 49
Optical Ethernet: Recent Developments
Ethernet technology evolving rapidly in the last 3-4 years
Multiple standards bodies have created valuable stds in: OAM
Interoperability
Scalability
Reliability
Security
New Services
Last-mile high speed access
Interworking
New capabilities in Ethernet – synchronization/timing, automatic SLA negotiation, Ethernet demarcation devices, Ethernet as xport
We summarize these next, and focus on key ones valuable for the military
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 50
Recent Advances in Optical Ethernet Standards: Snapshot
Area Standard and/or Activity Stds. Organization(s)
Reliability/Protection
Linear (G.8031) & ring (G.8032) protection ITU-T SG15
OAMConnectivity Fault Mgt. (802.1ag), Perf. Mgt. (Y. 1731)
IEEE, ITU-T SG 15
Security LinkSec, MACSec, Authentication IEEE
ScalabilityHierarchy via Shortest Path Bridging (PLSB)Provider Back-bone Bridging (802.1ah)
IEEE
InterworkingFCoE, Ethernet PWs, Circuit Emulation over
Ethernet (MEF 8)IETF, MEF
New Services E-Tree (p2mp communication for multicast) MEF
Higher-SpeedsFast last mile access (EPON, 802.11n), HS i/fs (40G,100G)
IEEE
New Capabilities
SyncE (link-layer clock distribution)1588v3 (network level time & clock distribution)Demarcation devices (MEF NID)Automatic SLA negotiation (MEF E-LMI)Ethernet as transport (PBB-TE)MPLS-TP (Transport Profile): applicable for COE
IEEE, MEF, IETF
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 51
Ethernet Security:LinkSec (MACSec, KeySec) Layer 2 link security standard defined by
MACSec (IEEE 802.1ae) KeySec (IEEE 802.1af)
MACSec provides: Origin authentication Data integrity checking Data confidentiality between two e2e Ethernet switches
MACSec defines a frame format that includes data encapsulation, encryption, authentication
KeySec defines key mgt. protocol for MACSec
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 52
MAC Sec Packet Format
TCI = Tag Control Info.AN=Association No.SL = Short Length (i.e. no SCI inserted)PN = Packet No.SCI= Secure Channel ID (optional)
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 53
Ethernet OAM
Ethernet OAM supports Layer (domain) Monitoring Up to 8 layer levels (domains) per VLAN
Ethernet OAM has tools for: Fault Management (802.1ag): CCM, LB, LT, AIS, RDI
CCM: Continuity Check Message – verifies one-way connectivity LB: Loop Back – checks 2-way (round trip) connectivity LT: Link Trace – provides path (nodes) between nodes A & B AIS: Alarm Indication Signal – provides fwd alarm propagation RDI: Reverse Defect Indication – provides rev alarm propagation
Performance Measurement (Y.1731): LM, DM LM: Loss Measurement – measures loss on an EVC DM: Delay Measurement – measures latency on an EV
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 54
Ethernet OAM & Maintenance Domains
Access Core Access
Customer CustomerService Provider
Customer OAM Domain
Provider OAM Domain
Operator OAM Domain Operator OAM Domain
Operator OAM Domain
Independent OAM can be run in each OAM domain for the same VLAN
IEEE provides for 8 levels of Maintenance Domains – allows a level to be assigned to each entity – customer, provider, operator
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 55
Ethernet OAM: Loopback (LB) Example for Provider & Operator Domains
Access Core Access
Customer CustomerService Provider
Customer OAM Domain
Provider OAM Domain
Operator OAM Domain Operator OAM Domain
Operator OAM Domain
Independent OAM can be run in each OAM domain for the same VLAN
We show operator, provider, and customer loopback examples above
E2e Ethernet path
Provider LB
Operator LBs
Customer LB
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 56
Synchronization in IEEE 1588
1588: a protocol designed to synchronize real-time clocks in the nodes of a distributed system that communicate using a network
Synchronizes both – clock & Time-of-Day (SyncE only synchronizes clock)
Network
Master Slave/Boundary Slave/Boundary
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 57
IEEE 1588 Synchronization Operation & Clock Offset Computation
Clock Offset Computation
MS delay = t2 – t1SM delay = t4 – t3
offset = {MS_delay –SM_delay}/2
1588 Operation
How Optical Ethernet Meets Key How Optical Ethernet Meets Key Technology Requirements of Technology Requirements of
Military NetworksMilitary Networks
Metanoia, Inc.Critical Systems Thinking™
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 59
Role of Ethernet Technology
Ethernet component provides several key capabilities
Native mp2mp communication
Easily creates communities of interest (COIs)
Segregation of traffic and users
Via VLANs (802.1ad) or B-VID/B-VLAN tags (802.1aq)
Enables use a common infrastructure for multiple virtual networks
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 60
Role of Optical Technology
Optical component complements Ethernet packet technology, providing strengths where Ethernet does not suffice
Robustness against interference/EMI
Tremendous bandwidth scalability Using an optical fiber transmission medium Via WDM technology, without needed additional fiber
Connection-oriented nature Allows for traffic engineering Sophisticated, ultra-fast protection/restoration
Long reach/range Reliable communication over long distances
Facilitates deterministic timing/performance
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 61
Suitability of Optical Ethernet for the Military (1)
Military Network Requirement
How Today’s Optical Ethernet Technology Meets It
1 Scalability
- Hierarchy – via MAC-in-MAC encapsulation - Routing & Topology capability – PLSB, TRILL (MAC learning in CP)
2 Security
- MACSec – providing e2e security between nodes - ACLs – based on address, VLAN, port, … - Queueing per VLAN, class, app., in systems
3 Manageability
- Extensive OAM for fault & perf. management - Service-level and link-level OAM, with hierarchy - OOB management capability - ELMI negotiation at UNI
4 Dynamic Setup & Control
- RSTP variants - MSTP - ELMI for negotiation at UNI - LACP helps setup link aggregation groups - IS-IS in control plane for network topology control
5 Mp2mp and p2mp communication
- Inherently mp2mp technology - E-Tree service from MEF
6 Low-Cost - Economical deployment - Capex $1/ 1 Mb/s, which is ~1/4th of TDM circuit cost
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 62
Suitability of Optical Ethernet for the Military (2)
Military Network Requirement
How Today’s Optical Ethernet Technology Meets It
7 Reliability - ITU-T link and ring protection - EAPS (Ethernet Automatic Protection Switching), RFC 3619 - Link Aggregation (LAG)
8 QoS
- VLANs for virtualization - Use of “p” bits for prioritization - Bandwidth profiles (MEF) for queueing - Per VLAN, per class traffic management - Policing, shaping, dropping, metering, marking within
systems for differentiation between traffic
9 Availability - Linear + Ring protection - EoWDM to increase reach, while decreasing BER - Dual homing in access & E-NNI (network interfaces)
10 Diverse Last-Mile Access
- P2p Ethernet - Wi-Fi access - WiMAX access - EPON
11 Support of Legacy Services
- Circuit Emulation over Ethernet (MEF8, SATOP, CESoPSN)
- Use of EtherType allows native encapsulation (of different traffic types) within Ethernet. E.g. FCoE, PPPoE
12 Clock Distribution - SyncE - IEEE 1588v2
Summary and ConclusionSummary and Conclusion
Metanoia, Inc.Critical Systems Thinking™
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 64
Wrapping it Up ...
Optical Ethernet is today a well-established & well-known technology, with many capabilities
New capabilities being rapidly added, due to its versatility and popularity
Usable in access, metro, core, in mobile backhaul, data centers, ...
Interoperable – so can be mixed-and-matched with other technologies
Suitable for net-centric, military applications
Adds value in many applications, and a strong candidate to be used where its characteristics fit the application or network segment under consideration
Thank You!Thank You!Questions? Questions?
Metanoia, Inc.Critical Systems Thinking™
GlossaryGlossary
Metanoia, Inc.Critical Systems Thinking™
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 67
Glossary (1)ACL Access Control List
BCB Backbone Core Bridge
BEB Backbone Edge Bridge
B-MAC Backbone MAC
BSC Base Station Controller
BTS Base Transceiver Station
B-VID Backbone Virtual ID
CAC Connection Admission Control
CE Customer Edge
COI Communities of Interest
COTS Common Off-The-Shelf
DA Destination Address
DCN Data Communication Network
DoD Department of Defence
DPI Deep Packet Inspection
DWDM Dense Wavelength Division Multiplexing
e2e End to End
ECMP Equal Cost Multi-Path
ELMI Ethernet Local Management Interface
EPON Ethernet Passive Optical Network
EVC Ethernet Virtual Circuit
GPON Gigabit-capable PON
H-QoS Hierarchical QoS
IEEEInstitution of Electrical and Electronic
Engineers
IETF Internet Engineering Task Force
IGMP Internet Group Management Protocol
I-SID Individual Service ID
LAG Link Aggregation Group
LC Line Card
LDP Label Distribution Protocol
MEF Metro Etherent Forum
MEN Metro Ethernet Network
mp2mp Multi-point to Multi-point
MPLS Multi Protocol Label Switching
MPLS-TPMulti-Protocol Label Switching -
Transport Profile
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 68
Glossary (2)MSTP Multiple Spanning Tree Protocol
NGN Next-Generation Network
NMS Network Management System
N-PE Network-facing-Provider Edge device
NSF Non-Stop Forwarding
NSR Non-Stop Routing
OAMOperations, Administration, and Maintenance
ODU Optical Data Unit
OOB Out of Band
OTN Optical Transport Network
p2mp Point to Multi-point
PB Provider Bridging
PBB Provider Backbone Bridging
PBB-TEProvider Backbone Bridging - Traffic Engineering
PE Provider Edge
PHY Physical Layer
PLSB Provide Link State Bridging
PON Passive Optical Network
POTs Plain Old Telephone Service
PSN Packet Switched Network
PW Pseudowire
QoS Quality of Service
RNC Radio Network Controller
RSTP Rapid Spanning Tree Protocol
RSVP-TEResource Reservation Protocol - Traffic Engineering (RSVP protocol with MPLS traffic engineering extensions)
SA Source Address
SDH Synchronous Digital Hierarchy
SONET Synchronous Optical Network
SPT Shortest Path Tree
STP Spanning Tree Protocol
TDM Time Division Multiplexing
TRILLTransparent Interconnection of Lots of Linkshttps://datatracker.ietf.org/wg/trill/charter/
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 69
Glossary (3)UNI User Network Interface
U-PE User-facing-Provider Edge device
VLAN Virtual LAN
VPN Virtual Private Network
Appendix: Word on Provider Appendix: Word on Provider Bridging (PB) and Provider Bridging (PB) and Provider Backbone Bridging (PBB)Backbone Bridging (PBB)
Metanoia, Inc.Critical Systems Thinking™
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 71
Native Ethernet in Metro Access
How does one create the notion of a virtual circuit? VLAN tagging with point-to-point VLAN
VLAN stacking Outer tag service instance; Inner tag individual customer
802.1Q in 802.1Q (Q-in-Q) - IEEE 802.1ad
C-DA: Customer Destination MAC
C-SA: Customer Source MAC
C-TAG: IEEE 802.1q VLAN Tag
C-FCS: Customer FCS
S-TAG: IEEE 802.1ad S-VLAN Tag
C-DA C-TAGC-SA Client data FCSS-TAG
6bytes 6bytes 4bytes 4bytes 4bytes
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 72
Customer Network
Customer Network
Customer Network
72
Provider Bridge (IEEE 802.1ad) Architecture
CE: Customer Equipment
UNI: User-to-Network Interface
CES: Core Ethernet Switch/Bridge
P-VLAN: Provider VLAN
UNI-B
CES
CES
CE-A
UNI-A
UNI-C
CE-C
Spanning tree
CE-B
CES
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 73
Provider Backbone Bridging (802.1ah)
Encapsulate customer MAC with provider MAC at edge
Edge switch adds 24-bit service tag (I-SID), not VLAN tag
Core switches need only learn edge switch MAC adds.
S-TAG: IEEE 802.1ad S-VLAN Tag
B-DA: IEEE 802.1ah Backbone Destination
B-SA: IEEE 802.1ah Backbone Source MAC
I-TAG: IEEE 802.1ah Service Tag
B-DA B-TAGB-SA I-TAG C-DA C-TAGC-SA Client data B-FCS
6bytes 6bytes 6bytes6bytes4bytes 5bytes 4bytes 4bytes
Copyright 2010All Rights Reserved Milcom’10, October 31-Nov 3, 2010, San Jose, CA 74
Provider Backbone Bridging (PBB) Architecture
CPE BCPE ACPE C
Provider backbone network (802.1ah)
CPE BCPE A
802.1ad
CPE BCPE B
802.1q
CPE C
Provider backbone network (802.1ad)
CPE D
CPE DCPE C
CPE A
Provider backbone network (802.1ad)
Provider backbone network (802.1ad)
Provider backbone network (802.1ad)
Recommended