Upload
jwirth779
View
227
Download
0
Embed Size (px)
Citation preview
8/7/2019 GIGABIT CAMPUS NETWORK DESIGN 1
1/26
Public
Copyright 2000 Cisco Systems, Inc. All Rights Reserved.
Page 1 of 26
WHITE PAPER
Gigabit Campus DesignConfiguration and Recovery Analysis
Introduction
Some of the most useful variations of the large-scale multilayer campus internetwork design were tested for failure recovery. Please refer to
the companion document Gigabit Campus DesignPrinciples and Architecture for an explanation of several implementations of the
multilayer design. In all cases, Gigabit EtherChannel links were used between distribution-layer switches and core-layer switches.
Redundant Gigabit Ethernet uplinks were used to connect the access-layer switches to the distribution-layer switches. The Catalyst 4000,
5500, and 6000 family switches were used in the wiring closet configurations. The Catalyst 6500 with Multilayer Switch Feature Card
(MSFC) and Catalyst 8540 were both used as the Layer 3 switch. The Catalyst 6500 is used as the Layer 2 switch in the core.
This paper documents configuration and recovery with three fundamental designs:
1. Layer 3 distribution with dual-path Layer 2 core (good)
a) Catalyst 6500 standard building block
2. Layer 3 distribution with Layer 3 core (better)
a) Catalyst 6500 standard building block
3. Layer 3 distribution with dual-path Layer 3 core (best)
a) Catalyst 6500 standard building block
b) Catalyst 6500 virtual LAN(VLAN) building block
The following two design variations are described in the appendix:
A1. Layer 3 distribution with Layer 3 core
a) Catalyst 8540 standard building block
A2. Layer 3 distribution with dual-path Layer 3 core
a) Catalyst 8540 standard building block
Layer 2 spanning-tree loops were eliminated in the core and in the standard building block. Avoidance of spanning-tree loops in the core is
important for high availability. Within the core and the standard building block all Layer 3 switches were configured with native routed
interfaces. VLAN trunks were only used within the VLAN building block. The VLAN building block is typically used in a server farm to
provide for redundant server connection with dual Network Interface Cards (NICs).The Enhanced Interior Gateway Routing Protocol (EIGRP) was tuned for fast convergence with a one-second-hello timer and a
three-second-hold timer. The Hot Standby Routing Protocol (HSRP) was tuned for fast convergence with a one-second-hello timer and a
three-second-holdtimer.When using protocolhello timers of onesecond, the limit of accuracyfor recoverymeasurement is about onesecond
8/7/2019 GIGABIT CAMPUS NETWORK DESIGN 1
2/26
Public
Copyright 2000 Cisco Systems, Inc. All Rights Reserved.Page 2 of 26
Thus, all results are rounded to units of whole seconds, representing the worst case. Network recovery times were very predictable, with some
variation due to the state of things such as the Address Resolution Protocol (ARP) cache. OSPF was configured with the same timer values
as EIGRP, with similar results.
Every router and switch in the network was provided an out-of-band Ethernet management interface on VLAN 99. One Ethernet port on
each device was configured in VLAN 99 and wired outside the test network to a separate switched management network. In addition to the
management IP address on VLAN 99, we also configured Cisco 2511 terminal servers with RJ-45 serial cables to access the console port of
every router and switch in the layout. With this arrangement, we have two out-of-band paths to every switch and router, as well as a number
of in-band paths.
Wherever possible Gigabit EtherChannel was configured with ports on two different cards to increase availability. For example,
port-channel 1 on CT85 (core top 8540) used interfaces gigabit 0/0/0 and gigabit 1/0/0, which are the first physical ports on two different
cards. EtherChannel recovery was not tested, as it is much faster than one second and falls below the measurement threshold of this testing.
Configuration of the Standard Building Block
The standard building block appears on the left side (west block) of Figures 3a, 3b, and 3c. Configuration of the standard building block is
very simplebecause all spanning-tree loopsareeliminated. Hence,there is no Layer 2 tuning, such as selecting the best root switch fora given
VLAN. No VLAN trunks are used and uplinks are connected to native routed interfaces on the Layer 3 switches in the distribution layer.
Catalyst 6000 Access Switch: Standard Building BlockSet prompt aw6 (access layer, west block, Catalyst 6000)
Set vtp domain west
Set vtp mode transparent
(no VLAN trunks, use transparent mode)
Set vlan 99 (used for out of band management)
Set vlan 99 3/48 (last physical port on switch used for oob management)
Set int sc0 99 172.29.196.51 255.255.254.0
(logical console port for oob management)
Set ip route default 172.29.196.1
(gateway router in oob management network)
Set port channel 1/1-2 mode off
(turn off channel negotiation on uplinks)
(not using EtherChannel in this configuration)
set trunk 1/1-2 off(turn off VLAN trunking on uplinks)
set vlan 10(VLAN 10 corresponds to subnet 10 ie 10.10.0.0)
set vlan 10 1/1-2(all other ports are part of VLAN 10)
set vlan 10 3/1-47
We disable channel negotiation on the uplinks to make connection faster after a failure has been restored. This is appropriate because the
uplinks are routed connections to Layer 3 switches in the distribution layer, not Layer 2 connections that require spanning tree. (Nothing in
the configuration indicates that all clients and uplinks attached to the switch are in subnet 10 (10.10.0.0) with mask 255.255.0.0. VLAN 10
is configured everywhere to match subnet 10 [10.10.0.0]).
Configuration of the Standard Building Block with Load Balancing
To achieve load balancing with thestandardbuilding block,a couple of changes are required. Configuretwo VLANs (two subnets)on a wiring
closet switch and use VLAN trunks for uplinks. For example, on switch aw6, configure VLAN 10 and VLAN 11. All outbound VLAN 10
traffic will take one uplink to the HSRP primary gateway router for subnet 10.10.0.0. All outbound VLAN 11 traffic will take the other uplink
to the HSRP primary gateway router for subnet 10.11.0.0. This load balancing configuration maintains all the advantages of the standard
building block and is only marginally more complex.
8/7/2019 GIGABIT CAMPUS NETWORK DESIGN 1
3/26
Public
Copyright 2000 Cisco Systems, Inc. All Rights Reserved.Page 3 of 26
Catalyst 6000 Access Switch: Standard Building Block with Load Balancing
Set prompt aw6 (access layer, west block, Catalyst 6000)
Set vtp domain west
Set vtp mode transparent
(use transparent mode, configure VLANs explicitly)
Set vlan 99 (used for out of band management)
Set vlan 99 5/48 (last physical port on switch used for oob management)
Set int sc0 99 172.29.196.51 255.255.254.0(logical console port for oob management)
Set ip route default 172.29.196.1
(gateway router in oob management network)
Set port channel 1/1-2 mode off
(turn off channel negotiation on uplinks)
(not using EtherChannel on this switch)
set trunk 1/1-2 on 10,11 dot1q
(VLAN trunking on uplinks, for VLAN 10 and VLAN 11)
set vlan 10 3/1-48(VLAN 10 corresponds to subnet 10 ie 10.10.0.0)
set vlan 11 4/1-48(VLAN 11 corresponds to subnet 11 ie 10.11.0.0)
Configuration of the VLAN Building Block
The VLAN building block appears on the right side (east block) of Figures 3a, 3b, and 3c. If dual-attached servers are used, then a VLAN
trunking configuration is required within the server distribution block. The two distribution-layer switches are the root bridges of the even andodd numbered VLANs. UplinkFast is configured on the access-layer switches. BackboneFast is configured on the access and distribution
switchesfor faster spanningtree recovery. Formaximum determinism, VTPtransparent modeis usedand allVLANs are configured explicitly.
HSRP at Layer 3 is configured to match the Layer 2 spanning tree configuration. This way, the HSRP primary gateway router for
even-numbered subnets is also the spanning tree root for even-numbered VLANs. The HSRP primary gateway router for odd-numbered
subnets is also the spanning tree root for odd-numbered VLANs.
Configuration of Catalyst 6000 Access Switch:
Set prompt ae6 (access layer, east block, Catalyst 6000)
Set vtp domain east
Set vtp mode transparent
(use transparent mode, configure all VLANs explicitly)
Set vlan 99 (used for out of band management)
Set vlan 99 4/48 (last physical port on switch used for oob management)Set int sc0 99 172.29.196.51 255.255.254.0
(logical console port for oob management)
Set ip route default 172.29.196.1
(gateway router in oob management network)
set spantree uplinkfast enable
(set uplinkfast on the access switch only)
set spantree backbonefast enable
(enable backbonefast on all switches in block)
Set port channel 1/1-2 mode off
(turn off channel negotiation on uplinks)
(not using EtherChannel on this switch)
set trunk 1/1-2 on 50,51,52,53 dot1q
(dot1q VLAN trunking on uplinks, state VLANs explicitly)
(these four VLANs are used in this building block)
set vlan 50 2/1-48(VLAN 50 corresponds to subnet 50 ie 10.50.0.0)
set vlan 51 3/1-48(VLAN 51 corresponds to subnet 51 ie 10.51.0.0)
8/7/2019 GIGABIT CAMPUS NETWORK DESIGN 1
4/26
Public
Copyright 2000 Cisco Systems, Inc. All Rights Reserved.Page 4 of 26
Configuration of Catalyst 6500 Distribution Switch:
The ARP cache timeout on the MSFC is four hours. However, the Layer 2 CAM table times out in 300 seconds by default. This may result
in some IP unicast traffic being flooded. In the following configuration the CAM timeout agingtime is set to 4hours*60min/hour*60sec/
min. = 14400 seconds to match the ARP cache timeout.
Set prompt det65 (distribution layer, east block, top, Catalyst 6500)
Set vtp domain east
Set vtp mode transparent(use transparent mode, configure all VLANs explicitly)
Set vlan 99 (used for out of band management)
Set vlan 99 3/48 (last physical port on switch used for oob management)
Set int sc0 99 172.29.196.51 255.255.254.0
(logical console port for oob management)
Set ip route default 172.29.196.1
(gateway router in oob management network)
set cam agingtime 1-1000 14400
(set CAM timeout to 4 hours to match ARP timeout)
set spantree backbonefast enable
(enable backbonefast on all switches in block)
Set port channel 1/1-2 mode off
(turn off channel negotiation on non-Etherchannel links)
set port channel 3/7-8 mode off
set port channel 4/7-8 mode offset port channel 3/1-2 mode on
(EtherChannel used for routed links to the core)
set port channel 4/1-2 mode on
(EtherChannel used for routed links to the core)
set vlan 40 3/1,4/1
(routed etherchannel link VLAN40=10.40.0.0 spans two cards)
set vlan 42 3/2,4/2
(routed etherchannel link VLAN42=10.42.0.0 spans two cards)
set trunk 3/7 on 50,51,52,53 dot1q
(set VLANs and dot1q trunking explicitly)
(3/7-8 and 4/7-8 are uplinks to wiring closet switches)
set trunk 3/8 on 50,51,52,53 dot1q
set trunk 4/7 on 50,51,52,53 dot1q
set trunk 4/8 on 50,51,52,53 dot1q
set trunk 1/1 on 50,51,52,53 dot1q
(this is the backup trunk to other distribution switch)
set spantree root 50,52
(make this root bridge for even VLANs)
set spantree root secondary 51,53
(make this backup root bridge odd VLANs)
set trunk 3/1-2 off
(no VLAN trunking on routed links to core)
set trunk 4/1-2 off
(no VLAN trunking on routed links to core)
As shown in Figures 3b and 3c, VLANs 50, 51, 52, and 53 correspond to the wiring closet subnets. VLANs 40 and 42 correspond to routed
links to the core switches. Refer to section three of the test results for the corresponding MSFC (router) configuration.
8/7/2019 GIGABIT CAMPUS NETWORK DESIGN 1
5/26
Public
Copyright 2000 Cisco Systems, Inc. All Rights Reserved.Page 5 of 26
Notes on Test Procedures
Test results are measured by recovery of multiple two-way PING traffic flows. In all cases, test traffic flows are between clients attached to
the network, not to and from the switches within the network. We measure the outage when the failure is created in terms of seconds of lost
traffic. Then we measure the outage when the failure is restored in terms of seconds of lost traffic. Results are rounded up to the next second,
which is the limit of accuracy with one-second-hello protocols. If recovery is given as three seconds in the table, that means we measured two
seconds and three seconds over several tests. The small routing table just reflects the subnets within the test bed. To stress routing protocol
convergence, the large routing table includes 3000 static host routes injected into the test bed from the six Layer 3 switches.
Summary of IP Routing with Small Routing Table
Summary of IP Routing with Large Routing Table
To stress the control plane (routing software running on the CPU) static routes are added to each of the routers (Layer 3 switches) in the test
bed by Trivial File Transfer Protocol (TFTP). TFTP configuration is accomplished with the configure network command. The files on theTFTP server consist of a series of commands of the form:
ip route 10.0.1.001 255.255.255.255 null 0 1
ip route 10.0.1.002 255.255.255.255 null 0 1
ip route 10.0.1.003 255.255.255.255 null 0 1
ip route 10.0.1.004 255.255.255.255 null 0 1
ip route 10.0.1.254 255.255.255.255 null 0 1
end
ctmsfc#sho ip route summary
Route Source Networks Subnets Overhead Memory (bytes)
connected 1 5 336 864
static 1 0 56 144
eigrp 1 0 11 616 1584
internal 2 2328
Total 4 16 1008 4920
ctmsfc#sho ip ro sum
Route Source Networks Subnets Overhead Memory (bytes)
connected 1 5 336 864
static 1 500 28056 72144
eigrp 1 0 2511 140616 361584
internal 2 2328
Total 4 3016 169008 436920
8/7/2019 GIGABIT CAMPUS NETWORK DESIGN 1
6/26
Public
Copyright 2000 Cisco Systems, Inc. All Rights Reserved.Page 6 of 26
1. Layer 3 Distribution with Dual-Path Layer 2 CoreCatalyst 6500
Please refer to Figures 1a, 1b, and 1c for this section. Clients test1 through test8 were attached to the wiring closet switches in order to
test network recovery times as different failures were induced. This campus design features redundant Layer 3 switches in each
distribution layer building block and redundant Layer 2 switches in the core. The dual-path Layer 2 core consists of two separate switched
VLANs with no loops andno VLAN trunks. The links into thecore VLANs arenative routedinterfaces on thedistribution-layer switches,
and VLAN trunks are not used.
8/7/2019 GIGABIT CAMPUS NETWORK DESIGN 1
7/26
Public
Copyright 2000 Cisco Systems, Inc. All Rights Reserved.Page 7 of 26
Configuration of MSFC: Client-Side Interface on HSRP Prim ary Distribution Switch
interface Vlan10
ip address 10.10.0.81 255.255.0.0
no ip redirects
no ip directed-broadcast
ip hello-interval eigrp 1 1
(set eigrp hello timer 1 second)
ip hold-time eigrp 1 3(set eigrp hold timer 3 seconds)
standby 10 timers 1 3
(set HSRP hello timer 1 hold timer 3 seconds)
(convention - HSRP group number 10 matches VLAN number)
standby 10 priority 200 preempt delay 60
(this is the primary gateway router for subnet 10)
(preempt delay 60 seconds allows EIGRP to stabilize before
HSRP switches back upon power recovery)
standby 10 ip 10.10.0.200
(10.10.0.200 is the HSRP gateway router address)
standby 10 track Vlan31 75
standby 10 track Vlan32 75
(if you lose both links to the backbone, drop priority
by 150 to initiate HSRP recovery)
Configuration of MSFC: Client-Side Interface on HSRP Secondary Distribution Switch
interface Vlan10
ip address 10.10.0.82 255.255.0.0
no ip redirects
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3
standby 10 timers 1 3
standby 10 priority 100 preempt delay 60
(HSRP secondary or backup gateway router for subnet 10)
standby 10 ip 10.10.0.200
standby 10 track Vlan31 25
standby 10 track Vlan32 25
Additional MSFC ConfigurationInterface to Backbone VLAN 31
interface Vlan31
ip address 10.31.0.81 255.255.0.0
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3
Additional MSFC ConfigurationInterface on Management VLAN
interface Vlan99
ip address 172.26.196.81 255.255.254.0
no ip directed-broadcast
Additional MSFC ConfigurationEIGRP with Passive Interfaces to Wiring Closets
router eigrp 1
passive-interface Vlan10
passive-interface Vlan11passive-interface Vlan12
passive-interface Vlan13
passive-interface Vlan99
network 10.0.0.0
8/7/2019 GIGABIT CAMPUS NETWORK DESIGN 1
8/26
Public
Copyright 2000 Cisco Systems, Inc. All Rights Reserved.Page 8 of 26
2. Layer 3 Distribution with Layer 3 CoreCatalyst 6500
Please refer to Figures 2a, 2b, and 2c for this section. Clients test1 through test8 were attached to the wiring closet switches in order to
test network recovery times as different failures were induced. Each distribution-layer switch has a single connection into the core, but
each building block as a whole has redundant connectivity into the core.
This campus design features redundant Layer 3 switches in the distribution layer. For building block west the redundant switches are
dwt65 (distribution west top 6500) and dwb65 (distribution west bottom 6500). Each distribution switch has a two-port GigabitEtherChannel connection to a Layer 3 switch in the core. The HSRP track is configured on dwt65 and dwb65 so that fast HSRP recovery
will take place if the routed link to the core is broken.
There is one important caveat with this design: because each distribution-layer switch has a single path into the core, a redundant routed
path must be provided in case this fails. If the physical link breaks, then HSRP will handle the recovery. However, in the event of some
logical failure of the MSFC within the core switch, HSRP will not be triggered. An example of this is when the MSFC in the core router
is reloaded. Eliminate the passive interface command on two of the wiring closet VLAN interfaces to provide two routed backup paths
between the distribution-layer switches.
Table 3 Results with Small Routing Table
Note: When power is restored to the core switch, the links out to the distribution layer come up before EIGRP has stabilized. Therefore,
HSRP on the distribution switch preempts the primary gateway router function a few seconds before EIGRP on the distribution switch has a
routing table built; hence the 14-second interruption as noted. The preempt delay command-line option refers to delay upon powerup of the
HSRP switch itself, and does not affect this value.
Test FailureTime of
InterruptionRecovery
Mechanism
Wiring Closet Uplink
Fail 4s HSRP
Restore 1s HSRP
Distribution Switch
Fail 4s HSRP
Restore 1s HSRP
Core Switch
Fail 4s HSRP
Restore 14s (See note below)
8/7/2019 GIGABIT CAMPUS NETWORK DESIGN 1
9/26
Public
Copyright 2000 Cisco Systems, Inc. All Rights Reserved.Page 9 of 26
Table 4 Results with Large Routing Table
Note: When power is restored to the core switch, the links out to the distribution layer come up before EIGRP has stabilized. Therefore,
HSRP on the distribution switch preempts the primary gateway router function a few seconds before EIGRP on the distribution switch has a
routing table built; hence the 30-second interruption as noted. The preempt delay command-line option refers to delay upon powerup of the
switch running HSRP in the distribution layer, and is not effective when the core switch is powered up.
Configuration of MSFC: Client-Side Interface on HSRP Prim ary Distribution Switch
interface Vlan10
ip address 10.10.0.81 255.255.0.0
no ip redirects
no ip directed-broadcast
ip hello-interval eigrp 1 1
(set eigrp hello timer 1 second)
ip hold-time eigrp 1 3
(set eigrp hold timer 3 seconds)
standby 10 timers 1 3
(set HSRP hello timer 1 hold timer 3 seconds)
(convention - HSRP group number 10 matches VLAN number)
standby 10 priority 200 preempt delay 60
(this is the primary gateway router for subnet 10)
(preempt delay 60 seconds allows EIGRP to stabilize before
HSRP switches back upon power recovery)
standby 10 ip 10.10.0.200
(10.10.0.200 is the HSRP gateway router address)
standby 10 track Vlan20 150
(if you lose the link to the backbone, drop priority
by 150 to initiate HSRP recovery)
Test FailureTime of
InterruptionRecovery
Mechanism
Wiring Closet Uplink
Fail 4s HSRP
Restore 1s HSRP
Distribution Switch
Fail 4s HSRP
Restore 1s HSRP
Core Switch
Fail 4s HSRP
Restore 30s (See note below)
8/7/2019 GIGABIT CAMPUS NETWORK DESIGN 1
10/26
Public
Copyright 2000 Cisco Systems, Inc. All Rights Reserved.Page 10 of 26
Configuration of MSFC: Client-Side Interface on HSRP Secondary Distribution Switch
interface Vlan10
ip address 10.10.0.82 255.255.0.0
no ip redirects
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3
standby 10 timers 1 3standby 10 priority 100 preempt delay 60
(HSRP secondary or backup gateway router for subnet 10)
standby 10 ip 10.10.0.200
standby 10 track Vlan23 50
Additional MSFC ConfigurationAny Routed Interface to Ba ckbone
interface Vlan21
ip address 10.21.0.81 255.255.0.0
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3
Additional MSFC ConfigurationInterface on Management VLAN
interface Vlan99
ip address 172.26.196.81 255.255.254.0no ip directed-broadcast
Additional MSFC ConfigurationEIGRP with Passive Interfaces to Wiring Closets
router eigrp 1
passive-interface Vlan10
passive-interface Vlan11
(no passive interface on VLAN 12 or VLAN 13)
(two VLANs are kept as redundant routed paths)
passive-interface Vlan99
network 10.0.0.0
3. Layer 3 Distribution with Dual-Path Layer 3 CoreCatalyst 6500
Please refer to Figures 3a, 3b, and 3c for this section. Clients test1 through test8 were attached to the wiring closet switches in order to
testnetwork recoverytimes as different failureswereinduced. Thisdesign has thehighest level of redundancyand thehighest linkcapacity
into the core.
This campus design features redundant Layer 3 switches in the distribution layer. For building block west the redundant switches are
dwt65 (distribution west top 6500) and dwb65 (distribution west bottom 6500). Each distribution switch has redundant two-port Gigabit
EtherChannel connections to both Layer 3 switches in the core. The HSRP track is configured on dwt65 and dwb65 so that fast HSRP
recovery will take place if both routed links to the core are broken.
8/7/2019 GIGABIT CAMPUS NETWORK DESIGN 1
11/26
Public
Copyright 2000 Cisco Systems, Inc. All Rights Reserved.Page 11 of 26
Table 5 Testing with the Standard Building BlockSmall Routing Table
Table 6 Testing with the Standard Building BlockLarge Routing Table
Test FailureTime of
InterruptionRecovery
Mechanism
Wiring Closet Uplink
Fail 3s HSRP
Restore 0s HSRP
Distribution Switch
Fail 3s HSRP
Restore 0s HSRP
Core Switch
Fail 0s Dual-path redundancy
Restore 6s EIGRP
Test FailureTime of
InterruptionRecovery
Mechanism
Wiring Closet Uplink
Fail 3s HSRP
Restore 0s HSRP
Distribution Switch
Fail 3s HSRP
Restore 0s HSRP
Core Switch
Fail 0s Dual-path redundancy
Restore 20s EIGRP
8/7/2019 GIGABIT CAMPUS NETWORK DESIGN 1
12/26
Public
Copyright 2000 Cisco Systems, Inc. All Rights Reserved.Page 12 of 26
Table 7 Testing with the Standard Building BlockSmall Routing Table
Note: Restoration of the distribution-layer switch causes a 15-second outage. This is a result of the fact that the Layer 2 and Layer 3 path
determination functions on the switch recover at slightly different moments. When the routing table is increased to 3000 routes in the next
test, this anomaly is eliminated.
Table 8 Testing with the VLAN Building BlockLarge Routing Table
Test FailureTime of
InterruptionRecovery
Mechanism
Wiring Closet Uplink
Fail 2s UplinkFast
Restore 0s UplinkFast
Distribution Switch
Fail 3s UplinkFast/HSRP
Restore 15s (See note below)
Core Switch
Fail 3s Dual-path redundancy
Restore 0s Dual-path redundancy
VLAN Backup Trunk
Fail 31s Spanning tree
Restore 28s Spanning tree
Test FailureTime of
InterruptionRecovery
Mechanism
Wiring Closet Uplink
Fail 2s UplinkFast
Restore 0s UplinkFast
Distribution Switch
Fail 3s UplinkFast/HSRP
Restore 5s (See note below)
Core Switch
Fail 3s Dual-path redundancy
Restore 0s Dual-path redundancy
VLAN Backup Trunk
Fail 31s Spanning tree
Restore 28s Spanning tree
8/7/2019 GIGABIT CAMPUS NETWORK DESIGN 1
13/26
Public
Copyright 2000 Cisco Systems, Inc. All Rights Reserved.Page 13 of 26
Note: When the distribution switch is restored, a five-second outage results because the Layer 2 topology stabilizes at a slightly different
moment than the Layer 3 topology. With the large routing table this difference is reduced from the previous test.
Configuration of MSFC: Client-Side Interface on Distribution SwitchHSRP Primary
interface Vlan10
ip address 10.10.0.81 255.255.0.0
no ip redirects
no ip directed-broadcast
ip hello-interval eigrp 1 1
(set eigrp hello timer 1 second)
ip hold-time eigrp 1 3
(set eigrp hold timer 3 seconds)
standby 10 timers 1 3
(set HSRP hello timer 1 hold timer 3 seconds)
(convention - HSRP group number 10 matches VLAN number)
standby 10 priority 200 preempt delay 60
(this is the primary gateway router for subnet 10)
(preempt delay 60 seconds allows EIGRP to stabilize before
HSRP switches back upon power recovery)
standby 10 ip 10.10.0.200
(10.10.0.200 is the HSRP gateway router address)
standby 10 track Vlan20 75standby 10 track Vlan21 75
(if you lose both links to the backbone, drop priority
by 150 to initiate HSRP recovery)
Configuration of MSFC: Client-Side Interface on Distribution SwitchHSRP Secondary
interface Vlan10
ip address 10.10.0.82 255.255.0.0
no ip redirects
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3
standby 10 timers 1 3
standby 10 priority 100 preempt delay 60
(HSRP secondary or backup gateway router for subnet 10)
standby 10 ip 10.10.0.200
standby 10 track Vlan22 25
standby 10 track Vlan23 25
(track both links to the backbone)
Additional MSFC ConfigurationAny Routed Interface to Backbone
interface Vlan21
ip address 10.21.0.81 255.255.0.0
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3
Additional MSFC ConfigurationInterface on Management VLAN
interface Vlan99
ip address 172.26.196.81 255.255.254.0
no ip directed-broadcast
Additional MSFC ConfigurationEIGRP with Passive Interfaces to Wiring Closets
router eigrp 1
passive-interface Vlan10
passive-interface Vlan11
passive-interface Vlan12
passive-interface Vlan13
passive-interface Vlan99
network 10.0.0.0
8/7/2019 GIGABIT CAMPUS NETWORK DESIGN 1
14/26
Public
Copyright 2000 Cisco Systems, Inc. All Rights Reserved.Page 14 of 26
4. Layer 3 Distribution with Layer 3 CoreCatalyst 8540
Please refer to Figures A1a, A1b, and A1c for this section. Clients target1 through target6 were attached to the wiring closet switches in
order to test network recovery times as different failures were induced. Each distribution-layer switch has a single connection into the
core, but each building block as a whole has redundant connectivity into the core.
This campus design features redundant Layer 3 switches in the distribution layer. For building block left the redundant switches are
dlt85 (distribution left top 8540) and dlb85 (distribution left bottom 8540). Each distribution switch has a two-port Gigabit EtherChannelconnection to a Layer 3 switch in the core. The HSRP track is configured on dlt85 and dlb85 so that fast HSRP recovery will take place
if the routed link to the core is broken.
Table 9 Results with Small Routing Table
Table 10 Results with Large Routing Table
Test FailureTime of
InterruptionRecovery
Mechanism
Wiring Closet Uplink
Fail 4s HSRP
Restore 1s HSRP
Distribution Switch
Fail 3s HSRP
Restore 1s HSRP
Core Switch
Fail 6s EIGRP
Restore 3s HSRP
Test FailureTime of
InterruptionRecovery
Mechanism
Wiring Closet Uplink
Fail 4s HSRP
Restore 1s HSRP
Distribution Switch
Fail 3s HSRP
Restore 2s HSRP
Core Switch
Fail 7s EIGRP
Restore 3s HSRP
8/7/2019 GIGABIT CAMPUS NETWORK DESIGN 1
15/26
Public
Copyright 2000 Cisco Systems, Inc. All Rights Reserved.Page 15 of 26
Configuration Details of HSRP Primary Distribution Router dlt85
interface Port-channel1
ip address 10.70.0.53 255.255.0.0
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3
hold-queue 300 in
!interface Port-channel2
ip address 10.71.0.53 255.255.0.0
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3
hold-queue 300 in
!
interface GigabitEthernet0/0/1
no ip address
no ip directed-broadcast
channel-group 1
!
interface GigabitEthernet1/0/0
ip address 10.60.0.53 255.255.0.0
no ip redirectsno ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3
standby timers 1 3
standby priority 200
standby preempt
standby ip 10.60.0.200
standby track Port-channel1 150
!
interface GigabitEthernet1/0/1
no ip address
no ip directed-broadcast
channel-group 1
!
interface GigabitEthernet2/0/0
ip address 10.61.0.53 255.255.0.0
no ip redirects
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3
standby timers 1 3
standby priority 200
standby preempt
standby ip 10.61.0.200
standby track Port-channel1 150
!
interface GigabitEthernet2/0/1
no ip address
no ip directed-broadcastchannel-group 2
!
interface GigabitEthernet3/0/0
ip address 10.62.0.53 255.255.0.0
no ip redirects
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3
8/7/2019 GIGABIT CAMPUS NETWORK DESIGN 1
16/26
Public
Copyright 2000 Cisco Systems, Inc. All Rights Reserved.Page 16 of 26
standby timers 1 3
standby priority 200
standby preempt
standby ip 10.62.0.200
standby track Port-channel1 150
!
interface GigabitEthernet3/0/1
no ip address
no ip directed-broadcast
channel-group 2
!
interface Ethernet0 (management interface)
ip address 172.26.196.53 255.255.254.0
no ip directed-broadcast
!
!
router eigrp 1
passive-interface GigabitEthernet0/0/0
passive-interface GigabitEthernet1/0/0
passive-interface GigabitEthernet2/0/0
passive-interface GigabitEthernet3/0/0
network 10.0.0.0!
end
5. Layer 3 Distribution with Dual-Path Layer 3 CoreCatalyst 8540
Please refer to Figures A2a, A2b, and A2c for this section. Clients target1 through target6 were attached to the wiring closet switches in
order to test network recovery times as different failures were induced. The dual-path design has the highest level of redundancy and the
highest link capacity into the core.
This campus design features redundant Layer 3 switches in the distribution layer. For building block left the redundant switches are
dlt85 (distribution left top 8540) and dlb85 (distribution left bottom 8540). Each distribution switch has redundant two-port Gigabit
EtherChannel connections to both Layer 3 switches in the core. The HSRP track is configured on dlt85 and dlb85 so that fast HSRP
recovery will take place if both routed links to the core are broken.
Table 11 Results with Small Routing Table
Test FailureTime of
InterruptionRecovery
Mechanism
Wiring Closet Uplink
Fail 3s HSRP
Restore 0s HSRP
Distribution Switch
Fail 3s HSRP
Restore 0s HSRP
Core Switch
Fail 3s EIGRP
Restore 1s EIGRP
8/7/2019 GIGABIT CAMPUS NETWORK DESIGN 1
17/26
Public
Copyright 2000 Cisco Systems, Inc. All Rights Reserved.Page 17 of 26
Table 12 Results with Large Routing Table
Note: Restoring the distribution switch results in an outage between the time that HSRP switches and the EIGRP routing table is built. This
can be corrected by adding a preempt delay as in the following command:
standby 10 priority 100 preempt delay 60
Configuration details of HSRP primary distribution router dlt85
interface Port-channel1
ip address 10.70.0.53 255.255.0.0
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3
hold-queue 300 in
!
interface Port-channel2
ip address 10.71.0.53 255.255.0.0
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3
hold-queue 300 in
!
interface GigabitEthernet0/0/1
no ip address
no ip directed-broadcast
channel-group 1
!
interface GigabitEthernet1/0/0
ip address 10.60.0.53 255.255.0.0
no ip redirectsno ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3
standby timers 1 3
standby priority 200
standby preempt
standby ip 10.60.0.200
standby track Port-channel1 75
standby track Port-channel2 75
!
Test FailureTime of
InterruptionRecovery
Mechanism
Wiring Closet Uplink
Fail 3s HSRP
Restore 0s HSRP
Distribution Switch
Fail 3s HSRP
Restore 11s (See note below)
Core Switch
Fail 10s EIGRP
Restore 1s EIGRP
8/7/2019 GIGABIT CAMPUS NETWORK DESIGN 1
18/26
Public
Copyright 2000 Cisco Systems, Inc. All Rights Reserved.Page 18 of 26
interface GigabitEthernet1/0/1
no ip address
no ip directed-broadcast
channel-group 1
!
interface GigabitEthernet2/0/0
ip address 10.61.0.53 255.255.0.0
no ip redirects
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3
standby timers 1 3
standby priority 200
standby preempt
standby ip 10.61.0.200
standby track Port-channel1 75
standby track Port-channel2 75
!
interface GigabitEthernet2/0/1
no ip address
no ip directed-broadcast
channel-group 2
!interface GigabitEthernet3/0/0
ip address 10.62.0.53 255.255.0.0
no ip redirects
no ip directed-broadcast
ip hello-interval eigrp 1 1
ip hold-time eigrp 1 3
standby timers 1 3
standby priority 200
standby preempt
standby ip 10.62.0.200
standby track Port-channel1 75
standby track Port-channel2 75
!
interface GigabitEthernet3/0/1
no ip address
no ip directed-broadcast
channel-group 2
!
interface Ethernet0
ip address 172.26.196.53 255.255.254.0
no ip directed-broadcast
!
!
router eigrp 1
passive-interface GigabitEthernet0/0/0
passive-interface GigabitEthernet1/0/0
passive-interface GigabitEthernet2/0/0
passive-interface GigabitEthernet3/0/0network 10.0.0.0
!
end
8/7/2019 GIGABIT CAMPUS NETWORK DESIGN 1
19/26
Public
Copyright 2000 Cisco Systems, Inc. All Rights Reserved.Page 19 of 26
1A: Dual-Path Layer 2 CoreLayout, Catalyst 6500
1B: Dual-Path Layer 2 CoreLogical, Catalyst 6500
aw3
aw4
aw5
aw6
ae3
ae4
ae5
ae6
Test1
Test2
Test3
Test4
Test5
Test6
West Block
Standard
East Block
VLAN
dwt65
dwb65
ct65
cb65
det65
deb65
Access Distribution Core Distribution Access
Gigabit VLAN Trunk
Gigabit Ethernet
Gigabit EtherChannelDual
aw3
aw4
aw5
aw6
ae3
ae4
ae5
ae6
10.10.0.101
10.11.0.101
10.12.0.101
10.13.0.101
10.50.0.101
10.51.0.101
10.52.0.101
10.53.0.101
Test4
Test5
Test6
West Block
Standard
East Block
VLAN
dwt65
dwb65
ct65
cb65
det65
deb65
Access Distribution Core Distribution Access
Gigabit VLAN Trunk
Gigabit Ethernet
Gigabit EtherChannelDual
10.10.0.0
172.26.196.11
10.x.0.25
10.x.0.200 Pri
172.26.196.25
10.x.0.29
10.x.0.200 Pri
172.26.196.29
10.x.0.26
10.x.0.200 Sec
172.26.196.26
10.x.0.30
10.x.0.200 Sec
172.26.196.30
10.31.0.0
172.26.196.27
10.32.0.0
172.26.196.28
10.11.0.0172.26.196.12
10.12.0.0172.26.196.13
10.13.0.0172.26.196.14
10.5x.0.0
172.26.196.21
10.5x.0.0172.26.196.22
10.5x.0.0
172.26.196.23
10.5x.0.0172.26.196.24
Test1
Test2
Test3
8/7/2019 GIGABIT CAMPUS NETWORK DESIGN 1
20/26
Public
Copyright 2000 Cisco Systems, Inc. All Rights Reserved.Page 20 of 26
1C: Dual-Path Layer 2 CoreBackbone, Catalyst 6500
2A: Layer 3 CoreLayout, Catalyst 6500
West Block East Block
10.31.0.0
10.32.0.0
VLAN 31
VLAN 31 VLAN 31
VLAN 31
VLAN 32 VLAN 32
VLAN 32 VLAN 32
ct65
cb65
dwt65 det65
dwb65 deb65
Gigabit VLAN TrunkGigabit Ethernet
Gigabit EtherChannelDual
aw3
aw4
aw5
aw6
ae3
ae4
ae5
ae6
Test1
Test2
Test3
Test4
Test5
Test6
West Block
Standard
East Block
VLAN
dwt65
dwb65
ct65
cb65
det65
deb65
Access Distribution Core Distribution Access
Gigabit VLAN Trunk
Gigabit Ethernet
Gigabit EtherChannelDual
8/7/2019 GIGABIT CAMPUS NETWORK DESIGN 1
21/26
Public
Copyright 2000 Cisco Systems, Inc. All Rights Reserved.Page 21 of 26
2B: Layer 3 CoreLogical, Catalyst 6500
2C: Layer 3 CoreBackbone, Catalyst 6500
aw3
aw4
aw5
aw6
ae3
ae4
ae5
ae6
10.10.0.101
10.11.0.101
10.12.0.101
10.13.0.101
10.50.0.101
10.51.0.101
10.52.0.101
10.53.0.101
Test4
Test5
Test6
West Block
Standard
East Block
VLAN
dwt65
dwb65
ct65
cb65
det65
deb65
Access Distribution Core Distribution Access
Gigabit VLAN Trunk
Gigabit Ethernet
Gigabit EtherChannelDual
10.10.0.0
172.26.196.11
10.x.0.25
10.x.0.200 Pri
172.26.196.25
10.x.0.29
10.x.0.200 Pri
172.26.196.29
10.x.0.26
10.x.0.200 Sec
172.26.196.26
10.x.0.30
10.x.0.200 Sec
172.26.196.30
10.x.0.27
172.26.196.27
10.x.0.28
172.26.196.28
10.11.0.0
172.26.196.12
10.12.0.0
172.26.196.13
10.13.0.0
172.26.196.14
10.5x.0.0
172.26.196.21
10.5x.0.0
172.26.196.22
10.5x.0.0
172.26.196.23
10.5x.0.0
172.26.196.24
Test1
Test2
Test3
West Block East Block
VLAN 20 VLAN 20
VLAN 30
VLAN 30
VLAN 40VLAN 40
VLAN 23 VLAN 23 VLAN 43VLAN 43
ct65
cb65
dwt65 det65
dwb65 deb65
Gigabit VLAN Trunk
Gigabit Ethernet
Gigabit EtherChannelDual
10.20.0.0 10.40.0.0
10.23.0.0 10.43.0.0
10.30.0.0
8/7/2019 GIGABIT CAMPUS NETWORK DESIGN 1
22/26
Public
Copyright 2000 Cisco Systems, Inc. All Rights Reserved.Page 22 of 26
3A: Dual-Path Layer 3 CoreLayout, Catalyst 6500
3B: Dual-Path Layer 3 CoreLogical, Catalyst 6500
aw3
aw4
aw5
aw6
ae3
ae4
ae5
ae6
Test1
Test2
Test3
Test4
Test5
Test6
West Block
Standard
East Block
VLAN
dwt65
dwb65
ct65
cb65
det65
deb65
Access Distribution Core Distribution Access
Gigabit VLAN Trunk
Gigabit Ethernet
Gigabit EtherChannelDual
8/7/2019 GIGABIT CAMPUS NETWORK DESIGN 1
23/26
Public
Copyright 2000 Cisco Systems, Inc. All Rights Reserved.Page 23 of 26
3C: Dual-Path Layer 3 CoreBackbone, Catalyst 6500
A1A: Layer 3 CoreLayout, Catalyst 8540
West Block East Block
VLAN 22
VLAN 22
VLAN 20 VLAN 20
VLAN 30
VLAN 30
VLAN 41
VLAN 42VLAN 21
VLAN 21
VLAN 40VLAN 40
VLAN 23 VLAN 23 VLAN 43VLAN 43
VLAN 42
VLAN 41
ct65
cb65
dwt65 det65
dwb65 deb65
Gigabit VLAN Trunk
Gigabit Ethernet
Gigabit EtherChannelDual
10.20.0.0 10.40.0.0
10.23.0.0 10.43.0.0
10.42.0.0
10.41.0.0
10.30.0.0
10.21.0.0
10.22.0.0
al4
al5
al6
ar4
ar5
ar6
Target1
Target2
Target3
Target4
Left Block Right Block
dlt85
dlb85
ct85
cb85
drt85
drb85
Access Distribution Core Distribution Access
Gigabit VLAN Trunk
Gigabit Ethernet
Gigabit EtherChannelDual
8/7/2019 GIGABIT CAMPUS NETWORK DESIGN 1
24/26
Public
Copyright 2000 Cisco Systems, Inc. All Rights Reserved.Page 24 of 26
A1B: Layer 3 CoreLogical, Catalyst 8540
A1C: Layer 3 CoreTopology, Catalyst 8540
al4
al5
al6
ae4
ae5
ae6
10.60.0.101
10.61.0.101
10.62.0.101
10.100.0.101
10.101.0.101
10.102.0.101
Target3
Target4
Left Block Right Block
dlt85
dlb85
ct65
cb85
drt85
drb85
Access Distribution Core Distribution Access
Gigabit VLAN Trunk
Gigabit Ethernet
Gigabit EtherChannelDual
10.x.0.53
10.x.0.200 Pri
172.26.196.53
10.x.0.59
10.x.0.200 Pri
172.26.196.59
10.x.0.54
10.x.0.200 Sec
172.26.196.54
10.x.0.60
10.x.0.200 Sec
172.26.196.60
10.x.0.57
172.26.196.57
10.x.0.58
172.26.196.58
10.60.0.0
172.26.196.50
10.61.0.0
172.26.196.51
10.62.0.0
172.26.196.52
10.100.0.0
172.26.196.61
10.101.0.0
172.26.196.62
10.102.0.0
172.26.196.63
Target1
Target2
Left Block Right Block
dlt85 ct85 drt85
10.90.0.0
10.80.0.0
10.70.0.0
Port-chan 1
int gig 0/0/1
int gig 1/0/1
Port-chan 1
int gig 0/0/0
int gig 1/0/0
Port-chan 3
int gig 0/0/1
int gig 1/0/1
Port-chan 1
int gig 0/0/0
int gig 1/0/0
dlb85 cb85 drb85
10.93.0.010.73.0.0
Port-chan 2
int gig 2/0/1
int gig 3/0/1
Port-chan 2
int gig 2/0/0
int gig 3/0/0
Port-chan 4
int gig 2/0/1
int gig 3/0/1
Port-chan 2
int gig 2/0/0
int gig 3/0/0
Port-chan 5
int gig 9/0/0
int gig 9/0/1
Port-chan 5
int gig 9/0/0
int gig 9/0/1
Gigabit VLAN Trunk
Gigabit Ethernet
Gigabit EtherChannelDual
8/7/2019 GIGABIT CAMPUS NETWORK DESIGN 1
25/26
Public
Copyright 2000 Cisco Systems, Inc. All Rights Reserved.Page 25 of 26
A2A: Dual-Path Layer 3 CoreLayout, Catalyst 8540
A2B: Dual-Path Layer 3 CoreLogical, Catalyst 8540
Left Block Right Block
al4
al5
al6
ar4
ar5
ae6
Target1
Target2
Target3
Target4
dlt85 ct85
dlb85 cb85
drt85
drb85
Access Distribution Core Distribution Access
Gigabit VLAN Trunk
Gigabit Ethernet
Gigabit EtherChannelDual
al4
al5
al6
ar4
ar5
ar6
10.60.0.101
10.61.0.101
10.62.0.101
10.100.0.101
10.101.0.101
10.102.0.101
Target3
Target4
Left Block Right Block
dlt85
dlb85
ct85
cb85
drt85
drb85
Access Distribution Core Distribution Access
Gigabit VLAN Trunk
Gigabit Ethernet
Gigabit EtherChannelDual
10.x.0.5310.x.0.200 Pri
172.26.196.53
10.x.0.5910.x.0.200 Pri
172.26.196.59
10.x.0.54
10.x.0.200 Sec
172.26.196.54
10.x.0.60
10.x.0.200 Sec
172.26.196.60
10.x.0.57
172.26.196.57
10.x.0.58
172.26.196.58
10.60.0.0
172.26.196.50
10.61.0.0
172.26.196.51
10.62.0.0
172.26.196.52
10.100.0.0
172.26.196.61
10.101.0.0
172.26.196.62
10.102.0.0
172.26.196.63
Target1
Target2
8/7/2019 GIGABIT CAMPUS NETWORK DESIGN 1
26/26
Cisco Systems has more than 200 offices in the following countries. Addresses, phone numbers, and fax numbers are listed on the
C i s c o C o n n e c t i o n O n l i n e W e b s i t e a t h t t p : / / w w w . c i s c o . c o m / o f f i c e s .
Argentina Australia Austria Belgium Brazil Canada Chile China Colombia Costa Rica Croatia Czech Republic Denmark Dubai, UAE Finland France
Germany Greece Hong Kong Hungary India Indonesia Ireland Israel Italy Japan Korea Luxembourg Malaysia Mexico The Netherlands New
Corporate Headquarters
Cisco Systems, Inc.170 West Tasman DriveSan Jose, CA 95134-1706USAhttp://www.cisco.comTel: 408 526-4000
800 553-NETS (6387)Fax: 408 526-4100
European Headquarters
Cisco Systems Europe s.a.r.l.Parc Evolic, Batiment L1/L216 Avenue du QuebecVillebon, BP 70691961 Courtaboeuf CedexFrancehttp://www-europe.cisco.comTel: 33 1 69 18 61 00Fax: 33 1 69 28 83 26
Americas
HeadquartersCisco Systems, Inc.170 West Tasman DriveSan Jose, CA 95134-1706USAhttp://www.cisco.comTel: 408 526-7660Fax: 408 527-0883
Asia Headquarters
Nihon Cisco Systems K.K.Fuji Building, 9th Floor3-2-3 MarunouchiChiyoda-ku, Tokyo 100Japanhttp://www.cisco.comTel: 81 3 5219 6250Fax: 81 3 5219 6001
A2C: Dual-Path Layer 3 CoreBackbone, Catalyst 8540
Left Block Right Block
Port 2
Port 1
Port 1 Port 1
Port 5
Port 5
Port 4
Port 3Port 1
Port 2
VLAN 40Port 3
Port 2 Port 2 Port 2Port 4
Port 2
Port 1
ct85
cb85
dlt85 drt85
dlb85 drb85
Gigabit VLAN Trunk
Gigabit Ethernet
Gigabit EtherChannelDual
10.70.0.0 10.90.0.0
10.73.0.0 10.93.0.0
10.92.0.0
10.91.0.0
10.80.0.0
10.71.0.0
10.72.0.0
Conclusion
Choose a deterministic, structured design model to achieve high
availability in your enterprise network. Apply redundancy in the
mission-critical parts of the network. Scale the enterprise
network to the size required by choosing the appropriate
building block model and combining with the right backbone
model. For a better understanding of the different choices
referred to in this paper refer to the paper Gigabit Campus
Network DesignPrinciples and Architecture.
Geoff Haviland ([email protected])Network Design
Engineer.