Cisco ASR1000 and Microsoft Azure ExpressRoute Design …on the Azure side. To connect to Microsoft Azure services using ExpressRoute, Microsoft provides best practices for network

© 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 1 of 35

Cisco ASR1000 and Microsoft Azure ExpressRoute

Design and Deployment Guide

Extend your enterprise network into Azure with

Cisco ASR®1000

Written by Jason Yang and Kevin Echols II

July 2018

Design and Deployment Guide


Contents

Executive summary ................................................................................................................................................. 3 Cisco Multicloud Portfolio: Overview ..................................................................................................................... 3 Cloud Connect: Overview ..................................................................................................................................... 4 Cloud Connect: Use cases.................................................................................................................................... 4 Cloud Connect: Benefits ....................................................................................................................................... 4

Introduction .............................................................................................................................................................. 5 Target Audience .................................................................................................................................................... 5 Purpose of This Document .................................................................................................................................... 5 Solution Overview ................................................................................................................................................. 5

Product Overview .................................................................................................................................................... 6

Preparation ............................................................................................................................................................... 7 Getting Started ...................................................................................................................................................... 8

Configuration: ExpressRoute Peering on Azure ................................................................................................... 9

Configuration: Cisco ASR1000 ............................................................................................................................... 9 Two Router Deployment vs. One Router Deployment .......................................................................................... 9 Interface Configurations ...................................................................................................................................... 10

802.1Q-in-Q VLAN ID Sample Interface Configuration .................................................................................. 10 802.1Q VLAN ID Sample Interface Configuration .......................................................................................... 11

BGP Configurations ............................................................................................................................................ 12 Setup eBGP Sessions .................................................................................................................................... 12 Advertise Prefixes Over the BGP Session to Azure ....................................................................................... 13 Filter Prefixes Received from Azure (Optional) .............................................................................................. 13 High Availability and Optimize Routing Configuration .................................................................................... 14 AS Path Prepending to Influence Routing ...................................................................................................... 15 Avoid Asymmetric Routing ............................................................................................................................. 16

NAT Configuration .............................................................................................................................................. 17 NAT Common Best Practices ......................................................................................................................... 18

Route Redistribution into EIGRP ......................................................................................................................... 18

Value-Added Feature Configurations ................................................................................................................... 19 Configure Flexible Netflow .................................................................................................................................. 19 Configure Quality of Service ............................................................................................................................... 20

Advanced Services Configurations ...................................................................................................................... 21 Configure Application Visibility and Control (AVC) .............................................................................................. 21 Configure IPsec VPN .......................................................................................................................................... 22

Test Connectivity ................................................................................................................................................... 24 Verify the BGP Neighbors ................................................................................................................................... 24 Verify ExpressRoute Connectivity ....................................................................................................................... 30 Verify NAT Translation Entries and Pool ............................................................................................................. 32 Verify Netflow Entries .......................................................................................................................................... 33

ASR1000 Proactive System Monitoring ............................................................................................................... 34

References ............................................................................................................................................................. 35


Executive summary

This guide focuses on how to extend your on premises network into the Microsoft Azure Virtual Private Cloud

(VPC) with ExpressRoute using the Cisco ASR1000 Series Routers with VPN connectivity back to a private data

center at the corporate site. The design uses an ExpressRoute managed VPN connection through a virtual private

gateway (VGW) attached to the VPC. Advanced features such as Application Visibility & Control (AVC)/Next-

Generation Network-Based Application Recognition (NBAR2), and Flexible NetFlow data export are also discussed

for traffic and application-level visibility at the ASR 1000 Series routers within the private data center.

Cisco Multicloud Portfolio: Overview

In a multicloud world, growing complexity is driving a cloud gap between what your customers require and what

your people, processes, and tools can support. With the Cisco Multicloud Portfolio, we make it simple: simple to

connect, simple to protect, and simple to consume.

The Cisco Multicloud Portfolio is a set of essential products, software, and services supported with simplified

ordering and design deployment guides to help you when it comes to multicloud adoption. The Cisco Multicloud

Portfolio consists of four component portfolios (Figure 1):

● Cloud Advisory: Helps you design, plan, accelerate, and remove risk from your multicloud migration.

● Cloud Connect: Securely extends your private networks into public clouds and helps ensure the

appropriate application experience.

● Cloud Protect: Protects your multicloud identities, direct-to-cloud connectivity, data, and applications,

including Software as a Service (SaaS).

● Cloud Consume: Helps you deploy, monitor, and optimize applications in multicloud environments.

Figure 1. Multicloud Portfolio: Cloud Advisory, Cloud Connect, Cloud Protect, and Cloud Consume

https://azure.microsoft.com/en-us/services/expressroute/

https://www.cisco.com/c/en/us/products/routers/asr-1000-series-aggregation-services-routers/index.html


Cloud Connect: Overview

Cloud Connect consists of essential products that help securely extend your private networks – including data

center, branches, and campuses – to public clouds and to help ensure that the application experience is optimal:

● Cisco Cloud Services Router (CSR) 1000V Series

● Viptela® vEdge with Cisco Umbrella™

For detailed use cases, see the section about Cloud Connect on the portfolio’s solution page at

https://www.cisco.com/go/multicloud.

Cloud Connect: Use cases

Cloud Connect delivers value in the following use cases:

● Securely extending a private network to single or multiple public cloud environments. Includes multiple

clouds (for example, multiple AWS and Azure), multiple regions in a cloud, or multiple VPCs in a cloud;

VPN; multicloud and multi-VPC connectivity; scaling; and performance optimization-transit VPC. Also

supports extending data centers into the cloud and enabling direct branch-to-cloud connectivity (when a

branch has a Cisco 4000 Series Integrated Services Router [ISR] and wants to connect the clouds or a

branch has vEdge and requires a software-defined WAN [SD-WAN] extension to the cloud).

● Optimizing data center and branch connectivity performance to cloud Infrastructure as a Service (IaaS) and

SaaS. Includes best path to destination (SD-WAN), cloud segmentation, monitoring to assure best

performance, visibility into traffic going to applications, and traffic shaping/Quality of Service (QoS). Also

supports extending data centers into the cloud and enabling direct branch-to-cloud connectivity (when a

branch has a 4000 Series ISR and wants to connect the clouds or a branch has vEdge and requires an SD-

WAN extension to the cloud).

● Securing access to the Internet and SaaS from the branch. Includes connecting and protecting branch office

users directly to the multicloud environment using Direct Internet Access (DIA), SD-WAN (vEdge), and

secure Internet gateways (Cisco Umbrella).

Cloud Connect: Benefits

Cloud Connect benefits include the ability to:

● Extend a private network to a multicloud environment while leveraging existing investments

● Apply consistent security policies across a private and public cloud footprint

● Enhance and secure the app experience on a cloud network by enabling visibility and path selection and

optimization

● Centralize management in a manner that is intuitive, fast, and easy to design, provision, and apply policies

across the entire network

● Achieve faster and more simple adoption of cloud

● Improve TCO

● Access a richer networking security feature set and higher performance

● Improve ease of use through consistency of management tools for both on-premises and cloud

● Simplify implementation through increased visibility into the public cloud network

https://www.cisco.com/go/multicloud


Introduction

The majority of enterprises have started deploying business-critical applications that span on premise equipment

and cloud infrastructure in what is known as a hybrid cloud deployment. These enterprises seek to benefit from

reduced Total Cost of Ownership (TCO), the ability to scale applications to meet growing demands, and an always

on guarantee via distributed workloads across multiple availability zones and geographic regions.

Establishing a reliable connection from on premise to the cloud has proven difficult for many of these enterprises

as the Internet does not guarantee the metrics required for crucial business applications. Cisco and Microsoft have

partnered to make the transition to a hybrid cloud deployment easier for our mutual customers by creating a

jointly-validated designs between Microsoft Azure ExpressRoute and Cisco CSR100v and ASR1000.

Microsoft’s ExpressRoute lets you extend your on premise networks into Microsoft Azure over a private connection

facilitated by a connectivity provider. With ExpressRoute, you can establish connections to Microsoft cloud

services, such as Microsoft Azure, Office 365, and Dynamics 365.

Target Audience

The intended audience for this document includes sales engineers, field consultants, professional services staff, IT

managers, partner engineering staff, and customers deploying the Microsoft Azure ExpressRoute with Cisco

ASR1000 routers. External references are provided wherever applicable, but readers are expected to be familiar

with the technology, infrastructure, and enterprise security policies of the customer installation.

Purpose of This Document

Cisco-Microsoft Joint Validated Designs provide guidelines for creating an end-to-end solution that enable you to

make informed decisions with the goal of successfully creating a hybrid cloud deployment.

This document describes the steps required to extend your on premises network into the Microsoft Azure with

ExpressRoute using the Cisco ASR1000 Series Routers on premise in your data center and the Cisco CSR 1000v

on the Azure side. To connect to Microsoft Azure services using ExpressRoute, Microsoft provides best practices

for network security, optimize routing, asymmetric routing, and NAT. This guide will focus on how to implement

these best practices with ASR1000 configurations, recommend advanced features and services on the ASR1000.

Please note that this guide is not meant to be a comprehensive overview of the ASR1000 platform and routing

technologies, see References section for platform and feature configuration guides.

Cisco validation provides further confirmation of solution compatibility, connectivity, and correct operation for the on

premise deployment. Although readers of this document are expected to have sufficient knowledge to install and

configure the products used, the Cisco-Microsoft Design and Deployment Guide provides configuration details that

are important to the deployment of this solution.

Solution Overview

ExpressRoute supports layer 3 connectivity between your on premise network and Microsoft Azure through a

connectivity provider in 3 connectivity models: CloudExchange Co-location, Point-to-point Ethernet Connection,

and IP VPN Connection. ExpressRoute connections do not go over the public Internet, which allows ExpressRoute

connections to offer more reliability, faster speeds, lower latencies, and higher security than typical connections

over the Internet. As shown in Figure 1, ExpressRoute circuits have multiple routing domains associated with them:

Azure private peering, and Microsoft peering. Each of the routing domains are configured in separate Virtual

Routing and Forwarding (VRF) domains on a pair of ASR1000 routers for high availability. In Figure 1, these

routers are shown located in the partner edge block.

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-introduction

https://www.cisco.com/c/en/us/products/routers/asr-1000-series-aggregation-services-routers/index.html

https://docs.microsoft.com/en-us/azure/best-practices-network-security

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-optimize-routing

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-asymmetric-routing

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-nat

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-connectivity-models


Figure 2. Common ExpressRoute Deployment

ExpressRoute capabilities and features are identical across all of the connectivity models. The ASR1000 physical

connectivity configuration to each of the service providers may vary, but the configuration to ExpressRoute will be

identical.

Product Overview

Cisco ASR1000 Series Aggregation Services Routers aggregate multiple WAN connections and network services,

including encryption and traffic management, and forward them across WAN connections at line speeds from 2.5 to

200Gbps. ASR1000 Series routers offer elastic service delivery; programmability and automation; up to five-nines

availability; comprehensive and flexible QoS; and advanced services, such as IPsec VPN and Application Visibility

and Control (AVC) for enterprise networks.

The Cisco ASR1000 Series platforms vary in I/O connectivity speed, density, system performance, and

redundancy options. All models use the Cisco Quantum Flow Processor and support the same feature set

available on the Cisco IOS XE Operating System. All this commonality simplifies management and operations.

ExpressRoute circuits are purchased based on a number of bandwidth options. Table 1 outlines ASR1000 platform

recommendations for each of the ExpressRoute bandwidth options.

Table 1. ExpressRoute Circuit Bandwidth to ASR1000 Platform Recommendations

ExpressRoute Circuit Bandwidths ASR1000 Platform Interface Type

50 Mbps ASR 1001-X GigabitEthernet




1 Gbps ASR 1001-X or

ASR1001-HX

GigabitEthernet or

TenGigabitEthernet

2 Gbps ASR 1001-HX TenGigabitEthernet

https://developer.cisco.com/site/ios-xe/

https://www.cisco.com/c/en/us/products/collateral/routers/asr-1002-router/solution_overview_c22-449961.html

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/sec_conn_vpnips/configuration/xe-3s/sec-sec-for-vpns-w-ipsec-xe-3s-book/sec-cfg-vpn-ipsec.html

https://www.cisco.com/c/en/us/td/docs/ios/solutions_docs/avc/guide/avc-user-guide/avc_config.html

https://www.cisco.com/c/en/us/td/docs/ios/solutions_docs/avc/guide/avc-user-guide/avc_config.html


ExpressRoute Circuit Bandwidths ASR1000 Platform Interface Type



The ASR1001X, pictured in Figure 2, is a 1 RU form factor, supports redundant power supplies, and the Embedded

Services Processor (ESP) has default throughput of 2.5Gbps that is upgradable to 5-, 10-, or 20Gbps via software

activation. The platform consumes 250W at max with front-to-back airflow. Onboard, the ASR1001X has 6 Gigabit

Ethernet SFP ports, 2 TenGigabit Ethernet SFP+ ports, and has a single half-height Shared Port Adapter (SPA)

that can be configured with a range of interfaces from a 2-port Gigabit Ethernet SPA to a T1/E1 NIM. See the

ASR1001X Datasheet for more details on the platform, and the ASR1001X Hardware Installation Guide from a

complete list of supported hardware.

Figure 3. ASR1001X

The ASR1001HX, pictured in Figure 3, is a 1 RU form factor, supports redundant power supplies, and the

Embedded Services Processor (ESP) has throughput up to 60Gbps. The platform consumes 360W at max with

front-to-back airflow. Onboard, the ASR1001HX has 8 Gigabit Ethernet SFP ports and 8 TenGigabit Ethernet SFP+

ports, where 4 of the TenGigabit Ethernet ports (Te4-7) are compatible with SFPs. See the ASR1001HX Datasheet

for more details on the platform, and the ASR1001HX Hardware Installation Guide from a complete list of

supported hardware.

Figure 4. ASR1001HX

Preparation

The configuration guide will include numerous value substitutions provided for the purpose of example only. Any

references to IP addresses, device IDs, shared secrets or keys account information or project names should be

replaced with the appropriate values for your environment when following this guide. Values unique to your

environment will be highlighted in bold.

This guide is not meant to be a comprehensive setup for entire device configuration for all network connectivity, for

example, the same device may also have connectivity to the enterprise data center, campus, or branches, the

configuration of which is outside the scope of this guide. This configuration guide will focus on the connectivity to

the ExpressRoute. List 1 provides a high-level overview of the configuration process that will be covered.


List 1: High-Level Overview of ASR1000 Configuration Process

1. Interface Configurations

a. 802.1Q-in-Q VLAN ID Sample Interface Configuration

b. 802.1Q VLAN ID Sample Interface Configuration

2. BGP Configurations

a. Setup eBGP Sessions

b. Advertise Prefixes Over the BGP Session to Azure

c. Filter Prefixes Received from Azure (Optional)

d. High Availability and Optimize Routing Configuration

e. AS Path Prepending to Influence Routing

f. Avoid Asymmetric Routing

g. NAT Configuration

h. NAT Common Best Practices

3. Route Redistribution into EIGRP

4. Advanced Feature Configurations

a. Flexible Netflow Configuration

b. Quality of Service Configuration

5. Advanced Services Configurations

a. Application Visibility and Control (AVC) Configuration

b. IPsec VPN Configuration

Getting Started

It is assumed that you met all the requirements in ExpressRoute prerequisites & checklist, the ExpressRoute

circuits have been created, and the ExpressRoute circuit provisioned by the service provider.

The first step in configuring your Cisco ASR1000 for use with the ExpressRoute connectivity is to ensure that the

following prerequisite conditions have been met:

The essential feature set (BGP, NAT, VRF-Lite, IPv4/IPv6 dual-stack) required for setting up the ExpressRoute

connectivity and the advanced features are supported by the ASR1000 universal image, that is, no additional

license is required.

The advanced services require AES license in addition to base licenses:

1. NBAR/AVC requires AVC feature license

2. The IPsec application requires:

a. Advanced Enterprise Services(SLASR1-AES) or Advanced IP Services Technology Package License

(SLASR1-AIS)

b. IPsec RTU license (FLASR1-IPSEC-RTU)

c. Encryption HW module (ASR1001HX-IPSECW) and Tiered Crypto throughput license which applies to

ASR1001-HX chassis

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-prerequisites

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-howto-circuit-portal-resource-manager

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-howto-circuit-portal-resource-manager


Refer to the ASR1000 Routers Ordering Guide for more details on ASR1000 Series Router license information.

The recommended software image is 16.6.2 and onward. Suggested images are recommended on the on

cisco.com download software page, where the suggested image is labeled by icon.

Configuration: ExpressRoute Peering on Azure

Follow the ExpressRoute peering steps in Azure portal.

Configuration: Cisco ASR1000

Two Router Deployment vs. One Router Deployment

We recommend the deployment of two ASR1000s in a redundant pair to connect to the ExpressRoute service.

Each router will need two QinQ subinterfaces on the physical interface. At the Microsoft Edge (see Figure 1) an

ExpressRoute service is terminated on a pair of Microsoft ExpressRoute Edge (MSEE) routers. The MSEE routers

hand off to a pair of Connectivity Provider routers, and then down to the customer’s ASR1000 routers. Microsoft

will always have two BGP sessions for each of the peering types.

As an example, assume Connectivity Provider defines an outer dot1Q tag of 10 for ER circuit, and the customer

requests an inner tag of 310 for the Microsoft peering, and 3101 for the private peering. Table 2 outlines the

example of mapping of Interfaces, subinterfaces, VRFs and their respective peering to ER in the customer edge

dual router design.

Table 2. Router, Interface, Subinterfaces, VRFs and Peering for Customer Edge Dual Router Design

Routers R1 R2

Interfaces TE0/1/0 TE0/1/1 TE0/1/0 TE0/1/1

Interface description

Connection to ER Primary Connection to customer corp network

Connection to ER Secondary

Connection to customer corp network

Subinterfaces 0/1/0.310 0/1/0.3101 0/1/1.10 0/1/1.101 0/1/0.310 0/1/0.3101 0/1/1.10 0/1/1.101

Subinterface description

Primary Microsoft Peering

Primary Private Peering

DMZ VLAN Corp VLAN Secondary Microsoft Peering

Secondary Private Peering

DMZ VLAN Corp VLAN

Encapsulation dot1Q 10 second-dot1q 310

or

dot1Q 310

dot1Q 10 second-dot1q 3101

or

dot1Q 3101

dot1Q 10 dot1Q 101 dot1Q 10 second-dot1q 310

or

dot1Q 310


or

dot1Q 3101

dot1Q 10 dot1Q 101

VRFs* C10 C101 C10 C101 C10 C101 C10 C101

IP Addresses 216.221.237.33/30

172.16.0.1/30 192.168.0.1/30

192.168.0.5/30

216.221.237.37/30

172.16.0.5/30 192.168.0.1/30

192.168.0.5/30

Note: It is best practice to separate private peering and Microsoft peering with two separate VRFs. The

private peering is considered trusted, whereas the Microsoft peering is a public network. The customer can

send each VRFs/VLANs to the appropriate security zone before entering/exiting their corporate VLANs.

Unless otherwise stated, this configuration guide provides configuration example on Router R1. Router R2 should

have the same configuration as R1, with the exception of IP addresses/subnets. The subinterface, IP address, and

VRF will use the example provided in Table 2.

Optionally, if the customer chooses to deploy one router for connection to ER circuit, Table 3 outlines an

example of mapping of Interfaces, subinterfaces, VRFs and their respective peering to ER in single customer

edge router design.

https://www.cisco.com/c/en/us/products/collateral/routers/asr-1000-series-aggregation-services-routers/guide-c07-731639.html

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-howto-routing-portal-resource-manager


Table 3. Interface, Subinterfaces, VRFs and Peering for Customer Edge Single Router Design

Interfaces TE0/1/0 TE0/1/1 TE0/1/2

Interface description Connection to ER Primary Connection to customer corp network

Connection to ER Secondary

Subinterfaces 0/1/0.310 0/1/0.3101 0/1/1.10 0/1/1.101 0/1/0.310 0/1/0.3101

Subinterface description Primary Microsoft Peering

Primary Private Peering

DMZ VLAN Corp VLAN Secondary Microsoft Peering

Secondary Private Peering

Encapsulation dot1Q 10 second-dot1q 310

or

dot1Q 310


or

dot1Q 3101

dot1Q 10 dot1Q 101 dot1Q 10 second-dot1q 310

or

dot1Q 310


or

dot1Q 3101

VRFs C10 C101 C10 C101 C10 C101

IP Addresses 216.221.237.33/30

172.16.0.1/30 192.168.0.1/30 192.168.0.5/30 216.221.237.37/30

172.16.0.5/30

Interface Configurations

This section provides the interface configuration of Cisco ASR1000 to connect to ER. At least one internal facing

interface is required to connect to your own network, and one external facing interface is required to connect to

ExpressRoute.

You will require a subinterface per peering in every router you connect to ER. A subinterface can be identified with

an 802.1Q-in-Q VLAN ID or 802.1Q VLAN ID based on the connectivity providers’ requirement and an IP address.

Follow ER IP address requirements for the BGP peering.

802.1Q-in-Q VLAN ID Sample Interface Configuration

ip vrf C10

rd 65021:10

!

ip vrf C101

rd 65021:101

!

interface TenGigabitEthernet0/1/0

description connection to ER Primary

no ip address

dot1q tunneling ethertype 0x9100

! The default ethertype is 0x8100, can be changed to 0x88A8|0x9100|0x9200 to meet

the connectivity provider’s requirement

!

interface TenGigabitEthernet0/1/0.310

description Customer 10 Primary Microsoft peering to Azure

encapsulation dot1Q 10 second-dot1q 310

ip vrf forwarding C10

ip address 216.221.237.33255.255.255.252

!


description Customer 10 Primary private peering to Azure


https://docs.microsoft.com/en-us/azure/expressroute/expressroute-routing



ip address 172.16.0.1 255.255.255.252

!


description Customer 10 Corporate facing interface

no ip address

!


description Customer 10 DMZ VLAN

encapsulation dot1Q 10


ip address 192.168.0.1 255.255.255.252

!


description Customer 10 Corp VLAN



ip address 192.168.0.5 255.255.255.252

802.1Q VLAN ID Sample Interface Configuration

ip vrf C10

rd 65021:10

!

ip vrf C101

rd 65021:101

!


description connection to ER

no ip address

!





ip address 216.221.237.33 255.255.255.252

!





ip address 172.16.0.1 255.255.255.252

!


description Customer 10 Corporate facing interface

no ip address


!





ip address 192.168.0.1 255.255.255.252

!


description Customer 10 Corp VLAN



ip address 192.168.0.5 255.255.255.252

Note: The MTU for the ExpressRoute interface is 1500 Bytes, which is the default MTU on ASR1000

Ethernet interface.

BGP Configurations

Setup eBGP Sessions

You must setup a BGP session with Microsoft for every peering. The sample below enables you to setup a BGP

session with Microsoft. If the IPv4 address you used for your subinterface was a.b.c.d, the IP address of the BGP

neighbor (Microsoft) will be a.b.c.d+1. The last octet of the BGP neighbor's IPv4 address will always be an even

number.

Follow ER ASN requirements for the peering.

router bgp 65021

bgp router-id 10.6.32.241

bgp log-neighbor-changes

!

address-family ipv4 vrf C10

neighbor 216.221.237.34 remote-as 12076

neighbor 216.221.237.34 description Microsoft peering to Azure

neighbor 216.221.237.34 local-as 394749

neighbor 216.221.237.34 activate

neighbor 216.221.237.34 password A1B2C3D4

neighbor 216.221.237.34 soft-reconfiguration inbound

redistribute connected

exit-address-family

!


neighbor 172.16.0.2 remote-as 12076

neighbor 172.16.0.2 description private peering to Azure

neighbor 172.16.0.2 local-as 64512

neighbor 172.16.0.2 activate

neighbor 172.16.0.2 password A1B2C3D4

neighbor 172.16.0.2 soft-reconfiguration inbound




exit-address-family

Note: Password configuration is an optional feature for the ER BGP peering and not enabled by default.

See BGP Command Reference for more information to set up a password on the BGP peering.

Advertise Prefixes Over the BGP Session to Azure

Use network statement or redistribution from IGP to advertise your internal network prefixes to Azure.

router bgp 65021

!


network 192.168.0.4 mask 255.255.255.252


redistribute static

Microsoft peering does not accept default route or private IP addresses (RFC 1918), the sample below use prefix-

list to filter them out.

router bgp 65021

!


neighbor 216.221.237.34 prefix-list rfc1918 out

!

ip prefix-list rfc1918 deny 0.0.0.0/8 le 32








ip prefix-list rfc1918 deny 0.0.0.0/0

ip prefix-list rfc1918 permit 0.0.0.0/0 le 32

Microsoft Azure has policy of accepting up to 4000 (10,000 for Premium ExpressRoute) route prefixes for private

peering and 200 route prefixes for Microsoft peering. It is your responsibility to manage and aggregate network

prefix while advertising your internal network, otherwise Microsoft will drop the BGP session once prefix count goes

above the limit.

Filter Prefixes Received from Azure (Optional)

You can use route-maps and prefix lists to filter prefixes propagated into your network. You can use the sample

below to accomplish the task. Ensure that you have appropriate prefix lists setup.

router bgp 65021

!


neighbor 216.221.237.34 route-map <MS_Prefixes_Inbound> in


neighbor 172.16.0.2 route-map <PP_Prefixes_Inbound> in

https://www.cisco.com/c/en/us/td/docs/ios/iproute_bgp/command/reference/irg_book/irg_bgp3.html#wp1108507


!

route-map <PP_Prefixes_Inbound> permit 10

match ip address prefix-list <PP_Prefixes>

!

route-map <MS_Prefixes_Inbound> permit 10

match ip address prefix-list <MS_Prefixes>

High Availability and Optimize Routing Configuration

We recommend that both ASR1000 routers have L3 peering to south bound corporate network router so that

customers can leverage High Availability or Equal Cost Multi-Path to load share traffic to ExpressRoute

Follow ER Optimize Routing from customer to Microsoft, BGP local preference is used to influence the routing.

Make sure you have the correct BGP community for region, e.g. USW is 12076:51006 and USW2 is12076:51026.

A detailed list of region to ER BGP communities can be found here under “Support for BGP Communities” section.

The sample below use BGP community “12076:51004” for the prefixes received from US East, and BGP

community “12076:51006” for the prefixes received from US West. We will assign US West region, e.g.

13.100.0.0/16, to higher local preference in the US West, and assign US East region, e.g. 23.100.0.0/16, to higher

local preference in the US East.

#US West ASR1000

!

router bgp 65021

!


neighbor 216.221.237.34 route-map Peer-USW in

!

ip bgp-community new-format

!

ip community-list 1 permit 12076:51006

!

route-map Peer-USW permit 10

match community 1

set local-preference 400

#US East ASR1000

!

router bgp 65021

!


neighbor 216.221.237.34 route-map Peer-USE in

!

ip bgp-community new-format

!

ip community-list 1 permit 12076:51004

!




route-map Peer-USE permit 10

match community 1

set local-preference 400

AS Path Prepending to Influence Routing

In order to optimize routing from Microsoft to your network, AS Path prepending is used to influence routing.

Microsoft removes private AS numbers in the AS PATH for the prefixes received on Microsoft Peering, so it is

important to append public AS numbers in the AS PATH to influence routing for Microsoft Peering. The sample

below did not follow the AS and IP scheme in Table 2, but based on the Microsoft ER example as shown in

Figure 4.

Figure 5. AS Path Prepending Sample

You can lengthen the AS PATH for 177.2.0.0/31 in US East so that Microsoft will prefer the ExpressRoute circuit in

US West for traffic destined for this prefix (as Microsoft network will think the path to this prefix is shorter in the

west). Similarly, by lengthening the AS PATH for 177.2.0.2/31 in US West so that Microsoft will prefer the

ExpressRoute circuit in US East.

#US West ASR1000

!

router bgp 345

!


neighbor 216.221.237.34 route-map Prepend-USW out

network 177.2.0.0 mask 255.255.255.254

network 177.2.0.2 mask 255.255.255.254



!

ip prefix-list prefix_USW seq 10 permit 177.2.0.2/31

!

route-map Prepend-USW permit 10

match ip address prefix prefix_USW

set as-path prepend 345

!

route-map Prepend-USW permit 20

#US East ASR1000

!

router bgp 345

!


neighbor 216.221.237.134 route-map Prepend-USE out

network 177.2.0.0 mask 255.255.255.254

network 177.2.0.2 mask 255.255.255.254

!

ip prefix-list prefix_USE seq 10 permit 177.2.0.0/31

!

route-map Prepend-USE permit 10

match ip address prefix prefix_USE

set as-path prepend 345

!

route-map Prepend-USE permit 20

Avoid Asymmetric Routing

Follow ER asymmetric routing solutions, in the example, if you want to use the Internet for authentication traffic and

ExpressRoute for your mail traffic or other public services, you should not advertise your Active Directory

Federation Services (AD FS) public IP addresses over ExpressRoute. This best practice can be enforced with an

outbound route-map configuration:

router bgp 65021

!


neighbor 216.221.237.34 route-map AD_FS_Prefixes out

!

ip prefix-list AD_FS permit 121.10.0.1/32

!

route-map AD_FS_Prefixes deny 10

match ip address prefix-list AD_FS

route-map AD_FS_Prefixes permit 20

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-asymmetric-routing


NAT Configuration

As per Microsoft NAT for ExpressRoute, Microsoft expects to support bi-directional connectivity on the Microsoft

peering. Traffic destined to Microsoft cloud services must be SNATed to valid public IPv4 addresses before they

enter the Microsoft network. You can use the sample configuration below to accomplish the task, it is using the MS

peering subinterface ip address as the NAT pool (216.221.237.33), so the returning traffic will be sent back to this

router, un-NATed before forwarded out of the DMZ VLAN.





ip address 216.221.237.33 255.255.255.252

ip nat outside

!





ip address 192.168.0.1 255.255.255.252

ip nat inside

!

ip route vrf C10 216.221.236.33 255.255.255.255 null0

!

ip nat pool Cust10_MSFT_Pool 216.221.236.33 216.221.236.33 netmask

255.255.255.252

!

ip nat inside source route-map Cust10_MSFT_sNAT pool Cust10_MSFT_Pool vrf C10

overload

!

ip access-list extended Local_BGP_C10

remark deny BGP session from being NATed

permit tcp host 216.221.237.33 host 216.221.237.34 eq bgp

permit tcp host 216.221.237.34 host 216.221.237.33 eq bgp

!

access-list 10 permit 216.221.237.34

!

route-map Cust10_MSFT_sNAT deny 5

match ip address Local_BGP_C10

!

route-map Cust10_MSFT_sNAT permit 10

description NAT any traffic in VRF C10 with NH 216.221.237.34 toward Microsoft

Peering

match ip next-hop 10

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-nat


It is your responsibility to ensure that the NAT IP pool advertised to Microsoft is NOT advertised to the Internet

(even as a subnet of the Internet advertisement, they must be completely non-overlapping). Failure to meet this

requirement may break connectivity to other Microsoft services.

NAT Common Best Practices

1. Set the NAT max-entries per system scale, which is 2M on ASR1001-X and ASR1001-HX. Other ASR1000

systems may have different NAT scale, please follow the relevant product datasheet.

ip nat translation max-entries 2000000

2. It is recommended to keep the default NAT timeout. If the user has specific needs to reduce the timer, for

example the pools are being exhausted, then the user can refer to the sample commands below to make

configuration changes:

The default NAT timeout values can be seen in show command

ASR1000#show platform hardware qfp active feature nat data time

Timeouts: default 86400; TCP 86400; TCP PPTP 86400; UDP 300; FINRST 60;

SYN 60; DNS 60; ICMP 60; Skinny 60; ICMP error 60; ESP 300

To change the timeout values for example:

ip nat translation tcp-timeout 10800

3. If there is the requirement that both NAT and non-NATted traffic must co-exist in the NAT outside interface,

then use Gatekeeper to optimize system performance:

ip nat settings gatekeeper-size 65535

Route Redistribution into EIGRP

In order to redistribute routes from the Private and Microsoft BGP Peerings to EIGRP, add the following

configuration

router eigrp 1

!


redistribute static route-map BGP_Private_to_App_EIGRP

redistribute bgp 65021 metric 1000000 100 255 1 1500

network 10.0.0.0 0.0.0.255

no auto-summary

autonomous-system 2

exit-address-family

!


redistribute bgp 65021 metric 1000000 100 255 1 1500

network 10.1.0.0 0.0.0.255

no auto-summary

autonomous-system 3

!

router bgp 65021

!


https://www.cisco.com/c/en/us/products/collateral/routers/asr-1000-series-aggregation-services-routers/datasheet-c78-731640.html

https://www.cisco.com/c/en/us/support/docs/ip/network-address-translation-nat/210869-ASR1k-NAT-intermittently-fails-to-transl.html


redistribute eigrp 2 route-map EIGRP_App_to_BGP

!

ip prefix-list BGP_Private_to_App_EIGRP seq 5 permit 10.3.0.0/23

!

ip access-list extended EIGRP_App_to_BGP

permit ip 10.0.0.0 0.0.0.255 any

!

route-map EIGRP_App_to_BGP permit 10

match ip address EIGRP_App_to_BGP

!

route-map BGP_Private_to_App_EIGRP permit 10

match ip address prefix-list BGP_Private_to_App_EIGRP

!

In order to NAT traffic from your corporate network, adjust the NAT configuration as follows

access-list 11 permit 10.1.0.0 0.0.0.255

route-map Cust10_MSFT_sNAT permit 10

description NAT any traffic in Corp_NET toward public peering

match ip address 11

Value-Added Feature Configurations

Configure Flexible Netflow

Flexible Netflow (FNF) is an embedded instrumentation capability within the ASR1000 to characterize network

operation, to characterize IP traffic, and understand how and where it flows is critical for network availability,

performance, and troubleshooting. The sample below shows how simple it can be to turn on FNF for ASR1000.

flow exporter C10_expo

destination 10.10.10.9 vrf C101

transport udp 9996

!

flow monitor C10_mon

exporter C10_expo

record netflow-original

!



ip flow monitor C10_mon input

ip flow monitor C10_mon output

!



ip flow monitor C10_mon input

ip flow monitor C10_mon output

To be able to see bi-directional traffic in the ASR1000 system, the user can turn on ingress NetFlow on all

interfaces, or if the user is only interested in the bi-directional traffic from and to ER, turn on ingress and egress

NetFlow on ER. We recommend the use of full NetFlow instead of sampled NetFlow.


Configure Quality of Service

Follow ER QoS requirements, a 6-class QoS model, as shown in Table 4, can be implemented to fulfill the

requirements while protecting the mission critical applications and network control traffic in the events of ER circuits

congestion. Use the sample QoS configuration below to accomplish the task.

Table 4. 6-Class QoS Model

Traffic Class DSCP Values Business workload Bandwidth % Congestion avoidance

Voice EF Skype / Lync voice 10 (PQ) -

Video AF41 Interactive Video, VBSS 30 remaining WRED

Network Control CS6 NET-CTRL* 5 remaining -

Transactional Data AF21 App Sharing 25 remaining WRED

Bulk Data AF11 File Transfer 25 remaining WRED

Class-default Catch-all Catch-all 15 remaining WRED

Note: BGP is always marked as CS6 by ASR1000 so it is protected in the NET-CTRL class.

class-map match-any VOICE

match dscp ef

class-map match-any VIDEO

match dscp af41

class-map match-any NETWORK-CONTROL

match dscp cs6

class-map match-any TRANSACTIONAL-DATA

match dscp af21

class-map match-any BULK-DATA

match dscp af11

!

! example of 500Mbps of ER circuit, adapt it to your circuit BW accordingly.

policy-map ER-500MBPS-POLICY

class class-default

shape average 500000000

service-policy ER

!

policy-map ER

class VOICE

priority level 1

police cir percent 10

class VIDEO

bandwidth remaining percent 30

random-detect

class NETWORK-CONTROL


class TRANSACTIONAL-DATA


random-detect

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-qos


class BULK-DATA


random-detect

class class-default


random-detect

set dscp 0

! Microsoft require user to rewrite all other DSCP to 0 before sending the

packets to ER

!



service-policy output ER-500MBPS-POLICY

Advanced Services Configurations

Configure Application Visibility and Control (AVC)

If the DSCP values for applications above have not been marked properly or not preserved in your network before

reaching the ASR1000, use the Solution Reference Network Designs (SRND) policy model to simply application

classification in NBAR, and mark the application to the DSCP specified by Microsoft.

class-map match-all VOICE

match protocol attribute traffic-class voip-telephony

match protocol attribute business-relevance business-relevant

class-map match-all BROADCAST-VIDEO

match protocol attribute traffic-class broadcast-video


class-map match-all INTERACTIVE-VIDEO

match protocol attribute traffic-class real-time-interactive


class-map match-all MULTIMEDIA-CONFERENCING

match protocol attribute traffic-class multimedia-conferencing


class-map match-all MULTIMEDIA-STREAMING

match protocol attribute traffic-class multimedia-streaming


class-map match-all SIGNALING

match protocol attribute traffic-class signaling


class-map match-all NETWORK-CONTROL

match protocol attribute traffic-class network-control


class-map match-all NETWORK-MANAGEMENT

match protocol attribute traffic-class ops-admin-mgmt


class-map match-all TRANSACTIONAL-DATA

match protocol attribute traffic-class transactional-data

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/qos_nbar/configuration/xe-3s/qos-nbar-xe-3s-book/clsfy-traffic-nbar.html#GUID-D4B22457-1195-4433-B68C-4CC9C96278E7

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/qos_nbar/configuration/xe-3s/qos-nbar-xe-3s-book.html



class-map match-all BULK-DATA

match protocol attribute traffic-class bulk-data


class-map match-all SCAVENGER

match protocol attribute business-relevance business-irrelevant

!

policy-map MARKING

class VOICE

set dscp ef

class BROADCAST-VIDEO

set dscp af41

class INTERACTIVE-VIDEO

set dscp af41

class MULTIMEDIA-CONFERENCING

set dscp af41

class MULTIMEDIA-STREAMING

set dscp af41

class SIGNALING

set dscp af41

class NETWORK-CONTROL

set dscp cs6

class NETWORK-MANAGEMENT

set dscp default

class TRANSACTIONAL-DATA

set dscp af21

class BULK-DATA

set dscp af11

class SCAVENGER

set dscp default

class class-default

set dscp default

!



service-policy input MARKING

!

Configure IPsec VPN

A common use utilizes the Cisco Cloud Services Router, CSR1000v, deployed as an application VNet gateway in

Azure to provide IPsec gateway for entire VNet. See Extending Enterprise Network into Public Cloud with Cisco

CSR1000v. The ASR1000 connecting to ER is the ideal on-premises gateway for the IPsec tunnel termination in

Enterprise network as the platform delivers embedded hardware acceleration for IPsec VPN. For details on

ASR1000 system IPsec throughput, refer to the relevant product datasheet.

https://clnv.s3.amazonaws.com/2017/usa/pdf/BRKARC-2749.pdf

https://clnv.s3.amazonaws.com/2017/usa/pdf/BRKARC-2749.pdf



The Cisco CSR1000v is ASR1000 in virtual form factor. They run the same IOS XE software release, inherit the

same IOS XE software architecture, support the same CLIs and feature sets of IPsec VPN.

Once you have deployed CSR1000v on Microsoft Azure, you would configure the IPsec VPN on the CSR1000v by

using the step-by-step procedure outlined in this video demo or as per the sample:

crypto isakmp policy 200

encryption aes

authentication pre-share

group 2

lifetime 28800

!

crypto isakmp key cisco123 address 0.0.0.0

crypto isakmp keepalive 10 10

!

crypto ipsec security-association replay window-size 1024

!

crypto ipsec transform-set csr esp-aes esp-sha-hmac

mode tunnel

crypto ipsec df-bit clear

!

crypto ipsec profile csr

set transform-set csr

!

interface Tunnel1

ip address 192.168.100.2 255.255.255.252

tunnel source GigabitEthernet1

tunnel mode ipsec ipv4

tunnel destination 172.16.0.1

tunnel protection ipsec profile csr

You should have the IPsec tunnel peer configuration on the ASR1000 enabled as per the sample:

crypto isakmp policy 200

encryption aes

authentication pre-share

group 2

lifetime 28800

!

crypto isakmp key cisco123 address 0.0.0.0

crypto isakmp keepalive 10 10

!

crypto ipsec security-association replay window-size 1024

!

crypto ipsec transform-set csr esp-aes esp-sha-hmac

mode tunnel

crypto ipsec df-bit clear

https://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/configuration/b_CSR1000v_Configuration_Guide/b_CSR1000v_Configuration_Guide_chapter_00.html#con_1092033

https://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/azu/b_csr1000config-azure.html

https://www.youtube.com/watch?v=U2-lc8oewhA


!

crypto ipsec profile csr1

set transform-set csr

!

interface Tunnel1


ip address 192.168.100.1 255.255.255.252

tunnel source TenGigabitEthernet0/1/0.3101

tunnel mode ipsec ipv4

tunnel destination 10.0.0.4

tunnel protection ipsec profile csr1

Test Connectivity

While there are steps to verify ExpressRoute connectivity with Microsoft, there are also verification steps can be

performed on ASR1000 and in the customer on-premises network.

Verify the BGP Neighbors

Use the following commands to verify the Microsoft peering and Private BGP peering are established and Up

ASR1000#show ip bgp vpnv4 vrf C10 neighbor 216.221.237.34

BGP neighbor is 216.221.237.34, vrf C10, remote AS 12076, local AS 394749,

external link

Description: Microsoft peering to Azure

BGP version 4, remote router ID 207.46.160.94

BGP state = Established, up for 00:39:52

Last read 00:00:16, last write 00:00:39, hold time is 180, keepalive interval

is 60 seconds

Neighbor sessions:

1 active, is not multisession capable (disabled)

Neighbor capabilities:

Route refresh: advertised and received(new)

Four-octets ASN Capability: advertised and received

Address family IPv4 Unicast: advertised and received

Enhanced Refresh Capability: advertised and received

Multisession Capability:

Stateful switchover support enabled: NO for session 1

Message statistics:

InQ depth is 0

OutQ depth is 0

Sent Rcvd

Opens: 1 1

Notifications: 0 0

Updates: 2 3

Keepalives: 45 45

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-troubleshooting-expressroute-overview


Route Refresh: 0 0

Total: 48 49

Do log neighbor state changes (via global configuration)

Default minimum time between advertisement runs is 0 seconds

For address family: VPNv4 Unicast

Translates address family IPv4 Unicast for VRF C10

Session: 216.221.237.34

BGP table version 1326, neighbor version 1326/0

Output queue size : 0

Index 17, Advertise bit 1

17 update-group member

Inbound soft reconfiguration allowed

Outbound path policy configured

Outgoing update prefix filter list is rfc1918

Route map for outgoing advertisements is AD_FS_Prefixes

Slow-peer detection is disabled

Slow-peer split-update-group dynamic is disabled

Sent Rcvd

Prefix activity: ---- ----

Prefixes Current: 1 144 (Consumes 19584 bytes)

Prefixes Total: 1 144

Implicit Withdraw: 0 0

Explicit Withdraw: 0 0

Used as bestpath: n/a 144

Used as multipath: n/a 0

Used as secondary: n/a 0

Outbound Inbound

Local Policy Denied Prefixes: -------- -------

prefix-list 3 0

Bestpath from this peer: 144 n/a

Total: 147 0

Number of NLRIs in the update sent: max 73, min 0

Last detected as dynamic slow peer: never

Dynamic slow peer recovered: never

Refresh Epoch: 1

Last Sent Refresh Start-of-rib: never

Last Sent Refresh End-of-rib: never

Last Received Refresh Start-of-rib: never

Last Received Refresh End-of-rib: never


Sent Rcvd

Refresh activity: ---- ----

Refresh Start-of-RIB 0 0

Refresh End-of-RIB 0 0

Address tracking is enabled, the RIB does have a route to 216.221.237.34

Route to peer address reachability Up: 4; Down: 1

Last notification 03:14:13

Connections established 5; dropped 4

Last reset 00:42:26, due to BGP Notification received, Connection Collision

Resolution

External BGP neighbor configured for connected checks (single-hop no-disable-

connected-check)

Interface associated: TenGigabitEthernet0/1/0.3101 (peering address in same

link)

Transport(tcp) path-mtu-discovery is enabled

Graceful-Restart is disabled

SSO is disabled

Connection state is ESTAB, I/O status: 1, unread input bytes: 0

Connection is ECN Disabled, Mininum incoming TTL 0, Outgoing TTL 1

Local host: 216.221.237.33, Local port: 48945

Foreign host: 216.221.237.34, Foreign port: 179

Connection tableid (VRF): 2

Maximum output segment queue size: 50

Enqueued packets for retransmit: 0, input: 0 mis-ordered: 0 (0 bytes)

Event Timers (current time is 0x153EE19F):

Timer Starts Wakeups Next

Retrans 47 0 0x0

TimeWait 0 0 0x0

AckHold 46 42 0x0

SendWnd 0 0 0x0

KeepAlive 0 0 0x0

GiveUp 0 0 0x0

PmtuAger 1521 520 0x153EE253

DeadWait 0 0 0x0

Linger 0 0 0x0

ProcessQ 0 0 0x0

iss: 2713507505 snduna: 2713508500 sndnxt: 2713508500

irs: 120760723 rcvnxt: 120762358


sndwnd: 15390 scale: 0 maxrcvwnd: 16384

rcvwnd: 16213 scale: 0 delrcvwnd: 171

SRTT: 998 ms, RTTO: 1014 ms, RTV: 16 ms, KRTT: 0 ms

minRTT: 0 ms, maxRTT: 1000 ms, ACK hold: 200 ms

uptime: 2392902 ms, Sent idletime: 16537 ms, Receive idletime: 16737 ms

Status Flags: active open

Option Flags: VRF id set, nagle, path mtu capable

IP Precedence value : 6

Datagrams (max data segment is 1460 bytes):

Rcvd: 93 (out of order: 0), with data: 47, total data bytes: 1634

Sent: 94 (retransmit: 0, fastretransmit: 0, partialack: 0, Second Congestion: 0),

with data: 47, total data bytes: 994

Packets received in fast path: 0, fast processed: 0, slow path: 0

fast lock acquisition failures: 0, slow path: 0

TCP Semaphore 0x7FAA555FB670 FREE

ASR1000#show ip bgp vpnv4 vrf C10 neighbor 172.16.0.2

BGP neighbor is 172.16.0.2, vrf C10, remote AS 12076, local AS 64512, external

link

Description: private peering to Azure

BGP version 4, remote router ID 207.46.160.94

BGP state = Established, up for 4d01h

Last read 00:00:19, last write 00:00:03, hold time is 180, keepalive interval

is 60 seconds

Neighbor sessions:

1 active, is not multisession capable (disabled)

Neighbor capabilities:

Route refresh: advertised and received(new)

Four-octets ASN Capability: advertised and received

Address family IPv4 Unicast: advertised and received

Enhanced Refresh Capability: advertised and received

Multisession Capability:

Stateful switchover support enabled: NO for session 1

Message statistics:

InQ depth is 0

OutQ depth is 0


Sent Rcvd

Opens: 1 1

Notifications: 0 0

Updates: 43 2

Keepalives: 6410 6400

Route Refresh: 0 0

Total: 6456 6403

Do log neighbor state changes (via global configuration)

Default minimum time between advertisement runs is 0 seconds

For address family: VPNv4 Unicast

Translates address family IPv4 Unicast for VRF C10

Session: 172.16.0.2

BGP table version 1326, neighbor version 1326/0

Output queue size : 0

Index 16, Advertise bit 0

16 update-group member

Inbound soft reconfiguration allowed

Outbound path policy configured

Route map for outgoing advertisements is Prepend-USW

Slow-peer detection is disabled

Slow-peer split-update-group dynamic is disabled

Sent Rcvd

Prefix activity: ---- ----

Prefixes Current: 1 1 (Consumes 136 bytes)

Prefixes Total: 1 1

Implicit Withdraw: 0 0

Explicit Withdraw: 147 0

Used as bestpath: n/a 1

Used as multipath: n/a 0

Used as secondary: n/a 0

Outbound Inbound

Local Policy Denied Prefixes: -------- -------

route-map: 0 7

Other Policies: 291 n/a

Total: 291 7

Number of NLRIs in the update sent: max 143, min 0

Last detected as dynamic slow peer: never

Dynamic slow peer recovered: never

Refresh Epoch: 1


Last Sent Refresh Start-of-rib: 04:36:39

Last Sent Refresh End-of-rib: 04:36:39

Refresh-Out took 0 seconds

Last Received Refresh Start-of-rib: never

Last Received Refresh End-of-rib: never

Sent Rcvd

Refresh activity: ---- ----

Refresh Start-of-RIB 1 0

Refresh End-of-RIB 1 0

Address tracking is enabled, the RIB does have a route to 172.16.0.2

Route to peer address reachability Up: 1; Down: 0

Last notification 4d01h

Connections established 1; dropped 0

Last reset never

External BGP neighbor configured for connected checks (single-hop no-disable-

connected-check)

Interface associated: TenGigabitEthernet0/1/0.3103 (peering address in same

link)

Transport(tcp) path-mtu-discovery is enabled

Graceful-Restart is disabled

SSO is disabled

Connection state is ESTAB, I/O status: 1, unread input bytes: 0

Connection is ECN Disabled, Mininum incoming TTL 0, Outgoing TTL 1

Local host: 172.16.0.1, Local port: 179

Foreign host: 172.16.0.2, Foreign port: 28211

Connection tableid (VRF): 2

Maximum output segment queue size: 50

Enqueued packets for retransmit: 0, input: 0 mis-ordered: 0 (0 bytes)

Event Timers (current time is 0x1542D0A0):

Timer Starts Wakeups Next

Retrans 6431 0 0x0

TimeWait 0 0 0x0

AckHold 6401 6287 0x0

SendWnd 0 0 0x0

KeepAlive 0 0 0x0

GiveUp 0 0 0x0

PmtuAger 0 0 0x0

DeadWait 0 0 0x0

Linger 0 0 0x0

ProcessQ 0 0 0x0


iss: 763195641 snduna: 763324044 sndnxt: 763324044

irs: 1048104958 rcvnxt: 1048226686

sndwnd: 15054 scale: 0 maxrcvwnd: 16384

rcvwnd: 16099 scale: 0 delrcvwnd: 285

SRTT: 1000 ms, RTTO: 1003 ms, RTV: 3 ms, KRTT: 0 ms

minRTT: 0 ms, maxRTT: 1000 ms, ACK hold: 200 ms

uptime: 349616252 ms, Sent idletime: 3404 ms, Receive idletime: 3203 ms

Status Flags: passive open, gen tcbs

Option Flags: VRF id set, nagle, path mtu capable

IP Precedence value : 6

Datagrams (max data segment is 1460 bytes):

Rcvd: 12802 (out of order: 0), with data: 6402, total data bytes: 121727

Sent: 12808 (retransmit: 0, fastretransmit: 0, partialack: 0, Second Congestion:

0), with data: 6435, total data bytes: 128402

Packets received in fast path: 0, fast processed: 0, slow path: 0

fast lock acquisition failures: 0, slow path: 0

TCP Semaphore 0x7FAA5454F230 FREE

BGP session is essential to maintain ER connectivity. To protect BGP packets in the ASR1000 punt path and

mitigate potential DDoS attacks, it is recommended you implement Control Plane Policing as per the Control Plane

Policing template on page 50 -53.

Verify ExpressRoute Connectivity

Follow the procedure here to verify ExpressRoute connectivity. The ExpressRoute circuit can be validated by using

the Azure portal “Home > ExpressRoute circuit”, and looking at the “Essentials” field. If you see “Circuit status” is

Enabled, then the ExpressRoute Circuit is up on the Microsoft side, and the “Provider status” as Provisioned, then

the circuit is up on the service provider side, as shown in Figure 5.

http://d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/BRKARC-2019.pdf


https://docs.microsoft.com/en-us/azure/expressroute/expressroute-troubleshooting-expressroute-overview#validate-circuit-provisioning-and-state


Figure 6. Verify ExpressRoute Circuit Status in Azure Portal Snapshot

To further validate that the circuit is up from the customer side, click “Home > ExpressRoute circuit > Azure

Private/Microsoft Private > Get route table summary” to see if your subinterface networks are reachable, as shown

in Figure 6 and 7 respectively.

Figure 7. Verify Private Peering Customer Networks are Reachable in Azure Portal Snapshot


Figure 8. Verify Microsoft Peering Customer Networks are Reachable in Azure Portal Snapshot

Verify NAT Translation Entries and Pool

Follow NAT monitoring and Maintaining guide to verify NAT translation entries are set up properly.

ASR1000#show ip nat translation

Pro Inside global Inside local Outside local Outside global

icmp 216.221.236.33:98 192.168.0.1:98 216.221.237.34:98

216.221.237.34:98

Total number of translations: 1

To monitor the pool stats:

ASR1000#show platform software nat fp active pool

Dump NAT pool config

ID: 1, Name: Cust10_MSFT_Pool, Type: Generic, Mask: 255.255.255.252

Flags: Unknown, Acct name:

Address range blocks: 1

Start: 216.221.236.33, End: 216.221.236.33

Last stats update: 02/13 17:35:39.556

Last refcount value: 1

ASR1000#show platform software nat fp active pool-stats id <id>

NAT Pool Statistics

Pool name Cust10_MSFT_Pool, id 1

Assigned Available

Addresses 0 1

UDP Low Ports 0 512

TCP Low Ports 0 512

UDP High Ports 0 64512

TCP High Ports 0 64512

(Low ports are less than 1024. High ports are greater than or equal to 1024.)

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/ipaddr_nat/configuration/xe-16/nat-xe-16-book/iadnat-monmain.html


Verify Netflow Entries

The ASR1000 exports the NetFlow cache entries directly from the Quantum Flow Processor to the external

collector via in-band interface. Do NOT connect the collector on the management interface (GigabitEthernet0). Use

the following command to verify the flow monitor is exporting data to the exporters.

ASR1000#show flow monitor C10_mon

Flow Monitor C10_mon:

Description: User defined

Flow Record: netflow-original

Flow Exporter: C10_expo

Cache:

Type: normal (Platform cache)

Status: allocated

Size: 200000 entries

Inactive Timeout: 15 secs

Active Timeout: 1800 secs

Trans end aging: off

Use the Top N talkers capability, which facilitates real-time traffic analysis of the most traffic volume consumers.

ASR1000#show flow monitor C10_mon cache sort counter packets top 3 format table

Processed 2 flows

Aggregated to 2 flows

Showing the top 2 flows

IPV4 SRC ADDR IPV4 DST ADDR TRNS SRC PORT TRNS DST PORT INTF

INPUT FLOW SAMPLER ID IP TOS IP PROT ip src as ip dst as

ipv4 next hop addr ipv4 src mask ipv4 dst mask tcp flags intf

output bytes pkts time first time last

=============== =============== ============= ============= ====================

=============== ====== ======= ========= ========= ==================

============= ============= ========= ==================== ========== ==========

============ ============

10.3.0.5 192.168.0.1 0 2048

Te0/1/0.3103 0 0x00 1 0 0

0.0.0.0 /0 /0 0x00 Null

91860 1531 17:16:36.049 17:42:19.371

192.168.0.1 10.3.0.5 0 0 Null

0 0x00 1 0 12076 172.16.0.2

/32 /23 0x00 Te0/1/0.3103 91620

1527 17:16:40.065 17:42:19.371

Note: ASR1000 does not support aggregate flows in Top N talkers.

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/fnetflow/configuration/15-mt/fnf-15-mt-book/cgf-topn.html#GUID-0340CBF8-EBAE-448D-B119-B345243C94C5


ASR1000 Proactive System Monitoring

Proactive monitoring system resources allows you to detect potential problems before they happen, thus avoiding

outages. Figure 8 highlights key system resources to monitor on ASR1000.

Figure 9. Key System Resources to Monitor - Summary

The system resources to be consumed by each of the features discussed in the configuration guide are listed in

Table 5.

Table 5. Feature to System Resources Consumption

Features System Resources Consumed

BGP IOS memory/CPU, RP memory/CPU

FIB IOS memory/CPU, RP memory/CPU

NAT QFP, resource DRAM, TCAM

Netflow QFP, resource DRAM

QoS QFP, TCAM

AVC QFP, resource DRAM, TCAM

IPsec IOS memory/CPU, RP memory/CPU, QFP, Crypto Assist, TCAM

The best practice is that during steady state the system should have minimum 25% of IOS memory, RP memory,

and resource DRAM available to accommodate potential network churning and reconvergence events; otherwise,

you should plan to upgrade system memory or upgrade to a higher performance ASR1000 variant such as the

ASR1002-HX.

For exact CLIs and MIBs to monitor each system source, follow the Operating an ASR1000 guide page 24 - 37.



References

Please refer to the following documentation for ASR1000 platform architecture, packet flow, feature configuration

guide and datasheet:

● ASR1000 System Architecture Overview

● BGP Configuration Guide

● NAT Configuration Guide

● QoS Configuration Guide

● Flexible Netflow Configuration Guide

● NBAR Configuration Guide

● AVC Configuration Guide

● Security for VPNs with IPsec

● IPsec Virtual Tunnel Interface

● ASR1000 Routers Datasheet

● ASR1000-X Router Hardware Installation Guide

● ASR1000-HX Router Hardware Installation Guide

● ASR1000 ESP Datasheet

● ASR1000 Ordering Guide

● IOS-XE NGE Support Product Tech Note

Refer to the following documentation for common error messages and troubleshooting notes:

● Troubleshooting of ASR1k Made Easy

● ASR1000 Troubleshooting TechNotes

● ASR1000 Error and System Messages

● Embedded Packet Capture for IOS-XE

Printed in USA C07-740698-00 07/18

https://clnv.s3.amazonaws.com/2017/eur/pdf/BRKARC-2001.pdf

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/iproute_bgp/configuration/xe-16/irg-xe-16-book.html

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/ipaddr_nat/configuration/xe-16/nat-xe-16-book.html

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/qos_plcshp/configuration/xe-16/qos-plcshp-xe-16-book.html

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/fnetflow/configuration/xe-16/fnf-xe-16-book.html

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/qos_nbar/configuration/xe-16/qos-nbar-xe-16-book.html

https://www.cisco.com/c/en/us/td/docs/ios/solutions_docs/avc/guide/avc-user-guide.html

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/sec_conn_vpnips/configuration/xe-3s/sec-sec-for-vpns-w-ipsec-xe-3s-book/sec-cfg-vpn-ipsec.html

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/sec_conn_vpnips/configuration/xe-3s/sec-sec-for-vpns-w-ipsec-xe-3s-book/sec-ipsec-virt-tunnl.html


https://www.cisco.com/c/en/us/td/docs/routers/asr1000/install/guide/1001-x/asr1hig-book.html

https://www.cisco.com/c/en/us/td/docs/routers/asr1000/install/guide/1001HX_1002HX/b_ASR1001HX-1002HX_HIG.html


https://www.cisco.com/c/en/us/products/collateral/routers/asr-1000-series-aggregation-services-routers/guide-c07-731639.html

https://www.cisco.com/c/en/us/support/docs/security-vpn/ipsec-negotiation-ike-protocols/116055-technote-ios-crypto.html

http://d2zmdbbm9feqrf.cloudfront.net/2017/usa/pdf/BRKCRS-3147.pdf

https://www.cisco.com/c/en/us/support/routers/asr-1000-series-aggregation-services-routers/products-tech-notes-list.html

https://www.cisco.com/c/en/us/support/routers/asr-1000-series-aggregation-services-routers/products-system-message-guides-list.html

https://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/ios-embedded-packet-capture/116045-productconfig-epc-00.html

Documents

Cisco ASR1000 and Microsoft Azure ExpressRoute Design …on the Azure side. To connect to Microsoft Azure services using ExpressRoute, Microsoft provides best practices for network