8001 Linxin Qian VoIP QoS Monitoring

CN 8001 Project/Case Study

M.Eng. Computer Networks, Ryerson University

VOIP QoS Monitoring Presented by Linxin Qian

A project Presented to Ryerson University In partial fulfillment of the Requirement for the degree of Masters of Engineering in Computer Networks At Ryerson University Toronto, Ontario, Canada, 2007

Under the supervision of Govindan Ravindran

Linxin Qian 2007

Revision Date: 14-Aug-2007

CN 8001 Project/Case Study University

M.Eng. Computer Networks, Ryerson

Authors declaration

I hereby declare that I am the sole author of this case study.

I authorize Ryerson University to lend this case study to other institutions or individuals for the purpose of scholarly research.

Signature: _____________________________

I further authorize Ryerson University to reproduce this case study by photocopying or by other means, in total or in part, at the request of other institutions or individuals for the purpose of scholarly research.

Signature: _____________________________




AbstractVOIP has become a hot topic in IT industry. To be able to provide acceptable VOIP performance, QoS monitoring is necessary not only during pre-deployment network assessment, but also during troubleshooting and maintenance.

Many commercial VOIP QoS monitoring tools are based on monitoring VOIP call servers. However, these solutions could only answer whether these calls meet some kinds of quality assurance or not. When the network congestion occurs, these solutions could not provide accurate information to locate network bottleneck. Because of IP packet routing unpredictability, the troubleshooting procedure for network administrators become tedious and error-pronep.

In this work, we propose and explain a monitoring procedure to quickly locate network and device bottlenecks from just monitoring two factors: queue drops for device interfaces, and monitoring end-to-end link quality for WAN.

We also investigate the possibility to automate this monitoring procedure, to enable monitored devices to send traps or alarms to SNMP managers. The SNMP managers can in turn notify network admins. Our implementation is twofold: i) fetch the number of drop packets for voice via SNMP polling on Cisco-Class-Based-QoS-MIB, and ii) set up Cisco IOS IP SLA and enable SLA threshold-based monitoring. These numbers and traps are received by SNMP managers, locally processed and mail notifications are sent to network administrators.




Acknowledgement

Special thanks to Dr. Govindan Ravindran for his time and assisstance on the technical review of this material.




Table of Contents1 Introduction......................................................................................................... 11.1 1.2 1.3 Purpose ................................................................................................................................1 Introduction to VoIP ............................................................................................................1 VoIP Requirements..............................................................................................................1 1.3.1 Bandwidth............................................................................................................1 1.3.2 Latency ................................................................................................................2 1.3.3 Jitter .....................................................................................................................2 1.3.4 Packet Loss .........................................................................................................2 QoS .......................................................................................................................................2 VoIP QoS ..............................................................................................................................3 Requirements of VoIP QoS Management Solution ..........................................................3

1.4 1.5 1.6

2 Protocols and Softwares.................................................................................... 42.1 2.2 2.3 2.4 2.5 2.6 SNMP ....................................................................................................................................4 Traps .....................................................................................................................................4 Net-SNMP .............................................................................................................................4 OidView Mib Browser..........................................................................................................5 Kiwi Syslog Daemon ...........................................................................................................5 HP OpenView Performance Insight ...................................................................................5

3 Cisco Class Based QoS ..................................................................................... 63.1 3.2 3.3 Cisco QoS Overview ...........................................................................................................6 Cisco Class Based QoS Model...........................................................................................6 Cisco-Class-Based-QoS-MIB..............................................................................................7

4 Cisco IOS SLA..................................................................................................... 84.1 4.2 Cisco IOS SLA Overview ....................................................................................................8 Cisco IOS SLA Operation ...................................................................................................8




4.3 4.4

SLA UDP Jitter Operation ...................................................................................................9 Cisco IOS SLA Threshold .................................................................................................10

5 Module Sample ................................................................................................. 125.1 Basic Analysis ...................................................................................................................12 5.1.1 Bandwidth..........................................................................................................12 5.1.2 Delay...................................................................................................................12 5.1.3 Jitter ...................................................................................................................14 5.1.4 Packet Loss .......................................................................................................14 Assumption & Summary...................................................................................................15 Case Study Network Diagram ..........................................................................................15 Implementation ..................................................................................................................16 Cisco Class Based QoS Configuration ...........................................................................16 Cisco Class Based QoS MIB Analysis ............................................................................18 Cisco Class Based QoS MIB SNMP-Query Analysis .....................................................21 Cisco Class Based QoS MIB SNMP-Query Result .........................................................24 Cisco IOS SLA Implementation........................................................................................25

5.2 5.3 5.4 5.5 5.6 5.7 5.8 5.9

5.10 Cisco IOS SLA Results .....................................................................................................26 5.11 HP Openview Performance Insight on SLA Results ......................................................28

6 Conclusions ...................................................................................................... 29 7 Future Works..................................................................................................... 31 8 References ........................................................................................................ 32




List of TablesTable 1 Codec Standard .............................................................................................................................. 1




List of FiguresFigure 1 QoS Traffic Flow ..............................................................................................................................7 Figure 2 SLA Operation Diagram...................................................................................................................8 Figure 3 End-to-End Delay...........................................................................................................................12 Figure 4 Network Diagram Sample..............................................................................................................15 Figure 5 Cisco-Class-Based-QoS via OidView............................................................................................20 Figure 6 cbQosCMDropPktOverflow in OidView .........................................................................................21 Figure 7 CbQosConfigIndex in OidView .....................................................................................................22 Figure 8 cbQosPolicyIndex & cbQosObjectsIndex in OidView....................................................................22 Figure 9 cbQosPolicyIndex2 & cbQosObjectsIndex2 in OidView................................................................23 Figure 10 cbQosCMDropPkt via Oidview ....................................................................................................23 Figure 11 VOICE configindex via NetSnmp ..............................................................................................24 Figure 12 cbQosPolicyIndex and cbQosObjectsIndex via Net-Snmp .........................................................24 Figure 13 cbQosCMDropPkt via Net-Snmp .................................................................................................25 Figure 14 Syslog configuration via Kiwi Syslog ...........................................................................................26 Figure 15 Trap Configuration in Kiwi Syslog................................................................................................26 Figure 16 Received Trap & Syslog in Kiwi Syslog .......................................................................................27 Figure 17 Mail Setup in Kiwi Syslog.............................................................................................................27 Figure 18 Jitter Summary in HPOVPI ..........................................................................................................28 Figure 19 Response Time in HPOVPI .........................................................................................................28 Figure 20 Jitter Results in HPOVPI..............................................................................................................29 Figure 21 Packet Loss in HPOVPI...............................................................................................................29




List of AppendicesAppendix A: ISP Network Latency ........................................................................... 32 Appendix B: Market VoIP Solution ........................................................................... 33 Appendix C: SLA Availability Computioantion .......................................................... 34 Appendix D: Cisco 4500 Switch QoS Configuration ................................................. 36




11.1

IntroductionPurpose

The purpose of this work is to fina a quick and efficient way to identify voice quality bottlenecks on the enterprise network during the VOIP pre-deployment and during post-deployment maintenance and troubleshooting. 1.2 Introduction to VoIP

Voice over IP (VoIP) enables a voice-enabled network device to carry voice traffic, such as telephone calls and faxes, over an IP network infrastructure. The increased efficiency of IP networks and the ability to statistically multiplex voice traffic with data packets allows companies to maximize their return on investment (ROI) in data network infrastructures. Decreased cost and an increase in the availability of differentiated services are two major reasons companies are evaluating the implementation of VoIP. 1.3 VoIP Requirements

To ensure that VoIP is an acceptable replacement for standard public switched telephone network (PSTN) telephony services, customers must receive the same consistently high quality of voice transmission that they receive with basic telephone services. Like other real-time applications, VoIP is extremely sensitive to issues related to bandwidth and delay. To ensure that VoIP transmissions are intelligible to the receiver, voice packets cannot be dropped, excessively delayed, or subjected to variations in delay (that is, jitter). A successful VoIP deployment must provide an acceptable level of voice quality by meeting VoIP traffic requirements for issues related to bandwidth, latency, jitter, and packet loss. 1.3.1 Bandwidth Standard, Codec G.711, PCM G.726, ADPCM G.728, LDCELP G.729, CS-ACELP G.729A, CS-ACELP Bit Rate(kbps) 64 16,24,32 16 8 8Table 1 Codec StandardPage 1

Voice Quality(MOS) 4.1 3.85(with 32kbps) 3.61 3.92 3.9




Different codec has different bandwidth requirement. To be able to achieve better performance, minimum bandwidth has to be met. 1.3.2 Latency

Callers usually notice roundtrip voice delays of 250ms or more. ITU-T G.114 recommends a maximum of a 150 ms one-way latency. Since this includes the entire voice path, part of which may be on the public Internet, your own network should have transit latencies of considerably much less than 150 ms. 1.3.3 Jitter IETF RFC 3550 RTP: A Transport Protocol for Real-Time Applications IETF RFC 3611 RTP Control Protocol Extended Reports (RTCP XR) Most VOIP endpoint devices, e.g. VOIP phones and ATA, have jitter buffers to compensate for network jitter. 1.3.4 Packet Loss

Jitter can be measured in several ways. There are jitter measurement calculations defined in:

VOIP is not tolerant of packet loss. Even 1% packet loss can significantly degrade a VOIP call using a G.711 codec, and other more compressing codecs can tolerate even less packet loss. One of Ciscos Whitepaper says, The default G.729 codec requires packet loss far less than 1 percent to avoid audible errors. Ideally, there should be no packet loss for VoIP. 1.4 QoS

Generally speaking, QoS is the ability of a network element (e.g. an application, a host or a router) to provide some level of assurance for consistent network data delivery. Some applications are more stringent about their QoS requirements than others. We have two basic types of QoS implementation available: Resource reservation (integrated services): network resources are apportioned according to an application's QoS request, and subject to bandwidth management policy. VoIP guarantees high-quality voice transmission only if the signaling and audio channel packets have priority over other kinds of network traffic. Since resource reservation takes too much device resource when core devices have to deal with hundreds of connection, this solution is not that scalable. Most networks go with prioritization to slove VoIP QoS.

Page 2




Prioritization (differentiated services): network traffic is classified and apportioned network resources according to bandwidth management policy criteria. To enable QoS, network elements give preferential treatment to classifications identified as having more demanding requirements. 1.5 VoIP QoS

VoIP guarantees high-quality voice transmission only if the signaling and audio channel packets have priority over other kinds of network traffic. Since resource reservation takes too much device resource when core devices have to deal with hundreds of connection, this solution is not that scalable. Most networks go with prioritization to slove VoIP QoS. 1.6 Requirements of VoIP QoS Management Solution

VoIP QoS management should not just focus on network IP quality of service (QoS) to monitor and manage the quality of IP services, such as packets loss, jitter, or delay, but also ensure a high-quality customer experience by monitoring and managing voice quality levels, because the quality of voice reproduction and transmission can often make or break customer satisfaction and service renewals. The mature VoIP QoS management should monitor not only end-to-end VoIP quality via control protocol, SIP/H.322, but also network device health for packet loss, jitter, or delay. It should be able to provide not only summary statistics of historical calls, and also detailed real-time end-to-end info of each call to troubleshooting. It should be able to provide assessment tool to evaluate the whole network device readiness before VoIP deployment. It should be able to monitor server health and network IP service, including gateway/call manager CPU/memory/TFTP/DNS/Database size/data query, etc. It should be able to provide report for capacity/trend/QoS monitor and summary. It should be less/easy to configuration, better to be appliance. It should be able to adopt different voice products from many vendors. It should integrate with other tools, such as HP openview NNM and Performance Insight.

Page 3




22.1

Protocols and SoftwaresSNMP

The simple network management protocol (SNMP) forms part of the internet protocol suite as defined by the Internet Engineering Task Force (IETF). SNMP is used by network management systems to monitor network-attached devices for conditions that warrant administrative attention. It consists of a set of standards for network management, including an Application Layer protocol, a database schema, and a set of data objects. In typical SNMP usage, there are generally a number of systems to be managed, and one or more systems managing them. A software component called an agent runs on each managed system and reports information via SNMP to the managing systems. 2.2 Traps

There are at least 4 methods to send a notification to a SNMP manager; 1) SNMP manager actively query specific MIB objects, and identify the breach on the managed device, then generate mail alert; 2) Post-process SNMP manager results and generate notification. 3) Router monitors itself locally via RMON alarm and events. Configure RMON events and threshold on local device, if threshold breach, trigger alarm to send a trap to SNMP manager; No wildcard is supported, each interface need individual configuration. 4) Router monitors itself locally via Event-MIB. If event is triggered, send trap to SNMP manager. Need SNMP Set to initialize some MIB values. 2.3 Net-SNMP

Net-SNMP is a suite of applications used to implement SNMP v1, SNMP v2c and SNMP v3 using both IPv4 and IPv6. This suite included a library, a selection of simple management commands, and an "agent" (which reported most of the standard management information, as defined in RFC 1213). The code was made publicly available, and used by a number of people (including several commercial companies).

Page 4




2.4

OidView Mib Browser

OidView developed by ByteSphere is a modular SNMP analysis toolset and MIB browser for network management administrators. It comes with a built-in MIB Browser, Performance OID Poller, Trap Manager and Trap Receiver, a high-performance MIB Compiler, and a variety of other tools (IGRID, PDUTrace, ENTITY, etc). The OidView system could analyze up to 10 live SNMP agents or pre-recorded mibwalks simultaneously, and implement agent-recognition code to autoload the correct MIBs for an agent analysis session. Loaded MIBs are automatically detected and loaded. 2.5 Kiwi Syslog Daemon

Kiwi Syslog Daemon is a freeware Syslog Daemon for Windows. It receives, logs, displays and forwards syslog messages from hosts such as routers, switches, unix hosts and any other syslog enabled device. 2.6 HP OpenView Performance Insight

HP OpenView Performance Insight can help enterprise implement service level management by monitoring and reporting on the systems, networks and applications that make up a service. It collects, analyzes and summarizes management data from any source and can be tailored for any audience. In addition to the out-of-the-box reports for popular HP OpenView tools such as Network Node Manager, Performance Agents and Internet Services, it also has prepackaged reports on a wide range of network protocols and devices. This gives enterprise the flexibility to report on the data that is most important their customers.

Page 5




33.1

Cisco Class Based QoSCisco QoS Overview

Typically, networks operate on a best-effort delivery basis, which means that all traffic has equal priority and an equal chance of being delivered in a timely manner. When congestion occurs, all traffic has an equal chance of being dropped. QoS selects network traffic (both unicast and multicast), prioritizes it according to its relative importance, and uses congestion avoidance to provide priority-indexed treatment; QoS can also limit the bandwidth used by network traffic. QoS make network performance more predictable and bandwidth utilization more effective. 3.2 Cisco Class Based QoS Model

Cisco CBQoS model includes classifying traffic, policing, and marking at the ingress interface: Classifying distinguishes one kind of traffic from another. The process generates an internal DSCP for a packet, which identifies all the future QoS actions to be performed on this packet; Policing determines whether a packet is in or out of profile by comparing the traffic rate to the configured policer, which limits the bandwidth consumed by a flow of traffic. The result of this determination is passed to the marker. Marking evaluates the policer configuration information regarding the action to be taken when a packet is out of profile and decides what to do with the packet (pass through a packet without modification, mark down the DSCP value in the packet, or drop the packet). Include queueing and scheduling at the egress interface: Queueing evaluates the internal DSCP and determines which of the egress queues in which to place the packet. Scheduling services the egress (transmit) queues based on the sharing and shaping configuration of the egress (transmit) port.

Page 6




Figure 1 QoS Traffic Flow

3.3

Cisco-Class-Based-QoS-MIB

This MIB provides read access to Quality of Service (QoS) configuration and statistics information for Cisco platforms that support the Modular Quality of Service Command-line

Interface (Modular QoS CLI). Configuration information available through this MIB includes all ClassMap, PolicyMap, Match Statements, and Feature Actions configuration parameters. Statistics available through this MIB include summary counts/rates by traffic class before and after any configured QoS policies are enforced. Features. In addition, detailed feature-specific statistics are available for select PolicyMap

Page 7




44.1

Cisco IOS SLACisco IOS SLA Overview

Cisco IOS IP Service Level Agreements (SLAs) is a capability embedded Cisco IOS Software, which allows Cisco customers to increase productivity, lower operational costs, and reduce the frequency of network outages. IP and SLAs are converging, and extending IP performance monitoring to be application-aware is critical for new IP network applications such as voice over IP (VoIP), audio and video, enterprise resource planning (ERP), customer relationship management (CRM), material requirements planning (MRP), VPNs, and other business-critical applications. Cisco IOS IP SLAs can perform network assessments; verify quality of service (QoS), ease deployment of new services, and assist administrators with network troubleshooting. Cisco IOS IP SLAs use unique service-level assurance metrics and methodology to provide highly accurate, precise service-level assurance measurements. 4.2 Cisco IOS SLA Operation

Network engineer can monitor the performance between any area in the network: core, distribution, and edge. Monitoring can be done anytime, anywhere, without deploying a physical probe.

Figure 2 SLA Operation Diagram

Page 8




Cisco IOS IP SLAs uses generated traffic to measure network performance between two networking devices such as routers. Cisco IOS IP SLAs starts when the Cisco IOS IP SLAs device sends a generated packet to the destination device. After the destination device receives the packet, and depending on the type of Cisco IOS IP SLAs operation, the device will respond with time-stamp information for the source to make the calculation on performance metrics. A Cisco IOS IP SLAs operation performs a network measurement from the source device to a destination in the network using a specific protocol such as UDP. 4.3 SLA UDP Jitter Operation

There are various types of Cisco IOS IP SLAs operations, what each operation measures, and for what purpose the operation is used. The type we used in the report is UDP Jitter operation. The IP SLAs UDP jitter operation was primarily designed to diagnose network suitability for real-time traffic applications such as voice over IP (VoIP), video over IP, or real-time conferencing. Jitter means inter-packet delay variance. When multiple packets are sent consecutively from source to destination, for example, 10 ms apart, and if the network is behaving ideally, the destination should be receiving them 10 ms apart. But if there are delays in the network (like queuing, arriving through alternate routes, and so on) the arrival delay between packets might be greater than or less than 10 ms. Using this example, a positive jitter value indicates that the packets arrived greater than 10 ms apart. If the packets arrive 12 ms apart, then positive jitter is 2 ms; if the packets arrive 8 ms apart, then negative jitter is 2 ms. For delay-sensitive networks like VoIP, positive jitter values are undesirable, and a jitter value of 0 is ideal. However, the IP SLAs UDP jitter operation does more than just monitor jitter. As the UDP jitter operation includes the data returned by the IP SLAs UDP operation, the UDP jitter operation can be used as a multipurpose data gathering operation. The packets IP SLAs generates carry packet sending sequence and receiving sequence information, and sending and receiving time stamps from the source and the operational target. Based on these, UDP jitter operations are capable of measuring the following: Per-direction jitter (source to destination and destination to source) Per-direction packet-loss

Page 9




Per-direction delay (one-way delay) Round-trip delay (average round-trip time) As the paths for the sending and receiving of data may be different (asymmetric), the perdirection data allow you to more readily identify where congestion or other problems are occurring in the network. The UDP jitter operation functions by generating synthetic (simulated) UDP traffic. The UDP jitter operation sends N UDP packets, each of size S, sent T milliseconds apart, from a source router to a target router, at a given frequency of F. By default, ten packetframes (N), each with a payload size of 10 bytes (S) are generated every 10 ms (T), and the operation is repeated every 60 seconds (F). Each of these parameters are user-configurable, so as to best simulate the IP service you are providing, or want to provide. 4.4 Cisco IOS SLA Threshold

IP SLAs includes the capability for triggering SNMP notifications based on defined thresholds. This allows for proactive monitoring in an environment where IT departments can be alerted to potential network problems, rather than having to manually examine data. IP SLAs supports threshold monitoring for performance parameters such as average jitter, unidirectional latency and bidirectional round trip time and connectivity. This proactive monitoring capability provides options for configuring reaction thresholds for important VoIP related parameters including unidirectional jitter, unidirectional packet loss, and unidirectional VoIP voice quality scoring (MOS scores). SNMP notifications (traps) for IP SLAs can be configuredas a triggered action, to be sent when monitored values exceed an upper threshold or fall below a lower threshold, or when a set of defined conditions are met. For example, an SNMP trap can be triggered by 5 consecutive timeouts during an IP SLAs operation. The sending of SNMP traps is one of the options for triggered actions that can be configured for IP SLAs violations. The monitored values (also called monitored elements), the threshold type, and the triggered action are configured using the ip sla monitor reaction-configuration global configuration mode command. SNMP traps for IP SLAs are handled through the system logging (syslog) process. This means that system logging messages for IP SLAs violations are generated when the specified conditions are met, then sent as SNMP traps using the CISCO-SYSLOG-MIB. The ip sla monitor logging traps command is used to enable the generation of these IP SLAs specific traps. The generation

Page 10




of IP SLAs specific logging messages is dependant on the configuration of the standard set of logging commands (for example, logging on). IP SLAs logging messages are generated at the informational system logging severity level.

Page 11




5

Module Sample

Enterprise networks could be divided into LAN and WAN components. Network condition and VoIP requirements is different for both. 5.1 5.1.1 Basic Analysis Bandwidth LAN Most enterprises have no bandwidth issue in their local networks. WAN Bandwidth is a critical, and precious resource. It should be fully and efficiently used. During congestion, low priority packet will be dropped before high priority packet. So it is necessary to monitor the drops on voice class at the edge device.

Solution: Cisco CBQoS need to be configured on edge router to prioritize voice packet. Cisco-Class-Based-QoS-MIB need to be polled for packet drops. 5.1.2 Delay

Figure 3 End-to-End Delay

Page 12




There are four kinds of delay: Processing Delay: The time it takes for a router to take the packet from an input interface and put it into the output queue of the output interface. The processing delay depends on various factors, such as: o CPU speed o CPU utilization o IP switching mode o Router architecture o Configured features on both input and output interface Queuing Delay: The time a packet resides in the output queue of a router. It depends on the number and sizes of packets already in the queue and on the bandwidth of the interface. It also depends on the queuing mechanism. Serialization Delay: The time it takes to place a frame on the physical medium for transport. Propagation Delay: The time it takes to transmit a packet. (This usually depends on the type of media interface.)

End-to-end delay equals a sum of all propagation, processing and queuing delays in the path. In best-effort networks, propagation delay is fixed, processing and queuing delays are unpredictable.

-

LAN Since serialization delay and propagation delay are relative small in the LAN, they can be ignored; however, processing and queuing delays are unpredictable here, need to be monitored and controlled. In LAN, most delay exists on hardware layer, so the network admin should consider prioritize voice packet. Class based QoS configuration is for this purpose. With proper QoS configuration, voice packet could get higher priority than other data flows. Since drops only occur in congestion, we believe voice QoS level could be kept high in this device as long as no such drops in voice queues occur, with the precondition that QoS configuration is done properly during deployment.

Page 13




-

WAN The delay in WAN is controlled by ISP. Generally enterprise network admin could not do much to lower this delay. However, currently some ISP guarantee certain SLA to their customers, allowing enterprise network admin to limit this kind of delay to some value and monitor it. Solution: Cisco CBQoS need to be configured on all devices to prioritize voice packet, Cisco-Class-Based-QoS-MIB need to be polled for packet drop; on edge device, SLA need to be configured for delay monitor.

5.1.3 -

Jitter LAN Jitter in LAN is relative small, and most VOIP endpoint devices, e.g. VOIP phones and ATA, have jitter buffers to compensate for network jitter. So it can be ignored.

-

WAN As per Cisco doc, Jitter buffers (used to compensate for varying delay) further add to the end-to-end delay, and are usually only effective on delay variations less than 100 ms.

Solution: Cisco IOS SLA need to be configured to monitor jitter that is more than 100ms. 5.1.4 Packet Loss LAN Since there is no bandwidth shortage in LAN, the drops here are mostly due to physical reason, such as link interfere, and interface mismatch and could be easily avoided.

-

WAN The drops occurs in the ISP network. Enterprise network admin need to monitor it.

Solution: Cisco IOS SLA need to be configured on the edge device to monitor packet loss.

Page 14




5.2

Assumption & Summary

As long as these 4 issues, bandwidth, latency, jitter, and packet loss, meet VoIP traffic requirements, we could reasonably believe we provide an acceptable level of voice quality.

With above analysis, we could summarize them as following:

All devices need to be polled for packet drops; Edge devices need to be configured for monitoring delay, jitter, and packet loss.

Of course, these summaries are based the below 3 reasonable assumptions: Assumption 1: Bandwidth is often not issue. The average link utilization in current enterprise is quite low. Assumption 2: Packet loss is well controlled. Packet loss could be reduced easily, because these devices are in the domain of same administration. Assumption 3: Jitter is minimized on end units.

5.3

Case Study Network Diagram

SNMP Polling Server OidView $ Net-SNMP Toronto access Router Cisco 2600 Cisco 3560 SLA Poller Indian access router Cisco2600

WASLA Responder

HeadquartersSNMP Syslog & Trap Server Kiwi Daemon

SNMP Report Server HP OVPI

Figure 4 Network Diagram SamplePage 15




5.4 -

Implementation Cisco 3560 used as access switch, configured with CBQoS and Polled for CBQoS MIB by SNMP Polling Server; Cisco 2600 @Toronto access Router Used as Local SLA poller, configured with Cisco IOS SLA; Cisco 2600 @Indian access Router Used as Local SLA responder, configured with Cisco IOS SLA; SNMP Polling Server for OidView & Net-SNMP This server is used for polling Cisco-Class-Based-QoS, to confirm there such MIB exist for our monitoring; SNMP Syslog & Trap Server for Kiwi Daemon This server is used to receive SLA trap and breach information; SNMP Report Server for HP OVPI This server is used to generate SLA report for delay, jitter, and packet loss;

Target: 5.5 Monitor Cisco Device CBQoS Voice packet drops; Set-up SLA threshold based monitoring of WAN delay, jitter, and packet loss. Cisco Class Based QoS Configuration

Although different hardware type and IOS version make QoS configuration different, the configuration step of classification , policing, and marking on ingress, queueing and scheduling on egress are still same. In our module, we used Nortel as example. This kind of phone use port 5200 for voice, use port 5000-5100 for voice management between call center and phone. Cisco 3560 Switch QoS Configuration! ! enable qos mls qos ! create qos mapping cos-dscp mls qos map cos-dscp 0 8 16 26 34 46 48 56

Page 16




! ! ip access-list extended voice remark UDP Ports for VoIP - Bearer permit udp any any eq 5200 ip access-list extended voice-control remark VoIP Control Traffic permit tcp any any range 5000 5100 ip access-list extended netmgt remark CW2K permit ip host 172.17.1.100 any ip access-list extended best-effort remark All other traffic permit ip any any !

!classify traffic ! class-map match-all VOICE description VoIP Bearer traffic match access-group name voice class-map match-all VOICE-CONTROL description Voice Control traffic match access-group name voice-control class-map match-all NETMGT description network management stations match access-group name netmgt ! ! policy-map Mark-Classify-AccessLayer description Mark-Classify traffic class VOICE set dscp ef class VOICE-CONTROL set dscp af31 class NETMGT set dscp af21 class class-default set dscp default ! !interface configuration example interface FastEthernet2/2 description Access Port Fa2/2 switchport access vlan 39 switchport mode access service-policy input Mark-Classify-AccessLayer

Page 17




udld port aggressive no snmp trap link-status srr-queue bandwidth share 30 20 25 25 priority-queue out spanning-tree portfast ! !Uplink Trunk example interface GigabitEthernet0/1 description Uplink switchport trunk encapsulation dot1q switchport trunk allowed vlan 39 switchport mode trunk priority-queue out udld port aggressive mls qos trust dscp end!

5.6

Cisco Class Based QoS MIB Analysis

Step 1 - MIB ID for Cisco-Class-Based-QoS is 1.3.6.1.4.1.9.9.166. Step 2 Go through MIB doc for Cisco-Class-Based-QoS-MIB , we could see-- Defines 2 groups of objects. ciscoCBQosMIBObjects OBJECT IDENTIFIER ::= { ciscoCBQosMIB 1 } ciscocbQosMIBConformance OBJECT IDENTIFIER ::= { ciscoCBQosMIB 2 }

So ciscoCBQosMIBObjects = 1.3.6.1.4.1.9.9.166.1 Step 3 Locate the MIB ID for cbQosClassMapStats-- object groups cbQosServicePolicy cbQosInterfacePolicy cbQosFrameRelayVCPolicy cbQosATMPVCPolicy cbQosObjects cbQosPolicyMapCfg cbQosClassMapCfg cbQosMatchStmtCfg cbQosQueueingCfg cbQosREDCfg cbQosREDClassCfg OBJECT IDENTIFIER ::= { ciscoCBQosMIBObjects OBJECT IDENTIFIER ::= { ciscoCBQosMIBObjects OBJECT IDENTIFIER ::= { ciscoCBQosMIBObjects OBJECT IDENTIFIER ::= { ciscoCBQosMIBObjects OBJECT IDENTIFIER ::= { ciscoCBQosMIBObjects OBJECT IDENTIFIER ::= { ciscoCBQosMIBObjects OBJECT IDENTIFIER ::= { ciscoCBQosMIBObjects OBJECT IDENTIFIER ::= { ciscoCBQosMIBObjects OBJECT IDENTIFIER ::= { ciscoCBQosMIBObjects OBJECT IDENTIFIER ::= { ciscoCBQosMIBObjects OBJECT IDENTIFIER

1 } 2 } 3 } 4 } 5 } 6 } 7 } 8 } 9 } 10 }

Page 18




cbQosPoliceCfg cbQosTSCfg cbQosSetCfg cbQosClassMapStats cbQosMatchStmtStats cbQosPoliceStats cbQosQueueingStats cbQosTSStats cbQosREDClassStats cbQosPoliceActionCfg --

::= { ciscoCBQosMIBObjects OBJECT IDENTIFIER ::= { ciscoCBQosMIBObjects OBJECT IDENTIFIER ::= { ciscoCBQosMIBObjects OBJECT IDENTIFIER ::= { ciscoCBQosMIBObjects OBJECT IDENTIFIER ::= { ciscoCBQosMIBObjects OBJECT IDENTIFIER ::= { ciscoCBQosMIBObjects OBJECT IDENTIFIER ::= { ciscoCBQosMIBObjects OBJECT IDENTIFIER ::= { ciscoCBQosMIBObjects OBJECT IDENTIFIER ::= { ciscoCBQosMIBObjects OBJECT IDENTIFIER ::= { ciscoCBQosMIBObjects OBJECT IDENTIFIER ::= { ciscoCBQosMIBObjects

11 } 12 } 13 } 14 } 15 } 16 } 17 } 18 } 19 } 20 } 21 }

So cbQosClassMapStats = 1.3.6.1.4.1.9.9.166.1.15 Step 4 Locate MIB ID for cbQosCMDropPktOverflowcbQosCMStatsTable OBJECT-TYPE SYNTAX SEQUENCE OF CbQosCMStatsEntry MAX-ACCESS not-accessible STATUS current DESCRIPTION "This table specifies ClassMap related Statistical information. " ::= { cbQosClassMapStats 1 } cbQosCMStatsEntry OBJECT-TYPE SYNTAX CbQosCMStatsEntry MAX-ACCESS not-accessible STATUS current DESCRIPTION "Each entry in this table describes the statistical information about ClassMap. ClassMap specific information you can find in this table are : pre/post policy pkt/byte counts, bit rates, drop pkt/bytes and no buffer drops. This table contains statistical information only, no configuration information associated with it. Therefore, it is indexed by the instance specific IDs, such as cbQosPolicyIndex and cbQosObjectsIndex. " INDEX { cbQosPolicyIndex, cbQosObjectsIndex } ::= { cbQosCMStatsTable 1 } CbQosCMStatsEntry ::= SEQUENCE { cbQosCMPrePolicyPktOverflow cbQosCMPrePolicyPkt cbQosCMPrePolicyPkt64 cbQosCMPrePolicyByteOverflow cbQosCMPrePolicyByte cbQosCMPrePolicyByte64 cbQosCMPrePolicyBitRate cbQosCMPostPolicyByteOverflow cbQosCMPostPolicyByte cbQosCMPostPolicyByte64 cbQosCMPostPolicyBitRate cbQosCMDropPktOverflow cbQosCMDropPkt

Counter32, Counter32, Counter64, Counter32, Counter32, Counter64, Gauge32, Counter32, Counter32, Counter64, Gauge32, Counter32, Counter32,

Page 19




cbQosCMDropPkt64 cbQosCMDropByteOverflow cbQosCMDropByte cbQosCMDropByte64 cbQosCMDropBitRate cbQosCMNoBufDropPktOverflow cbQosCMNoBufDropPkt cbQosCMNoBufDropPkt64 }

Counter64, Counter32, Counter32, Counter64, Gauge32, Counter32, Counter32, Counter64

So cbQosCMDropPktOverflow = 1.3.6.1.4.1.9.9.166.1.1.1.12 Step 5 Locate MIB ID for cbQosCMDropPktOverflow on interface Gi3/8 From the documentation, we know all of cbQosCMDropPktOverflow value are indexed by cbQosPolicyIndex and cbQosObjectsIndex. Using a standard MIB browser such as OidView Browser, we could easily find these two values.

Figure 5 Cisco-Class-Based-QoS via OidView

The above screenshot shows MIB to poll for Cisco-Class-Based-QoS on Switch:172.17.1.100.Page 20




For interface Gi3/8: ClassMap:VOICE on this interface has cbQosPolicyIndex = 1091, cbQosObjectsIndex =1093. So cbQosCMDropPktOverflow on int Gi 3/8 = 1.3.6.1.4.1.9.9.166.1.1.1.12.1091.1093 Confirmation:

Figure 6 cbQosCMDropPktOverflow in OidView

We could find 1.3.6.1.4.1.9.9.166.1.1.1.12.1091.1093 is counter for cbQosCMDropPktOverflow. 5.7 Cisco Class Based QoS MIB SNMP-Query Analysis

To be able to query this information, follow this below steps, you could locate indexes, cbQosPolicyIndex and cbQosObjectsIndex. 1. Locate cbQosConfigIndex in cbQosClassMapCfgTable to match class-map VOICE; Result: cbQosConfigIndex = 1051.

Page 21




Figure 7 CbQosConfigIndex in OidView

2. Find cbQosPolicyIndex and cbQosObjectsIndex in cbQosObjectsTable by using cbQosConfigIndex; Result1 : cbQosPolicyIndex = 1091 and cbQosObjectsIndex = 1093

Figure 8 cbQosPolicyIndex & cbQosObjectsIndex in OidView

Page 22




Result2 : cbQosPolicyIndex = 1127 and cbQosObjectsIndex = 1129

Figure 9 cbQosPolicyIndex2 & cbQosObjectsIndex2 in OidView

3. Access QoS statistics: cbQosCMDropPkt in table: cbQosClassMapStats by using cbQosPolicyIndex and cbQosObjectsIndex. Result: cbQosCMDropPkt = 0 for cbQosPolicyIndex = 1091/ cbQosObjectsIndex = 1093; cbQosCMDropPkt = 0 for cbQosPolicyIndex = 1127/ cbQosObjectsIndex = 1129.

Figure 10 cbQosCMDropPkt via Oidview

Page 23




5.8

Cisco Class Based QoS MIB SNMP-Query Result

Summary : From above analysis, we could conclude that 3 commands are used for programming. Command 1: snmpwalk 1.3.6.1.4.1.9.9.166.1.7 for class-map VOICE configindex

Figure 11 VOICE configindex via NetSnmp

Command

2:

snmpwalk

1.3.6.1.4.1.9.9.166.1.5

for

cbQosPolicyIndex

and

cbQosObjectsIndex related with class-map VOICE configindex

Figure 12 cbQosPolicyIndex and cbQosObjectsIndex via Net-Snmp

Page 24




Command 3: snmpget 1.3.6.1.4.1.9.9.166.1.15.1.1.15.x.x for cbQosCMDropPkt in cbQosClassMapStats

Figure 13 cbQosCMDropPkt via Net-Snmp

5.9

Cisco IOS SLA Implementation

The device memory limitation made us not use new IOS 12.4, so we could only configure and test old SLA features based on rtr commands. Local host configuration:snmp-server host 10.1.12.56 rtr snmp-server host 10.1.12.56 syslog ! rtr 10 type jitter dest-ipaddr 10.245.0.5 dest-port 3000 source-ipaddr 10.6.0.8 request-data-size 256 rtr schedule 10 life forever start-time now rtr reaction-configuration 10 connection-loss-enable threshold-falling 3 threshold-type immediate action-type trapOnly rtr 30 type echo protocol ipIcmpEcho 10.245.0.5 source-ipaddr 10.6.0.8 request-data-size 512 rtr schedule 30 life forever start-time now rtr reaction-configuration 30 connection-loss-enable threshold-falling 10 threshold-type immediate action-type trapOnly

Remote host configuration:rtr responder

Page 25




5.10

Cisco IOS SLA Results

The Kiwi Syslog manager is used as syslog and trap manager.

Figure 14 Syslog configuration via Kiwi Syslog

Figure 15 Trap Configuration in Kiwi Syslog

Page 26




After Kiwi start services, we could find some messages below.

Figure 16 Received Trap & Syslog in Kiwi Syslog

We could see each time when the threshold breach, the snmp manager received two messages, one from syslog, another from rtr trap, which followed Cisco documentation. And this could also be configured to send mail alert.

Figure 17 Mail Setup in Kiwi Syslog

Page 27




5.11

HP Openview Performance Insight on SLA Results

The report server should show report based on collected SLA data. Check delay

Since round-trip response time is equal to 344ms, the one-way delay is around 172ms that is more than 150ms. So we could receive alert shown above, and also conclude the voice quality between Toronto and India could not meet VOIP requirement. The voice quality is worse.

Figure 18 Jitter Summary in HPOVPI

Figure 19 Response Time in HPOVPI

* This is round-trip delay, not one-way delay.Page 28




- Check Jitter Jitter is good, since it is only around 20ms, less than 100ms. It should be able to compesate on end unit

Figure 20 Jitter Results in HPOVPI

- Check Packet Loss No packet loss occur, or minor that can be ignore

Figure 21 Packet Loss in HPOVPI

Page 29




6

Conclusions

VoIP QoS Monitoring in the enterprise network is an essential part of providing consistent service levels. In this work, we propose and explain a monitoring procedure to quickly locate network and device bottlenecks from just monitoring two factors: queue drops on device interfaces, and monitoring end-to-end link quality for WAN. Our proposal do not replace traditional VoIP QoS monitoring solution, but compliments nicely existing VoIP monitoring best practices. A good VoIP QoS monitoring should not limit itself monitoring just the end-to-end voice call quality, but also the low-level device and network health.

Page 30




7

Future Works

Automating our VoIP monitoring procedures will be a good contribution, thus making fetching cbQosCMDropPktOverflow value easier and automatically. One could also integrate the proposed two procedures for monitoring the LAN and WAN health. We could also set up some event correlation to analyze these two kinds of traps, thus getting more accurate information for further troubleshooting.

Page 31




8

References[1] Cisco Documentation Support : http://www.cisco.com/en/US/customer/support/tsd_documentation.html [2] SNMP introduction : http://www.net-snmp.org/ [3] Configuring Cisco Voice Over IP ( Second Edition), Michael E.Flannagan & Jason Sinclair, 2002 Syngress Publishing Inc. [4] VoIP Service Quality - Measuring and Evaluating Packet-Switched Voice, William C. Hardy, 2003 McGraw-Hill Com. [5] Best Practices for Monitoring Cisco System IP Telephony Networks with AppManager, White Paper, March 2006 NetIQ Corporation [6] A Handbook for Successful VoIP Deployment: Network Testing, QoS, and More, John Q. Walker, 2002, NetIQ Corporation [7] Monitoring Voice over IP Quality of Service, Document ID: 17962, Aug 10, 2006, Cisco Systems Inc. [8] Cisco IOS SNMP Traps Supported and How to Configure Them, Document ID: 13506, Jul 09, 2007, Cisco Systems Inc [9] Cisco Class-Based Qos Configuration and Statistics MIB, 2000-2002, Cisco Systems Inc. [10] Cisco IOS Network Mangement Configuration Guide, Release 12.4, 2005-2006, Cisco Systems Inc. [11] Cisco AVVID Network Infrastructure Enterprise Quality of Service Design, August 2002, Cisco Systems Inc.

Page 32




Appendix A: ISP Network LatencySome of ISP network SLAs specify their maxium latency Qwest SLA - 50ms maximum latency - Measured Actual Axiowave SLA - 65ms maximum latency Verio SLA - 55ms maximum latency Internap SLA - 45ms maximum latency

Page 33




Appendix B: Commericial VoIP Monitoring SolutionsVoIP is susceptible to various shortcomings in the network. Quality of Service (QoS) is paramount for the enterprise, as poor quality may affect their business continuity. VoIP, being a real-time application is more demanding on the network than the traditional data application. VoIP call is transmitted over User Datagram Protocol (UDP), which does not ensure all the packets are delivered in sequential order leading to problems relating to latency, delay and jitter. In extreme cases the packets may not reach the receiving end causing packet loss. This makes VoIP call management inevitable. After enterprise implements VOIP, people always want to know whether my implement could meet voice requirement and which bottleneck need to pay attention to before real issues come. Most of current available VOIP QoS Monitoring tool is based on SIP protocol. AdventNet Inc.Manage Engine VQManger monitors any device or user-agent that supports SIP. Avaya VoIP Monitoring Manager collects Real-Time Control Protocol (RTCP) packets sent from Avaya IP phones, media gateways, and media processor resources during actual calls, and makes the information available in real-time or stored for time-based analysis. Tools from Agilent Technologies, NetIQ, and fluke network are also available for this purpose.

Page 34




Appendix C: SLA Availability ComputationAvailability calculations are performed for ICMP, UDP, TCP Connect, DHCP, and DNS SLAs. The two types of availability noted are device and link. Device availability Device availability (or reachability) refers to the source device. It is calculated by the CM. It tracks changes in devices using SysUpTime. When the CM collects data, it samples data once every minute. The data is grouped into a sample for a 5-minute time period, which is reported to the SLM server. A device is considered available if it responds to at least one of the five sampled times. This yields a collection sense of 1. If any other collection sense results, the minute sample values (each of the five samples) are used to determine if the device was available. The basic calculation for availability is uptime minus downtime. Link Availability Link availability is computed using the rttMonHistory table MIB variable, rttMonHistoryCollectionSense.If the target device is unreachable from either a network path issue or disruption of service within the target device, that is counted as link availability. 1- For ICMP and UDP, rttMonHistoryCollectionSense values associated with link down errors are: disconnected (2) timeout (4) notConnected (6) 2- For DNS, rttMonHistoryCollectionSense values associated with link down errors are: disconnected (2) timeout (4) notConnected (6) dnsServerTimeout (11) 3- For DHCP, rttMonHistoryCollectionSense values associated with link down errors are:Page 35




timeout (4) 4- For TCP Connect, rttMonHistoryCollectionSense values associated with link down errors are: timeout (4) notConnected (6)

Page 36




Appendix D: Cisco 4500 Switch QoS Configuration! enable qos qos ! create qos mapping cos-dscp qos map cos 3 to dscp 26 qos map cos 4 to dscp 34 qos map cos 5 to dscp 46 ! ip access-list extended voice remark UDP Ports for VoIP - Bearer permit udp any any eq 5200 ip access-list extended voice-control remark VoIP Control Traffic permit tcp any any range 5000 5100 ip access-list extended netmgt remark CW2K permit ip host 172.17.1.100 any ip access-list extended best-effort remark All other traffic permit ip any any !classify traffic class-map match-all VOICE description VoIP Bearer traffic match access-group name voice class-map match-all VOICE-CONTROL description Voice Control traffic match access-group name voice-control class-map match-all NETMGT description network management stations match access-group name netmgt ! policy-map Mark-Classify-AccessLayer description Mark-Classify traffic class VOICE set dscp ef class VOICE-CONTROL set dscp af31 class NETMGT set dscp af21 class class-default set dscp default ! !interface configuration example interface GigabitEthernet3/8 description Access Port Fa3/8 switchport access vlan 39 switchport mode access service-policy input Mark-Classify-AccessLayer udld port aggressive tx-queue 3 priority high shape percent 30 spanning-tree portfast ! !Uplink Trunk example interface GigabitEthernet1/1 description Uplink switchport trunk encapsulation dot1q switchport trunk allowed vlan 39 qos trust dscp tx-queue 3 bandwidth percent 30 priority high shape percent 30

!

Page 37


Documents

8001 Linxin Qian VoIP QoS Monitoring