Maximizing Voice Services Uptime in VoIP Environments

Bill Caraher, von Briesen & RoperJim McCue, Rodey Law FirmJim McCue, Rodey Law Firm

OverviewOverview

• Phone systems and network designPhone systems and network design• VOIP components and services:

– What are they?What are they?– Where are they?– Essential in a disaster?Essential in a disaster?

• Vendor approaches to disaster recovery/business continuity recovery/business continuity

• What can go wrong?– How do we adjust or do we?How do we adjust or do we?

Rodey Phone System/Network Design

• Rodey Law Firm: – 70 attorneys, 150 people, 2 offices in Albuquerque and Santa Fe, New

Mexico

• Shoretel Phone System replaced Siemens 9751 PBXy p– Headquarters server in Albuquerque, Distributed Voice Services server

in Santa Fe (2 vm servers)

• Network: HP Procurve Gigabit POE SwitchesNetwork: HP Procurve Gigabit POE Switches– Vlans: data per floor, voice– Routing switches– MSTP: Multiple Spanning Tree Protocol per Vlan Spanning TreeMSTP: Multiple Spanning Tree Protocol, per Vlan Spanning Tree

• Interoffice connection: QMOE (Qwest Metropolitan Over Ethernet)

Rodey Phone Diagram:/Phones/Switches

Rodey Phone Diagram:External Lines

Rodey Network DiagramRodey Network Diagram

von Briesen Phone System/Network Design

• von Briesen Law Firm: 110 Attorneys 250 von Briesen Law Firm: 110 Attorneys, 250 total employees.

• 4 Offices in Wisconsin• 4 Offices in Wisconsin• Cisco VoIP Replaced (4) Legacy PBX Systems• Centralized Voice System at HQ• Dual P2P T1 between HQ, PRI & POTS @each, @• Voice VLANs & QoS Routing

von Briesen Voice Infrastructurevon Briesen Voice Infrastructure

von Briesen Network diagramvon Briesen Network diagram

Phone ServicesPhone Services• Need for in a DR scenario?• Switch to outside service? Switch to outside service?

– Conference Calling, faxing• Essential?:

– Internal 4 digit dialing– Outbound phone calls– Inbound phone calls: Inbound phone calls: – Voice mail– Unified Messaging– Auto attendant– Call forwarding– 911 services

• Less essential?– TAPI – PC interface with phone system– Conference Calling– Cost Recovery– Faxing: Inbound, outbound

O h d i– Overhead paging– Music on Hold– Extension monitoring– Presence and IM– Dial by Voice

Shoretel DR FeaturesShoretel DR Features• Distributed Call Control

– Distributed call control on each switch:Distributed call control on each switch:– Call setup, breakdown, conferencing, transfer, conference, forward– Peer to peer: switch to switch

• Distributed Voice Applications– Voice mail– Auto attendant– Unified messaging– Call control

• N+1 redundancy– Voice switch failure

• Phone-switch heartbeatC l i h d h d – Caveat: lose a switch and a headquarter server

• Voice mail failover/Auto attendant failover– DVS to headquarters, DVS to DVS– Headquarters won’t failover to DVS– Auto attendant to backup auto attendant

• S Hi h A il bilit• Server High Availability:– Doubletake– VMware with HA

• PSTN failover– DRS enabled?

Shoretel DR FeaturesShoretel DR Features

• Outbound Trunk Call Routing/Failover:– Selects from all lines marked for type of call (local, ld, international, 911, etc.)– Trunk group rollover

• Extension portability:Extension portability:– Different phone– Wireless phone– Soft phone– Office Anywhere– VPN accessible: phone, soft phone– Mobile Call Manager - blackberries

• 911 services:911 services:– Ensuring 911 calls and accurate location information are routed to the

appropriate Public Safety Answering Point (PSAP)– Native Shoretel 911

E911 Add – E911 Add on

Cisco DR FeaturesCisco DR FeaturesPhone System Disaster Recovery• Redundant Call Manager – Subscriber & Publisher• Supports Virtualization for Server Based Components• SRST Gateway Failover Local Survivability in Branch Offices & HQ• SRST Gateway Failover – Local Survivability in Branch Offices & HQ• Voice Mail

– Clustered Unity Server (Windows)– All Voicemail stored in Exchange –no data loss or data failover w/clustered Exchange

• Auto attendant failover Subscriber / Publisher• Auto attendant failover – Subscriber / Publisher• Failover trunk groups –from carrier• POTS backup• Service Advertisement Framework (SAF) New in CM 8.0 –services and DN can be advertised

by different hardware when a failure occures no manual switchoverby different hardware when a failure occures, no manual switchover.

Location Disaster Recovery• Centralized Phone System in Data Center• Soft phonesSoft phones• Take desk phones remote via Internet Proxy –no VPN Necessary• Power Loss – Have local UPS and bricks available when PoE is used.

Feature Disaster RecoveryFeature Disaster Recovery• Virtualize or Cluster important services –but weight cost vs. benefit

What can go wrong?What can go wrong?

What can go wrong?What can go wrong?

Phone failurePhone failure• Phone Failure

– Users value their phone service!:

– Rodey • Replace phone

L i h h (LAN Wi l VPN)• Log into another phone (LAN, Wireless, VPN)• Use Softphone (LAN, VPN)• Office Anywhere• Mobile remote (BB)( )

– von Briesen• Log into another phone – Cisco Extension Mobility• Replacement Hardware on Hand• Single Number Reach –calls go to Cell Phone Automatically• Cisco Mobility –outbound dial via Cell Phone• Proxy Desk Phone via Internet and ASA• SoftphonesSoftphones

Phone switch failurePhone switch failure• Phone switch failure (Jim)

– IP switch– Analog switch– T1 switch

• SolutionsSolutions– Rodey

• Shoretel IP Switch failover:– local, remote

• Analog Switch:Analog Switch:– Replace– Move extension in software, hard wire

• T1 Switch:– Redundant lines– Replace

Phone switch failurePhone switch failure

Phone Switch FailurePhone Switch Failure

– von Briesenvon Briesen• Cisco Core Network Routing & Switching• Cisco Endpoint Switching• PoE / Bricks• SRST: Phones register with Gateway (Router)

LAN FailureLAN Failure• LAN failure von Briesen

– To phone: Single network cable into core switch w/VLAN T M lti l t k bl i t it h /VLAN– To server: Multiple network cables into core switch w/VLAN

• Solutions– von Briesen

• Rely on Core Switch for fault tolerance on LAN• Long term plan is to Virtualize Servers, easy to spin up/restore.

– Rodey• Spanning Tree per vlan• Dual port switches• Dual port servers

S k h !• Spare network switches!• Wireless phones• DHCP

Interoffice line failuresInteroffice line failures

• Interoffice Line FailuresInteroffice Line Failures– MPLS, Frame Relay, T1, Metro over Ethernet, etc.

• SolutionsS– Rodey

• PSTN failover• Backup Internet VPN• Switch is call manager

– von Briesenvon Briesen• Multiple T1 lines between offices, Internet VPN backup, SRST local

survivability.• Long term failure Phone Proxy through Internet• Long term failure – Phone Proxy through Internet

External line failuresExternal line failures• External line failures

– External lines are the weakest link in your phone system!:

– Types:Types:• Analog• PRI• SIP

• Solutions:von Briesen– von Briesen

• Outbound – multiple PRIs, Call Routing over Data Lines, POTS• Inbound – multiple PRIs, Calls can originate from any PRI (Telco Trunk DID Trunk Group Failover)• SIP solution for Cisco

– Rodey• Outbound – PRI call routing/failoverOutbound PRI call routing/failover

– Alb PRI2 to Alb PRI1 in descending order– Alb PRI1 to Alb POTS line– SF PRI to Alb PRI1, PRI2– SF PRI to SF POTS line

• Inbound– Alb PRI1 to Alb PRI2 in ascending orderAlb PRI1 to Alb PRI2 in ascending order– SF PRI to SF POTS line– SF PRI via Qwest BCR to Alb PRI

• SIP solution for Shoretel?

• `

Server failure: von BriesenServer failure: von Briesen

Publisher & Subscriber Failure ScenarioPublisher & Subscriber Failure Scenario

The cool thing about Cisco is that you can lose both the Call Manager P blisher & S bscriber both the Call Manager Publisher & Subscriber and still have dial tone. Phones can re-register with the Gateway at HQ and Branch Officeswith the Gateway at HQ and Branch Offices.

*Some Features will not work when in SRST mode.

Server failure: RodeyServer failure: Rodey

• Headquarters down, DVS upq p• Voice Mail:

– Headquarters VM forwards to switch based backup auto attendant (not to SF DVS)attendant (not to SF DVS)

– Remote VM still functional• IP phone failover will not work at either location• PCM:

– Headquarters doesn’t work– Remote site worksRemote site works– Can’t change call handling mode/configuration

• Users can’t reassign extensions

Server failure: RodeyServer failure: Rodey

• DVS down, Headquarters upq p• Voice Mail:

– Headquarters VM still functionalR t f d t H d t VM– Remote users forward to Headquarters VM

– Remote users can’t use Office Anywhere, reassign extension• IP phone failover worksp• PCM:

– Headquarters works R t it d ’t– Remote site doesn’t

– HQ Users can reassign extensions, remote can’t (uses DVS VM)

Power FailuresPower Failures• Power failure

– Total– Computer Room– Network Switch Rooms– Personal Offices

• Solutions– Rodey

• POE, UPS, n+1 power supplies, n+1 ups, power bricks• Analog line hard wired for emergencies, power backup runs out

– von Briesen• 95% of all phones have local power bricks, not on PoE. Data Center on

UPS Power that will hold for 90 Minutes. After that, HQ is dead—but branch offices work on SRST. In ‘11 or ‘12 the DC will be moved to a co-location facility with generator backup power.location facility with generator backup power.

• Backup Analog lines will only work if the phones and gateway have power.

Office lossOffice loss

• Office lossOffice loss– Headquarters, remote office, collocation facility?

• S l ti• Solutions– von Briesen:

• A t t l l f th HQ ld b bl ti Th b h ffi ld • A total loss of the HQ would be problematic. The branch offices could operate but we would need to re-establish a Call Manager and ASA Proxy through another office or co-location facility. Voicemail is also at the HQ, so that and the feature servers would need to be rebuilt.

Office loss-RodeyOffice loss Rodey

• Redirect DIDS with BCR– BCR: Qwest Business Continuation Routing

• Call forwarding service• Predefined groups of numbers• Predefined groups of numbers• Forward to up to 2 alternate locations• Triggered by phone call

DID i f d• DID info preserved• Dependent on CO functionality

• Restore HQ server or DVS– Reconfigure effected extensions– Use either Doubletake or VMware to sync copy of HQ

server/DVSserver/DVS

Plan for the Minor DisastersPlan for the Minor Disasters• Troubleshooting a faulty VoIP system can be tricky.

Intermittent problems are the worst and hardest to resolve!e e p ob e s a e e wo s a d a des o eso ve!• Back into the problem – what doesn’t work and then think

about what is required for that part of the system to function.• Keep documentation up to date and available.

– Example: Intermittent one way audio on some Inbound long distance calls. Enable verbose logging, write output to a syslog server, QRG buttons on phones that are

bl ti it t d d l t th t f l i problematic –write an extended log to the system for analysis. – Open a ticket with the Telco carrier. Request a dispatch to rule out local equipment.

They always want to blame CPE. Had a scenario recently—closed my ticket twice stating it was CPE with a faulty DID. I finally got a tech dispatched, he agreed there was a problem on their end, they determined that the number got ported on accident.

– Open a ticket with the VoIP system vendor. Often times they will be helpful in helping to get to the root cause of the failure. Be prepared with version numbers, log files and lots of details to get the call going quicker.

SummarySummary• Jim

f f f (– Look for all the points of failure (phones, switches, lan, wan, external lines, power, operator) and test solutionssolutions

– Don’t create your own disastersDon’t forget o r network!!– Don’t forget your network!!

– Rodey weakness – dependence on HQ server for phone/switch assignment changes Alb VMphone/switch assignment changes, Alb VM

• Solution: VMWare with HA certified with version 11 • Solution: move VM in Albuquerque to new Alb DVSSolution: move VM in Albuquerque to new Alb DVS

SummarySummary

• BillBill– Successful Disaster Recovery involves planning and execution. Chose

the right telco partner that can automate as much fail-over as possible. K th WAN d LAN h lth w h l l i tw k Keep the WAN and LAN healthy—you now have a local voice network. Look for single points of failure in the design and system. Test your DR plan. Weigh the cost vs benefit on some redundancy scenarios—is it really necessary to have every feature workingreally necessary to have every feature working.

Audience questions and real life experiencesp