Unity Connection 2.0 Mixer

1© 2005 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialSession NumberPresentation_ID

Unity Connection 2.0 Mixer

UCBU Media Team

June 2007


Agenda

•Overview of Mixer Architecture and Features

•Placement of mixer in Unity Connection 2.0 architecture

•Mixer characteristics / supported features

•Administrative Interface

•Control Interface and Signaling•API used for Miu to CuMixer messaging•CuMixer communicates with mixer via API library calls


Agenda (continued)

•Debugging•Tracing mixer API requests and responses•Logging of debug and error messages•Interactive Diagnostic Interface (TAC CLI)•Troubleshooting audio problems


Mixer Overview

Mixer features: Mixer is software only, no DSPs are used.It runs as a user mode process (not a kernel driver).Supports G.711(u/a), G.726 (32kbps), and G.729a codecs (all mono, 8K sample rate)Voice Activity Detection (VAD, with comfort noise packet sent at start of silent periods)RFC-2833 DTMF relay (both send and receive)Performs Text To Speech (TTS) playback of text files using Nuance RealSpeakICMP port unreachable detection and reporting for media failureMaintains numerous statistics on channels


Mixer placement in Unity Connection 2.0

CsMgrTAC-CLI(Diag. Interface)

CuMixer Control Application (process)

GY_API

CuMixer makes function calls to the mixer API layer

Message Queue

Soft-mixer (gy_mixer) components

RTPFile I/OPlayer - Recorder Audio Mixing

API

Logging

•Recording from endpoints to disk•Playback of audio and text files. Log Files for debugging

•Decode/encode RTP streams•“fork” audio from EP to ASR

Nuance TTS Engine


Mixer Debugging

Options Accessible to AdministratorMixer Log Files:

• Mixer log files are found by using Real-Time Monitoring Tool (RTMT) “Trace & Log Central”

to collect the mixer log files.• Log contains error information and event messages, detail depends on selected level.• Debugging level can be set using the Unity Connection Serviceability web page:

• Go to Unity Connection Serviceability->traces->micro traces->mixer• Set level to value between 0 and 6 (6 is most detailed debugging traces)• Or, go to Unity Connection Serviceability->traces->macro traces->media (wave) traces,

which turns on Miu and Mixer tracing.• Error conditions are always logged, regardless of logging level. (Default debugging level = 4.)

Log file name: • Filenames are “diag_CuMixer_00000xx.uc” where xx=0 thru 39.• Log files close after 200K written, and next file is opened. • Log files wrap after 40 files.


Debugging Mixer Issues

Checking the mixer logs for errors:

1. search logs for keywords “failure” or “error”:

-- You may see ICMP port unreachable log msgs. These are reported when the mixer is streaming to an endpoint that has closed its RTP port. It usually means some sort of signaling problem, where the call has terminated but theCUC session has not.

-- Any other instance of “error” or “failure” is significant, andcould indicate: a cmd sequence error or invalid request received; Bad recorded prompt, or other issue.

2. Turn on more detailed logging if needed.


Debugging Mixer Issues

3. Search logs for API message types to follow progress of call.

Some Useful Message Types:GY_MSG_OPEN_CHN: Start of an endpoint sessionGY_MSG_CLOSE_CHN: End of an endpoint sessionGY_MSG_OPEN_FORKGROUP: Create a new fork groupGY_MSG_CLOSE_FORKROUP: Close a for groupGY_MSG_ADD_CHN_FORKGROUP: Add a channel to a FGGY_MSG_RM_CHN_FORKGROUP: Remove a channel from a FGGY_MSG_PLAY_FILE: Play an audio or text fileGY_MSG_STOP_PLAY_FILE: Pause playback of a fileGY_MSG_CLOSE_PLAY_FILE: Terminate and close playback of a fileGY_MSG_PLAY_FILE_EVT: From mixer – indicates play file doneGY_MSG_RECORD_FILE: Start recording to a fileGY_MSG_STOP_RECORD_FILE: Pause file recordingGY_MSG_CLOSE_RECORD_FILE: Stop recording and close file

NOTE: Mixer response messages to the Miu commands above have same mnemonic, but with “_RES” appended to the end of the message ID.


Mixer Debugging

Options Accessible with “root” CLI Access

Mixer Log Files:• Mixer log files can also be accessed directly in directory “/var/log/gy ” when logging into the

server as “root” and using “cd” to navigate to that directory.• Filenames are “cisco_gy_log_file_xx” where xx=0 thru 39.

:


TAC Command Line Interface

Text-based command line interface “taccli”:

- “root” logon required. Just type “taccli” to start it up.

- Provides access to mixer diagnostic information

- Allows control of mixer tracing level (alternative to CUCS screen)

- Display GYAPI messaging counters, errors, other stats

- Activate / deactivate channel diagnostic recording

- Display mixer return code table for error code lookup

- Display other information



Menu Screen:

[root@unityconnection root]# taccli

CLI to mixer queue created, Queue ID=0x12340001

*********************************************************Unity Connection Mixer Diagnostics Interface*********************************************************

Please enter a request:1 Display statistics for all channels 8 Activate tracing2 Display statistics for a channel 9 Deactivate tracing3 Display statistics for all fork groups 10 Show current date/time4 Display fork group statistics 11 Display status of a channel5 Display mixer status 12 Display mixer return code table6 Display fork group details 13 Start/Stop channel RTP recording7 Quit :



Displaying statistics for all channels:

******************************* - 118 - ****************************************Chn ID: 118Start Time: 05/01/07 16:37:34Elapsed Time: 0:00:41Total Pkts Recv: 0Total Pkts Sent: 2Total Bytes Recv: 0Total Bytes Sent: 161Total Pkts lost/Too late to play: 0Packet Size: 0Max. Jitter: 0Endpoint IP Addr: 172.19.248.66Endpoint RTP Port: 7774

************************* - End of channel display - *************************

All Channels statistics display (option 1)



******************************* - 119 - ****************************************Chn ID: 119Start Time: 05/01/07 16:37:22Elapsed Time: 0:00:53Total Pkts Recv: 425Total Pkts Sent: 1387Total Bytes Recv: 60527Total Bytes Sent: 218853Total Pkts lost/Too late to play: 1Packet Size: 160Max. Jitter: 1Endpoint IP Addr: 10.32.132.52Endpoint RTP Port: 28822

************************* - End of channel display - *************************

All Channels statistics display (option 1) continued



******************************* - 119 - ***********************************Chn ID: 119Start Time: 11/27/06 21:30:29Elapsed Time: 0:00:03Total Pkts Recv: 78Total Pkts Sent: 89Total Bytes Recv: 12480Total Bytes Sent: 14240Total Pkts lost/Too late to play: 0Packet Size: 160Max. Jitter: 4Endpoint IP Addr: 10.21.145.2Endpoint RTP Port: 49152

Single Channel Statistics Display (option 2)



Chn ID: Channel numberStart Time: Time at which the channel was openedElapsed Time: Amount of time the channel has been open (in seconds)Total Pkts Recv: Total packets received from the channel endpointTotal Pkts Sent: Total packets sent to the channel endpointTotal Bytes Recv: Total bytes received from the channel endpointTotal Bytes Sent: Total bytes sent to the channel endpointTotal Pkts lost/Too late to play: Total dropped packetsPacket Size: in bytesMax. Jitter: in millisec. This is the maximum observed value of the average pkt jitterEndpoint IP Addr: Remote IP address of the endpointEndpoint RTP Port: Remote RTP port used by the endpoint

Channel Statistics field explanations



Display all forkgroups:

******************************* - 249 - ****************************************ForkGroup ID: 249ForkGroup Sched. ID: 0Start Time: 05/01/07 17:09:18Elapsed Time: 0:02:00Num. Active Chn: 0Max. Active Chn: 2Join/Depart ForkGroup: 16Total Pkts Recv: 209Total Pkts Sent: 1737Total Bytes Recv: 18971Total Bytes Sent: 275694Max. Jitter: 1Max. Playout Delay: 0Last Playout Delay: 0Max. Record Delay: 0Last Record Delay: 0

*********************** - End of ForkGroup Display - **********************

All Forkgroups Display (Option 3)



Display forkgroup statistics:

Please enter a forkgroup ID between 0 and 249: 249

******************************* - 249 - ****************************************ForkGroup ID: 249ForkGroup Sched. ID: 0Start Time: 05/03/07 14:14:30Elapsed Time: 0:00:33Num. Active Chn: 2Max. Active Chn: 2Join/Depart ForkGroup: 14Total Pkts Recv: 136Total Pkts Sent: 84Total Bytes Recv: 9517Total Bytes Sent: 13122Max. Jitter: 2Max. Playout Delay: 0Last Playout Delay: 0Max. Record Delay: 0Last Record Delay: 0

Forkgroup Statistics Display (Option 4)



** Unity Connection Mixer Systemwide Statistics **

Mixer Start Time: 05/01/07 16:35:24Elapsed Time: 0:48:44ForkGroup Active: 1Max. Simultaneous ForkGroup: 1ForkGroup Total (cumulative): 4Current Active Channels: 2Channels Total (cumulative): 8Longest ForkGroup Duration: 128Total Packets Received: 2709Total Packets Sent: 14060Total Bytes Recv: 357438Total Bytes Sent: 2232905Total Packets Dropped: 18Max. Jitter Observed: 2

System-wide Statistics Display (Option 5)



Please enter a forkgroup ID between 0 and 250: 249

******************************* - ForkGroup Status - *******************************

forkgroup ID: 249ForkGroup Name: conf249ForkGroup State: OpenCurrently playing prompt: None Current record file: None Number of ForkGroup channels: 2ForkGroup channel IDs: 118 119 Number of contributing channels: 1Channel IDs: 119

ForkGroup Details Display (Option 6)


: 8Request: Activate Tracing:Trace levels are 0-6 (6=Maximum).Enter desired trace level: 6selection was: '6'

Mixer trace activated.

: 9Request: Deactivate Tracing:

Trace deactivated.

:10

Current system time: 05/03/07 14:25:03

Activate/Deactivate Tracing; Show Date-Time (Options 8, 9, 10)



******************************* - 119 - ****************************************Chn. ID: 119Channel Status: Active, not currently in conferenceConf. ID: NoneContribute to mixer: NoEndpoint IP Addr: 10.21.145.2Endpoint RTP Port: 49152Endpoint RTCP Port: 0Codec Type: G711aLawIngress frame Size: 20Bitrate: 64000Ingress Payload Type: 8Ingress DTMF Payload Type: 101Ingress DTMF Enabled: YesMixer IP Address: 172.19.136.149Mixer RTP Port: 19794Mixer RTCP Port: 19795IP TOS Precedence: 0x 2eTTL: 64Egress Codec Type: G711aLawEgress Frame Size: 20Egress Bitrate: 8000Egress Payload Type: 8Egress DTMF Payload Type: 101Egress DTMF Enabled: YesEgress Mute: NoEgress VAD: NoPrompt Active: YesPrompt File Name: /opt/cisco/meetingplace_express/afs//runprompt/en_US/s1739.wavRecord Active: NoDiagnostic Record Active: No

Channel Status Display (Option 11)


:13

Please enter a channel ID between 0 and 250. (To deactivate all diag. recording, enter 999): 118

Enter 1 to start or 0 to stop recording: 1

Recording activated, saving to file /tmp/chn.118.05.07.07.16.10.03.wav

:13



Recording deactivated for RTP stream 118.

:13



Recording activated, saving to file /tmp/chn.118.05.07.07.16.10.03.wav

:13



Recording deactivated for RTP stream 118.


Start/Stop Channel RTP Recording (Option 13)


Debugging Audio Quality Issues

Debugging audio stream problems:

• Locate the channel of interest, and check the mixer andchannel statistics for excessive jitter.

• Options if the problem can be isolated to a particular caller:1. Use diagnostic channel recording to capture what is coming into the mixer.2. Check channel stats for dropped packets, jitter, etc.

Try to see if the audio was already bad coming into the mixer:Use Ethereal to capture audio in and out of the mixer (eth0). Ethereal can

create an “.au” file from G711 RTP, and do stream analysis.-- Select any RTP packet from the problem endpoint-- Select: “Statistics”->”RTP”->”Stream Analysis”->”Save Payload”-- Select “forward channel”, then specify file name (c:\xxx.au), then save file.

-- The .au file can then be played with Windows media player. Is the incoming audio stream already bad? If so, then look for network problems.


Debugging TTS Issues

• The TTS Engine is a third-party product from Nuance Communications, Inc.

• From Cisco Unity Connection Servicabilty web page:Go to CUCS traces->macro traces->Text to Speech traces

Turn on tracing

Use RTMT to collect traces and search for “error”

• Or, with root access: Edit file

/usr/local/Scansoft/RealSpeak4.0/config/ttsserver.xml.

Make sure parameter “log_file_enabled” is set to “true”.

Parameter “log_file_base_name” is the name of the log file

Search for “error” in the log file (XML format)


Currently open bugs

The following are known mixer issues which are expected to be fixed in the next release:

1. CSCsi39808: VUI voice commands may occasionally not be recognized by the mixer’s ASR engine. This problem is under investigation.

2. CSCsi41515: Some .wav prompt files may contain other chunks preceding the “fmt “ chunk, and the mixer will not be able to play such .wav files. This would only be seen if a user has substituted their own prompt files for the standard ones provided, and if such provided files have this unusual characteristic. This issue is fixed in the next release.

3. CSCsi60118: If a user attempts to use the TTS feature with a language which is not installed on that system, they may experience a delay without any playback, any error response, or other feedback, until the maximum time-out for playback occurs. This is fixed in the next release.

4. CSCsj14600: Comfort Noise packets not using industry standard payload type value.


For More Information

• Related Docs:

- Lucas Messaging Software Architecture Proposal, EDCS-490444

- Lucas: Mixer and App Interfaces, EDCS-500086

- Bounty Hunter core audio mixer SUFS/SUDS – EDCS-415152

- Bounty Hunter core mixer diagrams: EDCS - 486998


Documents

Unity Connection 2.0 Mixer