SUM303: XenDesktop Advanced Troubleshooting

SUM303: XenDesktop Advanced Troubleshooting

Mick Glover

Senior Support Readiness Specialist

October 16, 2012

© 2012 Citrix | Confidential – Do Not Distribute

Agenda

2

• Overview

• XD5.x Architecture Basics

• Logging & Tracing

• Key Support Tools Overview

• Using PowerShell to monitor your site

• Troubleshooting Example(s)

Overview


Before we begin..A Question

Q. To date, what is the most common problem logged with technical support

worldwide in 2012?

A. VDA Connectivity issues (Registration, Connection/Disconnection…)

4


The Good News…

5

• Its not all about verbose Logging and Tracing…

• The following goes a long way…○ Good Understanding of Product Architecture○ Knowing what to expect○ Knowing what to trace and when

XD5.x Architecture Basics


Machine CreationServices

BrokerService

InfrastructureServices

XenDesktop 5 Architecture - .Net SOA

Broker

Broker Service

Machine CreationService

AD Identity Service

Machine Identity Service

Host Service

ConfigurationService

Desktop Studio Desktop Director

WCF [80]

Virtual Desktop

Agent (VDA)

WinRM 2.0

PVS for VMs Agent

[5985/5986]

WCF [80]

PoSH PoSH

Group Policy Engine

SQL Server SQL Server SQL Server7


ServicesKey Responsibilities

8

• Broker Service

○ VDA registration, Resource allocation, Power MGMT & Licensing enforcement

• Configuration Service

○ Provides Directory Services, MetaData storage & security

• Host Service

○ Manages Hypervisor connection & resources


ServicesKey Responsibilities

9

• Machine Creation Service

○ Creates & manages VMs and Master images

• Machine Identity Service

○ Creates & manages the disks attached to VMs

• AD Identity Service

○ Creates & manages AD Computer a/c’s


Registration Basics

• VDA registration○ Process of locating a controller and establishing ongoing communication○ WCF web service endpoints on both ends facilitate the registration process

• Registration methods○ Registry base registration○ AD base registration○ Quick Deployment registration

• Registration types ○ Soft Registration○ Hard Registration

10


WCF Registration portSpecifics

11

• XD5.x: TCP/IP port 80

○ Controller and VDA

• Ports can be set differently on controller and VDA

• Special consideration when changing the WCF port on controllers if using

Registry based registration i.e. ControllerRegistrarPort


Configuring Registry based registrationListOfDDCs

12


Checking the WCF port on a ControllerBrokerService.exe /show

13


Changing the WCF port on a ControllerBrokerService.exe –vdaport 89

14


Changing the WCF ports on the VDA

15

• Re-configure the VDA ○ (add/remove programs)

Or

• HDX Computer Policy


Changing the WCF ports on the VDAConfirming the changes (registry)

16


Changing the WCF ports on the VDAConfirming the changes (.Config file)

17


ControllerRegistrarPortDetails

18

• Hardcoded on VDA to use port 80

• Value must be manually changed if VDA

WCF port is changed on Controller

• When using AD based registration

○ WCF port is included as part of the URL

attribute and therefore this manual change is

not required as each controller will update its

own details in AD


ControllerRegistrarPortRegistry value created after re-configuring the VDA

19


ControllerRegistrarPort - Known IssueChanging Virtual Desktop Agent Port Using Add/Remove Programs Fails

20

• Cause: ControllerRegistrarPort is created

incorrectly with an ending space

• Resolution: Manually delete and recreate

Registry value

• Note: CTX131263 also includes updated

binary's which can be deployed to resolve

the issue without having to manually edit

the registry


VDA HeartbeatDetails

• Value: HeartbeatPeriodMs

○ Originally a 1 minute timeout (60000ms) with the VDA sending

a ping every 30 seconds (an interval of half the timeout value)

• Since the release of XD5 SP1, the default

timeout/ping interval has changed to 10 & 5 minutes

respectively…

21

Logging & Tracing


.NET <Service>.exe.Config filesAvailable logging mechanisms

• "LogToCDF" - if present and set to 1, will log to CDF using the supplied CDF

trace module

• "LogFileName" - if present, will log to the specified file name

○ "OverwriteLogFile" - if present, will overwrite existing log to the file if it already exists

• Note: More than one sink can be specified, in which case logs will be output to

all of them…

23


Out of the box…

• CDF Tracing enabled for all services

• <Service>.exe.Config <appSettings>

<add key="LogToCDF" value="1" />

</appSettings>

24


Out of the box…

• CDF Tracing enabled for all services

• <Service>.exe.Config <appSettings>

<add key="LogToCDF" value="1" />

</appSettings>

• Clear text logging disabled

25


WCF Service Logging - ExampleEnabling clear text logging

1. Edit WorkStationAgent.exe.Config

2. Add the lines in Green below to enable and control clear text logging

3. Set appropriate NTFS permissions

4. Restart Citrix Desktop Service (WorkstationAgent.exe)

26


Service configuration logging reference - XD4

27


Service configuration logging reference - XD5

28


PortICA logging

• PortICA can produce both CDF and

clear text output for debugging

purposes○ Disabled by default

• Create PorticaConfig.XML and place

under the following directory on the

VM○ C:\Program Files\citrix\ICAService\XML○ XML directory must be manually created Note: Once PorticaConfig.xml has been created and

placed in the correct directory, it must be configured29


Configuring PorticaConfig.XML

• LogFile○ Level 0,1,5,9 supported○ 0 = disabled

• CdfTrace○ Level 1,5,9 supported

• FunctionTrace○ Values 1,5,9 supported○ Master flow control

<?xml version="1.0" encoding="utf-8"?><Config xmlns="Portica.xsd"> <Portica> <LogFile> <LogLevel>5</LogLevel> </LogFile> <CdfTrace> <LogLevel>5</LogLevel> </CdfTrace> <FunctionTrace> <LogLevel>5</LogLevel> </FunctionTrace> </Portica></Config>

Note: Cdf tracing is enabled by default for PortICA. The above values just control the verbosity of the output.

30


PortICA clear text logs

• Logs files are automatically saved to

the following locations○ XP C:\Documents and Settings\

LocalService\Local Settings\Temp ○ Win7 C:\Windows\ServiceProfiles\

LocalService\AppData\Local\Temp

31


Making the process easier…Logging Enabler v2.0

32


Logging Enabler v2.0VDA

33


Don’t forget Windows Event ViewerApplication Log - VDA

•XenDesktop 5.5 - VDA Event Log Messages http://support.citrix.com/article/CTX133154

34

http://support.citrix.com/article/CTX133154



Don’t forget Windows Event ViewerApplication Log - Controller

•XenDesktop 4.0 - VDA and DDC Event Log Messages - http://support.citrix.com/article/CTX133152

35




Finding & Validating ControllersVDA.log (WorkstationAgent.exe)

36


Confirming Successful Soft RegistrationBroker.log (BrokerService.exe)

37


Tips for searching Logs con’t

38


Soft Registration

• Soft registration ○ VDAs are not yet belong to a desktop group○ Only partial registration action done○ No config sent (XML) or status monitoring initiated○ No session launching or management is permissible○ VDA appears in Desktop Studio as registered○ Avoids unwanted registration delays○ Avoids network traffic and event logs

39


Viewing Soft Registered MachinesXDDBDiag - XenDesktop Site Summary

40


Hard Registration

• Promotes soft to hard registration○ Once VDA is added to a group○ Configuration data send to site database○ Status monitoring initiated (ready or

power-off etc.)

41


CDF Tracing overview

• The acronym goes by several names○ Citrix Diagnostic Facility○ Common Diagnostic Facility○ Common Diagnostic Function

• CDF is a system to capture diagnostic output from specific modules in Citrix

products which can be very helpful when troubleshooting issues

• The trace messages are coded into the product and displayed with appropriate

status and error codes when run

42

CDF messages in the source code

[[

OR

43


Trace Message Format Files (TMFs)How it works

• Raw CDF events are captured as GUIDs from the Citrix modules. To convert

these to human-readable strings, the GUIDs need to be matched with their

corresponding descriptions.

• The mapping of GUIDs to descriptions is accomplished by the use of TMF files

which are similar to PDB debug files…

44


TMF FilesPublic & Private

45

•There are two varieties of TMF files: Public and Private○ Public TMF files reveal basic state and function information for customers and partners

○ Private TMFs are for internal Citrix use only and are used by escalation engineers and

development while debugging. These files contain references to source code.

•Public TMF server: http://ctxsym.citrix.com/tmfs/xaxd

• Public TMFs are also available at ftp.citrix.com for externally use

Key Support Tools


Key Support Tools

• Tracing & Logging○ CDFControl○ CDFCheck○ CDFMarker○ CDFMonitor○ LogEnabler

• Monitoring & Collection○ Scout○ XD Site Checker○ HDXMonitor

• Database and Services○ XDPing○ XDDBDiag

47

Tracing & Logging


CDFControl (1/2)

• The primary tool used to control and consume CDF traces

• All available Citrix modules

• Additional features include ○ performance data capture○ tracing on system startup○ common error code lookups○ expert control to zero-in on true errors and failures

• Available at http://support.citrix.com/article/CTX111961

49


CDFControl (2/2)

50

Kimberly

Check to see if trace categories dropdown have been updated to coinside with the new modules with XD 5.6 FP1


CDFControl remote tracing

• Start CDFControl

○ Select View -> Remote Providers

○ Verify no firewalls are in place

○ Verify remote registry service running

• Select modules

○ Tools -> Remote Tracing

○ Saved to sub directory of CDFControl

'CDFControl_Remote_Traces\%computername%'

51


Remote Providers Error

•Make sure you can connect to registry on remote machine…○ Ensure Remote Registry service is started on remote system○ Ensure firewall is not preventing communications

52


Multi-Domain/Non-Trust Workaround…

53


CDFControl startup tracing

• Start CDFControl – select module(s)○ Tools -> Options (to set trace file path)○ Tools -> Startup tracing -> Enable

○ ETL file located at Startup trace file path

54


CDFMarker v1.3

• Stand alone executable ○ CDFMarker.exe

• Injects a customizable CDF trace

statement into CDF trace to help locate

narrow search

• CDFMarker outputs its statements to

MF_Hook_MMHook and therefore this

module must always be selected

55

© 2012 Citrix | Confidential – Do Not Distribute 56


CDFCheckCDFChkCmd.exe

• Suspect a specific binary as root

cause of issue

57


CDFCheck

58

• BrokerService.exe○ CdsController○ CdsInterService○ ControllerDal

Monitoring & Collection


XenDesktop Site CheckerV1.1

• Features

○ Control Logging (Enable/Disable)

○ Restart Services

○ Check Service Instances

• Views

○ Site Details

○ Node Details

○ Active Site Services (leases)

○ Assignments (Desktop Groups)

61


ScoutV2.4.0.2

• Collects data points and CDF traces

• Data package(s) uploaded to Citrix

Technical Support

• Auto-Update

○ Help -> Check for updates

62


ScoutData points (1/2)

• Hardware

• Software

Computer Bios Memory Devices

Processor PNP device Logical drive Page file usage

Network adapter & configuration

Video controller Printers IDE controllerSCSI controller

OS/Service packs PNP driver Driver

Processes MS Hotfix Grop Policy Info (RSOP)

63


ScoutData points (2/2)

• Farm

Hypervisor Site Information Controllers Catalogs

Desktop groups Configured VDA Broker administrators Installed DB version

XDDBDiag output Broker hypervisor connection

Connection leg

64


ScoutConfiguration

65


ScoutCDF Tracing

66


ScoutCollecting & Uploading Data

67


ScoutCheck-Update

• Compares local version against

latest version online

• If the Local and Server versions

do not match, click Download…

69

Database and Services


XDPing Database & Services

• CLI based tool

○ Verify configuration settings on DDCs and VDAs

• Can be run on the local console or remotely

○ Xdping

○ xdping /host MyDDC01.citrix.com

71




XDDBDiag

• Performs consistency check against

DB and summarises findings…

• Outputs the following diagnostic

data into comma-separated value

(.csv) files

○ Site Information, Virtual Desktop Agent Information, Current Connections / Connection Log, Hypervisor Connections, Policy Information, Desktop Groups, Controller Information & SQL Information

74


XDDBDiag output

77

Using PowerShell to monitor your site

PowerShell - Reference

• Load XenDesktop Snap-ins

○ asnp citrix*

• Return all available XenDesktop cmdlets

○ Get-command –module citrix.*

• XenDesktop 5.6 SDK Cmdlet Help

○ http://support.citrix.com/article/CTX127254

80

PowerShell Remoting

• Built-in remoting functionality

• -AdminAddress Parameter

○ Specifies the host name or IP address of the controller to

which the PowerShell snap-in will connect

• Default = “LocalHost”

• Note: Once a value is specified by any command,

this value becomes the new default

81


PoSH - Validating Service Status

• Use Get-BrokerServiceStatus (Broker Service)

• Use Get-ConfigServiceStatus (Configuration Service)

• Use Get-HypServiceStatus (Host Service)

• Use Get-AcctServiceStatus (AD Identity Service)

• Use Get-ProvServiceStatus (Machine Creation Service)

• Use Get-PvsvmServiceStatus (Machine Identity Service)

82


Get-BrokerServiceStatus - Return Values (1 of 2)

Status Meaning

OK The broker is connected to a database that is valid, and the service is running.

DBUnconfigured The broker does not have a database connection configured

DBRejectedConnection The database rejected the logon from the Broker Service. This may be caused by bad credentials, or the database not being installed.

InvalidDBConfigured The database schema is missing (possibly just the stored procedures in it).

DBNotFound The specified database could not be located with the configured connection string.

DBMissingOptionalFeature The broker is connected to a database that is valid, but it does not have the full functionality required for optimal performance. Upgrading the database is advisable.

DBMissingMandatoryFeatureThe broker is connected to a database that is valid, but it does not have the full functionality required so the broker cannot function. Upgrading the database is required.

83


Get-BrokerServiceStatus - Return Values (2 of 2)

Status Meaning

DBNewerVersionThanService The broker is too old to use the database. A newer version is required.

DBOlderVersionThanService The database is too old for the Broker Service. Upgrade the database.

DBVersionChangeInProgress A database schema upgrade is in progress.

OK

PendingFailure Connectivity between the Broker Service and the database has been lost. This may be a transitory network error, but may indicate a loss of connectivity that requires administrator intervention.

Failed Connectivity between the broker and the database has been lost for an extended period of time, or has failed due to a configuration problem. The broker service cannot operate while its connection to the database is unavailable.

Unknown The Service's status cannot be determined84


Setting the DB connection string(s)

• Required for troubleshooting, DB Migration…

• To view existing DB connection string○ Use Get-BrokerDBConnection

• Server○ Name of server hosting DB

• Database○ Name of XD5 database (CitrixXenDesktopDB = Default)

85


Setting the DB connection string(s) con’t

• Trusted_connection=true○ Enables Integrated Windows authentication

• The following error is returned if above parameter is not specified:

• Why? ○ Service instances use the local DDC machine a/c to authenticate to the DB○ SA authentication is not supported…

86



• To Reset Broker Service DB connection string○ Set-BrokerDBConnection –DBConnection $null

• The Broker Services will reset and return to an idle state until a valid new

database connection string is specified

• New value = DBUnconfigured

87



• To Set Broker Service DB connection string:○ Set-BrokerDBConnection –DBConnection

"Server=dbserver;Database=XDDB;Trusted_Connection=True“

• New value = OK

88


Setting the DB connection string(s) - ReferenceOne Cmdlet for each Service

• Set-BrokerDBConnection (Broker Service)

• Set-ConfigDBConnection (Configuration Service)

• Set-HypDBConnection (Host Service)

• Set-AcctDBConnection (AD Identity Service)

• Set-ProvDBConnection (Machine Creation Service)

• Set-PvsvmDBConnection (Machine Identity Service)

89


Setting the DB connection string(s)

• The database connection string is updated to the specified value regardless of

whether it is valid or not

• Specifying an invalid connection string will prevent a service from functioning

until the error is corrected

• Once a connection is configured, it is not possible to alter it without first

clearing it (set the connection to $null)

• When connection is set to $null it is reset and service is returned to an idle

state90


Viewing Host Details through PoSHHost Service PoSH provider

• Name: Citrix.Hypervisor

• Drive: <XDHyp>

○ pseudo-drive

• Two Directories

○ Connections

○ HostingUnits

91


PS XDHyp:\connections> dir

• Displays detailed information about each defined host connection

92


PS XDHyp:\hostingunits> dir

• Displays detailed information about each configured host unit

93


Checking IntelliCache status UselocalStorageCaching

• set-item -path xdhyp:\hostingunits\res* -UseLocalStorageCaching $true/$false

94

Troubleshooting Example


Error Removing Controller from SiteTroubleshooting Example 2

96


Check Controller status in Desktop STudio

97


Use PoSH to check Controller stateGet-BrokerController

98


Enable Desktop Studio Logging

99

• Edit MmcSnapin.dll.Config under C:\Program Files\Citrix\Desktop Studio○ Enable Log To File mechanism “LogFileName”


What we see…

100

Inner exception:

System.Net.Sockets.SocketException No such host is known

at System.Net.Dns.GetAddrInfo(String name)

at System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6)

at System.Net.Dns.GetHostEntry(String hostNameOrAddress)

at Citrix.Console.Models.ServiceStateModel.<GetAddressesWithExclusions>b__3(ServiceInstanceStatus address)

at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext()

at System.Linq.Enumerable.<SelectManyIterator>d__31`3.MoveNext()

at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()

at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)

at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)

at Citrix.Console.Models.ServiceStateModel.GetAddressesWithExclusions(IEnumerable`1 addressesToExclude)

at Citrix.Console.PowerShellSdk.HealthMonitor.Scripts.ReloadServiceInstancesScript.<GetWorkingServiceInstanceSet>d__25.MoveNext()

at Citrix.Console.PowerShellSdk.HealthMonitor.Scripts.ReloadServiceInstancesScript.<ExecuteNext>d__c.MoveNext()

at Citrix.Console.PowerShellInteraction.PowerShellScript`1.<ExecuteNext>d__9.MoveNext()

at Citrix.Console.InteractionCore.ScriptBase.<ExecuteImplementation>d__1.MoveNext()

at Citrix.Console.InteractionCore.Scheduler.ExecuteNext(ScriptExecution execution)


What we see…

101

Inner exception:


















What we see…

102

Inner exception:


















Researching the Exception…www.google.com

103

• Exception Type

○ SocketException

• Condition

○ An error was encountered when resolving the hostNameOrAddress parameter

• Sources

○ http://msdn.microsoft.com/en-us/library/ms143998%28v=vs.85%29.aspx

○ http://

msdn.microsoft.com/en-us/library/system.net.sockets.socketexception%28v=vs.85%29.aspx


Researching the Exception…www.google.com

104

• Exception Type

○ SocketException

• Condition

○ An error was encountered when resolving the hostNameOrAddress parameter

• Sources

○ http://msdn.microsoft.com/en-us/library/ms143998%28v=vs.85%29.aspx

○ http://

msdn.microsoft.com/en-us/library/system.net.sockets.socketexception%28v=vs.85%29.aspx


CDF Output

106


Checking Service StatesGet-<Service>ServiceStatus

107

E.G Running Get-BrokerServiceStatus against DDC56.glover.com confirms

what the output from the CDf trace is telling us i.e. all services on the controller

we want to remove from the site have been reset:

Note: -AdminAddress parameter is used to specify which controller the cmdlet is

run against (built-in PoSH remoting capability…


Validating Service StatusReference

• Use Get-BrokerServiceStatus (Broker Service)

• Use Get-ConfigServiceStatus (Configuration Service)

• Use Get-HypServiceStatus (Host Service)

• Use Get-AcctServiceStatus (AD Identity Service)

• Use Get-ProvServiceStatus (Machine Creation Service)

• Use Get-PvsvmServiceStatus (Machine Identity Service)

108


Checking Registered Services

109

• As each service on DDC256.glover.com has been reset to DBUnconfigured,

running Get-ConfigRegisteredServiceInstance against the same controller will

return the following expected error:


Checking Service Instances

110


Checking Registered Services con’t

111


Checking Registered Services con’tGet-ConfigRegisteredServiceInstance –ServiceType

112


Step 1Manually cleaning up the database & removing orphan DB entries

113

• Take note of ServiceAccountSid used by service…

• Note: This is the objectSid of the machine the service is running on


ADExplorer

114

• ADExplorer can be used to

retrieve the ServiceAccountSid

assigned to the orphan services

• objectSid = ServiceAccountSid


Step 2Manually cleaning up the database & removing orphan DB entries

115

• Create DB eviction script for each service to fully remove them from the site…


Step 3Run config service eviction script against site database

116


Step 4Run the rest of the service eviction scripts against the Site DB

117


Step 5 Refresh Desktop Studio

118

• Problematic controller no

longer appears in Desktop

Studio…


Step 6The clean up…

119

• AD based Registration

• Run Set-ADControllerDiscovery –sync

• Registry based registration

• Update ListOfDDCs registry value via AD Group Policy

• Computer Policy\Controllers

Resources


ToolsTracing & Logging

• CDFControl


• CDFCheck 1.0 - Cmd


• CDFMarker On Demand - For XenApp and XenDesktop


• LogEnabler


122


ToolsMonitoring & Collection

• Citrix Scout


• XenDesktop Site Checker


• HDX Monitor 2.0


123


ToolsDatabase and Services

• XDPing


• XDDBDiag for XenDesktop


124


Before you leave…

• Conference surveys are available online at www.citrixsummit.com starting Thursday, 18 October○ Provide your feedback and pick up a complimentary gift at the registration desk

• Download presentations starting Monday, 29 October, from your My Organiser tool located in your My Account

129