Upload
syaamlive
View
144
Download
4
Tags:
Embed Size (px)
DESCRIPTION
Troubleshooting XenApp 6
Citation preview
XenApp 6 Case Studies and TroubleshootingRick Berry, Escalation EngineerMark Callahan, Escalation EngineerMay 24th, 2011
• Case study for UPM issue on XenApp 6
• Case study on XenApp 6 filtered policy issue
• Questions and wrap-up
Agenda
Case study for UPM issue on XenApp 6
• Customer was experiencing hung sessions at logon
• Some users could log in, others could not
Problem Definition
• “Black Hole”
• User Profile Manager process still running
• Logged in users would eventually be affected
Citrix Confidential - Do Not Distribute
Symptoms
Citrix User Profile Manager
Functional Overview - Logon
Local Windows Devices
XenApp Servers
XenDesktop Streamed/Delivered Desktops
Profile management ServiceFile Servers
Profiles stored via File Share My Settings
Functional Details
GPO\User Configuration\Windows
Settings\Folder Redirection\My
Documents
My Documents
\profiles\UserName\
\\server\UserHome\
Folder Redirection via Network
XenApp Server[User Logon Event Location]
Active Directory
Collect
Configuratio
n
File Server
File Server
Intelligent Sync
Profile management Service
User Logon
\HKLM\Software\Policies\Citrix\UserProfileManager.
• Complete System Dump
• PerfMon
• User Profile Manager Logs
Citrix Confidential - Do Not Distribute
Troubleshooting Methodology
• Examine Kernel memory
• Examine Winlogon process
Citrix Confidential - Do Not Distribute
Troubleshooting MethodologyComplete System Memory Dump
• Performance Monitor – monitor User Profile Manager and Winlogon threads
Troubleshooting MethodologyPerformance Monitor
NORMAL
PROBLEM
Troubleshooting MethodologyUser Profile Manager Logs
[PID];WaitUntilChangeJournalIsProcessed: Waiting to finish change journal processing of partition: C
Ah Ha! A suspicious log entry!
NTFS JournalingEvent NTFS file system action
Initial write operation The NTFS file system writes a new USN record with the USN_REASON_DATA_OVERWRITE reason flag set. For more information on possible reason flags, see the USN_RECORD structure.
Setting of the file time stamp
The NTFS file system writes a new USN record with the flag setting USN_REASON_DATA_OVERWRITE | USN_REASON_BASIC_INFO_CHANGE.
Second write operation The NTFS file system does not write a new USN record. Because USN_REASON_DATA_OVERWRITE is already set for the existing record, no changes are made to the record.
File truncation The NTFS file system writes a new USN record with the flag setting USN_REASON_DATA_OVERWRITE | USN_REASON_BASIC_INFO_CHANGE | USN_REASON_DATA_TRUNCATION.
Close operation If the user making changes is the only user of the file, the NTFS file system writes a new USN record with the following flag setting: USN_REASON_DATA_OVERWRITE | USN_REASON_BASIC_INFO_CHANGE | USN_REASON_DATA_TRUNCATION | USN_REASON_CLOSE.
• NTFS change journal was showing an increased size of the identification field.
Troubleshooting Methodology
SCREENSHOT
• Based on the data learned from the NTFS change journal examination, a code change was made to handle changes to the size of the Update Sequence Number record and a hotfix was developed.
Resolution
Resources discussed
Citrix Profile Manager Edocs Site
Citrix Profile Manager Logon Diagram
Citrix Profile Manager Logoff Diagram
CTX119791- Profile Management FAQ
CTX12559- Citrix Profile Manager Upgrade FAQ
CTX124455- How to Capture CDF Startup Traces on UPM 3.0
Resources – Citrix Profile Manager
Log Parser for Citrix Profile Management
Memory Dump File Not Being Generated on Provisioned Target
Microsoft Windows Change Journals
Resources – Citrix Profile Manager
Case study on XenApp 6 filtered policy issue
• Customer had a new XenApp 6 farm in place• XenApp 6 Citrix policies (both computer and user settings) were being applied via Active Directory Group Policy
Objects (GPOs)• Some of the Citrix policy settings were filtered for Access Gateway connections and others were filtered by client IP• When end users connect to the XenApp 6 server from an Access Gateway site, the filtered policy settings were not
applying to the session
Problem definition
XenApp 6 policies overview
• Manage XenApp servers collectively by grouping servers into worker groups
• You can assign published applications and Citrix policies to worker groups
• Servers added to worker groups inherit settings
XenApp 6 Group-based administration
Worker Group 1
Worker Group 2
XenApp Farm
Published Application:Notepad.exe
Citrix Policy:Enable Client Drive Mapping
• Worker Group is a new filter for applying Citrix policies
• Automatic configuration of new XenApp servers by placing them in an existing worker group
Applying Citrix Policies to Worker groups
Citrix policy creation and administration
1. Create policies as Citrix IMA-based policies using Delivery Services Console (Used if AD does not exist or access is limited)
2. Create policies as Active Directory-based policies using Group Policy Management Console (GPMC)
Note: All Citrix policy settings are configurable using either administration method
• Citrix policies added via the DSC are stored in the datastore
• Two types of policies categorized by computer policies and user policies
• Can be “filtered” for granular control or “unfiltered” to apply to all servers or users
• Policy settings are stored in the servers registry
Citrix policies via the Delivery Services Console
• Filtered policy• Applies to specific group of users or servers• Uses a variety of filters (IP, AG, Groups, Client name)• Use case: Disable CDM for the Marketing domain group
• Unfiltered policy• Applies to all servers or users• Used when filters or granular control isn’t necessary• Use case: Specifying the license server that all farm servers will use
Filtered versus unfiltered policies
• Allows integration of Citrix policies into the Windows GPO engine
• Adds a Citrix node in the Group Policy Management Console and Group Policy Object Editor
• Installed with Delivery Services Console• Must be installed on the same machine where
Group Policy Objects are administered• Can be installed on a standalone machine used for
administrative purposes
Citrix policy extension
• Computer policies • Enables or disables server settings that were once under the farm and server
properties in previous versions• Registry location:
• 32-bit components: HKLM\Software\Policies\Citrix
• 64-bit components: HKLM\Software\Wow6432Node\Policies\Citrix
• User policies • Enables or disables specific features for user sessions• Registry location:
• 32-bit components: HKLM\Software\Policies\Citrix\<SessionID>
• 64-bit components: HKLM\Software\Wow6432Node\Policies\Citrix\<SessionID>
Citrix policy settings on the server
GPO processing and precedence
Local PoliciesLocal Policies
Citrix Group Policy ObjectsCitrix Group Policy Objects
Site Group Policy ObjectsSite Group Policy Objects
Domain Group Policy ObjectsDomain Group Policy Objects
OU Group Policy ObjectsOU Group Policy Objects
PR
OC
ES
SIN
G
PR
EC
ED
EN
CE
Citrix policies general roubleshooting checklist
Identify how the policies are being applied (e.g. Active Directory, DSC, both)?
Are the Citrix policy files present on the server?
What does the group policy results wizard show?
CDF Tracing results (see CTX113199 for modules).
Setup and review Citrix policy debugging logs.
Are the Citrix policy registry settings in place?
Troubleshooting Methodology
• Identify how the policies are being applied (e.g. Active Directory, DSC, both)? Are they pulling down properly?
Troubleshooting methodology for the case
• Identify how the policies are being applied (e.g. Active Directory, DSC, both)? Are they pulling down properly?
• What does output from Group Policy Results Wizard show? Keep in mind GPMC has to be run from XenApp 6 server.
• Identify how the policies are being applied (e.g. Active Directory, DSC, both)? Are they pulling down properly?
• What does out from Group Policy Results Wizard show? Keep in mind GPMC has to be run from XenApp 6 server.
• Enable Citrix policy debugging (see CTX128413)
Troubleshooting Methodology
Setting these values to 0xFFFFFFFF writes the debug information to a log file: %SYSTEMROOT%\Temp\CitrixCseEngine.log
Setting these values to 0x0000FFFF writes the debug information to a debugger such as DebugView
NOTE: The same values have to be written to HKLM\SOFTWARE\Wow6432Node\Citrix\GroupPolicy
For more details see CTX128413
• Reviewing %SYSTEMROOT%\Temp\CitrixCseEngine.log we need to verify the logged in user
User Name = REDGETLAB\rickbeuser1, SID = S-1-5-21-3992822370-2973014269-1922904879-1172, Session ID = 3
Computer Identity - Name = 60426497M1
• Next we search on the display name of our policy so we can get the GUID since the GUID is referenced more in the log
Name={52243C73-ED52-4539-B484-02098F5A88F4}, DisplayName=Test Policies, Link=LDAP://OU=RickBe,DC=REDGETLAB,DC=CTX
Troubleshooting Methodology – Debug logs
• We know that the Access Gateway filter on the policy was using a wildcard (apply to any Access Gateway site), so for the Access Gateway filter we can search on AGInUse
FullArmor.GroupPolicyFramework:And(Citrix.Policy.Templates:AGInUse.isValid, Citrix.Policy.Templates:AGFarm.isValid
Citrix.Policy.Templates:WildcardMatch("*"
Citrix.Policy.Templates:AGTags.value,"*",true
Troubleshooting Methodology – Debug logs
• Our session in question was session 3:
HKLM\SOFTWARE\Policies\Citrix\3\Events
"LastUpdate"="2011-03-27 04:12:12Z“
• Looking at the Evidence key:
HKLM\SOFTWARE\Policies\Citrix\3\Evidence
“AGFarm”=
"AGInUse"=dword:00000000
Troubleshooting Methodology – Registry review
These are issues!!
• Reviewing the debug logs and comparing this to the registry entries being made allowed us to narrow down the issue to how the policy filters were being evaluated
• Through our analysis it was determined that there was an issue with the filter expression logic when the Access Gateway filter was being used
Root cause isolation
• The investigation into this issue resulted in code change for the Delivery Services Console which was tested successfully by the customer
• This code change is currently being packaged into a hotfix for the Delivery Services Console
Resolution
Resources discussed
CTX125152 - Citrix Group Policy Engine Facts in XenApp 6
CTX127612 - How Policies are Applied when an ICA Session Connects to XenApp 6.0
CTX127611 - How Citrix IMA Policies for XenApp 6.0 Fit in to Microsofts GPO Processing and Precedence Model
CTX124241 - Technical Guide for Upgrading/Migrating to XenApp 6
Citrix Blog Site - XenApp 6 Policies Deep Dive
Resources – Citrix Policy Architecture
CTX128413 - XenApp 6 and XenDesktop 5 Group Policy Tracing
CTX111961 - CDFControl Tool
CTX113199 - IMA Modules to Select When Obtaining a CDF Trace for a Policy Problem
Resources – Citrix Policy troubleshooting
Questions?
Before you leave…
• Session surveys are available online at www.citrixsummit.com starting Thursday, May 26• Provide your feedback and pick up a complimentary gift at the registration desk
• Download presentations starting Friday, June 3, from your My Organizer Tool located in your My Synergy Microsite event account