Within Citrix Consulting, we are delivering different types of engagements (see the Citrix website for more details). One frequently used in France is “Issue Resolution” wherein a consultant will provide onsite expertise to assist in troubleshooting customer issues. This does not replace Technical Support services and very often the onsite troubleshooting we perform acts as a complementary task working towards resolution of the support case opened by the customer.
Recently, I was involved in such engagement and, along with the application editor Consulting services, we performed different troubleshooting steps, trying to narrow down the issue. Given the outcome that isn’t much documented on the web, I decided to share the solution here, with all of you.
The issue
Customer was deploying a ERP solution over HDX, hosted on Citrix Virtual Apps 7.15 (formely known as XenApp) on Windows Server 2012R2 with PVS and performed extremly slowly : some actions could take up to 45 minutes (the preview before exporting to PDF for example).
That application (well an older version to be 100% accurate) worked absolutely fine on XenApp 5.0 (hosted on Windows 2008) : the same operation took around 5 minutes on this environment.
The troubleshooting steps
The most important when troubleshooting is to narrow down the issue as much as possible. Therefore, the customer already done several important steps : building a dedicated Windows 2012R2 with VDA7.15 on top of it, without any antivirus nor monitoring solution and also no PVS involved.
Remove the Citrix hooks from the application processes
When looking to disable Citrix hookings for troubleshooting purpose, most people refers to https://support.citrix.com/article/CTX107825. However, since the VDA7.9 this is no longer valid. Support for SecureBoot was introduced in this version and required to change the hooking mechanism Citrix has been using for more than a decade (based on AppInit_Dlls, see https://docs.microsoft.com/en-us/windows/desktop/dlls/secure-boot-and-appinit-dlls). Yes, hooking was first introduced in MetaFrame XP !!
So, in modern days, disabling Citrix hooks requires to use the UviProcessExcludes registry key on the VDA, in HKLM\SYSTEM\CurrentControlSet\services\CtxUvi (see https://support.citrix.com/article/CTX223973). This key is present by default from VDA7.14 onwards.
Unfortunately, disabling Citrix hooks for the application process didn’t change anything : performance was still extremly poor.
Connecting using RDP instead of HDX
A well known troubleshooting step when dealing with application performance over HDX : does the issue occur when connecting over RDP ? The main obvious difference is the presence of a full desktop (not many are using RemoteApp as a troubleshooting step, a full desktop RDP session is quicker to try!) along with the use of a different remoting protocol.
Why? because if the performance is good within a full desktop, then it’s most likely a seamless issue. Then, a new troubleshooting process kicks in, with all the seamless exceptions that exist : https://support.citrix.com/article/CTX101644.
In that case, we didn’t notice any improvment : performance was still poor within a full desktop and over RDP.
Test without a VDA
Next step was to confirm if the issue was caused by some service, DLL, feature included in the VDA. Uninstalling the VDA was already tested by the customer, and didn’t show any improvment. However, we needed to dig further :
Testing the application on a freshly installed WS2012R2 without VDA and without the RDS role showed excellent performance over RDP ! Therefore the enxt logical step is to enable the RDS session role on WS2012R2, which is automatically done by the meta-installer of the VDA.
After doing so, rebooting and testing the application over RDP, in a full desktop : the application performance was severely degraded. We were on the right track !
Dinging into the RDS changes betwen Windows 2008 R1 and Windows 2012 R2
RDS evolved a lot between Windows 2008 (yes, I am not talking about Windows 2008R2) and Windows 2012R2. One set of feature tickled our curiosity : in Windows 2008R2, Microsoft introduced the Dynamic Fair Share Scheduling (at Citrix, we recommended to use the XenApp 6.x integrated CPU fairshare system, leading to disable the Microsoft one as explained on our Knowledge base).
The feature is detailled on a Microsoft blog article and in Windows 2012 R2, the RDS team introduced also network and disk fairshare optimization. Unfortunately, not a lot of Microsoft articles mentions this (feel free to share in the comments if you find the TechNet source or an official Microsoft RDS team blog post!) besides a Microsoft Dynamics GP blog where the application was suffering also from bad performance in Windows 2012R2.
So, DFSS in Windows 2012R2 RDS added support for network and disk sharing to distribute load equally across all the sessions, to avoid excessive resource usage by a single session. This feature does not prevent the system from being exhausted but simply means users will still get some CPU, network or disk capacity at the cost of a degraded experience.
So the new registry key controlling these features in Windows 2012R2 (and Windows 2016 by the way) is EnableFairShare located in
HKLM\System\CurrentControlSet\Services\TSFairShare\Disk
HKLM\System\CurrentControlSet\Services\TSFairShare\NetFS
In our case, disabling DFSS for the disk subsystem did the job : performance wasn’t degraded anymore and maybe a bit faster than on XenApp 5.0.
Wrap-up
Setting the registry key HKLM\System\CurrentControlSet\Services\TSFairShare\Disk\EnableFairShare to 0 on the VDA resolved our performance issue with the application.
Hopefully, this real world experience can benefit to many and can provide some tips when troubleshooting performance issues. Remember to do such troubleshooting in a test environment whenever possible, to avoid further disruption of the production !
Frédéric Serrière,
Architect & Service Delivery Manager
Citrix Consulting Western Europe
Citrix TechBytes – Created by Citrix Experts, made for Citrix Technologists! Learn from passionate Citrix Experts and gain technical insights into the latest Citrix Technologies.
Click here for more TechBytes and subscribe.
Want specific TechBytes? Let us know! tech-content-feedback@citrix.com
