Over the last year, I’ve worked with customers to deploy and tune 3D graphics Citrix workloads leveraging graphics processing units (GPUs). Historically, GPUs have been reserved for graphics-intensive workloads, such as engineering modeling or modern workflows like machine learning.
But with the advent of more resource-intensive operating systems and common applications, graphics cards are not just for graphics-heavy users anymore. Options are available to either leverage a full or partial GPU depending on the requirements (keep reading for more on that).
The diverse set of potential graphics card use cases presents an excellent opportunity to emphasize new enhancements to the Citrix HDX 3D Pro feature. These include significant improvements in delivering high frames per second (FPS) sessions and reduced graphics CPU consumption, both of which were released in Citrix Virtual Apps and Desktops 2112 and are also available in Citrix DaaS (see our blog post on hitting new highs with HDX 3D Pro for more).
While there is plenty of documentation on GPUs and the HDX 3D Pro feature out there, in this blog post I wanted to share five lessons about Citrix HDX 3D Pro and GPUs.
1) Technology Options
When planning how a GPU will be used within Citrix (i.e., virtualized) workloads, there are multiple options available. Selecting the appropriate option will depend on the capabilities of the GPU processor model, your budget, and your workload requirements. This blog post will focus on the options and configurations for two of the largest graphics card vendors — NVIDIA and AMD.
GPU Passthrough: An entire physical GPU is directly assigned to one VM. This technology is consistent across the major vendors. While this may provide the highest level of performance, it is less flexible and quickly becomes costly because the GPU card is locked to one VM.
Virtual GPU: Here, multiple virtual machines have direct, simultaneous access to a single physical GPU, providing a more cost-efficient and scalable solution. This has become a very common deployment model for virtualized workloads. There are a few key differences between NVIDIA and ADM. See our product documentation on preparing hosts for graphics to review configuration options for Citrix Hypervisor with virtual GPU technologies.
- NVIDIA — NVIDIA leverages a software-based split of the GPU card via a management tool, NVIDIA Virtual GPU Manager, with the technology called vGPU. Each VM will have a vGPU profile attached to it within the hypervisor, which defines the series type; the amount of GPU memory per vGPU; and the intended workload class. Here’s an example:
- M60-1B profile: Tesla M60 graphics card, 1 GB memory per vGPU, and virtual desktop power workload (B).
- Other workload classes include virtual applications (A), compute intensive workloads such as AI (C), and designers requiring the Quadro technology (Q). The different workload classes require a specific edition license.
- NVIDIA has developed a virtual GPU software user guide. See the section on virtual GPU types for more on profiles.
- AMD — AMD offers a hardware-based GPU virtualization solution, called MxGPU, that is built into the chip with a technology called single root input/output virtualization (SR-IOV). This design foregoes an extra software layer to reduce complexity and overhead. You can load an optional driver, called AMD Radeon Pro Settings, onto hypervisors (only vSphere at time of writing) to enable/disable virtual functions and manage the MxGPU resources. Download the MxGPU whitepaper for additional details.
Virtual Shared Graphics: In this deployment model, the hypervisor conducts the splitting and allocation of the GPU capacity into virtual instances. An example of this is VMware Virtual Shared Graphics Acceleration (vSGA), which requires a vendor driver to be installed in the hypervisor. From here, each VM uses a proprietary VMware vSGA 3D driver that communicates with the vendor driver. This is an uncommon method for virtualized Citrix workloads and is generally not recommended.
2) GPU/vGPU Prerequisites
Before you put your shiny new GPU to the test within a Citrix session, there are several prerequisites that must be in place to successfully license and consume the graphics card.
Host Capability: The servers that will have the GPU cards must support the desired GPUs.
License Server: NVIDIA requires a software license in addition to their GPU cards to leverage the vGPU capability. To distribute these licenses, a NVIDIA vGPU License Server is required. This license server may be deployed as a Windows or Linux machine, and two are recommended for high availability. From here, workloads are directed to the license server via registry (Windows) or configuration file (Linux). AMD does not require separate licensing for their MxGPU technology, so no license server is needed.
See the NVIDIA License Server User Guide for details on the license server installation, configuration, and requirements. Learn about workload configuration in the NVIDIA Client Licensing User Guide.
Hypervisor: Separate hypervisor setup items are required for different GPU vendors.
- NVIDIA — Within the hypervisor, NVIDIA requires you to download and install the NVIDIA Virtual GPU Manager. See the documentation on installing and configuring NVIDIA Virtual GPU Manager for instructions for each of the main hypervisors.
- AMD — Does not require a client to be installed on the hypervisor. Optionally, consider installing the Radeon Pro Settings plug-in if your hypervisor supports it. Learn more in the AMD MxGPU and VMware Deployment Guide.
- System BIOS Settings — For both vendors noted, physical host BIOS settings may need to be adjusted. This may include enabling SR-IOV for certain GPUs or setting the correct display adapter.
Assignment: Assigning a GPU/vGPU to a VM is done via the machine settings within the hypervisor management console (e.g., XenCenter, vCenter). For a VMware deployment, the GPU is added as a “Shared PCI Device” under “Virtual Hardware.”
Driver in VDA: Each GPU vendor requires a specific driver to be installed in the operating system of the workload machine to broker communication between the hardware and software. This is the case for both GPU passthrough and virtual GPU. Download the drivers directly from the vendor. Additionally, AMD requires their MxGPU technology to be configured within the workload machine using a setup script. Learn more in the AMD MxGPU and VMware Deployment Guide.
Failure to install the correct driver is a common reason for GPUs/virtual GPUs not being successfully consumed. For example, the general NVIDIA driver downloads page is for physical machines, not virtual machines. Instead, download from the NVIDIA Application Hub (requires registration for a free account).
3) Public Cloud GPU Optimized VMs
Each public cloud provider has graphics optimized VM instances with different options available. GPU cloud instances still require the appropriate GPU driver to be installed. This may be completed either through a more cloud-native solution (e.g., GPU Driver Extension for Azure, use a driver available on Amazon S3) or from the public driver downloads. As is always the case with public cloud, capabilities and options change rapidly, and you should review them before design or deployment work.
- Microsoft Azure: Azure provides options to either leverage dedicated GPU card(s) or a fraction of a GPU, aligned with the virtual GPU capability. Azure has NVIDIA and AMD GPU instances providing both delivery options. See the documentation on GPU optimized virtual machine sizes for details.
- AWS and Google Cloud: Dedicated NVIDIA GPU card(s) are available in AWS and Google Cloud. See the AWS documentation on accelerated computing and the Google Cloud documentation on GPU platforms to learn more.
4) Citrix Policy Settings
You can configure various Citrix policy settings to use HDX 3D Pro and optimize your experience. The “best” policy setting will vary based on use case and application set, so you should isolate and test different setting options.
Setting: Optimize for 3D graphics workload
Recommended Value: Enabled
Notes: Required to enable HDX 3D Pro. There is no longer a feature that you must select during VDA installation to enable HDX 3D Pro.
Setting: Use video codec for compression
Recommended Value: “For actively changing regions” or “For the entire screen”
Notes: First test with “For actively changing regions.” This will optimize the overall bandwidth consumption. If graphics performance is not sufficient, test “For the entire screen.” See our product documentation on using video codec for compression.
Setting: Use hardware encoding for video
Recommended Value: Enabled
Notes: This setting is enabled by default and will only be used if specific GPU requirements are met. The benefit? Video codec can be offloaded to the GPU hardware.
Setting: Visual Quality
Recommended Value: Test for specific use cases
Notes: In cases where preserving image data is vital, leverage the setting “Always lossless.” Otherwise, use “Build to lossless.” For lighter graphics workloads, the “High” or “Medium” (default) values can also be tested to reduce session bandwidth.
Setting: Target frame rate
Recommended Value: 30 – 60 FPS
Notes: Test with the default value of 30. As noted earlier, this may now be increased up to 60 FPS.
Setting: Allow visually lossless compression
Recommended Value: Test for specific use cases
Notes: Test this setting with use cases that require pixel-perfect graphics.
Setting: Overall session bandwidth limit policy
Recommend Value: Only set limit if required
Notes: Consider leveraging this setting to limit bandwidth consumption if multiple users share a network connection with limited bandwidth.
5) GPU Usage
How do you know if your apps are successfully using the GPU? There are various ways to verify, but here are two I have used. Please note, beyond more typical graphics applications like AutoCAD and ArcGIS, many common apps can leverage GPUs with a built-in configuration setting (typically called “Hardware Acceleration” or “GPU Acceleration”). Examples of these include Chrome, Edge, Office, and Teams, which can all be observed as using GPUs when configured properly.
Task Manager: This method involves using the built-in Windows tool on the Citrix workload machine. First, add the GPU and GPU Engine columns in the “Processes” tab (right click a column header and select these options, shown below). You can monitor how individual apps are using a percentage of the GPU. You can also monitor overall GPU resource usage and various real-time graphs from the “Performance” tab.
Taskbar Notification Area: This option is specific to NVIDIA graphics cards. When you install a NVIDIA graphics driver package, an app called the “NVIDIA Control Panel” is also installed. This tool provides control of NVIDIA graphics driver settings for the local machine. One key feature will show which programs are actively using the GPU/vGPU. From the NVIDIA Control Panel app, click View → Display GPU Activity Icon in Notification Area. You now have a new NVIDIA icon in the taskbar notification area. Use this new icon to see which programs are actively using the configured graphics card.
Learn More
Wading into the waters of a graphics-rich world? This post should give you a good jumping-off point. Citrix HDX 3D Pro is here to help deliver an optimal end user experience along this journey. Learn more about Citrix HDX 3D Pro’s industry-leading capabilities in our product documentation and in this Citrix Tech Zone article on HDX graphics.
