A major challenge with Global Server Load Balancing (GSLB) implementations, may be affecting your users and disrupting your business.
In this post, I’m going to highlight this challenge. Then, I’m going to provide a solution.
But before we start, let me explain how Domain Name Service (DNS) based GSLB works, it might not be what you expect.
Most customers that I speak with, already understand that DNS-based-GSLB provided by NetScaler (or elsewhere – this isn’t NS specific), can balance incoming users across multiple geographically dispersed sites and do so using logic for site-to-site failover, logic to prevent congestion at a single site, or to route users to their closest datacenter for achieving optimal performance.
It is commonly understood that users are directed (if you can call it that) by DNS services running on the NetScaler and configured with the logic mentioned above.
While this is largely correct, it is – in my experience – not so commonly understood that the user’s computer never actually speaks with the NetScaler during this process and it happens well before a login page is even displayed. In-fact it’s the DNS resolver configured on the user’s computer that actually speaks with the NetScaler. For this reason, and because DNS resolvers don’t normally forward client IP addresses, the logic used by DNS-based-GSLB alone cannot make decisions based on who users are or where the client computers are located without making assumptions, it can only direct the DNS resolver from which it receives the request.
This said, with site-to-site failover or logic to prevent congestion at a single site the difference between what most people understand (Fig. 1.) and what is actually happening (Fig. 2.), is largely unimportant.
The Problem With Routing Users to Their Closest Datacenters
The problem with directing users to a specific location based solely on their DNS resolver is because this assumes the world still looks like it did in the 1990s. Previously, ISPs were regional, therefore using DNS resolvers in the UK was an excellent indicator that the client computer was also in the UK. In today’s world, this isn’t always true.
For example, while at my home in the UK (Fig. 3.), my personal computer will have a UK DNS resolver. However, when writing this article whilst staying at a hotel in Ireland on a BTOpenzone WiFi connection, my DNS resolver is not in Ireland as you might assume; instead both the DNS resolvers that I’ve been assigned via DHCP, are originating their requests from UK IP addresses. As all DNS-based-GSLB has to work with is the location of the resolvers, it will make the wrong assumption that I’m also in the UK, and I won’t be provided with a logon point in the correct country (Ireland).
Not a major issue, you may be thinking. This will only impact a handful of users. Let’s look at Google’s DNS resolvers, which apparently handle over 7% of the worldwide DNS requests.
If I manually configure my laptop only to query Google’s DNS resolver “8.8.8.8”, I see the query arrive at my NS from an IP address that appears to be in Belgium or California (Fig. 4.), the split seems to be 50/50 when I tested it from my home, the Hotel connection, and a number of other locations.
Google has servers everywhere, you likely won’t see the same split of servers as I do, but if you see something similar, this suggests that over 7% of your users (I say ‘over’ as Google DNS are not the only third party DNS resolver out there) are being routed to the wrong continent every other logon, not to the closest datacenter. That’s an average of 1 in 28 logons going to the wrong continent. Personally, I see that as a major issue.
This is an issue not only because the user’s XenApp or XenDesktop session may be delayed by 100-150ms needlessly crossing a transatlantic link, but also because user-specific data such as Home drives, profile data and Exchange Mailbox may not be replicated to remote sites and would need to traverse internal WAN links adding further delay.
Solution Overview
In solving this challenge, we need to first look at the bigger picture.
Users are being routed to their closest NetScaler Gateway logon point via the DNS-based-GSLB service as indicated by circle 1 in (Fig. 5.), through that logon point, they gain proxy access to backend StoreFront servers by circle 2 in (Fig. 5.), and finally establish an ICA connection as in circle 3.
The logon point to establish the ICA connection is actually controlled by an .ICA file which is downloaded from the StoreFront servers. Critically, these servers are aware of the real client IP address, which has been forwarded by the NetScaler within a header. This opens up the possibility of extending StoreFront to perform a second level of GSLB using an extension module and hence, resolves the issue.
Using the SDK that has been released for StoreFront 2.5 and above, a drop-in GSLB extension has been constructed as in (Fig. 6.). With the correct placement of just a few files in the correct directory across each StoreFront server, a user who is initially routed to an incorrect logon point, can now be rerouted via the downloaded .ICA file to the correct logon point before their actual remote access session begins.
Solution Operation
The StoreFront GSLB extension contains a configuration file and a gateways file. The configuration file allows debug logging to be turned on and off, and also the setting of a priority mode which controls how the gateways file should be interpreted.
If the priority mode is set to “location”, then the exact distance between the location of the user’s endpoint and the location of each matching gateway will be calculated, then the NetScaler Gateway logon point that would normally be sent within the .ICA file will be replaced with the gateway which is physically closest.
If the priority mode is set to “order”, then the NetScaler Gateway logon point will be replaced with the first match found within the gateways file.
In both modes, the locations of the IP addresses are determined using the free GeoLite2 database from MaxMind or, if you purchase it, their commercial database, which provides greater accuracy.
Location distances are determined via a Great Circle calculation, performed using each matching gateway and the endpoint IP address. The endpoint IP address itself is taken from an X-Forwared-For header that is always automatically inserted by the NetScaler Gateway.
Please note that as the users will be routed to a logon point, through which they have not been authenticated, it is imperative that global STA servers are used across all NetScaler Gateway logon points and StoreFront servers so that all NetScaler Gateways will accept the authentication tickets presented.
GSLB Installation
Here, I present all the installation instructions needed for implementing the GSLB extension solution. Take note that the MaxMind GeoLite2 database is not included; it is required and for licensing reasons must be downloaded separately and placed in the appropriate location as detailed in the instruction below. MaxMind offer the GeoLite2 database in multiple formats and this tool requires the uncompressed .MMDB file format.
The GSLB StoreFront extension is distributed as a zip file available here and has the following MD5 hash.:
6d664226cb486278372a6b1fb3992393
Before installation please take care to note this extension is not officially supported and is offered without warranty of any kind.
The zip contains the following.
First, the entire content of the zip file should be extracted:
Paste the GSLB extension files and the extension’s configuration directory into the StoreFront bin folder.
Next, extract the “GeoLite2-City.mmdb” from the zipped file downloaded from MaxMind and place it within the SF.Data configuration folder.
The configuration file for the extension “Config.JSON” can be edited in Notepad. Setting debug to true will append the entire ICA file which is sent to the user to the debug file within the configuration directory. Priority modes are described below.
The gateways file must list all locations that you expect the MaxMind database to return in a regular expression format, and the gateways to which those users should be directed. (.*) will match every remote location.
The gateways file above illustrates the benefit of “location”. No matter the location returned by the MaxMind database for the end user’s IP address the (.*) regular expression will match, the physical distance between that location and each of the gateways listed will then be calculated and the user directed to the closest.
Alternatively, with the priority mode set to “order” all users would be directed to “gbgateway” as this would be the first matching gateway. A better example for “order” would be a gateways file containing the following
EU@GB(.*), gbgateway.citrix.com
EU@(.*), degateway.citrix.com
In this example users with IP addresses determined to be within the UK by the MaxMind database would be directed to “gbgateway”, other EU users to “degateway”, and all other users would receive the gateway normally supplied by StoreFront without modification.
Final notes
As mentioned above this extension isn’t officially supported however it is extremely easy to remove should problems occur, to do so simply delete the extension files from the StoreFront bin directory into which they were installed. While there is absolutely no warranty, should you have problems I’d be very interested to know about them and can be contacted via Twitter.
Also, as this extension modifies data within StoreFront prior to the .ICA file being created it doesn’t violate CTX131547.
The .ICA file provided to internal users will not be modified by the StoreFront GSLB extension, as it doesn’t already contain a NetScaler Gateway. If no location matches within the gateway file no modification will be made.
