Kerberos and WorxWeb

Overview of Kerberos

Kerberos was originally started in the 1989 to provide authentication in an unsecure world.  Consider that Kerberos could be used with protocols that had no security for data in transit (think telnet where passwords are sent in clear text). In a Kerberos world, clients authenticate to the Key Distribution Center or KDC. In our example we will be leveraging Microsoft AD as our KDC. In an oversimplification, Kerberos is based on the premise that client and KDC:

1. Communicate a list of available encryption models and select highest in common.
2. The Client sends its password to the KDC in an encrypted format.
3. The KDC decrypts and authenticates the user
4. The KDC sends back a Ticket Granting Ticket or TGT.
5. The TGT is the used moving forward to gain access to specific Ticket Granting Services or TGS. The TGS gives the user credentials to access to a specific service.  A service may be a Telnet Server, Web Site, etc.
6. The TGT and TGS have short time spans. Typically 6-24 hours.

Even in today’s world of encrypted communication, Kerberos has some real value. It can simplify the number of passwords a user must remember. Transitions can be made from one account or authentication model to another. A more importantly, even if the SSL encryption is removed (think SSL offload), the users password is still protected.

There are two primary forms of authentication that are typically leveraged by WorxWeb.  These are Kerberos Impersonation and Constrained Delegation (KCD).  We will be covering Impersonation in this document. Impersonation means that the users credentials will be leveraged to get the TGT and subsequent TGS to gain access to service(s). Kerberos Constrained Delegation (KCD) is beyond the scope of this document.  KCD will be provided separately. When KCD is leveraged, a service account is used to get the TGT.  This TGT is then used to get a TGS for the associated user. KCD is typically used when the users credentials are not available. For example, when a user accesses the NetScaler Gateway with a certificate, the users password is not available.

Active Directory as the KDC

A key pillar of Microsoft Active Directory is the use of Kerberos. Basic Kerberos on Windows servers requires little to no configuration to enable to work with IIS.  It is on by default.  In a MS AD, your domain name will be your Realm.  Unlike other Linux/UNIX Kerberos implementations, Microsoft Kerberos realm names are NOT case sensitive.

All domain controllers will appear as a KDC. A view at the DNS MMC snap-in on your forest will reveal AD servers a KDC’s. By default, clients will query DNS to find a KDC for the requested Realm.  In the example below, the domain is bowlins.com. So, if a client wants to find the realm bowlins.com, it would query DNS for an SRV record for the FQDN _kerberos._tcp.bowlins.com. Note the port in the Data column below. The 88 is the port that Domain controllers will be listening for authentication requests.

Validating NetScaler and KDC connectivity

Before any real WorxWeb testing is done, various tests should be performed on the NetScaler shell to ensure basic connectivity and functionality. NetScaler does not offer the nslookup command that most windows users are familiar with using.  NetScaler leverages the linux dig command.  An example below (1) shows the format to perform a DNS query for our Kerberos service record in our test domain. The highlighted area (2) shows our AD server DNS entry and the port it will be listening for Kerberos request, port 88.

In some environments the NetScaler may not have Name Server records pointing to AD’s DNS services. If this is the case, you have two options:

1. Point the NetScaler to AD DNS with name server entries.
   
2. Manually create a DNS SRV record on the NetScaler.
   

The Kerberos port (88 by default) will often not be opened from the DMZ to internal network.   Validate via a telnet connection to verify that the port is open.

If the admin is experienced with Kerberos on Linux system, they will expect standard Kerberos command to be available, such as kinit and klist. These are standard commands used from the command line. Older 9.x and earlier NetScalers offered these commands directly from the command line as well. There is a fair amount of information about NetScaler via a Google search that still references this. As of NetScaler 10.1, these commands are arguments of the nskrb command. On a Linux system, we would enter the following command to get a TGT:

kinit mbbowlin@bowlins.com

We would use this same format on a NetScaler but would prefix the entry with nskrb like the example below:

nskrb kinit mbbowlin@bowlins.com

Below is an example of a successful run of this command. It will prompt for a password and if entered correctly and all prerequisites are in place, it will simply return to the command prompt.

If your DNS is not configured correctly, you would see an error like this

If port 88 is not opened on the firewall, you will receive an error like the following:

If you are successfully authenticated, you can query the NetScaler for a list of cached TGT’s that the NetScaler has acquired.

WorxWeb Client Considerations

WorxWeb single sign on (SSO) is achieved for the most part via the Secure Browse VPN mode. This allows the NetScaler to terminate the SSL stream and view the contents of the information inside. It is here that the beauty of the NetScaler SSO comes into play. When a user accesses a web page, the NetScaler is able to intercept the 401-authentication challenge and perform authentication on behalf of the user. In most cases, the NetScaler will have cached the users AD password during authentication to the NetScaler Gateway from WorxHome.

Thus, it is critical that WorxWeb MDX network access policies be set to Tunneled to the internal network and that the Preferred VPN mode is set to Secure Browse as shown below.

Validating NetScaler Session Policy for WorxWeb

Once your base configuration has been validated and connectivity confirmed.  Its time to ensure that based on your authentication methodology that your NetScaler Gateway session policy for MDX is configured correctly.

The key consideration is to configure the NetScaler Worx Session policy to allow it to properly identify the Kerberos Realm from your connection. Without proper identification, NetScaler will not recognize your traffic and know to perform Kerberos impersonation.

There are two ways that NetScaler will determine the Realm that the user is using.

1. The user authenticated to the Gateway utilizing their UPN. Thus the realm can be extracted from the domain portion of the UPN.
2. The user authenticated using SamAccountName, and thus needs more information to determine the realm. In this case, the Session->Published Application SSO domain will be used:

Additionally, the NetScaler needs to have a KCD account configured that has no password.  This account is a placeholder that informs the NetScaler that the specified Kerberos Realm should be leveraged by the NetScaler to perform Kerberos impersonation.

Your KCD account must now be bound to a traffic policy on your XM gateway.  Without this traffic policy configured, Kerberos will not be initiated on your WorxWeb traffic.

Create a new traffic policy with the expression set to ns_true for testing.  You can limit this down to only WorxWeb via more granular policy expressions.

Create your traffic profile similar to below. When the gateway receives traffic for the HTTP/HTTPS protocol it will attempt Kerberos authentication assuming that the domain/realm match that defined in the KCD account.

Debug Logs

The Kerberos engine provides extensive logging on the NetScaler. The log can be found in the /tmp folder. This is the same location that is used to view AAA (i.e. LDAP) authentication information. The Kerberos log is nskrb.debug. The administrator will need to cat the log file as the authentication is attempted.

cat /tmp/nskrb.debug

A successful Kerberos impersonation via WorxWeb will output a lot of information:

Note:  As of NetScaler release 11.1-51, Kerberos Ticket caching has been moved to in-memory for better performance.  Hence, after the first time, everything gets served from within packet engine. Hence, you will not see nskrb.debug output.  You will want to “tail -f /var/log/ns.log” for information after the first successful authentication.

https://support.citrix.com/article/CTX223494

A number of errors can occur when something goes wrong. A Kerberos error code reference can be found at the end of this document. They will generally be a negative number starting like this:   -1765328…

IIS Configuration

Ensure that the IIS site or virtual directory has anonymous authentication disabled and Windows integrated enabled.

You then need to verify that your IIS site/VD is configured to negotiate Kerberos/NTLM.  Right click on Windows Authentication and choose providers. Ensure that Negotiate is at the top. This tells IIS to attempt Kerberos first and then fall back to NTLM if it fails.

You can force only Kerberos with the following setting. But this typically is NOT something that would be done in a typical production environment. This can be helpful when NetScaler appears to be performing NTLM as it has already cached the users password and NTLM is the fallback for Negotiate.

By default IIS7 and higher assumes that when you enable “Integrated Windows Authentication”, that you will be using Network Service account and will enable kernel mode authentication. We will want our application pool for IIS using a domain account. We will need to click the “Advanced Settings” link and un-check (disable) kernel mode authentication on the site. This will cause the Kerberos tokens to be processed by the application pool account instead of the machine account.

Navigate to you IIS web site authentication settings. Select windows authentication -> Advanced Settings and uncheck “Enable Kernel-mode authentication”

Testing

You should now be able to open WorxWeb and connect to the your web site using Kerberos.  To ensure that NTLM is not getting in the way, you should terminate your NetScaler Gateway session and kill WorxWeb on the device.

Open your ssh session to the NetScaler and cat the nskrb.debug log in the /tmp directory.

You can verify that you are using Kerberos by checking the Windows security log and look for event 4624 with the “Authentication Package” field set to Kerberos.

WebSphere

WebSphere deployments typically leverage Simple and Protected GSSAPI Negotiation Mechanism (SPNEGO). This is often pronounced “spen-go”. NetScaler 10.5-55.8 and below do not support this negotiation mechanism. Thus, SSO to WebSphere servers will typically fail on these versions of the NetScaler. In this case, ensure that the customers NetScaler is at level 10.5-59.x+

Error Code Reference

https://andromeda.rutgers.edu/~sysmail/krb5_error.html