Upgrade your ADC from 10.5 to 11.x/12.x — Lessons from the field

Among the daily tasks of a network admin who manages an ADC environment is keeping equipment firmware updated. They have to make sure the environment is bug free and supported by the manufacturer.

We have articles that can show you how to upgrade Citrix ADC, but with this blog post, I wanted to share some of my experiences from the field upgrading Citrix ADC from version 10.5 to 11.x / 12.x.

Citrix ADC 10.5 has themes, features, and capabilities that differentiate it from newer versions, so upgrading requires some care when transitioning. Let’s take a look at some lessons we’ve learned from our Citrix Consulting engagements.

• Always take a snapshot/backup: Make a backup copy before you upgrade your ADC. And when it comes to VPX, always take a snapshot before starting the process. This can be helpful in case of a rollback or if you run into any problems during your upgrade.
• Have disk space available: Upgrading ADC firmware requires adequate disk space. If you find you’re running out of disk space while upgrading your ADC, you can delete logs from the default directories mentioned in this article or even delete old firmware as covered in here.
• Multiple firmware on local disk: The ADC firmware update consists of copying a new kernel version into the ADC flash and various boot and configuration files. But not everybody knows that the old kernel version is still in the ADC flash and can be used to boot the ADC flash at any time. This is useful if there are problems with the new firmware or when the upgrade process is successful but the ADC can’t initialize the right firmware; if this happens, you can edit the files “/ flash / boot / loader.conf ” and ” /flash/boot/loader.conf.local ” using the VI or follow the instructions in this article.

Update Path

If your ADC is not on version 10.5 build 57.x or later, you won’t be able to upgrade it directly to versions 11.x / 12.x via the GUI using the Upgrade Wizard. However, you can still upgrade it using the CLI. If you want to upgrade using the GUI, just upgrade to version 10.5 build 57.x and then to the desired version.

Remove Custom Themes

The theme structure of Citrix ADC 10.5 is different from newer versions. When you update to new versions, the gateway login page will be distorted. The username and password fields will be gone, as described in this article.

To remove the customization before upgrading to version 11.x / 12.x, follow these two steps:

1. Disable customization by using the unset vpn parameter -UITHEME command and saving the settings.
2. Edit the /nsconfig/rc.netscaler script and comment out any commands that are changing the login page. Additionally, to be sure, you can rename the file /var/ns_gui_custom/customtheme.tar.gz to another name, although the first step (unset vpn) is enough to disable the use of this file.

If you haven’t disabled the customization before upgrading to 11.x / 12.x, and the ADC Admin GUI no longer appears, you can enter the command line and perform the steps described in item 1 above, reboot Citrix ADC, and the GUI should appear again. I’ve done this to recover customers who upgraded without shutting down the old customization. If that doesn’t work, check out this article on theme troubleshooting.

Manual Changes

There are three configuration items that must be set manually when you upgrade from 10.x to 11.x / 12.x:

• Remove “-TLS11 disabled” and “-TLS12 disabled” from services because version 10.x doesn’t support TLS 1.1 and TLS 1.2 on some models. As a result, it automatically disabled these protocols. Therefore, the disabling is copied into version 11 .x / 12.x, though ADC supports TLS 1.1 and TLS 1.2 on all models.
• Correct the root DNS server address “h.root-servers.net” to 198.97.190.53. In the past, this server used the address 128.63.2.53; that changed in December 2015 to the new address. New versions of Citrix ADC have the correct address. When you upgrade Citrix ADC 10.5 or earlier versions, the upgrade doesn’t correct the DNS address and the configuration keeps the old address.
• In version 12.x, there’s a new parameter in SNMP alarm configuration called “-allPartitions ENABLED”. This parameter is not automatically added during the upgrade to version 12. For VPX instances running inside SDX, SDX adds SVM as an alarm destination. The result? After upgrading to 12.0, alarms for SVM are missing the parameter. To solve this, use “set snmp trap generic [SVM address] -allPartitions ENABLED” and “set snmp specific [SVM address] -allPartitions ENABLED”.

Cache in the User’s Browser

After the upgrade users will need to clean their browser cache for the page to load correctly. This behavior is expected because Citrix ADC modifies the page objects, and the cache on the user’s machine is “outdated.”

I developed a script with information that my colleague Marcelo Oguma gave me about the headers the Gateway has. The script forces the browser not to use the cache and request objects to ADC.

add rewrite action rw_request_cache delete_http_header If-Modified-Since
add rewrite action rw_request_cache2 delete_http_header If-None-Match
add rewrite action rw_response_cache delete_http_header ETag
add rewrite action rw_response_cache2 delete_http_header Expires
add rewrite action rw_response_cache3 delete_http_header Age
<strong>add rewrite action rw_response_cache4 insert_http_header Last-Modified "\"Tue, 12 Feb 2019 15:00:00 GMT\""</strong>
add rewrite policy rw_request_cache_pol true rw_request_cache
add rewrite policy rw_request_cache_pol2 true rw_request_cache2
add rewrite policy rw_response_cache_pol true rw_response_cache
add rewrite policy rw_response_cache_pol2 true rw_response_cache2
add rewrite policy rw_response_cache_pol3 true rw_response_cache3
add rewrite policy rw_response_cache_pol4 true rw_response_cache4
bind vpn vserver SW21CloudGW -policy rw_request_cache_pol -priority 100 -gotoPriorityExpression NEXT -type REQUEST
bind vpn vserver SW21CloudGW -policy rw_request_cache_pol2 -priority 110 -gotoPriorityExpression NEXT -type REQUEST
bind vpn vserver SW21CloudGW -policy rw_response_cache_pol -priority 100 -gotoPriorityExpression NEXT -type RESPONSE
bind vpn vserver SW21CloudGW -policy rw_response_cache_pol2 -priority 110 -gotoPriorityExpression NEXT -type RESPONSE
bind vpn vserver SW21CloudGW -policy rw_response_cache_pol3 -priority 120 -gotoPriorityExpression NEXT -type RESPONSE
bind vpn vserver SW21CloudGW -policy rw_response_cache_pol4 -priority 130 -gotoPriorityExpression NEXT -type RESPONSE
NEXT -type RESPONSE
bind vpn vserver SW21CloudGW -policy rw_response_cache_pol4 -priority 130 -gotoPriorityExpression NEXT -type RESPONSE

N.b.: The following line should be inserted with the latest possible date:

add rewrite action rw_response_cache4 insert_http_header Last-Modified "\"Tue, 12 Feb 2019 15:00:00 GMT\""
Note: Be sure not to leave policies in place too long because it will prevent the browser from caching the Citrix ADC page, which will cause an increase in bandwidth due to the page being requested for all access.

Summary

Upgrading Citrix ADC can be tricky, and I’ve covered a lot in this post to help with your transition. If you’re moving from Citrix ADC 10.5 to a newer version, here are the five things you should remember:

• Always take a snapshot (in the case of VPX). This has helped me when the ADC no longer booted during an upgrade. Check out this article for more information.
• Though you might skip over versions when you upgrade, older kernals will always be available in / Flash / Boot. You can boot into other versions at any time.
• When upgrading from version 10.5 to any version above 11.0, always remove any custom themes you have and add them after upgrading.
• Any theme changes will require cleaning the cache in the user’s browser.
• You cannot upgrade directly from the first build of firmware 10.5 to any version above 11.0 via the GUI. You have to use the CLI. If you do want to use the GUI, you have to updated to the latest build of 10.5 first, then to the desired firmware.

I would like to thank my colleagues Marcelo Oguma, Senior Enterprise Architect, and Edson da Luz, Principal Consultant, for their contributions to this blog post.