One of the key security controls in many customer DMZs is the logical or physical separation of management and production data traffic. This is especially important for our US Department of Defense customer set, who are required to abide by DISA STIG requirements (for an overview of this, see my DISA STIG Overview article).

This article will guide you through the process of insuring that NetScaler management and data traffic are completely separated using Policy-based Routing. To start, we need to understand how NetScaler IP addressing and routing works.

NSIP vs SNIP

NetScaler natively separates out management vs data communication through two different types of IP address. A NetScaler IP (NSIP) can be considered the management IP of the NetScaler and is used for management access (GUI, SSH, Telnet, etc.). It is also used to source various other management traffic including:

  • LDAP, Radius, TACACS, Kerberos authentication
  • Perl Monitors
  • SNMP
  • Syslog
  • NTP
  • GSLB synchronization
  • High Availability Traffic
  • Licensing
  • NetScaler MAS communication

A Subnet IP (SNIP) is mainly used to source data traffic from the NetScaler to other devices. When a NetScaler is initially setup, it will generally contain a single NSIP and a single SNIP. On creation of the NSIP, the NetScaler will create a default route (0.0.0.0/0) using the configured NSIP network gateway. When a SNIP is created, a direct route will be created from that SNIP to its Layer-2 network. This means that by default, to contact an IP that is not on a SNIP network, the NetScaler will always send data through the NSIP router. This is not desirable if you are attempting to separate all management and data packets!

NetScaler Routing

NetScaler utilizes a single routing table for all communication regardless of its type. By default, it will reference this table for all communication unless placed in Mac Based Forwarding (MBF) mode. In this mode the NetScaler will always send return traffic out of the same interface it was received on. I recommend not using MBF unless absolutely necessary. It is not designed for high throughput production use and generally acts as a band-aid for poor routing design.

Out of the box NetScaler will use the NSIP for its default route. This can lead to situations where data traffic is sent through the NSIP management network. To combat this, we will be changing the default route to utilize a SNIP instead. This then creates another problem for us…

If NetScaler needs to send a packet sourced from the NSIP, but does not have a route, it will reference its routing table to decide how to send it. For example, if the routing table deems that a SNIP should send the packet, NetScaler will do an internal handoff from the NSIP to the SNIP interface before sending the packet. Most importantly though, it may not change the actual source IP in the packet! Here is a capture of my SNIP interface sending a packet with the source IP of my NSIP:

This is hard to catch if you are just performing Layer-3 network traces and not paying attention to Layer-2 communication. To prevent this, we will be using Policy Based Routes.

 

Floating IPs

NetScaler also has the concept of “floating IPs”, meaning that an assigned NSIP\SNIP can potentially communicate out of any assigned NetScaler interface. This is easily changed by using VLANs to bind specific SNIP addresses to specific physical interfaces (see CTX136926). You can also bind specific NSIP traffic to a VLAN and interface by using a NSVLAN assignment (see here for instructions). I’ll show you how to complete this process after making the appropriate routing and Policy Based Route changes.

Now that we know the basics of NetScaler routing, let’s look at how to make the appropriate configurations.

Assumptions and Base Recommendations

A few things before I detail the configurations:

  • In this blog I am going to assume that you are utilizing separate interfaces for management and data rather than a single VLAN trunk. For example, NetScaler interface 0/1 is being used for management, and a separate interface such as 1/1, 10/1, LA/1, etc. is in use for data traffic.
  • I will be using a fresh NetScaler installation as the primary example; however, these configurations could be applied to existing NetScaler devices. Your environment may be more complex than my lab setup so use caution and good judgement when following this guide.
  • Never use the 0/1 or 0/2 interfaces for production data traffic! These were not designed for constant, high throughput traffic flow. If for some reason you need to combine management and data traffic over a single interface/channel, utilize any interfaces but 0/1 and 0/2.
  • Save your configuration before and after each step! This will allow you reboot the NetScaler and restore your configurations if you lose connectivity to the NSIP during this process.
  • Configuring your environment to completely separate management vs data traffic does not allow you to send NetScaler HA heartbeats across all your configured interfaces for redundancy (this is a leading practice). You should weigh the impact of this against full security compliance. In NetScaler 12.0 and above you can explicitly disable HA heartbeats on specific interfaces in System > Network > Interfaces and then selecting HA Heartbeat OFF for that interface.
  • By default, our NetScaler Perl monitors (such as built-in StoreFront monitor) are sourced from the NSIP IP. To source this from a SNIP while still maintaining granular StoreFront monitoring, check out my article Load Balancing Citrix StoreFront LTSR with NetScaler and DISA STIGs.

Lab Configuration

For reference, here is the networking configuration my lab environment is using:

Network Type Interface MAC Network IP VLAN
Management 0/1 00:50:56:99:7a:be 192.168.2.0/24 NSIP: 192.168.2.60 200
Data 1/1 00:50:56:99:21:82 192.168.8.0/24 SNIP: 192.168.8.70 800

Creating a Policy Based Route

The first we want to do is configure our Policy Based Route (PBR). This will ensure that when we change our default route, all NSIP data still has a route and will flow through the 0/1 interface. Warning: if you change your default route before doing this, you will lose connectivity to the management GUI. Save your configuration beforehand in case of a mistake.

Navigate to System > Network > PBRs and create a new Policy Based Route with the following configurations:

This configuration tells the NetScaler that for any packet it sends with a source IP it owns on the NSIP\management network, that is not destined for that local network, to send to the NSIP router instead of referencing its routing table (Note: This assumes that NetScalers in an HA pair have NSIPs on the same management subnet). With this configuration, even though our default route will be using a SNIP address, the NetScaler will send all its NSIP sourced traffic (the traffic types I detailed earlier in the article) through the management network interface. PBRs always take precedent over the NetScaler routing table.

Changing the default route

Now that we have a PBR configured, we can safely change our default route away from the NSIP to a SNIP (remember to save first). This will ensure that any data traffic that needs to use the default route will be sourced from the SNIP network. Navigate to System > Network > Routes and delete the default route (the first route in the list assigned to network 0.0.0.0 and netmask 0.0.0.0). Then add a new default route attached to one of your data networks gateway router IP:

After you’ve done this, your NetScaler routing table should look something like the following:

In my lab, the 192.168.8.0/24 network is now being used for any traffic that does not have an explicit route.

Through a combination of modifying the default route and adding a PBR, our management and data traffic should now be separated properly. However, to ensure that all traffic leaves the appropriate interfaces, I now recommend explicitly binding your IPs to interfaces using VLANs.

Binding IPs to Interfaces

Save your configuration and navigate to System > Network > VLANs. Add a new VLAN for your data network and bind the appropriate interface(s) and IP(s). If you have more than one data network, repeat for those SNIPs and save your configuration. If you need the NetScaler to add 802.1Q tags to packets, this is where it can be configured as well.

This only takes care of our SNIP networks. To properly bind the NetScaler HA heartbeats to an interface, navigate to System > Settings and select Change NSVLAN Settings.

Note: I only recommend configuring the NSVLAN if you must have complete data and management separation. This will lock NetScaler HA heartbeats solely to the management interface. Otherwise, Citrix Consulting always recommends have NetScaler heartbeats on each active interface for redundancy.

Configure the NSVLAN ID and assign the appropriate management interface(s). Turn off tagging if it is not required. Complete this on both nodes in an HA pair.

Now go to System > Network > Interfaces and disable the HA Heartbeat on any interface not bound to the NSVLAN above. Do this on both NetScaler devices in an HA pair.

All your SNIP\NSIP IPs should now be properly bound to their respective interfaces.

Testing

Before we declare success, let’s perform some testing to ensure our configurations are correct. To do this I recommend setting up an NTP Server under System > NTP Servers. This will attempt to send NTP data from the NSIP every few seconds for us to capture.

SSH to the NetScaler NSIP and run shell to bring up the NetScaler shell. Now run nstcpdump.sh -e port 123 to capture NTP traffic on Layer-2. We are looking to ensure that the MAC address of the NSIP interface is sending traffic to the router and not the SNIP interface.

In this capture, you can see my NSIP interface (00:50:56:99:7a:be) try to hand off to my SNIP interface (00:50:56:99:21:82) per the routing table. However, the PBR kicks in and we see that my NSIP interface sends the traffic directly to my router (80:2a:a8:8d:b6:6f).

On the contrary, if we did not setup a PBR, or it was done incorrectly, we would see this:

Here we see my NSIP interface pass the NTP packet to the SNIP interface. Then per the routing table, the SNIP interface (00:50:56:99:21:82) sends the packet to the router. This is undesirable as management traffic is traversing my data interface.

Summary

Here is a quick recap of the modifications we made to accomplish complete management and data traffic separation:

  • Save the NetScaler configuration first, then setup a Policy Based Route (PBR) with Source IPs = NSIP Network and Destination IPs != NSIP Network. Check that you still have connectivity to the NetScaler and then save the configuration again.
  • Delete the default route attached to the NSIP and add one attached to a SNIP data network. If you lose connectivity to the NetScaler after deleting the default route, then your PBR is incorrect. Reboot the NetScaler to return management connectivity and confirm the correct PBR settings.
  • Configure interface and IP bindings to specific VLANs for data traffic. Configure the NSVLAN interface and bindings for management traffic if necessary to meet security requirements.
  • Use nstcpdump.sh -e to check that traffic appropriately separated. Using nstcpdump.sh -e port 123 is an easy way to check that NTP traffic is leaving the appropriate management interface.

Here is a recap of the general recommendations made throughout the article:

  • Never utilize MAC Based Forwarding (MBF) for production NetScaler devices unless necessary.
  • Save your NetScaler configurations after each modification to ensure you will still have NSIP management access.
  • Use Layer-2 traces (nstcpdump.sh -e) instead of Layer-3 to determine what physical interface traffic is being sourced from. You can use show arp to verify which router/interface/VLAN the NetScaler is communicating with.
  • Always bind SNIPs to your data interfaces using VLANs. Only ID and bind your NSVLAN to specific interfaces if you must have completely management and data separation. Otherwise, leave this configuration blank to allow for HA heartbeats across all active interfaces.

While the lab setup in this example is relatively simple, the same principles can be applied to most network architectures. These configurations could be adapted to support multiple SNIP networks, or for more granular control of specific management traffic. Feel free to submit questions in the comments, but if your looking for more in-depth expertise to support your environment, I recommend contacting our Citrix Consulting team.

Nick Czabaranek
Lead Architect US Public Sector Consulting

Citrix TechBytes – Created by Citrix Experts, made for Citrix Technologists! Learn from passionate Citrix Experts and gain technical insights into the latest Citrix Technologies.

Click here for more TechBytes and subscribe.

Want specific TechBytes? Let us know! tech-content-feedback@citrix.com

How to Deploy XenDesktop LTSR with DISA STIGs

0