Cloud Software Group is aware of a recent series of password spraying attacks directed at various organizations around the globe. These attacks have been seen across various products and platforms, suggesting an industry-wide issue, as highlighted in the following resources:
Some of these attacks have targeted NetScaler appliances. Cloud Software Group has collaborated with affected customers to analyze the issues and recommend remediations. These attacks are consistent with password spraying attacks and are distinct from brute force attacks – instead of trying many passwords against a single account, attackers try a small set of common passwords against many accounts to avoid detection and account lockouts. When a NetScaler appliance is sized for handling a typical volume of authentication attempts, the high number of login attempts from large password spraying attacks can overwhelm the appliance, potentially leading to service and/or operational disruption in some cases. Cloud Software Group has developed a series of recommendations to help mitigate these attacks, as described below.
Issue summary
Cloud Software Group has recently observed an increase in password spraying attacks directed at NetScaler appliances. These attacks are characterized by a sudden and significant increase in authentication attempts and failures, which trigger alerts across monitoring systems, including Gateway Insights and Active Directory logs. The attack traffic originates from a broad range of dynamic IP addresses, making traditional mitigation strategies such as IP blocking and rate limiting less effective.
Customers using Gateway Service don’t need to take any remediating measures. Only NetScaler/NetScaler Gateway appliances deployed on premises or in cloud infrastructure require these mitigations.
Potential impact
While the use of multi-factor authentication (with nFactor) on NetScaler helps prevent unauthorized access, these attacks can cause significant operational impact through resource exhaustion:
- Excessive logging: High volumes of failed login attempts fill the NetScaler ns.log file consuming the /var directory space and potentially impacting GUI access.
- Management CPU overload: The surge in authentication requests consumes significant CPU resources, impacting device performance, and in some cases triggering High Availability (HA) failover due to missed heartbeats.
- Appliance instability: In some cases the AAA module can become overwhelmed, leading to appliance crash.
Attack characteristics
During our analysis, it was observed that these password spraying attacks primarily target user authentication against historical, pre-nFactor endpoints.
The following logs are then created in ns.log. If the following logs are seen in ns.log and the volume of such logs is beyond what’s observed on NetScaler under normal day to day operations, then it signifies that NetScaler/NetScaler Gateway is potentially under attack.
This is a sample of the logs from an internal lab environment:
Dec 5 13:33:09 < > 10.110.13.215 12/05/2024:08:03:09 GMT n-test 0-PPE-0 : default SSLVPN Me
local0.info
Dec 5 13:33:09 < > 10.110.13.215 12/05/2024:08:03:09 GMT n-test 0-PPE-0 : default AAA Messa
local0.info
Dec 5 13:33:09 < > 10.110.13.215 12/05/2024:08:03:09 GMT n-test 0-PPE-0 : default SSLVPN Me
local0.info
Dec 5 13:33:09 < > 10.110.13.215 12/05/2024:08:03:09 GMT n-test 0-PPE-0 : default AAA Messa
local0.info
Dec 5 13:33:09 <local0.notice> 10.110.13.215 12/05/2024:08:03:09 GMT n-test 0-PPE-0 : default AAA
Messag Dec 5 13:33:09 < > 10.110.13.215 12/05/2024:08:03:09 GMT n-test 0-PPE-0 : default AAATM Mesas
local0.info
Dec 5 13:33:09 < > 10.110.13.215 12/05/2024:08:03:09 GMT n-test 0-PPE-0 : default AAA Messa
local0.info
Dec 5 13:33:09 < > 10.110.13.215 12/05/2024:08:03:09 GMT n-test 0-PPE-0 : default SSLVPN Me
local0.info
Dec 5 13:33:09 <local0.warn> 10.110.13.215 12/05/2024:08:03:09 GMT n-test 0-PPE-0 : default AAA LOGIN_FA Dec 5 13:33:09 < > 10.110.13.215 12/05/2024:08:03:09 GMT n-test 0-PPE-0 : default AAA Messa
local0.infoRecommended mitigations
Cloud Software Group recommends the following mitigations:
→ Ensure that multi-factor authentication is enabled for Gateway and the MFA verification factor is configured before the LDAP factor, details for which can be found here.
→ Create a responder policy to allow requests only for desired FQDN, as attacks are frequently targeting IP addresses rather than Gateway FQDNs. Create the following responder policy:
add responder policy IP_Block "HTTP.REQ.HOSTNAME.EQ(\"\").NOT" DROP
bind vpn vserver Gateway_vServer -policy IP_Block -priority 100→ Create a responder policy to block the following end points if not utilizing historic pre-nFactor basic/classic authentication:
- /cgi/login
- /p/u/doAuthentication.do
- /p/u/getAuthenticationRequirements.do
Creating this responder policy blocks these authentication requests before they reach the AAA module, preventing them from being processed. These policies only work for NetScaler firmware versions greater than or equal to 13.0.
Here are the responder policies that need to be created:
add policy patset patset_block_urls
bind policy patset patset_block_urls "/cgi/login"
bind policy patset patset_block_urls "/p/u/doAuthentication.do"
bind policy patset patset_block_urls "/p/u/getAuthenticationRequirements.do"
add responder policy policy_block_urls
"HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS_ANY(\"patset_block_urls\")" DROP
bind vpn vserver Gateway_vServer_name -policy policy_block_urls -priority 100
-gotoPriorityExpression END -type AAA_REQUESTIf WAF is being used to protect Gateway vServers, please use the following commands:
set appfw profile ns-aaa-default-appfw-profile -denylist ON
bind appfw profile ns-aaa-default-appfw-profile -denylist
"HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS_ANY(\"patset_block_urls\")" -valueType Expression
-ruleAction log RESETPlease note that during our analysis of these issues, we have observed that requests targeting “/p/u/doAuthentication.do” & “/p/u/getAuthenticationRequirements.do” are getting blocked with a higher success rate if WAF for Gateway has been enabled. Hence, we recommend enabling WAF for Gateway.
→ Enable IP reputation to automatically block requests from known malicious IP addresses, reducing attack volume. On NetScaler CLI, use the following commands to enable IP reputation:
enable feature reputation
add responder policy policy_block_malicious_ip CLIENT.IP.SRC.IPREP_IS_MALICIOUS" DROP
bind vpn vserver Gateway_vServer_name -policy policy_block_malicious_ip -priority 50
-gotoPriorityExpression END -type AAA_REQUESTMore details on how to configure the feature can be found in NetScaler documentation here.
→ The default time interval configured for log rotation is set at 1 hour and can lead to the log file size growing rapidly and fill up the storage disk and contribute to crash. We recommend rotating the log files with a shorter time interval set at 30 minutes. You can view the configuration required to make this change here. Additionally, we recommend you to review your log rotation policy and ensure that logs are rotated at set time intervals to avoid filling up the disk.
→ Enable recaptcha on NetScaler, details for which can be found here.
Additionally, this blog from June 2024 shares some best practices for protecting NetScaler/NetScaler Gateway from a range of attacks. Please also refer to this article posted in Citrix community which is useful if post n-factor URL’s are being used for attacking NetScaler Gateway deployments.
This information is based on information currently available to Cloud Software Group and is provided on an “as-is” basis and does not imply any kind of guarantee or warranty, including the warranties of merchantability or fitness for a particular use. Your use of the information provided here is at your own risk. You should use your own discretion in determining what mitigation and/or other actions are appropriate for your unique circumstances. Cloud Software Group reserves the right to change or update this information at any time.