[a recording of our special webcast will be available soon.  [Download PPT Files]]

About two weeks ago, on December 17th, Citrix released a workaround for a critical vulnerability in its Application Delivery Controller (ADC) and Gateway products [1]. These are products that Citrix acquired from NetScaler in 2005, and the NetScaler name is still commonly used.

    Last week, on December 23rd, Positive Technologies released a blog post with additional information, emphasizing the impact of the vulnerability [2]. This blog post affirmed that the vulnerability is critical and needs to be addressed quickly. CVE-2019-19781 is used to track this vulnerability.

    Due to the urgency of this problem, and holidays affecting about 70% of the globe these two weeks, we will have a special webcast to discuss this vulnerability.

    Luckily, there is no public “Proof of Concept (PoC)” exploit available yet, and we have not detected any exploitation of the vulnerability yet. You may have a bit more time to apply the workaround published by Citrix. During a quick review of the Citrix ADC code this week, we found several weaknesses and were able to exploit them to at least upload files to the system. This did not require any special tools or advanced skills. A determined individual should be able to find a full exploit in about a week. The code, as well as the system configuration, showed several obvious weaknesses. This is unlikely the last time you will have to patch these devices. 

    According to Citrix’s advisory, exploitation of the vulnerability does not require authentication and allows arbitrary code execution. The affected products can be deployed in a large number of configurations, and not all configurations may be vulnerable. But neither Citrix nor Positive Technologies provide any guidance to identify vulnerable configurations. Most likely, configurations that expose the Citrix web interface to outside users are vulnerable. This would affect the use of Citrix Gateway as an SSLVPN. Still, it could very well be used in other scenarios, for example, if Citrix ADC is used to restrict access to internal APIs or web applications.

    The best “hint” as to the nature of the flaw is the workaround Citrix published [3]:

add responder policy ctx267027 "HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/vpns/\") && (!CLIENT.SSLVPN.IS_SSLVPN || HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/../\"))" respondwith403

The rule first checks if the URL contains the string “/vpns/." Next, it checks if the user is either not connected to the SSLVPN, or if the URLs include the string "/../”. The “decode_using_text_mode” overwrites the default URL encoding. This likely indicates that the ‘/’ and ‘.’ characters can not be URL encoded to make the exploit work. It is important to note that the “/vpns/” string alone is blocked in the URL. You should not include the “/../” string if you translate this signature to other security devices.

    Citrix notes that this policy may block some valid requests as well. The "/vpns/scripts/" directory, for example, is used to serve browser plugins. Access to this directory is blocked by the suggested policy. If you are using the Citrix ADC in front of other web applications, any URLs that contain "/vpns/” are part of the patch are blocked.

    The last part (“/../") is typical for a directory traversal vulnerability. Directory traversal vulnerabilities come in many shapes and severities. Attackers typically use them to gain access to restricted resources, and the impact depends a lot on what resources are accessible.

A simple example (and this is NOT necessarily how it works here): A web application restricting access to the “/admin” URL can be fooled into providing access to unauthenticated users as long as they use a URL like “/somethingelse/../admin." The URL no longer starts with "/admin," and a web application is vulnerable if it does not parse URLs correctly. Directory traversal issues can also happen if the application executed files on the system. For example, the developer creates a "tools" directory with various scripts the user is allowed to run. The application then uses code like:

execute(“/tools/$script”)

An attacker could now supply a script like “../usr/bin/bash” to execute additional commands. This command injection vulnerability does take advantage of directory traversal.

Typically, simple "blocklists" like the one Citrix implemented here are not ideal. An attacker may be able to find alternative paths to the vulnerable script, or the attacker uses a different encoding technique to bypass the rule. At this point, we do not know enough about this vulnerability to discern if the rule is sufficient or not. Citrix has not announced any plans for an actual patch. Based on our review of the code, a patch will likely reveal sufficient details about the vulnerability to make it trivial to find an exploit. The policy was likely designed to block the exploit while revealing as little as possible about the vulnerability.

After applying the recommended policy, any attacks should be logged in the Apache access and error log. For example:

/var/log/httpaccess.log
127.0.0.2 - - [30/Dec/2019:21:05:43 +0000] "GET [EXPLOITURL] HTTP/1.1" 403 639 "-" "[USERAGENT]" "Time: 439 microsecs"

/var/log/httperror.log
[Mon Dec 30 21:06:33.317132 2019] [core:error] [pid 2499] [client 127.0.0.2:24553] AH00037: Symbolic link not allowed or link target not accessible: {file attempted to access}, referer: {referrer header (if any)}

What should you do?

1. Apply Citrix’s workaround as soon as possible (today!)
2. Monitor your systems for any exploit attempts. A quick “grep” for requests that contain “vpns” and “..” should tell you if there are any.
3. Consider additional steps, for example, if you have additional security devices ahead of Citrix ADC.
4. Monitor any abnormal activities from the Citrix ADC and Gateway, particularly from those devices towards the internal network hosts.

Even if you do not use Citrix, take a moment to check up on your other perimeter devices to make sure they are up to date. Last year has seen several critical vulnerabilities in similar devices. For example, there are still plenty of unpatched Fortinet devices out there that suffer from a path traversal vulnerability. Exploit code is readily available and has been used in the wild. The Fortinet vulnerability isn't a "remote code execution" vulnerability, but can easily be used to retrieve privileged account credentials from the system.

[1] https://support.citrix.com/article/CTX267027
[2] https://www.ptsecurity.com/ww-en/about/news/citrix-vulnerability-allows-criminals-to-hack-networks-of-80000-companies/
[3] https://support.citrix.com/article/CTX267679
 

---
Johannes B. Ullrich, Ph.D. , Dean of Research, SANS Technology Institute
Twitter|