On December 17th 2019, CVE-2019-19781 was disclosed. The vulnerability allows for directory traversal and remote code execution on Citrix Application Delivery Controllers (ADC) and Gateways with firmware versions 10.5, 11.1, 12.0, 12.1, and 13.1. Some time around the evening of January 10th 2020, security researchers released a proof of concept for the exploit. The exploit allows attackers to exploit the directory traversal vulnerability and calls a perl script that appends XML files to the victim’s machine. This can result in remote code execution (RCE). There are approximately 25,000 vulnerable servers exposed to the internet. The Citrix ADC system is a popular product among businesses in a variety of sectors, including law enforcement, healthcare, military, and critical infrastructure.
I created a Citrix ADC VPX in AWS to manually test the exploit. After using netcat to create a listener, I was able to catch a reverse shell, with access as the user “nsroot”, and could view the contents of /etc/passwd. This directory contains the hashed passwords of the users on the system, among other data. An attacker could use this vulnerability as a beachhead into the victim’s network, and considering many of these appliances are VPNs and load balancers, it could be disastrous if the attacker began to manipulate the flow of network traffic on victim networks.
The reverse shell connecting to my machine, listing the directories on the appliance
The contents of /etc/password on the victim appliance
After verifying that the exploit worked, I created a honeypot in AWS with the intent of gathering data on the volume and frequency of attacks against Citrix appliances. The exploit uses a GET request to verify the directory traversal vulnerability. My honeypot saw 127,085 GET requests in a 48-hour period, with 249 of those requests attempting to exploit the vulnerability. Interestingly enough, the original exploit that was released only allowed for the testing of one server at a time. Shortly after the author announced on Twitter that they had updated the exploit to be capable of scanning CIDR blocks, there was a dramatic increase in connection attempts to my honeypot, as can be seen in the graphic below (the tweet was made at 0852 CST on January 11th, the first spike in activity was at 1005 CST).
The volume of connections made to the server, sorted by time (UTC)
After doing some auditing of the logs on the VPX appliance, I discovered that there were 87 IP addresses that were the most persistent in trying to gain access. Someone was also kind enough to schedule a cron job that downloaded a shell script and redirected output to /dev/null. Further investigation of this shell script using open-source file analysis tools revealed it to likely be a cryptominer.
The contents of crontab, created by user “nobody”
While current data shows that attackers are actively scanning for this vulnerable service, the volume and frequency aren’t outside of previously seen data for other vulnerable services.
I will continue to run a few different honeypots to gather more data and present those findings in another blog.
Citrix has provided a detailed mitigation process here: https://support.citrix.com/article/CTX267679, and are currently working on a patch for the vulnerable firmware versions.
The Department of Homeland Security CISA has released a tool on Github to check if your infrastructure is vulnerable. The tool requires Python 3.6 or higher and can be found here: https://github.com/cisagov/check-cve-2019-19781.
Indicators of Compromise
The nature of the exploit means that the remote code execution is performed by user “nobody”. Audit the /var/log directory on any of your vulnerable Citrix appliances to look for activity by this user. Commands in the bash.log file, such as curl, hostname, uname, whoami or attempts to access sensitive data can all be useful in determining if there has been an attack. The httpaccess.log and httperror.log files will contain useful information about the origin of attacks, since the RCE takes place against the web server. Backdoors may be placed in the /netscaler/portal/templates and /var/tmp/netscaler/portal/templates directories.