How Memory Forensics Revealed Exploitation of Ivanti Connect Secure VPN Zero-Day Vulnerabilities

February 1, 2024

by Andrew Case, Michael Hale Ligh, Volexity Volcano Team

In a recent series of blog posts related to two zero-day vulnerabilities in Ivanti Connect Secure VPN, Volexity shared details of active in-the-wild exploitation; provided an update on how exploitation had gone worldwide; and reported observations of how malware and modifications to the built-in Integrity Checker Tool were used to evade detection. A critical piece of Volexity’s initial investigation involved collecting and analyzing a memory sample. As noted in the first blog post of the three-part series (emphasis added):

“…Volexity analyzed one of the collected memory samples and uncovered the exploit chain used by the attacker. Volexity discovered two different zero-day exploits which were being chained together to achieve unauthenticated remote code execution (RCE). Through forensic analysis of the memory sample, Volexity was able to recreate two proof-of-concept exploits that allowed full unauthenticated command execution on the ICS VPN appliance.”

Collect & Analyze Memory ASAP

Volexity regularly prioritizes memory forensics when responding to incidents. This strategy improves investigative capabilities in many ways across Windows, Linux, and macOS. This blog post highlights some specific ways memory forensics played a key role in determining how two zero-day vulnerabilities were being chained together to achieve unauthenticated remote code execution in Ivanti Connect Secure VPN devices.

Utilizing memory forensics in digital forensics investigations saves valuable time and provides unparalleled visibility into system state. Using the right memory forensics tools to collect and analyze memory can take a fraction of the time it takes to image a hard disk, with a small percentage of the required storage space. When dealing with potential zero-days that are being actively exploited, every minute counts. Volexity’s ability to get a head start on the investigation while other sources of evidence were being captured in the background made a significant difference.

Reconstructing the Attack

As outlined in the previous blog series, while Volexity leveraged network packet captures and disk images to reconstruct parts of the attack, it was ultimately a memory sample that allowed Volexity to confirm exploitation. Additionally, memory analysis enabled Volexity to identify the exact commands run by the attacker because of the zero-day exploitation, pinpoint the vulnerable API endpoints exposed by the device, and create a fully working proof-of-concept script to exploit the vulnerability. This workflow has allowed Volexity to recover many zero-days in the past, such as an actively exploited zero-day Volexity discovered in Confluence last year.

Volexity’s Incident Response team analyzed the collected memory sample with Volexity Volcano. This section describes Volexity’s memory analysis workflow for this case.

Commands in uwsgi Memory

Volcano users can create reusable, sharable indicators of compromise (IOCs) that run automatically after importing a memory sample. One simple, effective analysis technique Volexity regularly leverages is string-based IOCs, which helps identify commands indicative of malicious hands-on-keyboard activity. In this specific case, Volexity discovered interesting hits when reviewing results associated with the base64 -d command, as shown below.

The results of this IOC provided nearly instant gratification. Several hits appeared on the heap or other writable user-mode memory regions within uwsgi PID 2952 (this is a web server process). Being able to easily attribute these hits to uwsgi was critical, because it reduced the likelihood of these strings being related to an antivirus signature database or other noise.

Note: Volcano also finds strings in kernel memory and free/deallocated regions, which is especially useful for revealing historical artifacts and traces of attacker activity that they tried to erase.

Volexity noticed many of the hits were formatted as JSON, which indicated a possible POST payload. Decoding the base64 data revealed some obviously malicious commands, including the command below that runs /tmp/5 as a bash script.

$ echo L2Jpbi9iYXNoIC90bXAvNSAyPiYxfGJhc2U2NA==|base64 -d

/bin/bash /tmp/5 2>&1|base64

Vulnerable API Endpoint

The evidence above highly suggests the device was compromised. Generally, where there is smoke, there is fire, so Volexity pivoted to a hex viewer in order to explore memory regions adjacent to the hits, within the virtual address space of the web server process. Here, Volexity found a more complete HTTP request header, as shown below:

A few subsequent searches confirmed the full path to trigger the exploit was POST /api/v1/totp/user-backup-code/../../system/maintenance/archiving/cloud-server-test-connection.  Furthermore, this segment of memory contained the remote IP address 98.160.48.170, which Volexity’s Threat Intelligence team attributed to compromised Cyberoam devices. This further solidified the fact that Volexity’s Incident Response team was on the right track.

Note: As mentioned in the first blog post, Volexity observed attackers clearing log files from disk. Even if that didn’t happen, the logs would only include the URL containing the somewhat innocuous directory traversal (which is generally atypical, but not a smoking gun). The vital POST payload is a memory-only artifact.

Connect-back SSH Shells

Volexity continued inspecting memory and confirmed similar web requests from another compromised Cyberoam IP address, 173.220.106.166. A subsequent search for that address yielded hits that suggested code running within another uwsgi process (PID 31694) was initiating SSH connections back to the attacker’s IP address.

Based on the command-line values in Volcano’s process tree (partially shown below), it was apparent the first uwsgi process (PID 2952) was running the restservice.api, and this new one (PID 31694) was running cav.api. This indicated a widening attack surface. This meant either multiple, different vulnerabilities were involved, or the same vulnerability was affecting multiple API endpoints across various services.

Furthermore, both services were running as root (UID 0), which means the SSH connect-back yielded superuser access. At this point, the attacker completely owns the device and can begin reconnaissance against other assets.

Hunting Advanced Adversaries

The external Ivanti Connect Secure VPN Integrity Checking Tool helps detect files introduced or modified by attackers. However, this tool reboots the system, so in order to preserve volatile evidence, capture memory first.

Volexity also recommends integrity checking the active operating system components and applications, not just the hard drive. Although these initial incidents didn’t involve memory-only implants or rootkits, that should not be taken for granted or assumed true in the future. Also, keep in mind the nature of these exploits does not require the attacker to write webshells or other backdoors to disk to gain unauthorized remote access. In this scenario, attackers could use built-in SSH to create a reverse tunnel and leverage other tools installed on the operating system. It would not be possible to recover evidence of this activity without a memory sample.

When hunting advanced adversaries in memory, Volcano’s automated IOCs perform an audit of the system call table (shown below), interrupt table, TTY handlers, Netfilter stack, loadable kernel modules, input event monitors, injected shared libraries, and more.

Given Volexity has already observed attackers attempting to evade the Ivanti Connect Secure VPN Integrity Checking Tool, more advanced methods of stealth and persistence are not too far behind.

Conclusion

Volexity recently disclosed details related to exploitation of Ivanti Connect Secure VPN, revealing how the attacker chained two zero-day vulnerabilities to achieve remote code execution. When investigating the source of compromise, Volexity employed memory forensics, analyzing a memory sample collected from a suspected compromised VPN device, which allowed Volexity to zero in on the source of the compromise.

Part of the mitigation strategy for detecting whether or not an Ivanti Connect Secure VPN appliance has been compromised is to use the in-built Integrity Checking Tool. However, in the third blog post of the series, Volexity shared evidence that a threat actor had surreptitiously modified that tool in order to evade detection. The lesson for analysts is to independently verify the integrity and trustworthiness of high-value targets using memory forensics, rather than only relying on tools that run on a potentially compromised device.

If you are interested in learning more about Volexity's leading memory forensics solutions, Volexity Surge Collect Pro for memory acquisition and Volexity Volcano for memory analysis, please contact us.