3CX Supply Chain Compromise Leads to ICONIC Incident

March 30, 2023

by Ankur Saini, Callum Roxan, Charlie Gardner, Paul Rascagneres, Steven Adair, Thomas Lancaster

[Update: Following additional analysis of shellcode used in ICONIC, in conjunction with other observations from the wider security community, Volexity now attributes the activity described in this post to the Lazarus threat actor. Specifically, in addition to other claims of similarity, the shellcode sequence {E8 00 00 00 00 59 49 89 C8 48 81 C1 58 06 00 00} appears to have been only used in the ICONIC loader and the APPLEJEUS malware, which is known to be linked to Lazarus. The original post has been left as written.]

On Wednesday, March 29, 2023, Volexity became aware of a supply chain compromise by a suspected North Korean threat actor, which Volexity tracks as UTA0040*. Endpoints with the 3CX Desktop application installed received a malicious update of this software that was signed by 3CX and downloaded from their servers. This was part of the default automatic update process and would result in information-stealing malware being installed on the victim’s host. It is possible that additional malicious activity may have taken place if the threat actor deemed the endpoint to be of sufficient interest.

3CX is a phone system company and claims to have more than 600,000 customers and 12 million users, including world-renowned brands. They have posted an update on their website acknowledging the compromise, though it should be noted the information in this post should not be deemed conclusive or entirely accurate based on Volexity’s analysis.

In a public post on Reddit, CrowdStrike identified signed 3CX installation files as being malicious and reported that customers were seeing malicious activity emanating from the “3CXDesktopApp”. Volexity further identified public forum postings on 3CX’s own website that stated various endpoint detection and response (EDR) and antivirus (AV) vendors began flagging malicious activity from updates as early as March 22, 2023. Volexity’s analysis suggests the malicious activity likely began much earlier.

Volexity was able to obtain multiple malicious installers for Windows and macOS directly from 3CX download servers. Analysis of installers from both platforms allowed Volexity to identify several new indicators of compromise and gain further insight into how the malware functions.

This post details what Volexity discovered from its analysis of the malicious installers and the additional files it downloads. Highlights of the findings include the following:

  • Both the macOS and Windows installers for 3CX are affected.
  • Based on data recovered from GitHub, infrastructure used by the Windows variant was activated on December 7, 2022.
  • Domains and web infrastructure used in the attacks were registered as early as November 2022.
  • A reconnaissance payload was deployed far and wide to Windows users.
  • The same functionality to download a payload was identified in the macOS sample, although Volexity could not confirm the final payload as the C2 was unresponsive at the time of analysis.

Any endpoint impacted by this malicious update should be isolated and investigated for further signs of compromise. Organizations should assess the potentially impacted information on these endpoints and look to cycle secrets to reduce the risk of additional future compromise.

ICONIC Analysis

Stage #1: Supply Chain Attacks - ICONIC

Volexity’s analysis began with one of the installers tagged as malicious in public discourse:

Name(s) 3CXDesktopApp-18.12.416.msi
Size 97.8MB (102555648 Bytes)
File Type Windows Installer
MD5 0eeb1c0133eb4d571178b2d9d14ce3e9
SHA1 bfecb8ce89a312d2ef4afc64a63847ae11c6f69e
SHA256 59e1edf4d82fae4978e97512b0331b7eb21dd4b838b850ba46794d9c7a2c0983

The installer contains a malicious version of ffmpeg.dll, an open-source video player library:

Name(s) ffmpeg.dll
Size 2.7MB (2814976 Bytes)
File Type Win32 DLL
MD5 74bc2d0b6680faa1a5a76b27e5479cbc
SHA1 bf939c9c261d27ee7bb92325cc588624fca75429
SHA256 7986bbaee8940da11ce089383521ab420c443ab7b15ed42aed91fd31ce833896

The library is loaded by 3CXDesktopApp.exe, and it is used to decode and inject a payload into memory:

Name(s) N/A
Size 288.0KB (294912 Bytes)
File Type application/x-dosexec
MD5 11bc82a9bd8297bd0823bce5d6202082
SHA1 894e7d4ffd764bb458809c7f0643694b036ead30
SHA256 f79c3b0adb6ec7bcc8bc9ae955a1571aaed6755a28c8b17b1d7595ee86840952

The purpose of this malware, which Volexity will refer to as “ICONIC”, is as follows:

  • Download various files that contain additional code, with names such as icon[0-15].ico, hosted at https://github[.]com/IconStorages/images/. (Note: the GitHub repository has since been taken down.)
  • Parse these files to identify a “$” character followed by a base64-encoded string appended to the end of the ICO files.
  • Decrypt the base64 string using the AES-GCM encryption algorithm. All values required to decrypt AES-GCM are derived from a complex function that third-party researchers have indicated is based on a publicly available gist.
  • Once the string is decoded, it contains the URLs with which the DLL will then communicate to receive a next-stage payload.
  • The next-stage payload is a JSON object that is then parsed and must further be decrypted (with the same AES-GCM decryption function). The next stage is expected to be a 64-bit PE that is reflectively loaded through a shellcode loader stored at the head of the file.

A script is provided with this post on GitHub that can be used to decrypt the base64 blobs appended to the ICO files. Volexity was able to clone the GitHub project, and through the commit history, was also able to retrieve files that had previously been deleted. The table below provides details of each file and the decoded URL from each one. Note that there are duplicate filenames due to deletions, and in some cases the files were identical.

Active Files

Filename Hash (SHA1) Decoded URL
icon0.ico 9c943baad621654cc0a0495262b6175276a0a9fb https://www.3cx[.]com/blog/event-trainings/
icon1.ico 96910a3dbc194a7bf9a452afe8a35eceb904b6e4 https://msstorageazure[.]com/window
icon2.ico ffccc3a29d1582989430e9b6c6d2bff1e3a3bb14 https://officestoragebox[.]com/api/session
icon3.ico 89827af650640c7042077be64dc643230d1f7482 https://visualstudiofactory[.]com/workload
icon4.ico b5de30a83084d6f27d902b96dd12e15c77d1f90b https://azuredeploystore[.]com/cloud/services
icon5.ico 3992dbe9e0b23e0d4ca487faffeb004bcfe9ecc8 https://msstorageboxes[.]com/office
icon6.ico caa77bcd0a1a6629ba1f3ce8d1fc5451d83d0352 https://officeaddons[.]com/technologies
icon7.ico 57a9f3d5d1592a0769886493f566930d8f32a0fc https://sourceslabs[.]com/downloads
icon8.ico f533bea1c0558f73f6a3930343c16945fb75b20f https://zacharryblogs[.]com/feed
icon9.ico 31d775ab577f3cc88991d90e9ae58501dbe1f0da https://pbxcloudeservices[.]com/phonesystem
icon10.ico 0d890267ec8d6d2aaf43eaca727c1fbba6acd16e https://akamaitechcloudservices[.]com/v2/storage
icon11.ico 0d890267ec8d6d2aaf43eaca727c1fbba6acd16e https://akamaitechcloudservices[.]com/v2/storage
icon12.ico b1dee3ebcffad01a51ff31ff495fef1d40fdfaa0 https://azureonlinestorage[.]com/azure/storage
icon13.ico 64ab912d0af35c01355430d85dd4181f25e88838 https://msedgepackageinfo[.]com/microsoft-edge
icon14.ico 8377fb40c76aa3ba3efae3d284fa51aa7748e010 https://glcloudservice[.]com/v1/console
icon15.ico 11ae67704ea0b930b2cc966e6d07f8b898f1a7d2 https://pbxsources[.]com/exchange

Deleted Files

Filename Hash (SHA1) Decoded URL
icon1.ico ad37112b302c5193e60f6f6f49f4df668f5d3eb9 https://msedgeupdate[.]net/Windows
icon2.ico ad37112b302c5193e60f6f6f49f4df668f5d3eb9 https://msedgeupdate[.]net/Windows
icon10.ico 3a2138cd38ff2cef246f122a97d3c8f85ab6fc94 https://pbxphonenetwork[.]com/voip
icon0.ico 3df119f322c5182bdbea4ab364eec8a0e23d888b https://msstorageazure[.]com/window
icon1.ico 9c943baad621654cc0a0495262b6175276a0a9fb https://www.3cx[.]com/blog/event-trainings/
icon0.ico 9c943baad621654cc0a0495262b6175276a0a9fb https://www.3cx[.]com/blog/event-trainings/

A summary of the created, last modified, and domain registration times for each of these files is provided as an attachment to this post here.

Volexity believes the www.3cx[.]com entries were used for testing because, at the time of analysis, these URLs would not return a payload that could be parsed by the malware. Volexity was not able to retrieve payloads from the msedgeupdate[.]net or pbxphonenetwork[.]com domains, while the remainder of the URLs all provided the same valid second-stage payload.

The first commit to the GitHub page containing an ICO file with an encrypted 3cx[.]com URL was added on December 7, 2022, which suggests that the attacker had potentially initiated their own testing of the backdoor at this time.

Stage #2: ICONIC Stealer

Once a URL is decoded from an ICO file, a specially formatted request is made to download a second-stage payload. The format of the request is below:

accept: */*
accept-language: en-US,en;q=0.9
accept-encoding: gzip, deflate, br
content-type: text/plain
cookie: __tutma={MachineGuid}

The MachineGuid is derived from the system’s registry via SOFTWARE\Microsoft\ Cryptography.

Volexity’s analysis determined that the “cookie” header is the crucial component to retrieving the second-stage payload. If this value is not sent, no payload is returned to the user; the C2 responds with a 204 (No Content) status code instead. Volexity also determined that the second-stage payload servers are forwarding requests upstream to central infrastructure. This assertion is based on the fact that any given MachineGuid sent in the cookie header will only work once, even when used with different C2s.

Below is a snippet of what the returned JSON looks like:

{"url":"","description":"","meta":"vyoAAL4D<snip>"}

Each of the live servers returned identical responses, consisting of shellcode followed by a 64-bit DLL, which Volexity refers to as “ICONICSTEALER”. The DLL was compiled on March 16, 2023, and is designed to collect information about the system and browser using an embedded copy of the SQLite3 library. Details of the DLL are given below:

Name(s) N/A
Size 1.1MB (1182208 Bytes)
File Type application/x-dosexec
MD5 7faea2b01796b80d180399040bb69835
SHA1 3b3e778b647371262120a523eb873c20bb82beaf
SHA256 8ab3a5eaaf8c296080fadf56b265194681d7da5da7c02562953a4cb60e147423

The DLL retrieves the hostname, domain name, and OS version. Then, it will retrieve the browser history (title and URL) of the following browsers:

  • Brave
  • Chrome
  • Edge
  • Firefox

It limits the output to the first 500 entries, and it passes this data back to the ICONIC malware that then POSTs the data back to the C2. It is likely that the attacker then serves a further payload to victims of interest. Volexity has not been able to retrieve a further payload at this time.

macOS ICONIC Analysis

The macOS installer for 3CX was also compromised. The following table shows the details of this installer:

Name(s) 3CXDesktopApp-18.12.416.dmg|3CXDesktopApp-latest.dmg
Size 164.2MB (172150545 Bytes)
File Type Macintosh Disk Image
MD5 d5101c3b86d973a848ab7ed79cd11e5a
SHA1 3dc840d32ce86cebf657b17cef62814646ba8e98
SHA256 e6bbc33815b9f20b0cf832d7401dd893fbc467c800728b5891336706da0dbcec

The backdoor component is libffmpeg.dylib located in /Contents/Frameworks/Electron Framework.framework/Versions/A/Libraries. It is worth noting that this is the equivalent of the same library that was abused in the Windows binary.

Name(s) libffmpeg.dylib
Size 4.7MB (4979136 Bytes)
File Type Mach-O
MD5 660ea9b8205fbd2da59fefd26ae5115c
SHA1 769383fc65d1386dd141c960c9970114547da0c2
SHA256 a64fa9f1c76457ecc58402142a8728ce34ccba378c17318b3340083eeb7acc67

The macOS version does not use GitHub to retrieve its C2 server. Instead, a list of C2 servers is stored in the file encoded with a single byte XOR key, 0x7A. Below is a list of the URLs it will attempt to contact. Note that the domains largely overlap with the Windows sample, but the URIs are different.

  • msstorageazure[.]com/analysis
  • officestoragebox[.]com/api/biosync
  • visualstudiofactory[.]com/groupcore
  • azuredeploystore[.]com/cloud/images
  • msstorageboxes[.]com/xbox
  • officeaddons[.]com/quality
  • sourceslabs[.]com/status
  • zacharryblogs[.]com/xmlquery
  • pbxcloudeservices[.]com/network
  • pbxphonenetwork[.]com/phone
  • akamaitechcloudservices[.]com/v2/fileapi
  • azureonlinestorage[.]com/google/storage
  • msedgepackageinfo[.]com/ms-webview
  • glcloudservice[.]com/v1/status
  • pbxsources[.]com/queue
  • www.3cx[.]com/blog/event-trainings/

It is interesting to note that IDA Pro is confused by the main malicious function used in the macOS malware. The decompiled pseudocode hides most of the features. This means analysts relying on this view may miss the malicious functionality. Figure 1 shows the pseudocode of the function in IDA Pro. Figure 2 shows the same function with Ghidra, with more than 800 lines of pseudocode.

Figure 1. IDA Pro Pseudo Code

 

Figure 2. Ghidra pseudo code (totaling more than 800 lines)

The malware randomly picks one of the servers from the list to retrieve the next stage. As with the Windows version of the malware, a specially formatted cookie must be included in the web request to retrieve a further payload:

3cx_auth_id=%s;3cx_auth_token_content=%s;__tutma=true

The user-agent is also hardcoded and may be used by the attacker to filter valid requests:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.5359.128 Safari/537.36

Volexity was not able to retrieve the next stage from the C2 servers, as the upstream C2 infrastructure had stopped responding by the time it made requests.

Infrastructure & Attribution

In terms of attacker infrastructure, the domains used in these attacks are hosted on shared infrastructure and appear to simply proxy requests to an unknown upstream C2. Domains were registered with several providers, including NameCheap, Public Domain Registry, and NameSilo. Some of the domains were not registered using WHOIS protection, and each was registered using a unique email address. The following emails were observed in WHOIS records for related domains:

  • cliego.garcia@proton[.]me
  • remey.simpson@outlook[.]com
  • jackiewcaudill@gmail[.]com
  • philip.je@proton[.]me
  • haroldjmarable@gmail[.]com

In terms of attribution, the original CrowdStrike post suggests the incident is related to LABYRINTH CHOLLIMA, which is related to the public Lazarus moniker (although Volexity does not have visibility of exactly which parts of Lazarus this maps to). Volexity cannot currently map the disclosed activity to any threat actor, so it will be tracked under UTA0040.

Conclusion & Mitigations

Volexity’s analysis concludes that both the Windows and macOS installers for the 3CX desktop application had malicious code inserted into them before being provided to customers. This suggests that 3CX was itself compromised by the threat actor for a period of time prior to the infection, allowing the attacker to develop an understanding, access, and malicious code for the development-update process of the company.

The end result for victims of this campaign was that information-stealing malware was installed on endpoints that installed this update, and for selected victims, an additional arbitrary payload may also have been delivered.

Supply chain attacks are relatively rare due to the high level of technical and operational capability required for success. However, organizations with a large customer base, such as 3CX, are attractive targets due to the broad level of access these attacks can grant threat actors.

Volexity assesses that it is likely UTA0040 is a nation-state-backed threat actor based on the level of capabilities utilized in this campaign, combined with a perceived intent to gather information from victims for further targeted compromise. Crimeware-based threat groups who have historically conducted supply chain attacks typically push ransomware payloads immediately with their access, rather than try to conduct reconnaissance to filter victims of their true payload. While Volexity cannot attribute this cluster to any known group at this time, CrowdStrike has attributed this activity to LABYRINTH CHOLLIMA, a North Korean group.

Supply chain attacks are complex issues for defenders to defend against. This instance highlights how large code bases can be backdoored with minor additions to existing code and remain undetected by the software provider and the end user. However, information in the public domain highlights the value of endpoint and network detection capabilities, which provided valuable identification of anomalous behavior that may have prevented further impact for end users.

The infrastructure registration and public artifacts (notably the GitHub page) suggests that the attacker had access to the software provider at least as early December 2022, and perhaps as early as November 2022. It is not clear when the first malicious update described in this post was downloaded by victims of this campaign, but the public discussions around detections suggest this could be as early as March 22, 2023. This does not rule out other potentially malicious activities having occurred before this time related to this software.

To detect and investigate these attacks such as these, Volexity recommends the following:

  • Use the YARA rules provided here to detect related activity.
  • Use the provided Suricata rules here to detect related activity. It should be noted that these requests take place over HTTPs, meaning they are only effective if this traffic is being decrypted prior to matching.
  • Block the IOCs provided here.

Appendix A – Third-Party Reporting

There is a great deal of third-party reporting on this subject covering various aspects of the campaign. A list of resources is provided below, note that this list was compiled on March 30, 2023, and inevitably more resources will become available after publication.

Social Media Posts

CrowdStrike

Sophos

SentinelOne

Symantec

3CX

Objective-See

Bleeping Computer

Huntress

* Beginning in December 2022, Volexity began to use the following schema to refer to smaller unclassified clusters of threat activity not significant enough to warrant their own name: UTAXXXX. In this schema, “UTA” refers to “Unclassified Threat Actor”, and the numbers that follow are a unique identifier for that group of activity.

3cx, APT, iconic, North Korea, supply chain, supplychain