Unsecured Kubernetes Instances Could Be Vulnerable to Exploitation

This post is also available in: 日本語 (Japanese)

Executive Summary

Between October 2020 and February 2021, Unit 42 researchers periodically scanned and analyzed unsecured Kubernetes (also known as k8s) clusters on the internet. Kubernetes clusters can and should be configured for greater security, but when left unsecured, these clusters can be accessed anonymously by anyone who knows their IPs, ports and APIs. Researchers identified 2,100 unsecured Kubernetes clusters that consist of 5,300 nodes, 31,340 CPUs and 75,270 pods. A wide range of applications were seen running in these unsecured clusters, operated by organizations in sectors including e-commerce, finance and healthcare. The abundant computational resources and large amount of sensitive data in the applications – such as API tokens, database credentials, source code and personally identifiable information (PII) – make these unsecured clusters attractive targets to adversaries. The biggest cluster that researchers encountered had more than 500 nodes and 2,000 active pods.

Unit 42 discovered malware and malicious activities in some of these unsecured Kubernetes clusters. While most malicious code we observed runs in pods as containers, some is directly deployed on the host. Every containerized application runs in an isolated workspace (namespace) with different process IDs, networks and file systems from all other applications. If malware bypasses the isolation and accesses the underlying host's resources, it also poses a risk to the cloud environment that hosts the cluster. With this level of access, researchers demonstrated that more lateral or vertical movements are possible, including extracting credentials, compromising registries and accessing the host's data and network.

Unsecured Kubernetes API servers and unsecured Docker daemons have a lot in common. When misconfigured, adversaries can exploit the exposed APIs that run there to control the entire container platform. Our previous research identified numerous malicious activities in unsecured Docker daemons (examples include Graboid, Cetus and Pro-Ocean). Interestingly, though we performed similar research, we have not seen as many malicious activities in Kubernetes as in Docker.

Despite the seemingly lower level of malicious activities in unsecured Kubernetes clusters, attackers operating there could gain access to more valuable targets. As a compromised Kubernetes cluster can have much more resources (CPU, Memory and Network) and application data, a capable attacker can easily host command and control (C2) servers, build a botnet or steal sensitive data. We believe there will be more adversaries targeting or leveraging unsecured Kubernetes soon.

Palo Alto Networks customers running Prisma Cloud or CN-Series are protected from this threat. Prisma Cloud’s Kubernetes Benchmark and Runtime Defense can alert on insecure configurations and detect malicious activities such as cryptojacking. CN-Series provides complete L7 visibility that can effectively detect and isolate containers with malicious traffic.

Misconfigured Kubernetes Components

Kubernetes is a highly complex platform that consists of multiple layers and components. The modular design makes Kubernetes flexible and scalable, as most components can be individually configured or swapped to meet a specific goal. The complexity, however, also makes it challenging to secure the platform properly. One misconfiguration could lead to compromised containers or even compromised clusters, as shown in the next section.

This blog looks into two Kubernetes components, kube-apiserver (API server) and kubelet. Both run as RESTful API servers and accept requests that manage the underlying containers. We focus on these two components because they’re present in all Kubernetes platforms and can lead to host compromise if misconfigured. We analyze kube-apiservers and kubelets on the internet that can be accessed without authentication.

Kube-apiserver is the front end for the Kubernetes cluster that takes RESTful requests from clients and translates the requests to operations on containers, pods or nodes. Kubernetes allows clients to authenticate with client certificates, bearer tokens, etc. An anonymous client may or may not be able to access the API server, depending on the configuration. Sometimes the administrators may expose a subset of APIs to anonymous users, so that all applications can read specific data from it. The current Kubernetes implementation allows unauthenticated users to access restricted API information by default.

All the kube-apiserver’s APIs are well-documented in its API reference. Figure 2 shows an example of sending requests to an API server using curl, and Figure 3 makes the same requests using kubectl, a command line tool maintained by the Kubernetes community. Note that the APIs shown in the examples are not open to anonymous users by default. We bound more permissions to anonymous users for demo purposes.

Kubelet is an agent running on every node in a Kubernetes cluster. It takes RESTful requests from various components (mainly kube-apiserver) and performs pod-level operations. Clients can authenticate with Kubelet using client certificates or bearer tokens. Similar to API server, Kubelet may or may not accept unauthenticated requests, depending on the configuration. The current Kubernetes implementation allows anonymous access to Kubelet by default.

Although Kubelet doesn’t have an official API reference page, its APIs can be found in the source code. The open-source project kubeletctl also details all the Kubelet's APIs. In Figure 4, we use curl to send two requests to a Kubelet server, and in Figure 5, we make the same requests using kubeletctl.

A Glance Into Misconfigured Kubernetes Clusters

Between October 2020 and February 2021, we identified 6,400 API servers and 1,030 Kubelets exposed on the Internet. Within these servers, we found 47,030 containers, 47,115 images and 83,902 secret keys, as shown in Figure 1.

Figures 6-9 provide an overview of the locations, service providers, leaked APIs and versions identified in these Kubernetes clusters. Figure 6 shows that 50% of the exposed servers are found in China, and 18.2% in the U.S.

Figure 7 shows that around 65% of the unsecured servers are hosted in public clouds. 39% of the servers are hosted in Chinese-based cloud service providers (CSPs), followed by 25.9% in the U.S.-based CSPs. These results are consistent with the country distribution, in which 68% of the exposed servers are hosted in China or the U.S.

Figure 8 shows the APIs that can be accessed anonymously. We observed that different servers exposed different sets of APIs. For example, 18.3% of the exposed API servers allow /api/v1/nodes to be accessed anonymously, but only 13.6% of the exposed API servers allow /api/apps/v1/deployments to be accessed anonymously. Regardless, gaining access to any of these APIs could leak sensitive application or platform information.

Figure 9 shows the versions of the Kubernetes clusters identified – v1.12, v1.13, and v1.16 are the most commonly seen versions, contributing to 50% of the deployments. The number indicates that 50% of the clusters have not been updated for more than two years. At the time of writing, v1.9 is the most recent version available, contributing to only 2.2% of the deployments.

Malicious Activities in Unsecured Kubernetes Clusters

Unsecured Kubernetes clusters are vulnerable to all kinds of attacks. Among them, cryptojacking, in which attackers deploy malicious cryptominers in compromised containers, is still the most commonly seen attack. The cryptojacking malware we observed was either deployed as a new container or launched within a hijacked container. Once gaining access to a container, some malware also attempts to move laterally or vertically. Moving laterally allows attackers to control more containers or applications, and moving vertically allows attackers to control the host or even the cloud environment where the clusters are deployed.

Deploying Malicious Containers

Deploying container images that contain malware is a common way to launch an attack on a container platform. As containers are generally operating system (OS) agnostic, a containerized malware can easily run in any environment. Public container registries such as Docker Hub and Quay also simplify the hosting and delivery of malicious containers. Prior research uncovered numerous malicious images in Docker Hub (see previous research on finding malicious cryptojacking images in Docker Hub; attackers’ tactics, techniques and procedures in unsecured Docker Daemons; and using Docker images to mine for Monero).

Nevertheless, benign container images can also be abused for malicious purposes. Legitimate container images such as Ubuntu or Centos can be deployed in unauthorized environments repurposed to run malware. Most container platforms, by default, trust these official images. Table 1 shows three attacks that leverage official Docker Hub images to build and deploy malware. Each of these attacks was observed in hundreds of unsecured Kubernetes clusters.

^{Table 1. Cryptojacking containers in unsecured Kubernetes.}

There are many legitimate cryptocurrency mining tools published in Docker Hub. Crypto hobbyists often run these containers to participate in a mining network. However, the same images can also be abused. Table 2 shows how an attacker performs cryptojacking operations using XMRig, a Monero mining tool. The image was deployed as a DaemonSet, such that every worker node runs at least one instance of the container and pods are automatically rescheduled if a node joins or leaves the cluster. To disguise the malicious processes, the attacker named the DaemonSet as “Kube-controller” and deployed it in the kube-system namespace. kube-system namespace is usually used only for objects created by the Kubernetes system. When an administrator checks the running pods and daemons, this malicious container can be easily overlooked because of its obfuscated name.

^{Table 2. The configuration of a cryptojacking container.}

Launch Malicious Processes in Hijacked Containers

It is possible to perform remote code execution (RCE) in a running container through unsecured API servers or Kubelets. API server's exec API and Kubelet's run API allow adversaries to run arbitrary commands in containers. Launching a malicious process in a hijacked container is more stealthy as it doesn't need to pull new images or run new containers. Container vulnerability scanners or static analysis tools can't detect this type of attack.

In recent research, Unit 42 researchers revealed a cryptojacking malware called Hildegard that targeted misconfigured Kubernetes clusters. The attacker compromised the first container via an internet-facing and unsecured Kubelet. It then moved laterally to other nodes inside the cluster and hijacked multiple containers. In each compromised container, the malware attempted to establish an IRC channel back to a C2 server and launched a malicious cryptominer. However, malicious miners injected into running containers are prone to a shorter lifespan as containers can be restarted or rescheduled frequently. The malicious miners can't persist after the host containers are restarted.

Potential Attacks Against Unsecured Kubernetes

In the previous sections, we showed our findings in unsecured Kubernetes clusters. This section will describe a few potential attacks that can also occur in such unsecured environments. We first define initial access for each possible attack and illustrate how an attacker can exploit the initial access.

• Overly permissive capabilities. Each container is granted a subset of capabilities depending on the application’s requirements. For example, if an application needs to bind to a privileged port (port number below 1024), it needs to be granted cap_net_bind_service capability, and if an application needs to send an ICMP packet, it needs the cap_net_raw capability. Capabilities such as cap_sys_module or cap_sys_admin grant highly privileged permissions and allow containers to access the host's kernel modules. If a container with these capabilities is compromised, attackers can easily gain access to the host.
• Shared namespaces. A containerized application can run in a sandboxed environment because every resource of the container, such as processes, networking and file system, is placed in isolated namespaces. In some cases, a container may need to share its namespaces with other containers or the host. For example, if a container needs to capture the host’s network traffic, it needs to run in the host’s network namespace. This shared namespace exposes a channel for attackers to move from the container to the host.
• Unpatched vulnerabilities. Unpatched container runtime or node OS poses a risk of container breakout. Containerized applications are scheduled and managed by the container runtime. Vulnerabilities in container runtime, such as CVE-2019-5736, can allow applications to break out the containers. Because containers share the host's OS kernel, most kernel exploits could lead to privilege escalation and allow container breakout.
• Stolen service account token. Service account tokens reside in containers and are used by applications to authenticate with API servers. If stolen, attackers can impersonate the applications and leverage API servers to move laterally or vertically.
• Stolen cloud account credentials. Attackers may access cloud credentials through compromised applications or containers, as described in the Hildegard research. With cloud credentials, attackers can gain access to the cloud environment that hosts the Kubernetes clusters, which leads to a much worse breach.

Protection and Mitigations

All the issues mentioned above can be prevented using strict security policies, including:

• Patch frequently.
  • Known vulnerabilities can be exploited for initial access or privilege escalations. Applications and container platforms should be scanned periodically and patched frequently.
• Secure the network.
  • Never expose any Kubernetes components, such as API server, Kubelet or etcd to the internet. Firewall rules should only allow a small subset of IPs to communicate with these components.
  • Enforce mutual TLS (mTLS) on every Kubernetes component that supports mTLS.
  • Enforce Network Security Policy to restrict the pods, namespace and IP blocks that a pod can communicate with.
• Secure the identity.
  • Enforce role-based access control (RBAC) and practice the principle of least privilege. Grant every role the permissions that are absolutely needed.
  • No components should accept or respond to anonymous/unauthenticated requests.
  • Rotate the credentials and certificate periodically to mitigate potential credential leaks.

Conclusion

Between October 2020 and February 2021, Unit 42 researchers identified a large volume of sensitive data leaked from unsecured Kubernetes clusters on the Internet. The leaked data includes API tokens, database credentials, source code and PII. While we haven’t seen any large-scale or sophisticated attacks, we observed an increasing trend of opportunistic attacks in the past five months. With the rapid adoption of container technology and the abundant computational resources in these Kubernetes clusters, we believe more adversaries will soon turn their attention to this unexplored space. Regardless, simple best practices and continuous monitoring can effectively prevent and detect most of these opportunistic attacks. With the complex and dynamic nature of Kubernetes technology, it is also recommended to oversee the clusters with cloud workload protection solutions such as Prisma Cloud.

Palo Alto Networks customers running Prisma Cloud are protected from this threat by the Runtime Protection and Cryptominer Detection features and by Prisma Cloud Compute Kubernetes Compliance Protection, which alerts on an insufficient Kubernetes configuration and provides secure alternatives. Prisma Cloud Compute’s vulnerability scanner can alert on application vulnerabilities inside containers and platform/OS vulnerabilities on the hosts.

Additional Resources

Highlights from the Unit 42 Cloud Threat Report, 1H 2021