Protecting Against an Unfixed Kubernetes Man-in-the-Middle Vulnerability (CVE-2020-8554)

This post is also available in: 日本語 (Japanese)

Executive Summary

On Dec. 4, 2020, the Kubernetes Product Security Committee disclosed a new Kubernetes vulnerability assigned CVE-2020-8554. It is a medium severity issue affecting all Kubernetes versions and is currently unpatched. CVE-2020-8554 is a design flaw that allows Kubernetes Services to intercept cluster traffic to any IP address. Users who can manage services can exploit the vulnerability to carry out man-in-the-middle (MITM) attacks against pods and nodes in the cluster.

Adversaries may utilize MITM attacks to masquerade as internal or external endpoints, harvest credentials from network traffic, tamper with a victim’s data before sending it to its intended target or block communications with specific IPs altogether. Using encrypted protocols such as Transport Layer Security (TLS) can help, as attackers cannot easily access their traffic.

Multi-tenant clusters are most at risk, as they are most likely to have non-admin users that can manage services. Attackers that compromised a single tenant may exploit the vulnerability to perform MITM attacks against other tenants in the cluster.

The Kubernetes Product Security Committee determined that patching CVE-2020-8554 would result in functionality changes to several Kubernetes features, so there is no plan to resolve the vulnerability in the short term. Instead, the committee recommended several mitigations that restrict access to the vulnerable features, either based on a custom admission controller or through an Open Policy Agent (OPA) Gatekeeper rule. Palo Alto Networks Prisma Cloud Compute customers can enable those mitigations through the built-in Admission support for OPA rules. Please refer to the “Prisma Cloud Compute Mitigation” section for instructions.

The Vulnerability

CVE-2020-8554 stems from a design flaw in two features of Kubernetes Services: External IPs and Load Balancer IPs. Kubernetes Services are an abstract way to expose an application running on a set of pods as a network service. A service is exposed on one or more IPs. Once deployed, nodes in the cluster will route traffic destined to the service IPs to one of the backing pods that make up the service.

When the cluster manages and assigns service IPs, all is well. The problem starts when a Kubernetes user is able to assign arbitrary IPs to their services. In that case, a malicious user can assign IPs that are already in use by other endpoints (internal or external), and intercept all cluster traffic to those IPs. There are two methods to control the IP of a service:

1. Assign it an external IP.
2. Assign it a Load Balancer IP by patching the status.loadBalancer.ingress.ip field. This method requires the patch service/status permission.

Below is a service that, when deployed to the cluster, will intercept all cluster DNS traffic to IP 8.8.8.8 (Google’s DNS Server) and route it to the evil-dns-server pod.

To receive the intercepted traffic, attackers must control the endpoints backing their malicious service. In most cases, this will be a pod, like the evil-dns-server pod in the example above. Services can also be backed by external endpoints (rather than cluster pods), which means attackers can also route intercepted traffic to an external endpoint outside of the cluster. This requires the attacker to create a Kubernetes endpoint that points to an external address, which requires the create endpoint privilege.

Am I Affected?

The vulnerability affects all Kubernetes versions and is currently unpatched. Clusters are affected if they meet these conditions:

• Allow non-admin Kubernetes users to create or update services, or to patch the status of services; AND
  • Allow those unprivileged users to control a pod (create, update or exec to pods); OR
  • Allow those unprivileged users to create or update endpoints.

Multi-tenant clusters are most at risk, as they are most likely to implement the vulnerable configuration above. Multi-tenant clusters often segregate tenants using Kubernetes namespaces, limiting each tenant’s permissions to its namespace. Unfortunately, even if a tenant can only manage service and pods in his own namespaces, it can still exploit CVE-2020-8554 to eavesdrop on traffic from the entire cluster. Attackers that comprise a single tenant may exploit the vulnerability to intercept other tenants’ traffic and compromise them.

Clusters using Cilium to replace kube-proxy aren’t affected at all.

Indicators of Compromise for CVE-2020-8554

You may have been attacked if one of the following is true:

• A service that shouldn’t expose external IPs or Load Balancer IPs does so.
• An external IP or Load Balancer IP of a service matches an internal IP in the cluster – a pod IP or a clusterIP of another service.
• An external IP or Load Balancer IP of a service points to a known external domain (e.g. 8.8.8.8). Run nslookup <externalIP> to identify if an IP points to a known domain.
• A Load Balancer IP of a service is 127.0.0.1 (indicates an attempt to hijack node localhost traffic).

To identify services in your cluster that expose external IPs, run the following command:

To identify services in your cluster that expose Load Balancer IPs, run the following command:

Mitigations

The Kubernetes Product Security Committee concluded that the vulnerability could not be patched without functionality changes to features Kubernetes users rely upon. Instead, it recommended enforcing mitigations that restrict access to the vulnerable features. To prevent the use of External IPs, the committee provided two solutions: a custom Admission Controller and an OPA Gatekeeper constraint as solutions.

No mitigations were provided for Load Balancer IPs. Exploiting the issue through Load Balancer IPs requires the patch service/status permission, which is considered privileged. Ensure that unprivileged cluster users do not possess this permission. System components are the ones that normally update the status of a service, so user accounts doing so should raise suspicions.

Conclusion

CVE-2020-8554 is a unique vulnerability that is rooted in the design of Kubernetes Services. If your cluster is multi-tenant, or allows unprivileged users to create and update services, you are impacted. If applications within your cluster communicate without enforcing encryption via TLS, you're at greater risk. Even though the vulnerability is unpatched, the mitigations proposed by the Kubernetes Product Security Committee can effectively prevent exploitation. Prisma Cloud Compute customers are encouraged to enable the admission rules in the ‘Prisma Cloud Compute Mitigations’ section to protect their clusters.

Appendix: Prisma Cloud Compute Mitigations

Prisma Cloud Compute’s built-in Admission support for Rego rules can be used to implement the mitigations proposed by the Kubernetes Product Security Committee.

Restrict Access to External IPs

Customers can use the following instruction to set up an Admission rule to block services from exposing external IPs. Customers can optionally provide a whitelist for allowed IPs.

1. Follow Prisma Cloud Compute documentation to enable Admission Control in your cluster.

2. Download the “Mitigation for Kubernetes CVE-2020-8554 - External IPs” rule template. You can download the rule by running the following command:

wget https://raw.githubusercontent.com/twistlock/k8s-cve-2020-8554-mitigations/main/PrismaExternalIPs.json

3. Navigate to ‘Defend/Access/Admission’ and hit ‘Import’, then choose the downloaded rule template. You should be presented with the dialog below. Press ‘Add’.

4. To whitelist certain IPs for use as external IPs, update the rule as outlined in Figure 3. The following rule whitelists IP 203.54.74.83.

Once the rule is in place, attempts to set up services that expose forbidden external IPs will fail and trigger an alert under ‘Monitor/Event/Admission audits’.

You can press on each alert to view the request’s details.

Alert on suspicious updates to Load Balancer IPs

Typically, only system accounts should assign Load Balancer IPs to services. User accounts doing so may indicate a user is trying to exploit CVE-2020-8554. Unless that is not the case in your cluster, we recommend following the instructions below to enforce an appropriate rule set on Alert.

1. Follow Prisma Cloud Compute documentation to enable Admission Control in your cluster.

2. Add services/status to the resources monitored by Prisma Cloud Compute’s admission webhook. Run the following command to update Prisma Cloud Compute’s webhook configuration:

Note: If you have custom Prisma Cloud Compute admission rules in place that handle service update operations, they’ll start receiving services/status update requests as well after running the above command. Add the following line to these rules to discard services/status requests:

3. Download the “Mitigation for Kubernetes CVE-2020-8554 - Load Balancer IPs” rule template. You can download the rule by running the following command:

wget https://raw.githubusercontent.com/twistlock/k8s-cve-2020-8554-mitigations/main/PrismaLoadBalancerIPs.json

4. Navigate to ‘Defend/Access/Admission’ and hit ‘Import’, then choose the downloaded rule template. You should be presented with the dialog below. Press ‘Add’.

5. If certain accounts are expected to configure Load Balancer IPs, the IP whitelist approach implemented in the external IPs rule can be used.

Once the rule is in place, Prisma Cloud Compute will alert on requests made by user accounts that patch the status of a Load Balancer service. Alerts will show up under ‘Monitor/Event/Admission audits’. You can press on each alert to view the request’s details.