TCP SACK Panics Linux Servers

This post is also available in: 日本語 (Japanese)

Executive Summary

The newly discovered Linux vulnerabilities, CVE-2019-11477, CVE-2019-11478, and CVE-2019-11479, affect all Linux operating systems newer than kernel 2.6.29 (released on March 2009) or above and can cause a kernel panic to systems with services listening on a TCP connection. This remote attack can put a server into a Denial of Service (DoS) state, but remote code execution is not of concern. The vulnerability roots on the flaws in the TCP Selective Acknowledgement (SACK) and Maximum Segment Size (MSS) implementation. The attack can be triggered by a remote user who sets the MSS to the lowest limit of 48 bytes and sends a sequence of specially crafted SACK packets to overflow the receiver’s socket buffer. Most of the Linux distributions have released a patch, RedHat, Ubuntu, and openSUSE. If immediately patching the system is not possible, the attack can also be blocked by firewall rules or traffic control policies. As the attack is fairly simple, a proof of concept (POC) exploit should be available soon and we expect to see large-scale DDoS attacks within a few days as attackers exploit unprotected systems.

TCP SACK

RedHat gives a detailed description of this attack; we’ve included a brief review of the attack. The TCP SACK attack is achieved by tipping the balance between the sender’s TCP selective acknowledgment (SACK) mechanism and the receiver’s maximum segment size (MSS) settings. SACK is a mechanism used to inform the sender system which TCP packets have been accepted successfully by the receiver and which packets need to be resent. MSS is a setting set by the receiver on how large of a packet can be received by its network. When a receiver’s MSS is set to its lowest setting of only 48 bytes, the sender is required to send more segments to fulfill the request. When a receiver requests a large network transfer, i.e. media, or a large file transfer, the sender begins the monitoring the successful receipt of transmitted packets. However, the imbalance between the receiver’s low MSS capability and the potential for a sender's limited volume to store a long SACK queue can cause an overflow in the sender's SACK queue. This overflow of unsent packets, when large enough, can trigger a kernel panic state causing the DoS.

Impact

Any server with Linux kernel 2.6.29 or above and with open ports running TCP services are vulnerable to this attack. We perform a few queries on Shodan to understand the potential impact of this vulnerability, see Image 1. There are currently 1,944,975 Linux-based servers discoverable on the internet. 543,797 servers have http running on port 80, 152,416 servers have sshd running on port 22, and 126,143 servers have https running on port 443. If no special firewall rules or traffic control policy is enforced on these servers, they can all be potential targets to the attack.

Image 1. Publicly discoverable Linux systems via Shodan

Resolution

This attack targets Linux kernel systems. If you have a Linux system that is running a kernel version of 2.6.29 or higher, the system should be upgraded to the latest Linux release available. If the client is using Google Cloud Platform (GCP), Microsoft Azure, or another Infrastructure as a Service (IaaS), please follow their Linux upgrade procedures to upgrade all Linux systems to the latest release.

Amazon has identified the following services which can fall victim to the identified SACK Panic attack:

• AWS Elastic Beanstalk
• Amazon Linux and Amazon Linux 2
• Amazon Elastic Compute Cloud (EC2)
• Elastic Load Balancing (ELB)
• Amazon WorkSpaces (Linux)
• Amazon Elastic Container Service for Kubernetes (Amazon EKS)
• Amazon ElastiCache
• Amazon EMR

Clients who use any or all of the identified Amazon services should update those services to their latest versions. Below is an abbreviated listing of actions to be taken for each of the listed Amazon Services.

AWS Elastic Beanstalk

• If clients are using Amazon’s Managed Platform Updates, these systems will be updated during the selected maintenance window. If the client has not enabled Managed Platform Updates, the following link will assist them in setting up that process, Managed Platform Updates Guide.

Amazon Linux and Amazon Linux 2

• New Linux kernels have been uploaded and are available within the Amazon Linux repository. Clients should run the following command in each of their EC2 instances to receive the kernel update.
  sudo yum update kernel

Amazon Elastic Compute Cloud (EC2)

• Perform the following command within each EC2 instance
  sudo yum update kernel

Elastic Load Balancing (ELB)

• If your ELBs are configured to be classic load or application balancers, or if they are configured with TLS termination enabled, no action required.If you are running Linux-based EC2 instances within AWS which are not terminating TLS sessions, update the Linux kernels for those systems. Instructions can be found here.

Amazon WorkSpaces (Linux)

• All new Linux WorkSpaces will be updated automatically and do not require any updates by the client.

Amazon Elastic Container Service for Kubernetes (Amazon EKS)

• All currently-running EKS clusters are protected against this attack. Clients should replace all worker nodes to use the latest EKS version.

Amazon ElastiCache

• ElastiCache VPCs to not accept untrusted TCP connections by default and are not affected.
• If a client has made changes to their default ElastiCache VPC they will need to ensure their ElastiCache security groups follow AWS best practices and configure them to block traffic from untrusted clients. Additional information can be found in Amazon’s VPC and ElastiCache Security page.
• If a client has an ElastiCache running outside of a VPC and has altered the ElastiCache from the default, should configure their instance using ElastiCache security groups. More information can be found in Amazon’s EC2-Classic Security Group page.

Amazon EMR

• Amazon EMR launches EC2 instances running Linux into a client’s VPC, and these do not accept untrusted TCP connections by default and are not affected.
• Any client who has made changes to the EMR VPC should follow AWS best practices and configure them to block traffic from untrusted clients. Additional information can be found in Amazon’s Control Network Traffic with Security Groups page.
• If clients choose not to configure EMR security groups according to AWS best practices can follow these guidelines:
  • New Clusters - use EMR bootstrap actions to update the Linux Kernel and reboot each instance. Assistance can be found in Amazon’s EMR Bootstrap Plan page.
  • Existing Clusters - Update the Linux kernel on each instance within the cluster and reboot the instances one at a time.

Conclusion

Due to the low complexity and high severity of this vulnerability, it won’t be surprising to see large scale DDoS attacks in a few days. It is critical to update the Linux kernel as soon as possible. For the servers running in the public cloud with TCP services open to the internet, it is even more critical to patch immediately or at least set up firewall rules to block the attack. The resolution guideline provided in this blog should help administrators secure their systems.