Misconfigured and Exposed: Container Services

This post is also available in: 日本語 (Japanese)

Executive Summary

There are 40,000+ unique container hosting devices which have default container configurations that allow for quick identification. Both Kubernetes and Docker container platforms each have over 20,000 unique instances apiece. This does not necessarily mean that each of these 40,000+ platforms are vulnerable to exploits or even the leakage of sensitive data: it simply highlights that seemingly basic misconfiguration practices exist and can make organizations targets for further compromising events. Seemingly simple misconfigurations within cloud services can lead to severe impacts on organizations. For example, Docker Hub losing keys and tokens for 190,000 accounts was the result of an attacker exploiting weak security configurations of key and token storage within a cloud environment. Ladders, leaking of over 13 million user records is a perfect example of a basic container misconfiguration having significant consequences. The lessons learned from the Ladders breach should be at the security configuration forefront for all organizations deploying container services.

In our research, we were able to easily find 20,353 Kubernetes containers globally using simple search terms. These instances were located in the United States, Ireland, Germany, Singapore, and Australia and the overwhelming majority of them were hosted on Amazon. We were also able to easily find 23,354 Docker containers globally using simple search terms. These instances were located in China, the United States, Germany, Hong Kong, and France. In the case of Docker, while Amazon was again the top hoster, there was a broader distribution of hosters for these instances.

We then did further research into some of these exposed instances to see what services were exposed and information being leaked. We found sites exposing database instances to the public and also sites easily exposing personal information.

The following research highlights the results from Unit 42’s research into misconfigured containers, methods for identifying services exposed to the public, and mitigation steps to secure container services. In this blog, we identify common misconfigurations in container services. This allows our readers to deploy their container platform structures in a more secure and private fashion, avoiding the methods of data gathering that we outline in this blog.

Cloud Services

Cloud services can be any service that is available over an internet connection. Cloud services allow organizations to quickly deploy, maintain, and support projects requiring dedicated availability for services that may experience demand fluctuations in their designed service offering; common examples are API gateways, web services, and databases. There are several orchestration platforms, facilitated by either a managed service provider or by a self-serviced internal organization, which provides configuration functionality for containers and can provide policy enforcement for the platform. This functionality covers security or audit logging, role-based access control, and network connection enforcement for cloud infrastructure. Selecting the appropriate orchestration platforms or service providers can greatly assist in the security of cloud containers.

Shodan

To identify which cloud services and container platforms were exposed to the public Internet, Unit 42 used Shodan to highlight the ease of identification of these services. For reference, Shodan has been called the “Google for finding IoT” (Internet of Things) devices. While this research is not focused on identifying IoT devices, several of the same techniques can be used to identify cloud systems and containers that are connected to the public Internet. Like IoT devices, containers are connected to the Internet and can be publicly exposed through misconfigurations.

Shodan continually scans the publicly accessible internet address space, requests and then records the responses from each port, or service, for each particular address. The responses are saved to a database which is then made available to Shodan members. Shodan provides services like enhanced system analysis, service categorization, and tagging features which provide additional insight into the functionality and scope of each identified system. Some of this data is available via a free Shodan account, typically with a rate limit to avoid automated abuse, while additional analytic content curated by the Shodan team is only available to paying account holders. Examples of the paid membership benefits are: increased rate limits for returned results, advanced filter capabilities for detailed search requests, increased alerting on changes to specific net ranges, automated scanning over pre-defined criteria, and the ability to query for specific analytic tags. To highlight the ease of use for finding container services identified within this blog, the examples below can be performed with a free Shodan account.

Hunting for Exposed Default Containers

Default container platforms with little customization have telling characteristics which can be used to easily identify them within a Shodan search. Searches as simple as using the key terms of “kubernetes”, “docker” or even “k8s”, which is an abbreviated term for Kubernetes, being that there are eight characters between the ‘k’ and the ‘s’, can all be used to identify default containers.

Here are some simple search examples and results using the Shodan website:

Kubernetes

• • Performing a Shodan search for the term “Kubernetes”, results in a total of 20,353 devices which contain the term. There is additional metadata information returned, allowing the researcher to drill into what could be interesting results. Items like top countries, services, organizations, operating systems, and products. As you can see from the returned information, the United States has the most number of devices containing the term “Kubernetes” and we can also see that Amazon[.]com is the organization which hosts the majority of these devices. Figure 1 illustrates the type of results Shodan pulls back.

Figure 1. Shodan webpage Kubernetes query results

1. • Figure 2 illustrates an example of using the Shodan CommandLine Interface (CLI) tool. It shows an example of the ‘count’ functionality of a particular keyword. In this case, the current count of devices which contain the keyword “Kubernetes”, as it displayed within the Figure, the total count of devices is the same result as Figure 1, 20,353. For additional information regarding available commands provided by the Shodan CLI, please direct your attention to the Shodan CLI page. Suffice to say, having scriptable capabilities of Shodan searches can be very beneficial for researchers who wish to perform complex searches.

Figure 2. Shodan CLI Kubernetes query results

k8s

1. • Figure 3 shows a total of 509 devices which contain the term “k8s”. It is also interesting to note that the United States still remains the top country for returned results and that Amazon[.]com is still the top organization hosting k8s devices.

Figure 3. Shodan webpage k8s query results

1. • Figure 4, Shodan’s CLI also shows the same count value for k8s devices.

Figure 4. Shodan CLI k8s query results

Docker

1. • Figure 5 shows there were 23,354 devices which contain the term “Docker”. This is 1,001 more than Kubernetes devices, which could be accounted for due to Docker being an older and more established container platform. However, it is interesting to note that the top 5 countries are closer in total count than either Kubernetes or k8s. China came in as the top country with 6,015 and the United States was second at 4,617. Amazon[.]com remained the top organization but the next 3 out of 4 remaining top 5 organizations were Chinese based organizations, and number 2 was a Hong Kong based organization. These numbers can lend insight into which organization’s are favored to host Docker containers within each country.

Figure 5. Shodan webpage Docker query results

These searches reveal that Kubernetes and Docker container platforms each have over 20,000 unique instances publicly exposed to the internet, and the Kubernetes platforms have up to an additional 509 unique instances configured using its commonly used abbreviation, k8s. This does not necessarily mean that each of these 40,000+ platforms are vulnerable to exploits or potentially leaking sensitive data, it simply highlights that misconfiguration practices exist even in the most basic form of failing to rename a container. However benign, basic misconfigurations are being used within the industry. If there are misconfigurations in basic container usage, how far can this misconfiguration practice carry an ambitious attacker? And more importantly, can sensitive information be gathered from these misconfigured container platforms?

According to our research, yes. There are instances where the failure to properly configured containers can lead to the loss of sensitive information. This type of misconfiguration practice is becoming more publicized yet reports like Docker Hub losing keys and tokens for 190,000 accounts dominate headlines and can mask the deeper underlying issue. A good example being the recent report of the recruitment site Ladders, leaking of over 13 million user records. This is the result of a simple misconfiguration practice, failure to implement basic authentication requirements.

Hunting for Exposed Default Services

In order to identify the exposure of misconfigured container services, we first need to build a baseline for our research and to understand the landscape we are researching. By default Elastic, Kibana, and MySQL services use the following ports of 9200, 5601, and 3306, respectively. Shodan can provide basic insights into the total number of how many of each service are publicly accessible.

Elastic

An Elastic database uses the TCP port 9200 as the default Rest API interface for the data contained within the database. When navigating to an Elastic database via a browser, the user would be presented with a JSON formatted response with the datasets allowed to be viewed. In Figure 6, we show the result of a Shodan search we performed to count the total number of devices which had the TCP port 9200 open to the public. There were 1.448 million devices identified, with 1.3 million located within the United States. Nearly half of all the devices, 775,284 entries, returned the domain of ‘googleusercontent[.]com’. When performing additional research correlating the domain in conjunction with the port 9200 across known issues within Google Infrastructure, resulted in this being an Elastic service hosted on Google Cloud infrastructure. This is a very strong indicator that nearly all of these identified devices are hosting Elastic services. This domain also accounted for 53% of all Elastic instances.

Figure 6. Shodan CLI statistics for port 9200 (Elastic)

Kibana

Kibana is a database visualization tool used to provide a user with a Graphical User Interface (GUI) to view database information. Kibana uses an HTTPS web browser connection over the TCP port 5601 by default. In Figure 7, we show the results of a Shodan search we performed to count the total number of devices which had the TCP port 5601 open to the public. There was a total of 349,403 devices identified, with 288,159 located within the United States. As was the case with the Elastic results before the majority of entries, 56,635, were from the domain of ‘incapdns[.]net’. This domain is a web site hosting service which provides web hosting services for organizations. This domain accounted for 16% of all port 5601 entries.

Figure 7. Shodan CLI statistics for port 5601 (Kibana UI)

MySQL

The MySQL protocol is used to facilitate network communication to a SQL database over TCP port 3306 by default. In Figure 8, we show the results of a Shodan search we performed to count the total number of devices which had the TCP port 3306 open to the public. There was a total of 4.816 million devices identified worldwide with 2.138 million located within the United States. Interestingly, Google Cloud services fell 7 spots within these results and Amazon was the domain which hosted the largest number of MySQL protocol enabled services. It is important to note that no hosting domain carried the vast majority of MySQL protocol enabled services, with the highest percentage of instances being only 4% of all identified MySQL enabled devices being hosted on Amazon.

Figure 8. Shodan CLI statistics for port 3306 (SQL DB)

In recap, these results provide us with a frame of reference on the total scope of the services available to the public. At the time of this writing, there were 1,448,102 devices listening on port 9200, 349,403 devices listening on port 5601, and 4,816,638 devices listening on port 3306. It is important to note that not all of the data contained within these unique systems are accessible to the public, research points to the vast majority of these systems requiring authentication credentials in order to access any contained data. Additionally, it is possible that several of these systems do not host an Elastic, Kibana, or MySQL protocol enabled service, but rather another service that is simply listening on the same port that these services do by default. As such, these results do not greatly assist researchers to identify container misconfigurations.

Narrowing our Scope

Now that we have identified the landscape, we will now narrow our scope to identify any misconfigured containers that may be present. We began by augmenting our initial baseline queries, see figures 6-8 within the previous section, by combining multiple searches together. For example, searching for default ports with default container names, “port:9200” (the Elastic database RestAPI port) with the term “Kubernetes”. This query format would allow us to identify defaults Elastic services running on default Kubernetes container instances.

To focus on Elastic services hosted on Kubernetes containers, we performed a search with both ‘port:9200’ and ‘kubernetes’. As you can see from Figure 9, the results were narrowed from 20,353 unique results (Figure 1) from Kubernetes alone, and 1.4 million results from port:9200 alone (Figure 6), to a total of 13. The majority of the devices are hosted within China and the United States has only 2.

Figure 9. Shodan CLI statistics for port 9200 (Elastic) and Kubernetes

To focus on Docker containers hosting Elastic services, we performed a search with ‘port:9200’ and ‘docker’ as the key terms. Figure 10 illustrates a total of 39 unique devices contained default Elastic services hosted on default docker containers.

Figure 10. Shodan CLI statistics for port 9200 (Elastic) and Docker

Instead of the 1.4 million default Elastic instances, we narrowed the results to 13 default Elastic instances running on default Kubernetes platforms and 39 instances on default Docker platforms. This quantity is much easier to analyze and perform deeper research.

Shodan records the IP addresses of each system and those addresses from the identified Elastic instances were gathered using the following Shodan CLI command:

shodan search –fields ip_str port:9200

With the resulting IP addresses, we investigated select systems to identify if they contained sensitive information.

Example #1

The following system did have an unprotected elastic database exposed to the public. This database did not contain any sensitive information, however, it is a good example of what types of information an Elastic database can provide. We navigated to the Elastic system and performed the search query ‘<ipaddress>:9200/_search?q=*’. This query returned the following JSON data, see Figure 11.

Figure 11. JSON results from a queries Elastic database

Example #2

Example #2 did contain sensitive information within a default container using default services. Researchers navigated to the Elastic instance using the previously defined URL query format:

<ipaddress>:9200/_search?q=*

When investigating, a database titled “users-registered-2018*” was identified. We added the database to the search query:

<ipaddress>:9200/users-registered-2018*/_search?q=*

We identified a total of 19 user email addresses contained within the database, see Figure 12.

Figure 12. User email addresses identified

Email addresses are not on the same level of sensitivity as items like credit card numbers, social security numbers, or other highly sensitive information. However, they are often targeted by malicious actors. An example of a malicious action which could be performed leveraging email addresses would be spear phishing attacks. Malicious actors could gather information on an organization like its geographic location, the number or type of office buildings, and most importantly, employee names. This information allows an attacker to construct targeted emails, Business Email Compromise (BEC), which contain familiar information for the victims, enticing them to click on malicious links resulting in compromised accounts.

Unit 42 further identified that this system not only hosted an Elastic database without any form of authentication mechanism to help secure the data it contained, but also hosted an additional unsecured service: an instance of Kibana running alongside the Elastic instance.

We queried the following URL ‘<ipaddress>:5601’, and were able to gain access to the Kibana dashboard which contained additional database information aside from what was initially found within the Elastic service. We found two additional databases, one which contained internal infrastructure IP addresses and the second which contained logging information for a custom internal integration system. Finally, the user-registry database appeared to only have a two-month active life span, between September and November 2018, but the services were still in operation as of the time of this writing. Additionally, at the time of this writing the owner of the database has yet to be identified; we are still trying to ascertain who the owner for victim notification.

Figure 13. User email addresses identified via Kibana

Mitigation Steps

Unit 42 proposes the following steps to improve the overall security of container platforms.

• Invest in cloud security platforms or managed services which focus in container security strategies.
• Limit access to services hosted on containers to internal networks, or prior designated personnel, through the use of firewall controls or container platform network policies. The following links can help assist secure access to containers:
  • Docker – iptable Configuration
  • Kubernetes – Network Policies
• Establish basic authentication requirements for your containers. The following two links provide helpful instructions for how to establish a basic authentication practice for either Docker or Kubernetes:
  • Docker – Tokens
  • Kubernetes – Authenticating
• Identify the type of data stored or managed within each container and use the appropriate security practices to keep these data types secure.
  • Your organization’s compliance policies will assist in dictating the protections required.
• Isolate services to their own containers. Do not host more than one service on a single container, this will improve the resource efficiency of the container itself, and will assist in implementing effective security policies.

Conclusion

Default configurations can be significant security risks for organizations. The research performed by Unit 42 demonstrated the ease of use in searching for default configurations and demonstrated how a potential attacker could leverage an open source tool like Shodan. Misconfigurations such as using default container names and leaving default service ports exposed to the public leave organizations vulnerable to targeted reconnaissance. Using the proper network policies, or firewalls can prevent internal resources from being exposed to the public internet. Additionally, investing in cloud security tools can alert organizations to risks within their current cloud infrastructure. With attacks like that of the Ladders data leakage, or the Docker Hub loss of keys and tokens, the risks facing organizations operating in the cloud are great.