Blinding the Watchmen: Abusing Cloud Logging Services for Defense Evasion and Visibility

Executive Summary

Cloud logging services provide comprehensive visibility into actions performed within cloud resources, making them essential for security monitoring. However, this reliance also makes logging services a high-value target for attackers. An attacker who exploits these services could create weak spots, evade detection, and in certain scenarios, establish continuous visibility within a target’s environment.

Services such as Amazon Web Services (AWS) CloudTrail and Google Cloud are powerful for defenders, and prime targets for attackers seeking to remain undetected by disrupting the flow of logs. Attack techniques against cloud logging services primarily fall into two categories:

• Defense Evasion: Attackers aim to bypass detection systems, to execute attacks unnoticed. This may involve modifying resources within the cloud logging service.
• Continuous Visibility: Attackers attempt to transfer logs to their own accounts, establishing continuous visibility over the victim's environment.

Understanding these attack scenarios enables organizations to implement the appropriate configurations and detect service misuse.

Palo Alto Networks customers are better protected from the threats discussed above through the following products and services:

• Cortex Cloud

The Unit 42 Cloud Security Assessment is an evaluation service that reviews cloud infrastructure to identify misconfigurations and security gaps.

If you think you might have been compromised or have an urgent matter, contact the Unit 42 Incident Response team.

Cloud Logging Services

Serving as the authoritative system of record for every event, cloud logging services provide complete visibility into all actions within cloud environments. This comprehensive data enables analysis of past behaviors for both operational debugging and security investigations.

Each cloud provider implements logging services in a unique way. Our recent Cloud Logging for Security article provides an overview of these various services across different cloud providers. In this article, we analyze and demonstrate attack techniques that target the primary logging services within each major cloud provider.

Before examining the logging capabilities offered by major cloud providers, we outline the fundamental components and mechanisms for log delivery. Our analysis focuses on AWS CloudTrail and Google Cloud Logging. Both of these widely used services are designed to provide comprehensive audit trails and operational insights. While this article focuses on specific services, the attack techniques presented may also apply to other cloud logging services.

Background: How Cloud Service Providers Handle Logs

AWS CloudTrail

AWS CloudTrail's primary resource for configurable log collection and delivery is known as a trail. A trail acts as a configuration that specifies how CloudTrail records AWS application programming interface (API) calls and related events in an AWS account. These events include actions taken by users, roles or AWS services.

The main function of a trail is to deliver these captured logs to an Amazon S3 bucket. S3 is a highly scalable, durable, secure object storage service. When a trail is configured, it continuously writes log files containing event records to the designated S3 bucket.

The S3 bucket serves as a centralized, long-term repository for CloudTrail logs. This enables auditing, security analysis and compliance efforts. CloudTrail supports sending native event trails to either the CloudTrail Lake feature, EventBridge or CloudWatch Logs. However, for enterprises with integrations to third-party products, these features might not be relevant or usable.

Google Cloud Logging

Cloud Logging is a fully managed service that collects logs from all of an organization’s Google Cloud resources. Cloud Logging leverages a resource called a sink as the primary mechanism for log delivery. A sink functions as a router that sends log entries to a specific destination. While a sink primarily routes and stores logs in centralized log buckets for analysis and retention, it also offers the flexibility to export logs to other powerful Google Cloud services. For example, logs can be routed to a cloud storage bucket for cost-effective long-term archival. When a sink is configured, it exports log entries that match specific criteria defined by a filter to a designated log bucket.

Defense Evasion

Sophisticated techniques enable attackers to remain undetected within a compromised cloud environment. These methods could involve manipulating or evading logging mechanisms, which are crucial for security monitoring and incident response. Obscured activities can extend an attacker's presence, facilitate data exfiltration or inflict more harm before being discovered.

A wide array of security products including the following are fundamentally dependent on this log data to function:

• Security information and event management (SIEM)
• Security orchestration, automation and response (SOAR) platforms
• Specialized cloud security posture management (CSPM) tools

By disabling, altering or deleting these logs, an attacker can effectively hide themselves from these defensive systems, compromising the security integrity of the entire cloud environment. Attackers often use the following five techniques to accomplish this:

1. Stop logging
2. Delete the log storage destination
3. Delete the log router
4. Impair logging via attacker-controlled encryption key
5. Log poisoning

Technique 1: Stop Logging

The most direct method to suspend log flows is to disable the logging mechanism itself. A wide array of security products is fundamentally dependent on this log data to function. By disabling these logs, an attacker can effectively blind defensive systems, compromising the security integrity of the entire cloud environment.

• In AWS, an attacker with cloudtrail:StopLogging permissions can invoke the stop-logging API call for a specific trail. Once this API is executed, no further logs for that trail are written to the S3 bucket, creating an immediate visibility gap for organizations that created their own trails.
• In Google Cloud, the equivalent action is disabling the sink. An attacker with logging.sinks.update permissions can set the sink’s disabled field to true to stop logs from being written. The Google Cloud console message shown in Figure 1 reflects this change.

Technique 2: Delete Log Storage Destination

Typically, logs are stored in a cloud storage resource. An attacker who obtains permission to this resource could delete the cloud storage, preventing logs from being written. The following scenarios demonstrate this ability.

• In AWS, an attacker with s3:DeleteBucket permissions can use the delete-bucket API to delete the S3 bucket; this action also requires the s3:DeleteObject permission to empty the bucket. A few minutes after the action, the associated CloudTrail trail's configuration will indicate this deletion, as the notification in Figure 2 shows.

• In Google Cloud, an attacker with logging.buckets.delete permissions can delete the log bucket. In the case of a log bucket, the deletion is not immediate. Once the delete command is issued, the log bucket enters a DELETE_REQUESTED state, remaining in that state for seven days. After this period, the bucket is deleted.
  Google Cloud offers a mechanism to protect against this deletion by providing the ability to lock the logging bucket. Once a bucket is locked, its retention policy becomes permanent and irreversible, meaning the bucket cannot be deleted by any user until every log entry within it has fulfilled the specified retention period.

Technique 3: Delete Log Router

Another defense evasion tactic involves deleting the log routing resource – for example, an AWS trail or a Google Cloud sink. Once deleted, new logs will cease to be written to the designated destination. An attacker can delete a log router by using the delete-trail AWS API or the google.logging.v2.ConfigServiceV2.DeleteSink Google Cloud method.

Technique 4: Impair Logging via Attacker-Controlled Encryption Key

​​An attacker could potentially render cloud logs unreadable by modifying their encryption key. An attack flow using AWS could unfold as follows:

• When a trail is created in AWS, one of the configuration parameters is the key management service (KMS) key, which is used to encrypt the logs delivered by CloudTrail.
• An attacker could prevent legitimate access to this key by updating the trail to use an attacker-controlled KMS key and then removing access to the key.
• Consequently, logs will not be written to the bucket due to the inability to encrypt them with the unusable key.
• To perform this attack, an attacker first creates an external KMS key with the policy shown in Figure 3 to ensure that the key is accessible to CloudTrail.

• The attacker uses the update-trail API to modify the KMS key used to encrypt the logs in the S3 bucket.
• Next, the attacker removes CloudTrail's access to the key by either deleting the key or removing the Allow CloudTrail to access the key statement from the policy. Subsequently, CloudTrail will indicate that there is a configuration issue due to denied bucket access, even though the bucket is correctly configured.
• From this point on, logs will not be written to the bucket because the KMS key will be inaccessible. Figure 4 shows the message that is displayed after access is disabled.

Figure 5 shows the attack flow using AWS.

An attack flow using this technique in Google Cloud is as follows:

• In Google Cloud, a similar attack scenario can be simulated where a log bucket is already pre-configured with customer-managed encryption keys (CMEK). An attacker can then exploit this existing configuration by modifying the CMEK to reference an external key using the following command (the external CMEK must have encrypt/decrypt access granted to the KMS service account): gcloud logging buckets update BUCKET_NAME --location=LOCATION --cmek-kms-key-name FULL_KMS_KEY_NAME

Subsequently, the attacker can remove the permissions granted to the external key. At this point, the victim will be unable to read the logs, as the Google Cloud panel in Figure 6 shows.

Any attempts to revert the key will result in the error message “rekeying requires that the CMEK service account has decrypt access to the current CMEK key,” as Figure 7 shows.

Technique 5: Log Poisoning

Another defense evasion technique is the direct modification of logs – known as log poisoning. This is an effective technique when logs are pre-configured to be written to a cloud storage resource. In this case, the logs are stored in JavaScript Object Notation (JSON) format and can be modified by an attacker. If stored logs are deleted, added or modified, there is a high likelihood that Security Operations Center (SOC) personnel or analysts would inadvertently use these poisoned logs to conduct log analysis.

• In AWS, CloudTrail logs are delivered as objects to an S3 bucket. An attacker with s3:GetObject and s3:PutObject permissions on the bucket could download a log file, remove or alter specific events and then re-upload it, overwriting the original file. This breaks the chain of custody and invalidates the audit trail.
• Figure 8 shows the response to a query from a scenario where an attacker alters a log to avoid detection, and then the victim inspects the log using Amazon Athena.

• In Google Cloud, sinks route the logs to cloud storage. An attacker with storage.objects.get and storage.objects.create permissions can perform the same download and overwrite technique.

To mitigate the risk of log poisoning, AWS provides CloudTrail log file integrity validation. This feature provides the ability to cryptographically verify whether log files were modified after they were delivered by CloudTrail. This ability is enabled by default when using the AWS Console to create Trails, but not when using the API or command line interface (CLI).

Continuous Visibility

Upon gaining an initial foothold in a victim environment, an attacker with advanced capabilities would aim to establish long-term, passive visibility into the victim's cloud infrastructure. Instead of running noisy discovery commands that might trigger alerts – or if they lack proper permissions – an attacker can target the log routing mechanism to route logs to their own environment, resulting in real-time visibility. This enables attackers to perform continuous discovery and passively monitor all activity, from new VM deployments and IAM policy changes to sensitive data access. In this way, attackers can map the environment, identify high-value targets, and escalate privileges while potentially remaining invisible to the victim's security monitoring. The following techniques achieve continuous visibility:

1. Configure new log routing resource
2. Log redirection

Technique 1: Configure New Log Routing Resource

A direct method for achieving continuous visibility involves creating a new log routing resource – for example, an AWS trail or a Google Cloud sink. The attacker configures the newly created resource to direct logs to an external, attacker-controlled destination.

• In AWS, the attacker configures CloudTrail logging to their own S3 bucket by using the create-trail API and specifying their bucket in the --s3-bucket-name parameter.
• In Google Cloud, the attacker utilizes the logging.sinks.create API to set the DESTINATION parameter to the attacker’s intended resource.

Both of the above steps result in all logs being directed to the attacker's chosen destination.

For certain AWS accounts, the CreateTrail operation shows up in CloudTrail. If EventBridge is configured upon AWS account set-up, defenders can use EventBridge to alert on creation events. In this setup, subsequent describe calls of CloudTrail configuration will show the attacker's destination bucket. However, for organizations that use third parties or have not applied these configurations, attackers can carry out adversarial activities without being detected.

Technique 2: Log Redirection

Using this technique, the attacker alters the log routing destination to one within their own environment. This redirects logs to an attacker-controlled resource, enabling the attacker to obtain continuous discovery.

• In AWS, the attacker updates the --s3-bucket-name parameter when invoking the update-trail API. After modifying the destination bucket, all logs are directed to the attacker’s bucket, providing continuous discovery capabilities on the victim’s account.
• In Google Cloud, the attacker uses logging.sinks.update permissions to update the destination parameter, achieving the same ability to redirect logs.

Small enterprises that manage their own alert telemetry may notice that Trails have stopped working in this situation, but larger organizations may not be using AWS-native services that would allow them to detect this behavior.

Risk and Impact Assessment

Table 1 summarizes the evasion and visibility techniques, the likelihood that the activity is malicious, and the impact on logging services.

Table 1. Risk and impact assessment of evasion and visibility techniques.

Prevention and Awareness

The attack scenarios discussed all stem from modifications to logging service resources. Given the high value of cloud logging service resources, access should be restricted to highly privileged users to help prevent these scenarios. This measure reduces the likelihood of an attacker altering the configuration of such resources.

Amazon Web Services

For every AWS account there is an immutable 90 day CloudTrail Event History of all management events. This fallback ensures that these records cannot be deleted or circumvented. However, data and network events do not appear in this history.

In AWS, limit the update-trail API invocation to highly privileged users. Configure the bucket policy of the associated S3 bucket to prevent non-admin users from making configuration modifications. It's also crucial to ensure that only the CloudTrail service can write objects to these buckets.

Google Cloud

Google Cloud provides a similar safety mechanism to AWS, through its built-in log buckets. The _Required log bucket serves as an immutable repository for essential logs – such as Admin Activity and System Event audit logs – that cannot be disabled, modified or deleted. Alongside this, the _Default log bucket automatically captures a broader range of log entries for a shorter period of time. When creating logging storage for external integration purposes, these built-in buckets will not be relevant. As a result, those manually configured buckets may remain exposed to the specific attack techniques described in this article.

In Google Cloud, restrict the permissions for logging.sinks.update, and protect the destination resource.

Conclusion

Cloud logging services are fundamental for maintaining security posture and operational awareness, providing the definitive record of all activities within a cloud environment. The integrity of the logging infrastructure itself is a critical control, and for this reason has become a primary target for threat actors aiming to operate undetected.

The misuse of cloud logging services can have serious negative outcomes, enabling adversaries to cause blind spots for security teams, exfiltrate sensitive data in real-time, or methodically cover their tracks to evade forensics. By understanding the specific TTPs that threat actors use against these services, defenders can build more resilient detection and prevention strategies.

Palo Alto Networks Protection and Mitigation

Palo Alto Networks customers are better protected from the threats discussed above through the following products and services:

Cortex Cloud customers are better protected from the topics discussed within this article with cloud runtime security operations, through the collection, analysis, detection, alerting and prevention of malicious operations on cloud platform and SaaS application audit logs. Using behavioral and static alerting techniques on cloud logs during cloud operations runtime, the techniques discussed within the article can be identified and trigger alerts which provide early warning – and in some cases, prevention operations to prevent further compromise from these attacks.

The Unit 42 Cloud Security Assessment is an evaluation service that reviews cloud infrastructure to identify misconfigurations and security gaps.

If you think you may have been compromised or have an urgent matter, get in touch with the Unit 42 Incident Response team or call:

• North America: Toll Free: +1 (866) 486-4842 (866.4.UNIT42)
• UK: +44.20.3743.3660
• Europe and Middle East: +31.20.299.3130
• Asia: +65.6983.8730
• Japan: +81.50.1790.0200
• Australia: +61.2.4062.7950
• India: 000 800 050 45107

Palo Alto Networks has shared these findings with our fellow Cyber Threat Alliance (CTA) members. CTA members use this intelligence to rapidly deploy protections to their customers and to systematically disrupt malicious cyber actors. Learn more about the Cyber Threat Alliance.

Additional Resources

• CloudTrail concepts – AWS
• CloudTrail Event History – AWS
• General purpose buckets overview – AWS
• What is Amazon Athena? – AWS
• Validating CloudTrail log file integrity – AWS
• Aggregated sinks overview – Google Cloud
• Configure CMEK for Cloud Logging – Google Cloud
• Log buckets – Google Cloud
• Logging Bucket Locking – Google Cloud
• System-created Log Buckets – Google Cloud
• What is Pub/Sub? – Google Cloud
• Cloud Logging for Security and Beyond – Unit 42

Cortex XDR Alerts on Cloud Logging Services Misuse

Table 2 shows the Cortex alerts that are triggered by this activity in cloud logging services.

Table 2. Cortex alerts that indicate malicious activity in cloud logging services.