{
	"id": "04458ee9-f143-4c52-b95d-1c4541984dc7",
	"created_at": "2026-04-06T01:29:55.341082Z",
	"updated_at": "2026-04-10T03:21:08.323781Z",
	"deleted_at": null,
	"sha1_hash": "abf66c09a92400ca109b76c8be572fdefe514e43",
	"title": "Bypassing CloudTrail in AWS Service Catalog, and Other Logging Research | Datadog Security Labs",
	"llm_title": "",
	"authors": "",
	"file_creation_date": "0001-01-01T00:00:00Z",
	"file_modification_date": "0001-01-01T00:00:00Z",
	"file_size": 1874572,
	"plain_text": "Bypassing CloudTrail in AWS Service Catalog, and Other Logging\r\nResearch | Datadog Security Labs\r\nBy Nick Frichette\r\nPublished: 2023-03-20 · Archived: 2026-04-06 00:35:20 UTC\r\nCloudTrail is a crucial AWS service that provides a record of API calls and other important activities in AWS\r\nenvironments. Teams can use this information for auditing purposes and to identify potential security incidents. If\r\nan attacker who has gained a foothold in an environment can perform actions without CloudTrail logging them,\r\nthey’ll be able to conceal their activities and become functionally invisible to the victim.\r\nBypassing CloudTrail for AWS services is an active field of research. In previous publications, we demonstrated\r\nhow we were able to enumerate IAM permissions without logging to CloudTrail for a number of services and\r\nperform a subset of IAM actions while bypassing CloudTrail, allowing an adversary to carry out reconnaissance\r\nactivities undetected.\r\nIn this blog post, we’ll share some of our latest research into bypassing CloudTrail. We’ll cover a method that\r\nallowed CloudTrail bypass with both read and write API actions for the Service Catalog service. This now-fixed\r\nvulnerability is noteworthy, because it was the first publicly known CloudTrail bypass that could permit an\r\nattacker to alter an AWS environment. In addition, we’ll discuss another bug we identified in CloudTrail logging\r\nfor AWS Control Tower.\r\nLike with all CloudTrail bypasses, this would be used as a post-exploitation technique. An adversary would need\r\nto gain initial access to an AWS account through some means and the compromised identity would need to have\r\nsufficient privileges to interact with Service Catalog. It is at this point that an adversary could have used this\r\nvulnerability to bypass CloudTrail logging, for both read and write API actions. CloudTrail bypasses are important\r\nbecause they allow an adversary to avoid detection during activities that might otherwise be suspicious.\r\nWe shared this information with AWS, who confirmed our findings and have since remediated both issues.\r\nDisclosure timeline\r\nJanuary 30, 2023: Datadog reports both issues to AWS.\r\nJanuary 30, 2023: AWS responds that they received the report.\r\nJanuary 31, 2023: Datadog sends two proof-of-concept scripts.\r\nFebruary 7, 2023: AWS confirms that fixes are in development for both issues.\r\nFebruary 7, 2023: AWS deploys fix to Service Catalog.\r\nFebruary 13, 2023: AWS deploys fix to Control Tower.\r\nMarch 20, 2023: Datadog releases public disclosure.\r\nCloudTrail bypass in AWS Service Catalog\r\nhttps://securitylabs.datadoghq.com/articles/bypass-cloudtrail-aws-service-catalog-and-other/\r\nPage 1 of 9\n\nAs cloud security researchers, we are constantly exploring attack paths in cloud environments, trying to\r\nunderstand how real-world adversaries could compromise or exploit cloud resources. This research is typically\r\nfocused on the customer side of the shared responsibility model—however, we sometimes identify opportunities\r\nto cross into the cloud service provider’s domain.\r\nWhile working on a research project, we noticed an interesting entry in the Content-Security-Policy meta tag for a\r\npage in the AWS Console.\r\n[…snip…]\r\nhttps://controltower.us-east-1.amazonaws.com\r\nhttps://sts.amazonaws.com/\r\nhttps://cloudformation.us-east-1.amazonaws.com/\r\nhttps://aws242-servicecatalog-beta.us-east-1.amazonaws.com\r\nhttps://widgets.marketplace.us-west-2.amazonaws.com\r\nhttps://clientlogger.marketplace.aws.a2z.com\r\n[…snip…]\r\nWhat caught our eye here was the reference to aws242-servicecatalog-beta.us-east-1.amazonaws.com . It looks\r\nsimilar to the normal API endpoint for AWS Service Catalog, servicecatalog.us-east-1.amazonaws.com , but it\r\nseemed strange that it had the aws242 prefix. Because Service Catalog is such a critical AWS service—it allows\r\norganizations to create and maintain catalogs of approved AWS resources—a suspicious endpoint for this service\r\ncould introduce additional attack surface, so we decided to dig deeper.\r\nWhen using the AWS Console, a number of JavaScript files are loaded in the user’s browser. These files contain\r\nthe logic for interacting with the various AWS APIs. After digging around, we eventually found the service\r\ndefinition information for AWS242ServiceCatalogService .\r\n{\r\n \"version\": \"2.0\",\r\n \"metadata\": {\r\n \"apiVersion\": \"2015-12-10\",\r\n \"endpointPrefix\": \"servicecatalog\",\r\n \"jsonVersion\": \"1.1\",\r\n \"protocol\": \"json\",\r\n \"serviceFullName\": \"AWS Service Catalog\",\r\n \"serviceId\": \"Service Catalog\",\r\n \"signatureVersion\": \"v4\",\r\n \"targetPrefix\": \"AWS242ServiceCatalogService\",\r\n \"uid\": \"servicecatalog-2015-12-10\"\r\n },\r\n[…snip…]\r\nArmed with the service definition information and the suspicious endpoint, we knew how to speak to the service\r\nand what endpoint to interact with. We used valid IAM credentials with the necessary privileges to sign our\r\nrequests using the SIGv4 signing protocol, and the results we got surprised us.\r\nhttps://securitylabs.datadoghq.com/articles/bypass-cloudtrail-aws-service-catalog-and-other/\r\nPage 2 of 9\n\nInteracting with the beta endpoint\r\nWhen we interacted with the beta endpoint, none of our requests failed or received an error message—however,\r\nthey did not return results that we would normally expect. For example, our attempts to call\r\nservicecatalog:ListPortfolios returned no results, even when we intentionally created a portfolio in the\r\naccount.\r\nWe found, however, that if we used the beta endpoint to create a portfolio, it would show up when we called\r\nservicecatalog:ListPortfolios with the beta endpoint. We hypothesize that the beta endpoint is in a\r\ndifferent namespace than the normal commercial environment, so resources created in this partition are not\r\naccessible by normal means. In terms of security impact, we could not identify any way this endpoint would be\r\nuseful to an attacker, since it seemed to be siloed from the normal production environment.\r\nHaving reached what appeared to be a dead end and looking for other opportunities, we checked if there was a\r\nDNS entry for a gamma endpoint. Through their research on the AWS console, our researchers knew that gamma\r\nis a relatively common suffix for various AWS API endpoints (along with alpha and beta ) and has been\r\nnoticed by the security research community on multiple occasions. Based on their naming conventions, we\r\npresume these endpoints are used to test new features and services. Sure enough, the aws242-servicecatalog-gamma.us-east-1.amazonaws.com domain resolved to an IP address.\r\nInteracting with the gamma endpoint\r\nThe first thing we noticed was that we got normal results when interacting with aws242-servicecatalog-gamma.us-east-1.amazonaws.com . We could, for example, perform servicecatalog:ListPortfolios , and the\r\nresult would show all portfolios we could normally see in the account from the console.\r\nWe could now interact with Service Catalog normally using the gamma endpoint and get results for our API calls.\r\nThe new question became, what does this activity look like in CloudTrail? From prior research on the subject, we\r\nhttps://securitylabs.datadoghq.com/articles/bypass-cloudtrail-aws-service-catalog-and-other/\r\nPage 3 of 9\n\nknew that when the AWS Console interacts with private AWS APIs, there is a potential to bypass CloudTrail.\r\nHere is what it looks like in CloudTrail when you invoke the servicecatalog:ListPortfolios call using the\r\nAWS CLI and the normal endpoint.\r\nHowever, by comparison, after waiting we saw that no traffic sent to the gamma endpoint would show up in\r\nCloudTrail. We had just found a new way to bypass CloudTrail.\r\nWrite actions\r\nThe next order of business was to try to perform write actions, which are API calls that create or alter resources.\r\nDuring this step, we encountered an interesting error message.\r\nHTTP/1.1 400 Bad Request\r\n[…snip…]\r\n{\r\n \"__type\":\"AccessDeniedException\",\r\n \"Message\":\"Access Denied - The caller doesn't have permission to call this API.\"\r\n}\r\nThis was discouraging—however, to our surprise, the request that returned this error message appeared to\r\ncomplete successfully. For example, when calling servicecatalog:CreatePortfolio , we received the same error\r\nmessage, but the portfolio was nevertheless created.\r\n[...snip…]\r\n{\r\n \"ProviderName\":\"my-provider\",\r\n \"DisplayName\":\"This was made while bypassing CloudTrail\",\r\n \"IdempotencyToken\":\"blahblahblah\"\r\n}\r\nhttps://securitylabs.datadoghq.com/articles/bypass-cloudtrail-aws-service-catalog-and-other/\r\nPage 4 of 9\n\nOther write actions such as servicecatalog:UpdatePortfolio also returned a similar result: We would get an\r\nerror as the response, but the action would complete normally.\r\nFor a practical example of how an attacker could abuse this flaw, imagine a company using AWS Service Catalog\r\nhas a portfolio called “Production Portfolio” that they use to share CloudFormation stacks of production-ready\r\ninfrastructure. An adversary who has compromised access to a role or user with sufficient permissions to modify\r\nthe portfolio and the products within it could do so by using the gamma endpoint—without leaving a CloudTrail\r\nlog of their actions.\r\nMissing CloudTrail logging in Control Tower\r\nAWS Control Tower is a service that makes it easy to manage common controls across a multi-account AWS\r\norganization by orchestrating the capabilities of AWS Service Catalog, IAM Identity Center, and other services.\r\nWhile using Control Tower from the AWS Console, the application will make requests to an additional service\r\ncalled AWSBlackbeardService . This service handles API requests for most of the console functionality of Control\r\nTower.\r\n{\r\n \"version\": \"2.0\",\r\n \"metadata\": {\r\n \"apiVersion\": \"2018-11-26\",\r\n \"endpointPrefix\": \"controltower\",\r\nhttps://securitylabs.datadoghq.com/articles/bypass-cloudtrail-aws-service-catalog-and-other/\r\nPage 5 of 9\n\n\"jsonVersion\": \"1.1\",\r\n \"protocol\": \"json\",\r\n \"serviceFullName\": \"AWS Control Tower\",\r\n \"serviceId\": \"ControlTower\",\r\n \"signatureVersion\": \"v4\",\r\n \"signingName\": \"controltower\",\r\n \"targetPrefix\": \"AWSBlackbeardService\",\r\n \"uid\": \"controltower-2018-11-26\"\r\n },\r\n[…snip…]\r\nWhile working with AWSBlackbeardService during our research, it quickly became clear that it behaved\r\ndifferently from other AWS services. Normally, services that log events to CloudTrail log both successes and\r\nfailures. Any action configured to log to CloudTrail will produce a log there, regardless of whether it succeeds or\r\nfails.\r\nWith AWSBlackbeardService , however, failures due to insufficient privileges were not logged to CloudTrail. We\r\ncan demonstrate this by making calls to the service with two different user accounts: admin and no-perm (for no-permissions). With both of them, we invoke controltower:ListManagedOrganizationalUnits .\r\nHere is the admin user’s response:\r\nhttps://securitylabs.datadoghq.com/articles/bypass-cloudtrail-aws-service-catalog-and-other/\r\nPage 6 of 9\n\nAnd here is the resulting CloudTrail event:\r\nHere is the no-perm user's invocation:\r\nAnd here, we see that the result does not appear in CloudTrail:\r\nKnowing when an API action fails due to a lack of privilege can sometimes be as important as knowing when an\r\naction succeeds. Attackers who have compromised IAM credentials will often use brute-force techniques to\r\ndetermine what permissions the compromised entity has—and in this scenario, because there is no log in\r\nCloudTrail, they could avoid detection when enumerating Control Tower permissions.\r\nYou may be wondering, is it possible to use a gamma endpoint with AWSBlackbeardService to bypass\r\nCloudTrail logging, similar to what we did with Service Catalog? As it turns out, it’s not, but the reason sheds\r\nfurther light on the differences between alpha / beta / gamma and the normal endpoints. gamma.us-east-1.blackbeard.aws.a2z.com resolves to an IP address (the normal endpoint is prod.us-east-https://securitylabs.datadoghq.com/articles/bypass-cloudtrail-aws-service-catalog-and-other/\r\nPage 7 of 9\n\n1.blackbeard.aws.a2z.com ). When you made an API call to the gamma endpoint with AWSBlackbeardService ,\r\nyou would get the following error:\r\n{\r\n \"__type\":\"AccessDeniedException\",\r\n \"Message\":\"Failed to assume role arn:aws:iam::123456789012:role/service-role/AWSControlTowerAdmin\"\r\n}\r\nNormally, when making requests through the Blackbeard service, Control Tower will assume the\r\nAWSControlTowerAdmin role in the account. When you execute this request on the normal production endpoint,\r\nthe service assuming the role is controltower.amazonaws.com . However, when you make this request via\r\ngamma , the AWSInternal identity assumes the role. This means the gamma endpoint does not assume the role\r\nwith the same identity that the production service does.\r\nThis can be validated by setting the trust policy on the AWSControlTowerAdmin role to allow * to assume the\r\nrole. (Never do this in your AWS account, as it will allow any AWS principal to assume the role.)\r\n{\r\n \"eventVersion\": \"1.08\",\r\n \"userIdentity\": {\r\n \"type\": \"AWSService\",\r\n \"invokedBy\": \"AWS Internal\"\r\n },\r\n \"eventTime\": \"2023-02-13T21:52:45Z\",\r\n \"eventSource\": \"sts.amazonaws.com\",\r\n \"eventName\": \"AssumeRole\",\r\n \"awsRegion\": \"us-east-1\",\r\n \"sourceIPAddress\": \"AWS Internal\",\r\n \"userAgent\": \"AWS Internal\",\r\n \"requestParameters\": {\r\n \"roleArn\": \"arn:aws:iam::123456789012:role/service-role/AWSControlTowerAdmin\",\r\n \"roleSessionName\": \"GetGuardrailComplianceStatus\"\r\n },\r\n[…snip…]\r\nIn the excerpt of the above CloudTrail log above, we can see that “AWS Internal” assumes the role, rather than the\r\nControl Tower service. When this happens, we get a different error message:\r\n{\r\n \"__type\":\"ResourceNotFoundException\",\r\n \"Message\":\"Unable to get details for account=123456789012 in namespace=123456789012\"\r\n}\r\nhttps://securitylabs.datadoghq.com/articles/bypass-cloudtrail-aws-service-catalog-and-other/\r\nPage 8 of 9\n\nThis further validates our theory that gamma and beta endpoints can be (though are not always) in a separate\r\npartition or namespace, where their resources are different from the normal production environment used by\r\ncustomers.\r\nFixes\r\nIn response to these findings, AWS has implemented a fix for each service. For Service Catalog, attempts to make\r\nrequests to the gamma endpoint (even with sufficient privileges) now return the following error:\r\nHTTP/1.1 400 Bad Request\r\n[...snip…]\r\n{\r\n \"__type\":\"AccessDeniedException\",\r\n \"Message\":\"Access Denied - The caller doesn't have permission to call this API.\"\r\n}\r\nFor Control Tower—and, by extension, the AWSBlackbeardService —failed requests will now appropriately log\r\nto CloudTrail.\r\nAWS's Response\r\nIn addition, AWS has provided the following statement in regard to this disclosure:\r\n\"AWS is aware of the issues reported by Datadog and has fully addressed them. No customer action is required.\r\nWe have conducted a review of the services' logs with respect to the reported issues, and did not discover any\r\nevidence of unauthorized activity related to either Service Catalog or Control Tower. We would like to thank\r\nDatadog for the coordinated disclosure of their security research and collaboration with us on this report.\"\r\nAcknowledgements\r\nWe’d like to give major thanks to Alexis Fahrney and the AWS Security Outreach Team for being pleasant to work\r\nwith, as always.\r\nSource: https://securitylabs.datadoghq.com/articles/bypass-cloudtrail-aws-service-catalog-and-other/\r\nhttps://securitylabs.datadoghq.com/articles/bypass-cloudtrail-aws-service-catalog-and-other/\r\nPage 9 of 9",
	"extraction_quality": 1,
	"language": "EN",
	"sources": [
		"MITRE"
	],
	"references": [
		"https://securitylabs.datadoghq.com/articles/bypass-cloudtrail-aws-service-catalog-and-other/"
	],
	"report_names": [
		"bypass-cloudtrail-aws-service-catalog-and-other"
	],
	"threat_actors": [],
	"ts_created_at": 1775438995,
	"ts_updated_at": 1775791268,
	"ts_creation_date": 0,
	"ts_modification_date": 0,
	"files": {
		"pdf": "https://archive.orkl.eu/abf66c09a92400ca109b76c8be572fdefe514e43.pdf",
		"text": "https://archive.orkl.eu/abf66c09a92400ca109b76c8be572fdefe514e43.txt",
		"img": "https://archive.orkl.eu/abf66c09a92400ca109b76c8be572fdefe514e43.jpg"
	}
}