{
	"id": "f5e5c884-1e5d-49da-95f7-6b7641932389",
	"created_at": "2026-04-10T03:20:38.184111Z",
	"updated_at": "2026-04-10T03:22:18.101724Z",
	"deleted_at": null,
	"sha1_hash": "f711104ab074244981df0552e80792901989d91a",
	"title": "Detecting CI/CD Supply Chain Attacks with Canary Credentials | Tracebit",
	"llm_title": "",
	"authors": "",
	"file_creation_date": "0001-01-01T00:00:00Z",
	"file_modification_date": "0001-01-01T00:00:00Z",
	"file_size": 458649,
	"plain_text": "Detecting CI/CD Supply Chain Attacks with Canary Credentials |\r\nTracebit\r\nArchived: 2026-04-10 02:01:33 UTC\r\nIn recent weeks, a single threat actor - TeamPCP - compromised a chain of widely-used open source tools: Trivy,\r\nKICS, LiteLLM, and Telnyx. Each compromise fueled the next, with stolen credentials from one victim unlocking\r\naccess to the next target. We've just shipped a GitHub Actions integration in our free Community Edition in\r\nresponse to this incident.\r\nThis post looks at the campaign and explores the question: once you’ve pinned your actions and hardened your\r\nrunners, what actually detects credential exfiltration from a compromised CI/CD pipeline?\r\nTeamPCP campaign timeline\r\nCompromising a GitHub Action or PyPI package, even for just a few hours, can result in a large number of\r\nrepositories executing malicious code, as demonstrated by this quantitative analysis of the TeamPCP incidents.\r\nAlso Rami from Wiz has published a nice technical deep dive with further details about the campaign, including\r\nthe number of downloads of the malicious PyPI packages.\r\nMarch 19: The threat actor compromised multiple GitHub actions of Aqua Security’s Trivy scanner.\r\nMarch 22: TeamPCP used stolen credentials to compromise dozens of npm packages using a self-propagating worm, dubbed CanisterWorm. They also published malicious versions of Trivy on Docker\r\nHub.\r\nMarch 23: GitHub actions and OpenVSX extensions of Checkmarx’s KICS scanner were compromised\r\nwith a similar credential-stealing malware used in Trivy’s intrusion.\r\nMarch 24: LiteLLM PyPI package was compromised, resulting in two malicious versions published by the\r\nthreat actor.\r\nMarch 27: Telnyx PyPI package was compromised, again with two malicious versions published by the\r\nthreat actor.\r\nWhat does the malicious code look for?\r\nSecrets, of course.\r\nAll these attacks deploy malware designed for mass credential harvesting. Anything discovered in the\r\nenvironment - SSH keys, cloud credentials, git credentials, package manager tokens etc - are located and sent to\r\nthe threat actor. Each incident deployed essentially the same self-described \"TeamPCP Cloud Stealer\", but with\r\nprogressive refinements.\r\nThe compromised GitHub actions of Trivy and KICS (here’s an example of the malicious commit affecting trivy-action) perform two types of credential harvesting based on the environment running the code:\r\nhttps://tracebit.com/blog/detecting-cicd-supply-chain-attacks-with-canary-credentials\r\nPage 1 of 5\n\nIn GitHub-hosted Linux runners, the action runs the code contained in this memdump.py script, which\r\ndumps Runner.Worker process memory to steal GitHub secrets.\r\nIn all other environments, it scrapes the filesystem searching for cloud credentials and other secrets across\r\nmany sensitive file paths.\r\nWith the LiteLLM compromise, TeamPCP introduced a new lateral movement stage: after harvesting credentials,\r\nif a Kubernetes service account token is present, the malware reads all cluster secrets across all namespaces and\r\nattempts to create a privileged “alpine:latest” pod on every node in kube-system, mounting the host filesystem.\r\nAccording to vxunderground, the LiteLLM attack alone resulted in approximately 300GB of data exfiltrated from\r\n500,000 infected machines.\r\nDeceptive detections with CI/CD canaries\r\nMost guidance after these incidents focuses on preventive controls - pinning actions to specific commit SHAs,\r\nreviewing dependencies, hardening runner configurations. GitHub Actions themselves offer some quite\r\ncomprehensive guidance.\r\nPreventive controls are worth implementing in pipelines, though they can involve more effort than they first\r\nappear. To give one obvious example: a pinned action or dependency does not necessarily make for an idempotent\r\none if it fetches anything dynamically at runtime.\r\nWhat about detective controls? Runtime monitoring of CI/CD environments is desirable, but there’s a lot to\r\nconsider. In both Trivy and KICS attacks, the malicious code starts before the legitimate scanner services, and\r\neventually the workflows appear to complete normally. The LiteLLM case is more pernicious: the malware is\r\nimplanted by installing a .pth file that executes on every Python process startup across the entire environment - not\r\njust when LiteLLM is explicitly used. Aside from the details of payload injection: the Trivy attack demonstrated\r\nfallback exfiltration to trusted hosts (by creating repos using the GitHub API) - so network monitoring alone is\r\ninsufficient. Other major supply chain attacks using GitHub as exfiltration path include tj-actions, s1ngularity,\r\nShai-Hulud.\r\nCanary credentials take a different approach: fake credentials injected into your build environment. If a\r\ncompromised action or package exfiltrates credentials from your runner, canaries give you a clear signal - either as\r\nan initial detection, or as additional context when you discover you’ve been affected by a wider supply chain\r\nincident. This isn't just a theoretical idea: Grafana wrote last year about their successes using this technique.\r\nWhen we saw the TeamPCP campaign unfold, we built a GitHub Action for Tracebit’s free Community Edition\r\nthat deploys canary AWS credentials into CI/CD workflows.\r\nHow it works\r\nThe idea is simple:\r\n1. At the start of your workflow, the action calls the Tracebit API to receive a unique, short-lived set of canary\r\nAWS credentials\r\nhttps://tracebit.com/blog/detecting-cicd-supply-chain-attacks-with-canary-credentials\r\nPage 2 of 5\n\n2. The credentials are written to ~/.aws/credentials , exported as environment variables, and held in the\r\nrunner process's memory - covering every common exfiltration surface: credential files, environment\r\nvariable dumps, and process memory scraping.\r\n3. Tracebit monitors for any use of those credentials. If they are used, you get an alert with full context.\r\nBecause canary credentials sit alongside real credentials in the runner’s memory, environment variables, and\r\ncredential files, they would be exfiltrated by the same code that steals legitimate secrets. In the TeamPCP example,\r\nthis means any workflow using a compromised action or malicious package could trigger an alert with full context\r\nabout the affected repository, workflow, and run.\r\nDetection coverage against TeamPCP techniques\r\nIn cases where the threat actor compromised a GitHub action, the canaries cover both exfiltration surfaces:\r\nOn GitHub-hosted runners, the malicious code would have exfiltrated the canary AWS credentials from the\r\nRunner.Worker process memory.\r\nIn other runner environments, the canaries would instead be exfiltrated from ~/.aws/credentials .\r\nWe reproduced the Trivy attack by creating a workflow that includes the Tracebit Action and uses the malicious\r\nsetup-trivy action. Instead of encrypting and exfiltrating stolen credentials, we modified it to print them in the\r\naction output and call STS:GetCallerIdentity. The workflow runs in a GitHub-hosted runner, like this:\r\nname: \"Trivy scan\"\r\non:\r\n workflow_dispatch:\r\njobs:\r\n build:\r\n runs-on: ubuntu-latest\r\n steps:\r\n - uses: actions/checkout@v4\r\n - name: Configure credentials\r\n uses: tracebit-com/tracebit-community-action@517c410eae144100a1995cd720094c010995994d\r\n with:\r\n api-token: ${{ secrets.SECURITY_API_TOKEN }}\r\n profile: administrator\r\n profile-region: us-east-1\r\n async: true\r\n - name: Run Trivy\r\n uses: ./.github/actions/setup-trivy\r\nAfter running the workflow, the malicious code inside the compromised setup-trivy action retrieved the GitHub\r\nhttps://tracebit.com/blog/detecting-cicd-supply-chain-attacks-with-canary-credentials\r\nPage 3 of 5\n\nsecrets from the memory of the Runner.Worker process, including the AWS canary credentials.\r\nUsing the exfiltrated credentials triggers an alert on the Community Edition dashboard, which can lead to further\r\ninvestigation.\r\nThe Tracebit alert has full context about the affected workflow: repository, workflow, job, commit SHA, and run\r\nID. It also captures the attacker’s IP address and user-agent, along with the CloudTrail logs generated by the API\r\ncalls made with those credentials. Canary credentials have no permissions, so any API call except\r\nGetCallerIdentity fails - but the attempt itself is the signal.\r\nhttps://tracebit.com/blog/detecting-cicd-supply-chain-attacks-with-canary-credentials\r\nPage 4 of 5\n\nGrabbing secrets from runner memory is a known technique, used in other impactful supply chain attacks, such as\r\nthe incidents that affected tj-actions and reviewdog last year.\r\nIf the workflow instead uses a malicious version of a compromised package, such as LiteLLM, canary AWS\r\ncredentials stored in ~/.aws/credentials would have been exfiltrated. The LiteLLM malware not only\r\nharvested credentials, but actively called AWS APIs (ListSecrets, GetSecretValue, DescribeParameters), as\r\ndocumented in this article by StepSecurity. Tracebit canaries would have been used in these API calls, resulting in\r\nimmediate detection of the attack.\r\nConclusion\r\nSupply chain attacks can be self-sustaining. Trivy credentials unlocked CanisterWorm, which spread to 66+\r\npackages. LiteLLM used Trivy in their CI/CD security scan workflow, handing over PyPI publishing tokens.\r\nTelnyx tokens were likely harvested from LiteLLM-compromised environments. Each victim’s valid credentials\r\nbecome the attack surface for the next target.\r\nPinning and hardening reduce the attack surface, but what made TeamPCP’s campaign so effective was the\r\npropagation - stolen credentials from one victim unlocking access to the next. Canary credentials can’t prevent\r\nthat chain from starting, but they can tell you that your credentials have left the building. Teams should treat their\r\nbuild infrastructure with the same detection rigor as production.\r\nThe Tracebit Community Action is free and takes a few minutes to add to a workflow. We hope it’s useful - and\r\nwe’d welcome feedback on where to take it next.\r\nSource: https://tracebit.com/blog/detecting-cicd-supply-chain-attacks-with-canary-credentials\r\nhttps://tracebit.com/blog/detecting-cicd-supply-chain-attacks-with-canary-credentials\r\nPage 5 of 5",
	"extraction_quality": 1,
	"language": "EN",
	"sources": [
		"MISPGALAXY",
		"Malpedia"
	],
	"references": [
		"https://tracebit.com/blog/detecting-cicd-supply-chain-attacks-with-canary-credentials"
	],
	"report_names": [
		"detecting-cicd-supply-chain-attacks-with-canary-credentials"
	],
	"threat_actors": [],
	"ts_created_at": 1775791238,
	"ts_updated_at": 1775791338,
	"ts_creation_date": 0,
	"ts_modification_date": 0,
	"files": {
		"pdf": "https://archive.orkl.eu/f711104ab074244981df0552e80792901989d91a.pdf",
		"text": "https://archive.orkl.eu/f711104ab074244981df0552e80792901989d91a.txt",
		"img": "https://archive.orkl.eu/f711104ab074244981df0552e80792901989d91a.jpg"
	}
}