Your AI Stack Was Backdoored for 3 Hours Yesterday. Were You Watching?

On Monday morning, while most teams were triaging Jira tickets and sipping cold brew, a threat actor called TeamPCP published two poisoned versions of litellm to PyPI. The packages contained a three-stage credential stealer, a Kubernetes worm, and a systemd backdoor. They stayed live for about three hours.

Three hours doesn't sound like much. But litellm pulls 3.4 million downloads per day. It's a transitive dependency of CrewAI, Browser-Use, DSPy, Mem0, Opik, Instructor, Guardrails, Agno, and Camel-AI. Wiz Research estimates it's present in 36% of cloud environments. If your team ran pip install or pip install --upgrade on any of those frameworks between roughly 09:00 and 13:30 UTC on March 24, you may have pulled the compromised package without ever touching litellm directly.

That's the thing about supply chain attacks. You don't have to be the target. You just have to be downstream.

The Kill Chain: Trivy to PyPI in Five Days

This wasn't a brute-forced password or a phishing email. TeamPCP ran a multi-hop campaign that weaponized trust between open-source projects.

March 19 — TeamPCP rewrote Git tags on Aqua Security's Trivy repository, pointing existing version tags to a malicious release containing credential-harvesting code. Any CI/CD pipeline pulling Trivy by tag (not by commit hash) silently picked up the poisoned version.

March 23 — The same group hit Checkmarx's KICS GitHub Action using the same tag-rewriting technique. They registered two C2 domains: checkmarx[.]zone and models[.]litellm[.]cloud.

March 24, ~08:30 UTC — LiteLLM's CI/CD pipeline ran the compromised Trivy action as part of its security scanning workflow. The irony is thick: the security scanner was the attack vector. The malicious Trivy code scraped environment variables from the GitHub Actions runner, including the PYPI_PUBLISH token. That token was enough.

March 24, 10:39 UTC — Version 1.82.7 appeared on PyPI.

March 24, 10:52 UTC — Version 1.82.8 followed thirteen minutes later, with an upgraded persistence mechanism.

March 24, ~11:25 UTC — PyPI quarantined both packages after automated detection flagged anomalies.

The attack window was roughly three hours. Enough.

Inside the Payload: Two Injection Techniques

The two versions used different delivery mechanisms, which suggests the attacker was iterating in real time.

Version 1.82.7 embedded a double-base64-encoded payload directly in litellm/proxy/proxy_server.py. When anything imports litellm.proxy — which happens when you start the litellm proxy, or when a dozen AI frameworks initialize their LLM backends — the payload drops to disk as p.py and executes.

Version 1.82.8 added a nastier trick: a file called litellm_init.pth (34,628 bytes) placed in site-packages/. Python's .pth file mechanism runs arbitrary code during interpreter initialization. Not during import. During startup. Every Python process on that system would execute the payload, including unrelated scripts, test runners, and even pip install of other packages.

Both payloads used double base64 encoding to dodge static analysis and ran via subprocess to avoid leaving traces in the Python stack.

The Three-Stage Payload

Once triggered, the malware executed three stages:

Stage 1: Credential Harvesting. The payload swept the entire filesystem and runtime environment for anything valuable:

• Environment variables (API keys, tokens, connection strings)
• SSH keys (~/.ssh/)
• AWS credentials (~/.aws/credentials, IMDSv2 metadata endpoint)
• GCP and Azure service account keys
• Kubernetes configs and service account tokens
• Docker config.json (registry credentials)
• Git credentials and shell history
• Database passwords from common config locations
• SSL/TLS private keys
• Cryptocurrency wallets (BTC, ETH, SOL, XMR)
• CI/CD secrets from runner environments

Stage 2: Encryption and Exfiltration. The harvested data was encrypted using AES-256-CBC with a randomly generated 32-byte session key. The session key itself was encrypted with a hardcoded RSA-4096 public key embedded in the payload. The bundle was packaged as tpcp.tar.gz and posted to https://models.litellm[.]cloud/. Only TeamPCP holds the RSA private key, meaning only they can decrypt the harvest.

Stage 3: Persistence and Lateral Movement. The payload wrote a backdoor to ~/.config/sysmon/sysmon.py and created a systemd user service (sysmon.service) that polls https://checkmarx[.]zone/raw every five minutes for new commands. If Kubernetes tokens were found on the system, the malware read all secrets across all namespaces and deployed privileged pods named node-setup-{node} in kube-system, each mounting the host filesystem at root. A single compromised developer laptop with kubectl access could cascade into full cluster compromise.

Who Was Actually Affected?

The blast radius has two layers.

Direct exposure: Anyone who installed or upgraded litellm to version 1.82.7 or 1.82.8 during the three-hour window. This includes automated CI/CD pipelines that use unpinned dependency versions.

Transitive exposure: Anyone who installed or upgraded any package that depends on litellm during that window, including CrewAI, Browser-Use, DSPy, Mem0, Opik, Instructor, Guardrails, Agno, and Camel-AI. If your requirements.txt says crewai>=0.100 without pinning litellm separately, pip would have resolved to the latest litellm, which for three hours was the poisoned version.

Users of LiteLLM's official Docker image (ghcr.io/berriai/litellm) were not affected because the image pins its dependencies.

How to Check If You're Compromised

Run these checks right now. Not after lunch.

Check the installed version:

pip show litellm 2>/dev/null | grep -i version

If it says 1.82.7 or 1.82.8, that environment is compromised.

Search for the .pth persistence file:

find "$HOME" /usr -name "litellm_init.pth" 2>/dev/null

Search for the systemd backdoor:

ls -la ~/.config/sysmon/sysmon.py 2>/dev/null
systemctl --user status sysmon.service 2>/dev/null

Check for suspicious Kubernetes pods:

kubectl get pods -n kube-system | grep "node-setup-"

Check network logs for C2 traffic: Look for outbound connections to models.litellm[.]cloud or checkmarx[.]zone in your firewall, proxy, or DNS logs.

Check for temp artifacts:

ls -la /tmp/.pg_state /tmp/pglog /tmp/tpcp.tar.gz /tmp/session.key /tmp/payload.enc 2>/dev/null

If You're Compromised: Rotate Everything

This is not optional, and partial rotation is exactly how TeamPCP maintained access after the initial Trivy breach. GitGuardian's post-mortem on the Trivy incident concluded that the attack persisted because the rotation process did not fully sever access.

Rotate:

• All SSH keys
• All cloud provider credentials (AWS access keys, GCP service accounts, Azure service principals)
• All API keys and tokens (OpenAI, Anthropic, Stripe, Twilio, etc.)
• All database passwords
• All Kubernetes service account tokens
• All CI/CD secrets and deploy keys
• All Docker registry credentials
• Any cryptocurrency wallet private keys that were on the system

Then remove the persistence mechanisms:

rm -f ~/.config/sysmon/sysmon.py
systemctl --user disable sysmon.service 2>/dev/null
rm -f ~/.config/systemd/user/sysmon.service
systemctl --user daemon-reload

Pin litellm to a safe version:

litellm<=1.82.6

The Deeper Problem

The technical details of this attack are worth studying, but the real story is structural.

LiteLLM's CI/CD pipeline pulled a security scanning tool by Git tag instead of by pinned commit hash. That unpinned reference was the single link TeamPCP needed to bridge from "compromised open-source security tool" to "compromised AI infrastructure package with 95 million monthly downloads."

This pattern is everywhere. Most GitHub Actions workflows use @v3 or @latest tags. Most Python projects pull CI/CD tools without pinning. The assumption is that the tool's maintainers will protect the tag. But tags are mutable in Git. They can be moved, rewritten, or deleted. A tag is a promise, not a guarantee.

The fix isn't complicated. Pin your CI/CD dependencies to full commit SHAs. Scope your PyPI tokens to individual packages. Use short-lived credentials where possible. Treat your publishing tokens the same way you treat your production database password.

None of this is new advice. But TeamPCP just demonstrated what happens when it's ignored.

Timeline Summary

Indicators of Compromise

What Aeroxis Recommends

For organizations running AI workloads in production:

1. Audit your dependency trees now. Not just direct dependencies. Use pip-audit or safety to scan for known compromised versions across all transitive deps.
2. Pin everything in CI/CD. GitHub Actions by commit SHA. Python packages by exact version. Docker images by digest. Tags are mutable. Hashes are not.
3. Scope your publishing tokens. PyPI supports per-project API tokens. Use them. A token scoped to one package can't publish to another.
4. Monitor for .pth files. These are a known persistence vector that most endpoint detection tools don't flag. Add find /usr -name "*.pth" -newer /usr -mtime -7 to your security sweeps.
5. Treat AI infrastructure as critical infrastructure. LiteLLM sits between your application code and every LLM API call. A compromise here means every prompt, every response, and every API key flows through attacker-controlled code. This is not a dev tool. It's part of your production attack surface.

If your team needs help assessing exposure or hardening your AI pipeline security posture, reach out to us.

Sources: LiteLLM Official Security Update, Snyk Analysis, Wiz Research, Sonatype Technical Breakdown, GitGuardian Post-Mortem