Update:

- Impacted versions (v1.82.7, v1.82.8) have been deleted from PyPI - All maintainer accounts have been changed - All keys for github, docker, circle ci, pip have been deleted

We are still scanning our project to see if there's any more gaps.

If you're a security expert and want to help, email me - krrish@berri.ai

> it seems your personal account is also compromised. I just checked for the github search here https://github.com/search?q=%22teampcp+owns%22

This must be super stressful for you, but I do want to note your "I'm sorry for this." It's really human.

It is so much better than, you know... "We regret any inconvenience and remain committed to recognising the importance of maintaining trust with our valued community and following the duration of the ongoing transient issue we will continue to drive alignment on a comprehensive remediation framework going forward."

Kudos to you. Stressful times, but I hope it helps to know that people are reading this appreciating the response.

>1. Looks like this originated from the trivvy used in our ci/cd

Were you not aware of this in the short time frame that it happened in? How come credentials were not rotated to mitigate the trivy compromise?

I put together a little script to search for and list installed litellm versions on my systems here: https://github.com/kinchahoy/uvpowered-tools/blob/main/inven...

It's very much not production grade. It might miss sneaky ways to install litellm, but it does a decent job of scanning all my conda, .venv, uv and system enviornments without invoking a python interpreter or touching anything scary. Let me know if it misses something that matters.

Obviously read it before running it etc.

Similar to delve, this guy has almost no work experience. You have to wonder if YC and the cult of extremely young founders is causing instability issues in society at large?

Kudos for this update.

Write a detailed postmortem, share it publicly, continue taking responsibility, and you will come out of this having earned an immense amount respect.

I just want to share an update

the developer has made a new github account and linked their new github account to hackernews and linked their hackernews about me to their github account to verify the github account being legitimate after my suggestion

Worth following this thread as they mention that: "I will be updating this thread, as we have more to share." https://github.com/BerriAI/litellm/issues/24518

we're using litellm via helm charts with tags main-v1.81.12-stable.2 and main-v1.80.8-stable.1 - assuming they're safe?

also how are we sure that docker images aren't affected?

You're making great software and I'm sorry this happened to you. Don't get discouraged, keep bringing the open source disruption!

Is it just in 1.82.8 or are previous versions impacted?

This is just one of many projects that was a victim of Trivy hack. There are millions of those projects and this issue will be exploited in next months if not years.

> - Krrish

Was your account completely compromised? (Judging from the commit made by TeamPCP on your accounts)

Are you in contacts with all the projects which use litellm downstream and if they are safe or not (I am assuming not)

I am unable to understand how it compromised your account itself from the exploit at trivvy being used in CI/CD as well.

[flagged]

The decision to block all downloads is pretty disruptive, especially for people on pinned known good versions. Its breaking a bunch of my systems that are all launched with `uv run`

There are hundreds of PRs fixing valid issues to your github repo seemingly in limbo for weeks. What is the maintainer state over there?

This is the security shortcuts of the past 50 years coming back to bite us. Software has historically been a world where we all just trust each other. I think that’s coming to an end very soon. We need sandboxing for sure, but it’s much bigger than that. Entire security models need to be rethought.

"Anymore" is right though. This should be a call to change the global mindset regarding dependencies. We have to realize that the "good ol days" are behind us in order to take action.

Otherwise people will naysay and detract from the cause. "It worked before" they will say. "Why don't we do it like before?"

DISA STIG already forbids use of the EPEL for Red Hat Enterprise Linux. Enterprise software install instructions are littered with commands to turn off gpgcheck and install rpm's from sourceforge. The times are changing and we need cryptographically verifiable guarantees of safety!

Would value your opinion on my project to isolate creds from the container:

https://github.com/calebfaruki/tightbeam https://github.com/calebfaruki/airlock

This is literally the thing I'm trying to protect against.

So... I'm working on an open source technology to make a literal virtual machine shippable i.e. freezing everything inside it, isolated due to vm/hypervisor for sandboxing, with support for containers too since it's a real linux vm.

The problems you mentioned resonated a lot with me and why I'm building it, any interest in working to solve that together?: https://github.com/smol-machines/smolvm

> Dev containers were never good enough, too clumsy and too little isolation.

I haven't kept up with the recent exploits, so a side question: Have any of the recent supply chain attacks or related exploits included any escapes from basic dev containers?

We need programming languages where every imported module is in its own sandbox by default.

> In such an environment the container would crash, we see the violations, delete it and dont' have to worry about it.

This is the interesting part. What kind of UI or other mechanisms would help here? There's no silver bullet for detecting and crashing on "something bad". The adversary can test against your sandbox as well.

That's no solution. If you can't trust and/or verify dependencies, and they are malicious, then you have bigger problems than what a sandbox will protect against. Even if it's sandboxed and your host machine is safe, you're presumably still going to use that malicious code in production.

Sandboxes yes, but who even added the dependency? Half the projects I see have requirements.txt written by Copilot. AI says "add litellm", dev clicks accept, nobody even pins versions.

Then we talk about containment like anyone actually looked at that dep list.

I agree in general, but how are you ever upgrading any of that? Could be a "sleeper compromise" that only activates sometime in the future. Open problem.

  > We just can't trust dependencies and dev setups.

Also, I typically only update dependencies when either an exploit is known in the current version or I need a feature present in a later version - and even then not to the absolute latest version if possible. I do this for all my projects under the many eyes principal. Finding exploits takes time, new updates are riskier than slightly-stale versions.

Though, if I'm filing a bug with a project, I do test and file against the latest version.

Containers prevent this kind of info stealing greatly, only explicitly provided creds would be leaked.

strongly agree. we keep giving away trust to other entities in order to make our jobs easier. trusting maintainers is still better than trusting a clanker but still risky. We need a sandboxed environment where we can build our software without having to worry about these unreliable factors.

On a personal note, I have been developing and talking to a clanker ( runs inside ) to get my day to day work done. I can have multiple instances of my project using worktrees, have them share some common dependencies and monitor all of them in one place. I plan to opensource this framework soon.

This stuff already exists - mobile phone sandboxed applications with intents (allow Pictures access, ...)

But mention that on HN and watch getting downvoted into oblivion: the war against general computation, walled gardens, locked down against device owners...

[dead]

Thanks for putting this together. I've been seeing the name TeamPCP pop up all over, but hadn't seen everything in one place.

This is interesting. How do you keep this up to date so quickly?

The same thing occurred on the trivy repo a few days ago. A GitHub discussion about the hack was closed and 700+ spam comments were posted.

I scrolled through and clicked a few profiles. While many might be spam accounts or low-activity accounts, some appeared to be actual GitHub users with a history of contributions.

I’m curious how so many accounts got compromised. Are those past hacks, or is this credential steeling hack very widespread?

Are the trivy and litellm hacks just 2 high profile repos out of a much more widespread “infect as many devs as possible, someone might control a valuable GitHub repository” hack? I’m concerned that this is only the start of many supply chain issues.

Edit: Looking through and several of the accounts have a recent commit "Update workflow configuration" where they are placing a credential stealer into a CI workflow. The commits are all back in february.

Reporting spam on GitHub requires you to click a link, specify the type of ticket, write a description of the problem, solve multiple CAPTCHAs of spinning animals, and press Submit. It's absurd.

i'm guessing it's accounts they have compromised with the stealer.

Or they're just bots. This repository has 40k+ stars somehow.

> Even still though, we can't really trust any open-source software any more that has third party dependencies, because the chains can be so complex and long it's impossible to vet everything.

This is why software written in Rust scares me. Almost all Rust programs have such deep dependency trees that you really can't vet them all. The Rust and Node ecosystems are the worst for this, but Python isn't much better. IMO it's language-specific package managers that end up causing this problem because they make it too easy to bring in dependencies. In languages like C or C++ that traditionally have used system package managers the cost of adding a dependency is high enough that you really avoid dependencies unless they're truly necessary.

Reflect in what way? The primary focus of that talk is that it’s possible to infect the binary of a compiler in a way that source analysis won’t reveal and the binary self replicates the vulnerability into other binaries it generates. Thankfully that particular problem was “solved” a while back [1] even if not yet implemented widely.

However, the broader idea of supply chain attacks remains challenging and AI doesn’t really matter in terms of how you should treat it. For example, the xz-utils back door in the build system to attack OpenSSH on many popular distros that patched it to depend on systemd predates AI and that’s just the attack we know about because it was caught. Maybe AI helps with scale of such attacks but I haven’t heard anyone propose any kind of solution that would actually improve reliability and robustness of everything.

[1] Fully Countering Trusting Trust through Diverse Double-Compiling https://arxiv.org/abs/1004.5534

Stop scaring me.

You're right though. There's been talks of a big global hack attack for a while now.

Nothing is safe anymore. Keep everything private airgapped is the only way forward. But most of our private and personal data is in the cloud, and we have no control over it or the backups that these companies keep.

While LLMs unlock the opportunity to self-host and self-create your infrastructure, it also unleashes the world of pain that is coming our way.

The only way to be safe is to constantly change internal APIs so that LLMs are useless at kernel code

If that would happen, The worry I would have is of all the sensitive Government servers from all over the world which might be then exploited and the amount of damage which can be caused silently by such a threat actor or something like AWS/GCP/these massive hyperscalers which are also used by the governments around the globe at times.

The possibilities within a good threat could be catastrophic if we assume so, and if we assume nation-states to be interested in sponsoring hacking attacks (which many nations already do) to attack enemy nations/gain leverage. We are looking at damage within Trillions at that point.

But I would assume that Linux might be safe for now, it might be the most looked at code and its definitely something safe.

LLVM might be a bit more interesting as it might go a little unnoticed but hopefully people who are working at LLVM are well funded/have enough funding to take a look at everything carefully to not have such a slip up.

You know that people can already write backdoored code, right?

Would a proper SOC2 audit have prevented this?

I've been through SOC2 certifications in a few jobs and I'm not sure it makes you bullet proof, although maybe there's something I'm missing?

Same experience with browser-use, it installs litellm as a dependency. Rebooted mac as nothing was responding; luckily only github and huggingface tokens were saved in .git-credentials and have invalidated them. This was inside a conda env, should I reinstall my os for any potential backdoors?

> i was lucky to see my processes before the system hard locked.

how do you do that? have Activity Monitor up at all times?

What is Harness?

Most his recent commits are small edits claiming responsibility on behalf of "teampcp", which was the group behind the recent Trivy compromise:

https://news.ycombinator.com/item?id=47475888

Or his company is trash and hes moved onto plain old theft.

Everyone’s (well, except Anthropic, they seem to have preserved a bit of taste) approach is the more data the better, so the databases of stolen content (erm, models) are memorizing crap.

This was a compromise of the library owners github acccounts apparently, so this is not a related scenario to dangerous code in the training data.

I assume most labs don't do anything to deal with this, and just hope that it gets trained out because better code should be better rewarded in theory?

By betting that it dilutes away and not worrying about it too much. Bit like dropping radioactive barrels into the deep ocean.

I am pretty sure that such measures aren't taken by AI companies, though I may be wrong.

The point of trusted publishing is supposed to be that the public can verifiably audit the exact source from which the published artifacts were generated. Breaking that chain via a private repo is a step backwards.

https://docs.npmjs.com/generating-provenance-statements

https://packaging.python.org/en/latest/specifications/index-...

this kind of compromise is why a lot of orgs have internal mirrors of repos or package sources so they can stay behind few versions to avoid latest and compromise. seen it with internal pip repos, apt repos etc.

some will even audit each package in there (kind crap job but it works fairly well as mitigation)

And easily bypassed by an attacker who knows about your static analysis tool who can iterate on their exploit until it no longer gets flagged.

Interesting tool, will definitely try - just curious, is there a tool (hexora checker) that ensures that hexora itself and its dependencies are not compromised ? And of course if there is one, I'll need another one for the hexora checker....

In the thread:

> It also seems that attacker is trying to stifle the discussion by spamming this with hundreds of comments. I recommend talking on hackernews if that might be the case.

Ton of compromised accounts spamming the GH thread to prevent any substantive conversation from being had.

These have been popping up on all the TeamPCP compromises lately

Those are all bots commenting, and now exposing themselves as such.

Bots to flood the discussion to prevent any actual conversation.

Wow, the postmortem for this is going to be brutal. I wonder just how many people/orgs have been affected.

I'm surprised to see nanobot uses LiteLLM: https://github.com/HKUDS/nanobot

LiteLLM wouldn't be my top choice, because it installs a lot of extra stuff. https://news.ycombinator.com/item?id=43646438 But it's quite popular.

Valid, but for all the crap that LangChain gets it at least has its own layer for upstream LLM provider calls, which means it isn't affected by this supply chain compromise (unless you're using the optional langchain-litellm package). DSPy uses LiteLLM as its primary way to call OpenAI, etc. and CrewAI imports it, too, but I believe it prefers the vendor libraries directly before it falls back to LiteLLM.

If you're talking about their proxy offering, I had this exact same issue and switched to Portkey. I just use their free plan and don't care about the logs (I log separately on my own). It's way faster (probably cause their code isn't garbage like the LiteLLM code - they had a 5K+ line Python file with all their important code in it the last time I checked).

I just found https://github.com/jasmedia/InferXgate which looks interesting although quite new and not supporting so many providers.

Bifrost is the only real alternative I'm aware of https://github.com/maximhq/bifrost

github.com/mozilla-ai/any-llm :)

pydantic-ai

[dead]

It'll only be disruptive to people who are improperly managing their credentials. Cattle not pets applies to credentials too.

Two URLs found in the exploit: https://checkmarx.zone/raw https://models.litellm.cloud/

Yeah, this was my team at FutureSearch that had the lucky experience of being first to hit this, before the malware was disclosed.

One thing not in that writeup is that very little action was needed for my engineer to get pwnd. uvx automatically pulled latest litellm (version unpinned) and built the environment. Then Cursor started up the local MCP server automatically on load.

(I am not a security expert either)

But, one of the arguments that I saw online from this was that when a security researcher finds a bug and reports it to the OSS project/Company they then fix the code silently and include it within the new version and after some time, they make the information public

So if you run infrequently updated versions, then you run a risk of allowing hackers access as well.

(An good example I can think of is OpenCode which had an issue which could allow RCE and the security researcher team asked Opencode secretly but no response came so after sometime of no response, they released the knowledge in public and Opencode quickly made a patch to fix that issue but if you were running the older code, you would've been vulnerable to RCE)

Not just as a gateway in a lot cases, but CrewAI and DSPy use it directly. DSPy uses it as its only way to call upstream LLM providers and CrewAI falls back to it if the OpenAI, Anthropic, etc. SDKs aren't available.

Do you feel as if people will update litellm without looking at this discussion/maybe having it be automatic which would then lead to loss of crypto wallets/ especially AI Api keys?

Now I am not worried about the Ai Api keys having much damage but I am thinking of one step further and I am not sure how many of these corporations follow privacy policy and so perhaps someone more experienced can tell me but wouldn't these applications keep logs for legal purposes and those logs can contain sensitive information, both of businesses but also, private individuals perhaps too?

It doesn't need to be subtle if the goal is just to drown out actual discussion.

That was massively more interesting, this is just a straight-up hack.

It's both. They got compromised by another supply chain attack on Trivy initially.

Just wrote up a quick article on how greywall[0] prevents this attack:

https://greyhaven.co/insights/how-greywall-prevents-every-st...

[0] https://greywall.io/

I can't tell which part of that is expensive unless many multiples of python are spawned at the same time. Are any of the payloads particularly large?

there's probably a more precise way, but if you're on uv:

  rg litellm  --iglob='*.lock'

these days, i just use a private llm. it's very quick and when i see the logs, it does a better job than me for this type of task.

no i don't let it connect to web...

And how would that work for single maintainer projects?

First line of defense is the git host and artifact host scrape the malware clean (in this case GitHub and Pypi).

Domains might get added to a list for things like 1.1.1.2 but as you can imagine that has much smaller coverage, not everyone uses something like this in their DNS infra.

Not sure that Trivy was doing that itself but zizmor is probably better than starting with an LLM :

https://github.com/zizmorcore/zizmor

Ironically, Trivy was the first known compromised package and its purpose is to scan container images to make sure they don't contain vulnerabilities. Kinda like the LLM in your scenario.

Oof. What exactly is supposed to be "lite" about this?

Yep, DSPy and CrewAI have direct dependencies on it. DSPy uses it as its primary library for calling upstream LLM providers and CrewAI falls back to it I believe if the OpenAI, Anthropic, etc. SDKs aren't available.

laughs smugly in vimscript

according to articles the first malicious version was published at roughly 8:30 UTC and the pypi repo taken down at ~11:25 UTC.

The 1.82.7 exploit was executed on import. The 1.82.8 exploit used a pth file which is run at start up (module discovery basically).

Move Slow and Fix Things.

LLMs recommend LiteLLM, so its popularity will only continue.

Not familiar with LangChain besides at a surface level - what makes it the worst software project known to man?

Compromised accounts. The malware targeted ~/.git-credentials.

Even just having an import statement for it is enough to trigger the malware in 1.82.8.

safetensors is just as vulnerable to this sort of exploit using a pth file since it's a Python package.

Attackers trying to stifle discussion, they did the same for trivy: https://github.com/aquasecurity/trivy/discussions/10420

Are they trying to slide stuff down? but it just bumps stuff up?

First thing I noticed too.

Potentially compromised?

In case you missed it, according to the OP, the previous point release (1.82.7) is also compromised.

Terrible code quality and terrible docs

> Remember that you are programmers and you can just program, you don't need a framework, you are already using the API of an LLM provider, don't put a hat on a hat, don't get killed for nothing.

Programming for different LLM APIs is a hassle, this library made it easy by making one single API you call, and in the backstage it handled all the different API calls you need for different LLM providers.

Comparing this project to is-odd seems very disingenuous to me. My understanding is this was the only way you could use llama.cpp with Claude Code for example, since llama.cpp doesn't support the Anthropic compatible endpoint and doing so yourself isn't anywhere near as trivial as your comparison. Happy to be corrected if I'm wrong.

Automated comments aren't allowed here. Please stop.

https://news.ycombinator.com/newsguidelines.html#generated

The xz compromise was not "at openSSH", and worked very differently.