CrowdSec Source Code Leak

160 pointsposted 3 days ago
by eccgecko
(crowdsec.net)

59 Comments

mewse-hn

3 days ago

Funny to see this after I spent the morning troubleshooting and fixing my crowdsec install on my debian 13 vps. Apparently they stopped supplying a community blocklist to my machine because I'm running the old debian packaged version instead of directly from them (http 500). I had a LLM build a blocklist from publicly available sources rather than tie myself more tightly to their SaaS platform.

getcrunk

3 days ago

Wait … that’s lame. If it’s in Debian repos I expect it to work with full functionality (up to the version number)

I can understand the myriad reasons for why a company would be inconvenienced by that.

But on the other hand, it seems like a reason for me to never use the vendor

idiotsecant

3 days ago

I feel like a massive blocklist is simultaneously exactly the kind of thing an LLM is likely to hallucinate and exactly the kind of thing you don't want to rely on an LLM for.

zamadatix

3 days ago

If they had it build it from public sources then I assume it was combined via tool usage rather than direct generation. Beyond the hallucination risk, the latter would just be a waste of tokens & time.

antonvs

3 days ago

Modern agent harnesses rely heavily on tools, as well as writing code, to avoid hallucinations affecting deterministic values.

shakna

3 days ago

Which means it'd be a lot cheaper, to use tools, and public sources, and just automate it the traditional way...

KetoManx64

2 days ago

I'd rather spend the $0.50 in API credits than waste a couple hours of my time. How much is your time worth?

shakna

2 days ago

Well, it'd cost me about 10 minutes, and it wouldn't delete half my data. So... How much is your infrastructure worth?

zamadatix

2 days ago

Maybe you value the few cents (the .50c above seems more like a phrase than an actual value, that much actual cost would be the same as nearly a full days worth of a ChatGPT Plus subscription to combine some lists) over the 10 minutes but I'm not sure I follow why you'd give it access or control of half your data to pull data from public sources on the internet and give you a combined blocklist out of it?

shakna

2 days ago

Our current models and harnesses have not been doing a fantastic job, of keeping things confined... The harness defaults to approving random commands, meaning if the model goes for the data, it tends to find the data. And a delete all before replacing it is fine, right?

zamadatix

5 hours ago

The sources are public lists on the internet and the result is a combined file you click to download, none of this requires the agent even run on your PC if you don't trust your sandboxes - you can hope the agent deletes OpenAI's cloud instead (even though that's never happened with a billion weekly users).

KetoManx64

2 days ago

My infrastructure is build from a git ansible repo that I tear down and rebuild regularly, so $0.

shakna

2 days ago

Did you take that time into your account of costs as well?

You're already using automation tools. Seems like you just don't... Want to learn... The tool that would save you hours of frustration, tomorrow or the day after.

KetoManx64

2 days ago

It won't. And there is no reason to learn the intricacies of every little script that is written because there is not enough time in the day for that. If the script fails you feed claude the logs and tell it to fix it while working on the things that I actually enjoy working on and matter in the long run.

shakna

2 days ago

Well, as it wouldn't be a script, congrats on already losing the skills you would need, to assess whether or not what you set out to do is actually accomplished.

And as we're discussing a security tool... Love to see how you judge it as something that doesn't matter in your stack. Incidentally, mind offering me your IP range?

antonvs

2 days ago

I take it you’ve never used a modern agent.

tecleandor

3 days ago

Ah, I have to check that, thanks!

tacomagick

3 days ago

Ah, I just learned this, thanks for the heads up.

6thbit

3 days ago

  > the Tanstack compromise is very likely to have been the leak vector
  ....appears to have been backdoored to extract an API key with authorization to read the private codebase.
  ...
  >  immediately rotated all required tokens & credentials to prevent further incidents.
Rotating the API key doesn't quite put them in a position to "prevent further incidents" does it? The next PyPI/npm supply chain issue will just get the new key?

I suppose whatever they use that key for should be reviewed and re-scoped if possible?

Does github let you restrict where you can originate requests using a given API key? or are we just not there yet?

sandeepkd

3 days ago

On the funny side, reading the website tagline, apparently they claim to know who is attacking you, they just happen to miss out on who attacked them.

Turns out they are not really a security company, just an aggregator of bad IPs. Ideally this kind of aggregator problem is best suited for a trusted not-for-profit company where providing the data needs some level of credibility and querying the data costs you nominal fee to keep the setup floating.

Meneth

3 days ago

I suppose CrowdSec isn't used to analyze GitHub's traffic, so they wouldn't have much info to go on.

If they had self-hosted their own repos, they might have had more luck.

sandeepkd

3 days ago

It wasn't the Github that was compromised, it was the access to their private repository that was compromised so somewhere down the line the security best practices are in question for sure. Self hosted repos available on public internet would have met the same fate, may be worse, given github does provides some level of security.

Even regarding the blast radius, I do not really believe any company is honest about it. They do not have tools to verify it, if the user information was accessed with leaked token or real token. The thing that works in their favor is that no one else can verify it either which absolves them from any responsibility. Any platform engineer knows that your CICD system has the keys to the kingdom.

strictnein

3 days ago

The article you're commenting on says how they were compromised.

> The thing that works in their favor is that no one else can verify it either which absolves them from any responsibility

That's not how this works, at all. You need to have enough evidence that you can confidently demonstrate that there is no sign of a broader breach. If you get sued and can't do that, you're in trouble, because being unable to do that shows that you were acting negligently.

nicce

3 days ago

GitHub Enterprise has at least some level audit log

strictnein

3 days ago

What, exactly, is the definition of a "security company" in your mind? Threat Intel companies definitely fall under that normally, so I'm curious what you think it is.

Also, the idea that this type of thing could just be stood up as a "not-for-profit" company and ran for peanuts is kind of silly. How would the nominal fee pay for the engineers and infrastructure? Or would this just be a volunteer effort and you'd like people to do this for free for you?

sandeepkd

3 days ago

Any company where understanding of security practices has a direct impact on its revenue from early phases can be considered as a security company in my view.

From what I have seen a large chunk of internet exists and stands on the shoulder of folks who did the volunteer work cause they were passionate about it and enjoyed that part. Once built, the nominal fee for API to check IP address should cover the costs way easily for the servers.

Letsencrypt is a great example, it did took away the big money from all these commercial CA's, who used to issue blue, green and what not kind of checkmarks. Thats one big reason reason why the migration to HTTPS happened faster.

itintheory

3 days ago

The basic software is open source, and the list is free if you're running the tool and contributing detections back. They do have some curated lists that you have to pay for.

It's quite a bit less expensive than most other commercial products of this kind that I've looked at.

strictnein

3 days ago

So I guess your answer is yes, you would like someone to do this work for you for free, because others have done other work for free. And then once the free work is done, you will happily pay a nominal fee to gain the benefit of all the free work?

> Any company where understanding of security practices has a direct impact on its revenue from early phases can be considered as a security company in my view.

This could, quite literally, be any company on earth then? But not CrowdSec, because they had a single security issue? Every company on earth has had those, including every security company.

sandeepkd

3 days ago

There is discussion and then there is argument. Discussion is more about sharing and learning where as argument becomes all about proving a point, specifically individuals targeted point of view.

My observation is generic and more about state of things rather than focusing on one entity.

Also lets not belittle the hard work by just labelling it out as free. If only money had been motivation for everyone then the world would have been a different place. And like a lot of people I have done my share of passionate work that provided satisfaction to me and money for others, thats way tangential though

strictnein

3 days ago

> My observation is generic and more about state of things

You are the one who stated a specific company wasn't a security company. When asked to define what one was, your definition included any company on the planet.

> Also lets not belittle the hard work by just labelling it out as free

Who is belittling it? My point is that it's hard work and should be rewarded, not expected by someone to be done for them.

mistrial9

3 days ago

fallacy and apologist point of view here. Shall we dig in?

fallacy is "Oh I guess you mean" .. insert large unsolvable and probably unpopular derailment. Bonus points for aggressive labeling of the opponent being opposed to money.

apologist - "Any company on earth" .. We all stand together, Every Company On Earth .. does this warrent serious replies?

itintheory

3 days ago

We implemented CrowdSec for bot/scraping mitigation. The architecture is sound, but it ended up having an unacceptable false positive rate for us. This may be an issue with any kind of IP reputation approach. After a couple of months of work getting it ready to go I had to turn it off after a couple of days.

Jnr

3 days ago

Not sure what you implemented.

They provide several IP blacklists. None of those seem to be false positives. You can also add custom 3rd party blocklists.

They also provide several different rulesets. It is up to you to choose which ones to use and fine tune. LLMs can be very helpful with that.

And there are 3rd party dashboards and tools that help you manage it more easily.

I use the free version as a simple WAF on multiple servers and it blocks a lot of bots. It did require some initial finetuning though.

Are there any better open source solutions?

itintheory

3 days ago

We had the main community blocklist and several of their pricey paid blocklists enabled in a PoC capacity. We had a lot of legitimate users end up blocked. In some cases these may have been VPN exit nodes, or users on CG-NAT, or devices on a shared network with some other compromised / bot device. I didn't get 100% of the details, just that we were inundated with support requests from real users that ended up blocked.

EvanAnderson

3 days ago

Given the number of residential proxies I see scraping the couple of sites I have responsibility for I don't find IP address-based blocking useful anymore. That ship has sailed.

Behavioral and client fingerprint analysis (ugh-- having to run Javascript just to view a static site) is the only way (at least until we get user "age" and identity attestation rammed down our throats).

specialp

3 days ago

I run an attractive site to AI scrapers and yes the IP ship has indeed sailed in favor of residential proxies. They are non-stop and could have ingested all of our content many times over every day. JA4 is useless too because they have the ability to spoof this or use a very common fingerprint like Chrome that you cannot block without blowback. These bots will only use a given IP a couple of dozen times and then use another node. They use 10s of thousands of IPs. You can tell because they will come in waves from a residential or mobile network

itintheory

3 days ago

What kind of fingerprinting are you thinking of? JA4? I haven't found a way to do that inexpensively at our scale, but we may have to go that route - looking at CloudFront bot mitigation.

For behavioral, we have Anubis honeypot functionality turned on, but it doesn't seem to be effective for 99% of scrapers. Anubis is also running behind TLS termination, so I don't think it can do full JA4. It does have the less robust JA4H apparently, but I'm not sure how effective that will be.

Edit: Oh yeah, forgot to mention - it's almost 100% residential proxies. Primarily China Telecom and China Unicom. Unfortunately those providers are HUGE and also host a ton of legitimate users all over Asia.

EvanAnderson

3 days ago

> What kind of fingerprinting are you thinking of?

I'm minding a proprietary application where this is being discussed. The vendor is talking about running Javascript in the client to detect headless browsers, bots, etc. I'm guessing they're going to license something off-the-shelf. The technical specifics aren't being discussed. They're at the "wow the stakeholders" phase.

I'd like to do something ourselves but, at the size of the org I'm talking about (with virtually no in-house dev resources), COTS rules the day.

UltraSane

3 days ago

I have written my own honeypots to reduce the false positive rate. I simply have things like a VM with RDP and SSH open to the internet and any IP that tries to login gets banned at the firewall for x days. It works really well.

mazzma

3 days ago

Interesting, did you implement only IP reputation (via blocklist) or did you deploy the WAF as well? Regarding bot scrapping, you would probably want to try the new bot detection feature recently released

itintheory

3 days ago

This was just blocklist based. We had the main community list and a handful of the curated paid lists enabled.

wrt bot detection - this sounds very much like Anubis which we're also using with some success.

sidcool

3 days ago

Security firm hacked.

steve1977

3 days ago

Security firm putting their source code on Github

user

3 days ago

[deleted]

pixl97

3 days ago

A tale as old as time.

giancarlostoro

3 days ago

Sounds like an exploit took the credentials needed to extract the code, makes me wonder if a Ubikey + SSL cert for git access would have prevented the entire leak.

Sweepline

3 days ago

Oof, a security company leaking its own source code is quite the look. Hope it's not their latest version of the engine.

doodlesdev

3 days ago

From TFA:

   > CrowdSec source code consists of two parts: a private one and another that hosts our Free Open Source Software (i.e., the Security Engine), which is public by design and therefore out of scope.

Sorrel47

3 days ago

Time to scrutinize those CrowdSec deployments even closer. Hopefully, it's nothing critical that gets exploited.

miffy900

3 days ago

> CrowdSec source code consists of two parts: a private one and another that hosts our Free Open Source Software (i.e., the Security Engine), which is public by design and therefore out of scope. The private part, though, contains the source code for our SaaS console, some AWS Cloud routines, some connectors, and automations.

This leak seems incredibly mundane. customers would have the security engine installed, but not the console.

okokwhatever

3 days ago

of course... Ask Fable if the private code is valuable or not...

xyst

3 days ago

yet another security oriented company that doesn’t practice what they preach.

On the flip side, there was allegedly no PII leaked. But this event is still a red flag as it means their internal ops are absolutely shit. So it’s another vendor receiving a PNG flag.

CrowdSec. CrowdStrike.