I sometimes still wonder exactly who these scrapers are

This has zero to do with AI (it's just a convenient scapegoat that fits the narrative). Absolutely everything to do with those pushing for centralised control of the Internet.

You need to account for the fact that a lot of scraping is delegated by the big players to smaller players who can take loss of reputation and can use questionable methods (residential IPs). Some of the scrapers from these AI companies have been written very poorly from performance perspective.

From what I understand the pressure has created even Google to be a lot more aggressive than what it was before. Not entirely sure but I believe google has two categories of scrapers, the regular one and a new one for AI

One of the defining characteristics of AI turned out to be utter and profound facelessness. The scraping, the data centers, the slop content, it's like it all comes from thin air. Nobody is talking about who is really doing all this or why, because it's always just coming from... somewhere else outside of our place.

Most of the nonsense comes through botnets/residential proxies, so it's very hard to say.

Well, some websites claimed China is behind it, which could make sense (I would not know either way). At the same time, though, I kind of doubt your carte blanche here for all those companies. Why would you think none of them are responsible for the AI slop spam?

> Our largest offenders seems to be mostly limited to South-East Asia, so probably mostly Chinese AI projects, but that's speculation

Ok. So you also don't know. Well, I don't know either, but I don't make a speculation by claiming x, y, and z companies to be exempt. In my book they are all responsible.

0.05 cents even. The beautiful thing is, minuscule amounts of micropayments are already enough to fix the incentives

and do you happen to know a zero friction payment system that works internationally like the Internet itself does? because I promise you that unless you already have a captive audience, even a minuscule amount of friction to access your service will cost you an overwhelming percentage of organic human visitors.

The question is what exactly are you trying to achieve. Do you want to prevent them from gathering data or do you want to save resources?

If it is the latter, then, perhaps, some p2p (torrent-like) content delivery system could help.

I think it's gonna kill off the internet not just for bots but to whole class of lower income countries, especially for younger learners

Perhaps the biggest problem is 5 cents or less isn't enough to cover minimum transaction fees.

Cloudflare is working exactly on enabling that, like it or not.

https://blog.cloudflare.com/monetization-gateway/

How many pages does the average software developer visit everyday? 1.000? Price at 0.001 per load and it’ll be completely impractical for crawlers but super cheap for the average connected human on earth.

One thing that surpised me about gentoo is just how low budget it is as an operation. They are doing everything with a $12k budget. [1]

[1] https://www.gentoo.org/news/2026/01/05/new-year.html

>We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time.

What's the point of this compared to letting cloudflare handle everything automagically? Presumably whatever heuristics they come up with are going to be better than you can, given limited time and budget?

How did you implement this? My site's getting hammered, any tips would be appreciated

> Most scrapers are relatively honest in some way shape or form.

Did you miss a "dis" in there?

>I don't really understand why there is resistance to in-browser Anubis-style gating using crypto mining for public sites like this.

But in the case of anubis it's not even used for crypto. It's just wasted.

>You mine crypto and my free service remains cost-neutral. Yes, that transfers some of the cost to you, but it is marginal.

No, the problem is time wasted. I don't care about the electricity cost. Spending 10s to solve a challenge on a 5W SoC translates to 0.0005 cents (yes, cents, not dollars). Meanwhile if I click on a link and it doesn't load in 5s, I'm seriously questioning the value of your blog or whatever, and will probably just close the tab.

Instead of punishing a criminal, you make everybody pay.

This is not only unfair to legitimate users having to pay.

It also needs to explain what actually happens when the Ddos succeeds (I know, you are talking about scrapers, but what’s the difference really?). Does that mean that the attacker just gets to shrug it off.

Don’t take my statements as facts, I just want to outline a few reason I could come up with that show a purely technical solution might not be enough.

Akin to “just use Cloudflare, it’s free”.

Yes it works but why conceding defeat and say “oh from now on you depend on a business to publish a server”?

The question is, even if you identify who is the source of the traffic, are they even in a place where you can realistically sue them? The problematic traffic generally is not the bots that identify themselves, but the ones that are using residential proxies and try to be a non-fingerprintable as possible.

I agree in principle. The tech community naturally has a tendency to reach for tech solutions to societal problems. But without jurisdiction, laws are meaningless.

If you create a law that says you have to honour robots.txt files, what do you do if an IP from another country fails to do so?

No, most of the load comes from armies of residential IP addresses that look like Google Chrome on the wire. The major chat UIs properly identify themselves. These waves of attacks do not.

At this point they've mostly run out of material, so ANY type of content is valuable. Your small personal website, why would they scrape that? It's 10.000 additional words, wouldn't want to miss that. My Github repos.... got to get buggy code from somewhere I guess.

I get what you're asking, and I'm wondering the same. Not all sources are created equally and we see the results all the time. LLMs outputs nonsense all the time, like Flock cameras containing 5 grams of gold and ounces of copper, because they are completely on critical of their sources. Perhaps there's some weights that says: Kernel mailing list, MariaDB documentation and Microsofts Learning sites are 100% trust, Reddit 50%, 4Chan 10%, but I doubt it.

Anthropic might care a little bit, seeing as they scan books, but again, is it just all books? Because other than some flowery language I don't really see the point in scanning a 1970s paperback only spy novel.

They're scraping everything. It doesn't matter what. It doesn't matter if it makes sense. They just scrape it all.

They're mining the written output of technical talent discussing troubleshooting, bug finding, patches, inter-dependencies, etc. It's an AI trainer's wet dream.

Also: where exactly are AI companies incentivized to be anything but shitty 'neighbors'?

Spam, DDoS attacks and other network abuse used to cause your hosting company or provider to call you and tell you to knock it off or you'll get disconnected, if your provider was reputable. If your provider wasn't reputable, it was likely a matter of time before they would get a nasty call from their upstream provider.

Now it just gets you a thank-you from the sales team for all the bandwidth you bought.

Meanwhile, do any of the cloud providers have any incentive to do anything about this? Hell no. They're making money off you having to ramp up extra or bigger instances. They're making money off the bandwidth. They're making money off the people doing the crawling, too. They're incentivized to do the exact opposite of effectively help you with your AI bot problem.

Lots of build failure detailed investigations and gcc/kernel expertise in debugging misbehaving or outright ICEs.

That's it, I guess?

Dumping the database won't stop the scrappers because humans running the scrappers are not even aware Gentoo exists. They're just scrapping every website their bots find.

The hardest-to-mitigate bot traffic tends to come from residential proxies, and most residential connections are still v4-only.

However, it is also easy to get large numbers of v6 addresses cheaply.

My residential ISP doesn't support IPv6 at all. If a site is ipv6 only, I can't reach it unless I use a VPN.

This is weird, every VPS I've rented in the last 8 years had IPv6, and some didn't even had IPv4.

That's the thing, they aren't.

Please write a function that returns true when the user is using a residential proxy and false when it is a normal user on a residential IP address.

Scrapers will ignore the dumps.

You can link to them in 10 million HTTP 429 responses, they will still ignore them.

Plot twist: the AI wins not through terminators but a kafka nightmare

1. You shouldn't need to be a professional sysadmin to be able to deploy a basic semi-dynamic website

2. Considering the spike of complaints over the past two years and the fact that they range from small time site operators to larger orgs (Linux Kernel, GNOME, Duke University, etc), do you think that maybe you're missing part of the problem space?

Are you volunteering to setup and host that extra complexity for the Gentoo project?

A bugtracker is inherently dynamic.