> 2. Not all requests are created equal. 20 requests a second for the same static HTML file? No problem. But if you have, say, a release page for an open source project with binary download links for all past versions for multiple platforms, each one being a multi megabyte blob, and a scraper starts hitting these links, you will run into bandwidth problems very quickly, unless you live in a utopia where bandwidth is free.

All of this is (and should) cached on a cdn. You can go 1000 QPS on this in that config.

Shouldn't such big blogs be put on something like CloudFlare R2 or BackBlaze or even S3 with their caching in front? Instead of having your server handle such file downloads.

>you live in a utopia where bandwidth is free.

It's called peering agreements and they are very common. There's a reason social media and sites like YouTube, Twitch, TikTok don't immediately go out of business. The bandwidth is free for most users.

The "shitbox computer in the basement" is something I would call a server. I mean, it is more capable than most VPSs (except in upload speed to the Internet).

> Like 'hey cool if I use this tool to interact with your site for this and that reason?'

They do ask: they make an HTTP request. How the server responds to that request is up to the owner. As in the article, the owner can decide to respond to that request however he likes.

I think that a big part of the issue is that software is not well-written. If you think about it, even the bots constantly requesting tarballs for git commits doesn’t have to destroy the experience of using the system for logged-in users. One can easily imagine software which prioritises handling requests for authorised users ahead of those for anonymous ones. One can easily image software which rejects incoming anonymous requests when it is saturated. But that’s hard to write, and our current networks, operating systems, languages and frameworks make that more difficult than it has to be.

oh yeah I was horrified recently starting up a "smart TV" and going through the installable apps to find a lot of repackaged youtube contents, even from creators I like, eg. a chess channel. The app just provides the same content as the youtube channel does but at the bottom of the long free-to-use license agreement there is a weirdly worded clause that says you grant the app the right to act as a proxy for partner traffic... So many smart TV users are unwittingly providing residential IP's for the app developer to rent out.

Do what the Arch Wiki did and install https://anubis.techaro.lol/

He’s saying that a modern web server setup should be able to handle the traffic without any bot-specific handling. Personally, I don’t follow.

My biggest recommendation is to just get familiar with the caching constructs that are available. I understand folks think CDNs are complicated and expensive, but they're honestly incredibly cheap and relatively easy to use.

99.9% of the time, just showing static content with a good cache-control header will solve the issue. If you have a restrictive IP provider, use a CDN to do it for you for cheap.

The more involved recommendation is trimming out features of hosted apps that aren't all that useful and are causing problems. A simple example of what I mean...

---

The author here is noting that his Gitea instance is seeing huge load from 20r/s, which just isn't reasonable (I actually host a Gitea instance myself and I know it can handle 10 times this traffic, even when running on a raspberry pi). So why is his failing?

Well - it sounds like he's letting bots hit every url of a public instance. Not the choice I'd make, but also not unreasonable, hosting public things is fine.

Buuut - it also sounds like he's left the "Download archive" button enabled on the instance.

That's not a good call. It's a feature that's used very rarely by real humans, but is tripmine for any bot scanning the site to trigger high load and network traffic.

Want a 5 second solution to the problem? Set `DISABLE_DOWNLOAD_SOURCE_ARCHIVES` in the Gitea config (see https://docs.gitea.com/administration/config-cheat-sheet). Problem solved. Bots are no longer an issue. They are welcome to scan and not cause problems anymore.

What if your app doesn't have an easy config option? Nginx will happily help, with far less complexity and frustrations than trying to blindly play whack-a-mole with IP addresses (this is terrible and does not work... period).

Configure nginx with a specific path rule that either blocks requests to that path entirely, or places it behind basic auth (it doesn't need to be clever, and you don't even need to make it secret - hell, put the basic auth user/pass directly in the repo's readme, or show it on your site.) The bots won't hit it anymore.

---

So I guess what I'm saying is really - if you're finding that bots on your sites are causing a problem, consider just treating them like users and solving the problem, instead of going mad and trying to remove the bots.

Be constructive instead of destructive.

Ultimately, a lot of those bots are scanning that content to show to users, many are even doing it directly at the request of the user currently interacting with them.

Falling into the trap of the "fuck the bots" mindset is a sure way to lose (although it can feel good emotionally). It's not understanding the problem, it's not solving the problem, and it's limiting access to a thing you intentionally made public. Users are on the other end of those bots.

He's choosing to play the "everyone loses" square of the prisoner's dilemma.

Because there's a market to serve, that you refuse to serve, so the stop gap solution is for third parties to acquire the liabilities and risks for compensation.

Why do you do it?

We were getting hammered on nofollow links. Someone made a change to stop putting the nofollow in and using JavaScript to traverse on interaction, but that person left sometime after the toggle went in and we discovered it had never been turned on much later, when we were rearranging our ingress load balancer and needed to tweak how we handled bot throttling. That was at least 10% of the requests during peak traffic and extended the duration of the crawls by a lot. So dumb.

> When data becomes truly democratized...

That is not at all what is happening.

Read this: https://jan.wildeboer.net/2025/04/Web-is-Broken-Botnet-Part-...

Basically everybody's a bot in a hidden botnet now. And we agreed to it. Phones, tablets, Windows appstores add the SDK in, and then drone. One of the big ones is "Infatica" - devs get paid to put this in legitimate 'apps'

AI companies, and whoever and whatever else use the reputation-good IPs to hammer sites and well, it's fair game for all malicious people.

Further info along the same lines at https://drewdevault.com/2025/03/17/2025-03-17-Stop-externali...

> If you think these [AI] crawlers respect robots.txt then you are several assumptions of good faith removed from reality. These bots crawl everything they can find, robots.txt be damned, including expensive endpoints like git blame, every page of every git log, and every commit in every repo, and they do so using random User-Agents that overlap with end-users and come from tens of thousands of IP addresses – mostly residential, in unrelated subnets, each one making no more than one HTTP request over any time period we tried to measure – actively and maliciously adapting and blending in with end-user traffic and avoiding attempts to characterize their behavior or block their traffic.

Sourcehut (the site described) used Anubis before swithing "to go-away, which is more configurable and allows us to reduce the user impact of Anubis (e.g. by offering challenges that don’t require JavaScript, or support text-mode browsers better)." https://sourcehut.org/blog/2025-05-29-whats-cooking-q2/

> "It's futile to block AI crawler bots because they lie, change their user agent, use residential IP addresses as proxies, and more." https://xeiaso.net/notes/2025/amazon-crawler/

If this isn't the case, then the bot detection systems big sites are using must be pretty bad, because I do almost all of my browsing on a desktop originating from residential ASN IP address, and I routinely run up against CAPTCHAs. E.g., any Stack Exchange site on first visit, and even Amazon. What reason would there be for this, unless these crawlers are laundering their traffic through residential IPs?

Speaking from my own experience, which is admittedly limited, but still — I had AI bots crawling my fediverse server, I added them to my robots.txt as "Disallow: *", they stopped.

As I said, it might be someone else entirely using OpenAI/Amazon/Meta/etc user agents to hide their real identity while ignoring robots.txt. What's to stop them? People blame those companies anyway.

However, there's no evidence those bots are really OpenAI et al.

> Therefore, unlike the traditional way of blocking an IP just because it is from a data center, tirreno suggests using a risk-based system that takes into account dozens or hundreds of rules.

If you ban by the /24, that really feels like you're coming back to the previous approach, just with extra steps.

I keep forgetting that metacritic just doesn’t work with any VPN I’ve tried, but in a way that’s really annoying (it loads the site then never loads any scores or anything you actually want to see).

It’s so annoying.

tirreno has a rule for Private Relay/Starlink IP addresses, which can be configured as either a positive or a negative signal, depending on specific needs.

There is a extra cost only if you choose to block users automatically, regardless of the tool used.

The issue here is with publicly hosted git repositories for open source projects. Everyone is authorized, including anonymous users arriving from Google.

You would end up blocking every network then. Almost no one wants malware on their machine or network. Unfortunately, people get hacked and network operators cannot always determine which machines are hacked (hackers are not known for letting people know that they have taken over a machine).

Security is hard and there are no easy solutions. People often do not know they have been hacked or even know if a computer on their network has been hacked. Also, it is often not easy to determine if traffic is legitimate, illegal, malicious, or abusive.

hmm, might as well extend this hypothetical a bit more... Epic would argue Apple as a whole is malware, and therefore all networks with apple devices on them are malware, block them all?

it does not work that way. the open internet goal is to be open. "AI" is generally meant to be intelligent, indistinguishable to humans; we created the problem for ourselves, dismissing so many (back then, and now) theoretical problems.

what i fear are increased operating costs for the end users to "prove they are human". what i hope happens is cheaper infra (bandwidth ect) to be able to support the increased demand. and, a big hope that "bots" which are respectful [but this too wont hold up, people are not 100% respectful, why will their bots be?]

so we will be stuck with more captcha related crap, more websites that must use javahelll, even a "trusted browser" (walling out any new browsers, and things like curl), maybe websites that want your ID too (the "think of the children!" folks would be pleased), and ways to "trick" the bots (and not the real users) to ban the bots.... yay

go-away has one, it only works in browsers that follow meta refresh though.

Some scummy scrapers are snowshoeing their way through scads of (residential) IPs, just for one example.

Assuming that there are simple universal solutions, eg based on IP or UA, suggests that you have been lucky enough not to deal with many varieties of this misbehaviour.

Usually though from time to time over many many years from when G was still wet behind the ears I have had to write specific defences effectively just to deal with it.

Kagi already behave really nicely, giving a small summary with numbered footnotes that links to where Kagi got its information from.

Which IMO shows that is already possible to do this and and it has been for a while already.

The OP notes at some point that the scrapers were generating lots of source tarballs individual commits. That takes server resources, and a small server can't sustain many of those requests per second. I wonder why you (apparently) don't suffer from this — in fact one can even wonder why such tarballs are generated in the first place, it sounds very pointless.