It's mainly that dealing with them is hassle, time that could be better spent doing other more useful things.

Over the weekend I got round to turning off some old viewvc (CVS and subversion) and hgweb (Mercurial) web interfaces. They'd been running fine for a decade or two with basic security updates etc, mostly only occasionally visited for real purposes, but that's fine!

But getting 2.7 million requests/day from residential proxy IPs was a bit taxing for the old uwsgi/cgi programs (that's average 30/s), making other sites on the server suffer. It was also getting close to 1TB/month which is the VPS quota.

So I ended up turning them off, one more cut towards the centralised internet. Sure I could put a cache in front, but there are probably millions of combinations of dynamic urls in the VCS tree so caching mightn't help much anyway, not worth the effort. I'm sure the web server could be tuned better too, but again it's a waste of time.

I have a personal site on a shared host that was recently suspended for excessive CPU use caused by incessant AI bot crawling. I don't have a problem with crawling per se, there's just so many and they're so bad at it.

For me it's just a fun exercise to see what generalizations I can make about bots that botters can not really avoid or code around such as javascript. Yeah this blog is all precompressed static content in a ram disk. I'm sure it could handle a few hundred thousand requests per second. This is to show methods people could use on their forums, chan sites, chat servers, etc... Every option is optional and tunable. Any option people consider should be tested on a test server. People could just point and laugh if they so desired and that works for me too.

> WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that.

Yes we know - I still had my site knocked out a couple of times by overeager bots. They can be really abusive.

Hundreds of fake applicants to a position

When I first experimented with that I was not expecting anything to happen. Within a few days one person in Amazon EC2 in Germany started trying to do zone transfers for some of my domains likely to figure out which records to avoid and then they just excluded my domains entirely. All of the scanning stopped shortly thereafter. The scanning noise was literally all coming from one person despite the source IP's being all over the internet.

I don't see why AWS would run fail2ban on IMDS. Do they not trust their implementation? Do they want lawsuits from their big customers?

I can accept that. I've sat with RMS a few times. He's an interesting and very smart person. There are things he and I agree and disagree on. I'm sure I would never hear the end of it on this one. I'm glad he put the cancer into remission and will have many many more years.

On the topic of browsing with any kind of browser, I would be extra careful using things people have vibe coded or more specifically have not been battle hardened to protect against malicious sites. The reader apps people use I think would be especially at risk of malicious servers if they have not gone through extensive third party code reviews by penetration testers.

The user agent string is harmful in general. Any new browser should just copy the chrome user agent.

Or behavior. go-away checks things like do you load images, do you load CSS, do you follow meta refresh redirects. Anubis basically checks you can run JavaScript for a few seconds.

Thanks. Regretful implementation*, being a normie on iOS Safari:

https://i.ibb.co/vCDH79d0/IMG-0303.png

…and not a bot… hoping not to turn off iCloud Private Relay to read, well, anything. (Maybe Google Scholar if I really needed it, the most mainstream thing I know of with a complete and effective iCloud relay ban.)

*edit: per author’s reply elsewhere, as a test site, rather a good implementation! But other webadmins, please don’t adopt all methods if you can avoid it :)

I couldn't. Amusing that archive.ph's own crawler, evidently, went through just fine.

I wonder why you'd be unable to read it? :-)

Real browsers send it. [1] Some of the reader apps do not. Most bots aside from those utilizing Chrome Headless do not.

People can see a few headers here [2]

[1] - https://caniuse.com/?search=sec-fetch-mode

[2] - https://nochan.net/.env

I didn't even make it that far, I got PR_END_OF_FILE_ERROR which indicates it couldn't even get past the tls handshake.

Techno-Thriller/Sci-Fi/Mystery novels

Nice, I will have to take a look at some point.

The comments can be misleading. About 2600 people thus far and a few bots have been able to view it. Sunday is the best day to see all the unusual browsers and applications people browse the web with. Weekdays tend to be more mainstream bog standard browsers. This is a good test.

load balancer / proxy, since you need a way to get the real ip

If it is at all helpful most layer-7 load balancers have an option to enable a header that contains the real IP address, then one need only reconfigure their web daemon to log that header. It's very similar to how CDN's send the real IP in a header.

Thankyou for the kind words. I am not for everyone which is the way I would like to keep it. I agree there need to be more alternatives to a big centralized MitM caching proxy even if they can be a bit rough around the edges like mine. There is always room for improvement if enough of us try to come up with alternatives and options.

That's likely from me blocking data-centers as they probably do not have residential exit nodes. There are a couple nftable rules that could block most VPN's as well but they also block some cellular networks so I left that out.

visibly ravaged by the more shady denizens of the global internet

That is both funny, sad and oddly appropriate. Indeed, decades of running forums, chan and chat servers. I came here to take a break from all of that. It's been a nice vacation so to speak, letting someone else do all the management of a user-content driven site. Now it's all just tinkering and experimenting for me.

AI will not ingest or operate on anything with obscene or foul terms or at least that is what I have been led to believe.

A guy gets sent to hell. The demons guide him into the lobby. Satan jumps out and lets out a big evil roar to no effect. Satan then glances down and sees the spot on the mans finger where there was a ring. "Oh... well you've been through worse. The break room is down the hallway to the left, mail room is upstairs to the right..." Satan just walks away.

Every step is optional of course. There are ways to make curl work on my site but I choose to add friction as some botters abuse libcurl. I doubt anyone else will do what I do. They could spoof the user-agent but most botters seem to not know how to do that despite the myth that they all do.

For what it's worth if I had a site specific to sharing code artifacts or archives I would not block curl and I would also enable native rsyncd.

There's this weird entitlement people have in which they think that the author of a site is not allowed to choose the method by which they protect their own site because it may be inconvenient to their specific, niche, use case.

Although @Bender's unique answer is better than my obvious observation.

I would be interested in some examples of valid traffic that they would block. For the purposes of the document I do not consider anything from a data-center to be valid traffic. People can of course skip any or all steps, experiment with one at a time on a test server as they should.

But they don't script it. Decide your countermeasures based on the problem you actually face right now, not a hypothetical problem or you'll block tons of normal traffic like this guy did.