Bots and spiders hang onto old URLs longer than I hold onto a grudge.

We had some SSR that followed up with JavaScript listeners to do interactive updates instead of loading a whole new page. But the team responsible didn’t add nofollow and some bots ignore it anyway. Some significant fraction of our cluster size was answering those requests. But one deployment someone broke the fuck out of those links and it showed up immediately in our 4xx error statistics and we rolled the change back within a week.

Three years later I was still seeing those busted URLs being submitted.

Then the fix for the nofollow got implemented and declared done but the flag never got made permanent. We were pretty peeved when someone learned that in some offhand comment in a conversation between teams.

We were already identifying “good” and “bad” bots at the border and telling them to fuck off, so all of this was coming from our supposed friends. And that’s when I realized that a company owning a search engine and a cloud service provider is a protection racket. Google cloud is selling you servers in order to be talked to by Google bots.

> I banned Alibaba's Cloud two years ago and it still hits one of my domains 1 million times per day, despite every route 4xx-ing during that time.

You may as well silently drop the packets if they don't adapt.

> spamming for vulnerabilities

This has been going on for FAR longer than LLMs have existed. We call it "internet background radiation", because it's everywhere; it usually shows up within minutes of a host coming online. Or just about immediately after your first LetsEncrypt certificate gets issued.

Pretty much every infosec company of non-trivial size runs one set. The old venerable services are Shodan and Censys. Then there are a various security "researchers" who like to point their crapware vulnerability "scanner" at a subnet or ASN and let it rip.

And then there are of course all the actual botnets and low rent access broker wannabes trying to pop shells where ever they can, so they can resell that access.

Application-level banning seems very inefficient, these domains should be banned at the network (IP) level. Just drop any incoming packets from these domains/IPs, send absolutely nothing back that even confirms that you exist.

Can you do something with fail2ban? I.e. after a small number of requests resulting in 401 or 404, or a series of requests that's too fast to be reasonably from a human, ban that address for 48 hours.

If a legitimate request later comes from that address, it will probably get through.

But not sure how well iptables handles checking every connection against a list of possibly hundreds of thousands of addresses.

Does that old slowloris thing work but in reverse? Up your connection limits and keep N alibaba ones open forever, drip feeding them a small stream of data until they leak or timeout.

> These clouds must get the abuse happening under their own roof under control.

Similar to ad click fraud and scam ads, the cloud provider gets paid, why would they turn down the money?

We spent a decade wasting cryptocurrency compute and still didn't come up with the decades old promise of a partial payment plan for web use. Seems like it'd make a lot of sense now that ad revenue is basically garbage and repetitive traffic is clearly just leeches and parasites that should pay for freeloading.

Someone hack a payment gateway into every webserver and MCP and let's get the bloodsuckers to pay for the internet.

> like leaked .env routes, brute forcing admin panels, and so on

That sort of thing was very common long before agent based abuse scanners. Volume would wax and wane but never went away.

Though other scans for vulnerabilities have shot up in frequency now everyone script kiddie and their flee-bitten mutt can ask an AI to do it for them.

This is a very smart thing to do! Thanks.

Why allow random requests coming from the clouds?

However, they could just use residential proxy botnets.

I believe the future is static resources with dynamic apps being only offered to people who can authenticate. And even then I’d recommend grandfathered accounts (eg telegram users that had an account for at least a year).

I’ve had to block similar on clients sites. Their search pages would start to get hammered with obvious agentic search spam. Searching for thing not even related to the site in all combinations of the queries. Some of the user queries contained potentially PII.

Meta went absolutely haywire on one of my sites a couple months ago.

Is there a handy list of all of this that users contribute to?

so you're saying I should use my smtp filters on my website? (to be fair that seems like a good idea)

AWS is terrible. We had another tenant on AWS attacking us on AWS. AWS support took nearly a week of slowly bouncing around a case to do anything.

[flagged]

This is a very smart thing to do! Thanks.

Why allow random requests coming from the clouds?

However, they could just use residential proxy botnets.

Isn’t scanning for vulnerabilities illegal? Traditionally security researchers had to be very careful about such things

Last year, I read your post and implemented it. I had no way to tell whether there is any benefit to it, so I discontinued it. Serving the data, even though it is compressed and the compression is precomputed, was still a waste of bandwidth. It could hit the browsers of legit user (false positive).

The main problem on my site is that bots hit the CGIT repo, where visiting files triggers syntax highlighting scripts.

I put in a skill-testing-question type system where you get a maximally long-lived access cookie if you answer right (so legit users are bothered as infrequently as possible).

So it ended up not having a noticeable impact on the bots?

I have a small website with a contact form that gets a lot of spam. I found a very effective way of filtering spam, but it only works because it's not mainstream, so I can't ever share it :(

Can that technique be applied to packet compression?

What is that improved strategy? :pray:

Thanks for posting this, as a paranoid self-hoster, I'll be adding it to my already extensive set of block lists.

[flagged]

:) ive been dreaming of a usable emacs browser for so long

If you are rolling your own it probably doesn't need to be as clever as Anubis is trying to be. Simple something relatively CPU heavy in JS will do the trick, bonus points if you can make it memory-heavy too. Most bots will give up running something CPU heavy currently (though they will run a lot of JS stuff as much of the web is inaccessible without) and they aren't going to optimise your challenge in WASM or similar because it isn't worth it for the small number of sites it protects (Anubis' challenges on the other hand may well be worth optimising for that way because of their wider use, hence that project is stuck in the cleverness war of attrition and definitely needs to move toward memory-hard problems and something more efficient than JS sooner rather than later).

My main reason for planning to implement something like that myself instead of something off the shelf is two-fold:

1. The points mentioned above

2. I can be a dick and make some subsets do harder work (longer challenges, adding time delays that wouldn't phase a bot) - “hi there North American IP, I know how much you like extra hassle hence the love of effectively paying for tarriffs, here, have a little extra work on me!”

I want mini games! "Score 5 points" style. Although, it would surely fail ADA compliance. I don't see how any of this could be practically effective, respecting ADA.

> valuable traffic

What's the value for the website maker in these? There's user value to it (rather than just value for Anthropic/OpenAI) but the whole point of these bots is to make sure nobody ever needs to visit your website.

Strong agree, a good way forward is letting user agent once again truly mean an agent on behalf of the user. And speaking anecdotally, I've seen different traffic "shapes" under the same UA, which leads me to suspect that some actors might be disguising traffic as more mainstream labs.

Even if you block the user-agent, the traffic returns under a fake browser user-agent. No way to keep the bad ones out.

[flagged]

[flagged]

[dead]

Well, potentially very little! It could be subscription usage, or just really inefficient code authored by an agent. It probably costs LESS in tokens to write a poor quality cloner that breaks upstream things.

I wonder if you could just blast them with ever changing content to burn their summarization tokens on BS.

Though they probably use the cheapest possible models they can so the damage from their dumb business model would be limited

I like the spirit of this approach; make them suffer! But I'm surprised the bots wouldn't have reasonable timeouts to counteract. How long do they typically wait on these requests?

EDIT: sorry, that was a dumb question before reading your post. You actually learn the timeout they tolerate. Nice!

I'm running some tests currently

https://deep.43z.one (how deep bots will crawl)

https://stalling.43z.one (how long they'll keep the connection open)

> 3000 concurrent crawlers

I gave up on tarpits when it got to 130k "waiting" connections.

haha im not naming names

> I'm trying to understand the reasoning behind the entity that was sending presumably hundreds or thousands of requests per minute to the same page.

The most likely answer is they don't give a crap because it's not their money and no one is going to do anything about it.

If it's an AI news site it's probably coded by AI, and by someone who doesn't know what they're doing, or what "rate limiting" is.

doesn't fix the issue

I run a few private MMO servers, we have a website with a support forum, in order to post, you need to register, log in, fill in account data, then create a thread, each of these pages is protected by Turnstile as well as a general bot check for the website, we frequently get automated spam from RMT (real-money trade) services for the OG game. Turnstile is maybe effective against simple bots that just try to call curl/requests/whatever but if you have bots that use a browser driver it's completely useless.

Turnstile is no longer effective. Anyone with a budget uses a real headless chrome instance and gets around it. You only need the spendy chrome instances to get the initial cookie, then you pass that to your workers and they scrape.

It’s really starting to suck. The worst part is these are coming from consumer vpn providers that we can’t just block (lots of real users paying real money on our service are vpn users) or they come from big residential isps. The whole residential relay thing is a hard adversary, since it’s spread across so many real users paying ips that you can’t whackamole it. I’d say 1% of residential ips are participating in a botnet (there’s prob some real numbers out there if I cared to look).

This is just going to push everyone harder into walled gardens. Web input biggest inbound but our content is our mote, we are going to have to turn off non logged in content viewing, which will have real business impacts.

I did this for alyph.ai, but was recently flooded with so many browser-using, Turnstile-defeating bots from Taiwan that I had to completely turn off sign ups. It's not fun anymore.

Not even third party login worked (Google, Linkedin, ...) to curb this.

They were logging in, scraping the client API for vulnerabilities/possibilities and then spammed erotic dog poems to Claude Fable until the $5 welcome credit got drained.

Fun New World.

not yet, but I think something like this would be a next step

[flagged]

[flagged]

Certificate transparency logs get scraped. However, they cannot scrape wildcards. Get a domain.example + *.domain.example cert and host your blog on blog.domain.example and you can keep the bots from finding you, at least for a little while.

it's org-html-export-to-html babyyy lmao

If you have an SSL certificate so such a domain can serve HTTPS traffic if needed, which it wouldn't surprise me if some parking services generate for you these days via an ACME provider, then your domain will be in the certificate transparency logs so can be found that way.

One possible way around this is to use a wildcard and not serve anything on the base domain. Unless the right subdomain is listed somewhere, they'll not get any scans. Maybe useful for projects that you have online for just yourself and a small closed circle.

That's a vague and borderline off-topic suggestion, imo. When the problem is too many malicious visitors hammering your server, the amount of time it takes the visitors' browser to run the JavaScript is not the issue; on the contrary, when dealing with this issue, you might as well do more rendering client-side to offload as much work from the server side as possible (hence the Anubis-esque defense of adding random JavaScript work that isn't even necessary to render the real page).

Can you elaborate more, please? I'm curious.

Yeah it's awful they keep coming months after you post anything as a Show HN now, definitely discourages me from doing it

> So you got hammered by curl requests! But isn't that part of your delivery static anyway? Were you regenerating the same HTML over and over?

i dont think this was the cause of the slowdown, it was just sort of the thing that led to the discovery that this was mostly bots.

in general, because the purpose of the website is to allow users to generate news feeds based on custom criteria, it does do a bit of meaningful work on each request (ones that do follow through and load the full page).

> How many was it? > Also interesting that it was a single "user". (user-agent?) Did it come from a different IP every time?

I'm intentionally leaving some of the details vague here, both because I don't want to be too harsh on the original author (it may have just been accidental/a buggy program/etc.), and because I don't want the traffic generators to know how im flagging them.

Possibly, possibly not. I think some people are used to passing around news/information as short-form videos these days but HN (mostly) isn't into that kind of content.

How could the specifics of TLS help deter/block bots?

yeah, it was a wake-up call that I needed to make certain parts of my application faster, and I've been working on it for the past few weeks

Once every 24 hours seems sufficient for the stated purpose.

Glad you're liking unslop.news :) If you ever find any bugs or want anything added, it's open source so feel free to open a ticket https://github.com/leiDnedyA/hn-without-ai

same, but I see degraded quality in the submissions thatbare left, do you experience the same?

I don't know HN's rationale, but: if you try to ban self-promotion, then in practice you're only banning honest self-promotion. And if you allow it, but try to ban accounts that only exist for that purpose, then in practice you're encouraging them to spam mediocre comments in other threads to dilute the signal. [edit: I see that the guidelines do say not to use HN primarily for promotion. I'm willing to bet this significantly increases the amount of insincere engagement, but obviously the admins/moderators here are more aware of the tradeoffs than I am, and they have decided they are worth accepting.]

IMO the important thing to try to crack down on is vote manipulation; if you have that under control, then you can probably happily allow self-promotion and trust that only the interesting stuff will rise to the top.

Any thoughts on what the alternative would be?

Perhaps a contribution requirement per post? Though this proposal just invites bots to actually post comments because that becomes a requirement for posting links.

Ostensibly, they aren’t. From the Guidelines (https://news.ycombinator.com/newsguidelines.html):

> Please don't use HN primarily for promotion. It's ok to post your own stuff part of the time, but the primary use of the site should be for curiosity.

The c10k problem is a different problem. It relates to simultaneous open TCP connections at a server. It would take a very very large number of crawler bots to force open 10k connections, I suspect.