Thanks, we'll put that in the toptext as well.

Interesting that babble.c doesn't compile (with gcc 14):

    babble.c: In function ‘main’:
    babble.c:651:40: error: passing argument 1 of ‘pthread_detach’ makes integer from pointer without a cast [-Wint-conversion]
      651 |                         pthread_detach(&thread);
          |                                        ^~~~~~~
          |                                        |
          |                                        pthread_t * {aka long unsigned int *}
    In file included from babble.c:77:
    /usr/include/pthread.h:269:38: note: expected ‘pthread_t’ {aka ‘long unsigned int’} but argument is of type ‘pthread_t *’ {aka ‘long unsigned int *’}
      269 | extern int pthread_detach (pthread_t __th) __THROW;

As this binary does a bunch of manual text parsing and string operations in C (including implementing a basic HTTP server), I'd recommend at the very least running it as an unprivileged user (which the author implicitly recommends via the provided systemd unit file) inside a container (which won't definitely save you, but is perhaps better than nothing).

The program also uses unsafe C functions like sprintf(). A quick look at one of the instances suggests that the use is indeed safe, but that sort of thing raises red flags for me as to the safety of the program as a whole.

And while it does process requests very quickly, it also appears to have no limit on the number of concurrent threads it will create to process each request, so... beware.

Very elegant and surprisingly performant. I hope the llm bros have a hard time cleaning this shit out of their scrapes.

> Can bot writers overcome this if they know the credentials?

Yes, instead of doing just a HTTP request, do a HTTP request with authentication, trivial really. Probably the reason they "can't" do that now is because they haven't came across "public content behind Basic Auth with known correct credentials", so the behavior hasn't been added. But it's literally loading http://username:password@example.com instead of http://example.com to use Basic Auth, couldn't be simpler :)

Not sure if I can follow you, why would credentials known by anyone stop bots?

Clever solution, but it will only work as long as it doesn't become mainstream, or even a tiny bit more popular.

LLMs can now detect garbage much more cheaply than humans can. This might increase cost slightly for the companies that own the AIs, but it almost certainly will not result in hiring human reviewers

your requirements.txt lists feedparser but where are you actually using it? https://github.com/search?q=repo%3Arumca-js%2FDjango-link-ar...

My bad. It's fixed now. (and yes, the gcc suggested fix is the right one.)

It's often one IP (v4!) per one request. It's insane how many resources are being burned on this stupidity.

Part of the reason I did this is to get good numbers on how bad the problem is: A link maze is a great way to make otherwise very stealthy bots expose themselves.

If the payload expands to something too large then it is easy to detect and ignore. Serve up thousands of 10kb or 100kb files that expand to 10s of MB with random garbage inside...possibly the same text but slightly modified. That will waste the time and CPU cycles and provide no value to them. Maybe also add a message you want to amplify so AI bots train on it.

What you're referring to are LLMs visiting your page via tool use. That's a drop in the ocean of crawlers that are racing to slurp as much of the internet as possible before it dries.

> Do they do any harm

Not to me, but I've known people who have had their sites DDoSed out of existence by the scrapers. On the internet, it's often the smallest sites with the smallest budgets that have the best content, and those are hit the worst.

> They do provide source for material if users asks for it

Not for material they trained on. Those sources are just google results for the question you asked. By nature, they cannot cite the information gathered by their crawlers.

> You still need to pay for the traffic

It's so little traffic my hosting provider doesn't bother billing me for it.

> and serving static content (like text on that website) is way less CPU/disk expensive than generating anything.

Sure, but it's the principle of the thing: I don't like when billion dollar companies steal my work, and then use it to make the internet a worse place by filling it with AI slop/spam. If I can make their lives harder and their product worse for virtually no cost, I will.

They certainly effect some services: https://status.sr.ht/issues/2025-03-17-git.sr.ht-llms/

1. The bots have essentially unlimited memory and CPU. That's the cheapest part of any scraping setup.

2. You need to send the data for the Markov chain generator to the client, along with the code. This is probably bigger than the response you'd be sending anyway. (And good luck getting a bot to cache JavaScript)

3. As the author said, each request uses microseconds of CPU and just over a megabyte of RAM. This isn't taxing for anyone.

Why? The point is not to train bots one way or another, it’s to keep them busy in low resource activities instead of high resource activities.

To what end? I imagine ad networks have pretty robust bot detection. I'd also be surprised if scrapers didn't have ad block functionality in their headless browsing.

What you describe sounds more like industrial farming than tricks played by third world farmers (whatever that means).

Industrial ag regularly treats product to modify the texture, color, and shelf life. Its extremely common to expose produce to various gases and chemicals to either delay or hasten ripening, for example. Other tricks are used while the plants are still in the ground or immediately after harvest, for example spraying grains with roundup to dry out more quickly.

The agricultural farmers did it to themselves, many are very wealthy already. Anything corporate America has taken over is because the farmers didn’t want to do the maintenance work. So they sell out to big corporations who will make it easier.

Same as any other consumer using Meta products. You sell out because it’s easier to network that way.

I am the son of a farmer.

Edit: added disclosure at the bottom and clarified as agricultural farming

In authors setup, sending Markova generated garbage is much lighter on resources than sending static pages. Only bots will continue to follow links to the next piece of garbage and thus he traps bots in garbage. No need to detect bots, they reveal themselves.

But yes, all bots start out on an actual page.

Ah, it is explained in another post - https://maurycyz.com/projects/trap_bots/

Clever

They don't especially want to be sneaky, they mostly want the crawlers to stop hammering their site. Getting blocked would be a win.

Good thinking.

One way to keep things mostly the same without having to store any of it yourself:

1. Use an RNG seeded from the request URL itself to generate each page. This is already enough for an unchanging static site of finite it infinite size.

2. With each word the generator outputs, generate a random number between, say, 0 and 1000. On day i, replace the about-to-be-output word with a link if this random number is between 0 and i. This way, every day roughly 0.1% of words will turn into links, with the rest of the text remaining stable over time.

Add "babble" to any url to get a page of nonsense:

https://maurycyz.com/babble/projects/trap_bots/

The follow on post explains:

> You don’t really need any bot detection: just linking to the garbage from your main website will do. Because each page links to five more garbage pages, the crawler’s queue will quickly fill up with an exponential amount of garbage until it has no time left to crawl your real site.

From: https://maurycyz.com/projects/trap_bots/

it does at a macroscopic level by making scraping expensive. If every "valid" page is scattered at random amongst a tarpit of recursive pages of nonsense, it becomes computationally and temporaly expensive to scrape a site for "good" data.

A single site doing this does nothing. But many sites doing this has a severe negative impact on the utility of AI scrapers - at least, until a countermeasure is developed.

That would work, but I'd really prefer not to force users to run JavaScript, break RSS readers and slow down page loads (round trips are expensive). Adding a link maze to a random corner of the site doesn't impact users at all.

Yes, this would be fine if you have an SPA or are otherwise already committed to having client-side JS turned on. Probably rot13 "encryption" would be enough.

OTOH, I doubt most scrapers are trying to scrape this kind of content anyway, since in general it's (a) JSON, not the natural language they crave, and (b) to even discover those links, which are usually generated dynamically by client-side JS rather than appearing as plain <a>...</a> HTML links, they would probably need to run a full JS engine, and that's considerably harder both to get working and computationally per request.

> it probably costs you as much or more than it costs them in terms of both network bytes and CPU

Network bytes, perhaps (though text is small), but the article points out that each garbage page is served using only microseconds of CPU time, and a little over a megabyte of RAM.

The goal here isn't to get the bots to go away, it's to feed them garbage forever, in a way that's light on your resources. Certainly the bot, plus the offline process that trains on your garbage data, will be using more CPU (and I/O) time than you will to generate it.

We should encourage number 2. So much of the content that the AI companies are scraping is already garbage, and that's a problem. E.g. LLMs are frequently confidently wrong, but so is Reddit, who produce a large volume of trading data. We've seen a study surgesting that you can poison an LLM with very little data. Encouraging the AI companies to care about the quality of the data they are scraping could be beneficial to all.

The cost of being critical of source material might make some AI companies tank, but that seems inevitable.

Not to mention they have to store the data after they download it. In theory storing garbage data is costly to them. However I have a nagging feeling that the attitude of these scrapers is they get paid the same amount per gigabyte whether it's nonsense or not.

You can easily generate a number of random images with ImageMagick and serve these as part of the babbled text. And you could even add text onto these images so image analyzers with OCR will have "fun" too.

Example code:

   for c in aqua blue green yellow ; do
      for w in hello world huba hop ; do
         magick -size 1024x768 xc:$c -gravity center -annotate 0 $w /tmp/$w-$c.jpeg
      done
   done

Edit: added example

The traditional approach is a link to the tarpit that the bots can see but humans can't, say using CSS to render it 0 pixels in size.

You don't need to classify bots. Bots will follow any link they find. Hide links on your pages and eventually every bot will greedily find itself in an endless labyrinth of slop.

if that floats your boat, sure. It's also home to most of the world's malware, and you usually don't need it.

Only low IQ folks are okay with having their traffic MITMed by Cloudflare (and the NSA). Also, they can extort you and cut you off at any time, as they have done to folks, which further supports the prior point.

Trying to detect "garbageness" with an LLM drastically increases the scraper's per-page cost, even if they use a crappy local LLM.

It becomes an economic arms race -- and generating garbage will likely always be much cheaper than detecting garbage.

“Build my website, make no mistakes” is about the same, and we all know how _wildly_ effective that is!