> But now I'm really reluctant to give more stuff to the free web. Because the fact that it gets scraped and added to a pile of training data to later be monetized really rubs me the wrong way.

The thing is, people were scraping and monetizing other people's websites long before the current LLM fad. The difference is the magnitude of the problem.

> And I can see why maintainers of sites like this, or other free but incredibly useful resources might start to get irate at that.

Those who object to the scraping fall into several camps, but the biggest complaint I am hearing is that it increases both maintenance costs and time. In other words: it sucks when people are using your work in a manner that you find offensive, but it goes beyond that by doing actual harm.

I also recently made an open-source project with 200 GitHub stars private. I never had a problem with others using it as the basis for their own projects. In fact, that happened, and I received credit for it. But LLMs just hoover up everything, process it, and then spit it back out as if it were their own.

I open source as much software as I can because I want the models to train on it and get better at it!

>But now I'm really reluctant to give more stuff to the free web. Because the fact that it gets scraped and added to a pile of training data to later be monetized really rubs me the wrong way.

I dont get this.

You provided something for free to help people, but dont want to do that anymore because it might go into training data and help many many more people?

So far LLMs have been loan funded donations of loss leading services. They might never actually make their first dollar of profit.

Meanwhile ISPs the world over have been monetizing access to your content.

I feel like what you mean is that you want control over attribution.

> Because the fact that it gets scraped and added to a pile of training data to later be monetized really rubs me the wrong way.

I suspect the next AGPL (if not normal GPL) will explicitly prohibit training of closed models against the source material. Ideally it will prohibit training of open weight models too. Either share the entire process, or go piss up a rope.

Why would you offer data for free if people are going to pay someone else to access it?

The whole internet is about to experience USA-level greed because people are too dumb not to use AI. Just like Google killed small sites because people were too lazy to look at page 2.

> “…gets scraped and added to a pile of training data to later be monetized…”

I feel you. It reminds me of the "Freedom for any purpose" doctrine of FOSS and the moral question of FOSS in military usage.

> But now I'm really reluctant to give more stuff to the free web.

I feel the same way too. But guess what, all the code you did not publish gets into the training corpus anyway (when you gave Codex or whatever full read access to your filesystem).

This attitude makes zero sense to me. You benefit from the "training data" just like everyone else does. If you don't, that's a problem with you, not a problem with AI models.

AI solves exactly the meta-problem you describe: "I had a problem and needed to write a one-off doo-dad utility program to solve it." Now you can do something with your time besides writing pointless one-off doo-dads.

As for monetizing the training data, (a) it cost hundreds of millions of dollars to generate the weights, so why begrudge the companies that made the investment and did the research necessary to make it happen?; and (b) rest assured, whatever your doo-dad does, an open-weight model like GLM 5.2 can generate it for free using your own hardware.

So you don't have to pay anyone in that case. Well, except nVidia, I guess. Point granted there.

If it's the case that prediction market traders were trying to front-run the market outcomes, the prediction markets should put the fix on their end by locking down trading before a market closes with some time buffer to prevent this.

BigQuery has "public datasets", so users can even run complex SQL on it, but it's them who pays for it, not you. You only pay for data storage.

For what it’s worth, a Hetzner dedicated server has unlimited ingress and egress. It seems like it’s the only provider that does. Egress fees suck.

You can get a beefy one for about $40/mo on their server auction site.

Just... don’t miss payments. Ever. Or they’ll delete your server within a week or so.

the AI scrapers decided it would be more efficient to download terabytes upon terabytes of raw HTML by paginating through every possible facet on the search endpoint.

No, they decided that would be a great way to convince you of the narrative and persuade you to pay for "security" services that further the incumbent browser monopoly.

They're not "AI scrapers", they're DDoS'ers manufacturing consent.

Building on AWS is a financial time bomb.

Do you have any view on why the AI scrapers resulted in a heavier load than existing crawlers from eg search engines?

Where they more exhaustive or more frequent?

That’s a traffic design problem. You should be happy that your work is valuable and also protected it against excessive requests. Simple.

Ya this problem was solved 20 years ago with Varnish cache and Coral cache/CDN.

I think what's really going on is that bots expose how underpowered web servers has gotten in recent years. In the 2000s, even poorly-architected PHP sites tended to serve about 200 requests per second, with 1000+ being common for static sites. I remember when Node.js came out and claimed that it could serve more like 100,000 RPS due to its cooperative threading model. But today sites have a remarkable slowness to them, running many hundreds or thousands of database queries due to ORMs and N+1 problems, so that response times can be 500 ms or more and even 1000 simultaneous users stresses servers.

What went wrong is that nobody solved stuff like Russian doll caching in a general way separate from the programming language and database. We should have had ways to make dependency graphs using Etag headers as keys with real cache invalidation of dependent data. I went down that rabbit hole 10 years ago using touch events in Laravel with callbacks to handle cache invalidation when class model data was saved to the database. Also a query cache using Redis which I think might have been handled better at the database level anyway. After that experience, I can honestly say that cache invalidation is so difficult to get right that it's effectively an open problem. Meaning that programmings should use a package instead of rolling it by hand, and it should be a major concern from the start (along with sharding by user id or using something like Firebase).

Don't get me started on how the web should have been a P2P content-addressable memory anyway. Nearly everything should be available from a nearby edge peer, similarly to BitTorrent. But nobody bothered to solve how to make that work with HTTPS/SSL. I suspect that has to do with early flaws in the browser security model where the whole page has to be behind HTTPS or warnings appear. So it was never clear what was personally identifiable information (PII) or merely public data being served over HTTPS. To really solve that, we probably need real trust networks and maybe even zero-knowledge proofs.

Since these problems are so challenging to fix, and big companies can't be bothered to do it since they pulled the ladder up behind them, we're probably stuck with banal "are you human" challenge screens for the foreseeable future.

Yeah this seems like one of the easiest access models to create an excellent security model for -- basically just static publishing. Sounds like they were in need of a rewrite anyway!

It probably seems daunting but to be honest this feels like a weekend's work at this point with LLM assistance. Not to be glib!

Or just use a cache.

Yes, a community effort against the botnets would be great.

The "good bots" which identify themselves and follow the rules are easy enough to block, so not a problem. It is the "bad bots" which pretend to be real users and hide behind residential proxies, and so are almost impossible to block at the moment, which are the problem.

Given that there are big companies openly (i.e. on the clearweb, not even darkweb) selling access to these residential proxy botnets of compromised smart TVs[0] and mobile phones and other devices, can we not simply get a database of the these IPs and block access from them? I would venture that almost every single one of those residential users are unaware that they have devices in their homes which have been compromised and are being abused in this way, so if they were to start seeing messages from more and more sites along the lines of "Access to this site has been blocked because unusual traffic has been detected from your computer network. Please check all devices on your network and remove any malware which may be routing this traffic." then maybe we could start addressing the problem at the source.

[0] https://news.ycombinator.com/item?id=49000864

The community response was respecting robots.txt, but since that was more of a gentleman's agreement than a legal requirement, hungry AI scrapers disregarded it.

Which brings us to the old fashioned flood control mechanisms. That is, the toolkit you propose already exists and has been used for decades in various iterations to protect against various forms of attack (slashdotting, DDOS attacks, overzealous search engines, and now AI scrapers).

Have you looked into those before? Companies like Cloudflare have been at the forefront of this field for a long time now.

I don't think there is a technocratic solution to this problem. Very rich people are using their money to DDoS the internet, for no good reason. We just need to identify these people and fine or imprison them until they stop doing it. Residential proxy providers would be a good start.

This only works for crawlers that play fair, unfortunately there's many that don't. Including ones that use consumer devices like TVs [0] and mobile phone apps that offer incentives to consumers (if they're even open about it) to use their internet connection to e.g. crawl websites. There's probably an army of hacked toothbrushes and the like doing the same thing.

[0] https://news.ycombinator.com/item?id=49000864

90% of bot traffic on my network of websites is through headless Chrome, via residential bots nowadays. Impossible to block. Not even for Google, as they inflate my Adsense numbers as well.

This is good to know, but a bit of all-or-nothing. It's a shame that, for example, Google doesn't support the crawl-delay field so you can tailor their crawling to your setup: https://developers.google.com/crawling/docs/robots-txt/robot...

I presume it would also cut you off even more from referral traffic.

Claude Bot still (at least last month, and it's been doing it for over a year now) seems to have a bug when traversing (at least my sites), wherein it drops the trailing slash of a directory (which is present in the a href tag), then makes the request to the subdirectory without the slash I put in the link, then Caddy automatically responds to that (via the built-in File handler) with a redirect telling it to add the trailing slash, and Claude Bot then makes the request again with the trailing slash that should have been there in the first place.

So most subdirectory URLs get two requests from Claude bot, the first one needless because that wasn't the URL in the tag.

I won't eat your lunch if you put a sticker on your lunch box telling me not to.

To a point; as the post mentions, they added instructions for LLMs to their website and increased their licensing inquiries tenfold (the article didn't state anything about actual licensing payments being made though).

I don't think there would be any issues per se if the scrapers just paid licensing fees to get the good / complete data. But the issue was that they started to try and find exploits to get to data earlier.

worse; it isn't just no return, it is also cost.

[flagged]

Every website I visit could get a fraction of a cent in my Cloudflare Wallet. A human browsing incurs a few dollars a month. Plus, any website I access in this way decides not to show me ads either and just charges me the cost of serving the data plus a nominal profit margin.

Disclosure: I am a shareholder and would love for them to solve the AI bot problem and the ad problem like this.

we've been working on basically the same problem in the email space for a few decades (legit email vs. mass spam). its a very hard (i think impossible) problem.

I set Cloudflare up a couple of months ago specifically to block bot traffic. It didn't do anything for me. Dumb bots were still hitting every special link on my wiki fast enough that the server was continually swamped running Lua scripts. 65% of the traffic for my English-language site was coming from Vietnam. But I didn't want to block Vietnam altogether, because my hobby site has genuine users from there too.

I eventually just shut down my mediawiki instance. I couldn't find a way to keep it online and still run on an affordable VPS.

It tried it. Hundreds of “genuine” visitors per day on a new website with no search engine presence and no links. That’s a very leaky fence..

Cost?

GPTBot crawled millions of pages on my website. I get a handful of visitors from them. Meanwhile Google sends me 10k visitors each day.

So GPTBot is now blocked.

Already happening. Frontier LLM vendors have been hiring human domain experts specifically to create their own proprietary training data in targeted verticals.

They are often from residential IP.there are even businesses that allow you renting such IPs

Nobody serious would ever use an AWS IP lol

You missed the speculated motivation: unrestricted prediction markets, which are an open and broad incentive to do whatever actions might provide a slight edge in betting.

Webscraping is among the more benign things that gambling-on-anything can drive. And even that has a negative impact, as seen here.

I have no idea why those sites are legal.

John Brunner and Alvin Toffler both made stark warnings wrapped in futuristic giddiness about things like it. People like Fuller no doubt thought polling on a large scale was terrific. There were old usenet groups and BBS subs (minus the money aspect) experimenting with the model. I do not believe they ever are or were good in a largescale model (money or not).

I mean wisdom of crowds, super-forecasters, calibration and pre-registration are useful tools that can result in better predictions, turning it into online gambling is where it went sideways.

It would not make any difference.

> Read the Docs, a non-profit that hosts documentation for open-source software, who watched a single crawler download 73 terabytes of zipped HTML in one month, costing it over $5,000 in bandwidth

From the article.

Not the same site, but an example of the same issue.

What if you aren't motivated solely by making money? There will still be new free content created by hobbyists, passionate creators, altruists, etc.

(Agree with you more generally.)

I agree. I've seen it happen before on a project I use. I decided to take a look at the repo for one of the plugins, and I saw a heinous issue that basically was TELLING (not even asking) the maintainer to fix it.

Deplorable behavior indeed

they both have a risk of harm which the site operator was no longer comfortable with.

[dead]

The actual solution is damn simple and painfully obvious: make things like so-called "prediction markets" illegal.

The intelligence community came to the conclusion (after research and experiments) that things like "prediction markets" were a bad idea in 1996.

Far worse now than then, with the net and AI of 2026 and the insane number of people now online that want to make an easy buck.

Don't get me wrong; I am guessing some of us on here would do well on those places. But they should not exist.

Isn't this the NRA's argument? The only way to stop a bad guy with a gun is a good guy with a gun.

Or at least somewhere between that and full on protection racket.

LLMs come on the scene, hammer the site until it goes offline or racks up bills that threaten bankruptcy, and the solution then is to pay for an LLM to fix it and monitor it.

That's a real nice site ya got there, it would be a shame if massive datacenters going up around the world were to start hammering it from tens of thousands of IP addresses....