I know for a fact companies in China do this. One paid me directly for work in this space, and in another, I found my own OSS project in their artifact cache while working there.

I've only ever gotten in trouble in the UK

Why do you feel entitled to navigate to a website you neither own nor pay for via a method the owner expressly forbids?

Since my grade-school mascot/team was "The Trojans" and my adopted hometown is firmly rooted in pre-Christian Greek culture and mythology, I am currently studying the Trojan War, and I may observe that "Trojan" is an extremely perspicacious brand-name for condoms, more than the average guy would expect.

in my defence, i did lobby hard for "GuyWhoWavesYouThroughTheGate"

I think this is the point that has become a moral dilemma. Increasingly, in this agentic age tasks like research and web exploration that would be done by humans and even buy things from an website is now done through agents. While I sympathize with the main author about Ddos, but blocking every agent regardless of intent seems cruel. This would allow an MCP doing deep research(almost identical to human behavior) to move forward. But there should definitely be more work done system that focus on agent intent(i.e some agent trying to find the best price for vacation or scraper building alternatives to Travel Advisor).

CDN security SWE here. I have functionally zero interest in what individual people do in terms of automation. I automated my gym's class signup forum during COVID so I get the validity of the use case. You want to work around the mitigations to do the same? Go with my blessings.

The killer is that everything that works for individuals trying to get through the day and make the web a bit smoother is immediately used by industrial crawlers strip mining the Internet, and you can't block one without blocking the other. A future where the web is only accessible via device attestation is extremely dystopian but so is an Internet where every drop of content goes into an LLM training set.

Things like this is why all of the worthwhile content is going to drain into balkanized spaces, and we're all the poorer for it.

Why did you need it from a script? Right click -> Save Page As... is still a thing and works quite well in my experience, even with complex React SPAs.

It's not just excess bandwidth. It's cpu cycles, poor site coding, database issues. Like I said, not everything is backed by a huge company that has the money/hardware/engineering resources to absorb the load.

That’s the point. For every legitimate though ultimately unimportant use case like yours there are a thousand others who would scrape the results 86,400 times a day to either try to sell or mirror and plaster with their own ads.