Ask HN: How do you protect self-hosted websites from LLM scrapers?

1 pointsposted 7 hours ago
by atmosx

Item id: 49460258

3 Comments

arbol

3 hours ago

Why do you expect a high level of abuse? Blocking CIDRs of major cloud providers will stop the genuine ChatGPT/Claude traffic. For the other scrapers masquerading as them, just block the agents' user agents.

If you want the LLMs to refuse to scrape your site you could inject lots of encoded trigger words but this would probably affect your SEO score :)

selfhoster11

7 hours ago

Why would you not put a CDN in front? I’d rather do that than CAPTCHA.

atmosx

5 hours ago

Because it’s a third party service, plus CDNs have Captcha anyway.