Ask HN: The amount of AI bot traffic is out control?

6 pointsposted 10 hours ago
by flyingcoder

Item id: 49705724

5 Comments

babyDoomer

3 hours ago

I doubt it's really Facebook creating fake accounts and using proxies, why would they keep their real UA then?

Anyhow, I occasionally suffer this same problem when happy crawlers find my site and use truesign.ai to block them, so far successfully. It detects residential proxies, fake emails and overall suspicious activity, and - something I really wanted to avoid - there are no captchas.

smallerize

10 hours ago

How do you know it's facebook's crawler if it going through residential proxies?

flyingcoder

10 hours ago

It had meta in the user-agent headers, it’s trivial to spoof but I’ve read of many others who are seeing the same thing. It’s not just them though, it’s also parallel web systems, and then a bunch of others who aren’t identifying themselves.

d3Xt3r

22 minutes ago

That's probably not Meta themselves, but their new agent called Muse. It's likely coming from actual individuals. Not sure why they're scraping your website though. Maybe time to turn off all the SEO stuff to make your site less discoverable?