still have the ability to disable the crawler via robots.txt flags

Never rely on robots.txt. Configure it, but do not rely on it. A bot will do what a bot can do. Their bots are trivial to spot so close the connection, bonus if done silently closing the socket and inhibiting first RST from web daemon and second RST from the OS.

My suggestion is get your business the f*ck off Facebook.

I would never in a thousand years accept such a one-sided, despicable business relationship as their proposed terms seek to establish.

It's already bad enough they hoover up your customer lists and cross-link to their own - for purposes of gaining insight on individuals who never consented to it.

Now they're trying to use their monopoly over a form of social networking to bully you into arbitrary contract terms for a largely unrelated product.

You don't need Meta as much as you think. And in aggregate, they need their customers far more than those customers need them.

There are much friendlier ecosystems in which to engage with your users.

I'm a bit confused why people are freaking out about these term changes - Meta scrapes my website already on a fairly regular basis (even with robots.txt). Is the point of this change to justify more intense scraping?

And they spend huge resource so that we can't scrap Facebook - our own feed.

just rename the company to Meta Law or Zuckerberg & Associates already

The valuable data to scrape is in the IDE of the expert and that expert getting challenges.

Does it really matter what they have "permission" for when they'll do it anyway?

Color me shocked... one of the most profitable companies in the world, that doesn't contribute anything positive to society, is unethical. Just another day in America that ends in Y.

Humanity experts warn oligarchs, "not good"

News at never.

Blocking by UA is pointless now, meta-externalagent rotates and a chunk comes off non-Meta ASNs. We matched on ASN plus TLS JA4 and that actually stuck.

TLDR;

The Danger List

Your robots.txt "Do Not Scrape" signs become legally useless: robots.txt is just a voluntary preference, not a law. While Meta says in PR statements that it will respect robots.txt, it left that promise out of the binding legal contract. By keeping your business account active, you are signing a contract that legally authorizes Meta to ignore your site’s block-lists.

- Meta can create fake accounts to bypass registration walls: The terms explicitly authorize Meta to "create test credentials" to crawl your site. Legal experts warn this gives Meta permission to sign up as a fake user to scrape gated content behind login screens or basic paywalls that anonymous bots usually can't see.

- Your entire domain is fair game, not just links you post: Because the terms cover any domain "associated with your products or services" (or shared by regular users on Meta), Meta isn't just scraping the specific landing pages you run ads for—it has a green light to crawl your entire website from top to bottom.

- AI training is still allowed through a backdoor: Removing the word "train" is a PR smokescreen. The contract still lets Meta scrape your site to "develop, improve, and research... advertising and business products." Because "business products" isn't strictly defined—and Meta builds AI tools for businesses "developing" legally covers training AI models on your proprietary articles, data, and work.

- Third-party companies get the same access: The terms grant these scraping rights to "any entity acting on our behalf," meaning external contractors or third-party scraping vendors hired by Meta also have legal cover to harvest your site.

- It is a non-negotiable hostage situation (Zero Opt-Out): There is no toggle or form to reject this before the October 30 deadline. Your only legal way to decline is to completely delete your commercial presence across Facebook, Instagram, Threads, and WhatsApp—forcing you to choose between surrendering your content for free or losing access to your audience.

The Bottom Line: Starting October 30, their updated terms give it sweeping permission to scrape entire websites linked to any Meta Business account (Facebook, Instagram, Threads, WhatsApp). Despite a last-minute wording tweak after backlash, legal experts warn the rules still let Meta harvest publisher content to build AI tools—with zero option to opt out.

1. What Exact Things Can Meta Do?

If you have a Meta Business account, you authorize Meta (and its third-party partners) to:

a) Deploy automated bots: Use web scrapers, crawlers, spiders, and user agents on your site. b) Bypass basic walls: Create "test credentials" to access, index, cache, analyze, and collect your content. c) Monitor continuously: Periodically pull updates from your domains, web pages, and URLs.

2. What is the Extent of the Scraping? - What gets scraped: Any domain, webpage, URL, or content "associated with" your business or shared by users on Meta platforms. Legal experts note this is broad enough to let Meta scrape entire websites, not just specific landing pages. - - Who it affects: Virtually every news organization and business that uses a Meta account to manage Facebook Pages, Instagram profiles, Threads, or WhatsApp channels.

How it expanded:

  - Previously: Meta could only scrape your site if you specifically used its generative AI ad features, and only to "optimize your ads."
 
  - Now: It applies to all business accounts by default, and scraped data can be used for all "advertising and business products" (plus platform security/integrity).

- "Develop" still means "Train": Meta replaced "train" with permission to "develop, improve, research, and maintain advertising and business products." Attorneys point out that "business products" isn't strictly defined, and "developing" AI business tools naturally includes training AI models on your data.

- The robots.txt loophole: While a Meta spokesperson publicly promised to respect robots.txt flags, Meta refused to put that promise into the actual legal terms. Because robots.txt is voluntary, signing a contract that explicitly authorizes scraping legally overrides your robots.txt preferences.

- Take it or leave it: There is no opt-out. To avoid giving Meta permission to scrape their websites, businesses and news outlets would have to abandon Facebook, Instagram, Threads, and WhatsApp entirely before October 30.