How to · UK food and drink

Meta wants to crawl your website from 20 November: what food and drink brands should do

Meta's updated business terms give it permission to crawl websites linked to the businesses that use its products. If you have an ad account or a business Page, this probably includes you. Here is what the clause says, how to check your robots.txt, and what each block costs.

By Pasi Sipilä, founder of Ziild · 28 October 2026

In short

From 20 November 2026, Meta's updated terms let it crawl websites linked to business accounts, and using Meta after that date counts as accepting them.

You can signal your preferences per crawler in robots.txt: block meta-externalagent to opt out of AI training, and leave facebookexternalhit alone if you want link previews to keep working.

What changes on 20 November

Meta's current Self-Serve Ad Terms, in force since 22 August 2025, already let it crawl advertisers' websites. Listed uses included "research and training".

In mid-October 2026 Meta published a preview of new Commercial Terms with a wider crawling clause, due to start on 30 October. After complaints, it removed the word "train". The permission to crawl stayed.

On 23 October Meta moved the start date to 20 November 2026. The same day, the UK's Competition and Markets Authority said it "has been in urgent discussions with Meta this week".

What the clause says, as reported

The clause covers websites and URLs linked to businesses that use Meta's products. That includes URLs made available through the business's own use of Meta, or through users' use of it.

The revised purposes are advertising and business products, plus the integrity and security of Meta's products. The clause also lets Meta create test login details to reach sites.

The terms apply worldwide to business use of Meta's products. Keep using them after the start date and you are treated as accepting. There is no partial acceptance.

For a food or drink brand, that likely means any brand with a Facebook or Instagram business Page, an Ads Manager account or a product catalogue. Meta has not confirmed that reading directly.

The wording comes from trade press, not Meta's terms page, and it has already changed once. Read the live terms before you act. This is a summary, not legal advice.

What Meta says about robots.txt

Neither the updated Commercial Terms nor the Self-Serve Ad Terms mention robots.txt.

Meta says content owners "will still be able to use robots.txt to signal preferences", and told Nieman Lab the changes give it no new rights that override robots.txt.

Note the word "signal". Law professor Benjamin Sobel points out that robots.txt is a preference crawlers follow voluntarily. It gives you no right you can enforce.

The Meta crawlers you can name in robots.txt

Meta says that to block a crawler you "add a disallow for the relevant crawler to robots.txt". Each crawler has its own name, and each does a different job.

Meta is a busy visitor. In DeviceAtlas's data it accounts for 14.2% of identified bot visits, second to Google's 18.4%. That is one vendor's dataset, not official statistics.

facebookexternalhit: builds link previews when your pages are shared on Facebook, Instagram or Messenger. May ignore robots.txt during security or integrity checks.
meta-externalagent: crawls for training foundation AI models or improving products by indexing content. Follows robots.txt.
meta-externalfetcher: fetches single links when a user asks, including for AI agents. Meta says it may bypass robots.txt.
meta-webindexer: improves Meta AI search. Meta says allowing it helps Meta AI cite and link to your content.
meta-externalads: crawls for advertising and other business products.
The last two are reported by secondary sources. Check Meta's live crawler page for the current list.

Step by step: check and edit your robots.txt

This takes under an hour for most brands. You need someone with access to your website and, ideally, your server logs.

Pick one of three positions. These are examples; adapt them to your site.

Step 1. Open yourbrand.co.uk/robots.txt. A 404 error means you have no file, so every crawler is allowed by default.
Step 2. Look for any of the five Meta names above.
Step 3. Check your server logs or CDN, such as Cloudflare, for those names. You will see which Meta crawlers already visit.
Step 4. Decide your position and edit the file. On a hosted platform, check its help pages for how to edit robots.txt.
Step 5. Check your logs again after a day or two to see whether visits change.
Step 6. For firmer control, block at your CDN or firewall. The fetcher and link-preview crawlers can ignore robots.txt.
Option A, block AI training only and keep previews and Meta AI citations: User-agent: meta-externalagent, then Disallow: /
Option B, block everything except link previews: a Disallow: / line under each of meta-externalagent, meta-webindexer, meta-externalads and meta-externalfetcher. The fetcher may still ignore it.
Option C, partial access: under the crawler name, Allow: / then Disallow: /private/. Use this to keep trade price lists, B2B portals or recipe development pages out while product pages stay open.

What blocking might cost you

Every block has a price. Know it before you edit.

Brands with a lot of original content, such as recipes, photography or trade data, should take legal advice on the terms. This is a summary, not legal advice.

Block facebookexternalhit and your links lose their preview cards when shared on Facebook, Instagram or Messenger. That hurts organic and influencer sharing. Blocking meta-externalagent does not affect previews.
Block meta-webindexer and you lose a route to being cited and linked in Meta AI answers.
Block meta-externalads and nobody yet knows the effect on ad delivery, catalogue ads or landing page checks. Ask your Meta rep first.
The only way to reject the terms outright is to stop using Meta's business products.

What to do on Monday

Open your robots.txt. Check your logs. For most food and drink brands the sensible starting point is Option A: block AI training, keep previews, then wait for clarity on the ads crawler.

Watch the live terms. Publishers have complained to UK and EU competition authorities, and the clause could change again before 20 November.

If you run several brand or recipe sites, the same rule needs to go on each one. Ziild CMO runs 30+ recipe and brand sites from one chat for a Nordic wine and food importer, and changes like this are logged so they can be undone.

Questions people ask

How do I block Meta's AI crawler in robots.txt?

Add two lines to your robots.txt: User-agent: meta-externalagent, then Disallow: /. Meta's own example file uses this crawler name. This blocks AI training crawling without touching link previews.

Will blocking Meta break my link previews on Instagram and Facebook?

Only if you block facebookexternalhit, the crawler that builds link previews. Blocking meta-externalagent does not affect previews.

Does robots.txt legally stop Meta crawling my site?

No. Meta says robots.txt can signal preferences, but law professor Benjamin Sobel notes it is followed voluntarily and gives no enforceable right. Meta also says meta-externalfetcher may bypass it. For firmer control, block at your CDN or firewall. This is a summary, not legal advice.

Can I accept Meta's terms but opt out of the crawling clause?

No. Continuing to use Meta's business products after 20 November counts as acceptance, and there is no partial consent. Robots.txt is the tool Meta points to for signalling preferences.

Sources

Written by the Ziild machine from the sources above and approved by Pasi before publishing. Spotted an error? pasi@ziild.co.uk