What Meta's new crawling terms mean for UK advertisers

From 30 October 2026, any business that keeps using Meta’s commercial products accepts terms letting Meta crawl the websites associated with its products and, in the UK English edition, use that content to train Meta products. UK brands advertising on Facebook and Instagram have 13 days to decide what their robots.txt allows.

Key takeaways

• Meta’s updated Commercial Terms take effect on 30 October 2026, replacing the version in force since 6 March 2024.
• The new clause lets Meta use crawlers and create test credentials to access, index, cache and collect from domains “associated with your products or services”.
• The UK English edition says Meta may use that content to “provide, develop, improve, research, train” its products, while the US English edition names advertising and business products and leaves out the word train.
• Neither edition mentions robots.txt, although Meta told Nieman Lab that content owners can still manage access with it.
• Meta’s crawler documentation separates Meta-ExternalAgent, used for AI model training, from Meta-WebIndexer, used for Meta AI search, so a brand can block one and allow the other.

The clause Meta added

The first section of the UK English edition of the new terms, headed licences and rights you provide Meta, says “You authorise us (or any entity acting on our behalf) to use automated software (such as web scrapers, bots, robots, spiders, crawlers and user agents) and create test credentials to access, index, cache, analyse, collect, retrieve or crawl the domains, web pages or URLs (and any content available through them) that are associated with your products or services”. It goes on to cover pages “made available through your or users’ use of Meta Company Products and services” and lets Meta pull updates from them periodically.

Nieman Lab reported the change on 15 October after three legal professionals reviewed the terms, according to Search Engine Journal. Meta’s spokesperson Andy Stone called that reporting false on X on 16 October and said the update is about pulling accurate product information from third-party sites for ad and content recommendations, “not about scraping news sites”. SEJ reports that Meta declined to say whether test credentials would get past paywalls or paid content.

Is the UK wording different?

Yes, on the page Meta serves today. We read both English editions on 17 October and the purpose sentence differs.

Meta hasn’t explained the difference. Stone’s description of the clause matches the US text more closely than the UK one, and a UK brand’s legal team should read the edition Meta shows in its own Business Suite account rather than assume either.

Who on the team is exposed?

Brand and content teams carry most of the exposure. A brand’s site holds product photography it may license from a studio, editorial and guides it paid to produce, customer reviews and athlete or creator content used under limited permissions. The clause covers “any content available through” the associated pages, and nothing in the terms says how a brand grants Meta rights to material the brand doesn’t own outright. Our customer content rights playbook logs the scope of every permission, including AI use, and that log tells you which pages carry the risk.

SEO and web teams own the practical control, robots.txt. Meta’s crawler documentation lists four user agents with different jobs, and blocking the wrong one costs a brand reach it wants.

Our view at Manual Focus is that UK brands shouldn’t pause Meta spend over this clause, but they should block Meta-ExternalAgent on any domain that carries licensed or customer content, keep Meta-WebIndexer allowed so Meta AI can cite them, and get Meta to confirm in writing that a robots.txt block limits the licence. Treat that block as a partial crawl control rather than a training opt-out, because the clause covers any automated software, content can still reach Meta through the crawlers you allow or through what you upload, and Meta hasn’t said the block narrows what it may train on. That mirrors the trade we set out in The citation moved off the page, where being retrievable by AI answers is worth more than the training data a single brand withholds.

Paid teams feel this least in the short term. Stone tied the clause to product information for ad and content recommendations, the same area as the agent and checkout tools in our guide to Meta and TikTok’s Advertising Week AI ad tools. Meta’s documentation gives shared-link fetching to FacebookExternalHit, not Meta-ExternalAgent, but Meta hasn’t said which crawler feeds each ad product, so watch catalogue and ad delivery for a fortnight after any block.

What to do before 30 October

1. Find every associated domain. List the sites linked from your Meta ads, catalogues, shops and Business Suite, including microsites, regional stores and help centres, because the clause covers all of them.
2. Set robots.txt per crawler. Add a Disallow rule for Meta-ExternalAgent where you hold licensed or customer content, knowing it limits one crawler and doesn’t by itself stop training, check Meta-WebIndexer and FacebookExternalHit remain allowed, and confirm your CDN’s bot rules match, a check we also recommended in our Adobe holiday forecast post.
3. Send the UK edition to legal. Ask them to read the clause against your photography, creator and review licences and to flag any agreement that forbids passing content to third parties for AI training.
4. Put the questions to Meta in writing. Ask your Meta representative whether robots.txt limits the licence, what test credentials can reach, and why the UK and US editions differ. Keep the reply on file.
5. Record the decision. Write down which crawlers you block on which domains and why, and add it to your brand rules so new microsites inherit it. Our brand guardrails as code playbook shows how to make a rule like that checkable.

What we don’t know yet

Meta says content owners can still manage access with robots.txt, but neither edition of the terms says whether a robots.txt block narrows the licence a business grants. “Associated with your products or services” and “test credentials” aren’t defined. The terms offer no partial opt-out, and their updates section says a business that doesn’t agree may stop all business use of Meta’s products. We also don’t know whether the UK and US wording will converge before 30 October, or whether the UK ICO will take an interest in Meta training on business websites under this licence.

Further reading

• The citation moved off the page, on why staying visible to AI answers matters more than any one crawler
• Meta and TikTok AI ad tools from Advertising Week, a UK guide, on the Meta products that read your site
• Customer content rights pipeline, for logging which content you can let others use
• What the CMA’s AI choice screen plan means for UK marketers, on checking which AI crawlers can read your site
• Brand guardrails as code, for turning a crawler decision into a rule every site follows

Our services page explains how we help brand and web teams set AI crawler policy.

I run marketing for [brand], [one line on what you sell], selling mainly in
[market].

Meta's updated Commercial Terms, effective 30 October 2026, let Meta crawl the
domains, web pages and URLs "associated with your products or services". I
need a complete list of our domains that could fall under that.

Attached are:
- an export of our Meta ads with their destination URLs
- our Meta catalogue or product feed settings, including product URLs
- the list of websites and domains connected in our Meta Business Suite

List every distinct domain and subdomain these point to, including
microsites, regional stores, help centres and campaign pages.

Give me a table with the columns: Domain | Where it appears (ads, catalogue,
Business Suite) | Example URL | Who owns it (us, a retailer, a partner, or
"unknown").

Don't guess at domains that aren't in the files. If a URL is shortened,
redirected or unreadable, list it in a separate "couldn't check" table.

Here is our list of domains: [paste the domain column from the first prompt].

For each domain, open its robots.txt file and tell me what each of these
Meta user agents is currently allowed to crawl:
- meta-externalagent (Meta says it crawls for uses such as training
  foundation AI models)
- meta-webindexer (Meta says it improves Meta AI search results)
- meta-externalfetcher (fetches links at a user's request, and Meta says it
  may bypass robots.txt)
- facebookexternalhit (fetches pages shared on Facebook, Instagram and
  Messenger, and Meta says it may bypass robots.txt for security or
  integrity checks)

Give me a table with the columns: Domain | meta-externalagent |
meta-webindexer | meta-externalfetcher | facebookexternalhit | Rule that
applies (quote the line, or "no rule, default allow").

Then, for each domain I mark as carrying licensed or customer content
([list those domains]), draft the robots.txt lines that would disallow
meta-externalagent while leaving meta-webindexer and facebookexternalhit
allowed. Note that this limits one crawler and isn't a confirmed opt-out
from training.

If a robots.txt file won't load or redirects somewhere odd, say so in that
row rather than assuming.

I run marketing for [brand] in [market]. Meta's updated Commercial Terms take
effect on 30 October 2026. Attached are the clause from the UK English
edition under "Licences and rights you provide Meta" and a list of the
content types on our websites and the licences we hold for them
(photography, creator content, customer reviews, editorial).

Draft two things.

1. An email to our Meta representative, under 200 words, asking in writing:
   whether a robots.txt block narrows the licence we grant under the clause,
   what "test credentials" can access, what "associated with your products or
   services" covers, and why the UK and US English editions word the purpose
   differently.

2. A one-page brief for our legal team that quotes the clause, lists each
   content type on our sites, and flags any licence in the attached list
   that might forbid passing content to a third party for AI training. Give
   the flags as a table with the columns: Content type | Licence or source |
   Possible conflict | Question for legal.

Only use what is in the attached clause and licence list. Where a licence's
terms aren't in the file, mark it "terms not supplied" rather than guessing.