Meta's new Commercial Terms, effective October 30, 2026, let it crawl with robots the sites associated with a business's products or services, create test credentials to access them and use their content, including to train its products. They apply to any business use, including managing a Facebook Page, and continuing to use the services counts as acceptance. Meta replies that robots.txt remains the control. Its documentation separates Meta-ExternalAgent (training) from Meta-WebIndexer (Meta AI): a site can block the first and allow the second.
Contents
Meta has published an updated version of its Commercial Terms, effective October 30, 2026.1 A new paragraph lets it send robots to the sites associated with a business's products or services, create test credentials to access them and use their content to train its products.1 Search Engine Journal detailed the text on October 16, and PPC Land compared it with the March 2024 version.23 This article covers what the clause says, how Meta replies and how to set robots.txt for a small business that manages a Facebook Page.
01What the crawling clause in the Commercial Terms says
The clause sits in the section on the licenses and rights a business gives Meta, in the English (UK) edition of the terms.1 It is new: the March 6, 2024 version contains no crawling clause, according to Search Engine Journal and PPC Land.23
What Meta can do
The text authorizes Meta, or any entity acting on its behalf, to use automated software (web scrapers, bots, robots, spiders, crawlers and user agents) and to create test credentials.1 These tools are used to access the domains, web pages or URLs associated with the business's products or services, or made available through the use that the business or its users make of Meta's products.1
Meta can access, index, cache, analyze, collect, retrieve or crawl these addresses, then return periodically to pull updates.1 The content can be used to provide, develop, improve, research and train Meta's products, services and features, and to maintain their integrity.1
Who is covered
The Commercial Terms apply to any business or commercial use of Meta's products: ads, marketing messages, selling products, a catalog, managing a Page or a group for business purposes, Ads Manager, Meta Business Suite and measurement services, whatever the type of entity.1 Search Engine Journal concludes that the clause also reaches businesses that have a Facebook Page or use Business Suite without buying ads.2
Acceptance is implied. Continuing to access or use Meta's products for business or commercial purposes after notice of an update counts as agreement, and a business that disagrees may stop all access or use for those purposes.1
What Meta's texts already provided
A right to crawl already existed in several documents tied to advertising and selling. The table lists four texts cited by Search Engine Journal.2
| Document | Date | What the text provides |
|---|---|---|
| Commercial Terms | March 6, 2024 | no crawling clause2 |
| Self-Serve Ad Terms | August 22, 2025 | placing an ad order gives Meta the right to crawl the advertiser's domains, pages or URLs, including for research and training2 |
| Seller Agreement | updated May 26, 2026 | automated software and test credentials to obtain information from URLs that sellers provide2 |
| Commercial Terms | October 30, 2026 | crawling of the sites associated with the business's products or services, tied to any business use2 |
The difference is the trigger. The ad terms applied to buying an ad, while the new terms tie the permission to any business use.2
02What Meta says and what lawyers read in the text
Nieman Lab reported three legal experts' views on the text on October 15, according to Search Engine Journal.2 Meta spokesperson Andy Stone replied on X on October 16.23
Meta's reply
Andy Stone called the summary of the Nieman Lab report false. He wrote that the update is meant to pull accurate information about products for sale on third-party websites, for ad and content recommendations on Meta's platforms, and that it is not about scraping news sites.2 He added that nothing changes in how Meta uses web crawlers and that content owners can indicate their preferences through robots.txt.3
A Meta spokesperson told Nieman Lab that the new terms will not change how Meta crawls publishers and that content owners can still manage access through robots.txt.2
What Search Engine Journal and PPC Land point out
Search Engine Journal notes that neither the Commercial Terms nor the Self-Serve Ad Terms mention robots.txt.2 It also writes that the new terms do not say whether a business's robots.txt settings limit what it agrees to share.2
PPC Land points out that the paragraph contains no limit to products for sale and does not define the phrase “associated with your products or services.”3 It adds that the terms do not explain what test credentials are.3
The view of the lawyers Nieman Lab consulted
According to Search Engine Journal, the legal experts say the text gives Meta an opening to argue that its commercial terms supersede other blocking instructions.2 James Grimmelmann, a Cornell law professor, said test credentials could technically let Meta sign up as a fake user to get past registration walls.2 Meta declined to answer whether the credentials would bypass paywalled content.2
Attorney Matt Topic said there is no assurance a court would enforce the clause, but that publishers should assume it will.2 Search Engine Journal adds that the European Publishers Council (EPC) is writing to the European Commission about the terms.2
03Setting robots.txt before October 30
Meta presents robots.txt as the control2, and its crawler documentation, updated May 21, 2026, describes each robot and its role.4 Search Engine Journal draws the practical conclusion: a site can block Meta-ExternalAgent and allow Meta-WebIndexer.2
Meta's robots and their roles
The documentation describes five robots.4
| Robot | Role according to Meta | robots.txt compliance |
|---|---|---|
Meta-ExternalAgent | crawls the web for use cases such as training foundation AI models or improving products by indexing content directly4 | a disallow blocks it |
Meta-WebIndexer | improves Meta AI search result quality; allowing it helps Meta cite and link to your content in Meta AI responses4 | a disallow blocks it |
Meta-ExternalAds | crawls the web to improve advertising and other business-related products and services4 | a disallow blocks it |
Meta-ExternalFetcher | fetches individual links at a user's request, including to evaluate and improve agentic AI capabilities4 | may bypass robots.txt |
FacebookExternalHit | crawls the content of a site shared on Facebook, Instagram or Messenger to display its title, description and thumbnail4 | may bypass robots.txt during security or integrity checks |
PPC Land explains that Meta-ExternalAgent both trains models and indexes content for Meta's products under a single robots.txt token, so a site cannot allow indexing while refusing training.3
The lines to add
To block a robot, the documentation says to add a disallow for that robot in robots.txt.4 The table applies that syntax to Meta-ExternalAgent and Meta-WebIndexer.
| Goal | robots.txt lines |
|---|---|
| Block Meta-ExternalAgent | User-agent: meta-externalagentDisallow: / |
| Allow Meta-WebIndexer | User-agent: meta-webindexerAllow: / |
Meta-ExternalAds is written the same way.4 Meta says a change can take up to 24 hours to take effect, because crawlers may cache the file for that long.4
What robots.txt does not cover
Meta-ExternalFetcher and FacebookExternalHit may bypass the file, according to the documentation.4 For the terms, acceptance happens by continuing to use Meta's products after notice of the update.1
The trade-off between blocking and visibility for AI robots is covered in the guide to Generative Engine Optimization. Once the file is changed, the AI crawler access test shows robot by robot whether robots.txt allows them, and the robots.txt generator writes the blocking lines from a list of robots.