Block the AI crawlers and you vanish

Publishers just beat Meta over a paragraph of contract text, and nothing about their business changed. The fight that matters is a different one, and they are losing it by default.

On 5 November, Meta paused its new commercial terms, which would have had every business with a Facebook or Instagram account agree in writing to let Meta crawl its website. It took three weeks. Meta removed the word “train” from the text on 16 October, pushed the start date to 20 November, lost to ten Norwegian media groups in an Oslo court on 28 October, lost again in São Paulo on 3 November, then pulled the text.

Meta was crawling the open web before the proposal. It is crawling it today. It would have crawled it under the new terms. What publishers defended was the right to say they never consented. That has value in a lawsuit or a licensing negotiation. It has none in traffic.

The choice the fight left untouched belongs to every business with a website: let the AI crawlers in, or keep them out. The choice is real. It also barely matters, and the reason is the uncomfortable part.

Two crawls, and a fight over the wrong one

AI companies run two different crawls. The public argument treats them as one.

The training crawl collects pages to build the model. Meta runs it as Meta-ExternalAgent, OpenAI as GPTBot. It is the crawl publishers sue over, and the one Meta’s terms were read as extending. For most of the web it already happened. Blocking it today protects what you publish from now on and nothing you published before.

The retrieval crawl fetches pages to answer a question someone is asking right now. Meta runs it as Meta-WebIndexer, OpenAI as OAI-SearchBot. This is the crawl that replaces the visit, because the reader gets the answer inside the chatbot. It is also the one you cannot comfortably block. OpenAI’s documentation says a site that opts out of OAI-SearchBot will not be shown in ChatGPT search answers. Meta’s says that allowing its indexer helps Meta AI cite and link to you.

A third category removes what control is left. Both companies state that fetches triggered by a user may bypass robots.txt, because a person asked for the page.

The lawsuits are about the crawl that built the model. The business is being lost to the crawl that answers the question.

The dilemma

Block the retrieval crawler and you are absent from the answer. Allow it and you are the answer’s raw material. Here is what allowing pays.

Cloudflare Radar measures it as pages crawled per visitor sent back. From July to September 2026, OpenAI’s monthly figure ran between roughly 250 and 420. Google’s stayed at about four.

The chatbots are offering the deal Google offered, at between one-seventieth and one-hundredth of the rate. The figure blends both crawls, and training is most of the volume. Strip it out and the rate improves. I have found no measurement that puts it near Google’s. The measure also counts only visits that arrive with a referrer, so it undercounts. I left out October, when ChatGPT links reportedly stopped sending one and the figure jumped to 800.

So each option costs little. Allow, and you receive a trickle of visitors. Block, and you give up a trickle. It would be a non-decision except for what everyone else does. If you block and your competitor does not, the chatbot answers the same question from their pages. You have removed yourself and changed nothing. So everyone allows, and everyone is summarised.

That is a prisoner’s dilemma in the textbook sense: the move that is rational for each player is ruinous for all of them. In May I wrote that Google had no rational move except to cannibalise its own clicks. This is the same game from the other side of the table.

It also explains the only thing that moved Meta. It was not a robots.txt file. It was ten media groups in one courtroom. The exit from a prisoner’s dilemma is to stop playing alone, and that exit is open to industries with a trade association and a legal budget. It is not open to you.

The setting is yours. The outcome is not.

The content being defended

Now the part the industry would rather skip.

A University of Maryland audit, first posted in October 2025 and revised in April 2026, ran 186,000 articles published by 1,500 American newspapers in the summer of 2025 through an AI detector. It flagged 9.1% as partly or fully AI-generated. Of 200 flagged articles checked by hand, seven told the reader. Use concentrates in small papers, at 9.3% against 1.7% for those with a circulation above 100,000. The authors note the irony themselves: media groups suing AI companies over training are publishing machine-written articles.

The same industry sells to the companies it sues. News Corp licensed its titles to OpenAI in 2024 for a reported $250 million-plus over five years, part of it paid in credits for OpenAI’s own technology.

So the position is this. The machine writes a share of the article. The publisher objects when the machine summarises it. The licensing fee is paid partly in access to the machine. This was never a dispute about principle. It is a negotiation over price, conducted by companies that are cutting the cost of the product while demanding more for it.

It fails for a mechanical reason. An article a model can write is an article a model can reproduce without visiting you. Automate the writing and you manufacture the exact commodity the dilemma prices at zero.

Publishers are fighting to protect content from the machines that wrote it.

Where the price survives

What still commands a price is whatever the machine could not have produced. There are three such things, and they do not last equally.

• The reporting goes first. Reading filings, monitoring data and cross-checking documents is already agent work. What remains is presence: the courtroom, the factory floor, the call someone did not want to take.
• The source lasts longer. A person with something to lose chooses whom to tell. That choice rests on trust in another person and in an institution that will protect them.
• The name goes last. A name is not a capability. It is someone who can be wrong in public, corrected, sued or fired. A model can write the article. It cannot be held to account for it.

Value retreats to whatever the machine has not reached yet, and the last thing standing is accountability, not skill. None of it is permanent. The line has moved in one direction only, and I see no reason it stops at the newsroom door. The order is what matters, because it tells you what to stop paying for and what to keep.

The publishers that used AI to cut the people who carried that accountability sold the one asset that was holding its price.

If you run a business, not a newspaper

Meta’s terms covered every business account, not only publishers. If your website is your sales funnel, you are in the same game with less leverage.

• Set the crawlers and move on. Block the training bots if the principle matters to you, allow the retrieval bots, and give it ten minutes. No setting changes the arithmetic above.
• Do not build a strategy on being cited. Optimising for the answer engine is the dependency you had on Google, at a fraction of the traffic.
• Sort your content by who could have written it. What a model can produce from public information has no price and will not regain one. Stop counting it as an asset. What exists only because you were there, measured it or signed it is the part to protect, paywall or license.
• Own the channel. A reader who comes to you by name does not pass through anyone’s crawler. That was the conclusion of Google Killed the Click. It has not changed. The cost of ignoring it has.

Your content was valuable while only people could make it. That scarcity is gone, and no court order brings it back.

Sources

• Nieman Lab, Meta pulls down controversial terms of service, 5 November 2026
• Mediebedriftenes Landsforening, Norwegian Media Companies Prevail in Court Against Meta, 28 October 2026
• Reporters Without Borders, Courts in Brazil and Norway block Meta’s bid to exploit news content without permission
• Meta, Meta Web Crawlers
• OpenAI, Overview of OpenAI Crawlers
• Cloudflare Radar, AI Insights: crawl-to-refer ratio
• Russell et al., AI use in American newspapers is widespread, uneven, and rarely disclosed, University of Maryland, 2025
• Press Gazette, News Corp says OpenAI deal marks ‘beginning of beautiful friendship’, 23 May 2024
• Fabio Lauria, Google Killed the Click. I Didn’t See It Coming., May 2026

Fabio Lauria

CEO & Founder, ELECTE

Every week, we explore AI without the hype — using data, analysis and an independent perspective.