For over twenty years, the web has operated on an unwritten pact: Google and other search engines indexed publishers' content for free, displaying snippets and summaries on their pages, but in return they sent massive traffic to the original sites. Publishers accepted this “use” of their content because they received visibility, readers, and, as a result, advertising revenue.

Google News perfected this model: it aggregates news from thousands of sources, displays headlines and short excerpts, but each click takes the user to the original site. A win-win balance that supported the digital information ecosystem.

Now imagine you run a news site and discover that every day thousands of AI bots scan your articles, use the content to feed their models, and serve complete answers to users, without ever sending traffic back to your site. AI that doesn't generate links, and doesn't generate clicks: it responds directly. Welcome to the reality of 2025.

According to Cloudflare data, by mid-2025, 80% of AI crawling activity will be for model training, while referrals to publishers (especially from Google) are declining dramatically. The model that has worked for twenty years is crumbling: Google itself, with its AI Overviews, is sending less and less traffic to the original sites. The “crawl-to-refer” ratio shows shocking numbers: Anthropic has a ratio of 38,000 scans for every visitor sent to the site, while OpenAI reaches 1,700:1.

But what if bots could become paying customers instead of just free consumers of content? It's time for a new deal: if AI can no longer guarantee return traffic as Google did, it must at least pay for access to content.

In July 2025, Cloudflare announced that it had become the first internet infrastructure company to block AI crawlers by default, simultaneously introducing the “Pay Per Crawl” system. What's new? Every new domain that registers with Cloudflare is now explicitly asked if it wants to allow AI crawlers to access its content.

But the real game-changer is the marketplace where publishers can request compensation from AI companies every time one of their pages is scanned. No longer just “allow” or “block,” but a third option: “charge.”

Even before Cloudflare, TollBit had begun building this infrastructure. The startup, which recently raised $24 million in Series A funding, has created a platform where AI bots can pay websites directly to use their content.

TIME and Adweek are among the first customers experimenting with this model, discovering that AI represents a new category of customers with specific needs.

According to a detailed analysis, TollBit could already be generating approximately $71 million per year for its publishing partners. Here's how:

Using a typical CPM of $15 for content sites, this translates to $195,000 per day in potential revenue for the TollBit ecosystem, or over $71 million per year.

TollBit suggests two pricing tiers:

The difference can be substantial, making this a particularly attractive model for premium and specialized content.

A TollBit analysis of 160 websites revealed that AI companies scanned an average of 2 million times in Q4 2024. Each page was scanned an average of 7 times.

Toshit Panigrahi, co-founder of TollBit, explains: “The bot traffic generated by these AI platforms is almost equivalent to the bot traffic of search engines that have been around for 20 years, which is incredible.”

DataDome has partnered with both TollBit and other platforms to offer an integrated solution: real-time protection from malicious bots and monetization of compliant ones.

Their data shows that AI traffic has tripled in just six months, from 2.6% to 8.2% of all verified bot traffic.

Skyfire represents a different approach, building an entire payment network for autonomous AI agents. With $8.5 million raised and partnerships with companies such as DataDome, Skyfire aims to become the standard payment infrastructure for the AI economy.

Before analyzing the numbers, it is important to note that Pay Per Crawl is not convenient for everyone. For sites with limited traffic, allowing free access to AI bots can be strategically more advantageous: it increases the discoverability of content in AI systems, builds relationships with emerging platforms, and positions the site as “AI-friendly” for future opportunities.

Direct monetization only becomes attractive when volumes justify the management complexity and implementation costs.

Prashanth Chandrasekar, CEO of Stack Overflow, sums it up perfectly: “Community platforms that power LLMs should be compensated for their contributions so they can reinvest in their communities.”

With AI-to-AI commerce projected to reach $46 billion over the next three years, publishers who position themselves early in this ecosystem could find themselves with a significant competitive advantage.

Pay Per Crawl is no longer science fiction, but operational reality.

For publishers, the question is not whether this model will take hold, but if and when they can adapt to capture its value.

The uncomfortable truth: not everyone should monetize right away. For sites with limited traffic (<50K visits/month), maintaining free access to AI bots may be more strategically advantageous, building visibility and relationships for future opportunities. The break-even point for direct monetization is generally reached with significant traffic volumes.

Projections show attractive potential revenues for medium and large publishers, especially for quality content and substantial traffic. However, success will depend on the ability to:

For publishers with substantial traffic who are willing to experiment, 2025 could be the year when AI ceases to be just a cost and finally becomes a business opportunity. For everyone else, it may be time to lay the groundwork to capture this opportunity in the future.

Main sources:

Welcome to Electe’s Newsletter - English

This newsletter explores the fascinating world of how companies are using AI to change the way they work. It shares interesting stories and discoveries about artificial intelligence in business - like how companies are using AI to make smarter decisions, what new AI tools are emerging, and how these changes affect our everyday lives.

You don't need to be a tech expert to enjoy it - it's written for anyone curious about how AI is shaping the future of business and work. Whether you're interested in learning about the latest AI breakthroughs, understanding how companies are becoming more innovative, or just want to stay informed about tech trends, this newsletter breaks it all down in an engaging, easy-to-understand way.

It's like having a friendly guide who keeps you in the loop about the most interesting developments in business technology, without getting too technical or complicated

Subscribe to get full access to the newsletter and publication archives.