Pay per crawl is Cloudflare's opt-in system that lets a website charge AI bots a fee, as low as $0.01 per successful fetch, before they can retrieve a page. Instead of the binary allow-or-block choice offered by robots.txt, a participating site returns an HTTP 402 Payment Required response to any crawler that has not cryptographically signed a request and agreed to pay the declared rate. For anyone tracking AI citation, this is the biggest change to crawler-access economics since GPTBot first appeared in robots.txt files.
Cloudflare has continued hardening the policy through 2026. From 15 September 2026, AI training and agent bots are blocked by default on ad-supported pages for any new domain onboarded to Cloudflare, unless the site owner explicitly allows them or the bot agrees to pay. For CiteRank users, this is not only a publisher-licensing story. It changes what crawler access, one of the core signal categories in every AEO audit, actually means in practice.
What pay per crawl actually does
A participating site declares a price per successful crawl through Cloudflare. A bot that wants the page must authenticate the request (typically via signed HTTP message signatures under the emerging Web Bot Auth standard) and pre-declare that it will pay. If it does, Cloudflare bills the crawler operator and lets the fetch through. If it does not, or cannot prove who it is, the origin server returns a 402 and the crawler gets nothing: no HTML, no text to extract, no chance of a citation.
- Training crawlers that harvest pages to improve a future model version, with no connection to a specific user query
- Agent or assistant crawlers that fetch a page in real time because a user asked a question the model needs to answer right now
- Index-building crawlers that continuously refresh a retrieval store used later to answer many different queries, closer in behaviour to a search engine crawler
The mixed-use crawler problem
Many bots blur these categories. The same user agent can be used for both bulk training passes and live, query-triggered fetches, which is exactly why Cloudflare's September policy singles out 'mixed-use' crawlers on ad-supported pages for a default block. Site owners now have to decide, bot by bot, which purpose they are willing to allow for free, which they want to charge for, and which they want blocked outright, rather than making one decision for an entire user agent.
Why this matters for AI citation, not just licensing revenue
If the default from September is to block AI bots on ad-supported pages, then failing to explicitly allow the query-time bots that retrieve content to answer a live user question means a site silently drops out of citation results. The failure looks identical to the crawler-access problems already covered in robots.txt for AI crawlers: the page renders fine for a human visitor and for Googlebot, but the specific bot an AI engine sends at answer time never gets past the gate.
Conflating a training-bot block with a retrieval-bot block is the single most likely way pay per crawl damages AI citation. Blocking GPTBot's training pass costs you nothing in citation terms. Blocking OAI-SearchBot, which fetches pages live to answer a user's question, can cost you the citation entirely.
Three crawler-access postures, and which one protects citation
- Block every AI bot: maximises control over training data and content licensing, but forfeits AI citation entirely since retrieval bots cannot read the page either
- Allow every AI bot free: maximises citation opportunity across ChatGPT, Perplexity, Claude, and Gemini, at the cost of no licensing leverage over large training-focused crawls
- Selective access: charge or block declared training crawlers, keep declared retrieval and agent crawlers on a free allow list, which preserves citation while still asserting some control over bulk training use
Training versus retrieval: get the distinction right before you flip the switch
In practice this means treating GPTBot, ClaudeBot's training crawl, and PerplexityBot's index-building pass as one category, and treating OAI-SearchBot, Claude's user-triggered fetch, and Perplexity's live-query fetch as a separate category that should stay allowed regardless of what pricing policy applies to training. The naming conventions differ by vendor and keep changing as platforms split their bots more granularly, so this list needs re-checking rather than set once and forgotten.
A checklist before you enable pay per crawl
- Pull recent server logs and list every distinct AI bot user agent actually hitting the site, using the method in AI crawler log analysis
- Classify each bot as training, retrieval or agent, and index-building, based on the vendor's published documentation rather than assumption
- Add every retrieval and agent bot to an explicit allow list at zero cost, independent of whatever pay-per-crawl rate is set for training bots
- Only apply pricing or blocking to bots explicitly documented as training crawlers if licensing leverage is the actual goal
- Confirm robots.txt, Cloudflare bot management rules, and pay-per-crawl settings agree with each other rather than silently overriding one another
- Re-check the bot list monthly, since vendor bot names and Cloudflare default policies have both changed multiple times in 2026
If there is no active AI licensing negotiation underway, the simplest safe default in July 2026 is to leave every retrieval and agent bot fully open while only pricing or blocking bots explicitly declared as training crawlers.
The economics behind the shift
Cloudflare Radar data shared in June 2026 put automated requests at 57.5% of HTML traffic across its network, with 51.8% of verified bot traffic going to AI model training rather than search indexing or live retrieval. Pay per crawl is evolving into a broader 'Pay Per Use' model, helped by Cloudflare's acquisition of Human Native, which converts unstructured web content into structured datasets for AI licensing deals. So far only Ceramic.ai and You.com are named Pay Per Use partners; OpenAI, Anthropic, and Google DeepMind have not committed to paying under the scheme.
That last point matters for anyone tuning crawler access for citation rather than licensing revenue. If the engines you most want to be cited by have not agreed to pay, a block-by-default policy without an explicit allow list can quietly shut them out, even though nobody intended to block them. Treat crawler-access settings as something to re-audit monthly through the rest of 2026, alongside the declarative signals in llms.txt, rather than a switch you flip once and leave alone.