RSL, short for Really Simple Licensing, is an open, XML-based standard launched in September 2025 by the RSL Collective that adds a machine-readable License directive to robots.txt, letting a publisher state the exact terms an AI crawler must follow before it trains on, retrieves, or cites a page, instead of the binary allow-or-block choice robots.txt has offered since 1994. Cloudflare's parallel Content Signals Policy, added to the robots.txt specification a few months later, does something narrower but just as consequential for AI citation: it lets a site separate 'you may build a search index from this' from 'you may use this in a live AI answer' from 'you may train a model on this', three permissions that used to be bundled into a single Disallow rule.
For a decade, robots.txt gave publishers exactly one lever: allow a crawler or block it entirely. That was workable while the only thing a bot did with your content was build a search index. It stopped being workable once AI companies started sending crawlers, GPTBot, ClaudeBot, PerplexityBot, Google-Extended, to do three very different jobs: indexing for citation, fetching a page in real time to answer a live prompt, and bulk downloading to train a model. A site owner who wanted to allow the first two and refuse the third had no way to say so in the old syntax. RSL and Content Signals are the two standards now closing that gap, and for AEO purposes they matter because getting them wrong can silently switch off the exact retrieval fetches that turn into citations.
How the RSL License directive works
RSL extends the Robots Exclusion Protocol with a single new line, License:, followed by the absolute URL of an RSL licence file, an XML document that states who may use the content, for what purpose, and under what terms. The directive can sit globally at the top of robots.txt, applying to every crawler, or be scoped inside a specific User-agent block for terms that differ by bot. A compliant crawler is expected to retrieve and honour the referenced licence before it processes the page.
The five licence models RSL supports
- Free - explicit permission with no payment required, for publishers who want citation but no per-crawl fee
- Attribution-only - use is permitted in exchange for visible credit and a link back to the source
- Subscription or purchase - a flat recurring or one-time fee negotiated outside the crawl itself
- Usage-based - a per-crawl or per-training-instance fee, the mechanism behind pay-per-crawl style products
- Custom - a link to the publisher's own licensing page for terms too specific to encode in the standard fields
Enforcement: the Crawler Authorization Protocol
RSL pairs its licence declaration with a Crawler Authorization Protocol that reuses HTTP's standard Authorization header under a new License scheme. A compliant crawler presents a licence token with its request, and the server validates it against the licence issuer's introspect endpoint before serving the page, returning a 401 or 402 status if the token is missing or the terms have not been met.
RSL is a declared-terms system, not a lock. A crawler that simply ignores the licence file can, mechanically, still fetch the page, since nothing forces compliance at the network level. RSL works best paired with actual technical enforcement, a WAF rule or Cloudflare bot management, for any crawler that will not honour it, and its value overall depends on how many AI companies choose to read and respect the licence in the first place.
Cloudflare's Content Signals Policy: three yes-or-no flags
Where RSL builds a full licensing and payment framework, Cloudflare's Content Signals Policy adds one simple field, Content-Signal, that carries up to three comma-separated preferences: search (index this and show it in results with an excerpt), ai-input (use this live to answer a generative query, the retrieval-augmented generation pathway that produces most AI citations), and ai-train (use this to train or fine-tune a model). Each can be set to yes, no, or left unstated.
The two standards are not competitors so much as different layers of the same problem. Content Signals carries no payment mechanism, it is a declared preference that a cooperating crawler reads and respects, already enforceable inside Cloudflare's own bot management for sites on that network. RSL is the heavier framework, covering payment terms and formal licensing, and it already has real publisher backing: the RSL Collective's editorial group includes Condé Nast, Ziff Davis, Yahoo, Fastly, Automattic, and O'Reilly, and outlets including Arena Group, BuzzFeed, USA Today, and Vox Media have separately joined its AI content licensing effort.
The AEO trap: confusing ai-train with ai-input
This is the part that matters most for AI citation strategy. Before Content Signals existed, a publisher who wanted to stop AI training crawlers had only one lever available: block the bot entirely in robots.txt. As covered in Cloudflare's Pay Per Crawl policy, the same crawler user agent is often used for both bulk training fetches and the live retrieval fetches that produce a real-time citation, so a blanket block aimed at training quietly removed the site from citation eligibility too. The training-time versus retrieval-time distinction explains why that trade-off is so costly: retrieval-time visibility is the pathway a site can still influence today, while training-time knowledge is baked in until the next model release.
Content Signals fixes this by moving the distinction from "which bot" to "what the bot may do with what it fetches". A site can now say yes to ai-input and no to ai-train in a single line, keeping itself eligible for live citation while still withholding consent for training, something the old Disallow-only vocabulary could never express cleanly.
The safe default for a site optimising for AI citation is search=yes, ai-input=yes, ai-train=no, or an RSL free or attribution licence stating the same split. That combination keeps a live AI answer able to quote and cite the page while a bulk training run is not permitted to use it.
Should you implement RSL or Content Signals right now
Adoption among the AI companies whose crawlers actually read these directives is still early, RSL only launched in September 2025, and most crawlers do not yet honour an arbitrary XML licence file the way major search engines have honoured robots.txt Disallow rules for thirty years. That makes the two standards worth different amounts of effort right now. Adding a Content-Signal line costs almost nothing, the syntax is a single line, and it is already recognised by Cloudflare's platform for any site behind it. Publishing a full RSL licence file with payment enforcement is worth the heavier lift mainly for publishers with the scale or leverage to negotiate directly with AI companies, or who already use a pay-per-crawl style product.
- Add a Content-Signal line to robots.txt with search=yes, ai-input=yes, ai-train=no, or whatever split matches your actual preference
- Decide deliberately whether AI training crawlers should be able to use your content at all - this is a business decision, not just a technical one
- If you want to license usage rather than simply prohibit it, publish an RSL licence file and add the License: directive
- Confirm the crawlers you are signalling to can actually reach the page in the first place; neither directive helps if GPTBot, ClaudeBot, and PerplexityBot are already blocked
- Re-check crawler behaviour in your server logs periodically, since a declared preference does not guarantee any given crawler will honour it
None of this replaces the more established groundwork. llms.txt is a separate, voluntary file most major AI crawlers still largely ignore, while RSL and Content Signals sit directly inside robots.txt, the file every crawler already has to fetch. Getting the ai-input flag right is a five-minute edit with an outsized downside if it is set wrong, which makes it one of the rare AEO fixes that is both nearly free and genuinely high-stakes.
Run a free CiteRank audit to check whether your robots.txt already blocks the AI crawlers that would otherwise cite you, before you layer RSL or Content Signals rules on top.