HTTP Caching Headers and AI Crawlers: Why Cache-Control and ETag Decide How Fast Updates Reach Citations

HTTP caching headers are the instructions a server attaches to every response that tell a client, whether that is a browser, a CDN, or an AI crawler, how long a copy of a page can be reused before it needs to be re-fetched, and how to check cheaply whether a cached copy is still current. The three that matter most for AI citation are Cache-Control, which sets the reuse window, and ETag and Last-Modified, the pair of validators a crawler sends back on a repeat visit to ask a single question: has this page actually changed since I last read it?

That question matters more for AI citation than it ever did for classic SEO. Recent AEO benchmarking puts the majority of commercial and evaluation-stage AI citations on pages updated within the past six to twelve months, so freshness is now a direct ranking-adjacent signal rather than a housekeeping detail. But an AI engine can only reward an update it actually sees. If your caching headers tell every crawler that a page has not changed, a genuinely rewritten page can sit invisible to the freshness signal for weeks after the edit went live, no matter how much content freshness work went into it.

How AI crawlers actually use caching headers

GPTBot, ClaudeBot, and PerplexityBot do not run a shared browser cache the way a person's browser does. What they do instead, on a repeat crawl of a URL they have already indexed, is send a conditional request: a GET with an If-None-Match header carrying the ETag value they saw last time, or an If-Modified-Since header carrying the last Last-Modified date. If your server determines nothing has changed, it can reply with a 304 Not Modified and an empty body instead of re-sending the full page. That exchange is cheap for both sides, and it is also the exact mechanism a crawler uses to decide, at scale, which of the millions of URLs in its queue are worth spending a full fetch on this cycle.

The practical consequence is that a page which correctly reports a change gets a full re-fetch and a chance to have its updated content re-evaluated for citation. A page that incorrectly reports no change, because a validator was generated wrong or a cache layer intercepted the request before it reached your application, gets skipped, and the crawler moves on believing it already has the current version.

Cache-Control directives that matter

• max-age=<seconds>: how long a cache is allowed to serve a stored copy without checking back at all, no conditional request involved
• no-cache: the cache must revalidate with the origin server on every request before reusing a stored copy, which is the setting that keeps conditional requests flowing for content that changes unpredictably
• no-store: nothing may be cached at any point in the chain, forcing a full fetch every single time, useful for pages that must never be served stale but expensive at crawl scale
• stale-while-revalidate=<seconds>: serves a slightly stale copy immediately while fetching a fresh one in the background, a good fit for high-traffic pages that update occasionally
• private vs public: whether intermediate caches such as a CDN may store the response at all, or only the requesting client

HTTP/1.1 200 OK
Cache-Control: no-cache
ETag: "a1b2c3d4-content-hash"
Last-Modified: Wed, 19 Aug 2026 09:14:00 GMT
Content-Type: text/html; charset=utf-8

ETag and Last-Modified: the two conditional-request signals

ETag and Last-Modified do the same job in slightly different ways. Last-Modified is a timestamp, coarse to the second, that reflects when the resource last changed. ETag is a validator, usually a hash of the content or a version identifier, that changes whenever the underlying bytes change, regardless of timestamp precision. A crawler that already has a copy sends whichever validator it received last time back to the server; if the server confirms the content still matches, it returns a 304 with no body.

# What a re-crawl request looks like
GET /blog/http-caching-headers-ai-crawlers HTTP/1.1
Host: citerank.dev
If-None-Match: "a1b2c3d4-content-hash"
If-Modified-Since: Wed, 19 Aug 2026 09:14:00 GMT

# Server response when nothing changed
HTTP/1.1 304 Not Modified

# Server response after a real content edit
HTTP/1.1 200 OK
ETag: "e5f6a7b8-new-content-hash"
Last-Modified: Fri, 21 Aug 2026 11:02:00 GMT

Why this matters for citation freshness, not just crawl efficiency

Two failure directions both cost you citations, and they pull in opposite ways. Cache headers that are too aggressive, a CDN configured with a long max-age and no revalidation path, can serve a stale cached copy to a crawler for days or weeks after a real edit, so the freshness that should have earned a citation update simply never reaches the retrieval index. Cache headers that never validate correctly, an ETag or Last-Modified that does not reliably reflect content state, waste crawl budget on unnecessary full fetches and can push a site further down a crawler's priority queue, the same underlying constraint covered in page speed and AI crawler timeouts. The fix is not to disable caching. It is to make the validators honest, so a crawler gets a fast, cheap 304 when nothing changed and an immediate, accurate signal the moment something did.

Common misconfigurations that hide freshness from AI crawlers

• A CDN or reverse proxy configured with a long max-age (hours or days) and no revalidation directive, serving a stale cached HTML snapshot to every crawler regardless of what the origin server now holds
• Last-Modified tied to the build or deploy timestamp of the whole application rather than the actual edit date of that specific page, so every page on the site reports the same date whether it changed yesterday or a year ago
• ETag generated per-request from something unstable rather than from a stable content hash, so no re-crawl ever resolves to a 304 and genuine changes cannot be distinguished from noise
• Cache-Control: no-store applied broadly to avoid a stale-content bug, which technically guarantees freshness but forces a full re-fetch on every crawl pass and burns fetch budget the crawler would rather spend elsewhere on the site
• A reverse proxy or WAF layer that strips incoming If-None-Match and If-Modified-Since headers before they reach the application, silently breaking conditional requests even though the origin server is configured correctly

How to audit your caching headers for AI crawler freshness

1. Run curl -I against a live URL and confirm Cache-Control, ETag, and Last-Modified are all present and that the Last-Modified date reflects the actual last content edit, not a deploy timestamp
2. Re-request the same URL with curl -H "If-None-Match: <etag-value>" and confirm the server correctly returns 304 Not Modified when nothing has changed
3. Edit the page, then repeat the same request and confirm the ETag and Last-Modified values both changed and the server now returns a full 200 response
4. Check the CDN or edge configuration for the max-age set on HTML responses specifically, since API and asset caching rules are often set separately and can hide an overly long HTML cache window
5. Cross-check against server logs, per AI crawler log analysis, to confirm GPTBot, ClaudeBot, and PerplexityBot are actually receiving 304s on unchanged pages and full 200s promptly after a real edit, rather than inferring it from configuration alone

curl -sI https://example.com/blog/your-post | grep -iE 'cache-control|etag|last-modified'

curl -sI -H 'If-None-Match: "your-etag-value"' https://example.com/blog/your-post

The fix checklist

• Generate ETag from a stable content hash of the rendered page body, not from a session ID, request timestamp, or randomised element
• Derive Last-Modified from the actual content edit time stored by your CMS or CI pipeline, not from the deploy or build timestamp of the whole site
• Use Cache-Control: no-cache, not no-store, on pages that update unpredictably, so every visit revalidates cheaply instead of forcing a full re-fetch
• Set a short, explicit max-age on CDN-cached HTML, seconds to low minutes rather than hours, if content changes are time-sensitive
• Confirm your reverse proxy or WAF passes If-None-Match and If-Modified-Since through to the origin server untouched
• After any fix, verify with the curl commands above and confirm in server logs that AI crawlers are receiving the expected 304 and 200 responses