AI Web Crawlers Explained: GPTBot, AI Scrapers & LLM-Powered Crawling

GitHub’s robots.txt opens with a deal for GPTBot: come in, wait one second between requests, obey the house rules. Two hundred lines down, the deal for ByteDance’s Bytespider is one line long — Disallow: /. Every site that faces AI crawling makes a choice like this, and on July 1, 2025, Cloudflare started making it for millions of them by default.

An AI web crawler is a bot that copies public pages not to rank them for searchers, but to feed a language model: training corpora, retrieval indexes, real-time answers. The operators, the user-agents, and the rules of engagement changed completely in the last two years. This page names every crawler you’re likely to see in your logs, shows what major sites actually say to them, and sorts the controls that work from the ones that don’t.

Contents: What is an AI web crawler? · The roster · How they differ from Googlebot · What major sites do · Why robots.txt fails · Controls that bite · LLM-powered crawling · FAQ

What is an AI web crawler?

Mechanically, an AI crawler is a search crawler with a different customer. Both keep a frontier of URLs, fetch pages, respect (in theory) the politeness rules of RFC 9309, and store what they find. The difference is what happens downstream. Googlebot feeds a ranking index whose whole product is sending you traffic. GPTBot feeds training runs whose product is a model that answers without you.

The crawl splits into three jobs, and the big operators run a separate user-agent for each:

• Training crawls. GPTBot and ClaudeBot sweep the web for corpora. Your content may shape the next model version. There is no link back and usually no citation.
• Retrieval crawls. OAI-SearchBot, PerplexityBot, and Claude-SearchBot build the indexes that answer engines quote from. These behave most like search: get crawled and you can be cited.
• User-triggered fetches. ChatGPT-User, Claude-User, and Perplexity-User fetch a page because a human asked that specific bot to. These aren’t sweeps at all.

The split is the whole point of the naming. OpenAI’s crawler documentation states the settings are independent: you can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot to keep your pages out of training data. Anthropic’s crawler page (updated April 2026) does the same for its three bots and honors Crawl-delay. Both publish their crawler IP ranges in machine-readable JSON, so you can verify a request really came from them.

The crawlers hitting your site right now

User-agent	Operated by	What it feeds	Honors robots.txt	How to refuse it
GPTBot	OpenAI	Model training	Yes	Disallow: /; IPs at openai.com/gptbot.json
OAI-SearchBot	OpenAI	ChatGPT search index	Yes	Disallow: / removes you from ChatGPT search
ChatGPT-User	OpenAI	Fetches a user asked ChatGPT for	May not apply (user-initiated)	WAF rule on the user-agent
OAI-AdsBot	OpenAI	Ad landing-page review	Visits only pages submitted as ads	Disallow: /
ClaudeBot	Anthropic	Model training	Yes, supports Crawl-delay	Disallow: /; IPs at claude.com/crawling/bots.json
Claude-SearchBot	Anthropic	Claude search index	Yes	Disallow: /
Claude-User	Anthropic	Fetches a user asked Claude for	Yes (per Anthropic)	Disallow: /
PerplexityBot	Perplexity	Answer-engine index	Claims yes; see below	WAF managed rule
Perplexity-User	Perplexity	Fetches a user asked Perplexity for	Claims yes	Disallow: /
CCBot	Common Crawl	Open crawl corpus that trains many models	Yes	Disallow: /
Bytespider	ByteDance	TikTok’s AI and search ambitions	Widely reported not to	Network block, not robots.txt
Applebot-Extended	Apple	Apple Intelligence training	Yes	Disallow: / (Applebot itself still crawls for Siri/Spotlight)
Amazonbot	Amazon	Alexa and AI services	Yes	Disallow: /
Google-Extended	Google	Gemini and Vertex training	Not a crawler; a token	Blocks training reuse, not crawling

One row deserves its own paragraph. Google-Extended is the trap people fall into. It is not a crawler and never visits your site; it’s a robots.txt token that tells Google not to reuse the data Googlebot already gathered for Gemini training. Block it and your search ranking is untouched. Block Googlebot and you vanish from search. Publishers who wanted “no Gemini training but yes to search” had no way to say that before this token existed in 2023. Its arrival is what taught the market that one User-agent line per purpose was the negotiation format.

How AI crawling differs from search crawling

The Google deal was explicit: copy my content, get traffic back. It worked well enough to fund the web for twenty years, and even it decayed: Cloudflare’s measurement puts traffic-per-crawl at 10x worse than a decade ago. Then the AI deal arrived with no traffic clause at all. Cloudflare’s crawl-to-refer data (June 2025) found it is roughly 750 times harder to get traffic from an OpenAI crawl than from the Google of old, and about 30,000 times harder from an Anthropic crawl. One company crawls you a thousand times to send one reader.

Volume and rhythm differ too. A search crawler revisits pages on a freshness schedule measured in days to months, because rankings tolerate staleness. An answer engine’s index is closer to a cache: stale answers are wrong answers, so AI crawlers re-request popular pages far more aggressively. TollBit, which runs a paywall for bot traffic across its publisher network, reported one AI bot visit for every 31 human visits by Q4 2025. That traffic arrives, takes bytes, and rarely returns a session.

Attribution is the quiet difference. Google’s ranking contract, however frustrating, at least guarantees you a visible link. Answer engines generate one paragraph. Whether you get cited depends on their citation policy, not on any web standard, and a page that gets quoted without a link is indistinguishable from a page that got ignored.

What major sites actually do about it

I pulled the live robots.txt files from seven major sites on 2026-10-16. The spread of strategies is the real data:

Site	What their robots.txt says about AI bots
nytimes.com	Disallow: / for GPTBot, ClaudeBot, Claude-User, Claude-SearchBot, ChatGPT-User, CCBot, Bytespider, Amazonbot, Applebot-Extended, and more
bbc.co.uk	Blocks the same family, including GPTBot, ClaudeBot, CCBot, Bytespider, Amazonbot, Applebot-Extended, Google-Extended
github.com	GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai, and PerplexityBot get Crawl-delay: 1 plus a curated Allow list; Bytespider gets Disallow: /
spiegel.de	GPTBot, ClaudeBot, anthropic-ai, cohere-ai, CloudVertexBot, Bytespider blocked; Claude-SearchBot and Claude-User explicitly test-enabled, with comments in German
en.wikipedia.org	No AI-bot directives at all among 35 user-agent groups
stackoverflow.com	Robots.txt itself is behind bot filtering; my curl got HTTP 418
arstechnica.com	HTTP 403 to curl; filtering happens before robots.txt

Three strategies, in ascending order of effort. The New York Times and BBC block training and answer-engine crawls wholesale. GitHub’s policy is the one worth copying if you want AI referrals: declared, verifiable crawlers get in at a throttled rate, while the one operator with a reputation for ignoring the rules is out. Der Spiegel runs the finest-grained version I’ve seen: block the training bots, experimentally allow the search and user-triggered bots, and annotate the file so humans understand why.

Wikipedia blocks nothing and gets hammered for it, which is a values decision, not an oversight. And Stack Overflow plus Ars Technica demonstrate the endgame: when enough crawlers misbehave, even the robots.txt file, the politeness document itself, sits behind a challenge that a plain HTTP client can’t pass.

Why robots.txt alone doesn’t hold

Robots.txt is a suggestion. RFC 9309 defines the syntax; nothing defines enforcement, and no authentication binds a request to a user-agent string. Anyone can write PerplexityBot in their fetcher’s headers, or write Mozilla/5.0 Chrome and look like a MacBook.

How much this actually happens is now measured. TollBit’s State of the Bots report found 30% of AI bot scrapes on its network bypassed explicit robots.txt permissions in Q4 2025, up from roughly 3% a year earlier. A tenfold jump in one year.

The canonical case is Perplexity. In August 2025, Cloudflare documented what its network saw: alongside the declared Perplexity-User (20–25 million requests a day), an undeclared crawler impersonating Chrome on macOS (3–6 million a day) hit tens of thousands of domains, rotating IPs and autonomous systems when blocked. Cloudflare had set up fresh domains with blanket Disallow rules; Perplexity answered detailed questions about their content anyway. The company disputed the findings. Cloudflare delisted Perplexity as a verified bot and wrote signature rules against the stealth crawler.

The same Cloudflare experiment ran against ChatGPT as a control, and ChatGPT passed it cleanly: fetched robots.txt, saw Disallow, stopped, sent nothing else. So this isn’t an “all AI companies cheat” story. It’s an honor-system story. Most declared crawlers honor the file; the honor system’s enforcement arm is whatever your host can fingerprint; and roughly a third of the traffic by volume no longer pretends to honor anything.

Two more gaps close the loop. User-triggered bots are exempt by design: when a human asks ChatGPT to read a page, OpenAI notes that robots.txt rules “may not apply” to that fetch. And robots.txt is not law, anyway; it has never been more than an industry norm. What actually binds a scraper is contract and copyright, and that story runs through the courts documented in is web scraping legal.

Controls that actually bite

Edge enforcement, now the default. On July 1, 2025, Cloudflare flipped the switch it calls Content Independence Day: AI crawlers are blocked by default on every plan, including free, unless the site owner opts in. A pay-per-crawl beta lets publishers charge bots per request instead. Within five weeks, the company reported over 2.5 million sites had opted to fully disallow AI training. If your site sits behind Cloudflare, this decision is one toggle, and the default just changed under you.

Fingerprint verification. The reason stealth crawlers can be caught at all is that Cloudflare scores requests on TLS fingerprints, IP reputation, and behavioral signals, not just headers. Akamai and Fastly sell equivalents. This is the only control class that works against the third of traffic that ignores robots.txt, and it’s why my curl got a teapot joke from Stack Overflow.

Authenticated agents. Web Bot Auth is the standards-track fix: crawlers sign their HTTP requests with a certificate, so a user-agent string becomes a claim you can verify instead of a costume. ChatGPT Agent already signs its requests. An IETF working group (aipref) is drafting vocabulary for expressing AI preferences in robots.txt itself, which would give tokens like Google-Extended a standard home.

Your own logs. Before any of that, measure what’s actually hitting you. One command over a day of access logs tells you which of these bots showed up:

grep -Eio "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|CCBot|Bytespider|Amazonbot|Applebot-Extended|Google-Extended" access.log | sort | uniq -c | sort -rn

Whatever shows up with thousands of hits and no corresponding traffic in analytics is your real answer to “should I block this.” On shared hosting without WAF access, a robots.txt block plus rate limiting on the known IP ranges (openai.com/gptbot.json and Anthropic’s bots.json publish theirs) is the honest ceiling of what you can enforce.

LLM-powered crawling: the other kind of AI based web crawler

Everything above describes crawlers run by AI companies. The phrase “ai based web crawler” has a second meaning: crawling where the model sits in your pipeline. The crawler doesn’t feed a model; the model drives the crawl. Three patterns dominate:

Agents that browse. ChatGPT Agent, Claude’s browser tool, and operator-style products navigate pages like a user would on request. These produce user-triggered traffic, the ChatGPT-User and Claude-User kind: one fetch per human intent, not a sweep.

LLM-assisted extraction. A classical crawler fetches and cleans; an LLM turns the page into a JSON schema, picks the main content, or synthesizes selectors when the site’s markup shifts. The crawl logic is 2015 technology; the parsing is 2026 technology. This is the pattern behind every AI web scraper pitch, and it changes the payload requirement: what you hand the model should be clean markdown, not raw HTML. A 392KB page of nav bars and cookie banners can cost 100,000 tokens where the article is 500; markdown vs HTML for LLMs measures the gap.

LLM-planned crawling. The model decides what to fetch next — follow this link, skip that one, this site looks promising — turning the crawl frontier into a reasoning problem. Early days, expensive per page, useful for narrow research tasks over a few dozen sites rather than thousand-page crawls.

One practical note if you build this yourself: the politeness rules don’t change because your crawler is smart. Tools in this space differ in how they treat robots.txt and rate limits (the local-first scrapers I use enforce both by design), and your own crawlers should too, because the enforcement arms described above fingerprint sloppy automation just as fast as they fingerprint Perplexity. If you’re evaluating tooling, the honest starter question is whether it shows you the robots verdict per URL before fetching; a concrete walkthrough is in scraping a site with ChatGPT, and what the cleaned output is worth downstream depends on AI-ready data.

FAQ

Is GPTBot a search engine crawler? No. GPTBot feeds model training and has no effect on Google or Bing rankings. OAI-SearchBot is the one that controls whether you appear in ChatGPT’s search results. Block GPTBot and you lose nothing in search; block OAI-SearchBot and ChatGPT search stops citing you.

Does blocking AI crawlers hurt SEO? Not directly. Google ranks you from Googlebot’s crawl; Google-Extended only stops reuse for Gemini training. The indirect cost is discovery: answer engines are a growing referral surface, and blocking every AI bot removes you from all of them.

What’s the difference between GPTBot and ChatGPT-User? GPTBot is the automatic training crawl. ChatGPT-User fetches a page because a specific user asked ChatGPT to read it, so it’s user-initiated and OpenAI says robots.txt rules may not apply to it. Treat them as different decisions in robots.txt.

Can I charge AI crawlers for access? The infrastructure exists in beta: Cloudflare’s pay-per-crawl bills bots per request (HTTP 402) on behalf of publishers. Licensing deals between individual publishers and AI labs are now common. There is no open standard that a random site can switch on yet.

Which AI crawler is the most aggressive? By reputation and by who gets banned: Bytespider. Every site in my seven-file check that blocks AI bots at all blocks ByteDance’s crawler, and it’s the one GitHub singles out while admitting OpenAI and Anthropic. TollBit’s compliance data suggests stealth crawlers are the bigger aggregate problem.

The decision, in one pass

If you want AI referrals, do what GitHub does: allow the declared, IP-verifiable training and search bots at a crawl delay, ban the rest, and check logs monthly. If you sell content, block everything at the edge and let pay-per-crawl negotiate. If you just found this page because your analytics shows a wall of GPTBot, run the log command above before deciding anything. The right answer depends on which bots you actually get, and that number takes thirty seconds to count.