What Is a Web Crawler (Spider)?

A web crawler is a program that visits web pages, follows the links on each one, and records what it finds, page after page. Search engines run the famous ones. If you have ever wondered what is a web crawler doing on your server at 3 a.m., the short answer is: working through a queue of URLs it built itself, one page at a time.

This page defines the term properly, runs a real crawl you can reproduce in 16 lines of bash, and shows how sites tell crawlers where they may and may not go.

So what is a web crawler, exactly?

The definition above hides four verbs: pick a URL from a queue, fetch the page, record it, pull out every link. Each link goes back into the queue. Repeat until the queue is empty or a stop condition fires. That loop is the whole machine.

“Spider” is the same thing, older slang: the program “spiders” the web the way a spider works its web. Both names stuck. One of the first crawlers, Matthew Gray’s World Wide Web Wanderer, launched in 1993. Google’s crawler is Googlebot, Bing runs Bingbot, and the Internet Archive has been crawling for the Wayback Machine since 1996.

What the crawler records depends on who runs it. Search bots, archive bots, AI bots, SEO site-auditors, and focused crawlers that chase a single topic all run the same loop; they differ in what they keep. A search engine stores enough to build its index: text, titles, links, freshness signals. Common Crawl stores everything it can and gives it away; its September 2026 crawl (CC-MAIN-2026-39) pulled 2.17 billion pages into 361 TiB from 40.4 million hosts. What happens after collection is a separate job, and crawling is often confused with it.

The crawl loop, run for real

Talk is cheap. Here is a working crawler in 16 lines of bash, pointed at quotes.toscrape.com, a site built for exactly this kind of practice. It keeps a queue and a visited list, fetches the front of the queue, and pushes every same-site link it finds onto the back.

#!/usr/bin/env bash
seen=" "
queue=("http://quotes.toscrape.com/")
found=0
for ((n=0; n<6 && ${#queue[@]}; n++)); do
  url=${queue[0]}; queue=("${queue[@]:1}")
  html=$(curl -s -A "demo-crawler/0.1" "$url")
  echo "fetched $url ($(printf %s "$html" | wc -c) bytes)"
  for p in $(printf %s "$html" | grep -o 'href="/[^"]*"' \
            | sed 's/href="//;s/"//' | sort -u); do
    u="http://quotes.toscrape.com$p"
    [[ $seen == *" $u "* ]] && continue
    seen+="$u "; queue+=("$u"); ((found++))
  done
done
echo "discovered $found new URLs; queue still holds ${#queue[@]}"

Real output from a run on 2026-10-16:

fetched http://quotes.toscrape.com/ (11064 bytes)
fetched http://quotes.toscrape.com/ (11064 bytes)
fetched http://quotes.toscrape.com/author/Albert-Einstein (286 bytes)
fetched http://quotes.toscrape.com/author/Andre-Gide (276 bytes)
fetched http://quotes.toscrape.com/author/Eleanor-Roosevelt (290 bytes)
fetched http://quotes.toscrape.com/author/J-K-Rowling (278 bytes)
discovered 49 new URLs; queue still holds 44

Six fetches, 49 URLs discovered, 44 still queued. Pages link to more pages, so a crawl’s queue grows faster than it drains; that ratio is the entire economics of crawling, and it is why a careless crawler can hammer a small site.

Now look at line two: the homepage got fetched twice. Its logo links to /, so the link grep re-discovered it, and nothing marks a fetched URL as seen. The demo checks the visited list before enqueueing but not before fetching. Every crawler author ships this bug once; production crawlers consult the visited set before the fetch, not just before the push. One wasted fetch in six sounds harmless until the crawl is 2.17 billion pages, which was Common Crawl’s September 2026 haul.

The fetch order is breadth-first: homepage first, then author pages in the order the homepage linked them. Spreading attention across a site beats tunnelling down one path.

What the demo leaves out

The demo is honest about the loop and silent about everything that makes crawling a discipline.

Politeness comes first. A crawler must identify itself (the -A flag above sets a user agent), rate-limit per host, and honor robots.txt before every fetch. Wikipedia’s robots.txt is the best free exhibit of why: it permanently bans MJ12bot, and the receipt stays pinned in the file: “ignoring 429 ratelimit responses, claims to respect robots.” Sites remember who behaved.

Then the engineering that only shows up at scale: retries with backoff when a server hiccups, JavaScript rendering for pages that build themselves in the browser (see what a headless browser is if that’s new), duplicate detection across millions of URLs, and scheduling so popular pages get refreshed more often than dead ones. A production crawler is the 16-line loop above plus a few thousand lines of exactly those concerns. If you want to build one properly, the full crawler walkthrough covers the architecture; the open-source crawlers compared shows who else has already built it.

For day-to-day crawling I use Magpie, a local CLI crawler with robots.txt respect and rate limiting built in. The loop it runs is the one above. When a crawl misbehaves, it is almost always the queue or the politeness rules, never the fetch.

Crawling is not scraping

The two terms get used interchangeably and they shouldn’t be. A crawler answers “what pages exist and what do they contain?” A scraper answers “given this page, what is the price, the title, the rating?” Crawlers discover and download. Scrapers parse and extract specific fields. The same tool often does both, in sequence: crawl to collect, scrape to extract. The crawling vs scraping comparison breaks the differences down page by page.

How sites say yes and no to crawlers

robots.txt is the handshake. A crawler that claims to be polite fetches /robots.txt first and obeys the User-agent blocks inside. Real rules from two real files, fetched 2026-10-16:

python.org bans offline-site rippers by name:

User-agent: HTTrack
User-agent: puf
User-agent: MSIECrawler
Disallow: /

Wikipedia bans one by name, with the evidence attached:

# Observed spamming large amounts of https://en.wikipedia.org/?curid=NNNNNN
# and ignoring 429 ratelimit responses, claims to respect robots:
# http://mj12bot.com/
User-agent: MJ12bot
Disallow: /

Three details worth knowing. First, no robots.txt at all means “no restrictions”: quotes.toscrape.com returns a 404 for /robots.txt, so a polite crawler may crawl the whole site. Second, robots.txt controls fetching, not indexing: a page disallowed in robots.txt can still appear in search results if someone links to it; indexing is controlled separately by meta robots tags. Third, robots.txt is voluntary. Well-behaved crawlers (Googlebot, Bingbot, Common Crawl’s CCBot, OpenAI’s GPTBot) obey it, and you verify claimed identities by reverse DNS, not by trusting the user-agent string. The crawlers that ignore it are the reason rate limiting exists on every serious server.

Sites can also help crawlers instead of blocking them, by publishing an XML sitemap that lists the URLs worth crawling. Google’s crawler reads it before it starts guessing. And the AI crawlers now collecting training data, GPTBot and its peers, follow the same robots.txt rules, with the current blocklist in the AI crawlers explainer.

FAQ

Is a web crawler the same as a web scraper? No. A crawler discovers pages by following links and downloads them. A scraper extracts specific fields from pages it is handed. Many tools do both in sequence: crawl to collect pages, then scrape each one for the data you actually want.

Is Googlebot a web crawler? Yes. Google’s, and the most famous one. It feeds the search index, identifies itself as Googlebot, and publishes its crawling documentation. You can confirm a visitor really is Googlebot with a reverse DNS lookup on its IP, since user-agent strings are trivial to fake.

Are web crawlers legal? Crawling publicly accessible pages is generally lawful in the US (the hiQ v. LinkedIn line of cases), but scraping private, paywalled, or authenticated content is a different matter, and rules differ by country. Ignoring robots.txt isn’t automatically illegal, but it weakens your position badly. This isn’t legal advice; ask a lawyer for your specific case.

How do I stop crawlers from visiting my site? robots.txt asks polite crawlers to stay out, and the major ones honor it. For crawlers that ignore robots.txt you need enforcement: rate limiting, bot filtering at the firewall or CDN, or a reverse-DNS allowlist. A meta robots tag stops indexing but still allows the fetch itself.

How often do crawlers revisit a page? It depends on how popular and how changeable the page is; crawlers spend attention where freshness matters. Google doesn’t publish per-URL schedules. Common Crawl revisits in large monthly sweeps.

Try it on your own site

The demo crawler above is safe to run as written: six fetches, one request at a time, a labeled user agent. Run it against any small site you have permission to crawl and watch the queue grow. Then read your own site’s /robots.txt the way a crawler does. It is the first conversation you have with every spider on the web, and some of them, like Wikipedia’s banned MJ12bot, get named and shamed in it forever.