Web Scraping vs Web Crawling: The Difference

Web crawling finds pages; web scraping pulls fields out of pages you already have. That is the entire web scraping vs web crawling question, and it trips people up because one tool usually ships both jobs. I ran each against the same 1,000-book demo site today; here is what each one produced, where each one breaks, and which one you need.

The verdict

Point at a page you can name and ask “what does it say?” That is scraping. Point at one URL and ask “what else is on this site?” That is crawling. If you know the URL, scrape it. If you don’t, crawl first and scrape second; most real projects chain the two.

	Web crawling	Web scraping
Question it answers	What pages exist here?	What does this page say?
Input	One seed URL plus scope rules	One URL (or a list a crawl produced)
Unit of work	The URL	The field
Output	URLs, usually with page content attached	Values: price, title, stock, rating
Code that matters	Queue, dedupe, scope, politeness	Selectors, parsing, types
What kills it	Infinite URL spaces: session IDs, filters, calendars	Redesigns: renamed classes, moved nodes
Etiquette document	robots.txt, written for crawlers	Same politeness norms, no crawler-style rulebook
Classic examples	Googlebot, Common Crawl, site auditors	Price monitors, change watchers, lead extractors

Web scraping vs web crawling, measured on one site

books.toscrape.com is a sandbox built for exactly this: a fake bookstore with 1,000 product pages and no anti-bot defenses. Same site, two programs, both written with Python’s standard library only.

The crawler starts at the homepage with a 25-page cap and a 1 request-per-second ceiling. Its whole job is bookkeeping:

fetched 25 pages in 58.5s (0 errors)
raw hrefs seen: 1946
unique URLs in frontier after 25 fetches: 377
queue still holding: 352 (crawling could continue)

Twenty-five fetches in, the frontier knows 377 URLs and I stopped it while 352 were still queued. That ratio is the crawler’s signature: output grows even when you stop it, and most of what it “found” it never fetched. (The demo above doesn’t filter static files, which is why CSS and favicon links are in there. Deciding what not to fetch is half of real crawler work; what a web crawler is covers the full anatomy.)

The scraper handles one product page from the same store. One request, no queue:

1 request, 1.73s, 9,275 bytes fetched
  title: A Light in the Attic
  price: £51.77
  UPC: a897fe39b1053632
  Availability: In stock (22 available)
  Number of reviews: 0
  (abridged: 10 keys in total)

Look at the shapes. The crawler’s output is a list of URLs. The scraper’s output is ten keys, and not one of them is a URL. If your deliverable is “prices for all 1,000 books”, you need both: the crawl produces the product-page list, the scraper runs on each entry. If your deliverable is “the price of this one book”, crawling is waste — one request does it.

Where the engineering goes

The two jobs are hard in different places, which is why job posts treat them as different skills.

A crawler is a queue with rules around it. Scope: same host, subdomains only, skip /tag/? Dedupe: that product page is reachable from the category page, the search page, and the “next” link, so fetch it once. Politeness: the demo crawl above spent 58.5 seconds on 25 pages, mostly waiting on purpose. Stopping: a page cap, a depth cap, or a URL pattern. Get any one wrong and you either miss the site or hammer it. At search-engine scale this becomes its own discipline: Common Crawl’s March 2026 archive alone holds 1.97 billion pages, 344.64 TiB uncompressed.

A scraper is parsing logic. Where is the price? Is it "£51.77" or 51.77 with the currency in a separate node? Is “In stock” in a <p> or inside an icon tag next to it? None of that exists in a crawler, and none of the crawler’s queue machinery helps here. The hands-on guides live almost entirely in this second problem.

How each one breaks

Crawlers die by expansion. My demo strips query strings from every URL it queues: one line, _replace(fragment="", query=""). Do that on a faceted-nav store and you skip every filtered view. Skip it on a site that issues session IDs (?sessionid=...) and every response “discovers” a brand-new URL forever; the crawl never ends and never repeats, which is the worst of both. Calendars and infinite pagination are the same trap. The bug is never “the crawler is slow”. It is “the URL space is bigger than you thought”.

Scrapers die quietly. In the run above, my parser caught the availability <p> and got an empty string, because the visible “In stock (22 available)” text sits in an <i> icon tag beside it. The script exited zero and printed a blank field. I only noticed because I read the output; a cron job wouldn’t. A month later the store renames price_color to product-price and ten thousand rows come back blank instead of broken. This is the scraping failure mode people learn the hard way: redesigns don’t throw errors, they emit nulls. (It is also why self-healing extraction exists; the tools list flags which tools attempt it.)

One boundary note: none of this applies when the data has a better front door. If the site offers an API, scraping vs an API is the comparison that actually matters.

Both jobs share miseries that don’t differentiate them: JavaScript-only pages, rate limits, blocks when you push volume. Vendors harp on those because they sell the unblocking. Those problems are orthogonal to the scraping-vs-crawling question, so they get this paragraph and nothing more.

robots.txt was written for crawlers

The clearest external evidence that these are different jobs: sites write rules to crawlers, not scrapers. CNBC’s robots.txt opens with “to prevent the crawling and indexing of certain parts of your site by web crawlers and spiders”. The document is a crawler contract: which paths may be fetched, by which bot, at what pace (Crawl-delay). My demo reads it before the first request, and books.toscrape.com returns a 404 for robots.txt: no file, no restrictions, crawl anything.

Scraping one page you were handed fits the document poorly. There is no robots.txt field for “extract the price”, because extraction isn’t what the file governs. Google’s own spec states it “supports the following fields (other fields such as crawl-delay aren’t supported)”. The file controls discovery, full stop. Politeness still applies to scrapers, obviously: one request is one request, and rate limits and terms of service apply to everyone. But when a vendor tells you their tool “respects robots.txt”, read that as a crawling claim.

One tool, two jobs

Since every crawler must parse pages to find links and every scraper pipeline eventually wants more pages, toolmakers merged the two. What used to be separate programs is now usually two modes of one:

# scraping mode: one URL, fields/markdown out
magpie scrape https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html --page-format markdown

# crawling mode: seed URL, queue, caps
magpie crawl https://books.toscrape.com --max-pages 25 --corpus --rate 1.0

Same binary, same site, two different outputs, which is exactly why the vocabulary keeps collapsing. The crawl line finished 25 pages in 24.5 seconds. (Those are Magpie, a local CLI; the crawl line needs a --schema or the --corpus flag, and the seed URL needs its https:// or it exits with “no records extracted” and no explanation. I tested both lines.) For the crawling half specifically, the crawler tools roundup splits the field by who it fits.

FAQ

Can you scrape without crawling?

Yes, and it’s the common case. Any time you fetch one known URL and parse it (a product page, an article, a dashboard), you are scraping with zero crawling. You only need a crawler when the URL list itself is unknown or too long to type.

Is Googlebot a crawler or a scraper?

A crawler. Googlebot discovers URLs and fetches pages to build its index; Google’s documentation calls its automated fetchers crawlers throughout. Pulling fields into your spreadsheet is not part of its job.

Does robots.txt stop web scraping?

Not exactly. robots.txt is addressed to crawlers and only controls which URLs they may fetch; it has no extraction rules and no enforcement. It is the standard politeness signal, but a single-page fetch sits outside its vocabulary.

Which one do I need?

Decide by the deliverable. A list of URLs, a sitemap, a site audit: crawl. Values from pages you can already name: scrape. All 1,000 products’ prices: crawl for the product URLs, then scrape each one. The chain, not a choice.

The takeaway

Crawler = frontier. Scraper = fields. When someone conflates them in a project kickoff, ask what the deliverable looks like: a list of links, or a spreadsheet of values? The answer picks the tool, the failure modes you should expect, and which document — robots.txt or the site’s markup — is about to become your problem.