How to Build a Web Crawler (Architecture + Code)

A web crawler is thirty lines of Python wrapped around a hundred decisions. The lines take an afternoon. The decisions decide whether the thing comes back with clean data, 200 copies of the same page, or nothing at all because it switched itself off on the first 403.

So this is how to build a web crawler in the order that matters: the architecture first, then the code, then a real run you can check the numbers against. The finished crawler visited 25 pages of a test site in 23.8 seconds and discovered 308 unique URLs along the way. Every number below came from that run, today, on Python 3.12.

The six parts every crawler needs

Strip away the brand names and a web crawler is six parts in a loop. Learn them once and every crawler you ever meet, from a 30-line script to Googlebot, is recognizable.

1. The frontier. The queue of URLs waiting to be fetched. A deque does it. Without a cap on it, one page with 5,000 links puts 5,000 URLs in your memory and your future.
2. The seen set. Deduplication. The same page gets linked from dozens of other pages; category pages link to each other in circles. Without this, you crawl forever and store every page twice.
3. Politeness. A minimum delay between requests to the same host, plus robots.txt. Skip it and the site bans your IP before the crawl finishes.
4. The fetcher. HTTP with a timeout, a descriptive User-Agent, and a decision about what status codes mean. This is where most crawlers lie to themselves.
5. The parser. Pull links out of HTML and turn them into absolute URLs. Also where you decide what a “page” even is.
6. Storage. One record per page, written as you go. JSONL is fine. Without storage you have no record of what you crawled, which means you can’t resume, audit, or dedupe across runs.

The loop that ties them together: pop a URL from the frontier, check politeness and the seen set, fetch, parse, store one record, push the newly discovered URLs back onto the frontier. Repeat until the frontier is empty or a limit fires. Finding every page on a site is exactly this loop with different limits.

What you need

• Python 3.12 (tested on 3.12.3), with requests 2.32.5 and beautifulsoup4 4.14.3: pip install requests beautifulsoup4
• A target that tolerates crawling. I used books.toscrape.com, a sandbox built for scraping practice: 1,000 book pages, 50 category pages, heavy cross-linking, no JavaScript required. It publishes no robots.txt, which turns out to be a useful test case.

Step 1: Fetch a page without hanging (timeout, status, User-Agent)

Three things go wrong on the first request. No timeout, and your crawler waits forever on a connection that will never finish. No status check, and you parse a 404 page as if it were content. No User-Agent, and you look like a bot that has something to hide, which gets you blocked faster, not slower.

import requests

USER_AGENT = "BuildAWebCrawlerTutorial/0.1 (runs on books.toscrape.com; contact: [email protected])"

def fetch(session, url):
    try:
        resp = session.get(url, timeout=10)
    except requests.RequestException as err:
        return None, f"network error: {err.__class__.__name__}"
    if resp.status_code != 200:
        return None, f"HTTP {resp.status_code}"
    return resp, None

The User-Agent deserves one more line than people give it. Name your crawler, say what it does, and leave a contact. That string is the only thing a site operator sees in their logs, and it is the difference between “unknown bot hammering us, block it” and “someone’s tutorial crawler, let it finish.” If you want the longer version of staying welcome, I cover the mechanics in how to scrape without getting blocked.

Failure case this step fixes: a ConnectionError mid-run used to kill the whole crawl. Now it becomes one row in the errors list and the loop moves on. On this run there were 0 errors, but on a real site expect a few percent of pages to time out or 500.

Step 2: Extract links and normalize them (no loops, no junk)

Links come in three shapes that break naive crawlers: relative paths (/catalogue/page-2.html), fragments (page.html#contact, which is the same page as page.html), and non-HTTP schemes (mailto:, tel:, javascript:). Normalize everything the moment you see it.

from urllib.parse import urldefrag, urljoin, urlparse
from bs4 import BeautifulSoup

def normalize(url, base=None):
    """Resolve relative links, drop fragments, keep http(s) only."""
    if base:
        url = urljoin(base, url)
    url, _ = urldefrag(url)          # page.html#section == page.html
    scheme = urlparse(url).scheme
    if scheme not in ("http", "https"):
        return None                  # mailto:, tel:, javascript:, data:
    return url

def extract_links(html, base_url):
    soup = BeautifulSoup(html, "html.parser")
    title = soup.title.get_text(strip=True) if soup.title else ""
    links = [normalize(a.get("href"), base_url) for a in soup.find_all("a")]
    return title, {u for u in links if u}

The fragment strip is the one people skip, and it bites on real sites. Books to Scrape barely uses them (its product pages contain exactly one, #addreview on the reviews link), but a docs site links page.html#section from its sidebar dozens of times per page, and without urldefrag every one of those is a new “distinct” URL in your frontier.

Failure case this step fixes: javascript:void(0) and mailto: links ending up in the frontier, where the fetcher burns a politeness slot and a timeout on a URL that was never a page.

Step 3: The frontier loop (BFS, depth caps, and the seen set)

The loop is breadth-first search. A deque of (url, depth) pairs, a set of everything ever discovered, and two hard limits: max pages and max depth.

from collections import deque

MAX_PAGES = 25               # hard stop
MAX_DEPTH = 2                # seed page is depth 0

frontier = deque([(seed, 0)])
seen = {seed}
visited = 0

while frontier and visited < MAX_PAGES:
    url, depth = frontier.popleft()
    if not allowed(robots, url):          # Step 4
        errors.append((url, "blocked by robots.txt"))
        continue
    if depth > MAX_DEPTH:
        continue
    resp, err = fetch(session, url)
    if err:
        errors.append((url, err))
        continue
    visited += 1
    title, links = extract_links(resp.text, url)
    if depth < MAX_DEPTH:                 # don't expand past the cap
        for link in sorted(links):
            if link not in seen:
                seen.add(link)
                frontier.append((link, depth + 1))

Now the part no demo shows you. My run hit the 25-page cap, and of those 25 pages, 24 were at depth 1. Zero pages at depth 2. The seed page links more than two dozen category and featured pages, breadth-first search drains that whole layer before it touches anything deeper, so the budget was gone before a single category’s book pages got fetched.

That is not a bug. It is BFS doing what BFS does, and it is the reason your page budget and your depth setting need to match what you actually want. Depth 2 and 25 pages means “index this site’s directory structure,” not “read its articles.” Want the deep pages first? Raise the cap, or trade BFS for a priority queue that scores URLs (path depth, keywords in the slug) instead of blindly popping the oldest.

Failure case this step fixes: without the seen set, the 50 category pages that link to each other form cycles and the crawl never terminates. The run’s numbers prove the set works: 308 URLs discovered, 25 fetched, 283 still queued, nothing visited twice.

Step 4: Politeness (delay and robots.txt)

Politeness is two mechanisms. The delay is one line: time.sleep(0.5) after every request to the same host. Use 1 second or more on sites you don’t operate; I used 0.5 because the sandbox is built for this and the run was timed.

Robots.txt is where the interesting bugs live, and I hit two of them while writing this crawler.

The first bug was mine. My first draft fetched robots.txt and called raise_for_status() on the response. Books to Scrape returns 404 for robots.txt, raise_for_status() raised, my error handler treated that as “robots unreadable” and the crawler refused to start. Per RFC 9309, a 404 means “no rules published, crawl allowed.” One defensive line had turned every robots-less site on the internet into a forbidden zone.

The second bug is in the standard library’s happy path. RobotFileParser has a read() method that fetches robots.txt for you, and it uses Python’s default urllib User-Agent. I tested that against en.wikipedia.org today: default UA, HTTP 403. And a 403 inside read() sets an internal disallow_all flag, so can_fetch() returns False for every URL on the site. No exception, no log line. Your crawler just visits one page and stops. This is the fetcher lesson from Step 1 coming back around: the robots.txt reader is also a bot, and sites treat it like one.

The version below fetches robots.txt through the same session the crawler uses (so it sends my honest User-Agent) and handles the status codes explicitly:

from urllib.robotparser import RobotFileParser

def load_robots(session, start_url):
    parts = urlparse(start_url)
    rp = RobotFileParser()
    try:
        resp = session.get(f"{parts.scheme}://{parts.netloc}/robots.txt",
                           timeout=10)
    except requests.RequestException:
        print("robots.txt unreachable -- refusing to crawl (fail closed)")
        sys.exit(1)
    if resp.status_code in (401, 403):
        rp.disallow_all = True         # site protects robots.txt: crawl nothing
        return rp
    if resp.status_code >= 500:
        print("robots.txt server error -- refusing to crawl (fail closed)")
        sys.exit(1)
    if 200 <= resp.status_code < 300:
        rp.parse(resp.text.splitlines())
        rp.modified()                  # without this, can_fetch() returns False
        return rp
    rp.allow_all = True                # 404/other 4xx: no rules published
    return rp

One more stdlib trap, measured: if you feed lines to parse() yourself and forget rp.modified(), can_fetch() returns False for everything, because the parser treats “never checked” as “check failed.” I verified both sides of this on Wikipedia’s robots.txt: with the lines parsed properly, /wiki/Special:Search comes back disallowed (it is in the file) and /wiki/Spain comes back allowed. The rules work. The plumbing just gives no signal when you skip a step.

Step 5: Store one record per page (JSONL)

Write one JSON object per line, as you go, so a crash costs you nothing already crawled. JSONL streams, greps, and loads line-by-line in pandas; a single giant JSON array does none of those while running.

import json

out.write(json.dumps({"url": url, "depth": depth,
                      "title": " ".join(title.split())}) + "\n")

The " ".join(title.split()) is not decoration. Raw <title> tags ship with newlines and padding spaces; the third record of my run, uncleaned, would have contained "Academic | \n Books to Scrape - Sandbox". Store the clean form and spare yourself a cleanup pass later.

How to build a web crawler: the full script and a real run

Everything above, assembled. This is the exact file I ran:

#!/usr/bin/env python3
"""A small, polite web crawler. Tutorial code for
"how to build a web crawler" -- runs as written, Python 3.12,
requests + beautifulsoup4 only."""

import json
import sys
import time
from collections import deque
from urllib.parse import urldefrag, urljoin, urlparse
from urllib.robotparser import RobotFileParser

import requests
from bs4 import BeautifulSoup

USER_AGENT = "BuildAWebCrawlerTutorial/0.1 (runs on books.toscrape.com; contact: [email protected])"
DELAY_SECONDS = 0.5          # minimum gap between hits on the same host
MAX_PAGES = 25               # hard stop
MAX_DEPTH = 2                # seed page is depth 0


def normalize(url, base=None):
    """Resolve relative links, drop fragments, keep http(s) only."""
    if base:
        url = urljoin(base, url)
    url, _ = urldefrag(url)          # page.html#section == page.html
    scheme = urlparse(url).scheme
    if scheme not in ("http", "https"):
        return None                  # mailto:, tel:, javascript:, data:
    return url


def load_robots(session, start_url):
    """Fetch robots.txt with the same session (and User-Agent) the crawler
    uses, then parse the lines. Status codes per RFC 9309: 5xx or network
    error = assume no crawl; 404 and other non-auth 4xx = no rules, crawl
    allowed. 401/403 get the stricter de-facto treatment: the site
    protects robots.txt, so crawl nothing."""
    parts = urlparse(start_url)
    rp = RobotFileParser()
    try:
        resp = session.get(f"{parts.scheme}://{parts.netloc}/robots.txt",
                           timeout=10)
    except requests.RequestException:
        print("robots.txt unreachable -- refusing to crawl (fail closed)")
        sys.exit(1)
    if resp.status_code in (401, 403):
        rp.disallow_all = True         # site protects robots.txt: crawl nothing
        return rp
    if resp.status_code >= 500:
        print("robots.txt server error -- refusing to crawl (fail closed)")
        sys.exit(1)
    if 200 <= resp.status_code < 300:
        rp.parse(resp.text.splitlines())
        rp.modified()                  # without this, can_fetch() returns False
        return rp
    rp.allow_all = True                # 404/other 4xx: no rules published
    return rp


def allowed(robots, url):
    return robots.can_fetch(USER_AGENT, url)


def fetch(session, url):
    try:
        resp = session.get(url, timeout=10)
    except requests.RequestException as err:
        return None, f"network error: {err.__class__.__name__}"
    if resp.status_code != 200:
        return None, f"HTTP {resp.status_code}"
    return resp, None


def extract_links(html, base_url):
    soup = BeautifulSoup(html, "html.parser")
    title = soup.title.get_text(strip=True) if soup.title else ""
    links = [normalize(a.get("href"), base_url) for a in soup.find_all("a")]
    return title, {u for u in links if u}


def main():
    seed = "https://books.toscrape.com/"
    session = requests.Session()
    session.headers.update({"User-Agent": USER_AGENT})
    robots = load_robots(session, seed)

    frontier = deque([(seed, 0)])    # (url, depth)
    seen = {seed}
    visited = 0
    errors = []
    out = open("pages.jsonl", "w")

    t0 = time.monotonic()
    while frontier and visited < MAX_PAGES:
        url, depth = frontier.popleft()
        if not allowed(robots, url):
            errors.append((url, "blocked by robots.txt"))
            continue
        if depth > MAX_DEPTH:
            continue
        resp, err = fetch(session, url)
        if err:
            errors.append((url, err))
            continue
        visited += 1
        title, links = extract_links(resp.text, url)
        out.write(json.dumps({"url": url, "depth": depth,
                              "title": " ".join(title.split())}) + "\n")
        if depth < MAX_DEPTH:        # don't expand the frontier past the cap
            for link in sorted(links):
                if link not in seen:
                    seen.add(link)
                    frontier.append((link, depth + 1))
        time.sleep(DELAY_SECONDS)    # politeness: same host, always wait

    out.close()
    print(f"visited:  {visited} pages in {time.monotonic() - t0:.1f}s")
    print(f"queued:   {len(frontier)} left in frontier (page cap hit)" if frontier
          else f"queued:   0 -- frontier drained")
    print(f"seen:     {len(seen)} unique URLs discovered")
    print(f"errors:   {len(errors)}")
    for url, why in errors[:5]:
        print(f"  {url} -> {why}")


if __name__ == "__main__":
    main()

The output, verbatim:

visited:  25 pages in 23.8s
queued:   283 left in frontier (page cap hit)
seen:     308 unique URLs discovered
errors:   0

And the first lines of pages.jsonl:

{"url": "https://books.toscrape.com/", "depth": 0, "title": "All products | Books to Scrape - Sandbox"}
{"url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html", "depth": 1, "title": "A Light in the Attic | Books to Scrape - Sandbox"}
{"url": "https://books.toscrape.com/catalogue/category/books/academic_40/index.html", "depth": 1, "title": "Academic | Books to Scrape - Sandbox"}

Raise MAX_PAGES and this same loop runs for hours. Whether it should is the next question.

What breaks at 10,000 pages

The script above is honest up to a few thousand pages, single host. Past that, four walls arrive in roughly this order.

The seen set outgrows RAM. I measured a Python set of 1,000,000 URLs with tracemalloc: 158 MB, and URL lengths vary, so call it 150 to 200 MB per million. At 100 million URLs you are spending tens of gigabytes on “have I seen this.” Real crawlers move dedup into a bloom filter (tiny memory, rare false positives, no removal) or a disk key-value store.

One delay per loop starves every host but one. time.sleep(0.5) after each request is correct for a single host and madness for twenty: while you sleep for host A, hosts B through T sit idle. The fix is per-host queues, each with its own last-request timestamp, plus a scheduler that always works on the host that has waited longest.

JavaScript pages come back empty. I fetched quotes.toscrape.com/scroll with this stack: HTTP 200, 2,671 bytes, zero quotes in the HTML. A browser loading the same page shows quotes because a script calls an API. A static crawler cannot see that content; it will happily store the empty shell and report success. If your targets render with JS, you need a real browser in the fetcher, and browsers are heavy: my crawler peaked at 41 MB of RSS; a bare headless Chromium idling at about:blank takes 85 MB on this box before it renders a thing. That decision is big enough that scraping dynamic websites is its own problem, and infinite-scroll pages add pagination on top (how to handle them).

Sites push back, in order of politeness. 429 with a Retry-After header, then temporary IP blocks, then permanent ones. A respectful crawler backs off when told; a scraping-focused tool adds retries with jitter, and proxies only when a site blocks a datacenter IP range you legitimately need. Sitemaps offer a shortcut around link discovery: XML sitemaps let a crawler seed its frontier from a list the site publishes instead of finding everything the slow way.

When to stop building

The architecture above is the portable part. The last 10 percent, per site, is where tutorials quietly end and maintenance begins: retries with backoff, content extraction that survives redesigns, resume after a crash, scheduling. Whether to build that yourself depends on how often the answer changes.

If the crawl is the product, keep coding. Scrapy owns the machinery for you (their tutorial takes an hour, not an afternoon, and its pipelines are worth it past a few thousand pages). If the data is the product, tools do the six parts for you. For one recent comparison I ran Magpie’s crawl command against the same sandbox with the same 25-page budget: magpie crawl https://books.toscrape.com/ --corpus --max-pages 25 --max-depth 2 --rate 1 finished in 26.8 seconds, 25 pages, 0 errors, and each JSONL record carried cleaned main-content markdown alongside the URL and title, not just the <title> tag. It also tried the site’s sitemap first (404 there, so it fell back to link-following), and runs are resumable via a --resume checkpoint, which my script is not. Magpie is AGPL-3.0-or-later or commercial, local-only, and it is one of the options in the web crawling tools comparison. Build when the crawling is the point. Run a tool when the data is.

FAQ

What is the difference between a web crawler and a web scraper?

A crawler discovers and fetches pages by following links; a scraper extracts specific fields from the pages it is given. They compose: crawl to find the URLs, scrape to pull the data. The split is drawn out in web scraping vs web crawling.

Is it legal to build a web crawler?

In the US, crawling publicly accessible pages has held up in court: the Ninth Circuit’s hiQ v. LinkedIn decision (2022) treated scraping public pages as lawful under the CFAA. Login-walled content, copyright, personal data rules, and a site’s terms remain separate risks. Respect robots.txt, rate-limit, and get advice for anything commercial. Not legal advice.

What is the best language for a web crawler?

The one you can debug. Python reaches a few thousand pages per host with the script above; Go or Java buy cheap concurrency when you outgrow one host. The six parts and their failure modes are identical everywhere, so the architecture transfers even if the code does not.

How do crawlers avoid visiting the same page twice?

A seen set checked before enqueueing, fed by normalized URLs: fragments stripped, relative paths resolved, schemes filtered. Larger crawlers add canonical-URL handling (respecting rel=canonical) and swap the in-memory set for a bloom filter or a disk store once URLs number in the millions.

Where to take it next

Run the script against a second host you control and watch the errors list earn its keep: real sites return 403s, 429s, and HTML that lies about its encoding, and each one is a two-line fix now instead of a weekend later. Then decide which side of the build-versus-run line you’re on, and put the page budget where the data actually lives.