Scrapy vs BeautifulSoup: One Parses, One Crawls

Every Scrapy vs BeautifulSoup thread ends up at the same reply: they’re not competitors. True, and useless without numbers: I installed both from scratch today on Python 3.12 and pointed them at the same job, crawling all ten pages of quotes.toscrape.com for 100 quotes. The parser stack finished in 3.6 seconds, Scrapy in 4.6, a near-tie that tells you almost nothing about which one to pick, because the interesting differences are everywhere this benchmark doesn’t look.

The verdict

Pick BeautifulSoup with requests when the job fits in one file: parse a page you already have, hit a handful of URLs, reformat some HTML. Pick Scrapy when the job is a crawl: many pages, unknown depth, politeness budgets, structured output. If you can’t tell which side you’re on, you’re on the BeautifulSoup side. Upgrade the day you catch yourself writing a pagination loop by hand.

	Scrapy 2.19.0	BeautifulSoup 4.15.0 + requests
What it is	Crawler framework: downloader, scheduler, item pipelines, interactive shell	HTML parser with a Python-friendly API
Fetches URLs	Yes, built-in async downloader (Twisted); requests isn’t even a dependency	No. You supply HTML from requests, httpx, urllib, or a file
Install, fresh venv, measured	49 packages, 117 MB, 67 s	11 packages, 21 MB, 3.5 s
Latest release	2.19.0, Sep 10, 2026	4.15.0, Jun 7, 2026
License	BSD-3-Clause	MIT
Selectors	response.css() / .xpath() via parsel	.select() via soupsieve; nearly the same dialect
JavaScript	Neither renders it. scrapy-playwright bolts a browser onto Scrapy	Neither. You run Playwright or Selenium yourself and hand BS4 the HTML
Concurrency	16 concurrent requests by default	Sequential unless you write threads or asyncio
Retries	2 per request, built in	Whatever you write
robots.txt	Ignored by default; one setting makes it obey	Lives in your fetcher, and requests ignores it too
Export	scrapy crawl -o out.json (json, jsonl, csv, xml)	Your own json.dump call
Project home	github.com/scrapy/scrapy, 64,853 stars	Launchpad and crummy.com; the GitHub mirror last moved Nov 2022 (227 stars)

First, the category error

BeautifulSoup has no opinion about where HTML comes from. Give it a string, it gives you a tree and .select() for querying it. That’s the whole product, and it’s good at it: 4.15.0 still installs in 3.5 seconds because it’s a couple of pure-Python packages plus soupsieve for CSS matching. Requests supplies the HTTP half, and the pair works so well that “requests + BeautifulSoup” is the de facto Python scraping stack.

Scrapy is the other shape entirely. It ships its own event-driven downloader built on Twisted, a scheduler that deduplicates URLs, middleware layers for retries, redirects, and cookies, and an item pipeline for cleaning and export. It doesn’t import requests anywhere in the dependency chain; I checked pip show scrapy, and the fetch layer is all its own. Asking which tool parses faster asks the wrong question. Parsing was never Scrapy’s job. Fetching, scheduling, and surviving the site is.

The odd side effect: BeautifulSoup barely exists on GitHub. Development happens on Launchpad, and the mirror repo people star has been quiet since November 2022. Version truth lives on PyPI, where 4.15.0 landed in June 2026. The project is alive; its social proof is just elsewhere.

Scrapy vs BeautifulSoup on the same crawl

The job: every quote from quotes.toscrape.com, ten paginated pages, text plus author plus tags. First the parser stack. This is the whole program:

import json
import time

import requests
from bs4 import BeautifulSoup

start = time.perf_counter()
session = requests.Session()

def parse(html):
    soup = BeautifulSoup(html, "html.parser")
    return [
        {
            "text": q.select_one("span.text").get_text(),
            "author": q.select_one("small.author").get_text(),
            "tags": [t.get_text() for t in q.select("a.tag")],
        }
        for q in soup.select("div.quote")
    ]

quotes, page = [], 1
while True:
    r = session.get(f"https://quotes.toscrape.com/page/{page}/", timeout=30)
    r.raise_for_status()
    batch = parse(r.text)
    if not batch:
        break
    quotes.extend(batch)
    page += 1

print(f"{len(quotes)} quotes from {page - 1} pages in {time.perf_counter() - start:.2f}s")
json.dump(quotes, open("bs4_quotes.json", "w"), indent=1)

That’s 34 lines of code (minus blanks and comments), and it ran clean: 100 quotes, 10 pages, 3.60 seconds.

Now Scrapy. The whole spider:

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]

    def parse(self, response):
        for q in response.css("div.quote"):
            yield {
                "text": q.css("span.text::text").get(),
                "author": q.css("small.author::text").get(),
                "tags": q.css("a.tag::text").getall(),
            }
        for href in response.css("li.next a::attr(href)"):
            yield response.follow(href, self.parse)

Thirteen lines, and the command scrapy runspider quotes_spider.py -o scrapy_quotes.json did the rest. Same result: 100 quotes from 10 pages, 4.56 seconds wall time.

Read those side by side and the real difference isn’t speed. The requests version carries the pagination logic on its back: the while loop, the empty-page stop condition, the raise-for-status call. The spider just says “follow the next link” in one line. Export is a hand-written json.dump in one and a flag (-o out.json, json or jsonl or csv or xml) in the other. Retries: none in the parser stack, two per request in Scrapy. At ten pages, a tie. At a thousand, the parser stack is a hand-maintained crawler and you are its unpaid maintainer.

Cold starts and RAM: the parser’s quiet win

Scrapy’s machinery isn’t free even when you barely use it. Same box, same venv, one page each, cold process:

• requests + BeautifulSoup: 1.15 seconds including Python imports, 40 MB peak RSS
• Scrapy, one page: 2.04 seconds, 82 MB peak RSS

The Twisted reactor and Scrapy’s own imports cost about a second and double the memory before a single byte of your page is parsed. Inside a normal script nobody cares. Inside a serverless function where every cold start is billed, or a cron job that fires every minute, the parser stack starts to look like the responsible choice. BeautifulSoup wins every category nobody benchmarks: startup, memory, and the ability to explain the whole program in one screen.

The JavaScript test both tools fail

quotes.toscrape.com has a second version of itself at /scroll, where the quotes load from an API as you scroll. I fetched it raw with requests: HTTP 200, 2,671 bytes of HTML, zero div.quote elements, two script tags, no quote text anywhere in the markup. Parse that with BeautifulSoup and the result is zero quotes, no error. Scrapy does the same: the spider runs green and yields nothing. Neither tool renders JavaScript. That’s not a Scrapy gap or a BS4 gap; it’s the shared wall.

Two details from that test matter. First, /scroll/ with a trailing slash is a 404, and if I hadn’t printed the status code I’d have read “0 quotes” as a broken selector and started debugging the wrong thing. Silent zero-results are how most scraping bugs actually show up. Second, the fixes diverge: Scrapy has scrapy-playwright, a maintained plugin that hands pages to a real browser inside the crawl. BeautifulSoup has no integration because it’s a parser; you drive Playwright yourself and pass page.content() to BS4 when the render finishes. If the site you’re targeting needs a browser at all, the comparison that matters moves to Selenium vs Playwright, not this one. And if the parser is the only part doing real work against browser-fetched HTML, that fight is BeautifulSoup vs Selenium.

Past 100 pages, the gap opens

Everything Scrapy charges you at install time, it pays back as the crawl grows:

• Concurrency. 16 requests in flight by default. Reproducing that on requests means a thread pool, semaphore discipline, and careful exception handling. I’ve written it; it’s half a day of edge cases.
• Politeness. Defaults are hot: 16 concurrent, no delay, robots.txt ignored. Two settings fix it (ROBOTSTXT_OBEY = True and AUTOTHROTTLE_ENABLED = True), and AutoThrottle then adapts delay per site from response latency. Turn both on before any crawl you didn’t write the server for.
• State. The scheduler won’t revisit a URL twice, no matter how many pages link to it. Your while loop trusts the site’s pagination not to loop.
• Export and structure. Items flow through pipelines: validation, cleaning, dedupe, database writes. A requests script accretes a quotes.append(...) dict until someone refactors it into a pretend pipeline.

This is also where the selector syntax stops mattering. soup.select("div.quote") and response.css("div.quote") are close enough that switching costs an afternoon, most of it spent on Scrapy’s project scaffolding. That scaffolding is the actual product. The CSS selectors are table stakes both ways.

One honest sidebar for readers whose goal is the data rather than the code: a third shape exists that writes neither spider nor loop. Local desktop and CLI scrapers bundle fetch, boilerplate cleaning, extraction, and watch mode in one binary, robots and rate limits already handled; the trade is flexibility, since you configure rather than program. The Python web scraping guide covers when writing code is the wrong tool. For the rest of this page, the question stays Scrapy versus BeautifulSoup.

FAQ

Is Scrapy faster than BeautifulSoup?

Not for parsing one page, no. Scrapy’s cold start ran 2.0 s against the parser stack’s 1.15 s on my machine. Scrapy wins wall-clock time on large crawls because it fetches 16 pages at once and retries failures, not because it parses faster. If raw parsing speed is the bottleneck, lxml beats both.

Can BeautifulSoup crawl a website?

No. It parses the HTML you hand it. Crawling means fetching, which means requests or httpx plus your own pagination loop, retry handling, rate limiting, and dedupe. That’s roughly the 34-line program above, and it grows with every site quirk.

Can I use BeautifulSoup inside Scrapy?

Yes, in the plain Python sense: BeautifulSoup(response.text, "html.parser") works in any spider callback. Most spider code doesn’t bother, because response.css() and response.xpath() cover the same queries through parsel. Mixing them adds a dependency without adding reach.

Which should a beginner learn first?

requests plus BeautifulSoup. You’ll write working code in an afternoon and learn selectors, HTTP status codes, and HTML structure without framework ceremony. Add Scrapy when a real project needs crawling; the BeautifulSoup tutorial starts you off, and the Scrapy tutorial is the step up.

Does Scrapy obey robots.txt?

Not by default, which surprises people who assume a big framework means polite defaults. Set ROBOTSTXT_OBEY = True in your spider settings. BeautifulSoup sidesteps the question by not fetching anything; whatever fetcher you pair it with decides.

One decision rule

If the job fits in one file and one HTTP response, write requests plus BeautifulSoup and stop. The first time you catch yourself writing a while loop that follows a next link, port it to Scrapy, because at that moment you’ve already started rebuilding the framework, one bug at a time. Both installs took me under a minute and a half combined. Point them at your actual target site and let the site pick.