Best Open-Source Web Scraping Tools (2026)

An open source web scraper is free code, not a free lunch. Modify Firecrawl and serve it over a network, and its AGPL-3.0 license says you publish your modifications. That detail hides in every feature table, and it is why this list of the best open source web scraping tools leads with licenses before star counts.

Below: nine tools, every license checked against the actual LICENSE file this week, plus a same-page speed test I ran this morning. Short answer: Beautiful Soup for one page, Scrapy for a hundred thousand, Crawl4AI or Magpie for an LLM corpus on your own hardware. The rest of this page justifies those picks.

Tool	Language	Best for	License	Runs JS
Scrapy	Python	Structured crawls at scale	BSD-3	Via scrapy-playwright
Beautiful Soup	Python	Parsing one page fast	MIT	No
Playwright	Python, Node, Java, .NET	JavaScript sites, screen scraping	Apache-2.0	It is the browser
Crawlee	Node/TS, Python	Production JS crawls	Apache-2.0	Yes
Crawl4AI	Python	Local LLM pipelines	Apache-2.0	Yes
Firecrawl	TypeScript	Self-hosted scraping API	AGPL-3.0	Yes
Colly	Go	Small static crawls	Apache-2.0	No
Katana	Go	URL and endpoint discovery	MIT	Headless mode
Magpie	Go	Local-first scraping, agents via MCP	AGPL-3.0 or commercial	Yes

Ranking logic: the job the tool is actually best at, then license freedom, then how much code stands between install and first result. Hosted versus local is a preference, not a rank.

The same-page test

Every listicle here ranks tools none of them ran. So: one page (quotes.toscrape.com, a site built for exactly this), one machine (WSL2 on a home connection), one morning (October 16, 2026), sequential runs with a pause between requests.

Tool	Code you write	Wall time	Result
requests + Beautiful Soup	13 lines	0.86s	10 Python dicts, selectors hand-written
Scrapy 2.19	15-line spider in a project	2.98s	Same 10 dicts, engine startup included
Magpie CLI	One command, zero selectors	2.61s	214-word clean markdown + JSON-LD sidecar

Read the honest way: Beautiful Soup “wins” because it does the least: one GET, your selectors, done. Scrapy’s extra two seconds is Twisted engine startup, which dominates at one page and disappears at ten thousand. The column that matters is the last one: Magpie hands back readable markdown with no selectors written at all. The others need you to already know the page.

1. Scrapy: the incumbent

Scrapy is the most durable of the web scraping tools open source has produced: the Python answer since 2008, around 65,000 stars, BSD-3 licensed. Version 2.19 (September 2026) shows the age in the good way: auto-throttling, per-domain rules, item pipelines, retry middleware, and job persistence that survives a crashed run.

The spider I ran this morning:

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]

    def parse(self, response):
        for q in response.css(".quote"):
            yield {
                "text": q.css(".text::text").get(),
                "author": q.css(".author::text").get(),
                "tags": q.css(".tag::text").getall(),
            }

Pros

• Battle-tested at millions of pages; restarts and resume are solved problems, not plugins you evaluate
• Auto-throttle reads response times and paces itself per host
• scrapy-playwright adds real browser rendering when a site needs it

Cons

• You write a project, not a script. Hello-world is an hour of scaffolding: spiders, items, pipelines, settings
• No markdown output. An LLM pipeline needs a conversion layer you build and maintain
• JavaScript sites require scrapy-playwright, which adds a browser and a second config surface

License: BSD-3-Clause. Price: free. At scale, for structured records, nothing open source beats it.

2. Beautiful Soup: the first scraper everyone writes

Beautiful Soup parses HTML you already have. Pair it with requests and you have the smallest working website scraper open source ever produced: one MIT-licensed library you can read over a lunch break. The 13-line run from my test above finished in 0.86 seconds, fetch and parse included.

import requests
from bs4 import BeautifulSoup

r = requests.get("https://quotes.toscrape.com/", timeout=30)
soup = BeautifulSoup(r.text, "html.parser")
for q in soup.select(".quote"):
    print(q.select_one(".text").get_text(strip=True),
          q.select_one(".author").get_text(strip=True))

Pros

• Ten-minute learning curve; .select() speaks CSS you already know
• Forgiving of broken HTML, which is most HTML
• Tiny footprint; runs anywhere Python runs

Cons

• No fetching, no JavaScript, no concurrency; you assemble all three
• Selector rot is entirely your problem; a site redesign breaks your script silently
• Per-page scripts don’t grow into crawls; you will rewrite in Scrapy eventually anyway

License: MIT. Price: free. Right tool for one page, a prototype, or a cron job over pages you control.

3. Playwright: the browser, scripted

When the target is a JavaScript app, scraping open source style means driving a real browser, and Playwright (Microsoft, Apache-2.0, ~98,000 stars, v1.64.1 as of October 2026) is the default. It launches real Chromium, Firefox, or WebKit, waits for elements to settle, and can intercept network traffic, often the cleaner path, since the data you want usually arrives as an XHR JSON response you can grab directly.

If you came here hunting screen scraping tools open source projects actually maintain, this is the category: a real browser, scripted end to end, including screenshots of the rendered page. Playwright is the open source screen scraper of record; see what screen scraping is for the vocabulary.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://quotes.toscrape.com/js/", wait_until="networkidle")
    page.screenshot(path="quotes.png", full_page=True)
    browser.close()

Pros

• Real browsers, real rendering; auto-wait kills most flaky-timing bugs
• Network interception gets you the JSON behind the page, not the page
• Four language bindings, one API shape

Cons

• It is a driver, not a scraper: extraction, cleaning, storage are all yours
• Each Chromium tab costs hundreds of MB of RAM; parallelism eats laptops
• Vanilla Playwright fingerprints get detected by serious anti-bot stacks; anti-detect forks like Camoufox trade on that gap, and it is an arms race, not a fix. Playwright is not a bypass tool and doesn’t pretend to be

License: Apache-2.0. Price: free.

4. Crawlee: production habits for Node teams

Crawlee is Apify’s open-source core (Apache-2.0, ~26,000 stars, v3.18.2): the queueing, retrying, proxy-rotating plumbing every crawler gets rewritten until someone extracts it into a library. CheerioCrawler handles static sites at speed; PlaywrightCrawler takes the JS ones, sharing the same queue and storage layer.

Pros

• Request queues, resume, and proxy management built in: the parts that hurt at 5,000 pages, not 5
• One codebase covers static and browser crawls
• TypeScript-first; types hold up in real projects

Cons

• Markdown conversion is DIY. Output is DOM; you assemble the cleaning
• The Python port lags the Node original in features
• Browser-heavy crawls have the same RAM appetite as raw Playwright

License: Apache-2.0. Price: free. Pick it when your team already lives in Node and wants the grown-up scaffolding without writing it.

5. Crawl4AI: built for LLM pipelines

Crawl4AI (Apache-2.0, ~86,000 stars, v0.9.5) is the fastest-growing of the Python projects because it aimed at the RAG-shaped hole: pages in, clean LLM-ready markdown out, no API meter running. Its BM25 content filter scores page sections against your query and keeps only the relevant ones before you spend tokens embedding them. Local extraction works through Ollama, so the whole pipeline can run on your hardware.

Pros

• Markdown output tuned for context windows, natively
• Deep crawling, adaptive reuse of working selectors, Docker server mode
• Zero per-page cost; everything runs where you put it

Cons

• Still pre-1.0, and the API has moved underfoot between minor versions. Pin your version or read changelogs as homework
• Python only; config surface is large for a first-timer
• Markdown quality on nested-sidebar docs sites still needs tag filters

License: Apache-2.0. Price: free. I compared it against the hosted options in the Crawl4AI alternatives breakdown.

6. Firecrawl: the scraper API you can own

Firecrawl holds the most stars in scraping (~192,000) and its core is AGPL-3.0 open source, which means the API everyone pays them to host is one you can also run yourself. If what you want is a web scraper API open source under AGPL (a REST endpoint that takes a URL and returns clean markdown), this is the closest fit on this page.

Self-hosting is Docker Compose against their repo: set USE_DB_AUTHENTICATION=false for the no-database baseline, bring the stack up, and point the SDK at your box:

from firecrawl import Firecrawl

app = Firecrawl(api_key="local", api_url="http://localhost:3002")
doc = app.scrape("https://example.com", formats=["markdown"])
print(doc.markdown[:500])

Pros

• One API surface across self-hosted and their paid cloud; swap by changing api_url
• JS rendering, pagination handling, and main-content stripping inside the service
• SDKs in four languages plus an MCP server for agent workflows

Cons

• The self-host stack is not light: Playwright service, PostgreSQL queue, Redis. This is a small server, not a laptop tab
• AI-backed features need you to attach a model provider yourself
• AGPL-3.0: modify it and serve it over a network, and you publish your modifications. Unchanged internal use is fine

License: AGPL-3.0 (core), hosted tiers are paid credits. I went deeper on the trade-offs in Firecrawl: alternatives and honest verdict, and compared the hosted field in best web scraping APIs.

7. Colly: the Go minimalist

Colly does one thing cleanly: callback-driven crawling in about fifteen lines of Go. License check, because listicles keep getting this wrong: Colly’s LICENSE.txt says Apache-2.0, not MIT as half the blogs claim. ~26,000 stars, last release v2.2.0 in March 2025. The cadence is measured in years, which reads as stable or abandoned depending on your temperament. The code is small enough that either reading works.

c := colly.NewCollector(colly.AllowedDomains("quotes.toscrape.com"))
c.OnHTML(".quote", func(e *colly.HTMLElement) {
    fmt.Println(e.ChildText(".text"), e.ChildText(".author"))
})
c.Visit("https://quotes.toscrape.com/")

Pros

• Smallest API here; any Go developer reads it in one sitting
• Real concurrency with per-domain limits built in
• Compiles to a static binary your cron job runs for years

Cons

• No JavaScript. A React site returns an empty shell
• No cleaning, no markdown; raw HTML fragments in your callbacks
• Selector breakage is silent; you add your own monitoring

License: Apache-2.0. Price: free. For static sites you control, on a schedule, it is still the Go answer.

8. Katana: discovery, not content

Katana (ProjectDiscovery, MIT, ~18,000 stars, v1.8.0) crawls to find URLs, parameters, and JS-referenced endpoints. katana -u https://example.com -d 2 -jc emits every discovered endpoint as JSON lines. It is the scraping tool open source security teams reach for, and it stops at the doorway: it finds pages, it does not clean them.

Pros

• Fastest endpoint discovery per unit of config in this list
• Scope filters, depth caps, and headless mode in the CLI
• JSON-lines output pipes straight into jq or your own tooling

Cons

• No content extraction at all; pair it with something that parses
• Scope discipline is your job; a loose glob crawls hosts you didn’t mean to touch
• Large runs produce a firehose; dedupe downstream

License: MIT. Price: free.

9. Magpie: the open source web scraper that fixes its own selectors

Full disclosure: this is our product. Magpie is a Go binary (CLI plus a desktop app) with the fetch, clean, and extract steps in one pipeline, and its license is dual: AGPL-3.0-or-later, or a commercial license if AGPL doesn’t fit your situation.

The command from this morning’s test:

magpie scrape https://quotes.toscrape.com/ --page-format markdown --out quotes.json

That returned 214 words of clean markdown in 2.61 seconds, boilerplate stripped, JSON-LD structured data alongside. No selectors written, no parser code. The feature that earns the slot: extraction selectors live in a per-domain cache, and when a redesign drives the null-rate up, Magpie re-synthesizes the selector instead of failing quietly, the failure mode that eventually claims every hand-written scraper above.

It also carries the pieces you otherwise assemble yourself: robots.txt respect with auto-throttling, login sessions with persistent cookies, and JS rendering with click-and-wait actions plus XHR capture. Twenty-seven zero-LLM vertical extractors (jobs, events, articles, and more) need no API key at all. A 13-tool MCP server lets coding agents scrape without leaving the terminal.

Pros

• Single static binary, no runtime dependencies; runs offline
• Self-healing selector cache: breakage gets repaired, not discovered
• The data never leaves your machine; BYOK for any LLM extraction, including local models through Ollama

Cons

• Local means local: no hosted API, no fleet. Multi-machine crawling is your orchestration problem
• The desktop app is free to download with 5 runs per install, then $49/month. The CLI itself has no run counter
• Young, single-team project. Fast release cadence, but no SLA, and it won’t touch CAPTCHA-walled targets by design

License: AGPL-3.0-or-later or commercial. Price: free (CLI), $49/month (desktop, after trial). If the constraint is “the data stays on this machine,” start here.

Honorable mentions

Three more, because web scraping open source tools multiply faster than listicles update:

• ScrapeGraphAI (MIT, ~32,000 stars): LLM-powered extraction graphs; you supply an API key or an Ollama model. A capable open source data scraper when you’d rather write a prompt than selectors, and costs per run reflect that.
• Maxun (AGPL-3.0, ~18,000 stars), a no-code recorder that turns site interactions into repeatable robots, self-hosted. The least code of any genuinely open web scraper here.
• Heritrix, Apache Nutch: archival-scale crawlers, the scraping software open source wrote before lead generation needed one. Overkill for scraping jobs, worth knowing exist. The crawler-focused siblings of this list live in free and open-source web crawlers compared.

What the licenses actually cost

Plenty of free web scraping tools open source their core and charge for the hosted version; the license tells you which side of that line you’re on. Three buckets decide:

• Permissive (MIT, BSD, Apache-2.0): Scrapy, Beautiful Soup, Playwright, Crawlee, Crawl4AI, Colly, Katana. Use them in products, keep the copyright notice, owe nothing else.
• AGPL-3.0: Firecrawl’s core, Maxun. Free for internal use. The network clause kicks in when you modify the software and offer it as a service: your modifications must ship with their source. Building a SaaS wrapper is where people get surprised.
• Dual (AGPL or paid): Magpie. Free under AGPL; the commercial license exists for teams whose lawyers hate the previous sentence.

Web scraper source code is the contract. Read the LICENSE file the day you pick a tool, not the day legal asks about it. Treat any listicle, including this one, as a lead rather than a verdict.

How to choose

Match the tool to the job. There is more web scraping software open source than any one team can evaluate, and every README claims the same three things, so the routing starts from the work. Web data extraction open source style comes down to five routes:

• Structured records at scale (thousands of pages, repeating fields): Scrapy. Its pipeline model is why it outlived everything, and among open source data scraping tools it is the default answer.
• One page, right now: Beautiful Soup plus requests. Thirteen lines.
• JavaScript-heavy sites: Playwright if you want control, Crawlee if you want scaffolding. The full workflow is in how to scrape dynamic websites.
• An LLM corpus on your own hardware: Crawl4AI or Magpie. Both emit markdown natively and run without sending data anywhere.
• A REST API you host yourself: Firecrawl self-hosted.
• Endpoint lists, not page content: Katana.
• A Go team and a static target: Colly.

Every one of these open source scraping tools survives production somewhere; the routing is about where yours starts. Two questions cut the field faster than any feature table. Does the site need a browser? That removes Colly, Beautiful Soup, and vanilla Scrapy from contention. Will this run more than once against a site you don’t control? Then selector maintenance decides, and you want a tool that notices its own breakage (Crawl4AI’s adaptive crawling, Magpie’s self-healing cache) over the one that was fastest to install. For the traffic-shaping side of that problem, how to scrape without getting blocked covers pacing and politeness.

The whole web scraper software open source question fits in that list. Everything else is README marketing.

FAQ

What’s the best free open source web scraper for beginners? Beautiful Soup plus requests. One page, thirteen lines of Python, MIT license, no account. Move to Scrapy when the job becomes a crawl.

Can I use AGPL-licensed scraping tools commercially? Yes, for internal use. The obligation appears when you modify the software and serve it over a network: you must publish your modifications. Dual-licensed tools like Magpie sell an opt-out.

Do these open source tools handle JavaScript-heavy sites? Playwright, Crawlee, Crawl4AI, Firecrawl, and Magpie render JS natively. Scrapy needs the scrapy-playwright plugin. Colly and Beautiful Soup see only the raw HTML, which for an SPA is an empty shell.

Open source web scraper or open source web crawler: which do you need? A scraper extracts data from pages; a crawler discovers pages by following links. Most tools here do both with different emphasis. The crawler-first options are compared in open-source web crawlers, and the distinction itself in web crawling tools.

Check one thing this week: point your current stack at a page it scraped a month ago and diff the output against last month’s. An empty diff is good news. A changed one tells you which tool you actually need. Run that diff monthly and the best open source web scraper question settles itself, with your data instead of someone’s star counts.