Free & Open-Source Web Crawlers Compared (2026)

Every listicle calls Colly’s license MIT. The repo’s LICENSE.txt is Apache-2.0, and that one-line difference decides whether you can ship Colly inside a product. So I pulled the license file from every repository below, audited defaults, and ran five crawlers on the same site under the same politeness cap. Picking a web crawler open source project on README badges alone is how that gap survives.

Ten crawlers made the cut, from a 4,786-star C mirror to an 85,639-star Python framework. All ten are genuinely open source and free to run.

Tool	Language	License (verified in-repo)	Best for	Renders JS
Scrapy	Python	BSD-3-Clause	Large structured crawls with pipelines	Via scrapy-playwright
Crawlee	TypeScript	Apache-2.0	Production crawls on JS-heavy sites	Yes, Playwright/Puppeteer
Crawl4AI	Python	Apache-2.0	LLM-ready markdown, zero API cost	Yes, Playwright
Colly	Go	Apache-2.0	Small, fast static crawls	No
Katana	Go	MIT	URL and endpoint discovery	Headless mode
Browsertrix Crawler	TypeScript	AGPL-3.0	High-fidelity WACZ archives	Yes, full browser
Heritrix	Java	Apache-2.0	Archive-grade month-long crawls	Limited
Apache Nutch	Java	Apache-2.0	Hadoop-scale search indexing	No
HTTrack	C	GPL-3.0	Mirroring a site to disk	No
Magpie	Go	AGPL-3.0 or commercial	Local-first markdown/JSON + MCP	Yes, go-rod

Ordering logic: general-purpose frameworks first, specialists after, the turnkey local option last. Star counts are from the GitHub API on 2026-10-16.

What makes a web crawler open source

Two checks separate real open source from marketing. First, read the LICENSE file, not the README badge. Heritrix’s GitHub license slot says “Other” because its LICENSE file carries an appendix; the file itself grants Apache-2.0. Colly shows MIT in half the blog posts written about it and Apache-2.0 in its own tree.

The license also decides reuse. Apache-2.0 and BSD let you embed the crawler in a proprietary product. GPL-3.0 (HTTrack) and AGPL-3.0 (Browsertrix, Magpie) require you to release source if you distribute modified versions, and AGPL closes the SaaS loophole: run a modified crawler as a network service and you still owe the source. More on what that means commercially below.

Free web crawling tools, and the open-core catch

“Free” also does double duty in this category, and the second meaning costs money. Several well-known free web crawling tools are open-core: Firecrawl’s repo is AGPL with paid cloud features, and Apify’s SDK is open while the platform meters you by the result. Legitimate business model, but the code you can self-host is a subset of the product in the demo. The ten crawlers in this piece have no metered cloud component; everything runs on your machine.

1. Scrapy: the incumbent framework

First released in 2008, now at 64.8k stars and BSD-3-licensed, Scrapy is still the default answer for structured crawls in Python. Spiders select with CSS/XPath, items flow through pipelines, auto-throttle and per-domain settings handle politeness.

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]
    custom_settings = {"DEPTH_LIMIT": 1, "DOWNLOAD_DELAY": 1.0}

    def parse(self, response):
        for q in response.css("div.quote"):
            yield {"text": q.css("span.text::text").get(),
                   "author": q.css("small.author::text").get()}
        for href in response.css("li.next a::attr(href)"):
            yield response.follow(href, self.parse)

Run it with scrapy runspider quotes_spider.py -O out.json. I ran this file today and hit two surprises. ROBOTSTXT_OBEY defaults to False in Scrapy 2.19, so out of the box it ignores robots.txt, and naming a class attribute start crashed my spider, because Scrapy 2.13+ reserves that name for its engine hook.

Pros

• Pipelines, middleware, retries, and auto-throttle handle what a multi-day crawl needs
• 18 years of edge cases fixed; the logs tell you why a request failed
• BSD-3: embed in proprietary products

Cons

• A project, not a script: the first crawl costs an hour of scaffolding
• robots.txt off by default, which is the wrong default for strangers’ sites
• No markdown output; JS sites need scrapy-playwright and a second moving part

2. Crawlee: production crawls in TypeScript

Crawlee wraps Cheerio for static sites and Playwright/Puppeteer for dynamic ones behind one interface, with queueing, proxy rotation, and retries already built. 26.2k stars, Apache-2.0, pushed the day before I wrote this. npx crawlee create scaffolds a working project in under a minute, and maxRequestsPerCrawl gives it the page cap Colly and wget lack.

import { CheerioCrawler } from "crawlee";

const crawler = new CheerioCrawler({
  maxRequestsPerCrawl: 20,
  async requestHandler({ $, request }) {
    $("div.quote").each((_, el) => {
      console.log($(el).find("span.text").text());
    });
  },
});
await crawler.run(["https://quotes.toscrape.com/"]);

Pros

• One request queue and storage layer across static and browser crawls
• Resume and deduplication built in, not bolted on as plugins
• The only framework here with fingerprint rotation for browser crawls

Cons

• Browser mode eats RAM; twenty Playwright tabs want 16GB, not a laptop
• Markdown conversion is DIY; you get DOM and you do the cleaning

3. Crawl4AI: markdown for LLM pipelines

At 85.6k stars since mid-2024, Crawl4AI is the fastest-growing project here, for a narrow reason: it turns pages into LLM-friendly markdown locally, with no API key. Deep crawling ships in the core config; BFSDeepCrawlStrategy(max_depth=2) means the seed page plus two link levels. Apache-2.0, verified against the repo at v0.9.5.

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, BFSDeepCrawlStrategy

async with AsyncWebCrawler() as crawler:
    results = await crawler.arun("https://docs.example.com",
        config=CrawlerRunConfig(deep_crawl_strategy=BFSDeepCrawlStrategy(max_depth=2)))

Pros

• Clean markdown tuned for RAG chunks, at zero marginal cost
• Runs fully local; Ollama covers extraction without cloud keys
• Adaptive crawling reuses what worked on repeat visits

Cons

• asyncio ceremony leaks into every calling script
• The deep-crawl API is young and still moving between minor versions

For the hosted alternative to this workflow, see the Crawl4AI alternatives comparison.

4. Colly: the Go minimalist (Apache-2.0, not MIT)

Colly gives you callback-driven crawling in about 15 lines of Go: Visit, OnHTML, per-domain rate limits. 25.6k stars, single static binary output, and the core is one package you can read in an afternoon. The license is Apache-2.0; the MIT label that follows it around the internet has never matched its LICENSE.txt.

This is the crawler whose defaults I’d trust for a first crawl: it fetches and obeys robots.txt unless you set IgnoreRobotsTxt, verified in the v2.3.0 source. Depth semantics are the trap: MaxDepth(1) means seed-only, because Visit() seeds at depth 1 and children land at depth 2, which requestCheck then rejects. My first run today fetched exactly one page because of this.

c := colly.NewCollector(
    colly.AllowedDomains("quotes.toscrape.com"),
    colly.MaxDepth(2),
)
c.Limit(&colly.LimitRule{DomainGlob: "*", Delay: 1 * time.Second})
c.OnHTML("a[href]", func(e *colly.HTMLElement) {
    e.Request.Visit(e.Request.AbsoluteURL(e.Attr("href")))
})

Pros

• Smallest API here; a Go developer is productive in ten minutes
• Compiles to a binary your cron job runs for years untouched
• Errors on blocked URLs instead of skipping them quietly

Cons

• No JavaScript rendering; a React site returns an empty shell
• No page cap, no markdown, no output layer: callbacks give you raw HTML and you build the rest
• A selector that stops matching fails silently; there is no alerting

5. Katana: discovery, not content

ProjectDiscovery’s Katana (17.7k stars, MIT) crawls to map attack surface: endpoints, parameters, JS file references. katana -u https://quotes.toscrape.com/ -d 1 -rl 1 -silent returned 165 unique URLs from the same seed that gave other crawlers 12 pages, in 64 seconds at one request per second. It parsed nothing; that’s the point. If the deliverable is a URL list, this find-all-pages workflow is Katana’s home turf.

Pros

• Highest URL yield per flag of any tool here
• JSONL output feeds straight into other tooling
• Field-tested by security teams, which means weird sites fail loudly

Cons

• Extracts no content, so you need a second tool for anything textual
• Scope control is manual; a loose glob crawls hosts you didn’t intend

6. Browsertrix Crawler: archives, not datasets

Browsertrix Crawler (Webrecorder, 1.2k stars, AGPL-3.0) drives a full browser per page and records what a real visitor would see into WACZ archives: replayable, format-standard captures used by libraries and courts. One Docker container runs it: docker run webrecorder/browsertrix-crawler crawl --url https://example.com --limit 10. If the question is “what exactly did this page look like on this date,” nothing else here answers it.

Pros

• Highest-fidelity capture available in open source, JS and all
• WACZ output replays offline and holds up as evidence
• Crawler behavior is configurable down to per-URL scope rules

Cons

• Heavy: browsers per page means storage and RAM scale linearly with ambition
• WACZ is an archivist format; extracting records from it is a separate job

7. Heritrix: the Internet Archive’s workhorse

Heritrix is what crawls for the Wayback Machine: Java, Apache-2.0 (the license file says so even though GitHub’s detector gives up), 3.3k stars, built for crawls that run a month and stay polite throughout. Jobs are Spring bean files, driven from a local web UI. Its frontier and politeness policy are the most mature in open source; its extraction story ends at WARC files.

Pros

• Decades-tested politeness at web scale; sites rarely notice it
• Precise scope control (surts, regex, depth budgets)
• WARC output accepted by every archival tooling chain

Cons

• Steep setup: JVM, bean configs, and a UI that assumes an operator
• No structured extraction; parsing WARC is your problem

8. Apache Nutch: search indexing at Hadoop scale

Nutch exists to feed search indexes, not analysts. It runs crawl cycles (inject, generate, fetch, parse, updatedb) across a Hadoop cluster, Apache-2.0, 3.3k stars, pushing into Solr or Elasticsearch. For an open-web corpus and a rack of machines, it remains the sane choice. For one site on one machine, it’s a Hadoop dependency looking for a reason.

Pros

• Horizontal scale on commodity clusters, proven for 20 years
• robots.txt compliance and re-crawl scheduling built into the cycle model
• Direct integration with Solr/Elasticsearch indexing

Cons

• Hadoop is the price of admission, even for small jobs
• Batch cycles, not interactive crawls; latency is measured in job queues

9. HTTrack: the mirror

HTTrack (GPL-3.0, 4.8k stars, C) copies a site to disk as an HTML tree, rewriting links so the copy browses offline. httrack "https://example.com/" -O "./mirror" "+*.example.com/*" -r2 is the classic incantation. It’s the desktop-app-era veteran of this list and it still works, but the output is raw HTML with boilerplate intact. wget (GPL-3.0, same job category) shows up in my crawl-off below; HTTrack adds resume and filters on top of the same behavior.

Pros

• Set-and-forget mirroring with filters and interrupted-crawl resume
• GPL gives you the whole source with no strings beyond copyleft

Cons

• No cleaning, no extraction, no structured output
• JavaScript-rendered pages mirror as empty shells

10. Magpie: the local-first turnkey option

Full disclosure: I work with this one, which is why its limits sit right next to its features. Magpie is a single static Go binary (AGPL-3.0-or-later or a commercial license; the CLI is a free download) that crawls into cleaned markdown or JSON with fetch, robots check, and rate limiting handled by flags instead of code.

magpie crawl https://quotes.toscrape.com/ \
  --max-pages 12 --max-depth 1 --rate 1 --corpus --out quotes.jsonl

That command is the Magpie row in the crawl-off below: 12 pages, zero errors, 15.6 seconds, 3,466 words of cleaned markdown, no LLM calls. Beyond crawling it carries login sessions, JS rendering with click/fill actions, 27 zero-LLM verticals, and a 13-tool MCP server so coding agents can drive it.

Pros

• One binary, no runtime dependencies; the crawl-off command runs offline
• Live progress, resumable runs via a SQLite visited-set, sitemap seeding with automatic 404 fallback
• robots.txt respected and --auto-throttle backs off on 429/5xx by design

Cons

• Local means local: no hosted fleet, no cloud dashboard; you schedule it
• The desktop app costs $49/month after 5 free runs per install (the CLI has no run counter)
• Single-maintainer project; release cadence over enterprise SLA

What free web crawler data extraction gets you

Free gets you the crawl. Extraction is where the tools diverge, so I ran five of them against quotes.toscrape.com on 2026-10-16 from a home connection: one request per second each, back to back. The site is a scraping sandbox with no robots.txt (404) and about 60 linkable pages.

Crawler	Config (trimmed)	Fetched	Wall time	Deliverable
Magpie 0.1.32	crawl --max-pages 12 --max-depth 1 --rate 1 --corpus	12 pages, 0 errors	15.6s	3,466 cleaned words as JSONL
Colly v2.3.0	MaxDepth(2), 12-page counter, 1s delay	12 pages	19s	37 quotes via CSS callbacks
Scrapy 2.19.0	DEPTH_LIMIT=1, next-page spider, 1s delay	2 pages	3.3s	20 JSON records
wget 1.21.4	-r -l 1 --wait 1	47 HTML files	82s	712K raw HTML
Katana v1.8.0	-d 1 -rl 1 -silent	165 URLs	64s	URL list, no content

Read the fetched column twice: the numbers differ by configuration semantics, not just quality. Magpie and Colly both “crawled depth 1” and got 12 pages, because Magpie’s --max-depth counts links from the seed while Colly’s MaxDepth(2) compensates for its 1-based seed. Scrapy got 2 because my spider followed the pagination chain instead of sibling links. wget took every direct link including tag and author pages, because nothing in its flag set caps page count, only depth. Katana doesn’t fetch content, so its 165 includes CSS and JS assets.

Three runs cost me a debugging round-trip each, which no feature table would have shown: Colly’s depth off-by-one, Scrapy’s reserved start attribute, and wget’s missing page cap. Expect one on your first afternoon with any of them.

Web crawler software free for commercial use

Every crawler here is free to run commercially at zero license cost. Differences start when you redistribute or modify. The permissive group (Apache-2.0: Crawlee, Crawl4AI, Colly, Heritrix, Nutch, StormCrawler; BSD-3: Scrapy; MIT: Katana) embeds into proprietary products with attribution notices and nothing else.

The copyleft group needs a decision first. HTTrack and wget are GPL-3.0: use them as standalone tools however you like, but ship modified versions of their code and you release that source. Browsertrix Crawler and Magpie are AGPL-3.0, which adds the network clause: offer a modified build as a service and you owe users the source, no distribution required. That clause is why Magpie sells a commercial license alongside the free one. Free still means zero dollars to operate; it doesn’t always mean “no lawyer ever reads your repo.”

Picking an open source site crawler

By deliverable, since that’s what the crawl-off actually measured:

• JSON records at scale: Scrapy. Nothing here matches its pipeline durability.
• Clean markdown for LLMs, locally: Crawl4AI or Magpie; the difference is flags versus asyncio.
• Production JS-heavy crawls in a team codebase: Crawlee.
• URL discovery: Katana, then hand the list to whichever extractor fits.
• Archives with replay: Browsertrix Crawler; Heritrix if the crawl is measured in weeks.
• A mirror on disk: HTTrack or wget.

Two checks before you commit. Does the target need JavaScript? That eliminates Colly, HTTrack, wget, and Nutch in one stroke. Will the crawl run more than once against a changing site? Then selector maintenance beats first-run speed, and you want scraping politeness settings configured before the second run, not after the first 429. For the crawling-versus-scraping line, see web scraping vs web crawling; the scraper-side siblings live in the open-source web scraping tools comparison, and the free scrapers list covers no-code options.

Where the web crawler source code lives

Every repository in this piece pushed commits within eleven days of 2026-10-16, unusually healthy for an “open source” category: scrapy/scrapy, gocolly/colly, projectdiscovery/katana, apify/crawlee, unclecode/crawl4ai, internetarchive/heritrix3, apache/nutch, webrecorder/browsertrix-crawler, xroche/httrack, and motherlodelab/magpie. Activity is not uniform. Crawlee, Crawl4AI, and Katana ship visible changes weekly; Colly’s master branch moves but tagged releases come slowly; HTTrack is in maintenance mode; Heritrix changes when the Internet Archive needs it to.

Hackability tracks codebase size. Colly’s core is one readable package; you will understand your fork. Scrapy rewards extension points (middlewares, pipelines, extensions) over forks. Heritrix and Nutch expect you to speak their plugin dialects. Magpie ships WASM plugins, so you extend without rebuilding the binary.

The practical advice with any web crawler source code: clone it before you depend on it, read the defaults (Scrapy’s robots setting is a one-line grep), and pin a version. The day a request fails silently, you’ll be reading that code at 2 a.m. anyway.

FAQ

Which web crawler tool is open source?

All ten in this piece. Licenses span MIT, BSD, Apache-2.0, GPL-3.0, and AGPL-3.0. The fastest CLIs to start with are Katana and Magpie; the rest are libraries or systems first.

Do open-source crawlers respect robots.txt?

Defaults decide it. Colly, wget, Heritrix, Nutch, and Magpie obey robots.txt out of the box. Scrapy does not: ROBOTSTXT_OBEY defaults to False in 2.19. Set it to True in any project that touches sites you don’t own.

Is Scrapy free for commercial use?

Yes. BSD-3-Clause imposes no restriction on commercial use or on selling products built with it; keep the license notice. The AGPL pair (Browsertrix, Magpie) requires source disclosure only when you offer modified versions as a network service.

Can a free open-source crawler render JavaScript?

Crawlee, Crawl4AI, Browsertrix Crawler, and Magpie drive a real browser. Scrapy needs the scrapy-playwright plugin. Colly, HTTrack, wget, and Nutch parse static HTML only; against a React site they return the empty shell the server sent.

Before pointing any of these at a site you don’t own, do the three-line setup that all five crawl-off runs used: cap the pages, set the rate to one request per second, and grep the robots default. The politeness budget costs you nothing at 1 rps and it’s the difference between a crawler and an incident.