Scrape a Documentation Site Into Clean Markdown

Rich’s Console API reference ships as 69,159 bytes of HTML. The words a reader needs fit in about 24,000 characters of markdown. The difference is the sidebar’s navigation tree, header and footer chrome, and markup soup that a tokenizer will happily eat as garbage before it reaches one sentence of prose.

A documentation scraper fixes this: fetch a docs site, strip the chrome, and hand back clean per-page markdown you can chunk into a retrieval pipeline, upload to a custom GPT, or drop into NotebookLM. Every command below ran today (2026-10-16) against rich.readthedocs.io (Read the Docs, Rich 14.1.0) from a home connection, with real timings and the junk I hit along the way.

Before you start. Magpie itself, one static Go binary (git clone https://github.com/motherlodelab/magpie && cd magpie && go build -o magpie ./cmd/magpie, about two minutes). A docs site you’re allowed to crawl. No API key, no account, no Python environment.

Check the escape hatches before you scrape

Scraping is the fallback, not the opening move. Since Jeremy Howard proposed the llms.txt convention in September 2024, a growing set of docs platforms publish machine-readable markdown on purpose. Mintlify generates llms.txt, llms-full.txt, and serves every page as markdown if you append .md to the URL. GitBook, Fern, ReadMe, and Docs7 auto-generate llms.txt too. Docusaurus, Starlight, and MkDocs need a community plugin, so most self-hosted installs have nothing.

Read the Docs is in the nothing camp. My checks against rich.readthedocs.io:

Escape hatch	Result on rich.readthedocs.io
/llms.txt	404
/robots.txt	Allows everything, declares a sitemap
/sitemap.xml	880 bytes, 5 URLs: the /en/latest/ and /en/stable/ roots plus 3 links to Read the Docs’ own marketing pages
.md suffix on a page URL	Not supported (Sphinx serves HTML)

If your target site runs on Mintlify or another auto-generating platform, take the free markdown and skip the rest of this guide. On Read the Docs, keep going.

Step 1: Probe one page before you crawl anything

Pick the most content-heavy page you actually care about and scrape it alone. Magpie is one static Go binary; the general clean-text pass it runs is covered in how to extract text from any website. The probe:

magpie scrape https://rich.readthedocs.io/en/stable/console.html \
  --render static --page-format markdown \
  --exclude a.headerlink \
  --out console.json

About 2.3 seconds. You get a JSON record with the final URL, the page title, and the content as markdown under a markdown key. The --exclude a.headerlink drops Sphinx’s pilcrow anchor links; without it every heading comes out as # Console API[¶](#console-api), which is pure noise once you split the text into chunks.

The numbers for this exact page:

Form	Size	Rough tokens (÷4)
Raw HTML	69,159 bytes	~17,000
Naive tag-stripping	19,455 chars, includes the sidebar nav	~4,900
Cleaned markdown	24,132 chars, no nav, real heading structure	~6,000

Read the middle row carefully, because it’s the honest part. Stripping tags mechanically gives you smaller text than the markdown, but it keeps the navigation tree as prose-shaped noise and flattens every heading. The markdown costs a little more here because this API page carries dozens of absolute cross-reference links.

What you’re buying is structure: headings a chunker can split on, code fences, zero boilerplate. On lighter pages the markdown is smaller too; the introduction page’s HTML is 18,995 bytes and cleans to 4,246 characters.

Failure cases I hit while probing:

• Guessed paths 404. quickstart.html doesn’t exist on this project. Don’t guess; open the index page and read the sidebar’s href values (console.html, tables.html, and so on).
• Empty or fragmented output. If a page comes back as a navigation shell, the docs render with JavaScript. Rerun without --render static; the default auto mode escalates to a real browser. That costs about 10 seconds per page instead of 2, which is why you probe before you multiply.

Step 2: Map the site, then expect less than you hoped

magpie map https://rich.readthedocs.io

Returns two URLs: /en/latest/ and /en/stable/. That’s Read the Docs’ autogenerated sitemap doing its job: version roots, not pages. On a site that ships a real sitemap, map hands you the full URL list in one call, which is the cheap way to find all pages on a website. On RTD you get nothing usable and must crawl with link-following, which is the next step.

One more trap hiding in those two URLs: /en/latest/ and /en/stable/ are separate builds of the same content. Crawl exactly one, or you’ll index every page twice and wonder why retrieval returns near-duplicates.

Step 3: Crawl one version, scoped to a path prefix

magpie crawl https://rich.readthedocs.io/en/stable/ \
  --path-prefix /en/stable/ \
  --max-pages 50 --rate 1 \
  --corpus --out corpus.jsonl

What each flag buys you:

• --path-prefix /en/stable/ keeps the crawler inside the stable build, so the latest-vs-stable duplication never enters the corpus.
• --rate 1 holds one request per second per host. Docs sites are static and famously crawlable; their robots.txt usually allows everything. Stay polite anyway. The wider tactics live in how to scrape without getting blocked, and for docs the main rule is: you will crawl this site again someday, don’t get your IP remembered as the one that hammered it.
• --corpus emits one JSONL record per page (url, title, depth, markdown) with zero LLM calls and no API key.

My first run: 12 pages, 0 errors, 11.9 seconds. Then I looked at the titles:

• _sources/index.rst.txt (699 chars, empty title): Sphinx exposes raw RST sources of every page under /_sources/. You just scraped the source code of the docs, alongside the docs.
• genindex.html (13,337 chars): the alphabet index. A link farm with a search result’s worth of weight.
• appendix.html (350 chars): a stub page that adds nothing.

None of this is broken behavior. Sphinx sites always expose _sources and genindex; the scraper can’t know you don’t want them. That’s your job, next step.

Step 4: Filter the junk, then validate the corpus

Rerun with URL-glob excludes:

magpie crawl https://rich.readthedocs.io/en/stable/ \
  --path-prefix /en/stable/ \
  --exclude '**/_sources/**,**/genindex.html,**/search.html' \
  --max-pages 50 --rate 1 \
  --corpus --out corpus.jsonl

Rerun result: 12 records, all real pages (Console API, Live Display, Logging Handler, Standard Colors). Now validate before anything downstream eats it. My checklist, in order:

1. Boilerplate grep. grep -ciE "edit on github|back to top|read the docs" corpus.jsonl. Expected on this run: 0 hits. Nonzero means the cleaner leaked chrome into content and you should inspect those records before chunking.
2. Size sanity. Sort records by markdown length. Anything under ~500 characters is a stub like the appendix page; keep it or drop it deliberately, not by accident.
3. Title dedupe. Two records with the same title and different URLs means version bleed or a redirect you didn’t expect. Zero here.
4. Spot-check a code-heavy page. This is where the one real loss shows up, detailed below.

The validated 12-page corpus: 76,087 characters of markdown, 7,991 words, an average of 6,340 characters or roughly 1,600 tokens per page. Against the raw console page at ~17,000 tokens, that’s the difference between a retrieval pipeline that fits a docs section in context and one that spends its budget on navigation.

Two output details worth knowing before you build on this:

• Crawl output keeps the pilcrow anchors. CSS-selector excludes are a single-page-scrape feature; the crawl’s --exclude takes URL globs, not selectors. The anchors cost 493 characters across the whole console page, so I leave them and strip downstream. If it bothers you, scrape by URL list instead of crawling.
• Titles carry the site suffix. Console API — Rich 14.1.0 documentation is the HTML <title>, all 42 characters of it. Truncate on the em-dash when you index.

Other ways people build a docs corpus

• docs-to-markdown (Python, MIT). Selenium plus BeautifulSoup, optionally asks OpenAI where the main content is, and compiles everything into one markdown file. For a custom-GPT knowledge file that’s fine; for chunking it’s an anti-pattern. One file means no per-page provenance, and re-scraping rewrites the blob.
• Firecrawl (cloud API). 1 credit per crawled page, 1,000 free credits per month, Hobby $19/month for 5,000 credits, Standard $83/month for 100,000 (billed annually; pricing effective September 4, 2026). A page that answers 403 or 404 still costs 1 credit. Your docs travel to their cloud and back, which is fine for public docs and a non-starter for the internal ones.
• curl plus pandoc. Fine for five pages. No boilerplate removal, no crawl scope, no dedupe, and by page 20 you’ve written half a scraper anyway.

For a corpus you re-ingest on a schedule, per-page records beat one blob, and local beats cloud when the docs are behind a login or just none of anyone’s business. That’s why I run this pass with Magpie, a local scraper with an MCP server, on my own machine: the crawl never leaves the box, and the same pipeline the CLI runs is exposed to agents on demand.

Where a documentation scraper falls short

Tested today, so you don’t have to discover it mid-pipeline:

• Code blocks collapse. Sphinx wraps samples in highlighted <div>s, and the markdown flattens them to single inline lines: from rich.console import Console console = Console() where the source has two statements on separate lines. Browser rendering does not fix it (tested: same collapse, 10.1 seconds instead of 2.3). Prose, tables, and console-output blocks survive fine. If your pipeline indexes runnable snippets, spot-check a code-heavy page first and consider keeping the raw HTML as a fallback for those pages.
• Link-heavy pages get heavier. Every cross-reference becomes an absolute URL with a fragment. Citable, good for provenance, expensive in tokens. The console page’s markdown outweighs its tag-stripped text for exactly this reason.
• No llms.txt out. The corpus is for you; it doesn’t publish anything back to the site.
• robots.txt is respected, full stop. There’s no bypass, and docs sites essentially never need one.
• Freshness is your job. Nothing watches the site for you on a crawl. More on that below.

Docs behind a login

Same pipeline, one extra step: save the session once (magpie profile save), then scrape and crawl with --as <profile>. Cookies persist per site in the local store. Internal docs portals, versioned wikis, staging copies of your product docs: all fair game, and the login-scraping walkthrough covers the setup. One warning from experience: you’re authenticated, so rate-limit yourself harder than you would in public. Account bans cost more than IP throttles.

Keep the corpus alive

Docs change slowly. Re-crawl monthly and diff on url plus title; the JSONL format makes that a five-line script. For the one page you care about daily (release notes, a changelog), skip crawling and use watch mode with a webhook instead. And if your actual consumer is an AI agent that needs one page on demand rather than a corpus, that’s the MCP server setup, which shares the same local store as everything above.

FAQ

Is scraping documentation legal? Public docs with permissive robots.txt are the friendliest target in scraping. Copyright still applies to the content, and a site’s terms can object; for training or AI-use questions, see the legal landscape. Not legal advice.

Why not just wait for llms.txt everywhere? Platform lock-in decides it. Mintlify-hosted sites have it today; Sphinx sites on Read the Docs don’t, and a community plugin is the only path. Until generation is standard, scraping is the portable option.

Does this work on MkDocs, Docusaurus, or GitBook? Yes, same pipeline. The junk differs: _sources and genindex are Sphinx things; Docusaurus has tag and category listing pages; GitBook publishes a content API that beats scraping. Probe one page, check the record, then adjust the exclude globs.

How current does the corpus stay? As current as your last crawl. Docs sites change weekly at most; monthly re-crawls are plenty for retrieval, and the diff tells you exactly what moved.

Start with the probe command on one page you know well. If the cleaned markdown reads like the docs you already trust, the scoped crawl is one command away, and the whole 12-page run you just read about took under 14 seconds.