Web Scraping With Claude: MCP Setup in 5 Minutes

Ask Claude to pull the rows off a pricing page and it will explain, politely, that it cannot open URLs. You paste the HTML yourself, Claude works on the paste, and you have become the fetch layer. Claude web scraping only works once Claude has a fetcher, and MCP (Model Context Protocol, the standard socket between AI clients and local tools) is how you attach one.

The fetcher in this guide is Magpie, a local Go CLI that also runs as a 13-tool MCP server. Setup is three commands and one client restart. I ran every command below today on the machine this article was written on, so the outputs are real, including the failure case in Step 5 that cost me an hour to diagnose.

What Claude gets when this is done

Magpie’s MCP server registers 13 tools. I confirmed the list against the live server with a tools/list call, not against documentation:

Tool	What Claude can do with it
scrape_url	Fetch one URL, return clean markdown (or JSON/HTML/text); escalates to a real browser when the page needs JS; optional LLM schema for typed fields
crawl_site	Crawl a site to a page/depth cap and extract per page
map	List sitemap-derived URLs for a site
batch	Scrape up to 100 URLs with bounded concurrency, markdown only
extract_structured	Pull typed JSON out of HTML or markdown it already has (no fetch)
summarize	Summarize one URL in at most N sentences
diff	Word-level diff of a URL against a previous snapshot
brand	Extract colors, fonts, logo, favicon
list_extractors	List the 27 zero-LLM vertical extractors (jobs, events, articles, RSS)
vertical_scrape	Scrape one URL with a named vertical extractor, no LLM
search	Web search via your own SERP API keys
research	Multi-step research with a cited report, cost-capped
get_cached_selectors	Show the selectors Magpie has cached and self-healed for a domain

Most of this needs no API key at all. The fetch-clean-extract pipeline is pure Go; an LLM enters only when you pass a schema or use search/research, and then it bills your key, not Claude’s context.

Before you start

• Go 1.26 or newer (the build is a single static binary, no CGO).
• Claude Code or Claude Desktop, installed and signed in.
• An LLM API key only if you want schema extraction or research. Everything in Steps 1 through 4 runs keyless.
• Chrome or Chromium, optional. Only browser-rendered pages need it; without a browser installed, Magpie downloads a pinned Chromium (about 150 MB) on first render.

Step 1: Build the CLI

git clone https://github.com/motherlodelab/magpie.git
cd magpie
go build -o magpie ./cmd/magpie
./magpie --help

Put the binary somewhere on your PATH. I keep mine at ~/.local/bin/magpie. The build is the slow part of the five minutes; the wiring that follows is two commands.

Step 2: Wire it into Claude Code

From inside your project directory:

magpie init --client claude-code

Output, verbatim from my run (I tested in a scratch directory):

wrote /tmp/claude-init-demo/.mcp.json

And the file it wrote:

{
  "mcpServers": {
    "magpie": {
      "command": "/home/domidex/.local/bin/magpie",
      "args": ["serve"]
    }
  }
}

Three things worth knowing about this command, because they are the difference between a tool you trust and one you run once:

• The path is absolute on purpose. GUI clients don’t inherit your shell PATH, so a bare "magpie" only works if you launched them from a terminal that has it. The init command resolves the real binary location for you.
• It merges, never overwrites. I tested this by planting a config with another server and a stray "extra": 42 key, re-running init, and getting both back intact alongside magpie. Your existing servers survive.
• The first write saves your original config as .mcp.json.magpie.bak next to it.

Start Claude Code in that directory. Project-scoped servers from .mcp.json need a one-time approval, so Claude Code will ask; approve magpie. Then run /mcp inside Claude Code. You should see magpie with a checkmark and a tool count of 13. If it shows as pending approval instead, rerun claude in that directory and accept the prompt.

One boundary to know: the WSL case. Claude Desktop runs on Windows and cannot execute a /home/you/... Linux binary. Either run Claude Code inside WSL (then everything above works as-is) or build a Windows magpie.exe and point Windows Claude Desktop at it. Match the OSes.

Step 3: Claude Desktop and Cursor take the same command

magpie init --client claude-desktop

That writes the same stanza into Claude Desktop’s config: ~/Library/Application Support/Claude/claude_desktop_config.json on macOS, %APPDATA%\Claude\claude_desktop_config.json on Windows. Then fully quit Claude Desktop and relaunch it; config changes only load on restart. Check the connectors picker (the “+” next to the chat box) or Developer settings for the magpie tools.

Cursor gets the third variant, magpie init --client cursor, writing to ~/.cursor/mcp.json. Open Cursor’s Settings → MCP, toggle magpie on, and note that later config edits need a Cursor restart; custom servers don’t hot-reload. The full walk-through, with Cursor’s quirks, is in the Cursor scraping guide.

Step 4: Make Claude scrape something

Now ask Claude in plain language:

Scrape https://quotes.toscrape.com/ and list each quote with its author.

Claude calls scrape_url with that URL and gets back clean markdown. Here is the actual tool response from my test run, abridged:

{"content":"“The world as we have created it is a process of our
thinking...” by [(about)](https://quotes.toscrape.com/author/Albert-Einstein)\n\n
“It is our choices, Harry, that show what we truly are..." by J.K. Rowling..."}

3,548 bytes of markdown, nav and scripts already stripped, and the usage envelope read prompt_tokens: 0, completion_tokens: 0, usd_estimate: 0. No schema was passed, so no LLM ran anywhere in that pipeline; the cleaning is deterministic Go. Claude reads the markdown from its context and answers. This is the cheap path, and for most “read this page” asks it is the right one.

If Claude ever answers from memory instead of calling the tool, add “use the magpie scrape_url tool” to the prompt. Models occasionally need the nudge.

Step 5: Hand it a crawl, and learn where the money goes

The crawl tool takes explicit caps, which Claude will set for you if you ask for “a small crawl”:

{"name":"crawl_site","arguments":{"url":"https://books.toscrape.com/","max_pages":3,"max_depth":1}}

My run returned in 5.2 seconds:

{"errors":3,"pages_crawled":3,"records":0,"run_id":"0304d7efae86fc1a-169161",
"status":"finished","usage":{"fetch_bytes":81932,"fetch_ms":2601,"fetch_pages":3,
"prompt_tokens":0,"usd_estimate":0}}

errors: 3 with pages_crawled: 3 is not a network failure. The fetch stage succeeded on every page (82 KB in 2.6 seconds). The extract stage failed on every page, and here is the part that cost me an hour: crawl_site always runs LLM extraction, even with no schema in the arguments. The code falls back to a permissive default schema and sends each page to your configured provider. My box’s provider token was dead, so all three pages failed with HTTP 401 from the extract stage while the fetches looked perfectly healthy in the usage numbers.

Two fixes, pick one:

• Give the server a working LLM key (any of the 7 supported providers, including a local Ollama) and crawls extract typed records per page.
• Stay on the zero-LLM tools: scrape_url and batch return markdown with no provider call at all. For “read these 40 pages” jobs, batch is the honest choice.

The sibling map tool has its own sharp edge: it reads /sitemap.xml, and the classic scraping sandboxes don’t ship one. Point it at books.toscrape.com and you get GET https://books.toscrape.com/sitemap.xml: HTTP 404. That is the site’s honest answer, not a bug; use a bounded crawl_site for URL discovery there.

When one server beats one per project

Two transports ship. The stdio setup above starts a Magpie process per client, which is what you want for personal use. If you want one shared server for several clients (or for curl), run the HTTP transport:

magpie serve --transport http --addr 127.0.0.1:8089

Then add it to Claude Code as a remote-style server:

claude mcp add --transport http magpie http://127.0.0.1:8089

The HTTP server is stateless, no sticky sessions, which I verified by firing raw JSON-RPC at it: initialize and tools/list work back-to-back with no session handle. It binds where you tell it, and the sane place is loopback.

Loopback is also where the SSRF guard earns its keep. Ask Claude to scrape http://127.0.0.1:8089 itself and the fetch refuses:

mcp: scrape_url: fetch: address "127.0.0.1" is not a public IP: fetch: non-public address

An agent with a fetch tool is a prompt-injection risk surface; a scraper that refuses private addresses by default and respects robots.txt is the correct shape for one. There is no flag to turn the guard off.

What Claude web scraping costs in tokens

Two separate meters, and conflating them is how people get surprised:

1. Claude’s context. Tool results land in the conversation. That 3,548-byte quotes page is roughly 900 tokens at the usual 4-characters-per-token arithmetic. When you only need the gist, a summarize call returns N sentences instead of the full page. Claude’s subscription prices this, not Magpie.
2. Magpie’s LLM calls. crawl_site extraction, scrape_url/extract_structured with a schema, and research touch a provider; search has keyless providers too. The keyed paths bill your key (BYOK, cost-capped), and every run envelope carries usd_estimate, so the spend is visible per call. Everything else on the 13-tool list is zero-LLM by construction.

Before you reach for a schema, ask whether you need one. A CSS-selector-free markdown scrape plus Claude reading it covers most asks; typed extraction earns its cost when you’re piping records into something else. The trade-off is its own topic: LLM extraction vs CSS selectors. If you want extraction with zero API spend, point the provider at a local model, which I tested end to end in the Ollama scraping guide.

The full trade-offs, plainly: Magpie is local-only (no hosted API, nothing runs in a browser tab), robots.txt respected, bot challenges detected and reported but never bypassed, and the walled platforms (Instagram, X, LinkedIn behind login, Amazon review text) are out of scope by design. The CLI is free, dual-licensed AGPL/commercial. The Desktop app is optional; it wraps the same engine with a GUI, an MCP toggle in settings, and shared history with the CLI, free for 5 runs per install, then $49/month.

FAQ

Is giving Claude a scraping tool safe? The tools run locally: fetching happens on your machine, keys stay in your environment, and the init command never writes secrets into client configs. The SSRF guard blocks private-network targets, robots.txt is respected, and challenges are detected, not bypassed. Check a site’s terms before scraping it; public data is the intended use.

Does web scraping with Claude use my Claude tokens? The tool result enters Claude’s context, so yes, at roughly 900 tokens for a 3.5 KB page. Magpie’s own LLM features are separate and bill your provider key, not Claude. Keep results lean with summarize or scoped crawls.

Can Claude scrape LinkedIn, X, or Instagram with this? No. Those platforms wall bots and Magpie does not defeat bot protection, by design. Public LinkedIn company pages load, but expect walled targets to fail loudly rather than return partial data.

Do I need the Desktop app for the MCP server? No. The 13-tool server ships in the free CLI and that is what magpie init wires up. Desktop adds a GUI, a settings toggle for the same server, and shared history, but the CLI alone covers everything in this guide.

Next move: point Claude at a site you actually care about and ask for one small crawl with max_pages: 5. Watch the usage envelope in the response; it tells you within seconds whether your provider key works before you commit a real job. For the tool-by-tool parameter reference, see the Magpie MCP page, and if you also ride ChatGPT, the same three-command pattern applies, walked through in the ChatGPT scraping guide.