Web Scraping in Cursor With a Local MCP Server

Asking Cursor to work with a web page usually ends one of two ways: you paste HTML into the chat and burn 10,000 tokens on boilerplate, or you wire up a cloud scraping MCP and start counting credits. There is a third option. This guide sets up cursor web scraping against a local MCP server called Magpie, which runs the whole fetch-clean-extract pipeline on your machine and hands the agent clean markdown instead of raw HTML. Setup is one command, the CLI is open source, and nothing you scrape leaves your machine.

The generic case for scraping through MCP I cover on the MCP web scraping overview. This page is Cursor-specific: exact config, exact tool list, and the failure modes I hit while testing it today.

Approach	Setup	Cost at volume	Data	Where it fails
Paste HTML into chat	None	Your context window, every time	Goes to the model provider	Pages over ~50KB get truncated; JS shells have no HTML to paste
Cloud scraping MCP (Firecrawl, Bright Data)	API key + mcp.json entry	Per-page credits, roughly $0.001–0.01+/page depending on plan and options	Pages travel to the vendor and back	Credit walls mid-job; walled sites need premium proxies
Local MCP server (Magpie)	magpie init --client cursor	Free, no credits; LLM-dependent tools are BYOK	Never leaves your machine	No CAPTCHA bypass; sites that wall datacenter-free home IPs can still refuse

Before you start. A current Cursor build (MCP is a standard Agent-mode feature; the Cursor docs don’t gate it by plan). Go installed, because you build Magpie from source. About five minutes.

Step 1: Build the Magpie CLI

Clone and compile:

git clone https://github.com/motherlodelab/magpie.git
cd magpie
go build -o magpie ./cmd/magpie

One static binary, pure Go, no runtime dependencies. The licence is AGPL-3.0-or-later or commercial, and no paid tier gates any tool in this guide. Drop it somewhere on PATH (mv magpie ~/.local/bin/ works on Linux and macOS). Smoke-test it outside Cursor first, so you know the pipeline works before adding the agent layer:

magpie scrape https://go.dev/blog/go1.24 --render static

You should get the article as clean markdown in a couple of seconds. If this fails, fix it now; Cursor will just inherit the failure.

Step 2: Connect it to Cursor

Magpie ships a writer for client configs, and Cursor is one of the three it knows:

magpie init --client cursor

Output, verbatim from my run:

wrote /home/you/.cursor/mcp.json

The file it writes (this is --dry-run output, which prints the stanza without writing):

{
  "mcpServers": {
    "magpie": {
      "command": "/home/you/.local/bin/magpie",
      "args": ["serve"]
    }
  }
}

Three properties of this command matter in practice:

• It writes an absolute path. GUI-launched apps don’t inherit your shell’s PATH, so a bare "magpie" would only work by luck. The init command resolves the executable’s real path and uses that.
• It merges, never replaces. I tested this with an existing config holding another server plus a custom key ("n": 12345): both survived the merge byte-for-byte, and Magpie saved my original file as mcp.json.magpie.bak before its first write.
• No env block, ever. The stanza carries no secrets. We deal with API keys in step 4.

One failure case to know: if you’ve never installed or launched Cursor, ~/.cursor may not exist, and the init command refuses to create it. The fix is mkdir -p ~/.cursor and re-run. I hit exactly this in a clean sandbox; the error names the path it couldn’t write.

Now open Cursor: Settings → MCP (or the MCP entry under Customize), and you should see magpie listed. Toggle it on. Cursor spawns the server over stdio and marks it ready once the handshake succeeds. If you edit the config later, restart Cursor; updated custom servers don’t hot-reload.

Step 3: First scrape from the Agent chat

Open the Agent chat and ask for something concrete:

Scrape https://news.ycombinator.com and summarize what's on the front page.

Cursor asks for approval before using MCP tools by default, shows the scrape_url call, then renders the result. Under the hood it made exactly the JSON-RPC calls I made while testing, so I captured the real exchange over stdio. Tool listing returned all 13 tools in 0.01s. The scrape call:

{"jsonrpc":"2.0","id":3,"method":"tools/call",
 "params":{"name":"scrape_url","arguments":{"url":"https://news.ycombinator.com"}}}

The result came back in 0.79s: a JSON envelope with final_url, from_cache, and 19,278 characters of cleaned markdown. That envelope shape is deliberate. The agent gets machine-readable metadata plus text it can quote, not a wall of HTML it has to wade through.

Why not just paste the HTML? Size. The same Go blog post is 36,933 bytes of raw HTML; Magpie’s cleaned main content is 4,330 characters, about an 8× reduction. At the usual ~4 characters per token, that’s roughly 9,000 tokens of boilerplate against 1,100 tokens of article. On an article-heavy site the cleaned markdown also strips cookie banners and nav, which is work the model would otherwise do badly.

Step 4: The 13 tools, and which ones you’ll actually use

tools/list returns these, and Cursor exposes every one to the agent:

Tool	What it does	Needs an LLM key?
scrape_url	Fetch, clean, extract one URL to markdown or a schema	Only with a schema
crawl_site	Crawl a site (runs synchronously) or poll a previous run	Only with a schema
batch	Up to 100 URLs, bounded concurrency	No (markdown only)
map	List sitemap-derived URLs for a site	No
extract_structured	Schema-driven extraction from HTML/markdown you already have	Yes (BYOK)
summarize	Summarize one URL in at most N sentences	Yes (BYOK)
diff	Word-level diff of a URL against its last snapshot	No
brand	Colors, fonts, logo, favicon from a URL	No
list_extractors	List the zero-LLM vertical extractors	No
vertical_scrape	Typed extraction (jobs, events, articles, RSS…)	No
search	Web search via SERP providers, optionally scrape top hits	SERP provider config
research	Multi-step research with per-quote verification	Yes (BYOK, cost-capped)
get_cached_selectors	Inspect the self-healing selector cache for a domain	No

Day to day, scrape_url and map do most of the work, with vertical_scrape quietly being the best deal: typed JSON with no LLM call at all. Two setup notes for the tools that do call an LLM:

magpie config set-key

stores provider keys in your OS keyring (config resolution order: flag, then environment, then keyring). The keyring path matters because Cursor spawns the server as a GUI child process without your shell’s environment, so a key exported in .bashrc never reaches it. Cost-capped by design: research refuses to run past your --max-cost ceiling.

A realistic agent session now looks like: “map docs.example.com, pick the pages about authentication, scrape them, and tell me which endpoints need session tokens.” The agent chains map → scrape_url per page → its own reasoning, and you approve each tool call.

Running it over HTTP instead of stdio

stdio is the right default for one client on one machine. Switch when you want several tools pointed at one server, or long crawls you don’t want bound to an editor session:

magpie serve --transport http --addr 127.0.0.1:8787

The HTTP transport is stateless Streamable HTTP, no sticky sessions. I verified the endpoint with curl (initialize handshake answered in one round-trip, then a stateless scrape_url completed in 1.73s) and confirmed the docs’ contract: what sits behind the transport is the same server. For HTTP servers, Cursor’s config takes a URL instead of a command, so the entry becomes:

{ "mcpServers": { "magpie": { "url": "http://127.0.0.1:8787/mcp" } } }

The init command writes the stdio shape only; if you go HTTP you edit this entry by hand. Keep the address on loopback unless you add auth, because an unauthenticated scraper on your LAN is a gift to other people.

What breaks (I hit most of these)

Cursor updates. After updating custom servers, Cursor requires a restart; the server shows disabled until you do. Not Magpie’s fault, but it will look like one.

map needs a sitemap.xml. It is sitemap-derived by design. On quotes.toscrape.com it returns a clean error, verbatim: mcp: map: crawl: map: GET https://quotes.toscrape.com/sitemap.xml: HTTP 404. On a big site like bbc.co.uk it works but returns truncated: true, because the full sitemap set is enormous. When a site has no sitemap, have the agent crawl with a tight max_pages instead.

Wrong arguments get caught. I called map with {"url": ...} and got a typed validation error naming the correct field (site) instead of a silent empty result. That’s the MCP schema system doing its job; the agent retries with the right argument on its own.

Localhost targets are refused. The SSRF guard rejects private and loopback URLs, so you can’t point the agent at your dev server. The error message blames robots.txt, which is misleading; it’s the SSRF guard, and there’s no CLI opt-out. Test local apps by feeding Cursor the HTML directly.

Walled sites still wall you. Magpie detects bot challenges and reports them; it does not defeat them. Worse, some walls look like success: old Reddit served my home IP a redirect to a locale stub page, which came back as a valid-looking scrape of 200 wrong words. When an agent reports data from a walled site, spot-check one value in a browser before you trust it. The extraction blocklist (Instagram, X, LinkedIn behind login, and about two dozen more) hasn’t changed.

Long crawls block the chat. crawl_site runs synchronously to completion, so the agent waits with you. Cap max_pages (I stay under 10 in chat), or run the crawl over HTTP transport with a generous client timeout.

Where Cursor web scraping falls short

A local server has real limits, and I’d rather you know them now. No CAPTCHA or bot-protection bypass, by design, so heavily defended sites stay out of reach. Data stays local, but the pages still come from your home or office IP, so volume politeness rules apply; Magpie respects robots.txt and auto-throttles, which is the right behavior and also a speed limit. And when a tool calls an LLM, your prompt and page content do go to that LLM provider, because that’s how LLMs work; the zero-LLM tools avoid even that.

If you need blocked-site coverage or per-request residential rotations, that’s what the cloud scraping MCP options are for, and mixing the two is a reasonable setup: local for the 95% of jobs that are ordinary pages, cloud for the rest.

FAQ

Does Cursor support MCP on the free plan? The current Cursor docs describe MCP as a standard Agent-mode feature and don’t gate it by plan. If your build hides the MCP panel, update Cursor first; server config moved between versions.

Why not just have Cursor’s agent fetch pages itself? Cursor’s built-in web search fetches through someone else’s infrastructure and returns summaries, not source data. A local scraper returns the actual cleaned page, works behind your own sessions, and doesn’t consume your context window with boilerplate.

Where does my data go? Nowhere, for the zero-LLM tools: the fetch happens from your machine and the markdown lands in your editor. Tools that call an LLM send page content to the provider you configured, under your key, with a cost cap you set.

Can it use my logged-in sessions? Yes. Magpie saves cookies per site, and scrape_url accepts a profile or raw cookie header, so login-walled pages you have access to work the same as they do from your browser.

Windows and macOS? Same story. The binary cross-compiles with CGO_ENABLED=0, and the init command writes the absolute path either way. The only OS-specific bit is the keyring backend used by magpie config set-key.

Where to point it next

Once the server is connected, the highest-value job isn’t one-page scraping, it’s monitoring: point the agent at a competitor’s changelog with diff, or set up automatic change tracking with webhooks so the news arrives before you ask. For pages that need JavaScript executed before the data exists, the same pipeline escalates to a real browser automatically; I cover that in scraping dynamic websites. And if you run Claude instead of Cursor, the identical server connects in one command; I wrote that up separately in web scraping with Claude. The selector cache that keeps these extractions working after site redesigns is the self-healing scraper mechanism, and politeness settings for hitting one host harder than a crawl should are in scraping without getting blocked.