Website Summarizer: Summarize Any URL in N Sentences

The typical website summarizer is a web form on someone else’s server. You paste a URL, wait in a queue, get a result capped at 600 words on QuillBot’s free tier or a daily allowance on TLDR This, and hand your reading list to a third party along the way. This guide replaces the form with one command: magpie summarize fetches the page, cleans it, and condenses it to at most N sentences using your own LLM key, printing JSON you can script.

Everything below ran on magpie v0.1.32 on 2026-10-16 from a home connection. Outputs are pasted verbatim. By the end you’ll have a command that turns any public URL into a fixed-length summary, tells you exactly what each page cost in tokens, and fails loudly when a page refuses to be read.

What a summarizer has to do before the model sees anything

Condensing is the easy part. The fetch decides quality. Feed a model the raw HTML of a news article and half the prompt tokens buy navigation menus, cookie banners, and a related-articles rail.

magpie summarize sits on the same pipeline as the Magpie scraper: fetch, clean, then one prompt call. Boilerplate gets stripped, the model receives clean markdown, and a real browser is spun up only when the page needs JavaScript to render. Same fetch path, same robots.txt respect, same SQLite history. If you want the cleaned markdown without any model at all, that’s magpie scrape; the summarizer is scrape plus one prompt.

That cleaning step decides more than any model choice will. If you care about the underlying mechanics, the markdown vs HTML for LLMs comparison shows what a model does and doesn’t read in each format.

Before you start

You need three things.

1. The magpie binary. Clone and build:

   git clone https://github.com/motherlodelab/magpie.git
   cd magpie
   go build -o magpie ./cmd/magpie

   Any machine with a Go toolchain works; the binary is static.

2. One LLM key. Seven providers are supported: anthropic, openai, openrouter, opencode-zen, opencode-go (all key-bearing), plus ollama (local, no key) and codex (uses your Codex CLI login, no separate key). A summary costs a fraction of a cent on small models; exact math comes in Step 4.

3. A reachable URL. magpie fetches like a polite scraper: robots.txt respected, no CAPTCHA or paywall bypass, by design.

Step 1: Give magpie an LLM key

Three ways to hand it over. For one command, use the --api-key flag. For a shell session, export an environment variable (MAGPIE_ANTHROPIC_API_KEY, MAGPIE_OPENAI_API_KEY, and so on per provider). Or store the key for good in your OS keyring:

magpie config set-key

Skip this and the command tells you exactly what’s missing:

Error: scrape: provider anthropic: missing API key: set via --api-key flag, MAGPIE_* env, or `magpie config set-key`

The key travels from your machine to your provider and nowhere else. There is no magpie account, no proxy server, no third party reading along.

Step 2: Summarize one URL

magpie summarize https://en.wikipedia.org/wiki/Web_scraping --max-sentences 3

Real output from that exact command (I ran it on a free OpenRouter model; the JSON shape is identical for every provider):

{
  "url": "https://en.wikipedia.org/wiki/Web_scraping",
  "final_url": "https://en.wikipedia.org/wiki/Web_scraping",
  "title": "Web scraping - Wikipedia",
  "summary": "Web scraping is the automated or manual extraction of data from websites using bots, crawlers, or specialized tools, typically saving the information into databases or spreadsheets for analysis. Techniques range from simple text matching and HTTP requests to browser automation via DOM parsing, while websites employ defenses like CAPTCHAs, IP blocking, and TLS fingerprinting to deter scrapers. The legality of web scraping varies by jurisdiction, with court rulings in the U.",
  "provider": "openrouter",
  "model": "nvidia/nemotron-3.5-super-vl:free",
  "usage": {
    "prompt_tokens": 7939,
    "completion_tokens": 436,
    "usd_estimate": 0
  }
}

Read the usage block before the summary. That’s your per-page cost receipt: 7,939 prompt tokens, 436 completion tokens, and a USD estimate magpie computes from the provider’s pricing. final_url shows where redirects actually landed, which matters when a shortlink goes somewhere unexpected.

Step 3: Cap the length at N sentences

--max-sentences accepts 1 to 20 and defaults to 3. The prompt asks the model for at most N sentences, then magpie hard-cuts the reply at the Nth sentence boundary with its own splitter. The model does not get the final word on length. In the run above it rambled for 436 completion tokens; what you received is three sentences.

One quirk to know, spotted in that same output: the splitter counts every period, question mark, and exclamation point, and abbreviations contain periods. My Wikipedia summary ends at “court rulings in the U.” because the period in “U.S.” consumed a sentence slot mid-thought. If your summaries keep dying one clause short, nudge N up by one. It’s a blunt instrument, and honest about being one.

The whole command, fetch to JSON, took 23 seconds against a Wikipedia article that maxed out the 4,000-word input cap, measured with time on a free-tier model. Most of that is the model call, not the fetch.

Step 4: Pick the provider, or let auto do it

--provider takes anthropic|openai|ollama|openrouter|codex|opencode-go|opencode-zen|auto. --model takes any model the provider serves; leave it off and magpie uses a sane default per provider (gpt-4o-mini for OpenAI, llama3.1 for Ollama, claude-sonnet-5 for Anthropic, and similar).

--provider auto tries keyed providers in order: anthropic, openai, openrouter, opencode-zen, opencode-go, then the keyless ones, ollama and codex. Providers with no key configured are skipped, so auto degrades gracefully instead of erroring on the first miss.

Budget math, so nobody gets surprised: input is capped at the first 4,000 words of cleaned markdown, my Wikipedia run consumed 7,939 prompt tokens after that cap, and gpt-4o-mini lists at $0.15 per million input tokens (checked 2026-10). That works out to roughly $0.0015 per page, or about 15 cents per 100 pages, plus almost nothing for the summary itself. Free tiers exist too: the demo above ran on a :free OpenRouter model at a usd_estimate of 0, and Ollama runs entirely on your machine for zero cost, if your hardware carries the model.

Two escape hatches round this out. Any OpenAI-compatible endpoint can be addressed by pointing MAGPIE_BASE_URL at it and using --provider openai. And --max-cost sets a USD ceiling per call, so a misconfigured expensive model aborts instead of quietly billing you.

Step 5: Wire it into scripts and agents

The JSON envelope is the automation surface. One URL per line, one file per page:

while read -r url; do
  magpie summarize "$url" --max-sentences 5 --out "$(echo "$url" | md5sum | cut -c1-8).json" \
    || echo "$url" >> failed.txt
  sleep 2
done < urls.txt

Three more surfaces share the same machinery. Every summarize writes a run row to the same SQLite store the desktop app reads, so your command-line summaries show up in Desktop history. The desktop app itself has a Summarize action on any page already in its workspace; it reuses the fetched markdown with zero re-fetch. And the built-in MCP server exposes summarize as one of its 13 tools, which is how an agent in your editor calls the exact command you just learned, minus the shell.

Where it breaks

• Long pages lose their tails. Input stops at 4,000 words. A 9,000-word investigation gets summarized from its first 4,000; the model never sees the rest. Split long pages yourself or accept the loss.
• Walled pages stay walled. Hard paywalls and bot challenges stop the fetch, and magpie reports the failure rather than pretending. That’s the product working as designed; there is no CAPTCHA bypass to disable.
• One URL per call. “Summarize this whole site” is a crawl job, not a summarize job. If your actual goal is site-scale reading, start with AI web crawlers instead.
• The splitter is naive, as Step 3 showed. Fine at N of 5 or more; noticeable at N of 1 or 2.

How this compares to hosted website summarizers

The hosted tools are fine for what they are. QuillBot summarizes pasted text, 600 words on the free tier, paragraph or bullet output. TLDR This runs a web tool plus a Chrome extension with daily limits on the free plan. Edge’s Copilot summarizes the page in front of you, and pasting a URL into ChatGPT works until it hits a wall. For “clear this one tab,” any of them will do.

Pick the CLI when any of these is true: the pages are behind a login you already have (profiles and cookies work per site), the job repeats nightly or in bulk, the text feeds a downstream script, or the URLs are nobody’s business but yours. A hosted form can’t read pages only you can see, and it can’t be cron’d. A website summarizer that runs locally can do both, with a token receipt attached to every result.

One legal note since this is a fetching tool: summarizing public pages for personal use is the low-risk end of the spectrum, but if you’re doing this at scale or republishing outputs, read the web scraping legality overview first. robots.txt is respected here; the law is not settled anywhere.

FAQ

Is there a free website summarizer? Yes, three kinds: hosted free tiers (TLDR This, QuillBot) with daily or word caps, free-tier models through OpenRouter as in this guide, and Ollama models running locally at zero cost. The last has no caps at all beyond your hardware.

How do I summarize a website with ChatGPT? Paste the URL into the chat. ChatGPT fetches pages with its own browser, so paywalled or bot-walled pages often come back empty or partial. Pasting the text works more reliably, but it’s one page at a time and returns prose, not structured output you can script.

Can my browser summarize a page for me? Edge’s Copilot does, grounded in the current tab, and Chrome and Arc have similar built-ins. They’re one page at a time, bound to that browser, with no JSON, no length contract, and no way to batch.

Does magpie store my pages or my key anywhere remote? No. The key sits in your OS keyring or environment, pages cache in a local SQLite store, and LLM calls go directly from your machine to the provider you chose. Summaries and their token usage land in the same local store.

The ten-second version

Install the binary, magpie config set-key, then:

magpie summarize https://en.wikipedia.org/wiki/Web_scraping --max-sentences 3

N between 1 and 20, receipt in the usage block, failure means failure. Point it at one URL today; point the same command at a cron job tomorrow.