The Magpie guides, page 4

Step-by-step guides to extracting text, finding pages and pulling web data into the tool you already use — every command in them was run against a live site first, and each one ends where its method breaks and what to switch to.

Magpie title card: How to Scrape a Website to CSV or JSON

How to Scrape a Website to CSV or JSON

Scrape a website to CSV or JSON with local commands - one-page scrapes, whole-site crawls, typed records, and the jq one-liners that finish the job.

Motherlode Lab · · 8 min read

Magpie title card: How to Scrape a Website That Requires Login, over a JSON product record fetched with an authenticated session

How to Scrape a Website That Requires Login

Scrape website with login pages intact: session cookies, saved profiles, scripted forms, MFA-safe capture, and CDP. Live-tested numbers inside.

Motherlode Lab · · 9 min read

Magpie title card: How to Scrape Websites Without Getting Blocked, over the faded HTML source of a product listing

How to Scrape Websites Without Getting Blocked (Politely)

Scrape without getting blocked politely: robots.txt, rate math, Retry-After, conditional requests, honest headers. Measured numbers, nothing bypassed.

Motherlode Lab · · 10 min read

Magpie title card: Self-Healing Web Scrapers — Selectors That Fix Themselves, over a broken CSS selector repairing itself

Self-Healing Web Scrapers: Selectors That Fix Themselves

Build a self-healing web scraper that catches broken selectors and repairs them. Real run: CSS learned from three pages, cached extractions at $0.

Motherlode Lab · · 10 min read

Magpie title card: Web Scraping Pagination: How to Scrape Multiple Pages

Web Scraping Pagination: How to Scrape Multiple Pages

Web scraping pagination without the traps: wrong page params that silently repeat page 1, 200s past the end, loops that never stop. Measured live.

Motherlode Lab · · 8 min read

Magpie title card: Web Scraping Tutorial: Step-by-Step Guides

Web Scraping Tutorial: Step-by-Step Guides

A web scraping tutorial that runs each step live: robots.txt, a 20-line first scraper, hidden JavaScript data, and 60 guides for what's next.

Motherlode Lab · · 14 min read

Magpie title card: Website Summarizer: Summarize Any URL in N Sentences

Website Summarizer: Summarize Any URL in N Sentences

Run a website summarizer from your terminal. magpie summarize turns any URL into at most N sentences with your own LLM key. Real output, real limits.

Motherlode Lab · · 9 min read

Magpie title card: How to Scrape a Website With ChatGPT, over a chat window turning pasted HTML into a table

How to Scrape a Website With ChatGPT (and Its Limits)

How to scrape a website with ChatGPT, three ways that actually work: paste the HTML, generate a script, wire a custom GPT. Plus the real limits.

Motherlode Lab · · 10 min read

Magpie title card: How to Find All Pages on a Website, over a sitemap tree spreading across a site's pages

How to Find All Pages on a Website (6 Methods That Work)

Find all pages on a website with sitemaps, crawls, and search operators. Six methods compared, with runnable commands and a way to trust the result.

Motherlode Lab · · 7 min read