Automated Web Scraping: Schedule Scrapes That Run Themselves
Automated web scraping with cron, watch mode, and webhooks. Runnable commands, real failure modes, and when a hosted scheduler beats your own machine.
Motherlode Lab · · 12 min read
Scraping how-tos, tested as written.
Step-by-step guides to extracting text, finding pages and pulling web data into the tool you already use — every command in them was run against a live site first, and each one ends where its method breaks and what to switch to.
Get Back-in-Stock Alerts for Any Product
Set a back in stock alert on any product page in 10 minutes: watch mode, phone push, a cron recipe that filters for stock, and the honest limits.
Motherlode Lab · · 8 min read
BeautifulSoup Web Scraping Tutorial (2026)
BeautifulSoup web scraping tutorial with every code block tested live: requests + find_all, CSS selectors, pagination, CSV output, and where BS4 fails.
Motherlode Lab · · 9 min read
How to Build a Web Crawler (Architecture + Code)
How to build a web crawler that survives real sites: the frontier, dedupe, politeness, robots handling, plus 110 lines of tested Python code.
Motherlode Lab · · 15 min read
Web Scraping With Claude: MCP Setup in 5 Minutes
Web scraping with Claude in three commands: build Magpie, run init, restart. Thirteen live-tested MCP tools, zero tokens on the markdown path.
Motherlode Lab · · 10 min read
Competitor Website Monitoring: Track Changes Automatically
Competitor website monitoring that fires on real changes, not junk. Pick pages, baseline them, watch plus webhooks, with runnable commands and real output.
Motherlode Lab · · 10 min read
Web Scraping in Cursor With a Local MCP Server
Cursor web scraping with a local MCP server: wire 13 scraping tools into Cursor with one command, keep data on your machine, pay no per-page credits.
Motherlode Lab · · 10 min read
E-commerce Product Scraper: Extract Catalogs, Prices & Specs
Product data scraping with real output - typed records, whole-catalog JSON endpoints, spec extraction, and CSV delivery, every command run on live stores.
Motherlode Lab · · 10 min read
Event Scraper: Collect Events From Eventbrite, Meetup & Luma
An event scraper workflow that pulls typed event records from Eventbrite, Meetup and Luma, plus a fallback ladder for sites that publish no event data.
Motherlode Lab · · 9 min read
How to Extract a Color Palette From Any Website
Extract color palette from website HTML, rendered CSS, or one JSON command. DevTools, grep, and magpie brand, tested on real sites, failure modes included.
Motherlode Lab · · 10 min read
How to Extract Clean Text and Main Content From Any Website
Extract text from website pages without the nav, ads, and scripts. Five methods compared: online extractors, BeautifulSoup, trafilatura, browsers.
Motherlode Lab · · 10 min read
Go (Golang) Web Scraping Tutorial
A Golang web scraping tutorial with runnable code: net/http, goquery, Colly crawling, rod for JavaScript pages, and real timings from every step.
Motherlode Lab · · 8 min read
Google Sheets Web Scraping: IMPORTXML and Beyond
Google Sheets web scraping with IMPORTXML, IMPORTHTML, and Apps Script — verified limits, real failure modes, and when to switch to a dedicated scraper.
Motherlode Lab · · 9 min read
HTML Scraper: Get the Source and Data From Any Web Page
Get any page's HTML three ways - view-source, curl, and a command-line html scraper that returns the server source and the rendered DOM. Measured runs.
Motherlode Lab · · 9 min read
How to Import Data From a Website Into Excel
Import data from a website to Excel with Power Query, JSON endpoints, or a scraper bridge. Tested steps, real failure modes, and a validation checklist.
Motherlode Lab · · 10 min read
Job Scraper: Collect Postings From Any Job Board
Build a job scraper that returns typed posting records from Greenhouse, Lever and Ashby boards, plus a fallback ladder for boards that publish nothing.
Motherlode Lab · · 9 min read
Web Scraping With LangChain Loaders
LangChain web scraping tested: what WebBaseLoader really returns on a live page, and two measured ways to wire in a local scraper via MCP or the CLI.
Motherlode Lab · · 10 min read
News & Article Scraper: Collect Articles From Any News Site
Build a news scraper that returns typed article records, full text and webhook alerts from any news site. Every command live-tested on 2026-10-16.
Motherlode Lab · · 8 min read
Get Price Drop Alerts on Any Website
Set a price drop alert on any website: where trackers stop, a page-watch pipeline for the stores they skip, and the threshold you add yourself.
Motherlode Lab · · 10 min read
Price Scraper: Extract Competitor Prices From Any Store
Build a price scraper that pulls competitor prices from any store: JSON-LD extraction, Amazon currency traps, an LLM fallback, and a loop that ships CSV.
Motherlode Lab · · 11 min read
Proxies for Web Scraping: When You Actually Need Them
Every guide to proxies for web scraping is written by a proxy vendor. Measured advice: when you need them, when politeness wins, what rotation fixes.
Motherlode Lab · · 10 min read
Real Estate Scraper: Collect Listings From Any Property Site
A real estate scraper tested against five portals: Zillow listings into CSV in two steps, which sites block you, and legit data routes for the rest.
Motherlode Lab · · 10 min read
How to Scrape Data From Any Website (No-Code and Code)
Scrape data from website pages with no-code tools, spreadsheet functions, or ten lines of Python. Six methods tested live today, with real timings.
Motherlode Lab · · 12 min read
Scrape a Documentation Site Into Clean Markdown
A documentation scraper turns a docs site into clean per-page markdown: probe one page, crawl a scoped section, filter the junk, validate what you got.
Motherlode Lab · · 10 min read
How to Scrape Dynamic Websites (JavaScript-Heavy Pages)
Get 200 but no content? Four tested routes to scrape dynamic websites: parse embedded JSON, call the hidden API, render in a browser, scroll actions.
Motherlode Lab · · 11 min read
How to Scrape Email Addresses and Contacts From a Website
Scrape email from website data without the junk: JSON-LD contacts first, crawl plus regex second, LLM schemas for the rest, plus CAN-SPAM and GDPR rules.
Motherlode Lab · · 10 min read
How to Scrape a Table From Any Website
Scrape a table from any website into Excel, Sheets, or CSV. Four routes tested today, with the pandas 3.0 fix and what breaks on JavaScript tables.
Motherlode Lab · · 10 min read
How to Scrape a Website to CSV or JSON
Scrape a website to CSV or JSON with local commands - one-page scrapes, whole-site crawls, typed records, and the jq one-liners that finish the job.
Motherlode Lab · · 8 min read
How to Scrape a Website That Requires Login
Scrape website with login pages intact: session cookies, saved profiles, scripted forms, MFA-safe capture, and CDP. Live-tested numbers inside.
Motherlode Lab · · 9 min read
How to Scrape Websites Without Getting Blocked (Politely)
Scrape without getting blocked politely: robots.txt, rate math, Retry-After, conditional requests, honest headers. Measured numbers, nothing bypassed.
Motherlode Lab · · 10 min read
Self-Healing Web Scrapers: Selectors That Fix Themselves
Build a self-healing web scraper that catches broken selectors and repairs them. Real run: CSS learned from three pages, cached extractions at $0.
Motherlode Lab · · 10 min read
Web Scraping Pagination: How to Scrape Multiple Pages
Web scraping pagination without the traps: wrong page params that silently repeat page 1, 200s past the end, loops that never stop. Measured live.
Motherlode Lab · · 8 min read
Web Scraping Tutorial: Step-by-Step Guides
A web scraping tutorial that runs each step live: robots.txt, a 20-line first scraper, hidden JavaScript data, and 60 guides for what's next.
Motherlode Lab · · 14 min read
Website Summarizer: Summarize Any URL in N Sentences
Run a website summarizer from your terminal. magpie summarize turns any URL into at most N sentences with your own LLM key. Real output, real limits.
Motherlode Lab · · 9 min read
How to Scrape a Website With ChatGPT (and Its Limits)
How to scrape a website with ChatGPT, three ways that actually work: paste the HTML, generate a script, wire a custom GPT. Plus the real limits.
Motherlode Lab · · 10 min read
How to Find All Pages on a Website (6 Methods That Work)
Find all pages on a website with sitemaps, crawls, and search operators. Six methods compared, with runnable commands and a way to trust the result.
Motherlode Lab · · 7 min read