Automated Web Scraping: Schedule Scrapes That Run Themselves

A scraper you run by hand is a demo. The moment you need the same data next Tuesday (prices, job postings, a competitor’s changelog), manual checking stops scaling, and one missed check means acting on stale data. Automated web scraping fixes the trigger: a clock or a detected content change fires the scrape, and you hear about it when something actually happens.

Honest scope first. Magpie’s watches run on a schedule inside the desktop app, but scheduled scrapes and crawls need its CLI plus cron, systemd, or Task Scheduler. Both halves are below, with runnable commands and the failure modes that only show up after a week of unattended runs.

What automated web scraping actually means

Automated web scraping is extraction that starts on a trigger you didn’t fire: a clock tick or a content change. The fetching and parsing stay the same. What changes is who starts the run.

Trigger	Fires when	Typical job	What runs it
Manual	you type the command	one-off research	you
Scheduled	the clock says so	daily price dump, weekly crawl	cron, systemd timer, Task Scheduler, hosted scheduler
Change-driven	the page’s content hash moves	back-in-stock, release alerts	watch mode plus a webhook

Match the frequency to the change rate. A product page that reprices monthly, checked hourly, costs about 720 fetches for one useful run — that is 99.9% wasted traffic on someone else’s server, and it’s how scrapers earn rate limits. Change-driven triggers exist for exactly these pages: you poll on the tool’s interval, but you only act when the content hash moves.

Three ways to schedule web scraping

Where does the schedule live? Three places, with different blast radii when they break.

In the app. Magpie Desktop’s watch scheduler checks each watched page on an interval from 1 minute to 365 days. The next check time persists in the store, so quitting and reopening the app doesn’t reset the clock, and webhook endpoints fire when a page changes. Two limits, stated plainly: the app has to be running for checks to fire, and watches are page-level. There is no scheduled crawl in the app; scheduled crawls and scrapes go through the CLI.

In the OS. cron on Linux and macOS, Task Scheduler on Windows, systemd timers if you want logs and catch-up runs. The scheduler knows nothing about scraping; it runs your command and nothing else. This handles everything the app can’t: single pages, 200-page crawls, schema dumps into a dataset. It’s the workhorse, and it’s free, but your machine has to be awake at the fire time.

In someone else’s cloud. Apify schedules run Actors on cron expressions whether your laptop sleeps or not. Per their docs: runs can start as little as 10 seconds apart, one schedule drives up to 10 Actors or tasks, and you get an email when a scheduled run fails to start. Web Scraper Cloud’s scheduler offers daily, interval, and raw cron types, and if the previous job is still running when the next slot arrives, the new run waits instead of stacking. changedetection.io takes the pure-watch approach: cloud checks and diff notifications, or self-host it.

Pick by two questions. Does the data need to stay on your machine? Is your box reliably awake at fire time? Local CLI plus cron when both answers favor it, the app’s watch scheduler for change alerts on pages, hosted when the machine can’t be trusted to be on.

Before you start

• The Magpie CLI: one static binary, free to download, AGPL-3.0-or-later or a commercial licence. Everything below ran on v0.1.32.
• A target page you have permission to scrape, checked manually once.
• cron or a systemd timer (Linux/macOS) or Task Scheduler (Windows). flock on Linux/macOS to prevent overlapping runs.
• The desktop alternative if you never want to touch a terminal: watches in Magpie Desktop, 5 free runs per install, then $49/month. A watch counts as one run.

Step 1: Build the one-shot automated scraper

Automate nothing until one manual run returns the data you want. magpie scrape fetches the page, cleans it, and returns markdown plus structured data:

$ magpie scrape https://quotes.toscrape.com
{
  "url": "https://quotes.toscrape.com",
  "title": "Quotes to Scrape",
  "markdown": "\"The world as we have created it is a process of our
  thinking...\" by [(about)](https://quotes.toscrape.com/author/Albert-Einstein)
  ..."
}

(abridged; the full record carries every quote plus a structured_data field.)

Two failure modes to check before you schedule anything:

• Walled or blocked pages can return a stub that looks like success. The request worked; the record is junk. Read the title field. If it says “Bienvenue sur Reddit” instead of the page you asked for, the site is walling your IP, and a schedule will just automate the junk.
• Pages that need JavaScript need --render browser. The dynamic-page guide covers when to escalate.

Step 2: Test the change trigger before you schedule anything

Watch mode stores a markdown snapshot of the URL and diffs against it on every check. In --once mode it runs a single check and exits. That’s the cron mode, and the first call stores the baseline:

$ magpie watch https://quotes.toscrape.com --every 5m --once
$ magpie watch https://quotes.toscrape.com --every 5m --once
$ echo $?
0

Silence and exit 0 mean unchanged, the same convention as magpie diff. When content does change, the output is one line plus a word diff:

changed=true old=3f9c1a… new=8b20e7…
…diff of the changed words…

(abridged; old and new are full content hashes.) Add --webhook and the same event POSTs {url, changed, old_hash, new_hash, diff} to an endpoint. This is the cheapest automatic web scraper you can build: one binary, one line of cron, and it only speaks when something moved. The package release alerts guide is built on exactly this pattern.

Step 3: Put the scrape on a cron schedule

crontab -e, then one line:

0 6 * * * flock -n /tmp/quotes.lock /home/you/.local/bin/magpie scrape https://quotes.toscrape.com >> /home/you/scrapes/quotes.history.json 2>> /home/you/scrapes/quotes.err

What each part does:

• 0 6 * * *: daily at 06:00 server time.
• flock -n /tmp/quotes.lock: skip this run if the previous one still holds the lock. No pile-up when a run hangs.
• >> quotes.history.json: each run appends one JSON document. Over months you get a time series for free — read it with jq -s '.[length-1]', which parses the concatenated documents in one pass.
• 2>> quotes.err: stderr goes where you’ll actually look. Cron itself mails nothing useful by default; the err file is how you learn a run failed.

Three failure modes that cost me evenings:

• Use absolute paths everywhere, the binary and the redirects. cron’s PATH is minimal, and its shell is often dash, which doesn’t expand ~ at all.
• Give the full URL with the scheme. A seed like quotes.toscrape.com without https:// quietly produces nothing.
• An empty run still appends an empty-looking record. Schedule the err-file check along with the data check.

systemd timers beat cron on two points if you’re on a modern Linux box: logs land in the journal (journalctl -u your-scraper), and Persistent=true fires a missed run on next boot instead of skipping it. On Windows, Task Scheduler with “Run whether user is logged on or not” plus the same absolute-path rule works fine.

Step 4: Where web crawler automation earns its keep

A scheduled scrape is one page. A scheduled crawl is a site, and web crawler automation is a different weight class: more coverage, more politeness obligations, longer runtimes.

0 4 * * 1 flock -n /tmp/qcrawl.lock /home/you/.local/bin/magpie crawl https://quotes.toscrape.com --max-pages 25 --max-depth 1 --auto-throttle --corpus --out /home/you/scrapes/quotes-$(date +\%F).jsonl 2>> /home/you/scrapes/qcrawl.err

• --corpus writes one JSONL record per page with cleaned main-content markdown; no LLM calls, no API key.
• --auto-throttle backs off when the server pushes back (429s and 5xx). robots.txt is respected by default; leave it that way.
• The date +\%F stamp gives one snapshot file per run. Diff last Monday’s file against this one to see what the site changed. Note the escaped %: cron treats a bare % as a newline, and your command will silently truncate without the backslash.

For scale, I measured the crawl half of that line: 25 pages of quotes.toscrape.com in 36 seconds on a home line, pages_ok=25 pages_err=0 (the site has no sitemap.xml, so magpie said so in stderr and went seed-only). A 200-page crawl with depth 2 is minutes, not seconds — don’t schedule a crawl tighter than its own runtime. If the scrape/crawl split is fuzzy, the scraping vs crawling comparison settles it, and build a web crawler covers the from-scratch route.

Step 5: Automated data scraping into a dataset

Markdown is for reading. Scheduled runs usually want fields, and that’s what --schema is for. Hand it a JSON or YAML schema, and extraction returns records with exactly those fields:

$ magpie scrape https://quotes.toscrape.com --schema quote.schema.json >> quotes-dataset.json

On a schedule, each run appends a JSON document with exactly the fields your schema names; jq -s '.[].price' quotes-dataset.json reads them all back. Three rules keep the dataset sane:

• Dedup on the way in (URL plus content hash) or you’ll warehouse the same row 24 times a day.
• Watch the null rate. A schema that starts returning nulls is the first symptom of a site redesign, which is the next section.
• One quirk, verified against v0.1.32: scrape --format jsonl is accepted but still prints pretty JSON, so the append-plus-jq -s pattern above is the way, not a true JSONL file. The crawl command’s --corpus output is real single-line JSONL if that’s what your importer needs.

Step 6: Get told when something changes

A schedule gives you data. A webhook gives you signal. With --webhook, watch mode POSTs one JSON body per change to any endpoint that accepts a plain JSON POST. ntfy does, so a topic URL turns changes into phone notifications:

$ magpie watch https://quotes.toscrape.com --every 5m --once \
    --webhook https://ntfy.sh/my-scrape-alerts

The same trick alerts on failure. Cron runs watch instead of scrape when the job is really change detection, and a separate line pings you on nonzero exit:

0 8 * * * /home/you/.local/bin/magpie watch https://quotes.toscrape.com --every 1h --once || /home/you/bin/alert "quotes watch failed"

In the desktop app, watch webhooks are configured as endpoints in settings and delivered whenever a watched page changes; that’s the no-terminal version of this section. For the full monitoring playbook, the competitor website monitoring guide goes deeper.

Step 7: Make it stick

Unattended scrapers die in one of three ways. Plan for each:

• The site redesigns. Selectors break, records go empty. Magpie’s selector cache is self-healing: broken selectors get re-synthesized against the new markup, per domain and schema. That’s the difference between a hiccup and a dead pipeline. The self-healing scraper guide explains the mechanism. On any other tool, schedule a weekly record-count check that alerts on zero.
• The schedule outruns the politeness. One crawl at a time per site, auto-throttle on, robots.txt respected. The scraping without getting blocked guide covers the rest of the etiquette.
• Silence gets mistaken for health. Cron failing quietly for three weeks is the default outcome, not the exception. The err file, the record-count alert, and the nonzero-exit ping from Step 6 are the minimum viable monitoring.

Which automated web scraping tools fit which job

Approach	Runs where	Scheduling	Alerts	Cost
Magpie CLI + cron	your machine	any cron/systemd/Task Scheduler interval	webhook on change, DIY exit-code alerts	free (AGPL-3.0-or-later or commercial)
Magpie Desktop watches	your machine, app open	1 min–365 days per watch, survives restart	built-in webhook endpoints	5 free runs, then $49/month
Apify schedules	Apify cloud	cron expressions, ≥10 s between runs	email on failed start, webhooks	free plan carries a $5 monthly usage credit
Web Scraper Cloud scheduler	their cloud	daily, interval, or cron; overlapping runs queue	dashboard job history	$50/month (billed annually) for 5,000 URL credits
changedetection.io	self-hosted or their cloud	per-watch intervals	diff notifications by email/Slack/webhook	free self-hosted; hosted plans paid

Prices checked October 2026 against vendor pages. The honest read: if your machine is on when the schedule fires and the data shouldn’t leave it, the free CLI plus cron does everything the paid rows do, with more setup. Hosted earns its fee when the laptop sleeps, the targets need proxy rotation, or nobody owns a server.

FAQ

What is an automatic web scraper?

A scraper that runs without a person triggering it, on a clock (cron, hosted scheduler) or on change (watch mode with a content hash). The extraction itself is identical to a manual run; only the trigger differs. The simplest working example is magpie watch <url> --every 5m --once on a cron line.

How do I set up automated data scraping from websites without writing code?

Page-level change alerts: a desktop app’s watch feature or a hosted recorder does it in a few clicks. Whole-site scheduled crawls with structured output still need either a local tool’s CLI (Magpie, Scrapy) or a hosted platform; no-code recorders generally stop at single pages or fixed templates.

What’s the best automated web scraping solution?

Decide on two constraints, not features: data locality (does it matter if pages leave your machine?) and whether a machine of yours is awake at fire time. Local-first and always-on → CLI plus cron. Never-on machine → hosted scheduler. Change alerts on a handful of pages → watch mode anywhere.

Are paid automated scraping tools worth it?

Do the run math. Magpie Desktop is $49/month flat after 5 free runs; Web Scraper Cloud is $50/month for 5,000 page loads; Apify’s $19 Starter is prepaid credit that expires. At 200 pages a day, the flat desktop licence wins. At 50,000 pages needing proxies, the cloud rows win.

Is scheduled scraping legal?

Reading publicly accessible pages is generally lawful, with the usual constraints: respect robots.txt, rate-limit politely, never scrape personal data without a basis, and check the site’s terms. Schedules make all of this matter more because violations repeat automatically. The legality guide has the detail. None of it is legal advice.

---

One page annoys you more than the rest. Tonight: run watch --once on it twice, add the cron line, point the webhook at your phone. Ten minutes of setup, and the checking stops being your job.