Competitor Website Monitoring: Track Changes Automatically

Your competitor changed their pricing at 9am on a Tuesday. You found out eleven days later, from a lost deal. Competitor website monitoring closes that gap: a script watches their key pages, compares every visit to the last one, and tells you the moment something moves.

This guide builds that pipeline with Magpie’s free CLI: page discovery, baselines, watch mode, webhooks, scheduling. Every command ran against real sites in October 2026. If you’d rather not touch a terminal, the tool landscape section covers what Visualping, changedetection.io, and Distill.io cost and where they fall short.

What competitor website monitoring should watch

Whole-site alerts are noise. A competitor’s site changes every day in ways that mean nothing: new testimonial, A/B test, blog post, copyright year. The pages worth a monitor are the ones that move only when strategy moves.

Page	What a change means	Cadence
Pricing page	New tier, price move, packaging shift, new “contact sales” gate	Daily
Changelog / product updates	Feature launches, roadmap signals	Weekly
Careers / jobs page	Hiring is where the budget goes	Weekly
Homepage hero	Repositioning, new target customer	Weekly
Docs / release notes	API surface changes, deprecations	Weekly
Integrations directory	New partnerships, ecosystem plays	Monthly

Six monitors per competitor is fine. The cadence carries the load: pricing daily, everything else weekly, and you catch every move that matters without drowning.

Two ways to catch a change (and the noise problem)

Dedicated change detectors diff the rendered page (screenshots or HTML hashes). Visualping, changedetection.io, and Distill.io all work this way. The failure mode is false alarms: urgency banners that rotate, CMS timestamps, A/B tests that shuffle a headline every third visit. I pointed a whole-page watch at Hacker News’ /newest feed and every check after the first came back changed, because new posts renumber every row. Busy pages produce a diff every time, which trains you to ignore alerts.

The other way is field monitoring: extract the fields you care about (plan names, prices, feature bullets) and watch those. The alert fires when the number moves, not when a banner rotates. That’s the approach in Step 7, and it’s the difference between an alert inbox and intelligence.

The tool landscape, with real numbers:

• Visualping: 150 free checks/month across 5 pages, minimum 60-minute interval. Personal 1K runs $14/month (or $120/year). Per-page pricing punishes the six-pages-per-competitor pattern.
• changedetection.io: open source, ~31k GitHub stars, self-hosted free; hosted version $8.99/month. Best value if you’ll run Docker.
• Distill.io: free tier covers 25 pages with generous check intervals; paid tiers add pages and faster checks.
• Magpie CLI: free, AGPL-3.0-or-later OR commercial license, runs on your machine. Webhook POSTs, markdown snapshots, login profiles, JS rendering. Cons, stated plainly: no hosted dashboard, no browser extension, you own the cron.

For a deeper cut on the self-hosted option, see the changedetection.io alternatives breakdown. If your competitor list is long, per-page pricing math matters; that’s where a local runner wins.

Step 1: List the pages worth watching

Guessing URLs is how you miss the /pricing/new page they shipped last quarter. Ask the site instead:

magpie map https://basecamp.com

map prints every URL the site declares in its robots.txt and sitemaps, one per line (add --format json if you’re piping into a script). It’s zero-LLM, so it’s free and instant. From the output, pick the six page types from the table above.

Failure case: some sites declare no sitemap at all. Then crawl the origin shallowly — magpie crawl https://basecamp.com --max-pages 25 --max-depth 1 --corpus — and pull candidate URLs from the records. The guide to finding all pages on a website has the full walkthrough.

Step 2: Take a baseline snapshot

A watch compares each visit to the previous one, so store what “normal” looks like first:

magpie scrape https://basecamp.com/pricing --page-format markdown --out baseline/basecamp-pricing.md

What you should see: --out writes the full JSON record, and the cleaned page sits in its markdown field. On Basecamp’s pricing page that was 1,871 words of plans and prices, with navigation, cookie banners, and footers stripped. The record also carries a structured_data sidecar: the JSON-LD the page embeds. Basecamp’s includes an ItemList of navigation and a WebPage block with their own description of the page. That sidecar is free structured data most people never look at, and it’s the fastest way to see whether a site already hands you machine-readable prices.

Failure case: pages that fail the quality gate (challenge stubs, near-empty shells) return a typed error instead of bytes. If you just need proof of what the page showed, --page-format screenshot writes a PNG and skips that gate.

Step 3: Put the page under watch

watch is the core of the pipeline. It stores a markdown snapshot, then reports a word-diff whenever the content changes:

magpie watch https://news.ycombinator.com/newest --every 1m --once

The first run is silent: it stores the baseline. Run it again after a minute and, on a page that moved, you get:

changed=true old=4161152d6041c996... new=7560f3996423ba61...
+ [A team of Gemini 4 Argon agents beats Minecraft](https://arxiv.org/abs/...)
- 2.
+ 3.
- 3.
+ 4.

That output is real and abridged: hashes identify the exact snapshots, the +/- lines are a word diff of what changed. --every takes values like 30s, 5m, 1h (30 seconds is the floor); drop --once and the loop runs until Ctrl-C. It’s zero-LLM, so a 24/7 watch costs nothing but electricity.

Two behaviors to know before you trust it. Silence on no-change is deliberate: no news is no news, and your cron logs prove the run happened. And note the renumbering noise in that diff: one new post shifted every list index. On a competitor’s quiet pricing page you won’t see that; on a busy feed you will. Watch quiet pages.

Step 4: Get changes pushed to you

A diff you never see is a diff that didn’t happen. Add --webhook and every detected change arrives as one POST:

magpie watch https://basecamp.com/pricing --every 1h --once \
  --webhook https://hooks.example.com/your-endpoint

The payload is {url, changed, old_hash, new_hash, diff}, enough for a Slack or Discord relay to post the URL and the diff text straight into a channel. The guide to sending scraped data to a webhook covers wiring it into n8n or a tiny receiver. An hourly check on a static pricing page produces about one webhook a quarter. The signal rate is the point.

Step 5: Schedule it

--once exists precisely for schedulers. One crontab line runs all your competitor watches on their own cadences:

0 6 * * *  magpie watch https://basecamp.com/pricing --every 1h --once
0 6 * * 1  magpie watch https://basecamp.com/blog --every 1h --once

The first line checks pricing daily at 6am; the second checks the blog weekly. Failure case: % starts a newline inside crontab and silently breaks any command that contains it; percent-encode URLs or keep them out of cron entirely. The deeper version of all this, including systemd timers and Windows Task Scheduler, lives in the automated web scraping guide.

If you’d rather not own cron: Magpie Desktop has a watch scheduler (intervals from 1 minute to 365 days) that persists schedules across restarts, but the app has to stay running. Licensing differs between the two: the CLI is free with no run counter, while Desktop gives you 5 free runs per install, then $49/month, and a watch counts as one run. Pick the CLI for a fleet of monitors; pick Desktop when you want the schedules managed for you.

Step 6: Monitor pages that need a login or JavaScript

Competitors hide real prices behind logins (usage-based tools that show pricing only to signed-in users) or render them client-side. watch takes no credentials, so the login case runs through scrape plus a file diff:

# save the session once
magpie profile save competitor --cookies "session=..."

# baseline, then one per scheduled check
magpie scrape https://app.competitor.com/billing --as competitor --out baseline.json
magpie scrape https://app.competitor.com/billing --as competitor --out check.json

diff <(jq -r .markdown baseline.json) <(jq -r .markdown check.json)

The markdown field holds the cleaned page text, so the diff reads like a word diff. After you triage an alert, roll the baseline forward: mv check.json baseline.json.

Pages that need JavaScript get it automatically: Magpie escalates to a real browser when the static fetch comes back empty, and --render browser forces it. Click-to-reveal pricing tabs are an actions job, covered in the scraping with login and dynamic websites guides.

One limit, plainly: Magpie respects robots.txt, throttles itself, and does not defeat CAPTCHAs or bot protection. If a competitor’s pricing page sits behind a hard challenge, your monitor will report the challenge, not the price. That’s a feature working as designed. Read the web scraping legality guide before you get creative around it.

Step 7: Upgrade to competitor page monitoring

A word diff tells you something changed. A field-level comparison tells you what. Two ways to get there, both built in:

Zero-LLM verticals. For competitor stores, product pages already ship structured data: magpie scrape <url> auto-fires verticals like shopify_product or amazon_product and returns clean JSON with title, price, and availability. Monitor that JSON’s price field and you get alerts that read “Team plan: $79 → $99”, not forty lines of reflowed HTML. The price scraper guide goes deep on extraction; the Amazon price tracker guide covers the big-store case.

Schema extraction. For pricing pages with no structured data, hand Magpie a JSON Schema and it extracts exactly those fields through any LLM you configure (BYOK, cost-capped; local Ollama works too):

type: object
properties:
  plans:
    type: array
    items:
      type: object
      properties:
        name: { type: string }
        price: { type: string }

Run that against each check’s snapshot and diff the plan names and prices across runs. Four fields instead of 1,871 words — that’s competitor page monitoring you can act on.

Make it stick: monitor competitor website changes week after week

Three failure modes show up once the pipeline has run for a while, and all three have the same fix: watch quieter pages.

1. The rotating hero. Homepage carousels and testimonial rotators guarantee a diff on every visit. Watch the changelog and pricing page instead; visit the homepage yourself once a week.
2. The redesign. A competitor relaunch floods your diff with 1,500 changed words. It’s still a signal (repositioning usually starts with the website), so triage it manually. Because watch compares content hashes of cleaned markdown, a redesign can’t break the monitor the way it breaks CSS selectors.
3. The geo variant. Some sites show different prices by region. If your competitor does, run the watch through --proxy-file from another region and you’ll see what their other market sees. For retail price floors specifically, the MAP monitoring guide is the specialized version of this whole pipeline.

The same pipeline pattern extends past competitors: the terms-of-service change tracker watches legal pages, and the GitHub releases guide watches repos; if your competitor is open source, their release feed is the cheapest roadmap intelligence there is.

FAQ

Is competitor website monitoring legal?

Monitoring publicly accessible pages (pricing, changelogs, careers) is normal practice, and respecting robots.txt plus rate limits keeps you polite. Magpie does both by default. The full picture, including where logins and terms of service change the analysis, is in the web scraping legality guide.

How often should you track competitor website pages?

Daily for pricing, weekly for changelog, careers, and docs, monthly for integrations. Faster is wasted: most hosted tools floor free plans at 60-minute intervals anyway, and pricing pages rarely move more than once a day.

What’s the best tool for monitoring competitor websites?

Non-technical teams: Visualping ($14/month once you outgrow 5 pages). Self-hosters: changedetection.io (free in Docker, $8.99/month hosted). Teams who want diffs, structured extraction, and webhooks on their own machine: Magpie’s CLI, free.

Can it monitor pages behind a login?

Yes. Save the session once with magpie profile save, then let scheduled scrape --as runs fetch the page and diff it against a baseline, exactly as in Step 6 (watch itself takes no credentials). You’re using your own account, so the competitor’s terms still apply.

Does it work on sites that need JavaScript?

Yes: fetches escalate to a real browser automatically when the static response comes back empty, and click-to-reveal pricing works with scripted actions before extraction.

Start with three pages

Their pricing page, their changelog, their careers listing. Three watches, one crontab entry, one webhook into your team channel — under an hour of setup, and it runs on your machine forever at zero marginal cost. The first real diff will land in your channel within a day, and you’ll never again learn about a competitor’s price change from a lost deal.