Price Scraper: Extract Competitor Prices From Any Store
Build a price scraper that pulls competitor prices from any store: JSON-LD extraction, Amazon currency traps, an LLM fallback, and a loop that ships CSV.
On this page
Stores publish every price twice: once as schema.org JSON-LD for machines, once in the DOM for humans. Most hosted scrapers charge you per page to read what the store already says about itself. This guide builds a price scraper that reads the structured copy first, pays a fraction of a cent for LLM extraction only where the markup fails, and ships a CSV — every command below ran this morning against four live sites, and every output is what my terminal printed.
Before you start
- The
magpieCLI. One Go binary, no runtime dependencies, AGPL-3.0-or-later or a commercial licence. Grab a build from the repo releases. jqandcurlfor URL discovery and the CSV step.- An LLM API key for Step 4 only. Steps 1 through 3 call no model and cost nothing.
- One legal sentence: prices on public product pages are public data,
magpierespects robots.txt and throttles per host, but a store’s terms can still forbid scraping. That call is yours; I’m not a lawyer.
Step 1: Scrape prices from website structured data
Nearly every serious store ships a schema.org/Product with an Offer on the page. The ecommerce_product vertical reads that markup and returns a typed record. No LLM, no selectors, nothing to break when the theme changes:
magpie scrape "https://www.allbirds.com/products/mens-wool-runners" \
--vertical ecommerce_product{
"url": "https://www.allbirds.com/products/mens-wool-runners",
"title": "Allbirds Wool Runners, Men's | Reviews, SIzing Info | Casual Walking, Running Shoes",
"vertical": "ecommerce_product",
"record": {
"availability": "https://schema.org/InStock",
"brand": "Allbirds",
"currency": "USD",
"name": "Men's Wool Runner",
"price": 110,
"url": "https://www.allbirds.com/products/mens-wool-runners"
}
}Price 110, currency USD, availability in stock. That record came from the store’s own markup, so it costs zero API credits and zero LLM tokens.
What you should see on stores without usable markup is the failure mode, and it’s honest. I pointed the same vertical at a WooCommerce cup store and got back "price": null, "currency": "". The page has JSON-LD, but its Offer carries no price the parser can read. On a page with no product markup at all, the vertical refuses with a typed error instead of returning zeros. Either way you find out immediately, which is exactly what you want from a price scraper. The fix for both cases is Step 4.
Step 2: Store endpoints are faster, but watch the currency
Shopify stores expose a JSON product endpoint, and Magpie has a vertical for it:
magpie scrape "https://www.allbirds.com/products/mens-wool-runners" \
--vertical shopify_product{
"vertical": "shopify_product",
"record": {
"images": [],
"price": 110,
"title": "Men's Wool Runner - Natural Grey (Light Grey Sole)",
"type": "Shoes",
"url": "https://www.allbirds.com/products/mens-wool-runners",
"vendor": "Allbirds"
}
}Same 110, no currency field. The .js endpoint doesn’t carry currency; the store renders it client-side. If your competitors all sell in one currency that’s a non-issue. If they don’t, prefer ecommerce_product (its JSON-LD path returned "currency": "USD" on the same product) or pin the currency yourself per store.
A cheap cross-check that catches silent breakage: compare the vertical against the store’s own catalog JSON.
curl -s "https://www.allbirds.com/products.json?limit=4" \
| jq -r '.products[] | "\(.handle)|\(.title)|\(.variants[0].price)"'womens-allbirds-flip-flop-dusty-pink|Women's Allbirds Flip Flop - Dusty Pink|25.00
mens-canvas-runner-nz|Men's Canvas Runner NZ - Deep Navy Stripes (Blizzard Sole)|100.00
womens-canvas-runner-nz|Women's Canvas Runner NZ - Deep Navy Stripes (Blizzard Sole)|100.00
mens-cruiser-terry|Men's Cruiser Terry - Warm White (Warm White Sole)|115.00Same numbers the vertical returns, straight from the source. If those two ever disagree, the store changed something and your pipeline needs a look. (/products.json is the whole-catalog route; the e-commerce product scraper guide goes deep on catalog extraction. This page stays on prices.)
The WooCommerce vertical behaved differently on the one WooCommerce store I tested: it returned SKU, title, and images but no price field, because that store’s product API omits it. A vertical that can’t find a price leaves the field out rather than inventing one. That’s the right behavior, and it means your loop needs a Step 4 fallback for those stores.
Step 3: Amazon amounts are geo-priced and currency-less
The amazon_product vertical works against product pages, but understand what Amazon is pricing:
magpie scrape "https://www.amazon.com/dp/0135957052" \
--vertical amazon_product{
"vertical": "amazon_product",
"record": {
"asin": "0135957052",
"availability": "In Stock",
"list_price": { "amount": 1837.66 },
"price": { "amount": 1536.23 },
"rating": 4.8,
"reviews": 3661,
"title": "The Pragmatic Programmer: Your Journey To Mastery, 20th Anniversary Edition (2nd Edition)"
}
}1,536 for a paperback? My connection lands on Amazon’s Thai storefront, so that’s ฿1,536, about $47 and right in the price class for that book. The record drops the currency, so nothing in the output tells you that. I re-ran it with a Chrome TLS fingerprint and header profile; identical numbers, because fingerprinting changes how you ask, not where you’re asking from.
Two rules keep Amazon data sane. Treat amount as unitless until you’ve pinned the marketplace (use amazon.co.uk/dp/... or amazon.de/dp/... to force a market). And sanity-check every amount against the product class before it reaches a spreadsheet. The full Amazon walkthrough covers session gating, delivery-zip pricing, and the pages that flip with a browser render.
Step 4: Pay a third of a cent where markup fails
When the vertical returns null, hand the page to an LLM with a schema. Write the schema once:
{
"type": "object",
"properties": {
"name": { "type": "string", "description": "Product name" },
"price": { "type": "number", "description": "Current selling price, number only" },
"currency": { "type": "string", "description": "ISO 4217 currency code" }
},
"required": ["name", "price", "currency"]
}Then run it against the store where Step 1 found no usable markup:
magpie scrape "https://jococups.com/product/active-flask-17oz" \
--schema price-schema.json --provider openrouter{
"extracted": {
"currency": "AUD",
"name": "Active Flask 17oz",
"price": 59.95
},
"usage": {
"provider": "openrouter",
"model": "gpt-4o-mini",
"prompt_tokens": 1887,
"completion_tokens": 19,
"usd_estimate": 0.00029445
}
}59.95 AUD, correctly labeled, for $0.0003. At that rate, re-pricing 1,000 SKUs costs about 30 cents. The key stays yours (env var or OS keyring), and --max-cost 0.50 caps a runaway run in dollars.
Use this path only where the markup path fails. The JSON-LD route is free, deterministic, and costs zero tokens; spending an LLM call on a page that already has structured data is waste. A sensible pipeline tries Step 1, falls back to Step 4 on price: null, and logs which stores needed which.
Step 5: Run the price scraper loop: a category to a CSV in one pass
One product proves the pipeline. A category pays for it. Grab product URLs from discovery, scrape each with the vertical, and emit CSV:
for h in $(curl -s "https://www.allbirds.com/products.json?limit=4" | jq -r '.products[].handle'); do
magpie scrape "https://www.allbirds.com/products/$h" --vertical shopify_product
sleep 2
done | jq -s -r '(["title","price"], (.[] | [.record.title, (.record.price|tostring)])) | @csv'"title","price"
"Women's Allbirds Flip Flop - Dusty Pink","25"
"Men's Canvas Runner NZ - Deep Navy Stripes (Blizzard Sole)","100"
"Women's Canvas Runner NZ - Deep Navy Stripes (Blizzard Sole)","100"
"Men's Cruiser Terry - Warm White (Warm White Sole)","115"The sleep 2 matters. Magpie already respects robots.txt and rate-limits per host; a tight loop on top of that is how you earn a block. Two seconds between requests pulled four prices with zero failures, and the CSV matched the store’s own catalog JSON from Step 2 exactly.
One honest gap: Magpie’s batch command (100 URLs, bounded concurrency) outputs markdown only and takes no --vertical flag, so the bash loop is the current way to get typed price records in bulk. For a few hundred SKUs the loop with a sleep is fine; for thousands, split the list and run segments. If you need fancier delivery than CSV (JSON, Excel imports), the scrape-to-CSV-and-JSON guide covers the formats.
Step 6: Verify every record, because error pages look like products
This is the step that separates a price scraper that works from one that quietly lies. eBay is the cautionary tale. The ebay_item vertical exists, so I tried it today. A dead item ID failed cleanly:
Error: vertical: ebay_item: no item data at https://www.ebay.com/itm/286663680524A live ID returned something worse, a record that looked real:
{
"url": "https://www.ebay.com/itm/186601073855",
"title": "Error Page | eBay",
"vertical": "ebay_item",
"record": {
"availability": "OutOfStock",
"condition": "RefurbishedCondition",
"item_id": "186601073855",
"price": { "amount": 12200.22, "currency": "THB" },
"title": "Apple iPhone 15 A2846 US Cellular Only 128GB Green Very Good"
}
}eBay served my IP an error page in Thai, and the vertical parsed leftover markup into a plausible-looking record: ฿12,200 for a used iPhone. The tells were there (the page title says “Error Page”, the amount is absurd for the product class), but nothing in the pipeline flagged it automatically.
So make verification mechanical. Every price row passes three checks before it ships: the page title and record title agree, the amount is in the expected class for that product, and the currency is known. On marketplaces, expect walls: eBay and Etsy both resisted extraction from a home connection in my testing, and Magpie doesn’t defeat bot protections by design; it detects challenges and reports them. If marketplace coverage is the job, budget for a hosted extraction service or the marketplace’s official API instead of fighting it.
Price scraping tools compared
A price scraper is a means; the market sells you three different means plus this guide’s local route.
What web price scraping software charges
| Approach | Examples | Cost (Oct 2026) | Where it breaks |
|---|---|---|---|
| Hosted scraping APIs | ScrapingBee, Oxylabs Web Scraper API | From $49/mo, credit-based; JS rendering costs ~5 credits/page, protected sites 10–75 | Credits burn on retries and bot walls; you pay per page for markup stores give away |
| Monitoring SaaS | Price2Spy (~$27/mo), Prisync ($99/mo), Pricefy (free ≤50 SKUs, $49/mo at 100) | Subscription per SKU count | They own the extraction; SKU caps and per-market limits decide your budget |
| Visual tools and extensions | Web Scraper (free extension, cloud from $50/mo), Octoparse (free plan, Standard ~$69/mo annual) | Freemium | Point-and-click setup per site; pagination and login flows get tedious fast |
| Local CLI (this guide) | Magpie | Core free (AGPL-3.0-or-later or commercial); Desktop app 5 free runs then $49/mo; failed runs don’t count | Your IP hits their walls; no CAPTCHA bypass, by design |
Vendor pricing from their pages and current roundups; credit multipliers change often, so recheck before committing.
When price scraping services earn their cost
Hosted price scraping services justify their bill on the sites that genuinely need proxy rotation and CAPTCHA handling: big marketplaces, mostly. For the long tail of Shopify and WooCommerce stores that publish clean JSON-LD, paying per page for it is hard to justify when a local vertical reads the same record in about two seconds. That’s the trade the free route makes concrete: you swap their infrastructure for your own IP and patience.
Extraction is not monitoring
Once rows land in your CSV, the next want is alerts and history. Different job, different pages. Price-drop alerts with webhooks live in the price drop alerts guide, broader competitor tracking in competitor website monitoring, and Amazon-specific watch setups in the Amazon price tracker guide. If you police reseller pricing for a brand, MAP monitoring is the compliance angle. This page stops at getting the number out of the page and into a row.
FAQ
What are the best free price scraping tools?
A local CLI reading JSON-LD (this guide, free at any volume), the store’s own JSON endpoints via curl, and a browser extension like Web Scraper for one-off manual grabs. Hosted “free trials” just defer per-page costs. Structured-data extraction covers most stores without spending anything.
Can I run a free price scraper against Amazon?
The amazon_product vertical is free and needs no LLM, but amounts come back currency-less and priced to your IP’s marketplace. Workable for relative price tracking on one pinned marketplace. Verify title and amount class on every record, and read the caveats in Step 3 first.
Is scraping prices from a website legal?
Public prices are generally collectible in many jurisdictions; robots.txt and the site’s terms still bind you, and login-gated or personal data changes the analysis completely. Jurisdictions differ. This is orientation, not legal advice. Ask a lawyer before anything you’d defend in a meeting.
Why does my scraped price show the wrong number?
Five usual suspects: geo-pricing to your IP, a dropped currency making the value unreadable, sale versus list price, a variant you didn’t target, and error pages that parse into plausible records. Mechanical verification (title match, amount class, known currency) catches four of the five.
How often should I re-scrape competitor prices?
Most categories change prices daily at most; hourly runs mostly collect cache noise and throttles. Start with a daily pass, tighten the interval where you actually see churn, and keep the per-request delay in place no matter the frequency.
Start with the structured pass
Run Step 1 against one competitor product right now. If the store ships schema.org markup (most Shopify and WooCommerce stores do), you’re one command from a price row with brand, availability, and currency attached. Where the markup is missing or null, Step 4 picks it up for a third of a cent. Build the loop last, verify every record mechanically, and the spreadsheet takes care of itself.
