E-commerce Product Scraper: Extract Catalogs, Prices & Specs

Before you pay a hosted API by the page for product data scraping, look at what stores publish about themselves: schema.org markup on nearly every product page, a /products.json catalog endpoint on Shopify stores, a /wp-json store API on most WooCommerce installs. Every command in this guide ran this morning against two live stores, a Shopify footwear shop and a WooCommerce cup maker, and every output below is what my terminal printed. One local binary plus jq. No account, no API key until Step 4.

Before you start

• The magpie CLI. Single Go binary, free, AGPL-3.0-or-later or a commercial licence. Grab a build from the repo releases.
• jq for the CSV step.
• An LLM API key only if you want spec extraction in Step 4. Everything before it costs nothing and calls no model.

Check what your binary knows about product pages first:

magpie vertical --list

27 extractors ship today. Three matter here: ecommerce_product (any page with schema.org Product JSON-LD), shopify_product (Shopify’s /products/{handle}.js endpoint), and woocommerce_product (WooCommerce markup with a DOM fallback).

Step 1: Pick the exit route before you scrape product data from any website

The store’s platform decides your fastest path. Check that first, because it changes the whole job:

Store runs	Fastest route	You get
Shopify	shopify_product vertical, or the /products.json catalog	Typed records; full catalog as JSON
WooCommerce	woocommerce_product vertical, or /wp-json/wc/store/products	Typed records with SKU; catalog JSON
Anything with JSON-LD	ecommerce_product vertical	Price, currency, availability, brand
Custom platform, no markup	LLM schema extraction (Step 4)	Whatever fields you define
Marketplace (Amazon, eBay, Etsy)	Verticals exist, but these sites wall bots	A different fight (see failure modes)

One failure case to clear up front: robots.txt. Magpie respects it, and its crawler throttles to one request per second per host by default, so a store that disallows /products refuses politely instead of half-working. That’s the intended behavior, not a bug to route around.

Step 2: One page, one record: the website product scraper baseline

Point the vertical at a product URL. This is Allbirds, a Shopify store, with the generic JSON-LD extractor:

magpie scrape "https://www.allbirds.com/products/mens-wool-runners" \
  --vertical ecommerce_product

{
  "url": "https://www.allbirds.com/products/mens-wool-runners",
  "title": "Allbirds Wool Runners, Men's | Reviews, SIzing Info | ...",
  "vertical": "ecommerce_product",
  "record": {
    "availability": "https://schema.org/InStock",
    "brand": "Allbirds",
    "currency": "USD",
    "name": "Men's Wool Runner",
    "price": 110,
    "url": "https://www.allbirds.com/products/mens-wool-runners"
  }
}

Same page, platform-specific extractor instead (abridged):

magpie scrape "https://www.allbirds.com/products/mens-wool-runners" \
  --vertical shopify_product

{
  "vertical": "shopify_product",
  "record": {
    "images": [],
    "price": 110,
    "title": "Men's Wool Runner - Natural Grey (Light Grey Sole)",
    "type": "Shoes",
    "vendor": "Allbirds"
  }
}

Two shapes, same product. The JSON-LD route carries availability and currency; the Shopify route adds product type and vendor but came back with an empty images array on this store. Pick one shape per pipeline and stick to it, or your downstream code will chase both.

WooCommerce works the same way (abridged). Joco Cups, an Australian store on WooCommerce:

magpie scrape "https://jococups.com/product/active-flask-17oz" \
  --vertical woocommerce_product

{
  "vertical": "woocommerce_product",
  "record": {
    "images": ["https://jococups.com/wp-content/uploads/sites/7/2023/01/ActiveFlask_Dune_hero_Web.png"],
    "sku": "AUFLA17",
    "title": "Active Flask 17oz"
  }
}

Note what is missing: no price. More on that in the failure modes, because it is the most common surprise in this whole job.

Step 3: Scrape all products from website catalogs without touching HTML

Per-page requests do not scale to a catalog. Shopify stores expose the whole thing at one URL, and the fetch is a plain GET:

magpie scrape "https://www.allbirds.com/products.json?limit=3" \
  --page-format raw

The output is the store’s raw catalog JSON, abridged to one product:

{
  "products": [{
    "title": "Women's Allbirds Flip Flop - Dusty Pink",
    "handle": "womens-allbirds-flip-flop-dusty-pink",
    "vendor": "Allbirds",
    "product_type": "Shoes",
    "tags": ["gender => womens", "material => sugar", "hue => pink"],
    "variants": [{
      "sku": "A12513W050",
      "available": false,
      "price": "25.00",
      "compare_at_price": "50.00"
    }],
    "images": [{"src": "https://cdn.shopify.com/.../...png", "width": 4000}]
  }]
}

Every variant carries its own SKU, price, compare-at price, and stock flag, and the tags field often holds merchandising attributes (gender, material, color) that the product markup doesn’t carry. Page through with &page=2, up to 250 products per request, Shopify’s documented page cap. A 400-SKU catalog is two requests.

The WooCommerce webshop scraper route: /wp-json

WooCommerce’s store API plays the same role:

magpie scrape "https://jococups.com/wp-json/wc/store/products?per_page=100&page=1" \
  --page-format raw

[{
  "name": "Active Flask Utility Insulated 16oz",
  "sku": "AUIFL16",
  "permalink": "https://jococups.com/product/16oz-insulated-flask/",
  "on_sale": false,
  "prices": {
    "price": "6045",
    "currency_code": "AUD"
  }
}]

Two WooCommerce gotchas, both visible in that record. Prices arrive as strings of minor units: "6045" means A$60.45, and code that floats it without dividing by 100 reports ten-figure revenue. And store endpoints can disagree with the rendered page; this same flask shows A$59.95 on its product page, a 50-cent gap between the store’s own systems. Reconcile before a price feeds any decision.

Some shops disable the store API. When /wp-json 404s or returns a stub, fall back to crawling product pages: magpie crawl with an --include '**/product/**' glob fetches the pages, and Step 2’s vertical handles them one by one. The WooCommerce scraper guide covers that crawl path in detail.

Step 4: Extract specs the markup doesn’t carry

Catalogs give you prices and SKUs. They rarely give you capacity, material, or whether the thing is dishwasher-safe, because stores bury specs in description prose or comparison tables. That’s the one place an LLM earns its keep, and on this page it cost a third of a cent:

cat > specs.json <<'EOF'
{
  "type": "object",
  "properties": {
    "name": {"type": "string"},
    "capacity_ml": {"type": "number"},
    "material": {"type": "string"},
    "price_aud": {"type": "number"},
    "dishwasher_safe": {"type": "boolean"},
    "colours_available": {"type": "array", "items": {"type": "string"}}
  },
  "required": ["name", "capacity_ml", "material", "price_aud"]
}
EOF
magpie scrape "https://jococups.com/product/active-flask-17oz" \
  --schema specs.json --provider openrouter

{
  "extracted": {
    "capacity_ml": 500,
    "colours_available": ["Dune"],
    "dishwasher_safe": true,
    "material": "Glass",
    "name": "Active Flask 17oz",
    "price_aud": 59.95
  },
  "usage": {
    "model": "gpt-4o-mini",
    "prompt_tokens": 1899,
    "completion_tokens": 147,
    "usd_estimate": 0.00037305
  }
}

(url, title, and cache fields omitted from that output for length.)

Bring your own key, any of seven providers including local Ollama models, and cap the spend with --max-cost before a big run. At this writing, roughly 2,700 spec extractions cost a dollar on gpt-4o-mini. Keep the zero-LLM verticals for the fields markup already carries and spend tokens only on the gap.

Step 5: Extract product data from website pages into CSV rows

magpie scrape emits JSON or JSONL; CSV comes from one jq line. The loop below walks a URL list, one request per second, and lands typed records on disk:

for u in $(cat urls.txt); do
  magpie scrape "$u" --vertical ecommerce_product
  sleep 1
done | jq -c '.record' > products.jsonl

jq -r '[.name, .brand, .price, .currency] | @csv' products.jsonl

"Men's Wool Runner","Allbirds",110,"USD"
"Men's Canvas Runner NZ","Allbirds",100,"USD"
"Active Flask 17oz","",,""

That third row is the pipeline working correctly. Joco’s product markup omits the offers block, so price and currency come back null and print as empty cells, exactly where you will spot them. Missing data should look missing. For bigger jobs, magpie crawl --format csv writes CSV natively, and the site-to-CSV guide has the jq patterns for wider or narrower rows.

Product data scraping failure modes I hit today

Everything above worked. Here is what did not, because this is the part that costs you a weekend:

• No markup, no record. The practice site books.toscrape.com serves products as bare HTML tables with zero JSON-LD. --vertical ecommerce_product returns a typed error, no product data at <url>, not an empty object. Fix: platform vertical, catalog endpoint, or Step 4’s schema extraction.
• Thin markup, null fields. Joco carries Product JSON-LD but no price inside it. The extractor honestly reports null. Averages built on nulls are how pricing dashboards lie.
• Marketplaces are a different sport. Verticals named amazon_product, ebay_item, and etsy_listing exist, and on paper they parse those sites. In practice the marketplaces wall bots aggressively, and Amazon’s pricing is session- and region-gated; the same product page can render different prices by delivery ZIP. The Amazon scraper guide documents what actually survives contact. Plan on the catalog endpoints for Shopify and WooCommerce, and on far more machinery for marketplaces.
• JS-only stores. If a record comes back empty on a store you can see rendering in a browser, retry with --render browser. Allbirds needed no such thing; some headless-SPA shops do.

What a free product scraper really costs

The honest ledger for this stack: the Magpie CLI is free and runs local, the verticals make zero model calls, and today’s entire session, catalog pulls included, cost $0.0004, all of it the Step 4 spec extraction. The desktop app is separate: five free runs per install, then $49/month. That buys a GUI over the same engine, plus history, profiles, and scheduling; watch mode and webhooks are core, not paywalled.

Compare the hosted route, where an Apify actor advertises $3.00 per 1,000 items and Oxylabs’ e-commerce API prices by request, per their public listings at this writing. Per item, the numbers can look close. They diverge on two things: every hosted page transits someone else’s servers, and a thousand-page supplier-catalog job is a one-off cost you will repeat monthly. Local means the data never leaves the machine and reruns are free. There is no CAPTCHA bypass here either way; if a store challenges bots, both routes stop.

Choosing product scraper software that survives a redesign

Storefronts redesign. Theme updates reshuffle markup, and the scraper you built on CSS selectors dies quietly, returning rows with empty columns until someone notices the revenue report. This is where product scraper software separates: Magpie caches extraction selectors per domain and schema, watches the null rate, and re-synthesizes a broken selector instead of failing silently. The verticals sidestep the problem differently, by reading the store’s own structured data, which tends to survive redesigns because the store needs it for Google Shopping.

For recurring jobs, add watch mode: point it at a product page or a catalog endpoint and a webhook fires on change, which is the whole backbone of a price tracker without new infrastructure. And when the question outgrows tools, the e-commerce industry page covers the team process around feeds, MAP enforcement, and assortment tracking.

FAQ

Is it legal to scrape product data from online stores? Reading publicly accessible product pages is generally lawful; risk concentrates in what you do with the data (copyright, personal data, terms of service) and how you fetch (robots.txt, rate limits). Respect both. Store APIs exist precisely because stores are usually fine with structured reads of their own catalogs.

Can I scrape Amazon or eBay product prices with this? The verticals exist, but both marketplaces wall bots hard, and Amazon prices vary by session and region. Expect fights, not clean records. The Amazon guide above documents what worked and what refused as of its test date.

What is the best free product scraper? For Shopify and WooCommerce stores, the free verticals and catalog endpoints in this guide beat anything paid, because the store hands over structured data willingly. For arbitrary custom sites, you are choosing between writing selectors yourself and paying per page for a hosted API.

How do I get product data into Excel? Step 5’s jq line writes CSV, which Excel opens directly. Keep prices as numbers in your jq projection (they arrive as numbers in JSON-LD, minor-unit strings in WooCommerce’s API), and the sheet sorts correctly.

How do I monitor price changes automatically? Two ways: cron the catalog endpoint daily and diff the JSON, or point watch mode at the product page and let a webhook tell you when the price node changes. The second catches changes the catalog only shows on the next pull.

Start here: run magpie vertical --list, pick a store you actually buy from, and pull its /products.json. The first CSV takes about four minutes, and you’ll know within the first request whether the store plays straight with its markup.