Proxies for Web Scraping: When You Need Them (and When You Don’t)

Search for advice on proxies for web scraping and every result on the first page is written by a company that sells proxy bandwidth. Residential traffic runs $5–15 per GB (Scrapfly’s 2026 pricing overview), so a million-page crawl of 200 KB pages is roughly 200 GB, or $1,000–3,000 of bandwidth. Before you spend that, ask the question the vendor-funded guides skip: do you need a proxy at all?

Most of the time, you don’t. This guide gives you a test-first procedure: measure what a plain, polite crawl actually gets, read the failure codes, and only then buy egress. I’ll use Magpie for the commands, but the decision logic works with any scraper.

What a proxy actually buys you

A proxy changes exactly one input your target sees: the IP address. That’s the whole product. Blocks get decided by several inputs, and a proxy touches one of them:

• Rate per IP: how fast you’re hitting the site from that address.
• IP reputation: whether the address belongs to a datacenter range or a consumer ISP.
• Fingerprint: TLS handshake, HTTP headers, browser behavior.
• Behavior: what URLs you request, in what order, at what rhythm.

If your block was caused by rate-per-IP or datacenter reputation, a proxy helps. If it was caused by a bot fingerprint or a CAPTCHA gate, a proxy just moves your failure to a different address. Magpie detects bot challenges and reports them; it doesn’t defeat them, and no honest tool claim will say otherwise.

One more number before we start: free proxy lists are close to worthless. Scrapfly measured 5–15% of free proxies working at any given moment, which means your crawler spends its budget retrying dead sockets. If egress is genuinely needed, it’s a paid purchase.

Step 1: Run without proxies and measure what happens

Spend fifteen minutes here before spending money. The sequence:

1. Read the site’s robots.txt (10 seconds, and the robots.txt rules deserve their own read).
2. Pick a polite rate; 1 request per second is a reasonable default that most sites never notice.
3. Crawl 20–50 pages and record the status codes.

Here is a real run from today, no proxy configured, Hacker News front page:

$ magpie scrape https://news.ycombinator.com

{
  "url": "https://news.ycombinator.com",
  "final_url": "https://news.ycombinator.com",
  "title": "Hacker News",
  ...
}

844 words in the record, 4.0 seconds wall time, HTTP 200, zero egress cost (output abridged above). HN serves everyone, including your scraper. Sites like this (and it’s most sites) need politeness, not infrastructure.

The status codes from your sample run are the actual decision. Here’s the decoder:

What you got	What it means	The fix that costs $0
All 200s	No meaningful protection	Keep scraping politely. Done.
429 with Retry-After	Rate cap, the site is negotiating	Slow down to the stated pace
403 on your user agent	UA filtering, not IP filtering	Send honest headers
403 or challenge on every request	Fingerprint or IP-reputation gate	Maybe a proxy. Read Step 3.
Different content per country	Geo-fencing	Egress location matters. Read Step 3.

Failure case to watch for: the sample works but the full crawl dies at page 400. Rate caps are cumulative: a rate that’s invisible for 50 requests can trip a sliding-window ban at 500. Sample at your real crawl volume, not just a handful of pages.

Step 2: Exhaust the free fixes first

429s and UA-based 403s are instructions, not walls. The full playbook lives in my guide to scraping without getting blocked, but the short version: honor Retry-After, keep requests to one per host at a sane interval, identify yourself honestly, and revalidate with conditional requests instead of refetching unchanged pages.

The math is friendlier than people assume. At one request per second, a single IP pulls 3,600 pages per hour. That’s 86,000 pages a day from one polite connection. If your target list fits inside that envelope, the proxy conversation is over before it started.

Magpie applies this automatically where it can: robots.txt is respected, per-host throttling is built in, and the crawler refuses to run when a site’s robots.txt is unreachable (a 5xx means “no stated rules right now”, which I treat as no).

Step 3: When you need proxies for web scraping

When polite crawling measurably fails at the rate you need, these are the situations where egress is the actual answer, and how to tell them apart.

1. Per-IP rate caps below your target rate. Some sites cap each IP at, say, one request every 2 seconds. If you need 10 requests per second, no amount of politeness gets you there from one address; you need 20 IPs each staying under the cap. This is arithmetic: IPs needed = target rate ÷ per-IP allowance. Buy datacenter proxies, rotate across them, and each address looks like a slow, polite visitor.

2. Datacenter IP reputation walls. Some targets 403 everything from known cloud-hosting ranges before your first request even carries a header. If you’ve verified this (same request, same headers, different network, different result), you need addresses that look like homes. That’s the residential market: $5–15/GB, and you should buy the smallest pack you can while you prove the target actually unblocks consumer IPs.

3. Geo-fenced content. Prices, catalogs, and search results differ by country, and some content is flat-out unavailable outside a region. The proxy here isn’t evading anything; it’s choosing which country’s view you see. Pick a vendor with egress in the specific country, not “worldwide coverage”.

4. Genuinely large crawls of sites that do enforce caps. Search engines and big marketplaces sit here. Be honest with yourself about this bucket: the site’s terms usually restrict automated access, and a proxy doesn’t change what the terms say. The legal frame is its own topic — start with is web scraping legal — and know that a proxy letting you ignore a block has moved you from “scraping” into “bypassing”, a different and worse conversation.

What proxies do not fix: browser fingerprints, CAPTCHAs, TLS analysis, login-gated data (Magpie handles those with persisted sessions; see scraping behind a login), and JavaScript rendering (that’s a browser need, covered in scraping dynamic sites). Budget for the actual problem, not the one the vendor’s landing page describes.

Rotating proxies: what rotation fixes and what it breaks

A rotating proxy is a pool of addresses, typically behind one gateway host, where each request, or each timed session, exits from a different IP. That’s the standard definition, and it tells you both halves.

Rotation fixes the per-request case: when a site counts requests per IP, handing every request a fresh address resets the counter. It also fixes dead-address resilience; a pool with failover shrugs off an IP that stops responding.

Rotation breaks state. Anything the target ties to the source IP (a login, a paginated listing that lives in a server-side session, a multi-step form) fails when the address changes mid-flow. You’ll see empty pages, lost carts, and sessions that vanish between request 3 and request 4. The fix is sticky sessions: hold one address for a bounded window (typically minutes, on the vendor’s clock) and rotate between windows.

This is exactly why Magpie’s proxy pool has two strategies instead of one. round-robin (the default) spreads requests evenly and is right for the rate-cap case. sticky-host hashes the target hostname so the same target always rides the same entry, the right call when the target keeps state. For residential vendors whose credentials embed a session token, Magpie substitutes a stable per-host token wherever you write {{session}} in the pool line, so the vendor’s rotating gateway gives you stickiness too.

How to wire proxies into Magpie

All of this is CLI surface, verified in the current build. A pool file, one proxy per line:

$ cat proxies.txt
# datacenter pool, one per line
198.51.100.7:8000:alice:s3cret
proxy2.vendor.io:8000:alice:s3cret
# residential gateway with sticky sessions
gw.vendor.io:7000:alice-sess-{{session}}:s3cret

$ magpie crawl https://example.com --proxy-file proxies.txt

Line format is http(s)|socks5(h)://host:port or the host:port:user:pass paste that most vendors hand you. Comments start with #. For a single proxy, skip the file: MAGPIE_PROXY=host:port:user:pass. Precedence runs per-run flag → pool file → MAGPIE_PROXY → the standard HTTP(S)_PROXY environment variables, so nothing you already have breaks.

Failover is automatic and narrow by design. A proxy that won’t connect gets skipped for 60 seconds and re-enters rotation after; if every entry is dead, the error lists the endpoints without ever printing credentials. But an HTTP 403 or 429 from the site does not trigger failover: a site error is not a proxy error, and switching IPs to dodge it would be bypass, not resilience.

Two behaviors I verified live today. A dead proxy (connection refused) fails the run with a clear error and credentials absent:

$ magpie scrape https://news.ycombinator.com --proxy-file dead.txt
Error: fetch: Get "https://news.ycombinator.com": proxyconnect tcp:
  dial tcp 127.0.0.1:9: connect: connection refused

A malformed pool line is a usage error, exit code 2, with the line number and the expected grammar:

$ magpie scrape https://news.ycombinator.com --proxy-file typo.txt
Error: fetch: proxy pool line 1: fetch: bad proxy config:
  want http(s)|socks5(h)://host:port or host:port:user:pass

Picking a type when you do need one

The 2026 market, per Scrapfly’s overview:

Type	Typical price	Looks like	Buy it when
Datacenter	$1–3 per IP/month	A server	Rate caps are your only problem
ISP / static residential	$2–5 per IP/month	A home, but stable	You need consumer IPs with sessions that persist
Residential (rotating)	$5–15 per GB	A home	Datacenter ranges are 403’d on sight
Mobile	$10–20 per GB	A phone carrier	The target is aggressive enough to gate datacenter and residential
Free lists	$0	Anything	Never, seriously: 5–15% work

My ordering: start datacenter, because the failure you’re fixing determines everything else. If datacenter addresses get 403’d while the same request from your laptop works, you’ve proven IP-reputation blocking and residential is justified. Jumping straight to mobile proxies for a docs-site crawl is the most common budget mistake I see, and the bandwidth math explains why: heavy browser-rendered pages run 250–500 pages per GB against a residential plan, while static 200 KB pages stretch the same gigabyte across 5,000 pages.

FAQ

Are proxies for web scraping legal? A proxy is networking equipment; the legality question attaches to what you access and how, not to which address you access it from. Ignoring a block via rotation can however strengthen a case that you knowingly bypassed access controls. The full frame is in is web scraping legal.

Do I need rotating proxies for a small site? No. A small site has small per-IP budgets and a polite crawler stays far under them. Run the Step 1 test first; most small targets never send a single 429.

Can I use free proxies for scraping? You can, and you’ll regret it: 5–15% work at any moment, shared IPs carry other people’s bad reputations, and free proxy operators have been caught inspecting traffic. Paid datacenter IPs start at $1–3 a month.

How many requests before a site blocks me? There’s no universal number: every site sets its own per-IP budget, and some set none. Measure: sample crawl at your real volume, read the status codes, and let the site tell you its cap via Retry-After.

Is a VPN a substitute for a scraping proxy? For testing geo-fenced content, sure; it’s just a static egress address. For crawling, no: one VPN IP is one IP, with the same per-IP rate budget and the same single point of failure you already had.

The order that saves you the money

Run polite with no proxy. If it works, you’re done, and most targets will. If it 429s, slow down to the stated pace. If it 403s on headers, fix the headers. Only when you’ve measured a per-IP cap below your target rate, or a verified IP-reputation wall, or a geo-fence, do you open a pool file, and start with two datacenter entries, not a residential gigabyte plan. The first proxy you buy should be purchased because a status code told you to.