How to Scrape Websites Without Getting Blocked (Politely)

Search Google for ways to scrape without getting blocked and the first page of results sells proxy rotation. Every one of them treats the block as an enemy to sneak past. I tested the opposite path today: obey the site’s signals, do the rate math, and measure what actually happens. The block arrived in 15 milliseconds when I was rude and never arrived when I wasn’t.

One disclosure before the numbers. The blocking experiments below ran against a 7-page site on my own machine (127.0.0.1) with a 3-requests-per-5-seconds limiter, because hammering a real site to demonstrate rudeness would be a strange way to open a politeness guide. The crawls and conditional-request tests ran against real sites, one polite pass each.

What a block actually looks like

The test site allows 3 requests per rolling 5-second window and answers the fourth with an HTTP 429. I fired 8 parallel requests at it:

1: 200   2: 200   3: 200   4: 429
5: 429   6: 429   7: 429   8: 429

The whole burst took 15ms. That is what a block looks like from the client side: not a dramatic IP ban, just a status code your code probably ignores. The response carried instructions:

HTTP/1.0 429 Too Many Requests
Retry-After: 5

Ignore those instructions long enough and the escalation ladder on real sites goes 429, then 403 for your user agent, then a firewall rule for your IP. Wikipedia’s robots.txt publicly names one crawler, MJ12bot, banned for “ignoring 429 ratelimit responses”. The site wrote the bot’s name into a permanent shame file. That is the reputational cost of pushing through.

The fix costs nothing and most tutorials skip it. Seven steps, in the order I’d apply them.

Step 1: Read robots.txt before you write a line of code

robots.txt is the site’s written statement about automated access. It takes 10 seconds to read:

curl -s https://en.wikipedia.org/robots.txt | head -20

Three things matter in that file. Disallow lines mark paths the site keeps out of crawlers; treat them as walls. Crawl-delay: 5 means one request every 5 seconds for your agent; that number is a floor, not a suggestion. And the comments are intelligence: Wikipedia’s file notes that SemrushBot gets Crawl-delay 5 because the site’s backend budget is around 20 requests per second for everyone.

Failure case: the file is missing. A 404 on robots.txt means “no stated rules”, not “no rules”. Convention (and RFC 9309) says crawl freely but politely. A 5xx on robots.txt means the site can’t answer questions right now, and serious crawlers refuse themselves until it can. Magpie follows exactly that split: 4xx allows the crawl, 5xx or a network failure refuses it. I verified this in crawl/robots.go and watched it refuse a crawl of a host whose robots.txt it couldn’t fetch.

For the full directive grammar, see robots.txt and web scraping.

Step 2: Look for the API, feed, or sitemap before you scrape HTML

The politest request is the one the site built for you. Before parsing HTML, check for structure:

• curl -s https://example.com/sitemap.xml: a crawlable URL list the site hands you, no link-extraction needed.
• RSS or Atom feeds for anything that updates (blogs, news, releases).
• JSON endpoints behind the app: open the browser’s network tab, reload, look for application/json responses.

Each of these replaces dozens of HTML fetches with one structured one. A sitemap also protects you from over-crawling: you request exactly the URLs that exist instead of crawling into calendar archives from 2011. I cover the endpoint hunt in find a website’s hidden API.

Magpie’s crawl does this automatically: it tries sitemap expansion first and falls back to seed-only. Against quotes.toscrape.com (no sitemap) it printed one line, GET /sitemap.xml: HTTP 404; continuing seed-only, and carried on. No wasted requests guessing at URL patterns.

Step 3: Do the rate math before you pick a delay

This is the step that separates polite scrapers from lucky ones. “Polite” is not a vibe, it is arithmetic, and my limiter proved it the hard way. I re-ran the same 8 requests at 1-second spacing, which feels careful, and still collected 429s from request 4 onward:

200 200 200 429 429 429 429 429   <- paced 1/s, still blocked

The site’s budget was 3 per rolling 5 seconds, a sustainable rate of 0.6 requests per second. A steady 1/s stuffs 5 requests into every 5-second window. Three fit, two don’t, and once the window fills it never drains. The first three requests pass forever; everything after fails forever.

At 2-second spacing the same 8 requests all passed. Six real pages returned 200 (the last two probes hit paths that don’t exist, hence 404s, which proves nothing about pacing).

The procedure: find the site’s stated budget in robots.txt Crawl-delay, or estimate one by watching when 429s start, then at least double your interval against it. If Crawl-delay says 5, send one request every 5 seconds or slower. If nothing is stated, 1 request per second per host is a defensible ceiling for a short crawl; 1 every 2 to 3 seconds is safer. And space requests evenly. Eight requests with 7-second gaps between pairs but 0-gap within pairs is a burst wearing a costume.

Step 4: Treat 429 as instructions, not failure

When the 429 arrives, it tells you what to do. Retry-After: 5 means wait 5 seconds. No header means back off exponentially: 2s, 4s, 8s, 16s, cap at 60s, give up after 5 tries. The polite client keeps a per-host counter so one angry endpoint doesn’t stall the whole queue.

import time
import requests

session = requests.Session()

def fetch_with_backoff(url, tries=5):
    delay = 2.0
    for attempt in range(tries):
        r = session.get(url, timeout=15)
        if r.status_code == 200:
            return r
        if r.status_code in (429, 503):
            wait = int(r.headers.get("Retry-After", delay))
            time.sleep(wait)
            delay = min(delay * 2, 60)
    raise RuntimeError(f"{url}: gave up after {tries} tries")

Two properties of 503 and 429 matter: they are the site asking you to slow down, not a hard no; and they are always your fault at current volume. Successful backoff looks like nothing: your crawl finishes later. Magpie ships this as --auto-throttle, which doubles the per-host interval on 429/5xx and decays it back down on success. I verified the flag’s documented behavior against its source (backoff.go consumes 429/503 per attempt) but I did not fire artificial 429s at a real site to watch it work, because that would defeat the entire premise of this article.

Step 5: Revalidate instead of refetching

If your scraper runs on a schedule, most runs re-download pages that never changed. Conditional requests fix that with two headers the server already sent you:

$ curl -s -o /dev/null -w "%{http_code} %{size_download}B\n" https://books.toscrape.com/
200 51294B

$ curl -s -o /dev/null -w "%{http_code} %{size_download}B\n" \
    -H 'If-None-Match: "63e40de8-c85e"' https://books.toscrape.com/
304 0B

The ETag is the page’s fingerprint. Send it back in If-None-Match and the server answers 304 Not Modified with zero body when nothing changed. If-Modified-Since with the Last-Modified date works identically on sites without ETags; books.toscrape.com honored both when I tested.

Store the ETag per URL in your scraper’s state. On the next run a 304 costs you a header round trip instead of a full page, which is kinder to the site and faster for you. The measured saving here was 51KB per unchanged page; multiplied by a 10,000-page crawl on an hourly schedule, revalidation is the difference between a neighbor and a load generator.

Step 6: Identify yourself honestly

A user agent that names your tool and a contact route is the politest signal you can send. It tells the site operator who to throttle or email instead of firewall. Magpie’s default, visible in fetch/http.go, does exactly this:

User-Agent: github.com/motherlodelab/magpie/1.0 (+https://github.com/you/magpie)

Sites can and do allowlist well-behaved agents. Wikipedia’s robots.txt offers SemrushBot a real Crawl-delay it honors; the banned bot got a Disallow-everything. Identification is what makes that difference possible.

The opposite move, rotating fake browser user agents per request, buys you nothing here. It does not change your request rate, which is the thing being measured, and it reads as evasion to anyone reviewing logs. Header details that do matter for compatibility, like Accept and Accept-Language, are covered in web scraping headers.

Step 7: Crawl at the site’s pace, not yours

A single page is easy. A crawl is where politeness gets tested, because the default instinct (fetch as fast as the thread pool allows) is exactly the burst from the top of this article. I ran Magpie’s crawler against quotes.toscrape.com at half its default rate:

$ magpie crawl https://quotes.toscrape.com/ --corpus \
    --max-depth 1 --max-pages 8 --rate 0.5

crawl: sitemap expansion: HTTP 404; continuing seed-only
crawl: run_id=9b760c02... pages_ok=8 pages_err=0 records=8
real 13.16s

Eight pages, zero errors, 13.16 seconds. The --rate 0.5 flag caps the crawler at one request per 2 seconds per host; seven gaps of 2 seconds is 14 seconds of pure pacing, so the wall clock is mostly waiting, which is the whole point. The defaults are already conservative: rate 1 per second per host, and --auto-throttle available for sites that push back. Robots is on by default; --ignore-robots exists and prints a warning when you use it, which is the correct amount of friction.

One behavior worth knowing: the crawler refuses hosts it cannot safely reach, including localhost and private-network addresses, and it reports that as a robots failure. That refusal is an SSRF guard working as designed, and there is no flag that turns it off. Your scraper cannot be recruited against your own router.

Full crawl mechanics, including resuming and scoping, live in build a web crawler and scrape data from a website.

When politeness can’t save you

Some sites ban all automated access regardless of manner. Their terms of service say so, their responses say so, and no delay will change it. If the data matters, the honest options are: find an official API or data-license route, contact the operator, or walk away. What you do not get from me is the proxy-rotation-and-fingerprint-spoofing route that dominates this keyword; that is bypass, it violates the terms you just read, and it escalates the arms race for everyone. On sites that serve bot challenges, Magpie detects and reports the challenge rather than defeating it. The legal frame for all of this is in is web scraping legal.

Scrape without getting blocked: the checklist

Everything above, compressed to what I actually check before a first run:

1. robots.txt read, Disallow paths and Crawl-delay noted (or 404 noted as “no stated rules”).
2. Sitemap, feed, or JSON endpoint checked before HTML parsing.
3. Rate computed against the site’s budget, then halved; requests spaced evenly.
4. 429/503 handler with Retry-After and exponential backoff in place before launch.
5. Conditional requests (ETag or Last-Modified) for anything on a schedule.
6. User agent that identifies the tool and a contact.
7. Crawl concurrency bounded per host, auto-throttle on for unknown sites.

FAQ

How many requests per minute is safe when scraping? There is no universal number. Derive it: robots.txt Crawl-delay sets a floor (Crawl-delay 5 means 12 requests per minute), and otherwise start at 1 request per second per host and halve it if you see any 429s. My test limiter blocked a steady 60 requests per minute; 30 per minute passed cleanly.

Does changing my user agent prevent blocking? Rarely, and never against rate-based blocks, which measure volume, not identity. An honest user agent plus a slow rate beats a fake browser UA plus a fast rate. Sites can allowlist identified agents; nobody allowlists an unidentified one.

What does HTTP 429 mean for a scraper? The server is rate-limiting you and usually says when to return via the Retry-After header. Treat it as an instruction: wait the stated time, or double your interval, and retry. Repeatedly ignoring 429s is how crawlers get named and banned in robots.txt files.

Is scraping a website without robots.txt allowed? A missing robots.txt (404) means the site states no crawler rules; convention and RFC 9309 treat that as permission to crawl politely. A 5xx response means the rules are temporarily unknowable, and well-built crawlers refuse themselves rather than guess.

Do I need proxies to avoid getting blocked? Not for polite scraping. Proxies exist to distribute volume past rate limits, which is the bypass path this article doesn’t cover. One IP, one honest user agent, and a rate under the site’s budget kept every real-site test in this article at zero blocks.

The first thing to change in your scraper

Open your current project and find where HTTP requests happen. If there is no sleep between requests to the same host, add one tonight: time.sleep(2) is the whole fix for the majority of blocks. Then add the 429 handler, because the site that rate-limits you gently today is the site that hard-bans you the week your job gets important. Slow is a feature.