Web Scraping Bots: What They Are and How to Run a Polite One

Imperva counted the 2024 web and found that bots out-requested people for the first time: 51% of all traffic was automated, and 37% was hostile automation. A scraping bot lives somewhere in that automated half — and if you’re about to write one, the honest question isn’t whether you can fetch 10,000 pages. It’s whether the site will still be speaking to you next week.

This page explains what a scraping bot is, how one works step by step, and the six politeness rules that decide whether yours is a good tenant or an outage waiting to happen. The politeness part comes with real arithmetic, because “don’t hammer the server” is useless advice until you’ve seen what one request per second means to a small site.

What a scraping bot actually is

A scraping bot is a program that fetches web pages automatically and copies the data out of them. It sends HTTP requests, reads the HTML (or JSON, or a rendered page), pulls out the fields it cares about, and stores them somewhere: a CSV, a database, a feed for another system. No human clicks anything. The bot might fetch one page on a schedule, or a million pages across a thousand sites.

Mechanically, that’s the same thing Googlebot does. The difference is purpose and permission. A search crawler copies your pages so it can rank them and send traffic back; it announces itself, publishes its IP ranges, and its visits are the deal every public site signs up for. A scraping bot copies your pages to take the data somewhere else: a price comparison, a research dataset, or an AI training corpus. Its permission comes from whatever the robots.txt and the law say, not from an industry pact.

Not every bot that copies data deserves the villain framing, either. Journalists tracking page changes, academics collecting public records, a founder watching a competitor’s pricing page: all scraping bots. So are the contact-harvesting spam bots Cloudflare’s Learning Center catalogs alongside them. The tool is the same; the tenancy isn’t.

Bot	What it wants	Typical footprint
Search crawler (Googlebot, Bingbot)	Copies pages to rank them	Steady, announced, respects robots.txt
Scraping bot	Takes specific data home	Bursty or scheduled, permission varies
One-off scraper script	One dataset, one job	A few dozen requests, then gone
AI crawler (GPTBot, ClaudeBot)	Pages for training or answers	Heavy since 2023, default-blocked by many sites

The last row is its own story; the AI crawlers and the sites blocking them get a whole page of their own. The broader craft of taking data apart is the subject of our data scraping guide; this page stays on the bot itself.

How a web scraping bot works, step by step

Every scraping bot, from a 20-line script to a distributed fleet, runs the same loop. The details are where they differ.

1. Fetch. The bot requests a URL and gets bytes back. Static fetching is one HTTP GET: cheap, fast, and enough for server-rendered pages. When the target loads its content with JavaScript, the bot escalates to a real headless browser, which renders the page before extraction. That escalation costs 10 to 100 times more per page: a browser instance holds hundreds of megabytes of RAM and fires dozens of extra requests for scripts, styles, and fonts.

2. Parse. Raw HTML becomes a tree the code can query: DOM, XPath, CSS selectors, or regex for the reckless. Parsing is where format surprises live: malformed HTML, encoding lies in the meta tag, tables built from nested divs.

3. Extract. The bot pulls specific fields: prices, titles, dates, links to follow next. This is the fragile part of any scraping project. Selectors rot when the site redesigns, which is why serious setups monitor extraction quality (zero records extracted is a symptom, not an empty result) rather than trusting the run that “succeeded.”

4. Store and repeat. Fields land in a file or database, the queue grows with newly discovered URLs, and the loop continues until the budget, the queue, or the site’s patience runs out.

The loop is boring. The engineering is everything wrapped around it: the queue, the deduplication, the retries, and the throttle, which is what the rest of this page is about.

What “data scraping bot” means

A data scraping bot is the same machine with a different emphasis. “Web scraping” usually implies copying pages or content; “data scraping” implies the target is structured values (prices from a shop, stats from a league table, rows from a directory), often from pages that were never meant as an API. Some older usage means scraping desktop applications, but on today’s web the phrase almost always points at fields, not pages. If your deliverable is a spreadsheet rather than a mirror of the site, you’re doing data scraping, and our data extraction guide covers where the two crafts split.

The politeness rules that actually matter

Politeness gets sold as ethics. It’s also self-interest: a bot that respects load limits runs for months, and a bot that doesn’t gets IP-blocked within hours. Bright Data’s own glossary makes the same case, which tells you something: even the proxy industry admits throttled crawlers outlive aggressive ones. Here are the rules with numbers attached; the longer list of field-tested habits lives in our best-practices guide.

1. Read robots.txt before the first request. The Robots Exclusion Protocol has been an official internet standard since RFC 9309 (September 2022). Fetch /robots.txt on every host before you fetch anything else. Two edge cases people get wrong: a missing robots.txt (404) means everything public is allowed, but a server error (5xx) means assume everything is disallowed until it comes back. Python’s urllib.robotparser implements both rules correctly. Note that Crawl-delay is advisory only: Googlebot ignores it (Google tunes crawl rate in Search Console instead), while Bing honors it up to 30 seconds. For everyone else’s crawler, it’s the site’s stated speed limit.

2. Do the rate math per host. Concurrency of 1 and a 1-second delay sounds restrained. It isn’t:

Your delay	Requests per day
1 second	86,400
5 seconds	17,280
10 seconds	8,640

A modest personal site serves a few hundred requests a day. Against that baseline, your “slow” one-request-per-second bot is hundreds of times busier than every human visitor combined, and one request per second is roughly what a $5 shared host can absorb while still answering real users promptly. For anything smaller than a large company’s infrastructure, start at one request every 5 to 10 seconds and watch response times. If p95 latency climbs, you’re not polite — you’re an incident.

3. Announce yourself. A polite bot sends a user-agent like price-watch/1.0 (+https://yoursite.example/bot): name, version, and a URL where the site admin can reach you. Python’s requests library defaults to a user-agent that starts with python-requests, which reads as noise and gets filtered generically. A real identity buys you the difference between an angry email and a silent IP block. It costs you something too, and the next section covers that honestly.

4. Fetch less, not just slower. The politest request is the one you don’t make. Re-validate with conditional requests: store the ETag each response gives you, send it back as If-None-Match, and the server answers 304 Not Modified. That’s a few hundred bytes instead of the full page, and no re-parsing. Sites that serve ETags effectively let your bot poll hourly at near-zero cost. The same logic says skip images, stylesheets, and tracking scripts entirely; your scraper needs HTML, not the 40 extra requests a browser would make.

5. Obey the server’s signals. A 429 Too Many Requests with a Retry-After header is the site telling you the wait time in seconds. Honor it exactly. Repeated 503s mean the server is drowning: stop the crawl, don’t retry harder. Rising latency without errors is the same message in a softer voice. A bot that backs off when times degrade behaves like a good monitoring system; a bot that pushes through behaves like an attack, because that’s exactly what it looks like in the logs.

6. Schedule around the site, not around yourself. Poll at 4 a.m. site-local time when you can. Crawl incrementally: the 30 pages that changed, not the 30,000 you already have. If the target publishes a sitemap with lastmod dates, that’s the site doing your change detection for you.

Tools that enforce this by default: Scrapy’s AUTOTHROTTLE setting adjusts delay from response times automatically, and Magpie (the scraper behind this site) checks robots.txt, throttles per host, and escalates to a browser only when a page needs one: politeness as the default config rather than a checklist. Neither tool bypasses CAPTCHAs or bot protection; that’s not what the polite end of this craft is for.

A polite bot in 35 lines of Python

This is the whole loop (robots.txt, pacing, identification, revalidation, backoff) against a real site with a real robots.txt. python.org’s file names and bans known scraper tools (HTTrack gets its own Disallow: /), so it demonstrates a live allow decision for your honest user-agent.

import time
import requests
from urllib.robotparser import RobotFileParser

BASE = "https://www.python.org"
UA = "price-watch/1.0 (+https://mysite.example/bot)"

rp = RobotFileParser(f"{BASE}/robots.txt")
rp.read()  # 404 -> allow all; 5xx -> disallow all (RFC 9309)
delay = rp.crawl_delay("*") or 5  # seconds; your own cap if unset

urls = [f"{BASE}/blogs/", f"{BASE}/jobs/", f"{BASE}/events/"]
etag = {}

with requests.Session() as s:
    s.headers["User-Agent"] = UA
    for url in urls:
        if not rp.can_fetch(UA, url):
            print(f"SKIP (robots): {url}")
            continue
        headers = {"If-None-Match": etag[url]} if url in etag else {}
        r = s.get(url, timeout=10, headers=headers)
        if r.status_code == 304:
            continue  # unchanged since last visit
        if r.status_code == 429:
            time.sleep(int(r.headers.get("Retry-After", "60")))
            continue
        if r.status_code >= 500:
            print(f"Server struggling ({r.status_code}); stopping.")
            break
        if new_etag := r.headers.get("ETag"):
            etag[url] = new_etag
        print(f"OK {r.status_code} {len(r.text)} bytes  {url}")
        # ... your parse/extract step goes here ...
        time.sleep(delay)

Output (abridged):

OK 200 49210 bytes  https://www.python.org/blogs/
OK 200 48877 bytes  https://www.python.org/jobs/
OK 200 49032 bytes  https://www.python.org/events/

About 20 lines of logic, and it already behaves better than most of the traffic Imperva counted. The 304 branch only fires on servers that send ETag headers; python.org doesn’t on every path, so treat that branch as the pattern, not a promise about this host. Add a queue and a database and you have a production bot.

What politeness does not buy you

Most “be polite” advice stops right before the uncomfortable part. Here are the three things it skips.

Polite bots get blocked anyway. On July 1, 2025, Cloudflare began blocking known AI crawlers by default for new customers: bots that honor robots.txt, announce themselves, and behave. Blanket rules and industry pressure don’t distinguish a well-behaved scraper from a greedy one; they see a bot and act. If your project depends on a site whose owner has said no in any form, politeness doesn’t unlock the door. Find a different route: many sites offer RSS or an API that beats scraping for freshness and stability, and our data extraction guide lists the legitimate routes into common data types.

Disguise is the failure mode to refuse. The moment your polite bot gets filtered, the tutorial-industry answer is: rotate proxies, spoof the user-agent, fingerprint-match a real Chrome TLS handshake. That’s not politeness with extra steps. It’s a different activity with an arms race attached, and detection vendors fingerprint TLS handshakes (JA3/JA4) precisely to catch it. My line: if a bot only works while pretending to be a human with a credit card, the site has told you no. The sustainable answers are the boring ones: public datasets, official APIs, asking.

None of this is legal advice, either. Permission and legality run on separate tracks (robots.txt is a convention, not a law), and the legal picture, which our plain-English guide covers properly, turns on what data you take, where the servers are, and what you do with it. Polite and legal usually travel together, but they’re different checklists.

FAQ

Is Googlebot a scraping bot? Mechanically yes: it fetches pages and copies their content. Culturally no: it identifies itself, supports publisher controls, and its copying feeds an ecosystem that sends traffic back. That’s why most sites whitelist it while blocking other scrapers.

How fast is too fast for a scraping bot? Anything faster than a page request every few seconds to the same host deserves scrutiny. A one-second delay is 86,400 requests a day; a small site sees a few hundred. Start at 5–10 seconds per request with concurrency 1, then adjust to the server’s response times.

Can a polite bot still get blocked? Yes, and increasingly does. Default-deny bot policies, user-agent filters, and TLS fingerprinting block well-behaved crawlers along with abusive ones. Politeness reduces your footprint and preserves the relationship where one exists; it doesn’t override a site’s decision.

How do I stop scraping bots on my own site? Rate limiting that kicks in above human speeds, a well-formed robots.txt for the crawlers you want, and bot management for the rest. Cloudflare and Imperva both sell versions. Publishing an API or licensed data feeds removes the incentive for the polite ones entirely.

Where can I test a scraping bot safely? On sites built for it. Sandboxes like quotes.toscrape.com and books.toscrape.com exist so you can practice against targets that want the traffic; we keep a list of practice sites and real data sources for exactly this.

The politeness checklist

Run every bot you ship against this list:

1. robots.txt fetched and obeyed per host, including the 5xx rule
2. Concurrency 1 against small sites; delay 5–10 seconds until data says otherwise
3. User-agent with a name, a version, and a contact URL
4. ETag revalidation for anything polled on a schedule
5. Retry-After honored exactly; 503 stops the crawl
6. Incremental crawls, off-peak schedule, sitemap lastmod where offered

The one-sentence version: fetch slowly, say who you are, take only what changed, and stop when the server says stop. Do that and your bot stays welcome for years — which is longer than most scraping projects need, and exactly as long as a good one should last.