Web Scraping Tutorial: Step-by-Step Guides

Every web scraping tutorial on page one starts the same way: pip install, paste forty lines, done by minute five. None of them survive contact with the second website you try, because the code was never the hard part. This one runs the whole path live: the robots.txt check, a 20-line first scraper against a real bookstore, a JavaScript page that looks empty until you know where to look. Every number below comes from a run I made today, not from imagination.

The path:

1. What a web scraper actually does
2. Step 1: Pick a target you’re allowed to scrape
3. Step 2: Read robots.txt before you write code
4. Step 3: Look at the HTML before choosing a method
5. Step 4: Your first scraper, in 20 lines
6. Step 5: When the page needs JavaScript
7. Step 6: Store the data somewhere useful
8. Step 7: Scrape politely or don’t scrape at all
9. How to actually learn web scraping
10. Which guide to open next
11. Four mistakes that kill first projects
12. Web scraping tutorial FAQ

What a web scraper actually does

The bookstore homepage I scraped this morning is 51,294 bytes of HTML holding 20 books, each with title, price, and star rating. A scraper makes four moves. Fetch the HTML. Parse it into a tree. Pull out the fields you named. Store them somewhere a spreadsheet or a program can read.

Everything that makes scraping look scary is one of those moves getting harder. A page that renders with JavaScript hides the HTML from you at first. A login wall hides it behind cookies. An angry server hides it behind a 403. The fixes change; the shape of the problem never does. Keep the four moves in mind and every tool you’ll ever meet is just a different answer to one of them.

Step 1: Pick a target you’re allowed to scrape

For learning, pick a site built to be scraped. The two I use are books.toscrape.com and quotes.toscrape.com, sandboxes that exist so beginners can practice without anyone’s lawyers waking up. For a first real project, scrape a site you own or one with a published open-data policy.

If your target is somebody’s business, read the ground rules first: the site’s terms, and the law that applies to the data (personal data has rules that product prices don’t). The full picture, with the actual court cases, lives in Is Web Scraping Legal?. One sentence of practical law: publicly available data, scraped politely, at a human-ish pace, for analysis, is the case US courts have repeatedly sided with. Scraping behind a login you paid for, to resell someone’s content, is the other kind of day.

Step 2: Read robots.txt before you write code

robots.txt is a file at the root of most sites that states which paths the owner asks bots to leave alone. It takes thirty seconds to read and it settles most “am I allowed” questions before they start:

$ curl -s https://news.ycombinator.com/robots.txt
User-Agent: *
Crawl-delay: 30
Disallow: /reply?
Disallow: /vote?
...

That real file says two things. Disallow lists paths a bot shouldn’t touch: reply and vote endpoints, which are account actions, not pages. Crawl-delay: 30 says one request every 30 seconds at most. Any scraper I point at Hacker News waits 30 seconds between requests; a human reading the site takes about that long between clicks anyway.

Two things robots.txt does not do. It isn’t enforced (nothing stops a rude bot, which is why ignoring it marks you as one). And it can be missing entirely: both scraping sandboxes return 404 for /robots.txt (books.toscrape.com answers with a bare nginx 404 page). No file published means no stated restrictions, not “do anything at any speed.” The politeness rules in Step 7 still apply.

Step 3: Look at the HTML before choosing a method

This is the step every code-first tutorial skips, and it’s the one that decides which tutorial you need. Open the page, hit Ctrl+U (view source), and search for the data you want.

Today’s example. The books homepage view-source shows 51,294 bytes with <article class="product_pod"> wrapping every book. Data in the source means a plain HTTP request will get it: no browser, no JavaScript engine, milliseconds per page.

Now the counter-example, same sandbox family. curl -s https://quotes.toscrape.com/js/ brings back 5,806 bytes, and searching it for quote markup finds nothing:

$ curl -s https://quotes.toscrape.com/js/ | grep -c 'class="quote"'
0

The page you see in a browser has ten styled quotes. The page the server ships has none of them. That gap is the static-vs-JavaScript fork in the road, and Step 5 walks through it. The point of Step 3 is that five minutes of Ctrl+U tells you which branch you’re on before you’ve installed anything.

Three page shapes cover almost every site you’ll meet:

What view-source shows	What it means	Where you’ll end up
Your data in the markup (books.toscrape.com)	Plain HTTP is enough	Step 4
Your data in a <script> tag, as JSON (quotes.toscrape.com/js/)	Parse the script tag, skip the browser	Step 5, no browser
Your data nowhere in the source	It arrives after the page loads; the DevTools Network tab shows where from	Step 5, with a browser

While you’re in the source, note the selectors you’ll need: the class names wrapping each item. If CSS selectors are new to you, the CSS selectors glossary entry covers the ten you’ll use constantly. They are the single most transferable skill in this whole trade: BeautifulSoup, Scrapy, Playwright, Cheerio, and every no-code tool speak them.

Step 4: Your first scraper, in 20 lines

Prerequisites: Python 3.9 or newer, then pip install requests beautifulsoup4. I ran this today against Python 3.12.3, requests 2.32.5, and beautifulsoup4 4.14.3.

import time

import requests
from bs4 import BeautifulSoup

start = time.perf_counter()
resp = requests.get("https://books.toscrape.com/", timeout=30)
resp.raise_for_status()
resp.encoding = "utf-8"                      # why this line exists: below
soup = BeautifulSoup(resp.text, "html.parser")

books = []
for article in soup.select("article.product_pod"):
    books.append((
        article.h3.a["title"],                          # attribute
        article.select_one("p.price_color").get_text(), # text
        article.p["class"][1],                          # "One".."Five"
    ))

for title, price, rating in books[:3]:
    print(f"{price:>8}  {rating:<5}  {title[:52]}")
print(f"... {len(books)} books total, {time.perf_counter() - start:.2f}s")

What you should see (abridged):

   £51.77  Three  A Light in the Attic
   £53.74  One    Tipping the Velvet
   £50.10  One    Soumission
... 20 books total, 1.87s

Twenty books, fetch included, in under two seconds — on a sandbox that’s deliberately slow (the same fetch took 8.4 seconds once this morning; production sites are kinder, but the variance lesson holds: never assume one timing is the timing).

The failure case worth an hour of your life: my first run printed £51.77. That garbage character before the pound sign means an encoding mismatch: the page is UTF-8, but requests guessed ISO-8859-1 from the response headers and decoded the two-byte £ (0xC2 0xA3) as two wrong characters. resp.encoding = "utf-8" fixes it. When scraped text shows mojibake like £ or ’, don’t debug your parser. Check the encoding first; it’s that, almost every time.

One line worth understanding rather than pasting: soup.select("article.product_pod") returns every <article> whose class is product_pod, and everything inside the loop walks relative to one of those. Beginners who select from the document root for every field get pages of scrambled rows. Select the repeating container, then reach down.

Step 5: When the page needs JavaScript

Step 3’s counter-example deserves its own walkthrough, because “the data wasn’t in the HTML” has two very different causes, and only one of them needs a browser.

View the source of quotes.toscrape.com/js/ again and scroll into the <script> block. The quotes are there, as JSON, in a JavaScript array:

<script>
    var data = [
    {
        "tags": ["change", "deep-thoughts", "thinking", "world"],
        "author": {"name": "Albert Einstein", ...},

The server ships the data with the page; a script then document.writes it into styled divs. The rendered DOM has quotes. The raw HTML has raw JSON. You don’t need a browser — you need to parse the script tag instead of the page. That’s a five-line change, and it’s faster and more stable than any headless browser would be, because JSON schemas change less often than CSS classes.

The genuinely-browser-required case is different: data that arrives after the page loads, via a background request triggered by scrolling or clicking, or markup assembled from fragments where no JSON payload exists. You spot it in the browser’s DevTools Network tab: XHR requests returning exactly the data you want (in which case, call that endpoint directly) or nothing usable at all (in which case, automate a real browser). Both paths have step-by-step guides: how to scrape JavaScript-heavy sites covers the diagnosis, and the Playwright tutorial covers the browser automation. The JavaScript rendering glossary entry explains the vocabulary. A headless browser costs seconds per page instead of milliseconds, so earn your way to it; don’t start there.

Step 6: Store the data somewhere useful

A list of tuples in memory dies with the script. Two lines of stdlib make it survive:

import csv

with open("books.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.writer(f)
    writer.writerow(["title", "price", "rating"])
    writer.writerows(books)

CSV opens in any spreadsheet; JSON keeps nested structure (ratings per tag, quotes per author) without flattening gymnastics. Pick by consumer: humans and Excel want CSV, programs want JSON. The full decision, with the gotchas (Excel’s encoding quirks, nested fields, deduplication), is its own guide: scraping a website to CSV or JSON. When the raw rows need cleaning (prices as text, duplicate entries, missing fields), how to clean scraped data covers the pass that comes next.

Step 7: Scrape politely or don’t scrape at all

One page, one request: almost anything passes. The moment you loop (pagination, whole catalogs, monitoring), you’re a load on someone’s server, and the difference between a scraper that lasts and one that gets its IP banned is about five lines of manners.

The rules, in the order they matter:

1. Obey robots.txt and Crawl-delay (Step 2). Hacker News asks for 30 seconds between requests. That’s the speed limit, not a suggestion.
2. Throttle yourself even where no rule is published. time.sleep(1) between requests is the floor, not the ceiling. A request per second, all day, is not a human pattern and eventually gets treated like one.
3. Identify yourself. A User-Agent string with contact info (“my-scraper/0.1; [email protected]”) gets you emails from admins instead of silent bans. The User-Agent glossary entry explains what servers read.
4. Send sane headers. Requests without an Accept header or with a mismatched Accept-Language are bot-shaped in ways you can’t see. HTTP headers for web scraping lists the four that matter.
5. Cache. If you already fetched a page this hour, don’t fetch it again. Plenty of accidental DDoS is a retry loop with no memory.

None of this is about evading detection. It’s about being the kind of traffic a site admin never has to think about, which is also how you avoid the blocking arms race entirely. When you do hit walls (rate limits, challenges, IP blocks), scraping without getting blocked walks the polite countermeasures, and the rate limiting glossary entry decodes the 429s you’ll meet along the way.

How to actually learn web scraping

People ask how long it takes. Honest answer from having watched a few dozen people do it: a weekend to reach “I can pull clean data from a static page,” and the static-page case is most pages you’ll actually want. Weeks, not months, to handle JavaScript and logins. The anti-bot arms race is the only deep end, and you can have a useful scraping life without swimming in it.

The order matters more than the hours. The path that works:

1. Run the seven steps once against a sandbox, exactly what this tutorial just did, with your own hands on the keyboard.
2. Learn CSS selectors properly. Not “vaguely recognize them.” The 30 minutes on child selectors, attribute selectors, and :nth-child pays out in every tool you’ll ever open. It’s the transferable skill; everything else is syntax.
3. Pick one language and go deep with its tutorial (map below). Python has the gentlest on-ramp and the most answers on the internet when you get stuck, which is a real learning advantage.
4. Do one hard page type. A JavaScript page or a login-gated one. The first time it works, you’ll understand why the four moves matter.
5. Only then scale up to crawling many pages, where politeness and state management start to bite.

Most people invert steps 2 and 3. They learn a framework’s API first and can’t scrape anything when the framework is out of fashion two years later. Selectors outlive frameworks.

Which guide to open next

This page is the front door; the step-by-step guides for each branch live below it. The map, by what you’re trying to do:

Your next step	The guide
Scrape with Python (or decide when not to)	Python web scraping tutorial
Parse HTML step by step, deep on BeautifulSoup	BeautifulSoup tutorial
Build a proper framework-grade spider	Scrapy tutorial
Scrape from JavaScript/Node	Node.js web scraping
Automate a real browser	Playwright, Puppeteer, or Selenium
Scrape from Go	Golang web scraping tutorial
Skip the code entirely	Scrape data from any website (no-code methods) or the alternatives hub
Get through a login wall	Scraping a site that requires login
Handle “next page” links	Web scraping pagination
Pull a whole site, not one page	Scrape an entire website and find all pages on a site
Extract tables specifically	Scraping tables

There are language-specific tutorials beyond the big ones (Rust, Java, PHP, Ruby, R, C#, even Excel VBA), each with runnable code, plus a command-line scraper guide for terminal people and the conceptual ground floor in What Is Data Scraping?. If you’re coming here from a spreadsheet background, Google Sheets web scraping starts with a function you already know how to type.

Sidebar: when you’d rather not code it

One honest note from the tool I use daily. Magpie’s core is a free, local CLI (AGPL-3.0-or-later or a commercial license, from Motherlode Lab) that does the four moves in one command: magpie scrape https://quotes.toscrape.com/ returned the ten quotes as clean markdown in seconds when I ran it today, boilerplate gone. It also refused books.toscrape.com with quality blocked (empty). Its quality gate won’t return pages that clean to under 200 words, and a catalog grid is mostly links, not prose. A reasonable guardrail that occasionally gets in the way; worth knowing either way. Self-healing selectors, code that rewrites its own broken selectors, is its real party trick, and the reason I reach for it on sites that redesign often.

Four mistakes that kill first projects

Writing code before viewing source. Every hour at Ctrl+U saves three at the keyboard. The /js/ page above would have eaten a beginner’s whole evening “debugging” a parser that was never the problem.

Assuming the browser shows you the HTML. View-source is the truth; DevTools’ Elements panel is the truth after JavaScript ran. Diagnose against the wrong one and login walls and JS pages look like parser bugs. Same confusion, different flavor: a page you see logged-in returns logged-out HTML to your scraper; cookies are data too, and the login guide covers them.

Hardcoding selectors that rot. Sites redesign. A selector like div.results > div:nth-child(3) span.price dies the day someone inserts a wrapper div. Prefer stable handles (IDs, data attributes, semantic classes), and when you maintain scrapers across many sites, read how self-healing scrapers approach the problem.

Hammering the target. The polite-scraper rules in Step 7 exist because the failure isn’t hypothetical: no throttle, no cache, no backoff ends in a 429 or a silent IP block, usually mid-run, always at the worst time. A permanent ban from your best data source is a career expense you chose.

Web scraping tutorial FAQ

Is web scraping legal? Publicly available data scraped politely, at modest volumes, for analysis or research: yes, and the public-data line has US court precedent behind it (hiQ, Van Buren). The lines that move: personal data, login-gated content, and terms that explicitly forbid it. Full breakdown: is web scraping legal.

Is web scraping hard to learn? No, for static pages: the four moves plus CSS selectors, learnable in a weekend from this page and the language guide of your choice. JavaScript-heavy targets and login flows add real complexity; call it weeks of part-time practice, not months.

Can I scrape a website without coding? Yes. Spreadsheet functions handle simple tables, and no-code tools like Browse AI or ParseHub train on clicks, not code. Trade-offs: quotas, monthly fees, and workflows that break quietly on redesigns. Compared in the no-code methods guide.

What’s the difference between web scraping and web crawling? Scraping extracts fields from pages. Crawling discovers and fetches many pages, following links; that’s indexing behavior. Scrapers usually ride on top of crawlers. The measured comparison: web scraping vs web crawling.

Do I have to use Python? No. It has the biggest community, so beginners find answers fastest, but the concepts transfer: the same selectors and HTTP calls exist in Go, Node, Rust, and C# tutorials. Pick the language you’ll use for whatever comes after the data.

The next ten minutes

Open a terminal and run the two checks from Steps 2 and 3 against your real target: curl -s https://example.com/robots.txt, then view-source and search for your data. If the data’s there, the Python tutorial turns it into a scraper this evening. If it isn’t, you already know which branch you’re on. The JavaScript guide starts exactly where this page left off.