BeautifulSoup Web Scraping Tutorial (2026)

Most BeautifulSoup tutorials end at soup.find_all() and a demo page. The scraper you actually need has to fetch, handle pagination, survive pages that load data with JavaScript, and save output someone can use. That whole path is this BeautifulSoup web scraping tutorial, and every code block in it ran against quotes.toscrape.com on 2026-10-16 with BeautifulSoup 4.14.3 on Python 3.12.3. The outputs you see are real, not sketched.

How BeautifulSoup web scraping works

BeautifulSoup is a parser, not a fetcher. You hand it an HTML string and it builds a tree you can query with Python: “give me every div with class quote”, “give me the text inside this tag”. It never touches the network. The fetching is requests’ job, and knowing where the boundary sits is the difference between a working script and an hour of confusion.

Two things to get right at install time:

pip install beautifulsoup4 requests lxml
python -c "import bs4; print(bs4.__version__)"

4.14.3

The package is beautifulsoup4. The one called beautifulsoup is the dead 3.x line from over a decade ago; installing it by accident is still a common way to waste twenty minutes.

BeautifulSoup also needs a parser to sit on top of. The choices:

Parser	Depends on	Notes
html.parser	Nothing (stdlib)	Always available. Fine for small pages.
lxml	C library libxml2	Fastest. Use this unless you can’t install C deps.
html5lib	Pure Python	Parses like a browser, lenient with broken HTML. Slowest.

On the 11 KB test page, I measured html.parser at 7.65 ms per parse and lxml at 6.12 ms (100-run averages). On small pages the difference is noise. It widens with document size, which is why production code defaults to lxml.

Step 1: Fetch a page and parse it

One prerequisite: you need Python 3.7 or newer and the installs above. That’s it. No accounts, no API keys.

Fetch the page with requests, parse it with lxml underneath:

import requests
from bs4 import BeautifulSoup

r = requests.get(
    "https://quotes.toscrape.com/",
    headers={"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) MyScraper/1.0"},
)
soup = BeautifulSoup(r.text, "lxml")

print(r.status_code, soup.title.string)
print(len(soup.select("div.quote")), "quotes on the page")

What you should see (timed run: 0.98 s):

200 Quotes to Scrape
10 quotes on the page

Failure case: soup.title.string comes back None on pages where the title tag has nested elements, and .string only works when a tag has exactly one child. Use .get_text() when in doubt.

Step 2: Take one quote apart

Before looping over a page, pull one element apart and look at it. This is the step every tutorial skips and every debugging session needs.

quote = soup.select_one("div.quote")

text = quote.select_one("span.text").get_text()
author = quote.select_one("small.author").get_text()
tags = [a.get_text() for a in quote.select("a.tag")]

print(repr(text))
print(author, tags)

'“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'
Albert Einstein ['change', 'deep-thoughts', 'thinking', 'world']

Two details worth catching now. The curly quote marks (“”) are part of the text; strip them with .strip('“”') or you’ll ship them into your CSV and wonder about them later. And if you want an attribute instead of text, it’s square brackets: quote.select_one("span.text")["class"] returns the class list.

Step 3: find_all or select — pick one and know the other

BeautifulSoup gives you two query APIs. find_all() takes keyword arguments; select() takes CSS selectors. They overlap almost completely:

You want	find_all	select
Tags with a class	find_all("div", class_="quote")	select("div.quote")
Tag by id	find(id="pagination")	select("#pagination")
Nested match	find_all("span", class_="text")	select("div.quote span.text")
First match	find("div", class_="quote")	select_one("div.quote")

The overlap ends at advanced CSS. BeautifulSoup’s select() supports most selector syntax (attribute matches, :nth-child, sibling combinators) but not :contains() or :has(). Searching by text content means find_all(string=re.compile("...")) with the re module, not CSS. If you need a refresher on selector syntax itself, the CSS selectors for web scraping glossary page covers it line by line.

Pick select() if you already know CSS; pick find_all() if your filters are about text and attributes more than structure. Mixing both in one file is fine. Mixing both in one line is how bugs hide.

Step 4: Links, relative URLs, and urljoin

Every href you scrape is usually a relative path. requests will not resolve it for you; urljoin from the standard library does:

from urllib.parse import urljoin

BASE = "https://quotes.toscrape.com"
links = set()

for a in soup.select("div.quote span a[href^='/author/']"):
    links.add(urljoin(BASE, a["href"]))

for url in sorted(links)[:3]:
    print(url)

https://quotes.toscrape.com/author/Albert-Einstein
https://quotes.toscrape.com/author/Andre-Gide
https://quotes.toscrape.com/author/Eleanor-Roosevelt

Page 1 holds 10 author links but only 8 unique URLs, because Einstein shows up twice. That’s the third detail: dedupe with a set before you crawl anything, or you’ll request the same page twice per run and get to know 429 errors personally.

Step 5: Paginate without being a jerk

quotes.toscrape.com paginates with /page/N/ URLs, so pagination is a loop. Two politeness rules apply the moment you write one: sleep between requests, and stop when the site tells you to.

import time

rows = []
for page in range(1, 6):
    resp = requests.get(f"{BASE}/page/{page}/", timeout=15,
                        headers={"User-Agent": "MyScraper/1.0"})
    s = BeautifulSoup(resp.text, "lxml")
    for q in s.select("div.quote"):
        rows.append({
            "text": q.select_one("span.text").get_text().strip("“”"),
            "author": q.select_one("small.author").get_text(),
            "tags": ",".join(t.get_text() for t in q.select("a.tag")),
        })
    time.sleep(1)

print(len(rows), "quotes")

50 quotes

Five pages, fifty quotes, 10.5 seconds wall time including the sleeps. One second between requests is a floor, not a ceiling; for a site you don’t operate, check its robots.txt first. (This sandbox serves none; I verified, it 404s, which is exactly why it exists for practice.) On real targets, a 429 response means “slow down or stop”, and the rate limiting and HTTP 429 page explains the headers that tell you by how much. Retrying harder is how an IP gets blocked.

Step 6: Save the output — CSV or JSON

Fifty quotes in a Python list helps nobody. Write them somewhere:

import csv

with open("quotes.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["text", "author", "tags"])
    writer.writeheader()
    writer.writerows(rows)

That produced an 8,315-byte file, 50 data rows plus the header:

text,author,tags
The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.,Albert Einstein,change,deep-thoughts,thinking
"It is our choices, Harry, that show what we truly are, far more than our abilities.",J.K. Rowling,"abilities,choices"

The csv module quotes fields containing commas automatically, which is why the tags column survives. Set encoding="utf-8" explicitly or Windows will hand you a charmap error the first time a quote contains a character code page 1252 doesn’t know. For feeding other programs rather than spreadsheets, json.dump(rows, f, ensure_ascii=False, indent=2) is the better shape.

Where BeautifulSoup stops working

Same library, same site, different URL:

r = requests.get("https://quotes.toscrape.com/scroll")
soup = BeautifulSoup(r.text, "lxml")
print(r.status_code, len(r.text), "bytes,", len(soup.select("div.quote")), "quotes")

200 2671 bytes, 0 quotes

HTTP 200, 2,671 bytes, zero quotes. The page’s HTML contains an empty <div class="quotes"> and a “Loading…” placeholder; JavaScript fetches the data after load, and BeautifulSoup never sees it. This is the most common “my scraper returns nothing” report, and it has two fixes:

1. Find the API endpoint. Open browser DevTools, watch the Network tab, reload, and look at the XHR request that carries the data. The /scroll page pulls JSON from a quotes API; hitting that endpoint directly is faster and more stable than any parsing. Most “scraping” jobs quietly end as API clients.
2. Drive a real browser. When there’s no clean endpoint, you need Selenium or Playwright to render the page first, then hand the rendered HTML to BeautifulSoup. The trade-offs live in the BeautifulSoup or Selenium comparison. Check a site’s terms before automating it; browser automation on sites that forbid it is a policy violation, not a technicality.

When to graduate from BeautifulSoup

BeautifulSoup has no memory. Every run re-fetches, re-parses, and hopes your selectors still match. Three signals say it’s time for something more:

• Selector rot. The site redesigns, .quote becomes .quote-item, and your script returns zero rows with no error. This is normal and permanent; plan for it.
• Scale. Past a few thousand pages you want concurrency, retries, and resumable crawls, and building those on requests is a project of its own. That’s the point where Scrapy earns its learning curve.
• Boilerplate. If the goal is “main content of these 200 pages” rather than “the price field on this page”, per-site selectors are the wrong tool. Content extractors and LLM-based extraction handle it without site-specific code, which is the selectors-versus-LLM trade-off in one link.

For the boilerplate case specifically, I often skip writing parsers entirely and run the pages through Magpie, a local Go CLI that fetches, cleans, and extracts in one pass. It’s the right tool when the job is whole-page content; when you need one specific field from a specific layout, BeautifulSoup’s per-element control still wins.

The Python web scraping overview puts BS4 in the wider toolbox, and the tutorials hub has sibling walkthroughs for Scrapy, Selenium, and Playwright.

FAQ

Is BeautifulSoup still worth learning in 2026?

Yes. It’s actively maintained (4.15.0 is current as of October 2026), has the gentlest learning curve of any HTML parser I’ve used, and remains the default answer to “parse this HTML in Python”. JavaScript-rendered targets are the limit, not a reason to skip it.

Does BeautifulSoup work with JavaScript-heavy sites?

Not by itself. It only parses the HTML string you give it, so content loaded by scripts is invisible. Either hit the underlying API directly or render the page with Playwright/Selenium first, then parse the result.

BeautifulSoup or Scrapy?

One page or one site, quick jobs: BeautifulSoup plus requests. Multi-site, production, scheduled crawls with pipelines: Scrapy. The full comparison of both is linked in the graduation section above.

Is web scraping with BeautifulSoup legal?

Public data generally is, with caveats: terms of service can forbid it, login-walled and personal data raise GDPR-style issues, and copyright limits what you republish. Jurisdiction matters. This is orientation, not legal advice; ask a lawyer for a real answer on a real project.

Start here

The working skeleton from steps 1 and 2, minus everything you’ll replace:

import requests
from bs4 import BeautifulSoup

soup = BeautifulSoup(
    requests.get("https://quotes.toscrape.com/", timeout=15).text, "lxml"
)
for q in soup.select("div.quote"):
    print(q.select_one("span.text").get_text().strip("“”"))

Run it, confirm ten quotes, then swap in your target and your selector. Everything else in this tutorial is that loop wearing a raincoat.