CSS Selectors for Web Scraping

CSS selectors are patterns that address elements in an HTML document. .quote means “every element whose class list contains quote”. span.text narrows that to spans. Every web scraping stack reads the same notation, which is exactly why the selector is the part of your scraper you rewrite most: the site changes and the selector is where it shows. On 2026-10-16 I fetched quotes.toscrape.com and ran selector after selector against it, including a test where one inserted banner div kills a selector that was matching seconds earlier. This page is the vocabulary, the engine differences, and the results.

What a selector actually addresses

A selector is an address into the DOM, the tree a parser builds from the HTML bytes. It points at nodes the way a path points at files, except a good selector names the thing by what it is rather than where it happens to sit.

The notation is shared across the tools you already argue about: the browser’s querySelectorAll(), BeautifulSoup’s .select(), Scrapy’s .css(), Cheerio, jsoup, Playwright locators. Learn it once and it transfers. The differences live at the edges, and I’ll show those measured further down.

The CSS selectors that do almost all of the work

Counts below come from https://quotes.toscrape.com/, fetched 2026-10-16 with curl and parsed with BeautifulSoup 4.14.3. The page is a scraping sandbox, so counts are stable by design; a real site will drift, and that drift is the whole subject of this page.

Selector	What it matches	Count on that page	Reach for it when
.quote	elements with class quote	10	the site gives you a semantic hook, take it
span.text	spans with class text	10	same, plus the tag to cut false positives
[itemprop="text"]	elements carrying the attribute	10	schema attributes exist and nobody redesigns them casually
a[href*="/tag/"]	links whose href contains /tag/	40	URL structure is more durable than markup
li.next a	a anywhere inside li.next	1	pagination walking; that href is your crawler’s next URL
.quote:nth-child(2n) .text	even-positioned quotes	5	slicing rows, never as a primary hook
.quote + .quote	a quote immediately following one	9	confirming structure, not extracting
div.quote:has(small.author)	quotes that contain an author line	10	filtering containers by their contents

Eight selectors cover maybe 95% of scraping work. The craft is not knowing more syntax; it’s choosing among these so the choice survives the next deploy.

Combinators, briefly

Four connectors do the joining:

• Descendant (space): .quote small.author matches the author anywhere inside a quote. 10 hits.
• Child (>): div.quote > span matches only direct children. Each quote holds two spans, so this returns 20, not the 10 you’d guess from counting quotes.
• Adjacent sibling (+): .quote + .quote returns 9, every quote that follows one.
• General sibling (~): same idea without the adjacency requirement.

The child combinator is the one worth a deliberate habit. Descendant selectors over-match on nested layouts; > forces you to know the shape you’re claiming.

Pseudo-classes worth knowing, and one to skip

:first-of-type, :nth-child(2n+1), and :not() are old, everywhere, and boring in the good way. The recent one is :has(), which lets a selector make claims about children and siblings: div.quote:has(small.author) is real CSS now. It shipped in Safari 15.4 in March 2022 and Chrome 105 that August; Firefox held out until 121 in December 2023, so anything newer than that supports it.

Then there’s :contains("text"), which you will meet in tutorials and should treat as a dialect, not CSS. It was never in the browser spec. jQuery has it, Cheerio kept it, and BeautifulSoup’s engine supported it for years; as of soupsieve 2.8.3 it emits a FutureWarning pointing you at the vendor-specific :-soup-contains. I watched that warning fire during the runs for this page. Matching by text feels natural and breaks the moment an editor fixes a typo, which is why no engine wants to promise it.

The fragility test: one banner kills a selector

Here is the experiment. Take three selectors that all currently work on the quotes page, including the positional monstrosity in the shape that browser DevTools’ “Copy selector” produces. Then simulate the most common redesign event there is: one promo banner div inserted as the first child of the page container.

Selector	Before	After one banner div
body > div.container > div:nth-child(2) > div.col-md-8 > div.quote:nth-child(1) > span.text	1	0
div.row > div.col-md-8 > div.quote	10	10
.quote	10	10

The positional path describes the page’s accident: that the row happens to be the second child of the container. Insert anything above it and every index shifts, silently. The class hooks describe the page’s intent, and intent is the thing redesigns preserve.

Four rules fall out of runs like this:

• Hook on names, not positions. Classes, ids, data-* and schema attributes. Position is a last resort for ties.
• Assert a count. After every fetch, check len(soup.select(sel)) against the expected number and raise on zero. A selector returning 0 rows is a redesign; 300 rows on a page that had 10 is a duplicated hook. Both deserve a loud failure, not an empty CSV.
• Treat DevTools’ “Copy selector” as a starting point, never the answer. It’s built for uniqueness on today’s DOM, and it pays for that in brittleness.
• Keep the HTML you tested against. When a selector dies next month, the diff between old and new markup is the diagnosis. Without it you’re guessing.

Same selector, three engines

Shared notation, different dialects. I ran the interesting cases through the two Python engines and checked them against browser behavior:

Selector	Browser (evergreen, 2026)	BeautifulSoup 4.14 (soupsieve 2.8.3)	lxml + cssselect 1.6.0
:has()	yes, since Firefox 121	yes	yes
:contains("x")	no	deprecated, warns	yes
:nth-child(2n)	yes	yes	yes
:has-text(), :visible	no	no	no; Playwright defines its own

Two footnotes matter. Scrapy’s .css() goes through parsel into cssselect, so the cssselect column describes Scrapy too. And testing a selector in the browser console before pasting it into your parser is only half a test: the console supports what the parser may not and vice versa. Run it in the engine that ships.

FAQ

CSS selectors or XPath? CSS for most work: shorter, readable, and everything above applies. XPath pays when you must walk upward (parent of a match) or match on raw text, which CSS never promised. Both break equally on redesigns. The syntax side by side: /glossary/xpath.

Why does my selector work in DevTools but return nothing in my scraper? The page you inspected was built by JavaScript after load. Your scraper read the raw HTML, where the node does not exist yet. Confirm with view-source, then you need rendering: /glossary/javascript-rendering.

How do I find a good selector? Inspect in DevTools, but pick your own hook: a class, id, or attribute that names the thing. Skip the copied positional path. Then run it in your actual parser with a count assertion before writing any parsing code; the BeautifulSoup tutorial shows the loop.

How do I test CSS selectors quickly? Save the page once, iterate offline:

curl -s https://quotes.toscrape.com/ -o page.html
python3 -c "from bs4 import BeautifulSoup; print(len(BeautifulSoup(open('page.html').read(),'html.parser').select('.quote')))"

Fast, and no site rate-limits your debugging loop.

A note on tooling

One step removed from all this: my lab’s scraper, Magpie, accepts include/exclude CSS selectors at its cleaned-HTML stage and keeps a per-site selector cache that re-synthesizes selectors when match rates fall to zero. That last part matters because everything above decays. It’s a local CLI/desktop tool, not a hosted API, so I’ll stop there; this page stands on its own.

Pick your primary selector, write the count assertion first, and only then write the parser. The assertion is what tells you the site changed. The parser just believes whatever it’s told.