XPath for Web Scraping: Syntax & Examples

XPath is the query language for selecting nodes out of an HTML or XML document, and in web scraping it is the selector for the jobs CSS cannot do: match an element’s text, walk backward through the tree, or filter a node-set by position. Most XPath web scraping tutorials stop at toy examples against clean XML. This one runs its expressions against a live page and prints the counts, then walks through the three gotchas that break scrapers in production.

How XPath sees a page

A browser or parser turns the document into a tree, and an XPath expression is a path through that tree with optional filters in square brackets. / means direct child, // means any descendant, [@class="x"] filters by attribute, and each expression evaluates to a node-set, which your code sees as a list.

The version matters more than people expect. The W3C published XPath 1.0 in November 1999, and it is still what browsers (document.evaluate), Scrapy’s parsel, and Python’s lxml (libxml2) evaluate. XPath 3.1 arrived in 2017 with functions like lower-case(), but it lives in the XSLT and Saxon world. Copy a slick 3.1 expression from a forum post and it fails on the page you actually scrape, because the engine only speaks 1.0.

The syntax, in one table

Piece	Meaning	Example
/	direct child only	//div/span
//	any descendant	//div//span
tag	element by name	//small
*	any element	//*[@class="author"]
@attr	an attribute	//a/@href
[n]	nth match, 1-indexed	//li[2]
[last()]	the last match	//li[last()]
[@a="v"]	exact attribute match	//a[@class="tag"]
[contains(@a,"v")]	substring attribute match	//a[contains(@class,"tag")]
text()	direct child text nodes	//small/text()
. / string(.)	the whole subtree as one string	//div[contains(string(.),"x")]

XPath web scraping: ten expressions that do real work

I ran these against the quotes.toscrape.com homepage on 2026-10-16: 11,064 bytes of static HTML, parsed with lxml 6.1.3 (libxml2 2.14.6). The page holds 10 quote cards, each with a quote <span>, a byline <span> wrapping the author, and a tags block.

#	Expression	What it selects	Live result
1	//div[@class="quote"]	quote cards	10 elements
2	//span[@class="text"]/text()	the quote strings	10 strings
3	//small[@class="author"]/text()	author names	10, first “Albert Einstein”
4	//a[contains(@class,"tag")]	tag links	40 elements
5	//li[@class="next"]/a/@href	pagination link	/page/2/
6	(//span[@class="text"])[2]/text()	the second quote	the J.K. Rowling quote
7	//div[@class="quote"][position() mod 2 = 1]	odd-numbered cards	5 elements
8	//div[@class="quote"][last()]//small/text()	last card’s author	“Steve Martin”
9	//span[@class="text"]/following-sibling::span[1]//small/text()	author under each quote	10 names
10	//*[@class="author"]	authors, tag name unknown	10 elements

Two of these rows carry a hidden lesson. Row 6 wraps the whole path in parentheses: (//x)[2] picks the second match in the document, while //x[2] picks elements that are the second such child of their parent. On this page //span[@class="text"][2] returns zero, because each card contains exactly one .text span. And in row 9, following-sibling::span[1] means “the nearest following sibling span”, not “the second one”. The index inside an axis step counts from the current node, which is why the expression lands on the byline every time.

The three gotchas that eat the afternoon

/ and // are not interchangeable. Here is the real markup of a quote card, abridged:

<div class="quote">
  <span class="text">…</span>
  <span>by <small class="author">Albert Einstein</small>
    <a href="/author/Albert-Einstein">(about)</a>
  </span>
  <div class="tags">…</div>
</div>

The <small> is a grandchild, not a child. On the live page, //div[@class="quote"][1]/small/text() returns nothing and //div[@class="quote"][1]//small/text() returns “Albert Einstein”. Same tree, one slash of difference. The DevTools “Copy XPath” command has the opposite problem: it hands you absolute paths like /html/body/div[2]/div/div[1] that snap on the first layout change. Rebuild anything it gives you as a relative path anchored on something stable.

contains(text(), …) reads the first text node only. text() selects direct child text nodes, and contains() tests just the first of them. Take <p>Hello <b>bold</b> world</p>: the <p> has two direct text nodes, "Hello " and " world", so //p[contains(text(),"world")] matches nothing even though the paragraph plainly contains the word. //p[contains(string(.),"world")] matches one, because string(.) flattens the whole subtree. Any element that wraps other elements needs string(.) or normalize-space(.).

XPath 1.0 has no lower-case(). All matching is case-sensitive. On the same page, //span[contains(text(),"WORLD")] returns zero results while the quote text contains “world”. The 1.0 fix is translate():

//span[contains(translate(., "WORLD", "world"), "world")]

That returns one element. Ugly, correct, and the only portable option, because browsers never shipped anything newer than 1.0.

The small print, quickly: indexes start at 1, so //div[0] matches nothing. @class is one string, so class="btn tag" fails @class="tag" and you want contains(@class,"tag"). A leading // scans the whole tree, so on big pages scope first (//div[@id="results"]//h2). And when debugging, the DevTools console takes $x('//h1') directly in Chrome, Firefox, and Safari; $x('//h1')[0].innerText prints what matched.

XPath vs CSS selectors

CSS selectors handle downward traversal and read better, which is why every scraper library defaults to them. These days :has() and the +/~ combinators even close part of the upward and sideways gap. What CSS still cannot do is match on text content or walk backward (preceding::). “The <dd> that follows the <dt> labeled Price” is pure XPath: //dt[text()="Price"]/following-sibling::dd[1]. The full contrast lives in CSS selectors, and if you are deciding whether to write selectors at all, LLM extraction vs CSS selectors has that tradeoff measured.

A note from the site you’re on, since this page doesn’t sell anything: Magpie, the scraper this blog belongs to, skips hand-written selectors for most extraction. Schema-driven extraction routes an LLM at the cleaned page, 27 zero-LLM verticals (JSON-LD, job postings, RSS) cover common site types, and where selectors are used, a self-healing cache re-synthesizes the ones whose null rate climbs after a redesign. The XPath above is for the cases you drive yourself.

FAQ

Is XPath better than CSS selectors for web scraping? Neither wins overall. CSS reads cleaner for simple downward paths and is the default in every library. XPath earns its keep for text matching, backward traversal, and positional filters. Most production scrapers carry both and pick per site.

What XPath version do browsers support? XPath 1.0, exposed through document.evaluate, in Chrome, Firefox, and Safari. Versions 2.0 through 3.1 never shipped in browsers; they live in XSLT processors such as Saxon. DevTools’ $x(), Scrapy, and lxml all evaluate 1.0.

How do I test an XPath expression in Chrome? Open DevTools and run $x('//h1') in the console. You get an array of matched nodes; $x('//h1')[0].innerText prints the first one. The Elements panel’s right-click “Copy XPath” also works, but the path it builds is absolute and index-heavy, so it breaks on redesigns.

Do Selenium and Playwright support XPath? Yes. Selenium takes driver.find_element(By.XPATH, "//button[text()='Next']"). Playwright accepts a xpath= prefix: page.locator("xpath=//button[text()='Next']"). Both evaluate against XPath 1.0, so the same portability limits apply.

What’s the difference between text() and string()? text() selects an element’s direct child text nodes as a node-set. string(.) flattens the whole subtree, descendants included, into one string. contains(text(),"x") only ever inspects the first text node, so mixed-content elements need contains(string(.),"x").

Where to take this next

Run one adapted expression against a page you already scrape and count what comes back before you trust it; a zero that should be ten is the bug, and it is almost always a / where // belonged. For markup that stays stable while you learn, the practice targets in websites to practice web scraping beat pointing at production sites, and the Scrapy tutorial is where these expressions start running at scale.