Web Scraping Pagination: How to Scrape Multiple Pages

The hockey-stats sandbox at scrapethissite.com has 582 rows of team data split across 24 pages. I asked it for page 26 and it served page 1, byte for byte, with HTTP 200 and no warning. Five minutes earlier I had asked for ?page=3 and gotten page 1 back three times in a row, across three different user agents and a cookie jar, because the real parameter is page_num and the server ignores wrong ones in silence. Web scraping pagination is full of traps exactly like these: the requests all return 200, the data looks plausible, and your dataset quietly contains the same first page over and over.

This guide covers the four pagination patterns you will meet, the stop conditions that do not lie, the one-request trick that skips looping entirely, and the crawler settings that walk a paginated section for you. Every number comes from runs against two live sandboxes today, 2026-10-16, on a home connection.

Before you start

A browser with developer tools (to read the site’s own pagination markup), curl or Python for the requests, and a scraper that can crawl with URL filters. The commands here use magpie. Two sandbox sites carry the whole guide: quotes.toscrape.com (path-based pagination) and scrapethissite.com (parameter-based pagination). Both exist to be scraped; their robots.txt allows it.

The four web scraping pagination patterns

Pattern	What it looks like	How to walk it
Numbered URL paths	/page/2/, /page/3/	Generate the URLs, fetch each (Step 1)
Page parameters	?page_num=3, ?p=3, ?offset=50	Same, but read the real name from the Next link (Steps 1-2)
Big page size	?per_page=1000 alongside the page param	One request instead of N (Step 3)
No URLs at all	A Load more button, or content appended on scroll	Not pagination in the URL sense; see infinite scroll

Cursors and tokens (an opaque string the server hands you with each response) behave like the parameter pattern: you cannot guess them, you must take each one from the previous response. The trick from Step 2, using the site’s own links instead of guessing, covers those too.

Step 1: Read the site’s own Next link

Pagination markup is a spec written by the site’s developers, and it is the first thing to read. Every paginated page on scrapethissite.com carries this in its HTML:

<a href="/pages/forms/?page_num=1" aria-label="Next">

That single line answers the two questions every loop depends on: the parameter is page_num (not page, not p), and the next page is one more than the current. Follow the Next links and you never guess a parameter name, never overwrite a working URL scheme, and never loop against a server that ignores you.

The alternative, guessing ?page=2 and seeing a 200 with plausible-looking content, is how I ended up with five identical copies of page 1. The server accepted my wrong parameter, returned its default page, and said nothing. A wrong URL path 404s loudly; a wrong query parameter 200s quietly. Check the Next link before you trust any response.

Step 2: Pick a stop condition that cannot lie

Your loop needs to know when to stop, and the HTTP status code is the least reliable signal on the board. I probed both sandboxes past their last page:

curl -s -o /dev/null -w "%{http_code}\n" https://quotes.toscrape.com/page/11/
# 200   (3,050 bytes, zero quotes in it)
curl -s "https://www.scrapethissite.com/pages/forms/?page_num=25" | grep -c 'class="team"'
# 0

Same status code, three different truths. On quotes.toscrape.com, page 11 exists as a URL and returns an empty page: a 200 with no content. On scrapethissite, the honest parameter past the end returns the page with zero data rows. And with the wrong parameter name, every page past the end returns page 1 in full, with a 200 and a smile. Three stop conditions, in order of reliability:

1. The data itself: fetch, count the records, stop when the count is zero. Works on every pattern above.
2. The site’s own navigation: stop when the Next link disappears or the last page has none. Equally reliable, and it saves you one empty request.
3. Status codes: never alone. A 200 past the end is the norm, not the exception.

A dataset with silent duplicates is worse than a short one: it poisons every average built on top of it. Count your records at the end and compare against the site’s own totals if it publishes any.

Step 3: Try to replace the loop with one request

Before looping 24 times, ask whether the server will hand you everything at once. Many paginated endpoints carry a page-size parameter (per_page, page_size, limit, results_per_page) and most accept values far above the default. The hockey sandbox did:

curl -s -o /dev/null -w "%{size_download}B, %{time_total}s\n" \
  "https://www.scrapethissite.com/pages/forms/?page_num=1&per_page=1000"
# 716495B, 2.1 s - all 582 teams, one request

One request of 716 KB against 24 requests taking 10.7 seconds in a plain loop. The dataset arrived complete in 2.1 seconds. This is the entire trick of the hidden-API method applied to pagination: the page shows you 25 rows because a designer chose 25, not because the server can only count to 25.

Two caveats from the same run. First, servers cap the parameter: per_page values of 600, 700, 1000, and 2000 all returned exactly the same 582 rows, so the ceiling existed and 1000 was safely above it: probe with a large value and check the row count stabilizes. Second, a big page size is still one request to the same server; if you do not need everything, the default page size with a polite loop is the kinder option.

Step 4: Let a crawler walk the paginated section

For repeatable jobs, hand the walking to a crawler with a URL filter. Magpie’s crawl command takes glob include patterns, and the paginated chain on quotes.toscrape.com is /page/N/ paths:

magpie crawl https://quotes.toscrape.com/page/1/ \
  --include '**/page/**' --max-pages 10 --corpus --out pages.jsonl
# pages_ok=10 pages_err=0 records=10   (9.0 s)

Ten pages, zero errors, 1,332 words of cleaned text across the set, all in one JSONL file. Look closely at what the glob caught, though: along with /page/2/ and /page/3/, the run picked up /tag/choices/page/1/: the site paginates its tag pages with the same path shape, and ** crosses directories. That may be exactly what you want; if it is not, the fix is a tighter pattern or a post-filter on the output. Crawl output is cheap to inspect; check the URLs you actually got before you trust the set.

Crawler etiquette on paginated sites, in one breath: magpie respects robots.txt and rate-limits per host by default (one request per second), and the full politeness guide covers the rest. The same-host default keeps the crawl from wandering off the site. And magpie map is worth knowing about: it discovers URLs from a site’s sitemap.xml, which is often the fastest way to enumerate paginated or archived pages, when the site has a sitemap. This sandbox does not, and the command says so plainly instead of guessing.

Make it stick

The working order of operations: read the Next link for the real parameter name, probe one page past the end to learn the stop behavior, probe for a page-size parameter that collapses the loop, and only then write the loop, with a record count rather than a status code as the stop signal. Cursor-based feeds (each response carries the token for the next) and POST-based pagination follow the same discipline; you just read the token instead of incrementing a number.

When the pagination turns out to be a scroll trigger instead of a URL scheme, that is a different guide: how to scrape infinite scroll pages: same sandboxes, measured the same way. And when the data turns out to ride a JSON feed rather than the HTML, the loop disappears entirely into a few direct requests.

FAQ

How do I find the last page of a paginated site? Ask one page past where you think the end is, and read the data, not the status code. On the sandboxes here, the page past the end returned either an empty page with HTTP 200 or zero data rows. Alternatively, the last page simply has no Next link; check the pagination markup.

Why does my scraper keep getting the same page? You are using the wrong page parameter name, and the server is ignoring it silently. ?page=3 returned page 1 six times on a sandbox in this guide; the real parameter was page_num. Copy the parameter name from the site’s own Next link instead of guessing.

Is it better to follow Next links or generate page URLs? Generated URLs (/page/2/, ?page_num=2) parallelize and are simpler to reason about, but only after you have confirmed the pattern from the site’s markup. Following Next links is more resilient against odd numbering and works on cursor-based feeds. Start by generating; fall back to following.

What is a good request rate for paginated scraping? One request per second per host is a defensible default, and magpie’s crawler applies it automatically. The 24-page walk in this guide ran at that rate. Watch for 429 or 5xx responses and back off when they appear: a site telling you to slow down means exactly that.

Does pagination hurt SEO? Handled well, no: paginated series with real, crawlable URLs are how large sites stay indexable. The risk cases are infinite scroll with no URL per page, and pagination chains that return duplicated content (your scraper’s problem is the search engine’s problem too).