How to Find All Pages on a Website (6 Methods That Work)

Every attempt to find all pages on a website fails in a specific direction: sitemaps undercount, crawls overcount, and search operators round to whatever Google noticed. The fix is not one perfect tool but two cheap lists diffed against each other. This guide runs the methods in the order you should try them, with runnable commands and honest failure modes for each.

Before you start

You need 15 minutes, a terminal (for the crawl methods), and no permission from the site if you’re only reading public pages and respecting robots.txt. Everything here works on someone else’s site; two methods get better if the site is yours.

One boundary first: this page is about discovering pages across a whole site. If you want the links inside one specific page, that’s a different job with a smaller tool, covered in the URL and link scraper guide.

Method 1: read the sitemap (5 minutes)

Most sites tell you where their pages are. The sitemap is a machine-readable page list, and it lives in predictable places:

• https://example.com/sitemap.xml
• https://www.example.com/sitemap.xml
• https://example.com/sitemap_index.xml (a sitemap of sitemaps, common on WordPress and Shopify)

If the direct guesses 404, read https://example.com/robots.txt instead. Sites that use a non-standard sitemap location declare it there:

User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemaps/products-en.xml

Two things a sitemap will not tell you: whether the URLs still work, and whether the list is complete. Sites ship sitemaps that lag weeks behind reality, split content across a sitemap index with entries missing, or omit sections entirely. Treat the sitemap as the site’s wish list. The next methods check it against the actual site.

Deep background on the format, if you want it, lives in the XML sitemap glossary entry.

Method 2: make a tool read every sitemap for you

Opening sitemaps by hand dies the moment the site uses a sitemap index pointing at 40 child files. Magpie’s map command reads the sitemap tree, expands the index, and prints one URL per line. Zero LLM, zero cost, sitemap-only:

magpie map https://example.com

Output is plain lines, ready to redirect into a file:

https://example.com/
https://example.com/pricing
https://example.com/blog
https://example.com/blog/crawl-budget
...

(abridged — real sites return hundreds of lines)

For scripts, take JSON and check the truncated flag. map stops at a 25-second budget; a huge or half-dead sitemap tree comes back partial and marked, which is exactly what you want to know before trusting the list:

magpie map https://example.com --format json | jq '.urls | length, .truncated'

Method 3: crawl all pages of a website (the sitemap lies sometimes)

A crawl ignores the site’s claims and follows links from page to page. It finds what the sitemap forgot, and it also finds what the sitemap never should have listed. Run a scoped crawl with a page cap so a calendar-archive loop can’t run for hours:

magpie crawl https://example.com --corpus --max-pages 500 --out corpus.jsonl

Magpie respects robots.txt and auto-throttles per host, and a run is resumable, so a dead connection at page 400 costs you a resume flag, not a restart. Content comes back as markdown and JSON per page, which matters if the page list is step one of a bigger job.

No-code alternative: Screaming Frog’s desktop crawler is the standard in SEO shops, and its free tier covers 500 URLs per crawl, enough for small and mid-size sites. Above that it’s a paid licence. If you already live in Python, Scrapy or Crawlee do the same job with more code; the web crawling tools comparison ranks those.

The honest limit of crawling: link-driven discovery only reaches pages something links to. Orphan pages (nothing points at them) will not appear in a crawl, a sitemap that omits them, or a search operator. If orphans matter, and they usually only matter on your own site, get the list from the CMS or server config instead of from the outside.

Method 4: find all URLs on a website that has no sitemap

Some sites ship no sitemap at all, or one so stale it’s a museum piece. Then the crawl is the primary method, not the check:

1. Start the crawl at the page most likely to link to everything, usually the homepage or an HTML archive page.
2. Raise the page cap for large sites (--max-pages 5000), and use path scoping when the site mixes content you don’t want: magpie crawl https://example.com/blog --path-prefix /blog --max-pages 5000 keeps the run inside one section.
3. Diff the result against magpie map. On no-sitemap sites, map returns little or nothing, which at least confirms the diagnosis.

This is also where a generic “website page finder” tool stops being enough. Point-and-click finders scrape the links off pages you feed them; discovery on a sitemap-less site needs a real crawler walking the graph, because the whole job is following links across pages you haven’t seen yet.

Method 5: quick estimates (search operators and Search Console)

Two fast, free, unreliable methods worth knowing so you don’t mistake them for answers.

site:example.com in Google or Bing shows what the search engine has indexed. Good for a ten-second sanity check (“does the site have a blog?”), bad as an inventory: the index lags, applies its own deduplication, and caps what it shows at a few hundred results no matter the claimed count.

If the site is yours, Google Search Console’s Pages report is the ground truth of what Google found, including submitted-versus-indexed gaps. Bing Webmaster Tools does the same for Bing. Neither shows pages that exist but were never linked or submitted, so the CMS route still beats them for completeness on your own properties.

Method 6: validate the list before you trust it

Whatever produced your list, one step makes it trustworthy: take the union of methods 2 and 3 and check it.

• Deduplicate. Sitemaps list both http:// and https://, with and without trailing slashes. Normalize before counting or you’ll report 3,000 pages on a 1,400-page site.
• Check status codes. A sitemap full of 301s and 404s is common. A single-URL fetch per candidate is enough: magpie scrape https://example.com/some-url returns the page or an error, and one failed fetch doesn’t eat the budget a crawl would spend stumbling through the same list.
• Diff map against crawl. URLs in the crawl but not the sitemap are the sitemap’s gaps. URLs in the sitemap but not the crawl are either orphans, robots-blocked sections, or dead links. Each of those is a finding, not noise.

The diff is the actual answer to “how many pages does this site have”: every page reachable, listed, or both. Any single method is a subset wearing a confident file format.

Which method should you use to find all pages on a website

• Someone else’s site, quick look: Method 1, then Method 2 if there’s a sitemap index.
• Complete inventory of a site you don’t control: Method 2 + Method 3, diffed per Method 6.
• No sitemap: Method 4, and lower your expectations for orphans.
• Your own site: Search Console plus the CMS’s own page list, then Methods 2 and 3 as the outside view.
• Recurring monitoring: rerun Method 2 on a schedule and diff against last week’s list. Magpie’s watch mode (URL monitoring with webhooks) covers the new-page case for single pages; for the whole-site list, a cron job plus a diff file does it.

To get all pages of a website into a spreadsheet from here, the crawl output is JSONL, one page per line, and jq -r '.url' corpus.jsonl > urls.txt is the whole export step.

FAQ

How do I export all URLs from a website to CSV? Sitemap route: magpie map gives you a plain URL list you can paste anywhere. Crawl route: the output file is JSONL; one jq line converts it to a URL column, and any spreadsheet opens a CSV from there. Screaming Frog exports CSV natively.

How do I list all pages on a website for free? Read sitemap.xml or robots.txt by hand, or run magpie map from the terminal. Both cost nothing. The free crawl tiers (Screaming Frog, 500 URLs) cover smaller sites; open-source crawlers have no cap at all.

Is finding pages on a website legal? Reading publicly accessible pages and sitemaps is generally legal; the usual constraints (copyright, personal data, terms of service) apply to what you do next. Respect robots.txt and rate limits. The details are in the web scraping legality guide.

Does a sitemap list every page? No. Sitemaps are manually or semi-automatically generated and routinely omit sections, lag behind new pages, and include dead URLs. That’s why the two-list diff in Method 6 exists: the sitemap is a claim, the crawl is evidence, and the answer is the union.

Run Method 1 right now on any site you’re curious about: open /robots.txt, follow the Sitemap: line, and count what comes back. Then run magpie map on the same site and compare the two numbers. The gap between them is the part of the site nobody’s list tells the truth about, and it’s the reason one method is never enough.