Map the site before you commit.
Magpie lists the pages in a site’s sitemap first, so you narrow the crawl to the paths you want before a single page is fetched.
Live progress, and an honest failure report.
Watch pages complete as they land. A page that fails shows up in the list with the reason it failed.
Paced to respect the site.
Crawls follow robots.txt and space out their requests, so a big crawl doesn’t hammer the site you’re reading.
Crawl a site in three steps.
- 01
Start from one URL
Give Magpie a start URL and preview the pages its sitemap lists.
- 02
Choose what to crawl
Keep the paths you want and drop the rest with a prefix or a pattern, before anything runs.
- 03
Watch it land
Pages arrive live in your local cache, and failures are reported as they happen.
Crawling, answered.
More in the docsDoes Magpie respect robots.txt?
Yes. Crawls follow robots.txt and pace their requests so the site isn’t overloaded.
How big a site can I crawl?
As big as you need — there’s no page cap. Each crawl sets its own page and depth limits (25 pages, two links deep by default), and a stopped crawl can be resumed.
What happens when a page fails?
The crawl carries on. The page is marked in the live list with the reason it failed and counted as an error, and a run that fails outright waits on the Overview with a retry.