Crawl
Follow links from one seed URL and collect every page as records you can export. Crawls run in the background and survive a stop.
On this page
Start a crawl
- Open Crawl. Pick Mode: Corpus (no LLM) collects pages as Markdown records and bills nothing; Extracted (schema) applies a schema to every page.
- In Extracted mode, pick a schema from the library (saved from the Scrape screen’s schema panel).
- Enter the Seed URL — where the frontier starts. The sitemap feeds the queue first.
- Set Max pages (starts at 25) and Max depth (starts at 2; 0 is the seed page only). Leave Same host on.
- Press Start crawl.
Preview seed URLs runs the sitemap read without crawling: the URL list, a truncation badge when the sitemap is large, and a copy button. Author your scope globs against this list before committing a crawl to them.
Corpus mode needs no key and no account. One crawl counts as one run against the trial; failed and stopped runs do not count.
Scope
| Option | Default | What it does |
|---|---|---|
| Same host | on | Follow only same-host links |
| Path prefix | empty | Only URLs under this path, e.g. /docs |
| Include globs | empty | Comma-separated URL globs kept (** crosses /, * stays in a segment) |
| Exclude globs | empty | URL globs dropped — wins over include |
| Allow subdomains | off | Also follow subdomains of the seed host |
| No sitemap | off | Skip the sitemap feed — pure link discovery |
| Sitemap only | off | Frontier is the sitemap; no link-following |
| Concurrency | 0 = core default | Fetch workers |
| Rate (rps) | 0 = core default | Per-host requests per second |
| Cost ceiling $ | 0 = config default | LLM spend guard in Extracted mode |
| TLS fingerprint / Language / Proxy | Settings defaults | Same meaning as on Scrape |
| Auto-throttle on errors | off | Doubles the delay on 429/5xx, decays back on success |
robots.txt is respected by design. A disallowed crawl reports Blocked by robots.txt instead of fetching.
The run panel
While the crawl runs: a status pill, the run id, live counters (ok, err, records), a progress bar against the page cap, and a per-page feed with each failure’s reason. Stop ends the run and keeps what it collected — the footer chip tracks it from any screen.
After it ends, the panel names the outcome — Finished — 42 pages, 42 records, or Stopped with the same counts — and the records table shows the first 50 rows with a format picker and Export…. Full telemetry (LLM calls, cost, per-page counts) is in History for the run id.
Records and export
Records live on disk as per-run JSONL — <runId>.jsonl, plus numbered .part files after a resume — beside the store (see Data and cache). From the finish panel or History’s Output tab:
- Preview the first 50 rows, long cells truncated for display only.
- Export as jsonl, json, csv or sqlite through the native save dialog. Corpus runs export jsonl/json only.
- Open folder reveals the run’s files.
What each format contains is on Export formats.
Resume and re-run
A stopped or failed crawl keeps its checkpoint. From History’s drawer, Resume continues the run without refetching done pages — earlier records stay in the export — and Re-run fills the form fresh for a clean start. The CLI equivalent is magpie crawl <url> --resume <run_id>; both share one store, so a CLI crawl shows up here too (CLI reference).
For pages that change, add a Watch instead of re-crawling.