Job Scraper: Collect Postings From Any Job Board

Job postings hide in two kinds of places, and the difference decides your whole architecture. Company boards (the Greenhouse, Lever and Ashby pages that host most tech hiring) publish structured data begging to be collected, while the big aggregators (Indeed, LinkedIn, Glassdoor) wall the same data behind login flows and bot checks. This guide builds a job scraper that exploits the first group properly: typed records with titles, salaries and locations, no CSS selectors, no LLM bills — plus a measured fallback for boards that publish nothing structured, all commands verified against live boards on 2026-10-16 from a home connection.

Before you start: Magpie, a single static Go binary (AGPL-3.0-or-later or commercial; the CLI is free with no run counter), and jq for the feed parsing. No accounts, no API keys. If your target turns out to be an aggregator, skip to the walls section.

Job boards come in two species

Open any company’s careers page and look at the URL. If it contains greenhouse.io, lever.co, ashbyhq.com, workable.com, smartrecruiters.com or teamtailor.com, the company is renting an applicant tracking system (ATS), and the board you see is one of a handful of templates served millions of times. Six ATS families cover a large share of tech-company job boards, which means one working recipe repeats across thousands of sites.

Aggregators are the other species: Indeed, LinkedIn, Glassdoor, Google Jobs, ZipRecruiter. They index postings from the ATS boards and monetize the attention in between. Their data is the same data, wrapped in more anti-scraping. The short version of this entire guide: collect from the ATS boards, where the data lives and the walls don’t.

Step 1: Check the JSON feed before you scrape anything

The best job scraper is a curl command. Greenhouse, Lever and Ashby all expose public JSON endpoints that power their own frontends, and none of them require auth. Try it:

$ curl -s "https://boards-api.greenhouse.io/v1/boards/affirm/jobs" | jq '.jobs | length'
211

$ curl -s "https://api.lever.co/v0/postings/spotify?mode=json" | jq 'length'
82

$ curl -s "https://api.ashbyhq.com/posting-api/job-board/ashby" | jq '.jobs | length'
25

The Greenhouse feed gives you every posting’s URL, title and location in one request:

$ curl -s "https://boards-api.greenhouse.io/v1/boards/affirm/jobs" \
  | jq -r '.jobs[] | [.absolute_url, .title, .location.name] | @tsv' | head -3
https://job-boards.greenhouse.io/affirm/jobs/7850544003	Administrative Assistant IV	Remote US
https://job-boards.greenhouse.io/affirm/jobs/7808667003	Analyst	Remote US
https://job-boards.greenhouse.io/affirm/jobs/7832318003	Analyst II, Credit	Remote UK

Replace affirm with the board name from the company’s careers URL, same for the Lever and Ashby paths. Three failure modes to know: a dead board name returns {"status":404,"error":"Job not found"} from Greenhouse; some companies (Stripe, Duolingo, Databricks) moved their boards to their own domains, so the feed’s absolute_url points at their site instead of boards.greenhouse.io (the feed still works, the detail pages are just someone else’s problem); and none of these endpoints are documented as public APIs, so treat them as read-only and throttle your requests. What you should see is a JSON array; what it means is you just skipped the entire scraping problem for the listing layer.

Step 2: Get typed records with –vertical job_posting

Feeds give you listings. Postings, with their description, salary, employment type and date, need the detail page, and here the structured data moves into the page source. Schema.org JobPosting JSON-LD is a JSON blob embedded for Google’s job rich results, and where it exists you read fields instead of writing selectors. Magpie ships a zero-LLM extractor for exactly this block, named job_posting:

$ magpie scrape "https://jobs.ashbyhq.com/ashby/7458d4e9-da2e-47bd-98cb-adfda43d42b2" \
    --vertical job_posting --format json

You should see a "vertical": "job_posting" block like this (abridged):

{
  "vertical": "job_posting",
  "record": {
    "title": "Engineering Manager - EU",
    "organization": "Ashby",
    "locations": ["European Union"],
    "remote": true,
    "employmentType": "FULL_TIME",
    "datePosted": "2024-03-04",
    "salary": {
      "currency": "EUR", "min": 110000, "max": 185000, "unit": "YEAR"
    },
    "description": "Hi 👋 I'm Colin, Director of Engineering, Europe. …"
  }
}

That is a typed record: salary normalized to min/max/unit/currency, remote set only when the page actually declares TELECOMMUTE (never guessed), description stripped of HTML tags. Fields appear only when the board publishes them: the Spotify posting I pulled from Lever the same day had no salary key at all, because Lever’s JSON-LD for that job carries none. That refusal to invent values is the difference between a dataset and a guess.

Two things the vertical will not do. First, it refuses pages without JobPosting data with a typed error, vertical: job_posting: no JobPosting data at <url>, which is diagnostic rather than a bug: it tells you to drop to Step 3. Second, it is explicit-only: unlike some Magpie verticals that auto-fire on matching URLs, job_posting does nothing unless you pass --vertical job_posting. It never misfires on non-job pages, and it never runs without you asking.

Coverage as measured on 2026-10-16: Lever and Ashby detail pages carry JobPosting JSON-LD; Greenhouse’s own board pages carry none. I pulled the 87,758-byte HTML of an Affirm posting and counted zero ld+json blocks. Greenhouse is not a dead end (Step 3), but if a blog told you Greenhouse pages have schema.org job data, nobody in its chain opened the page source.

Step 3: No JSON-LD? Chain the feed with plain scraping

Greenhouse detail pages have no structured data, but they are server-rendered, so a plain fetch returns the full posting as clean Markdown. Chain the feed from Step 1 with a per-URL scrape and you get one JSONL record per job:

$ while IFS=$'\t' read -r url title loc; do
    magpie scrape "$url" --page-format markdown --format jsonl >> affirm_jobs.jsonl
    sleep 2
  done < <(curl -s "https://boards-api.greenhouse.io/v1/boards/affirm/jobs" \
    | jq -r '.jobs[] | [.absolute_url, .title, .location.name] | @tsv')

Each record looks like this (abridged):

{"url":"https://job-boards.greenhouse.io/affirm/jobs/7850544003","title":"Job Application for Administrative Assistant IV at Affirm","markdown":"# Administrative Assistant IV\n\nAt Affirm, we exist for the moments that matter…"}

The <title> tag is wordy (“Job Application for X at Y”), but the Markdown’s # heading holds the clean job title, and the feed TSV you looped over already gave you title and location as structured columns. At roughly three seconds per posting (sleep plus fetch), the full 211-job board takes about twelve minutes; for a weekly snapshot that’s nothing. If a posting’s Markdown comes back empty, the page probably needs JavaScript: re-run that URL with --render browser and check Magpie’s dynamic-page guide for what that changes.

Step 4: Automate across boards

The Step 3 loop is your automation unit. Wrap it: a boards.txt file of feed URLs, one scrape pass per board, sort -u on record URLs to dedupe reruns. Two honest limits. magpie crawl has no --vertical flag: crawling a board tree gives you Markdown per page, not typed records. Per-URL scrape loops are the right shape, not lazy ones. And these loops are yours to schedule: cron it weekly, diff on datePosted, and you have new-posting detection, which is the whole game for sales intelligence or market mapping. The watch-mode route to per-board alerts is its own recipe: get alerts when a company posts new jobs.

Where a job scraper hits the walls

Sooner or later someone asks why you don’t just scrape Indeed, since it indexes everything. Because the same request that works on an ATS board usually doesn’t work there. I tested Indeed’s search results on 2026-10-16: the listing page renders about six card summaries (1,571 words of company names and snippet text), but zero job-detail links exist in the served HTML — no /viewjob, no /cmp/ URLs, no posting dates. Indeed puts the real result set behind its own app traffic, and the teaser is all anonymous visitors get. Glassdoor and Google Jobs run their own walls; Magpie’s live tests on both came back walled or challenged. LinkedIn serves public job pages but gates search and feeds behind login. These are walls by design, not failures of tooling. Magpie detects bot challenges and reports them; it does not bypass them. For polite-crawling practice on the boards that do let you in, see scraping without getting blocked.

If you need aggregator coverage anyway, two options work. JobSpy is open-source Python (4,496 stars, actively maintained as of October 2026) against Indeed, Glassdoor, LinkedIn and ZipRecruiter; expect maintenance churn, because every aggregator redesign breaks it until someone patches it. Hosted Apify actors cover the same boards, paying per result and letting someone else absorb the breakage. For most ATS-board use cases, though, Step 1’s curl replaces both.

Make it stick

Three extensions, in the order I’d build them. Convert the JSONL dataset to CSV or Excel columns with jq or the website-to-CSV guide. Recruiters ask for spreadsheets, not JSONL. Add the weekly cron from Step 4 and keep a per-board snapshot directory, one file per run, so diffs are diff not archaeology. And when a target company’s board is one of the named ATSes, the per-board pages go deeper than this generic guide: Greenhouse and Lever each have quirks worth knowing before you loop.

FAQ

Is it legal to scrape job postings? Public postings on company boards are publicly accessible data, and US case law (the hiQ v. LinkedIn line) has favored scraping public pages. But ToS, the CFAA’s edges and your jurisdiction matter; this is information, not legal advice. The legality deep-dive covers the docket, and counsel signs off before commercial use.

Can you scrape Indeed? Not the result set. The anonymous teaser page renders a few card summaries with no job-detail links in the HTML; pagination and full results require the app traffic of a logged-in session. Tools that do it (JobSpy, Apify actors) reverse-engineer internal endpoints and break when Indeed ships changes.

What is JobPosting JSON-LD? A schema.org JSON block embedded in a job page’s HTML, intended for Google’s job rich results. It names the object a JobPosting and spells out title, hiringOrganization, jobLocation, baseSalary, datePosted as machine-readable fields. If a page has it, you don’t need CSS selectors.

Does the salary field always come through? No. The extractor reads baseSalary only when the board publishes it: the Ashby posting carried a €110k–185k range, the Lever posting carried none, and Greenhouse boards have no JSON-LD to read at all. A missing salary key means the source lacked it, and the extractor never estimates one.

Which boards should I start with? Whatever your target company actually uses; check the careers URL for greenhouse.io, lever.co, ashbyhq.com or a sibling. Greenhouse, Lever and Ashby all expose the public JSON feeds from Step 1, and between them they host a large share of tech hiring.

A last concrete move: pick one company you actually care about, run the three curl commands from Step 1 against its board, and see which rung of the ladder it lands on. That answer decides everything downstream, and you’ll have it in under a minute.