Google Search Scraper: Extract SERP Results to CSV/JSON

Point a scraper at google.com/search and Google hands you a CSS shell that redirects to a JavaScript check; run it through a real browser and the reply is the “unusual traffic” page, my home IP stamped at the bottom. I ran both experiments today (October 16, 2026, from a home connection) because the first job of any google search scraper is admitting what it can’t do: read Google’s HTML. Everything that works starts from an API or a different engine, and this page tests each route, then finishes the part the SERP pages skip: getting the results out as CSV or JSON.

The rules, one paragraph. Google’s robots.txt disallows /search for every automated crawler; I pulled the live file today and the only carve-outs are /search/about, /search/howsearchworks, and social-media preview bots. The terms of service bar automated queries on top of that. People scrape results anyway at low volume for rank checks and research, and where your use case sits on that line is a question for Is Web Scraping Legal?, not for me. The API routes exist precisely because Google would rather sell structured results than be scraped.

What happens when webscraping Google directly

Static fetch first. One request, no JavaScript, the way most tutorials imply it should work:

{
  "url": "https://www.google.com/search?q=google+search+scraper&num=10",
  "title": "Google Search",
  "markdown": "<style>table,div,span,p{display:none}</style> <meta http-equiv=\"refresh\" content=\"0;url=/httpservice/retry/enablejs?sei=…\"> Cliquez ici si, d'ici quelques secondes, vous n'avez pas été redirigé."
}

Abridged, but that is the whole payload: a style tag hiding every element, a meta-refresh to an enablejs retry URL, and fallback text that arrived in French on my machine. Exit code 0. The record says success while carrying zero results, the same trap I hit scraping Reddit last week, and the reason I check the title field before trusting any run.

Second attempt, real browser render (--render browser):

**About this page**

Our systems have detected unusual traffic from your computer network.
This page checks to see if it's really you sending the requests, and
not a robot.

Google’s CAPTCHA wall, served to a headless Chrome on a residential IP after exactly two requests. The tool reported the wall instead of pretending to have results. Webscraping Google directly from any single IP is a losing game in 2026; today’s two requests were proof enough.

The honest limit of this Google search scraper

The commands below use Magpie, a local CLI (and desktop app) I use for this kind of work. It does not scrape Google either, and I’m not going to blur that. Its search command is a front end for six SERP backends: SerpApi and Serper, which return Google’s organic results through their APIs, plus Brave, Exa, DuckDuckGo, and self-hosted SearXNG, which return results from their own indexes. When a section here says “Google results,” it means one of two things: Google organic via a provider key, or results from a different engine for free. Both are labeled everywhere they matter.

Four routes to Google SERP data

Route	Whose results	Cost to start	Cap	My test today
SerpApi	Google organic	Free: 250 searches/mo; then $25/1,000	50/hr on free tier	Verified in code + recorded fixtures; no key on this box
Serper	Google organic	2,500 free credits; then $50 per 50,000	Credits expire after 6 months	Verified in code + recorded fixtures; no key on this box
Google Custom Search JSON API	Programmable Search index, not google.com organic	100 queries/day free; $5 per 1,000	10,000/day	Docs-checked, not demoed
Magpie search, zero-key	DuckDuckGo	Free, no account	10 hits default	Ran live
Magpie search, keyed	Brave / Exa / SearXNG indexes	Brave has a free tier; Exa paid; SearXNG free self-hosted	Brave: 20/request	Error paths tested live

Routes are ordered by Google-ness. The first two return what google.com shows. Custom Search is official but searches a different corpus with different ranking (more on that below). DuckDuckGo is a different engine entirely, which is either the point or the dealbreaker depending on why you’re here. The cloud no-code tools that fill this SERP (Octoparse, Browse AI) run these same routes in someone else’s cloud, point-and-click on top of proxy pools and SERP APIs; untested here, because nothing about them runs locally.

The Google search scraper API route: Serper and SerpApi

Same command for both; the provider flag is the only difference:

# key from env, or store it once in the OS keyring:
export MAGPIE_SERPER_API_KEY="…"
magpie config set-key serper

magpie search "best erp for small business" --provider serper --limit 20 --out serp.jsonl
magpie search "best erp for small business" --provider serpapi --limit 20 --out serp.jsonl

Without a key, the command fails loudly instead of silently returning junk. I ran the error path today, verbatim:

Error: scrape: search provider serper: missing API key: set MAGPIE_SERPER_API_KEY
or `magpie config set-key serper`   (exit 7)

Output is JSON lines, one record per hit, the same shape from every provider:

{"position":1,"title":"…","url":"https://…","snippet":"…"}

Serper: cheapest per query

Serper sells prepaid credit packs: $50 for 50,000 credits works out to $1.00 per 1,000 queries, dropping to $0.30 per 1,000 at the 12.5-million pack ($3,750). New accounts get 2,500 free credits, and packs expire six months after purchase, so don’t bulk-buy a year ahead, because unused credits are a donation. I checked the current packs today against Serper’s own page and three pricing trackers that agreed within cents.

SerpApi: the free tier and the 25x premium

SerpApi’s plans, from their pricing page today: Free 250 searches/month (capped at 50/hour), Starter $25/month for 1,000, Developer $75/month for 5,000. That starter price is roughly 25x Serper’s per-query cost. What the premium buys is parsing breadth: SerpApi’s raw responses include ads, People Also Ask, AI Overviews, and knowledge-panel data. Through Magpie you get the organic slice only, so unless you’re also using SerpApi’s raw API for SERP-feature research, the budget feed makes more sense here.

The free Google scraper that isn’t Google

No key, no account, no card:

magpie search "google web scraper" --limit 10

That ran live on this box today and returned ten hits, DuckDuckGo’s default provider. The parser reads DDG’s HTML endpoint directly, unwraps its redirect links to real URLs, and drops its ad results (identified by the /y.js click-path). First two records, abridged:

{"position":1,"title":"Web Scraper - Free Web Scraping - Chrome Web Store","url":"https://chromewebstore.google.com/detail/web-scraper-free-web-scra/jnhgnonknehpejjnehehllkliplmbmhn","snippet":"Once the data is scraped, download it as a CSV or XLSX file…"}
{"position":2,"title":"Google Search scraper and SERP API · Apify","url":"https://apify.com/apify/google-search-scraper","snippet":"Google Search Results Scraper crawls Google Search Results Pages (SERPs)…"}

It is the best free google scraper for “who writes about this topic” jobs. It is the wrong tool for rank tracking: DuckDuckGo’s index and ordering are not Google’s, and pretending otherwise poisons your data. For Google positions, pay the dollar per thousand.

The other keyed providers earn their keep differently. Brave Search has its own index and a free tier, capped at 20 results per request. Exa is a neural search built for “pages similar to this one,” not exact-query lookups. SearXNG is the self-hosted meta engine; point MAGPIE_SEARXNG_URL at your instance and it’s the only zero-key path that can aggregate Google, if your instance queries it. Without that variable set, the command tells you plainly: “searxng needs a self-hosted instance; it is the zero-key path.”

Google data extraction, finished as CSV or JSON

JSON is the native output: one object per line, ready for --out to file or stdout to a pipe. CSV takes one jq filter.

A bare Google URL scraper with jq

Ran today against the real file from the DDG demo:

jq -r '[.position,.title,.url] | @csv' serp.jsonl > urls.csv

1,"Web Scraper - Free Web Scraping - Chrome Web Store","https://chromewebstore.google.com/…"
2,"Google Search scraper and SERP API · Apify","https://apify.com/apify/google-search-scraper"
3,"Web Scraper - Chrome Web Store","https://chromewebstore.google.com/detail/web-scraper/…"

Add .snippet to the array if you want descriptions in the sheet; import the CSV into Excel or Sheets and it opens as a table. The full set of JSON-to-CSV patterns, nested records and whole-site crawls included, lives in How to Scrape a Website to CSV or JSON.

Chain the SERP into full-page scrapes

A SERP is usually the beginning of a job, not the end. The --scrape-top flag chains discovery into extraction: it pushes the first N result URLs through the normal scrape pipeline and attaches each page’s clean markdown to its hit record:

magpie search "how to export google search results" --limit 5 --scrape-top 2 --out chain.jsonl

Today’s run: positions 1 and 2 came back with full page records — 2,225 and 3,469 words of markdown respectively — while positions 3 through 5 carried no page key, exactly as requested. One command, SERP plus source material, all local. This is the part the SERP-API landing pages never demo, and it’s the reason the flag exists.

If your Google web scraper is a browser extension

The live results for that same query split two different jobs. Three of the top ten hits today were Chrome extensions; the rest were SERP APIs and guides like this one. If you want to click around a results page in your own browser and collect rows by hand, an extension does that, and the tested list lives in Web Scraper Chrome Extensions. The moment the job becomes automated (queued queries, scheduled runs, a headless box), extensions stop being an option: they run where your browser runs, and the JavaScript check and the CAPTCHA wait there just the same.

What a Google data scraper will not return here

The API routes in this guide return organic results only. Magpie’s Serper parser reads the organic array; its SerpApi parser reads organic_results. Neither surfaces ad copy, People Also Ask boxes, AI Overviews, the local pack, or the knowledge panel. A Google data scraper built this way answers “who ranks for this query” and “what do the snippets say.” It will not mine SERP features. For those, call the provider’s raw API, which returns the full payload at the same price per search.

And the official option deserves its honesty paragraph: Google’s Custom Search JSON API costs nothing for 100 queries a day and $5 per 1,000 up to 10,000/day, but it searches Programmable Search Engine, a different corpus with different ranking than google.com. Using it for rank checks is measuring a shadow.

Where this Google scraping tool runs out of road

Each search request runs with a 15-second timeout, so a provider having a slow day fails visibly rather than hanging your batch. Brave caps at 20 results per request by documented API limit. SearXNG requires running your own instance. Keys come from environment variables or the OS keyring (magpie config set-key <provider>), and the --api-key flag you see in the help is for LLM providers only; the search engines get their own named keys, never your model credentials. Pagination is your job: the command returns up to --limit hits per query, and crawling through pages of results means adjusting each provider’s parameters through it, not looping Magpie flags.

FAQ

Is scraping Google legal?

Pulling results off Google’s public pages sits in the same gray zone as any scraping: robots.txt forbids it, Google’s terms bar automated access, and enforcement against small-volume researchers is rare but not unheard-of. The API routes exist to move the question off your plate. The plain-English breakdown is in Is Web Scraping Legal?; not legal advice.

Will a Google web scraping tool beat the CAPTCHA?

Not honestly. Google’s bot check is aimed at exactly the traffic patterns bypass tools produce. Tools that advertise CAPTCHA solving rotate residential proxies and burn money per request; my tool detects the challenge and reports it, which is the difference between a bad day and a banned IP.

What is the best Google search scraper tool?

Budget and volume: Serper at $1.00 per 1,000 queries. Occasional small jobs: SerpApi’s 250 free monthly searches. Zero cost and engine-agnostic: magpie search on its DuckDuckGo default. SERP-feature research (ads, PAA, AI Overviews): your provider’s raw API, not this pipeline.

Does Google scraper software need proxies?

The API routes need none. The provider’s infrastructure absorbs the blocking problem, which is what the fee buys. Direct scraping is where proxy rotation enters, and Google still wins that fight most days, as the CAPTCHA screenshot above shows.

Should you trust a Google URL scraper online?

Safe from malware, mostly; they’re just web forms. But every query you type goes through someone else’s server and logs. Competitive research you’d rather keep private — the queries themselves are the sensitive part — belongs in a local command where queries and keys never leave your machine.

Which Google data extractor tool for which job

Rank tracking on real Google positions: Serper key, $1.00 per thousand, done. Free monthly volume under 250 searches: SerpApi’s free tier. Topic research and link discovery where the engine matters less than the sources: magpie search with no key at all. SERP-plus-content pipelines: the same command with --scrape-top. Ad and SERP-feature mining: your provider’s raw API.

Run the two free moves, magpie search "your query" and jq -r '[.position,.title,.url] | @csv', and you’ll know within a minute whether the free index is good enough or the Google-shaped dollar is necessary.