Google News Scraper: Collect Headlines by Topic

Google News has no official API, and its web app is a JavaScript shell that hands a static fetcher exactly nothing. What it still serves (undocumented, keyless, and fast) is a family of RSS feeds: top stories, eight topic sections, geo feeds, and a search endpoint that accepts operators. This guide is the scorecard from today’s runs (2026-10-16): which feed URLs work, what each one returns, and the three traps every Google News scraper walks into. One ground rule before anything else: news.google.com’s robots.txt disallows all crawling except a short whitelist, Google’s terms bar automated access, and I pull these feeds the way a feed reader would: one polite request at a time. Not legal advice.

What a Google News scraper can actually reach

I pointed Magpie at news.google.com/home with a static fetch and got quality blocked (empty) back. The headlines render in JavaScript, so the HTML path is dead on arrival; you would need a real browser to see the page at all. The RSS endpoints are the whole game. They have existed for years without fanfare. They need no key, and every pull I ran today returned in under 3 seconds.

The written rules are stricter than the technical ones. The live robots.txt at news.google.com says User-agent: * Disallow: / with a short allowlist: /home, /nwshp, /topics/, /publications/, /stories/, /about. The /rss paths are not on it. Google also names and bans the AI crawlers outright (GPTBot, ClaudeBot, PerplexityBot, CCBot, ChatGPT-User, anthropic-ai, Claude-Web, all under Disallow: /). When I ran Magpie’s crawl command against a feed URL, it refused before fetching: crawl: seed news.google.com: robots.txt disallows crawl. The single-URL scrape command does not consult robots, so whether any individual fetch suits your use is your call, not the tool’s.

My position, for what it is worth: RSS exists to be fetched by readers, and a pull every 15 to 30 minutes looks like a feed reader to any server. What I would not do is treat the site as a crawl target, walking /topics/ links page after page. That is what the robots file is written against.

Five feed URLs that work today

All of these returned clean XML when I pulled them on 2026-10-16 from a home connection:

Feed	URL	What I got
Top stories	news.google.com/rss?hl=en-US&gl=US&ceid=US:en	38 items in 1.2 s
Topic	news.google.com/rss/headlines/section/topic/TECHNOLOGY?hl=en-US&gl=US&ceid=US:en	48 items in 2.9 s
Search	news.google.com/rss/search?q=when:1d%20data%20center&hl=en-US&gl=US&ceid=US:en	100 items raw; 50 in Magpie’s record
Per-publisher	news.google.com/rss/search?q=site:reuters.com%20when:1d&hl=en-US&gl=US&ceid=US:en	100 raw, 50 via Magpie, all Reuters
Geo	news.google.com/rss/headlines/section/geo/Bangkok?hl=en-US&gl=US&ceid=US:en	50 items

The topic slugs are the eight standard sections: WORLD, NATION, BUSINESS, TECHNOLOGY, ENTERTAINMENT, SPORTS, SCIENCE, HEALTH. I verified TECHNOLOGY, BUSINESS, and WORLD live (48 to 50 items each); the rest follow the same pattern.

Three parameters localize every feed. hl is the interface language, gl is the country, and ceid bundles them as country:language. The same top-stories URL with hl=en-GB&gl=GB&ceid=GB:en handed me the UK edition, also 38 items. Swap in your own locale and the feed follows.

The q= parameter on the search feed accepts the operators the site’s search box does. The two that do all my work: when:1h, when:1d, when:7d for time windows, and site:reuters.com to turn any publisher into a feed. URL-encode spaces as %20 and you are set.

Feeds also beat browser rendering on the clock. Every browser-rendered pull I have timed on news sites ran 10 to 30 seconds per page; every Google News feed pull today ran under 3. At a 15-minute polling cadence, that difference compounds fast.

Pull any feed in one command

Magpie’s rss vertical parses these feeds with zero LLM calls and no API key. The command is one line:

magpie scrape "https://news.google.com/rss?hl=en-US&gl=US&ceid=US:en" --vertical rss

Real output from my top-stories pull today (abridged; the summary field normally carries the full story cluster):

{
  "vertical": "rss",
  "title": "Top stories - Google News",
  "record": {
    "items": [
      {
        "title": "Teen shooter kills 3 children, teacher in Oregon elementary school before being shot by bystander",
        "link": "https://news.google.com/rss/articles/CBMie0FVX3lxTE56TDBxdnR3ZVZa...",
        "published": "Fri, 16 Oct 2026 10:50:00 GMT",
        "summary": "<ol><li><a href=\"...\">Teen shooter kills 3 children...</a>&nbsp;&nbsp;<font color=\"#6f6f6f\">CNN</font></li></ol>"
      }
    ]
  }
}

That summary field deserves a second look. Each top-stories item wraps the whole cluster: every related piece from every outlet covering the story, as list items with their own links. If you are tracking one story’s coverage across outlets, the cluster is the product and the title is just its label.

Four fields per item: title, link, published, summary. The source name is not a field of its own in Magpie’s record; it survives inside the summary HTML as <font color="#6f6f6f">CNN</font>. The raw XML actually carries a proper <source url="https://www.cnn.com">CNN</source> element on every item, so if per-item publisher homepages matter to you, fetch the XML directly or regex it out of the summary:

import re
source = re.search(r'<font color="#6f6f6f">([^<]+)</font>', item["summary"])

For a CSV of headlines, scrape’s --format only does json and jsonl, so hand the JSON to jq:

magpie scrape "https://news.google.com/rss/search?q=when:1d%20data%20center&hl=en-US&gl=US&ceid=US:en" --vertical rss \
  | jq -r '.record.items[] | [.published, .title] | @csv' > data-center-headlines.csv

If you would rather not install anything, Python’s stdlib parses the same XML:

import urllib.request, xml.etree.ElementTree as ET

url = "https://news.google.com/rss/headlines/section/topic/TECHNOLOGY?hl=en-US&gl=US&ceid=US:en"
req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
root = ET.fromstring(urllib.request.urlopen(req).read())
for item in root.findall("channel/item"):
    print(item.findtext("pubDate"), "|", item.findtext("title"))

Three traps, tested today

Feeds cap at 100, and Magpie shows 50 of them. My site:reuters.com search feed measured exactly 100 items in the raw XML. Magpie’s rss vertical returns the first 50, its own documented cap rather than Google’s. Top-stories feeds sit under both numbers at 38. I have never needed the tail: narrowing the query with when: beats paging through a longer dump, and if you truly need items 51 through 100, pull the XML with curl.

The same story shows up twice. Google News aggregates syndication, so one wire story arrived in my 50-item search pull from two outlets under near-identical titles (“Data center moratorium bills stack up ahead of Virginia legislative se…”, twice). Every title ends in - Source, so dedupe on the title with that suffix stripped and you get one row per story.

Article links do not resolve over plain HTTP. The link field points at news.google.com/rss/articles/CBMi..., not the publisher. I followed one with curl: a 302 to the same URL with locale parameters appended, then a 200 serving a 583 KB JavaScript page titled “Google News”. The old base64-decode trick is dead for the new-format tokens; I decoded one and found no URL inside. Even a full browser render through Magpie refused: text and comments are not long enough: 0 0. For headline collection this usually does not matter, because title, source, and timestamp are the payload. If you genuinely need publisher URLs, that is the hard 10 percent: dedicated decoders exist, they break whenever Google rotates the scheme, so retest anything you build on one.

From headlines to a pipeline

A one-off pull is one command. A standing collection is a cron line and a dedupe, and the same rss feed URL works as Magpie’s watch input, where each run diffs against the last and can fire a webhook on change. One warning from my own testing on other news feeds: Magpie’s watch treats a reorder as a change, and Google News topic feeds reorder constantly as stories age. Prefer a when:1h search feed for watching, or dedupe titles before you treat a webhook as a new story. The full watch setup lives in RSS Feed Monitor: Get a Webhook for Every New Item.

For archival pulls, cron plus a dedupe pass does the job. The dedupe keys on the title with the trailing - Source stripped; on my 50-item when:1d data center pull it dropped exactly 2 syndication duplicates, leaving 48:

import re

seen, deduped = set(), []
for item in items:
    key = re.sub(r" - [^-]+$", "", item["title"])
    if key not in seen:
        seen.add(key)
        deduped.append(item)

Where to go from here depends on what you are collecting. For other publishers’ feeds and full-article extraction, see News & Article Scraper: Collect Articles From Any News Site. For the search-results page, a different animal with consent walls and SERP features, see Google Search Scraper: Extract SERP Results to CSV/JSON. For a single-publisher example of this same feed-first approach, see BBC News Scraper: Headlines & Articles. And for turning any of these into spreadsheets, How to Scrape a Website to CSV or JSON covers the export side.

FAQ

Is scraping Google News legal? The feeds are public, but robots.txt disallows /rss and Google’s terms bar automated access without permission. Low-rate pulls that mimic a feed reader are common practice; large-scale collection needs permission or a licensed data provider. Not legal advice.

Does Google News have an API? No. There is no official public API. The RSS endpoints are the de facto interface: undocumented, but stable for years. Anything sold as “the Google News API” is a wrapper around these feeds or around scraping.

How do I get the original publisher URL instead of the Google redirect? You mostly do not, cheaply. Plain HTTP resolves to a JavaScript page, the base64 trick fails on new-format tokens, and decoders break when Google rotates the scheme. For headline pipelines, work with title, source, and timestamp.

How often do Google News RSS feeds update? Continuously: the newest item in my top-stories pull carried a timestamp minutes old. Polling every 15 to 30 minutes catches everything worth catching; polling faster just burns requests.

Why does my feed show the same story twice? Google News aggregates syndicated coverage, so one wire story can appear from several outlets. Dedupe on the title after stripping the trailing - Source suffix.

Start with the top-stories URL, narrow it with when: and site: until the feed only contains what you care about, then put the same magpie scrape line in cron at a 15-minute cadence. That is a working headline pipeline: one command, no key, no rendering, no cost.