News & Article Scraper: Collect Articles From Any News Site

News is the worst-case scraping target and the easiest one, in the same hour. A newspaper’s homepage reorders itself every few minutes, yet every story it publishes carries the same five facts — headline, byline, publish date, publisher, link — in machine-readable form. A news scraper is just the thing that collects those facts at scale, and you don’t need a Python project to build one: the commands in this guide ran against live BBC and NPR pages on 2026-10-16, and the slowest one took 29 seconds.

Before you start: Magpie, a single static Go binary (AGPL-3.0-or-later or commercial; the CLI is free, no account, no API key for anything below). Install it, and every command here runs as written.

Pick a route before you pick a news scraping tool

Most news sites offer three doors, and the difference is what each one hands you:

Route	You get	Speed (measured)	Breaks when
RSS feed	Links, titles, dates, summaries, no full text	33 BBC items in one call	Site has no feed (rare)
Article JSON-LD	Typed record: headline, authors, dates, image	5–6.5s per article	Page omits schema.org markup
Full-page clean text	The article body as markdown	634 words in 16.5s	Paywalls, bot walls

The routes stack. The feed gives you URLs, the JSON-LD gives you metadata, the clean text gives you the body. A complete news dataset is all three merged per article. Step 3 does exactly that.

Step 1: Pull the feed

Every major publisher still runs RSS, including the ones that hide it. BBC News’s main feed:

$ magpie scrape "https://feeds.bbci.co.uk/news/rss.xml" --vertical rss

That returned 33 items, each with title, link, published and summary:

{
  "title": "Children aged 5, 8 and 11 killed in US school shooting before suspect shot by bystander",
  "link": "https://www.bbc.co.uk/news/articles/cm89zp48k4kzo?at_medium=RSS&at_campaign=rss",
  "published": "Fri, 16 Oct 2026 03:45:09 GMT",
  "summary": "A teacher was also killed in the attack at the Silver Rail Elementary School in the city of Bend, police say."
}

What you should see: a "vertical": "rss" block with an items array. The vertical reads RSS 2.0 and Atom and caps output at 50 items, so poll it on a schedule rather than expecting an archive. NPR’s top-news feed returned 10 items the same way.

Failure case: the feed URL 404s or returns HTML. Publisher feeds usually live at /rss, /feed, or under a /feeds index, or check the page source for type="application/rss+xml". Strip tracking parameters (?at_medium=RSS&at_campaign=rss) before you store the links, or your dedupe logic will count the same story twice.

No feed at all? Skip to Step 4.

Step 2: Run the article scraper on any article URL

The article scraper route reads the schema.org JSON-LD that news sites embed for Google. NewsArticle markup spells out the fields an LLM or a database wants, and Magpie’s article vertical parses them without selectors:

$ magpie scrape "https://www.bbc.co.uk/news/articles/cm89zp48k4kzo" \
    --vertical article --render static

Output (the real record from today’s run):

{
  "headline": "Children and teacher killed in Oregon primary school shooting",
  "authors": ["Rebecka Pieder"],
  "datePublished": "2026-10-16T01:29:24.694Z",
  "dateModified": "2026-10-16T03:45:09.909Z",
  "publisher": "BBC News",
  "image": "https://ichef.bbci.co.uk/...jpg",
  "url": "https://www.bbc.co.uk/news/articles/cm89zp48k4kzo"
}

One flag matters here. With default auto rendering the same call took 29 seconds; --render static brought it down to 6.5 on BBC and 5.2 on NPR. The metadata sits in the page source, so static is the fast path. Drop the flag only when a site refuses to serve the markup without JavaScript.

Failure case: Error: vertical: article: no Article data at <url> means the page carries no Article/NewsArticle/BlogPosting JSON-LD. Plenty of regional and local outlets skip it. The record also contains no body text, by design. That’s Step 3’s job.

Step 3: Get the full text and merge

The default scrape path cleans the page down to its main content and returns GFM markdown:

$ magpie scrape "https://www.bbc.co.uk/news/articles/cm89zp48k4kzo" --page-format markdown

The Oregon story came back as 634 words of clean text: no nav, no “more stories” links, no cookie banner. If you’re feeding articles to an LLM, this is the output that matters; the website summarizer workflow starts exactly here.

Merging metadata and body per article is a two-call pattern:

$ magpie scrape "$URL" --vertical article --render static > meta.json
$ magpie scrape "$URL" --page-format markdown > body.json
$ jq -s '.[0].record + {text: .[1].markdown}' meta.json body.json > article.json

One article.json per story: typed fields plus text. That file is what you load into a database, a search index, or a RAG pipeline.

Step 4: Collect in bulk when there’s no feed

Some sites, or whole sections of sites, publish no feed. Crawl the section page and take the text from every page it reaches:

$ magpie crawl "https://www.bbc.co.uk/news/business" \
    --max-pages 6 --max-depth 1 --corpus --out corpus.jsonl

That run finished in 17 seconds: 6 pages fetched, 0 errors, one {url, title, depth, markdown} JSONL record each, the section page itself cleaning to 916 words.

Now the honest part, because it will bite you: five of those six records were BBC Food pages. Same-domain crawling follows every same-domain link, and news navs link everywhere: recipes, programmes, whatever the masthead advertises. A news crawl without URL filtering is a crawl of the whole site. Fix it by filtering after the fact (grep '/news/articles/' corpus.jsonl) or by seeding from a page whose links stay in-section. Note also that crawl has no --vertical flag: corpus mode returns clean text, not typed records. Pair it with Step 2 on the URLs you keep.

Step 5: Make it an automatic news scraper

Everything so far is one-shot. The automatic news scraper is the same commands on a timer, with Magpie’s watch mode doing the change detection:

$ magpie watch "https://feeds.bbci.co.uk/news/rss.xml" \
    --every 5m --webhook https://you.example/hook

When the feed changes, watch POSTs {url, changed, old_hash, new_hash, diff} to your webhook and stays silent otherwise. I ran two --once checks a minute apart on the live BBC feed and the second one produced a diff: the feed had already reordered items between runs. That’s the mode’s honesty and its caveat in one. You get a diff for any change, reordering included, so new stories and reshuffles fire alike. Compare links against what you’ve stored if you want new-items-only semantics. For cron instead of a long-running loop, --once is the timer mode:

*/10 * * * * magpie watch "https://feeds.bbci.co.uk/news/rss.xml" --every 10m --once --webhook https://you.example/hook

The RSS feed monitor guide goes deeper on the webhook side. Magpie also ships dedicated verticals for sources that aren’t plain feeds: see the Hacker News scraper and Substack scraper guides for those.

What breaks, and what to do instead

• Paywalls and bot walls. Magpie detects bot challenges and reports them; it does not defeat them, and neither should your pipeline. Sites that wall aggressively will return errors. That’s the correct result. Politeness is built in (robots.txt respected, per-host throttling), which matters when you’re polling fifty feeds an hour.
• No JSON-LD, no feed. Regional outlets sometimes offer neither. Step 4’s corpus route still works, but your records carry no typed bylines or dates; you’re regexing titles out of text. Budget for that before you promise structured data downstream.
• Feed reorder noise. Shown in Step 5. If your webhook fires on reorders and your on-call reads every payload, dedupe before you alert.
• Scale. One crawl runs at a time per Magpie instance, and the CLI takes no --vertical on crawls. For thousands of articles a day, run scrapes from a queue instead of one mega-crawl. If you want a hosted service to do the fetching instead, that’s a different trade-off; the news scraping API comparison covers it.

On legality: facts, headlines and dates are generally fine to collect; republishing full article text is a copyright question, not a technical one. Read is web scraping legal, then ask a lawyer if money is involved. This article is not legal advice.

FAQ

Is scraping news sites legal? Collecting factual metadata (headline, byline, date, link) from public pages is generally accepted practice; republishing full text implicates copyright. Robots.txt and terms of service still apply. The legal guide above has the detail, and a lawyer signs off on real deployments.

Can I scrape Google News the same way? Google News is a JavaScript-heavy aggregator, not a publisher. Scraping its pages is its own project. Go to the source: every publisher in its results has an RSS feed, which is what Step 1 reads. Site-specific walkthrough: Google News scraper guide.

RSS or scraping: which should I use for news? RSS if you need links and summaries (one call, fifty items). Scraping when you need the body text or typed per-article fields. Most serious pipelines do both: feed for discovery, scrape for content.

How do I collect only new articles? Watch the feed with --webhook and dedupe on the link field, since watch fires on any feed change including reordering. Store links you’ve processed and diff before ingesting.

---

Pick by what you have: a feed URL means Step 1 plus Step 5 and you’re done in ten minutes; a list of article URLs means Steps 2 and 3; neither means Step 4 with a URL filter. All three routes ran on live news pages the day this was written.