Reddit Scraper: Posts, Comments & Subreddits

Reddit’s Data API is free for non-commercial use, capped near 100 requests a minute per app, and priced at $0.24 per 1,000 calls for everyone else. The public .json endpoints most tutorials lean on instead still exist, but only when Reddit likes your IP: a request that pulled a full comment tree on October 11 returned a login wall in French when I re-ran it today. A reddit scraper in 2026 needs more than one route, and this page gives you three, each with the exact commands and the output I actually got.

The rules, in one paragraph: public threads are fine to read, Reddit’s User Agreement and Data API terms still govern what you do with the data, and republishing users’ personal information is where people get burned. That is not legal advice; Is Web Scraping Legal? covers the plain-English version. Everything below ran October 16, 2026, from a home connection, against r/selfhosted and r/golang. Reddit changes its mind hourly. Re-test before you depend on any of it.

Which Reddit scraper route fits your job

Route	What you get	Needs	My test today
Official API, app-only OAuth	Typed records, full comment trees, subreddit listings	A free “script” app + 3 env vars	Verified in code and tests; no creds on this machine, so not demoed live
Public .json, no key	Same record shapes	Nothing	Walled today (HTTP 302 to a login page); worked October 11
Browser render	Post body as clean Markdown	Local machine, nothing else	Worked: 1,067 words in 10.0s where static fetch got 0

Same tool runs all three: Magpie, a local CLI (and desktop app) whose reddit vertical is one of 27 zero-cost extractors that return typed records instead of raw HTML. The vertical rewrites your URL to Reddit’s machine-readable endpoints, parses them, and hands back post and comment fields. No LLM, no API key, no per-record cost.

What Reddit serves, and what it walls

Reddit has three doors and locks them at different times.

The official Data API is the front door. A free “script” app (create it at reddit.com/prefs/apps) gets you read access to public data at roughly 100 queries per minute on the free non-commercial tier. Commercial use goes through paid tiers at the $0.24/1,000 rate Reddit introduced in 2023. Listing responses top out at 100 items per page.

The second door is the old .json trick: append .json to almost any old.reddit.com URL and the server returns the raw data structures behind the page. A permalink returns two listings, the post and its comment tree. No key, no signup. The catch is that Reddit gates anonymous access by IP reputation. Datacenter IPs are walled solid. Residential IPs get through until they don’t, which is exactly what I hit: October 11, full thread; October 16, a 302 to a login page.

The third door is the rendered HTML page itself, which needs a real browser and gives you text, not typed fields.

Route 1: a post and its full comment tree, no key

The command for a reddit scraper run against a single thread:

magpie scrape "https://www.reddit.com/r/selfhosted/comments/1wzngic/what_i_run_oct_2026/" \
  --vertical reddit --format json

The vertical rewrites this to old.reddit.com/r/selfhosted/comments/1wzngic/what_i_run_oct_2026/.json, parses the two-listing shape, and emits one record. The shape (values illustrative, abridged):

{
  "kind": "post",
  "title": "What I run - Oct 2026",
  "author": "some_homelabber",
  "score": 1204,
  "selftext": "Proxmox as the hypervisor, debian VMs to run Docker...",
  "comments": [
    {
      "author": "another_user",
      "score": 312,
      "body": "How do you handle backups across those hosts?",
      "created": 1792100000,
      "replies": [
        { "author": "some_homelabber", "score": 98, "body": "PBS nightly...", "replies": [] }
      ]
    }
  ]
}

Each comment carries author, score, body, a UTC timestamp, and its own nested replies array, so you can walk the thread the way it actually branched. Two hard caps apply: 200 comments total and 10 levels deep, with the “more replies” placeholders skipped. That covers most threads. A 3,000-comment megathread will come back truncated; the upgrade path is API pagination, which the extractor doesn’t do yet.

On October 11 this exact command returned the post, its score, and the comment tree. Today it returned this instead (real output, abridged):

{
  "vertical": "reddit",
  "record": {
    "kind": "post",
    "title": "Bienvenue sur Reddit",
    "author": "",
    "score": 0,
    "selftext": "Connecte-toi ou inscris-toi pour personnaliser ton flux..."
  }
}

That is the login wall, scraped. Reddit served the anonymous request a signup page in French, the .json decode failed, and the HTML fallback scraped the stub. Nothing crashed, which is the problem: a wall looks like bad data, not an error. The extractor’s own source notes that anonymous listings are IP-blocked “in practice”. If you see a record titled “Bienvenue sur Reddit” (or “Welcome to Reddit”) with zero scores, your IP is walled and you need Route 2.

Route 2: free app credentials make it reliable

This is the route I recommend for anything you will run more than once. It takes about two minutes:

1. Go to reddit.com/prefs/apps, click “create another app…”, and choose the script type. Name it anything, point the redirect URI at http://localhost.
2. Note the client ID (the string under the app name) and the secret.
3. Export three variables before running Magpie:

export MAGPIE_REDDIT_CLIENT_ID="your-client-id"
export MAGPIE_REDDIT_CLIENT_SECRET="your-secret"
export MAGPIE_REDDIT_UA="windows:my-reddit-scraper:v1.0 (by /u/yourname)"

With those set, the same magpie scrape command switches endpoints: instead of anonymous old.reddit, Magpie exchanges your client credentials for an app-only bearer token and reads oauth.reddit.com, which is the same interface the Data API serves. Permalinks and subreddit listings both go through it. Set the variables once in your shell profile and forget them.

Three behaviors worth knowing, all verified in the source:

• Partial configuration fails loudly. Set one variable and forget another and you get reddit credentials partially configured, missing: MAGPIE_REDDIT_CLIENT_SECRET instead of a silent fallback to the walled anonymous path.
• If the token exchange fails, you get an error with the HTTP status. No quiet degradation.
• The --user-agent variable matters because Reddit’s API terms expect an identifiable agent, and a distinctive one keeps you off the abuse radar at volumes that would look robotic otherwise.

I could not demo this path live: this machine has no Reddit credentials, and I won’t invent output. The token exchange, endpoint switch, and record shapes are pinned by the extractor’s unit tests, and the app-only flow uses Reddit’s documented client-credentials grant.

Route 3: no key, no API, render the page

When credentials are a blocker (a shared machine, a quick one-off), the plain page pipeline with a real browser got through the wall today:

magpie scrape "https://www.reddit.com/r/selfhosted/comments/1wzngic/what_i_run_oct_2026/" \
  --page-format markdown --render browser --out post.md

Static fetch: 0 words. Browser render: 1,067 words in 10.0 seconds, and the Markdown was clean, post title as H1, the homelab tables preserved as tables, links intact. How the browser escalation works, and when to reach for it, is the subject of How to Scrape Dynamic Websites.

What this route gives up: typed fields and the comment tree. The Markdown ends at “Ask me anything!”, the post body, because Reddit loads comments lazily in the browser. Scores and usernames come embedded in page text rather than fields. For “what does this post say”, it’s perfect. For “give me every comment with its score”, it is the wrong tool and Route 2 is the answer.

Subreddits: what a listing URL returns

Point the vertical at a subreddit instead of a permalink:

magpie scrape "https://www.reddit.com/r/golang/" --vertical reddit --format json

You get a subreddit record: kind, title, the sidebar description, and the subscriber count in the score field. Useful for a quick “who is this community” card.

What you do not get is the list of posts. The listing record is the community, not its front page, and the extractor does not paginate /new or /top. If your job is collecting every post from a subreddit, this vertical is the wrong shape; the Data API’s listing endpoints with pagination are the honest answer, and Magpie doesn’t wrap them yet. For “tell me when something new shows up”, Monitor a Subreddit for New Posts covers watch mode, with its own stated limit: it alerts on any new post, with no keyword filter.

Three boundary cases I tested so you don’t have to:

• sh.reddit.com shortlinks are rejected with vertical extractor does not handle URL. Use the full www.reddit.com/r/... form.
• User profiles (/user/name) don’t match the vertical at all; it only handles /r/ paths.
• old.reddit.com and new.reddit.com URLs work identically to www.

Keep it polite, keep it running

One thread per run, results to JSON with --out thread.json, and modest intervals: Reddit rate-limits aggressively, and Route 2’s free tier of about 100 requests a minute is far more than a polite scraper needs. When the job is change detection rather than collection, magpie watch <url> --every 1h stores a Markdown snapshot and only speaks when the page changes, posting a word-diff to your webhook. The politeness specifics (delays, caching, identifying yourself) live in Web Scraping Best Practices.

The CLI costs nothing and has no run counter; it is AGPL-3.0-or-later, with a paid commercial licence option. The Desktop app wraps the same engine on the plan: 5 free runs per install, then $49/month, where failed and stopped runs are free but a successful one counts, including a wall run that returns a French stub.

Magpie is not the only reddit scraper worth knowing. Hosted actors (Apify and its cousins) will run Reddit jobs for a per-record price, and their whole value is proxy rotation, because Reddit walls datacenter ranges even harder than it walled my home IP. HN’s community runs on the same listing-plus-comments shape, and the Hacker News scraper page shows the sibling vertical that reads the Algolia API, no credentials, no walls, tested this week. For the wider set, the scrapers by site hub lists every site guide on this blog.

FAQ

Is it legal to scrape Reddit?

Reading public threads is generally lawful, and Reddit’s own User Agreement plus Data API terms set the contractual rules. The real risk is reuse: republishing personal data, selling user content, or hammering the servers. Not legal advice; see Is Web Scraping Legal?.

Does the Reddit API cost money?

The Data API is free for non-commercial use within rate limits (about 100 requests/minute per app). Commercial access runs on Reddit’s paid tiers, priced from $0.24 per 1,000 calls after the 2023 changes. A personal script app costs nothing.

Why does reddit.com/.json return a login page?

Reddit gates anonymous access by IP reputation. Datacenter ranges and IPs that have scraped too hard get a 302 to a login wall, sometimes localized. An app-only OAuth token (Route 2) bypasses the roulette because you’re authenticating, not anonymous.

Can I get all comments from a huge thread?

With Magpie’s vertical, up to 200 comments and 10 nesting levels; deeper threads truncate. For complete trees on megathreads, paginate the Data API’s comment listing yourself, since the extractor doesn’t do pagination yet.

Can I scrape Reddit without the API?

Yes, two ways: the public .json endpoints when your IP is in Reddit’s good graces, or a browser render of the page, which got through the wall in my test today but yields prose and tables rather than typed comment fields.

Start with the credentials

Create the script app, export the three variables, and run the permalink command from Route 2 against any thread. It is the two minutes of setup that turn a reddit scraper from a coin flip into infrastructure. Then check the thread you actually care about, and if the record comes back with a comments array, you’re done.

Live-tested October 16, 2026. Reddit’s walls move; the commands don’t.