How to Extract Clean Text and Main Content From Any Website

View-source on a news article I pulled this morning: 923 lines of HTML, 1,158 words of visible text, and a story that runs 773. The gap is navigation, cookie-dialog copy, and “related reads” modules. That gap is also the whole job when you extract text from website pages: keep the words a human reads, drop everything the layout invented.

Five methods below, ordered by effort: a paste-the-URL web form, two Python scripts, a headless browser, and a crawler. The extraction numbers come from runs I made today on that live article, not from documentation.

Pick your route before you install anything

Your job	Reach for	Why
One page, one time	An online text extractor	No install; the URL and content pass through someone else’s server
The same pages every week, in your code	BeautifulSoup, about 10 lines	You decide exactly what counts as text
Articles, main content only	trafilatura (a main-content extractor library)	Boilerplate stripping that survives redesigns better than any tag list you write
Pages that render with JavaScript	A browser: Playwright, or a tool that escalates on its own	The server’s HTML has no story in it yet
A whole section, hundreds of pages	A crawler with page caps	Per-page scripts rot somewhere around 20 URLs

To follow the Python steps: Python 3.9 or newer and pip install requests beautifulsoup4 trafilatura. That is the entire setup. I tested against trafilatura 2.3.1 today; the two functions used here are the same in current 2.x releases.

Step 1: Paste the URL into a website content extractor online

For a page you will read once, install nothing. Sites like urltotext.com and YourGPT’s webpage content extractor take a URL and return the page’s text in a textarea, ready to copy. Paste, wait a second or two, copy the result.

What you should see: the headline and body text, usually with the navigation gone. What you often get instead, in order of how much it annoys me:

• Nothing. The tool fetched server-side HTML and the page renders client-side. You got the skeleton of the page with no story in it.
• A challenge page. The site blocked the tool’s fetcher, and you got 35 words of “verify you are human” instead of the article. That happened to me today on Ars Technica with a plain HTTP fetch: a 2 KB page, 35 words, no story.
• Cookie-dialog text. The extractor grabbed the consent wall because that was the first text in the DOM.

One more thing worth saying out loud: the URL you paste, and the page content that comes back, travel through the tool operator’s servers. Fine for public articles. Do not paste internal dashboards.

The online route earns its keep for one-offs. The moment you are shuttling the same site through a web form every week, you have outgrown it.

Step 2: Write a ten-line text scraper with BeautifulSoup

This is the honest way to scrape text from website HTML you control or that serves plain HTML: a fixed tag kill-list, then get_text.

import requests
from bs4 import BeautifulSoup

r = requests.get(
    "https://www.npr.org/2026/10/15/nx-s1-5996678/midlife-wall-reset-test-guide",
    headers={"User-Agent": "Mozilla/5.0"}, timeout=30,
)
soup = BeautifulSoup(r.text, "html.parser")
for tag in soup(["script", "style", "noscript", "svg", "form",
                 "nav", "footer", "aside", "header"]):
    tag.decompose()
text = soup.get_text("\n", strip=True)
print(text[:400])

What you should see: the headline, then body text, one block per line. No tags, no script content. On the NPR page this exact script returns 944 words: the story, plus photo captions, plus section labels and a footer or two the kill-list missed.

Two failure modes, both cheap to fix:

• 403 or an empty response. The default python-requests user agent is on every blocklist ever written. The Mozilla/5.0 header in the snippet is there because I have been 403’d without it more times than I can count.
• Mojibake. Quotes come back as ’ because the page declared the wrong charset. Set r.encoding = r.apparent_encoding before you parse.

The deeper problem is the 944-versus-773 gap. A tag list is a guess about how one site names its wrappers, and every site guesses differently. Selectors beyond get_text are a whole skill; the BeautifulSoup tutorial covers them properly.

Step 3: Keep the main content, drop the boilerplate

Content extraction from web pages gets its own step because “everything except script tags” is not the same as “the article”. The general-purpose fix is main-content extraction, also known as boilerplate removal: the tool scores blocks of the page by text density, link ratio, and position, then keeps the cluster that looks like an article.

The text scraping tool I reach for first is trafilatura, and the reason is simple: its heuristics descend from Readability, which has been eating edge cases since Arc90 shipped it in 2009.

import trafilatura

downloaded = trafilatura.fetch_url(
    "https://www.npr.org/2026/10/15/nx-s1-5996678/midlife-wall-reset-test-guide"
)
print(trafilatura.extract(downloaded))  # plain text

Same page, three passes, real numbers:

Pass	Words	What you get
Every visible string (get_text, no cleanup)	1,158	Story plus nav, cookie text, related reads
Tag kill-list (Step 2)	944	Story plus captions, labels, leftovers
trafilatura	773	Headline through final paragraph

The trafilatura output starts at the headline and ends at the last paragraph of the story. That is the product. Two honest caveats: it also trimmed the photo captions, which on some pages are text you want, and when it cannot find a main content block it returns None rather than guessing. Check what it drops before you depend on it.

If you would rather stay closer to the DOM, python-readability and Mozilla’s Readability.js are solid. Both have a reputation, earned in my experience, for keeping more of what trafilatura trims, related-reads modules included. Pick one and learn its failure modes; do not run all three and merge.

Step 4: When JavaScript hides the text

A fetch-based web text scraper sees exactly what curl sees. On client-side-rendered sites that is a beautiful empty shell: status 200, megabytes of framework code, no article.

Test before you reach for anything heavier. Run curl -s <url> | grep -o "a sentence from the article". If curl cannot see the sentence, neither can requests, BeautifulSoup, trafilatura, or any online extractor.

When that grep comes back empty, you need a browser:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://example.com/quarterly-report", wait_until="networkidle")
    print(page.inner_text("body")[:400])
    browser.close()

One caveat on that snippet: networkidle hangs on pages that keep a connection open forever. If it stalls, switch to wait_until="load" plus page.wait_for_timeout(2000) and move on with your life. The JavaScript rendering glossary entry explains what is happening while you wait.

The cost is real: a Chromium instance takes a couple hundred megabytes of RAM and seconds per page instead of milliseconds. Escalate only where the static fetch failed. If scrolling or clicking reveals the content, that is an action step before extraction, and the FAQ below has the pattern.

Step 5: Extract content from website sections, not single pages

Somewhere past 20 URLs, the script-per-page approach rots: no resume, no rate limiting, one slow page stalls the whole run, and you have no idea which of the 200 files came back empty. This is crawler territory, and the caps matter more than the speed.

magpie crawl https://docs.python.org/3/tutorial/ --corpus \
  --max-pages 50 --out tutorial.jsonl

That command produces one JSONL record per page with the cleaned main-content markdown, zero LLM calls, no API key. robots.txt is respected and per-host throttling is automatic, which matters when the target is not your own site. Build a web crawler walks through the general case if you would rather assemble one from Scrapy or Colly; my only hard rules are page caps, a resume story, and politeness defaults you could explain to the site’s admin with a straight face.

What a local web content extractor adds

Everything above is code you maintain. A command-line web scraper like Magpie bundles the same pipeline into one binary, and this is the part where I disclose that I use it daily, so weight that as you like.

The single-page case, the actual command I ran for this article:

magpie scrape https://www.npr.org/2026/10/15/nx-s1-5996678/midlife-wall-reset-test-guide \
  --render static --page-format text

2.9 seconds later, 4,401 bytes of text (abridged):

Hit a midlife wall? This test identifies your strengths and what matters to you

Ready for a midlife reset? First, take stock of your strengths. Westend61/Getty Images

When's the last time you asked yourself: What do I love to do? What makes me feel alive?

The default output is GitHub-flavored markdown with a JSON-LD sidecar, which is the shape you want if the text is headed into an LLM pipeline; markdown versus HTML for LLMs covers why. When the main-content guess needs a nudge, --include and --exclude take CSS selectors. When the page needs a browser, --render auto escalates on its own instead of returning a shell. Around that core: watch mode with webhooks for pages you monitor, batch runs up to 100 URLs with no LLM in the loop, login sessions for pages you have legitimate access to, and a 13-tool MCP server so coding agents can drive it.

The cons, plainly: it is a local binary, about 60 MB, so there is no hosted API and nothing runs from a browser tab. The download costs nothing; you get 5 free runs per install, then it is $49/month. robots.txt is respected and bot challenges are detected and reported, never bypassed; Instagram, X, and Amazon review text are out of scope by design. If your job is one of those platforms, none of this article’s local tools will save you.

Where “extract text from website” goes wrong

• Regex to extract web page content. HTML is not a regular language. The first time the CMS adds a class attribute, your pattern breaks, and it breaks silently.
• Invisible text. get_text() happily returns display: none blocks: hidden spam, template stubs, preview text. Spot-check for strings you cannot see on the rendered page.
• Boilerplate that reads like content. “Related stories” sections and author bios pass every naive filter. Only main-content scoring catches them consistently.
• Pagination. A five-page article extracts as five half-stories unless you follow the “next” links and concatenate.
• Selector rot. Redesigns change wrapper class names; scrapers built on those selectors die without an error. Magpie’s answer is a selector cache that notices rising null rates and re-synthesizes; the trade-offs are in LLM extraction versus CSS selectors. The DIY answer is alerting on output length.

Make it stick

Put the script in cron and make it fail loudly: exit non-zero when the output drops below a length floor, because an empty file at 3 a.m. is a bug you want to hear about. Add a leak check, too. If “Subscribe” or “Sign in” starts appearing in your extracted text, your filter died and nobody told you.

When the job is noticing changes rather than collecting pages, flip it around: magpie watch <url> --every 1h --webhook https://your-hook.local/magpie tells you when a page moved instead of re-reading it all day. Whatever collects the text, cleaning scraped data is the step after extraction, and automated web scraping covers the scheduling patterns in depth.

FAQ

Is it legal to extract text from a website? Public pages, polite rates, robots.txt respected: generally treated as acceptable, and copyright still applies to what you copy. This is not legal advice. Paywall circumvention and personal-data scraping are different legal animals entirely.

Do you need a web content extractor download at all? For one page, no: an online extractor is faster. For recurring jobs, internal URLs, or anything past about 20 pages, install a local tool. You lose the per-page quotas and the third-party hop, and your output lands directly in files you own.

Is a web content extractor free of charge good enough? For static public pages, yes: BeautifulSoup and trafilatura are free and open source. Hosted free tiers run out, Firecrawl’s is 1,000 credits a month, so light use is covered and daily crawls are not. Past that you pay in quota or in setup time.

How do I extract content from webpage sections that load on scroll? Infinite scroll needs a browser plus actions before extraction. In Playwright, loop page.mouse.wheel() until document height stops growing. For click-to-load patterns, Magpie’s actions DSL does it inline: --action 'click .load-more' runs before the page is captured.

What’s the fastest way to extract website content on a schedule? Cron plus the Step 3 script for a handful of URLs. A capped crawl for whole sections. Watch mode with a webhook when you want change alerts instead of re-reading entire pages every morning.

Point trafilatura at five URLs from your real target tonight. A None result usually means JavaScript or a challenge page, so jump to Step 4. A word count suspiciously close to the naive pass means boilerplate is still riding along, and a capped crawler with an --include selector will beat any amount of additional guessing.