HTML Scraper: Get the Source and Data From Any Web Page

Ask for the HTML of a web page and you can get three different documents back: the source the server sent, the DOM a browser assembled from it, and the content someone cleaned out of the markup. On boring pages they differ by kilobytes; on JavaScript-heavy ones they barely overlap. I measured a demo site this morning: the server shipped 6,136 bytes of source, and after two scrolls in a real browser the page held 10,241 bytes the source never contained.

An html scraper is only useful once you know which document you want. This guide gets you each one: view-source, curl, a rendered fetch, then the repeatable versions of all three. Every command ran today from a home connection; every byte count comes from those runs.

Before you start. A browser and a terminal. curl is preinstalled on macOS and nearly every Linux; Magpie is optional for the first half and ships as a single static binary from github.com/motherlodelab/magpie. If you can run ls, you can run everything here.

One page, three documents

Server source. The bytes the web server returned. This is what view-source: shows and what curl saves. Scripts have not run; nothing has moved.

Rendered DOM. The live document after JavaScript executes. The browser added nodes, fetched data, mutated markup. This is what the Elements panel shows and what “Copy outerHTML” hands you.

Cleaned content. The article text, tables, and links after the nav bars, cookie banners, and footers are stripped. This is what parsers and LLMs actually want, and it is usually 60–90% smaller than either of the others.

Most arguments about scraper tools are two people asking for different documents. Decide which one you need before you pick a tool. If the gap between source and DOM is your real problem, the full toolkit lives in How to Scrape Dynamic Websites.

Get the source in your browser, first

Type view-source: before any URL in the address bar. Thirty seconds, zero tools, and you are looking at exactly what the server sent. Ctrl+S saves a copy with its assets; DevTools’ Elements panel plus right-click → Copy → outerHTML gives you the rendered DOM if that is what you needed.

Stop signs: if view-source shows a stub with a <script> tag and little else, the page is client-rendered and no amount of source-staring will find your content. If the browser tooling answers the question, close the tab and move on. For one page, fetched once, it usually does.

Scrape HTML from a website with curl

The terminal version of view-source:

curl -s https://example.com -o page.html && wc -c page.html
2359 page.html

Same 2,359 bytes I got from view-source: minutes earlier. Add -L to follow redirects.

The catch is that servers sort their visitors. A default curl user agent gets stubbed or refused by sites that only bother serving real browsers. One large retailer I test against refuses plain client fingerprints from my connection outright; switching the fetch to Chrome’s fingerprint flips its product pages from wall to full page. That’s fingerprint matching, not evasion. If a site answers with a bot challenge, that’s a stop sign, and a scraper that keeps knocking is a problem you created. Pages behind logins need cookies, and that’s a different setup: see How to Scrape a Website With Login. To scrape HTML from website URLs that repeat daily, skip ahead to the rerunnable setups below.

For a command-line html scraper with more judgment than curl, Magpie’s raw format returns the decoded response body untouched:

magpie scrape https://example.com --page-format raw

One deliberate behavior to know before you lean on it: the quality gate runs before raw bytes leave the program. A bot-challenge page or an empty extraction comes back as a typed error, never as bytes. Today it blocked three JavaScript shell pages I tried (quotes.toscrape.com/js/, demoblaze.com, and scrapethissite.com’s AJAX demo) with quality blocked (empty) and exit code 8. Raw mode is for debugging real pages, not a key to walls.

When the source is a shell: fetch the rendered DOM

The infinite-scroll demo at the-internet.herokuapp.com/infinite_scroll ships 6,136 bytes and ends its content at a “next page” link pointing to page 3. Here is the same URL with a real browser doing the scrolling:

magpie scrape https://the-internet.herokuapp.com/infinite_scroll \
  --page-format raw --render browser \
  --action "scroll 2000" --action "wait 1500" \
  --action "scroll 2000" --action "wait 1500" > dom.html
wc -c dom.html
10241 dom.html

The saved DOM now contains four pages of paragraphs and a next-link to page 5. None of those bytes existed in the server’s response. That is the whole trick: same URL, but the fetcher executed the page’s code before capturing.

Friction log from the same hour, because this is where scrapers die: the shortcut action scroll bottom threw TypeError: Cannot read properties of undefined (reading 'apply') on this exact page. Plain pixel scrolls (scroll 2000) worked. When an action dies, the run stops with the page’s own error, so try the other spelling before you conclude the site broke.

The gate still applies to rendered output. Demoblaze.com, a shopping-cart SPA demo, renders to under 200 words of extractable text, and Magpie refuses it rendered or not. If what you need is the shell itself, DevTools’ Copy outerHTML remains the honest tool for one page.

Take only part of the page: include and exclude

The html page format accepts CSS selectors. On Hacker News, stripping the score spans:

magpie scrape https://news.ycombinator.com --page-format html --exclude "span.score"

Every “N points” span is gone from the output; grepping the result for score_ or points returns zero hits. The reverse works too:

magpie scrape https://news.ycombinator.com --page-format html --include "table"

That produced 66,872 bytes from a 34,459-byte page, because include emits every matching subtree and HN’s tables nest, so inner tables repeat. Use narrow selectors, and prefer exclude as your first move. Both flags apply to the cleaned document, not the raw body.

An HTML web scraper you can rerun

One page fetched once is a lookup. The moment a task repeats daily, you want the same bytes saved the same way every time, and that is where an html web scraper earns the name.

Save instead of print:

magpie scrape https://example.com --page-format raw --out snapshots/$(date +%F)-example.html

Loop over a URL list, politely:

while read -r url; do
  magpie scrape "$url" --page-format raw --out "pages/$(echo "$url" | md5sum | cut -c1-12).html"
  sleep 3
done < urls.txt

For whole sections of a site, crawl instead of looping by hand. Magpie’s crawl respects robots.txt and paces itself per host; there is an --auto-throttle flag that backs off when a server pushes back, and a --corpus mode that writes one record per page:

magpie crawl https://quotes.toscrape.com/ --max-pages 10 --max-depth 1 \
  --concurrency 2 --corpus --out quotes.jsonl

Measured today: 10 pages, 0 errors, 22.5 seconds including a sitemap probe that 404’d and was skipped with a warning. Each record carries url, title, depth, and cleaned markdown, not raw source. That distinction matters: crawl is for collecting content, and for finding the pages in the first place there is a separate discovery command (magpie map, which prints a site’s robots-declared sitemap URLs). How to Find All Pages on a Website covers discovery in depth, and How to Scrape Without Getting Blocked covers the politeness mechanics.

HTML screen scraper: when pixels are the point

Screen scraping is the older term for grabbing what a screen shows rather than what markup says. It predates clean APIs and it still wins in three cases: content drawn to canvas, layout you need to verify visually, and pages where the markup is a hostile mess but the rendering is fine.

magpie scrape https://example.com --page-format screenshot --viewport 1280x800 --out page.png

That renders the page in a real browser and writes the PNG to disk. Because the screenshot path skips the cleaning pipeline entirely, a thin or JavaScript-shell page still gets its picture; the gate that blocks raw output does not apply here. An html screen scraper is also the honest fallback when a site’s DOM resists every selector: if you cannot parse it, at least you can prove what it looked like on a date.

HTML scraper online tools: when hosted is the right call

Search this topic and most results are hosted: BrowseAI’s page-source extractor, Apify’s page-source scraper, opengraph.io, and a dozen lookalikes. For a one-off public page they’re fine: paste a URL, get source, no install. An html scraper online also sidesteps whatever is blocking your home or office IP, which is sometimes the whole problem.

The costs are structural, not cosmetic. Your URL list and the page contents transit someone else’s servers. Free tiers meter you. And several of these tools fetch server-side only, which means they hand you the same JavaScript shell you were trying to escape.

Magpie is not in this category and does not pretend to be. It is a local binary; the pages it fetches never leave your machine, and there is no hosted API to call. If your scraping is occasional and public, hosted tools are reasonable. If it repeats, involves anything you would not paste into a stranger’s form, or needs rendered DOM, local wins.

FAQ

Is an HTML page scraper the same as a parser?

No. An html page scraper fetches the document: source, rendered DOM, or both. A parser turns that document into fields, rows, and typed values, which is a bigger job with its own tooling. The parsing side has its own guide; start there once the bytes are on disk.

How do I scrape HTML online without installing anything?

For one page, view-source: in the address bar. For a rendered copy, DevTools → Copy outerHTML. Hosted page-source tools work too, but you are sending URLs to a third party, and the server-side ones return pre-JavaScript source. For anything recurring, a local fetcher is the safer default.

Why does view-source show different HTML than DevTools?

View-source shows the server’s response, frozen. DevTools shows the live DOM after scripts ran and mutated it. On client-rendered sites the source is a stub and the DOM is the real page. Neither is wrong; they are snapshots at two different moments.

Can a site tell that I scraped its HTML?

Usually, yes: user agent, TLS fingerprint, request rate, and header order all separate scripts from browsers. Politeness is the practical answer: identify honestly, throttle, and honor robots.txt. A challenge page is a boundary, not an obstacle course.

Is scraping HTML legal?

Publicly available data is generally the safer ground, but courts weigh terms of service, copyright, and how you got in, and the case law has moved several times since 2021. The details and docket history are in Is Web Scraping Legal?; this is not legal advice.

Where to start

Classify the page before you pick a tool: view-source for thirty seconds, curl for the raw body, a rendered fetch if the source is a stub. Save the bytes with --out so you can diff tomorrow’s fetch against today’s. When it stops being one page, crawl. The shortest useful start:

magpie scrape https://page-you-actually-need.com --page-format raw --out page.html