How to Scrape Dynamic Websites (JavaScript-Heavy Pages)

curl the quotes page at quotes.toscrape.com/js/ and you get HTTP 200, 5.8 KB of HTML, and zero quotes. Open it in a browser and ten appear, written into the DOM by a script that loops over a JavaScript array. That gap is the entire problem when you scrape dynamic websites: the content you want does not exist until a browser executes the page’s code.

It is also a smaller problem than most tutorials make it. There are four routes, and the right one is usually the cheapest: parse data the server already embedded in the HTML, call the hidden API the page itself calls, render the page in a real browser, or drive that browser with clicks and scrolls. I ran all four today against three sandbox sites built for exactly this; every number below comes from those runs.

Four ways to scrape dynamic websites, cheapest first

What curl shows you	What it means	Your route
Full content in the HTML	The site is static; close this tab	requests + BeautifulSoup, no browser ever
The data sits inside a <script> tag	JavaScript renders data the server already sent	Parse the embedded JSON (Step 1)
An empty shell, but the data arrives from an XHR call	The page talks to a hidden API	Call the API directly (Step 2)
An empty shell, and nothing appears without a click or scroll	True client-side rendering	Browser, maybe with actions (Steps 3–4)

The test that sorts all of this out costs one command and thirty seconds.

Step 0: Diagnose with curl before installing anything

curl -s https://quotes.toscrape.com/js/ | grep -c 'class="quote"'
# 0

Zero quote cards in the HTML, ten in the browser: this page is JavaScript-rendered. Then check whether the data still rode along in some other form:

curl -s https://quotes.toscrape.com/js/ | grep -o 'var data = \['
# var data = [

That second grep changes everything. The quotes are in the HTML, just not in the DOM, which means no browser is needed at all. Two more diagnostics worth knowing: a grep -c that returns a challenge message (“Just a moment…”, “verify you are human”) means the site blocked your fetcher, and an empty shell whose source references id="root" or data-reactroot means a framework will mount the page client-side.

What curl cannot tell you is which XHR calls carry the payload. For that, open the page in your browser once, hit F12, and read the Network tab with the XHR filter on. Ten minutes there usually saves an afternoon of browser automation.

Step 1: Parse the data the server already sent

The /js/ sandbox embeds its quotes as a JavaScript array in a <script> tag. It is right there in the view-source. Extract the array and parse it:

import re, json, requests

html = requests.get(
    "https://quotes.toscrape.com/js/", timeout=30,
    headers={"User-Agent": "Mozilla/5.0"},
).text
m = re.search(r'var data = (\[.*?\]);', html, re.S)
quotes = json.loads(m.group(1))
print(len(quotes), quotes[0]["author"]["name"])

That prints 10 Albert Einstein, and the whole script, fetch included, takes about a second. No browser, no rendering engine, and it ran verbatim today.

This route is underused because people pattern-match “JavaScript site” to “need Selenium”. Check the script tags first. Many sites ship their initial state as JSON: sometimes as a tidy application/ld+json block, sometimes in a window.__INITIAL_STATE__ blob, sometimes (like this sandbox) in a hand-rolled variable. Two failure modes: the regex grabs too little or too much, so anchor it on the variable name and the closing ];; and some blobs are JavaScript object literals, not strict JSON (unquoted keys, trailing commas), which json.loads rejects. For those, demjson-style lenient parsers or a careful regex per field beat a browser.

Step 2: Call the API the page calls

The second sandbox, scrapethissite.com’s AJAX demo, shows Oscar data after you click a year button. The click triggers a plain GET request. I found it in the Network tab, and it is about as hidden as a screen door:

curl -s -o /dev/null -w "%{time_total}s %{size_download}B\n" \
  "https://www.scrapethissite.com/pages/ajax-javascript/?ajax=true&year=2013"
# 0.425026s 1183B

425 milliseconds, 1,183 bytes of clean JSON with film titles, nominations, and awards. Compare that with the browser route for the same data: 9.1 seconds of Chrome, and the HTML table still needs parsing afterward. Whenever you can hit the endpoint, the browser is the slow path.

Reading the page’s own JavaScript confirmed the endpoint and revealed a bonus: the page checks document.location.hash on load, so ...ajax-javascript/#2013 auto-loads that year’s data with no click at all. State in the URL is a scraping gift; it means you can often deep-link instead of scripting interactions.

The failure case here is authentication. Endpoints that require cookies or tokens will 401 for curl but work in your browser, where the session lives. The fix is to send the same cookies: export them from DevTools, or use a tool that keeps a login session per site. Finding a website’s hidden API walks the full discovery process, including signed requests.

Step 3: Render the page in a real browser

Some pages have no embedded data and no friendly endpoint: a framework mounts an empty div and builds everything client-side. Now you need a browser, and the only question is who drives it.

The tool I use daily for this is Magpie, a local Go binary (57 MB on disk, measured on the build I ran today) whose fetch pipeline starts static and escalates on its own. --render auto scores the HTML for shell signals, an SPA mount node, script density, “enable javascript” markers, and launches a real Chrome via go-rod when the score crosses 2. You can force either mode:

magpie scrape https://quotes.toscrape.com/js/ --render static
# Error: quality blocked (empty) for https://quotes.toscrape.com/js/

magpie scrape https://quotes.toscrape.com/js/ --render browser \
  --page-format screenshot --out quotes.png   # 183 KB PNG of the rendered page

Read that first line twice, because I think refusing is the correct behavior. A static fetch of this page produces a shell, and a tool that quietly writes you a shell produces the worst kind of bug: the pipeline runs green forever and the data was never there. Magpie’s quality gate compares extracted text against the raw body and fails loudly instead. The screenshot route bypasses extraction entirely and shows you what the browser actually rendered.

The DIY equivalent, if you would rather own the code, is Playwright:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://quotes.toscrape.com/js/", wait_until="load")
    page.wait_for_selector(".quote")
    print(page.locator(".quote").count())  # 10
    browser.close()

wait_for_selector beats wait_until="networkidle" here: network-idle hangs on pages that keep a socket open (analytics, websockets, long-polling). The costs are real either way. A Chromium instance is a couple hundred megabytes of RAM and seconds per page instead of milliseconds, so escalate only where the static fetch actually failed; the static toolkit has its own writeup in how to extract text from a website. The JavaScript rendering glossary entry covers what the browser is doing during those seconds.

Step 4: Drive the browser with clicks and scrolls

When the data arrives only after an interaction, infinite scroll, “Load more” buttons, tab switches, you need actions before extraction. Magpie takes them as DSL lines that run inside the browser session, then captures the final DOM:

magpie scrape https://the-internet.herokuapp.com/infinite_scroll \
  --render browser \
  --action 'scroll 2500' --action 'wait 700' \
  --action 'scroll 2500' --action 'wait 700' \
  --action 'scroll 2500' --action 'wait 700' \
  --action 'scroll 2500' --action 'wait 700' \
  --action 'scroll 2500' --action 'wait 700' \
  --page-format text

Measured today: the page loads with 602 words already present, and the five scroll-and-wait passes take it to 1,802 words in 14.2 seconds end to end. Each scroll hits the viewport bottom, the page fetches another batch, the wait gives it time to land. Same shape for click-to-load: on the scrapethissite page, --action "click [id='2013']" followed by --action 'wait-for .film-title' clicks the 2013 button and blocks until the loaded table exists.

Four things broke on me while testing that, and each fix cost seconds once known:

• IDs that start with a digit. #2013 is not a valid CSS selector (querySelector throws); the attribute form [id='2013'] works.
• Quoting inside --action. The DSL parses lines CSV-style, so an embedded double quote breaks flag parsing; single quotes in CSS attribute selectors avoid the whole class of problem.
• Scroll helpers. scroll bottom threw a rod eval error on one page; scroll 2500 (pixels) worked everywhere. When a verb fails on one site, try the explicit form.
• Waiting. wait-for is bounded by the fetch budget, about 20 seconds in my runs; a slow endpoint plus a long page-side delay (the sandbox adds an intentional 1.5-second loading spinner) can exceed it. Add explicit wait lines rather than hoping.

There is a third data source worth knowing about here: while the browser is open, the page’s own XHR responses can be captured directly. Magpie’s --capture-xhr 'regexp' records matching response bodies into the output envelope, which is the honest middle ground between full rendering and endpoint archaeology: you let the page make its calls, then keep the JSON instead of the HTML.

When even the browser route fights back

Here is the part tool marketing leaves out. On both sandbox pages, the browser rendered perfectly, screenshot-verified, and the text extraction still failed, because main-content extractors are tuned for prose, and a page whose body is a card list or a data table scores like boilerplate. Magpie’s answer is the same loud refusal:

Error: quality blocked (empty) for https://www.scrapethissite.com/pages/ajax-javascript/

I tried rescuing it with --include '#tableDisplay' (scrape only the table’s subtree) and got a different honest error: text and comments are not long enough. The tool will not hand me the table, by design. My honest read: for table-shaped payloads, this failure is the tool telling you to go back to Step 2, where the same data arrived as clean JSON in 425 milliseconds. When a screenshot or a rendered DOM is the goal (visual monitoring, proof of render), --page-format screenshot works fine on both pages.

The rest of the trade-offs, plainly: Magpie is local-only, so there is no hosted API and nothing runs in a browser tab; it needs Chrome or Chromium, which go-rod manages for you; robots.txt is respected and per-host throttling is automatic; bot challenges are detected and reported, never bypassed, so Cloudflare-hardened pages are out of scope by design. The download is free, you get 5 free runs per install, then it is $49/month. If your target is one of the walled platforms (Instagram, X, Amazon review text), no local tool in this article will help, and you should say so in your plan doc before writing any code.

Make it stick

Whatever collects the data, give the pipeline a length floor and make it exit non-zero below it. Today my first test run “succeeded”, exit code 0, and returned one paragraph: “The requested URL was not found on the server.” The sandbox page had died between my first fetch and my second. An empty-or-tiny output alarm is the difference between noticing in a minute and noticing in a month; cleaning scraped data covers the other checks worth automating.

For pages you monitor rather than harvest, flip the model: magpie watch <url> --every 1h --webhook https://your-hook.local/magpie tells you when a page changes instead of re-reading it all day, and it renders JavaScript the same way scrape does. Scheduling patterns for the general case are in automated web scraping.

FAQ

Can you scrape a website that uses JavaScript? Yes. In order of preference: parse data embedded in the HTML, call the XHR endpoint the page itself calls, or render the page in a headless browser and extract from the final DOM. Only the last needs a browser, and most sites never get that far.

How do I know if a website is JavaScript-rendered? curl -s <url> | grep -c "a phrase visible on the page". Zero matches plus a 200 status means the content is built client-side. Also check the HTML for id="root", data-reactroot, or __NEXT_DATA__, the fingerprints of React and Next.js apps.

Is scraping dynamic websites legal? The rendering technique changes nothing legally: public data, polite rates, robots.txt respected remains the accepted baseline, and copyright still applies to what you copy. This is not legal advice. Logins, paywalls, and personal data are different regimes entirely.

Why does my scraper see different content than my browser? Your browser carries cookies, a real viewport, and prior clicks; the scraper sees a cold first paint. A/B tests and lazy loading widen the gap. Render with a waited-for selector, and compare screenshots, not DOM counts, when debugging.

Do I need Selenium for JavaScript-heavy sites? Often not. Steps 1 and 2, embedded JSON and hidden endpoints, handle most sites without a browser. When rendering is unavoidable, Playwright is the current default over Selenium; Magpie wraps the same idea in a CLI with actions, capture, and a quality gate.

Before you pip install anything heavier than requests, spend ten minutes in the Network tab of your real browser. If the data crosses the wire as JSON, Step 2 gets it in half a second, and if it never does, at least you will know exactly which selector to wait for.