How to Scrape Data From Any Website (No-Code and Code)

The Hacker News front page I pulled an hour ago is 34,513 bytes of HTML holding 30 perfectly structured rows: rank, title, link, score, comment count, every one. That gap between what you see in the browser and what you can hold in a spreadsheet is the entire reason people want to scrape data from website pages, and closing it takes less code than you’d expect. Six methods below, ordered by effort from point-and-click to a capped crawler, and every timing and failure in this piece comes from runs I made today.

Pick by what you need back

Your job	Reach for	The catch
One page, one time, nothing installed	A spreadsheet function (Step 2)	One page per pull; the site can block the fetcher
The same fields every week, no code	A trained no-code tool (Step 1)	Quotas, and workflows break quietly on redesigns
Fields inside your own program	Ten lines of Python (Step 3)	You own the selectors when they rot
Pages that only exist after JavaScript runs	Any method above plus a browser (Step 4)	Seconds per page instead of milliseconds
Whole sections, hundreds of pages	A capped crawler (Step 5)	Caps and politeness are not optional
A pipeline that survives site redesigns	A local scraper with self-healing selectors (the Magpie section)	Local binary, no hosted API

To follow the Python steps: Python 3.9 or newer and pip install requests beautifulsoup4. I tested against Python 3.12.3, requests 2.32.5, and beautifulsoup4 4.14.3 today.

Step 1: Point and click with no-code software

If you want software to scrape data from website pages without writing any, the category works like this: the tool opens the page in an embedded browser, you click the first item in a list, it infers the repeating pattern, and you export to CSV or Excel. Setup runs 5–15 minutes for a simple listing page.

Three I’d shortlist, with the numbers that decide it:

• Octoparse (desktop app, Windows/macOS). Free plan is permanent: 10 scraping tasks, 1 device, local extraction, up to 50,000 rows exported per month. Paid starts at $69/month. Per its pricing page, checked today.
• Browse AI (cloud). You record yourself clicking fields and it builds a “robot” that re-runs on schedule and emails you when values change. Free tier: 50 credits a month across 2 sites; paid from around $19/month billed annually, per mid-2026 reviews of its pricing page.
• ParseHub (desktop app). Free plan: 200 pages per run, 5 public projects, 14-day data retention. Paid jumps to $189/month for 10,000 pages per run. And its own pricing page ships as 2,627 bytes of JavaScript shell; Step 4 covers why that matters, because it cost me an extra step today.

What you should see: a clean table preview inside the tool, then a CSV that opens without fixing. What you get instead when it goes wrong: the tool trained on 3 items out of 30, because you clicked one item without teaching it the list pattern. Re-train by clicking the second item too: every one of these tools needs two examples to generalize.

The honest cons. Cloud tools route your URLs and your data through their servers, so keep internal dashboards out of them. Free tiers are sized for evaluation, not weekly jobs; Browse AI’s 50 credits vanish in four weekly runs of one robot. And a trained workflow dies silently when the site redesigns; the browser-extension breed of website data grabber fails the same way, with even less visibility. If the output suddenly looks half empty, assume the site moved before you assume you did.

Step 2: Get website data straight into a spreadsheet

For a one-page pull, a spreadsheet function beats every installer. In Google Sheets:

=IMPORTXML("https://news.ycombinator.com/", "//a[@class='titleline']")

It returns the 30 front-page titles, one per row, if Google’s fetcher gets through: HN wraps each story link in a titleline class, markup verified this morning. The two ways it fails are below. The first argument is just a link in a cell, which makes this the cheapest way to extract data from link lists you already keep: put URLs in column A, point the formula at A2, drag down. To extract information from website tables and lists, the sibling call is IMPORTHTML.

Two limits that decide everything. The function fetches server-side, so a page that renders with JavaScript returns #N/A and no amount of formula cleverness fixes it. And popular sites block Google’s fetcher outright, which looks identical to a broken formula. Test the URL in an incognito tab with JavaScript disabled: if you can’t see the data there, neither can your sheet.

The deep version of this route (refresh schedules, pairing formulas, Excel’s Power Query equivalent) lives in the dedicated Google Sheets web scraping guide, and Excel’s side is covered in importing website data to Excel.

Step 3: Scrape data from website HTML in ten lines of Python

When the fields need to land in your own code, or the spreadsheet format is the wrong shape, this is the shortest honest version. To scrape information from website pages you fetch, parse, and pull fields by selector:

import csv
import requests
from bs4 import BeautifulSoup

r = requests.get("https://news.ycombinator.com/",
                 headers={"User-Agent": "Mozilla/5.0"}, timeout=30)
soup = BeautifulSoup(r.text, "html.parser")

rows = []
for tr in soup.select("tr.athing"):
    link = tr.select_one(".titleline a")
    sub = tr.find_next_sibling("tr")            # score/comments live here
    score = sub.select_one(".score")
    comments = sub.select("a")[-1].get_text()
    rows.append({"title": link.get_text(), "url": link["href"],
                 "score": score.get_text().split()[0],
                 "comments": comments.split()[0]})

with open("hn.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=rows[0].keys())
    writer.writeheader()
    writer.writerows(rows)

Ran today against the live front page: 30 rows in 0.80 seconds, a 3,534-byte CSV, no failures. The first three rows, abridged:

title,url,score,comments
Muse Spark Open,https://research.meta.ai/blog/muse-spark-open,663,364
RPKI now protects 70% of internet routes,https://ipinfo.io/blog/rpki-protects-70-internet-routes,67,18
Business accounts will consent to Meta scraping their websites...,https://www.niemanlab.org/2026/10/...,79,24

Each selector is a claim about the page’s structure, and that’s the deal you sign: tr.athing marks story rows, .titleline a the link, and the score sits in the row after the story row, which is why find_next_sibling is there. Scrape data from webpage HTML long enough and one of those claims breaks. The failure cases below are where the real time goes.

Two you’ll hit this week:

• 403 Forbidden. The default python-requests user agent is on every blocklist ever compiled. Send a browser one, like the snippet does.
• Empty output on a page that obviously has data. JavaScript rendering. Don’t debug your selectors; jump to Step 4.

Step 4: When the page renders with JavaScript

ParseHub’s pricing page is today’s example, chosen because it embarrassed me. curl fetched 2,627 bytes, status 200, and a search for the word “free”, which definitely appears on that page in a browser, found nothing. The server sent an application shell; the data arrives after JavaScript builds the page. Try to extract data from webpage markup at that point and every plain-HTML method returns the same thing: a clean, confident, empty result.

The test takes ten seconds and should precede every method above:

curl -s "https://www.parsehub.com/pricing" | grep -oi "200 pages per run"

No match means the data isn’t in the server HTML, and you escalate to a real browser. Playwright or Selenium drive one from code; Octoparse and ParseHub embed one and the cloud tools run one for you. For the command-line route, tools that escalate automatically are the pragmatic pick: magpie scrape https://www.parsehub.com/pricing --render auto fetched the full plan table (free tier: 200 pages per run, 5 public projects) in one step this morning, no flags changed.

That one line is the whole idea; the browsers cost RAM and seconds, so escalate only where the static fetch failed. The full patterns (waiting for network idle, scrolling, clicking before extracting) are the territory of scraping dynamic websites.

Step 5: Hundreds of pages need a crawler, not a loop

Web scraping pages in bulk with for url in urls: scrape(url) works right up until it doesn’t: no resume after a crash, no rate limiting, no record of which of the 300 files came back empty, and one slow page stalls everything. Past roughly 20 URLs, use a crawler with three non-negotiables: page caps, a resume story, and politeness defaults you could explain to the site’s admin:

magpie crawl https://docs.python.org/3/tutorial/ --corpus --max-pages 50 --out tutorial.jsonl

One JSONL record per page, robots.txt respected, per-host throttling automatic, zero LLM calls. If you’d rather assemble the crawler yourself, building a web crawler walks Scrapy and Colly through those same three requirements, and finding all pages on a website covers the discovery step that decides your URL list in the first place.

What a local web extractor adds

Everything so far is either a rented workflow or code you now maintain. The third option is a local tool: Magpie is a single Go binary that bundles fetch, browser escalation, and extraction, and I’ll disclose I use it daily; weight that however you like.

The same HN front page:

magpie scrape https://news.ycombinator.com/ --render static --page-format text

2.21 seconds, slower than the raw Python script and worth saying so. The extra 1.4 seconds buys the cleaning layer: boilerplate stripped, output as a markdown table instead of raw HTML you’d parse again. Output, abridged:

| 1. | Muse Spark Open (meta.ai) |
| | 663 points by bigwheels 9 hours ago | 364 comments |
| 2. | RPKI now protects 70% of internet routes (ipinfo.io) |
| | 67 points by emot 2 hours ago | 18 comments |

Where it earns its keep is the parts nobody wants to own: a selector cache that watches its own null rate and re-synthesizes selectors when a redesign breaks them; schema-driven extraction of structured fields through any LLM provider, including local models via Ollama; watch mode with webhooks when monitoring beats re-collecting; and CSV or JSON exports that drop straight into scraping a website to CSV or JSON pipelines.

The cons, plainly: it runs on your machine, so there’s no hosted API and no browser-tab version. Download is free with 5 runs per install, then $49/month. robots.txt is respected and bot challenges are detected and reported, never bypassed — and the platform list it won’t touch (Instagram, X, Amazon review text, and friends) applies to every method in this article, no-code ones included.

You cannot scrape a database from a website

Searches for “scrape database from website” deserve a straight answer: there is no database to reach. A site’s data store sits behind its application, and every tool on this page, browsers included, sees the same rendered surface you do. Queries like “extract database from website” hit the same wall: the closest thing to a database a site exposes is its rendered pages, plus whatever JSON its APIs return. What people usually mean by a website database extractor is the output side: scrape many pages, deduplicate, and land the rows in your store, a CSV today and a SQLite file next month. That’s a scraper plus a destination, not a different technique. If a vendor implies direct access to a site’s backend data, close the tab.

Where web data scraping goes wrong

• Selector rot. HN renamed its title class from .storylink to .titleline in a routine deploy a few years back, and most tutorial scrapers written before it died without an error: they returned empty lists, which look like success. When a scrape comes back short, check for silent empties before anything else.
• Success-shaped failures. An empty result and a truncated result both exit zero. Put a floor under your output (row count, byte count) and make the script fail loudly through it.
• Pagination. “Page 1 scraped fine” is five half-datasets wearing a trench coat when the listing runs 40 pages deep. Handling pagination covers the follow-the-next-link pattern.
• Logins and rate limits. Pages behind a login need session handling, not heroics; scraping a website that requires login shows the cookie approach. And if the target isn’t your site, throttling is a cost of doing business; scraping without getting blocked is the politeness playbook.

Make it stick

Whatever route you pick, the shape of web data extraction is identical: fetch, pull fields, land rows, then keep them landing. Put the working script in cron the same day you write it, and wrap it with the two checks that matter. First, a floor on row count. Second, a leak check: if “Sign in” or “Subscribe” starts appearing in your output, your extraction died and nobody told you. For schedules fancier than cron, automated web scraping covers the patterns, and whatever lands in your CSV will need cleaning and deduplication before it’s analysis-grade.

Tonight’s assignment: pick the real page you need data from, run the ten-second curl test from Step 4, and let the result choose your method. Server HTML with your fields in it means the Python script or the spreadsheet function, done in an evening. A JavaScript shell means browser escalation from day one, and no amount of selector tweaking will save you from that fact.

FAQ

What’s the best way to scrape data from a website? For one page, a spreadsheet function or a no-code tool’s free tier. For recurring jobs or anything past about 20 pages, the best way to extract data from website pages is the ten-line script or a capped crawler, because you own the failure modes instead of renting them.

Is it legal to scrape data from a website? Public pages, polite rates, robots.txt respected is the widely accepted baseline. This isn’t legal advice, and copyright applies to whatever you copy, so treat permission to extract information from a website as a question of its terms, not of your tooling. Paywall circumvention and personal-data scraping are different legal categories entirely.

Can I extract data from any website without coding? For static public pages, yes: the Step 1 tools and the Step 2 function need no code. JavaScript-heavy sites are where no-code tools earn their price, since they run real browsers. Login-walled pages work only where you have legitimate access.

How do I scrape information from website tables into Excel? If the table is in the server HTML, Excel’s Power Query (Data → Get Data → From Web) or Google Sheets’ IMPORTHTML pulls it directly. For anything repeating-but-not-a-table, or JavaScript-loaded, use a no-code tool that exports XLSX natively.

What’s the best way to web scrape pages that need JavaScript? Skip plain fetchers entirely; they see the empty shell. Use a browser-driven tool: Playwright from code, --render auto from the command line, or any cloud tool from Step 1. Budget seconds per page, and verify the data actually arrived before parsing; that goes double when you extract information from web apps, where a widget can fail without an error.