How to Scrape a Website With ChatGPT (and Its Limits)

Paste a URL into ChatGPT, ask for the prices, and half the time you get a tidy table back. The other half you get confident numbers for a page the model never actually read. This guide covers the three methods that genuinely work when you scrape a website with ChatGPT, what each one costs, and the point where I stop using a chatbot and reach for a real scraper.

Your job	Use	Why
Read one public page right now	Paste the URL, ask ChatGPT to read it	Fast, zero setup, works on static pages
Extract specific fields from one page	Paste the HTML, prompt for the fields	The model can only work from what you gave it
A script you can run again	Prompt ChatGPT for Python + BeautifulSoup	You own the code; reruns cost nothing
The same extraction every week, over many pages	A scraper with a schema	ChatGPT has no memory between chats and no scheduler

Before you start. A ChatGPT account (free tier handles the one-page jobs; Plus is $20/month if you do this daily). For Method 2, Python 3 installed locally. For Method 3, an account with a hosted scraping provider. All three methods assume a public page: if the data sits behind a login, skip to the limits section.

Can ChatGPT visit a URL at all?

Signed-in ChatGPT can follow a link you paste and summarize the page. Treat it as a person browsing, not a scraper. It fetches one page at a time, on someone else’s schedule, through infrastructure that many sites flag. Cloudflare-fronted stores, news sites with hard paywalls, anything behind a login: the fetch fails or returns a challenge page.

The failure that matters is quieter. When the fetch fails, the model sometimes answers anyway, from training data or plausible-sounding guesswork. The table looks perfect. The prices are from 2023, or they never existed. For that reason alone, never use a ChatGPT summary as source data for anything you’d defend in a meeting. Ask it to quote the exact text it fetched, and spot-check one value against the live page.

Method 1: Paste the HTML and ask for the fields

This is the method I use most. It removes the fetching problem entirely: you hand the model the page, so it cannot invent one.

Step 1: Get the real HTML

Open the page, press Ctrl+U (view-source), select all, copy. Save it to a file if it’s big. For the examples here I’ll use books.toscrape.com, a fake bookstore that exists precisely to be scraped, so nobody’s terms of service get bent in the making of this tutorial.

One warning: view-source shows what the server sent. A JavaScript-heavy site (most React shops) shows a near-empty shell there. If the data you want isn’t in view-source, this method needs a different fetch, which is Method 2’s territory.

Step 2: Give the model a structured prompt

Don’t paste 100KB of HTML with “extract the books” on top. Paste the prompt first, then the HTML, and say what output you want:

Below is the HTML of one page from a bookstore site.
Extract every product as JSON with these fields:
- title (string)
- price (string, keep the currency symbol)
- rating (string, e.g. "Three")
- in_stock (boolean)

Rules:
- Use ONLY the HTML below. If a field is missing for a product, use null.
- Do not add products that are not in the HTML.
- Output a JSON array, nothing else.

<HTML>
...paste here...
</HTML>

The “only the HTML below” rule is not decoration. It’s the difference between extraction and improvisation.

Step 3: Check the output against the page

You should get a JSON array where the first entry matches the first product on the page, currency symbols included. Spot-check two entries against the live page every single time. If the model “found” 25 products on a 20-product page, you caught a hallucination, and the fix is a stricter rule line, not a re-roll.

What it costs

ChatGPT Plus runs $20/month, and pasting a 100KB page burns through your usable context fast on a busy day. The API is the honest way to price it: OpenAI’s small current model (GPT-5.4 Mini) costs $0.75 per million input tokens and $4.50 per million output tokens. At roughly 4 characters per token, a 100KB page is about 25,000 input tokens, so a bit under 2 cents per page. Fine for ten pages. For a thousand pages you’d pay around $19 in input alone, plus output, plus the retries when a response comes back truncated.

That last number is the real limit. Paste-into-chat has no loops, no retries, no schedule. It’s a one-page tool with a linear price curve.

Method 2: Have ChatGPT write the scraper

For anything repeatable, the right output is not data, it’s code. ChatGPT writes a working BeautifulSoup script in one prompt if you do the one part that actually requires skill: bring the selectors.

Step 1: Find the selectors yourself

Right-click a title on the page, choose Inspect, note that titles live in article.product_pod h3 a and prices in p.price_color. Thirty seconds of work. If you skip this, the model guesses selectors from thin air and you debug its guesses instead of your data.

Step 2: Prompt with specifics

Write a Python script using requests and BeautifulSoup.

Target: https://books.toscrape.com/catalogue/page-2.html
Extract from every product card:
- title: text of `article.product_pod h3 a` (use its title attribute)
- price: text of `p.price_color`
- rating: the second class of `p.star-rating` (e.g. "Three")

Output: print a JSON array to stdout.
Requirements: 1-second delay between requests, a real
User-Agent header, retry once on non-200 responses,
handle encoding explicitly.

The delay and User-Agent lines are not politeness flourishes. A scraper without them is the reason sites build bot walls.

Step 3: Run it and fix what breaks

python -m venv venv && source venv/bin/activate
pip install requests beautifulsoup4
python scrape.py

You should see JSON for all 20 products on the page. If you want the longer version of this workflow, the BeautifulSoup tutorial on this site walks the whole script line by line: How to Scrape With BeautifulSoup.

Failure case, and it’s the big one: the script works today and dies on the day the site ships a redesign. ChatGPT wrote code for the HTML you pasted, not for the site’s future mood. Expect to re-paste new HTML and ask for a fix a few times a year on any site that actively develops.

Method 3: Wire a custom GPT to a scraping API

Custom GPTs can call external APIs through actions, which means you can build a GPT whose action hits a hosted scraping service and returns clean HTML or markdown. The scraping service deals with JavaScript rendering, proxies, and blocks; the GPT just relays your request.

Setup is honest work: an OpenAPI schema for the action, an API key from the scraping provider. Costs stack on both sides, the ChatGPT subscription and the scraping API’s per-page pricing, which at typical providers starts around free-tier pennies and reaches hundreds of dollars a month at volume. If that’s the direction you’re heading, compare the hosted options first in Best Web Scraping APIs rather than picking the one with the loudest ads.

This method earns its keep in one case: a non-coder on your team needs recurring data and shouldn’t be pasting HTML into a shared chat. Everyone else is paying two subscriptions to avoid writing twenty lines of Python.

The limits of scraping a website with ChatGPT

Four walls, and no prompt engineering gets you through them.

Logins and paywalls. ChatGPT will not log into a site as you, and handing it your session cookies is a security incident, not a technique. Pages behind auth are out.

Scale. No loops, no scheduling, no resumption. A 2,000-page job is not 2,000 chats; it’s the wrong tool.

JavaScript rendering. If view-source is an empty shell, Methods 1 and 2 both starve. You need a headless browser, which means real infrastructure, not a chat window.

Silent hallucination. The worst limit is the one that doesn’t error. A blocked fetch can still produce a plausible table, and nothing in the output tells you it’s fiction. This is why every method above ends with the same instruction: verify against the live page.

The legal floor, in one sentence: public data scraped politely is generally fine where I sit, but terms of service, robots.txt, and personal-data law (GDPR and friends) all apply, and none of them care that a chatbot did the typing. If you’re working with Claude instead, the same methods and walls apply; we cover the differences in Claude Web Scraping.

When to graduate from ChatGPT to a scraper

The signal is repetition. The day you re-run the same prompt for the third time, you’ve built a pipeline by hand, badly, one chat at a time. That’s the moment for a tool whose job is extraction.

This is where I’ll be straight about the tool behind this site. Magpie does schema-driven extraction: you write the fields once as a JSON Schema, and every scrape returns typed JSON instead of prose. It can use OpenAI models for that extraction, with your own key.

The limit, plainly: Magpie does not plug into the ChatGPT app. Your ChatGPT Plus subscription does not work with it. It calls the OpenAI API, which means a separate key from platform.openai.com, metered per token, billed separately from ChatGPT. If the plan was “use what I already pay $20 a month for,” this isn’t that.

The workflow, since the bookshop example is already loaded:

First the schema, book.yaml, modeled on the repo’s own example:

$schema: "https://json-schema.org/draft/2020-12/schema"
type: object
additionalProperties: false
required: [name, price]
properties:
  name:
    type: string
    x-magpie: { css_hint: "h1", trim: true }
  price:
    type: string
    x-magpie: { css_hint: "p.price_color", trim: true }

Then the run, with your OpenAI key in the environment:

export MAGPIE_OPENAI_API_KEY=sk-...

./magpie scrape https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html \
  --schema book.yaml --provider openai --model gpt-5.4-mini

Output (abridged):

{"name": "A Light in the Attic", "price": "£51.77"}

Run that against 100 product URLs and you get 100 identical JSON objects, no prompt re-rolling. Where the LLM step is overkill, Magpie also extracts zero-LLM verticals like JSON-LD and RSS with no API key at all, and there’s a fuller tour in AI Web Scraper.

The honest cons: it’s a local tool, so there’s no hosted API to call from a phone; the core is AGPL, and the desktop app costs $49/month after 5 free runs; and it deliberately doesn’t touch bot-walled sites like Instagram or Amazon reviews, so if that’s your target, none of this applies.

FAQ

Can ChatGPT scrape a website directly? It can read one public page you paste and summarize it, but that’s browsing, not scraping. It can’t log in, paginate at scale, or schedule runs, and a failed fetch sometimes produces invented data. For anything repeated, have it write a script instead.

Is it legal to scrape a website with ChatGPT? The legality hinges on the data and the site, not the tool. Public data is generally safe to collect politely; personal data, paywalled content, and sites whose terms forbid bots are not. Check robots.txt and the site’s terms before any bulk job.

Why does ChatGPT give me wrong prices for a page? Usually because the fetch failed and the model answered from memory instead of admitting it. The fix is procedural: require it to quote the fetched text, and verify one value against the live page before trusting the rest.

Can ChatGPT scrape pages behind a login? No, and don’t paste your cookies or session tokens to try. You’d be leaking credentials into a third-party chat log. For authenticated pages, run a scraper locally under your own session.

What’s the cheap way to scrape a thousand pages? A script, not tokens. ChatGPT-written Python over 1,000 pages costs the price of electricity; the same pages through the paste-the-HTML method run about $19+ in input tokens at GPT-5.4 Mini pricing, before retries.

Pick by repetition, not by capability. A one-off page: paste the HTML and be done in five minutes. The same page every Monday: write the schema once, and let the machine re-run it without you.