Real Estate Scraper: Collect Listings From Any Property Site

Five major property portals, one polite scraper, one afternoon. Zillow and Rightmove handed over listings; Redfin, Realtor.com, and Apartments.com refused in three different ways. That ratio is the first fact to check before buying any real estate scraper that promises “any property site”.

Anyone who has done web scraping for real estate data has met at least one of those walls. Here is the payoff: a tested matrix of which portals serve a robots-respecting scraper as of October 16, 2026, a two-step build that turns Zillow’s hidden listing feed into a clean CSV, and the legitimate data routes for the portals that lock their front doors.

What a real estate scraper can reach (tested October 2026)

Same tool, same home connection, one request per site, October 16, 2026. I used Magpie in default static mode, which respects robots.txt and throttles itself, and I read each record’s title and word count before believing any exit code.

Portal	Result	What came back
Zillow	Works	41 San Francisco listings in structured data, every one with price, beds, sqft, and URL
Rightmove (UK)	Partly	Real records: 60,459-result search page, 25 listing links with titles, descriptions, agents. No prices in the text
Realtor.com	Silent wall	Exit code 0, but the “record” is a 25-word block notice with a support email
Redfin	Blocked	Typed error, empty page, exit 8
Apartments.com	Blocked	Typed error, access denied, exit 8

Two working sites out of five, and one of those with a catch. Sites change their defenses monthly, so treat this as a snapshot, not a law.

The deeper lesson from the test: an exit code of 0 proves a request succeeded, not that you got data. Realtor.com returned HTTP success wrapping a refusal. That failure corrupts a dataset quietly, and it gets its own section below.

Before you build: terms, robots, and volume

Check two documents per site before the first request. robots.txt says which paths the site asks automated clients not to fetch; the terms of use say what you promised the site when you loaded a page. Zillow’s terms bar “any robot, spider, scraper or other automated means”, and Rightmove’s bar automated programs in nearly the same words.

The Zillow city page tested below is robots-allowed and terms-prohibited at the same time. That combination is common in real estate data scraping, public facts served on private contract terms, and it is exactly why I keep volume polite: one request per page, a few pages, nothing that looks like a harvest. I am not a lawyer and this is not legal advice; for a commercial project, get one.

You need 30 minutes, Go if you want to build Magpie from source, and Python 3 for the CSV step. No accounts, no API keys, no paid plan for anything in the first three steps.

Step 1: Scrape the Zillow city page

Zillow buries its listing data where scrapers rarely look. The visible page text shows zero prices; the structured data blocks in the HTML carry everything. Build the CLI from source, then pull one city page:

git clone https://github.com/motherlodelab/magpie.git
cd magpie && go build -o magpie ./cmd/magpie
./magpie scrape https://www.zillow.com/san-francisco-ca/ --out zillow.json

The record came back with 1,631 words of cleaned markdown plus 38 structured-data blocks (JSON-LD is the machine-readable metadata sites embed for search engines). One of those blocks is an ItemList with 41 entries. Every entry is a RealEstateListing with the full payload: address, price in USD, bedrooms, square footage, and the listing URL.

This is the single biggest trick in scraping Zillow: parse the JSON, ignore the prose. The full Zillow walkthrough covers the detail pages, robots map, and the Zestimate gap.

Failure case: repeat this too fast and Zillow’s bot wall answers instead. In a follow-up render during testing, the same session got a “quality blocked (empty)” error a few minutes after the first pull. That is Magpie refusing to hand over a challenge page, not a bug. Back off and come back later; there is no setting that makes heavy scraping safe, only slower ways to get blocked.

Step 2: Turn the listing JSON-LD into a CSV

The ItemList flattens into a spreadsheet with 30 lines of Python:

#!/usr/bin/env python3
"""Flatten Zillow's ItemList JSON-LD into listings.csv."""
import csv, json, sys

def flatten(entry):
    it = entry.get("item", {})
    offer = it.get("offers", {})
    prop = offer.get("itemOffered", {})
    addr = prop.get("address", {})
    street = addr.get("streetAddress", "")
    city = addr.get("addressLocality", "")
    region = addr.get("addressRegion", "")
    zip_ = addr.get("postalCode", "")
    floor = prop.get("floorSize", {})
    return {
        "address": f"{street}, {city}, {region} {zip_}".strip(", "),
        "price": offer.get("price", ""),
        "beds": prop.get("numberOfBedrooms", ""),
        "sqft": floor.get("value", "") if isinstance(floor, dict) else "",
        "url": it.get("url", ""),
    }

rec = json.load(open(sys.argv[1]))
if isinstance(rec, list):
    rec = rec[0]
rows = []
for b in rec.get("structured_data", []):
    if isinstance(b, dict) and b.get("@type") == "ItemList":
        rows.extend(flatten(e) for e in b.get("itemListElement", []))
with open("listings.csv", "w", newline="") as f:
    w = csv.DictWriter(f, fieldnames=["address", "price", "beds", "sqft", "url"])
    w.writeheader()
    w.writerows(rows)
print(f"{len(rows)} listings -> listings.csv")

Run it against the record from Step 1:

python3 zillow_to_csv.py zillow.json

Output from the actual run, abridged:

41 listings -> listings.csv
address,price,beds,sqft,url
"2070 46th Ave, San Francisco, CA 94116",1495000,3,1564,https://www.zillow.com/homedetails/...
"4077 23rd St, San Francisco, CA 94114",1595000,2,1248,https://www.zillow.com/homedetails/...

All 41 rows carried price, beds, and square footage. No cleaning passes, no selector maintenance, nothing to break when Zillow reshuffles a CSS class. More on turning records into spreadsheets, including JSON output, lives in the CSV and JSON guide.

Step 3: Run the same play on Rightmove

Rightmove’s robots.txt allows the search pages (it disallows contact forms, map views, and market-information paths, and it publishes sitemaps for coverage), so the same one-liner works:

./magpie scrape "https://www.rightmove.co.uk/property-for-sale/London.html" --out rightmove.json

What came back: a real record for a page advertising 60,459 results, with 25 listing links, property titles, descriptions, and agent names. What did not come back: a single pound sign. The prices render in the browser through JavaScript, and they were absent from the markdown in both static and rendered tests.

So the honest use here is discovery: build a list of listing URLs from the search page, then decide per detail page whether your terms and volume budget allow going further. If the search page ever comes back short, the sitemap at rightmove.co.uk/sitemap.xml is the alternative door.

The three portals that refused

Redfin returned an empty page and a typed error, exit code 8. Apartments.com returned access denied the same way. Those are the clean failures: the tool tells you the site refused, and you move on.

Realtor.com is the dangerous one: exit code 0, title field “realtor.com”, word count 25. The full text:

Please refresh the page to try again

If this issue persists, please copy the Request_ID below and contact [email protected].

That is a block page that returns HTTP success. If your pipeline logs “scraped 1 page, 25 words” and moves on, you now have a gap in your dataset that no crash ever flagged. Before you trust any scrape, check the record’s title and word count, or grep the text for phrases like “refresh the page” and “Request_ID”.

All three refused portals run PerimeterX-class bot protection; Magpie detects challenges and reports them, and it will not bypass them, by design. If someone tells you a rotating proxy fixes this, read when proxies actually help first: better IPs change who gets asked the CAPTCHA question, not whether the question gets asked.

Real estate data scraping without scraping the portal

For Redfin, Realtor.com, Apartments.com, and any portal that locks up, the legitimate routes beat the technical arms race:

• Redfin’s own data center. Redfin publishes monthly housing-market datasets as free downloads at redfin.com/news/data-center. Market-level metrics, not listing-level, but free, legal, and CSV-shaped.
• County records. Assessor and recorder offices sell or give away the ground truth: deeds, sale prices, tax parcels. Formats range from clean CSV dumps to 1990s FTP. Ugly, authoritative, and nobody can wall it off from you.
• MLS through Bridge Interactive. The Zillow Group program that replaced the retired 2021 Zillow API serves licensed brokerages and MLS partners, and its terms explicitly forbid reverse engineering or spidering Zillow. If you have an MLS relationship, this is the sanctioned feed; if you do not, it is not a workaround.
• Paid listing-data vendors. Companies like ATTOM sell licensed listing and ownership data with actual contracts. It costs real money, which is precisely why the free-but-blocked portals stay free-but-blocked for their own sites.

The pattern: the portals own the display layer, the counties and MLSs own the source. Any real estate project gets easier the moment you stop trying to win the display layer.

Which real estate web scraper fits the job

Match the tool to the frequency, not the marketing:

• One-off export, one site, today. The two commands in Steps 1 and 2. A script you can read beats a platform you must trust.
• Prebuilt scrapers (Apify actors, browser extensions). Fastest to start, and the extension route works fine on small portals. You inherit someone else’s selectors, so when the site redesigns, their actor breaks and your queue waits on a stranger’s fix.
• Recurring pulls, multiple sites, no infrastructure. A local real estate web scraper like Magpie handles the fetch, clean, extract pipeline, keeps a resumable crawl state in SQLite, and watches pages for changes with webhooks. It is a single Go binary, free under AGPL-3.0-or-later with a commercial option, and the desktop app runs 5 free scrapes before $49/month. Its limits are the ones in the matrix above: it works where sites serve robots-allowed pages, it detects and reports bot walls rather than defeating them, and there is no hosted version, your machine does the fetching.
• Blocked portals at scale, commercially. Hosted extraction vendors or licensed data feeds. You are paying for infrastructure and legal cover, not cleverness.

FAQ

Is it legal to scrape real estate listings?

Listing facts are generally not copyrightable in the US, but every major portal’s terms prohibit automated access, so scraping them risks a breach-of-contract claim regardless. Robots-allowed pages reduce one kind of complaint, not all of them. Courts have narrowed automated-access claims in recent years. This is context, not legal advice.

Does Zillow have an official API?

No public one. Zillow retired its Web Services API on September 30, 2021. Bridge Interactive handles data feeds for MLS-affiliated brokerages and partners, and the Zestimate is not included. The structured data on Zillow’s own pages is the only public machine-readable listing source.

Why does my scraper say success but return no listings?

The site served a block page with HTTP status 200, the way Realtor.com did in our test: exit code 0, 25 words, a support email. Check the record’s title and word count before storing it, and alert on records below a word threshold instead of on failures alone.

Can you scrape Redfin or Apartments.com?

Not through the front door from a home connection; both refused our polite static requests outright in October 2026. Redfin’s free data-center downloads cover market-level statistics. Listing-level Redfin data has no public API; partners get it through agreements.

Where to start

If your market is the US, build Step 1 against one Zillow city page and stop when the CSV has what you need; the JSON-LD does the work selectors used to do. If it is the UK, take the listing URLs from Rightmove and go site by site from there. For the walled portals, skip the proxy shopping and go to the county or the data center, the data is better anyway. And whatever you point at a portal, set a word-count floor on every record first; Realtor.com taught me that one the hard way, quietly.