How to Scrape a Website That Requires Login

Every tutorial that teaches you to scrape website with login pages stops at the same happy path: post the form with requests.Session(), get a cookie, scrape on. What they skip: the site you actually need has MFA, and the form post died the day your company turned on Duo. Five routes still work, and today I ran every one of them.

In order of effort: copy a cookie, save a profile, script the form, capture the session by hand, or borrow your own logged-in browser. The test target was quotes.toscrape.com, the scraping practice site with a real CSRF-protected login form, so the numbers below are measurements, not documentation.

Five ways to scrape a website that requires login

Your situation	Route	Why this one
One page, right now	Copy the session cookie from your browser	Five minutes, no tools, dies when the cookie does
The same site every week, no MFA	Save a credential profile	One command per week to refresh, secrets stored masked
Login form with no MFA, fresh session each run	Script the form with browser actions	Self-contained, but one redesign away from breaking
MFA, SSO, CAPTCHA on the login	Capture the session interactively	You do the logging in; the tool reuses the cookies
Your everyday browser is already logged in	Point the scraper at it over CDP	Zero setup; the site sees your real fingerprint

You need a site you have an account on and permission to read. If the terms of service forbid automated access, the honest move is to ask for an API export instead. Everything below assumes your own account, your own data, human-paced requests.

Step 1: Copy a session cookie out of your browser

Log in normally. Open DevTools (F12), go to the Network tab, refresh, click the first document request, and find the Cookie request header. Copy the whole thing, or just the one cookie that matters, usually something like session=....

Then hand it to any scrape:

magpie scrape https://quotes.toscrape.com/ \
  --cookies "session=eyJjc3JmX3Rva2VuIjoxNzYw..."

What you should see: the authenticated page, not the login wall. When I pulled the demo site this way, the anonymous homepage weighed 11,064 bytes and the same URL with a valid session came back at 11,928 bytes with the nav switched from Login to Logout. That byte difference is your proof the session rode along.

Failure case: the page still shows logged-out content. Either the cookie expired, the site binds sessions to your IP or user-agent, or you copied a cookie that is not the auth one. Re-copy from a request made seconds before, not from a tab that has been open since Tuesday. Playwright’s storage_state and Crawlee’s login helpers are this same move with more ceremony; reach for them when the scrape already lives in Python.

Step 2: Store the cookie as a profile so it survives the week

A raw --cookies flag lives and dies with one shell command. A profile persists:

magpie profile save quotes-demo --cookies "session=eyJjc3JmX3Rva2VuIjoxNzYw..."
magpie scrape https://quotes.toscrape.com/ --as quotes-demo

profile ls lists what you stored with the secret masked; mine printed quotes-demo cookies=sess…f2lU. The store is profiles.json in the config directory, permission 0600. Explicit --cookies or --header flags override the profile field-by-field, which is how you test a refreshed cookie without re-saving.

One honest failure from my runs: this demo site is so small that the authenticated page extracts to 176 words of markdown, and the scraper’s quality gate refuses to ship anything under a 200-word floor. The anonymous page squeaks through at 210. On a real dashboard with a thousand words of table, the gate never fires. When it does fire on a thin internal page, --page-format screenshot is the workaround; the gate only judges extracted text, and a screenshot skips text entirely.

Step 3: Script the login form when there is no cookie yet

Some sites hand you a fresh session each login and invalidate old ones fast. Then the cookie-copy dance is a treadmill. The actions DSL types into the form in a real browser and captures whatever the login lands on. Save this as login.actions:

# run with: magpie scrape https://quotes.toscrape.com/login --actions login.actions
type #username demo
type #password demo
click input[type=submit]
wait-for a[href="/logout"]
wait 800

type selects the field and replaces its contents; wait-for blocks until the logout link exists, which is the cheapest possible “did I log in” check. Run it with --page-format screenshot and you get the authenticated page as an image: my run returned a 1265×1852 PNG showing the Logout nav and per-quote links that only render signed-in. That PNG is the proof, and for “archive what my account sees” jobs it is the whole deliverable.

Failure case one: the site rotates its login form and your selectors 404. That is the same disease as any scraped page, and the same cure as self-healing scrapers. Failure case two, the serious one: the form sits behind MFA or an SSO popup. The script types your password, the site asks for a TOTP code, and there is no selector for your phone. Which is why Step 4 exists.

Step 4: Log in by hand, let the tool capture the session

This is the route the scripted tutorials never mention, and in 2026 it is the main one. Magpie’s desktop app calls it Sessions, and the mechanics matter more than the name: the app opens a real browser window pointed at the site’s login URL, you log in yourself, and the moment it detects a sign-in it saves the cookie jar, sealed, into your OS keyring, or into a 0600 encrypted file where no keyring exists. The app never sees your password. 2FA works because you type the code. SSO popups work. CAPTCHAs work because you solve them, and the tool is not in the bypass business.

After that, every scrape of that host attaches the saved session automatically; the scrape form shows a “Session ✓” marker next to the URL field when one will ride along. Sessions die, and the app says so: when an attached scrape hits an auth wall, the Sessions row gains an expired badge and the run offers a Re-login button that opens a fresh capture window. Capture windows time out after five minutes, so nobody’s password sits in an open browser overnight.

Worth knowing: cookies are scoped per host, so a session captured for your analytics portal never attaches to a different domain, and the research feature reads pages with your login only when you opt in per domain, per run, at a throttled one page per ten seconds. The CLI and MCP surfaces refuse login-backed reads entirely; this is a desktop feature by design.

Step 5: Borrow your real browser over CDP

If Chrome is already logged in, skip capture entirely. Start Chrome with a debug port:

google-chrome --remote-debugging-port=9222

Then point the scraper at it instead of launching its own browser:

magpie scrape https://quotes.toscrape.com/ --cdp-url ws://127.0.0.1:9222

The desktop app exposes the same thing as a CDP endpoint field on the scrape form. The site sees your genuine fingerprint, your extensions, your day-old cookies. The cost is real: your main browser profile is now driving an automated fetch, so keep the pacing polite and don’t point it at anything you wouldn’t click by hand.

When the data is not in the page at all

Half the dashboards I’ve scraped render their numbers from a JSON endpoint the page calls after login. Scraping the DOM is the hard way. Capture those background responses directly:

magpie scrape https://app.example.com/dashboard \
  --as work --capture-xhr "api/metrics"

--capture-xhr takes Go regexps, records matching response bodies during a browser render, and hands them back alongside the page. The JSON is cleaner than any HTML extraction you were about to write. A static API curl with --header "Authorization: Bearer ..." often finishes the job without a browser at all.

Make it stick: the expiry question

Every route here rots on its own clock. Plan for it.

Route	Typical lifespan	When it breaks
Copied cookie	Hours to weeks	Page shows logged-out content; re-copy
Saved profile	Same cookie, same clock	Refresh with profile save under the same name
Scripted login	Fresh every run	Selector breakage, or MFA appears
Captured session	Days to months	Expired badge; press Re-login, log in once
CDP to your browser	As long as you stay logged in	You logged out; log back in

Two limits I hit and can’t fix for you: the CLI’s crawl command accepts no credentials at all, so authenticated crawling is a desktop-app job with a saved session, and sites on the never-list like LinkedIn are out by policy, not by bug. For whole-site collection behind a login, capture the session, then scrape the URLs you need one by one, or work from the dynamic-render playbook once you are through the wall.

Keep the request rate human. A logged-in scrape at 20 requests a second is the fastest way I know to convert a working session into a banned account. When a selector dies three weeks from now, the self-healing cache is the difference between a note in your log and a broken pipeline; and if the goal is clean article text rather than account data, the clean-text guide starts one step earlier in the pipeline.

FAQ

Is scraping behind a login legal? Logging in does not grant scraping rights; the account’s terms do. Reading your own data at human pace is a normal, defensible practice. Bulk-harvesting a site that forbids automation is how people lose accounts and occasionally more.

How do I scrape a website that requires 2FA? Scripted logins cannot type your TOTP code. Capture the session interactively: log in yourself in a visible window, let the tool save the cookies, reuse them on every run. Re-capture when the badge says expired.

Why does my session cookie stop working overnight? Some sites invalidate sessions on new IP, user-agent change, or a server-side rotation schedule. Copy the cookie from a fresh request right before the run, or move to interactive capture, which reuses the jar exactly as the browser held it.

Can I scrape with my normal Chrome profile? Yes, over CDP. Start Chrome with --remote-debugging-port=9222 and pass --cdp-url ws://127.0.0.1:9222. The scraper drives your real logged-in browser, fingerprint and all.

Does the site know I’m scraping while logged in? It sees one account requesting pages in a pattern no human produces. Keep delays human, respect robots.txt, and stay inside what your account is for. The login never makes scraping invisible.

Where to start

If the job is one page tonight, Step 1 and a browser DevTools window are enough. If you will run it weekly against a no-MFA site, save the profile and stop re-copying. The moment MFA enters the picture, stop scripting forms entirely; log in by hand once, capture the session, and let the expiry badge tell you when to do it again. That last route is the only one I have never had to debug at 2 a.m.