Go (Golang) Web Scraping Tutorial

Go’s standard library ships an HTTP client with connection pooling and timeouts, so the floor for Golang web scraping is about twenty lines. The ceiling is a polite, concurrent crawler, and the middle is full of sites that 403 Go’s default User-Agent or render their content with JavaScript. Every step below ran today on Go 1.26.5 against quotes.toscrape.com, a sandbox built for scraping practice, and I kept the timings: 1.2s for the first page, 3.4s to crawl nine of them, 3.6s to render the JavaScript one.

You get three working programs, one measured gotcha per step, and a decision rule for when to stop writing Go and run a tool instead.

What you need

• Go 1.26 or newer (tested on 1.26.5, linux/amd64)
• A throwaway module: mkdir scrape-demo && cd scrape-demo && go mod init demo
• Three libraries, added with go get as they appear below: goquery v1.13.0 (BSD-3), Colly v2.3.0 (Apache-2.0), and rod v0.116.2 (MIT)

The sandbox is quotes.toscrape.com, a site that exists to be scraped: ten quotes per page, pagination, and a JavaScript variant at /js/ for the hard case.

The Golang web scraping baseline: net/http and goquery

Fetch with net/http, parse with goquery, which puts CSS selectors on top of golang.org/x/net/html:

package main

import (
	"fmt"
	"net/http"
	"time"

	"github.com/PuerkitoBio/goquery"
)

func main() {
	start := time.Now()

	res, err := http.Get("https://quotes.toscrape.com/")
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()
	fmt.Println("status:", res.Status)

	doc, err := goquery.NewDocumentFromReader(res.Body)
	if err != nil {
		panic(err)
	}
	fmt.Println("quotes on page:", doc.Find("div.quote").Length())

	first := doc.Find("div.quote").First()
	fmt.Println("first quote:  ", first.Find("span.text").Text())
	fmt.Println("author:       ", first.Find("small.author").Text())
	fmt.Printf("total: %v\n", time.Since(start).Round(time.Millisecond))
}

Output from today’s run:

status: 200 OK
quotes on page: 10
first quote:   “The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”
author:        Albert Einstein
total: 1.236s

That 1.236s includes the TLS handshake and the parse. Cached connections after this one are faster.

One failure case to know early: http.Get identifies your scraper as Go-http-client/1.1, and some sites refuse that on sight. I tested Wikipedia with curl while writing this: both Go default UA strings return 403, a Chrome UA returns 200. The fix is to send the request through http.NewRequest and set the header yourself:

req, _ := http.NewRequest("GET", url, nil)
req.Header.Set("User-Agent", "my-scraper/0.1 ([email protected])")

Use a UA that names your scraper when the site’s terms allow scraping at all. If a site still blocks you with a real UA, the block is probably TLS-fingerprint based, and no header will fix that.

Crawl a section with Colly

goquery handles one page. Colly adds the crawler: a queue, per-host rate limits, and callbacks that fire on matching HTML.

package main

import (
	"fmt"
	"time"

	"github.com/gocolly/colly/v2"
)

func main() {
	start := time.Now()
	quotes := 0
	pages := 0

	c := colly.NewCollector(
		colly.AllowedDomains("quotes.toscrape.com"),
		colly.MaxDepth(10),
	)
	err := c.Limit(&colly.LimitRule{
		DomainGlob:  "quotes.toscrape.com",
		Delay:       500 * time.Millisecond,
		RandomDelay: 250 * time.Millisecond,
	})
	if err != nil {
		panic(err)
	}

	c.OnHTML("div.quote", func(e *colly.HTMLElement) {
		quotes++
	})
	c.OnHTML("li.next a", func(e *colly.HTMLElement) {
		if pages < 9 {
			e.Request.Visit(e.Attr("href"))
		}
	})
	c.OnRequest(func(r *colly.Request) {
		pages++
	})
	c.OnError(func(r *colly.Response, err error) {
		fmt.Println("error:", err)
	})

	if err := c.Visit("https://quotes.toscrape.com/"); err != nil {
		panic(err)
	}
	c.Wait()
	fmt.Printf("pages: %d, quotes: %d, total: %v\n",
		pages, quotes, time.Since(start).Round(time.Millisecond))
}

pages: 9, quotes: 90, total: 10.103s

Nine pages, ninety quotes, about 1.1s per hop. Most of that is the delay I asked for: 500ms fixed plus up to 250ms random between requests to the same host.

Two things in that code matter more than the callback plumbing.

Colly reads robots.txt by default. It imports temoto/robotstxt and refuses disallowed paths; IgnoreRobotsTxt: true is the opt-out, and you have no business using it on sites you don’t operate. This makes Colly more polite out of the box than most tutorials imply.

The limit rule is where I lost ten minutes so you don’t have to. My first draft used DomainGlob: "quotes.toscrape.com/*". The glob runs against the host, not the URL, so that pattern matched nothing, c.Limit returned no error, and the rule silently did nothing: same nine pages in 3.43s, zero delay. A missing error return is not the failure mode here; a glob that never matches is. If your politeness settings matter, print the hop timing once and confirm the delay is real. For the full arithmetic on delays, 429s, and Retry-After, see how to scrape without getting blocked.

JavaScript pages: where net/http goes blind

The same sandbox has /js/, where a script writes the quotes into the DOM after load. The baseline program reports quotes on page: 0. The HTML arrived; the content didn’t.

First response: don’t reach for a browser yet. View-source the page and check DevTools’ Network tab. Often the data is embedded in a <script> tag or arrives from an XHR you can call directly, which turns a 3.5s render back into an ordinary GET. The decision tree is in how to scrape dynamic websites.

When the page genuinely needs a browser, the two options that matter in Go are rod and chromedp. Both drive Chrome over the DevTools protocol. Rod (v0.116.2, MIT) ships a downloader that caches a browser build under ~/.cache/rod; chromedp expects you to bring an installed Chrome. I use rod, so that’s what the demo runs:

package main

import (
	"fmt"
	"net/http"
	"time"

	"github.com/PuerkitoBio/goquery"
	"github.com/go-rod/rod"
)

func main() {
	url := "https://quotes.toscrape.com/js/"

	// What net/http sees: an empty shell.
	res, err := http.Get(url)
	if err != nil {
		panic(err)
	}
	doc, _ := goquery.NewDocumentFromReader(res.Body)
	res.Body.Close()
	fmt.Println("net/http quotes:", doc.Find("div.quote").Length())

	// What a browser sees.
	start := time.Now()
	page := rod.New().MustConnect().MustPage(url)
	page.MustWaitLoad().MustElement("div.quote")
	count := len(page.MustElements("div.quote"))
	first := page.MustElement("div.quote .text").MustText()
	fmt.Println("rod quotes:     ", count)
	fmt.Println("first quote:    ", first)
	fmt.Printf("render cost: %v\n", time.Since(start).Round(time.Millisecond))
}

net/http quotes: 0
rod quotes:      10
first quote:     “The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”
render cost: 3.572s

That 3.572s is a cold start: browser process launch, page load, then the query. Warm runs are faster, but a browser render is always the slow path. Budget for it, close your pages, and reuse the browser across URLs if you’re rendering more than a handful. Rod’s Must* helpers panic on error, which is fine for a script and wrong for a service; switch to the explicit-error variants before anything ships.

What the tutorials skip: selectors rot

Every selector in this tutorial is a promise the site can break. div.quote works until a redesign renames it, and no compiler will catch that. Three things actually help:

• Fail loudly. Treat zero results as an error, not an empty dataset. My colly OnError callback prints, but a zero-result page should wake someone up, not quietly export an empty CSV.
• Watch for changes. A monitoring pass that diffs the page catches breakage the day it happens, not the month after.
• If you maintain scrapers for a living, look at self-healing scrapers, which re-synthesize broken selectors instead of just reporting them.

Login-gated sites add a second tax: cookie handling and session persistence. That’s a separate problem with its own walkthrough on scraping sites that require login.

When to stop writing Go and run Magpie

Magpie is itself a Go program, so this section is first-hand. It’s a single static binary, pure Go, no CGO, licensed AGPL-3.0-or-later with a commercial option. The packages are importable if you want them: github.com/motherlodelab/magpie/scrape exports Run, which both the CLI and its 13-tool MCP server call. But the entry point takes a Deps struct that expects you to bring a SQLite store and fetcher constructors. It’s product wiring, not a framework. For most jobs the CLI is the better interface.

Same sandbox, one command:

magpie scrape https://quotes.toscrape.com/ --page-format markdown

That returned in 2.06s today: a JSON envelope with the final URL, the title, and a markdown field holding all ten quotes as clean Markdown with author links. My forty lines of Go counted div.quote elements. That command produced the whole page, boilerplate removed, ready for an LLM pipeline.

Now the honest limit, demonstrated on the /js/ page my rod script just scraped:

magpie scrape https://quotes.toscrape.com/js/ --render browser --page-format markdown
# Error: quality blocked (empty) for https://quotes.toscrape.com/js/  (exit 8)

Magpie’s thin-page gate refuses any page whose cleaned content is under 200 words (ThinPageWords = 200 in clean/quality.go). Ten short quotes score below the line, so the tool returns nothing where my rod script returned ten quotes. A comment in Magpie’s own source admits the “empty” wording misleads. That’s the trade in one sentence: quality gates protect your pipeline from junk pages, and hand-rolled scripts have no gates. Neither behavior is wrong; pick per job.

What you get free in the CLI beyond this: TLS-impersonating browser fingerprints (--browser chrome), robots.txt and auto-throttle by default, watch mode with webhooks, saved login sessions, and 27 zero-LLM vertical extractors for things like JSON-LD, job postings, and events.

My rule after a year of writing scrapers in Go: write Go when the scraping logic is the product. Run Magpie when the data is the product.

FAQ

Is Go good for web scraping? For static pages, yes: a fast HTTP client, goroutines for concurrency, and a single-binary deploy. JavaScript-heavy sites need a CDP driver such as rod, which costs seconds per page instead of milliseconds.

Colly or rod? Static crawl, use Colly. Client-rendered page, use rod. They compose: Colly discovers URLs, rod renders only the ones that need a browser.

How do I scrape a JavaScript site in Go? Check view-source and the Network tab first. Embedded JSON or a callable XHR beats a browser render on speed every time. Only open rod when the content truly requires it.

Does Colly respect robots.txt? Yes, by default, through the temoto/robotstxt package. IgnoreRobotsTxt: true opts out. On sites you don’t operate, don’t.

Why did my Colly rate limit do nothing? The DomainGlob in a LimitRule matches against the host, so a pattern with a path suffix like example.com/* never matches. Use example.com or *.example.com, then verify the delay with hop timings.

Start with the forty lines above. The day you catch yourself writing a third cookie jar, retry loop, and pagination queue, run the same URL through magpie scrape --page-format markdown and compare the output to yours. It refused the JavaScript page I gave it today, and at least it told me why.