Web Scraping Statistics (2026): Every Number Sourced

Bots served 53% of all web traffic in 2025, the second straight year automated requests outnumbered people. That is the number that puts web scraping statistics in every budget meeting this year: more than half of what any website ships goes to software, not to readers. Here are the load-bearing numbers for 2026, each one dated and linked to where I found it.

How I collected these web scraping statistics

I don’t aggregate other statistics roundups; those are where numbers go to lose their citations. Every figure below comes from a named primary source: the Thales 2026 Bad Bot Report, Cloudflare’s crawler telemetry, a market model by Mordor Intelligence, court records, and surveys I read in full. Where a number is a model output rather than a measurement, the heading says so. GitHub star counts I pulled live on October 16, 2026, and dated inline, because stars drift.

One habit worth stealing: while auditing links for this page I found several 2025-vintage roundups still passing off the 2024 report as current. If you cite a stat in a deck or a pitch, check the source’s data period, not the headline. Reports lag their titles by a year in this space.

Bots are most of the web, again

The 2026 Bad Bot Report covers full-year 2025 traffic across Imperva’s network (Imperva is owned by Thales, hence the branding). The topline:

• Automated traffic: 53% of all web traffic, up from 51% in 2024. Humans fell to 47%.
• Inside automation: bad bots 40%, benign crawlers and uptime monitors (“good bots”) 13%.
• AI-driven bot traffic grew 12.5× year over year; daily blocked AI-bot requests went from 2 million to 25 million.
• 27% of bot attacks hit API endpoints.
• Financial services absorbed 24% of attacks and 46% of account takeovers.
• The US drew 59% of attacks. Chrome was the most-impersonated browser, with Android Browser second.

Treat these as one telescope, not the sky. Imperva only counts what crosses its network, and its customers skew toward enterprises that bother buying bot protection. But it is the same methodology every year, so the trend line is real even if the absolute share is debatable.

The 40% bad-bot number is the practical one for scrapers. Your targets assume they are under attack, because they are. When a site adds fingerprinting challenges or tightens rate limiting, it is reacting to that 40%, and your polite script wears the same uniform. Anything you ship in 2026 needs to look like a real browser, because the defenses were built for worse than you.

AI crawlers: the fastest-moving numbers on the page

Cloudflare published its crawler telemetry comparing May 2024 with May 2025 across its network (“From Googlebot to GPTBot”). Shares below are within Cloudflare’s cohort of tracked crawlers, not of all traffic.

• Total crawler traffic grew 18% year over year on a fixed set of customers, 48% including new ones. April 2025 peaked at +32%.
• Googlebot: 30% → 50% of tracked crawl requests, with raw requests up 96%.
• GPTBot (OpenAI): 2.2% → 7.7% of crawl requests, a 305% jump in raw volume, from rank 9 to rank 3. Within the AI-crawler-only view it went from 5% to 30%.
• Bytespider (ByteDance): 22.8% → 2.9%, raw requests down 85%.
• ClaudeBot (Anthropic): 11.7% → 5.4%.
• ChatGPT-User requests grew 2,825%; PerplexityBot grew 157,490%.

I kept the Perplexity number because it is the honest kind of ridiculous: real growth, off a base so small the percentage is close to meaningless. That is exactly the sort of stat most roundups strip of its base and let run wild. Percentage growth without a base is a headline, not a measurement.

The shape of the data matters more than any single row. Search crawling still dwarfs AI crawling, but AI fetches are the growth curve, and they behave differently: they fetch on demand, answer-shaped, and do not send click-through back. Googlebot sends traffic. GPTBot takes text.

Publishers block the crawlers. The crawlers come anyway.

The Tow Center at Columbia tracked robots.txt blocks across the top 50 US news websites as of May 2025 (“Journalism Zero”):

• ~60% blocked Anthropic’s crawlers.
• 58% blocked Google-Extended (Gemini’s training feed).
• 56% blocked Perplexity’s crawler.
• 50% blocked GPTBot.
• 40% blocked ChatGPT-User and 32% blocked OAI-SearchBot.

Note the pattern: publishers block training crawlers hardest and search/search-crawlers least. They want to be cited, they just don’t want to be absorbed.

And the enforcement reality, from Tow Center research published in March 2025: several AI companies’ crawlers were observed accessing content from publishers who had blocked them. A robots.txt line is a request, honored by companies that care about their crawl reputation and ignored by those that don’t. If your own data matters, treat robots.txt as a signal, not a wall, in both directions.

Web scraping market size, honestly labeled

Market-size figures are model outputs, not measurements, so here is the model stated plainly. Mordor Intelligence sizes the web scraping market at $1.34 billion in 2025, $1.56 billion in 2026, growing to $3.49 billion by 2031 at a 17.39% CAGR (source). Other research shops put the absolute number elsewhere; the shape (a market of roughly $1.5B compounding around 17%) is the part I’d trust.

Mordor’s internal splits are more interesting than the total:

• Cloud-delivered scraping is 67.45% of 2025 revenue and growing fastest (17.80% CAGR).
• Software is 58.35% of solution revenue; services grow at 18.62%.
• Banking, financial services, and insurance are the biggest buyer at 29.40% of end-user revenue.
• Data scraping and ETL pipelines are 36.20% of use-case revenue.
• Price and competitive monitoring is the fastest-growing use case, at an 18.34% CAGR through 2031.
• North America holds 34.08% of 2025 share; Asia-Pacific grows fastest (18.66% CAGR).

Read the buyer split with the bot data above and the market makes sense: the industries getting hit hardest by bots are the ones paying most for scraping infrastructure, on both sides of the arms race. If you’re buying rather than building, the vendor landscape and its pricing traps are covered in best web scraping APIs, and the self-hosted end of the market in best web scraping tools.

Who scrapes: a survey and some star counts

Censuswide polled 506 US and UK technical professionals whose jobs include web scraping, for Oxylabs (method and results). Respondents skewed working practitioners: 41% software developers, 22% full-stack developers, 20% data engineers. Almost three-quarters reported rising demand for public web data in their businesses. That tracks with everything else on this page: the demand is not a trend, it’s a baseline.

For adoption of the open-source core, I pulled live counts from GitHub on 2026-10-16:

• scrapy/scrapy: 64,850 stars, commits pushed the day I checked.
• gocolly/colly: 25,552 stars.

Stars measure mindshare, not deployments. But 64,850 stars plus commits on the day I checked is not a museum either. The library layer of this market is old, stable, and still growing.

Common Crawl: the public baseline

The September 2025 Common Crawl release contains 2.39 billion pages, 421 TiB of uncompressed content, one monthly crawl of an archive collected regularly since 2008 and now measured in petabytes. It is the substrate most famous LLM pretraining datasets were built from: C4, the Pile, and CCNet all start with Common Crawl.

Common Crawl also explains the crawler numbers above. A free, stale corpus got the models to where they are; the next quality jump wants fresher data, so the labs crawl directly. That is the demand curve behind GPTBot’s +305%, and behind every robots.txt standoff in the news.

The courts, in five numbers

The legal story of 2024–2026, reduced to the outcomes:

• hiQ v. LinkedIn: the Ninth Circuit ruled in 2019 (938 F.3d 985) that scraping public LinkedIn profiles likely does not violate the CFAA. The Supreme Court vacated it in June 2021, the Ninth Circuit reaffirmed in April 2022 (31 F.4th 1180), and the parties settled in November 2022. Public data stayed defensible.
• Meta v. Bright Data: in January 2024 a California federal judge refused to block Bright Data’s scraping of public Facebook profiles, reading Meta’s terms as governing “use” of the site, which collection of public data is not.
• X Corp v. Bright Data: in May 2024 a judge dismissed X’s contract claims over scraping of public posts, and X dropped the remainder within days.
• Ryanair v. Booking.com: a Delaware jury found on July 18, 2024 that Booking violated the CFAA “with intent to defraud” by scraping Ryanair fares. On January 23, 2025 the judge erased the verdict because Ryanair couldn’t show Booking profited enough from it.
• Clearview AI: the Dutch DPA fined it €30.5 million in May 2024, bringing combined EU fines to more than €90 million across four regulators. The UK’s £7.5 million fine was overturned on jurisdictional grounds in October 2023.

Public data, scraped without login, kept winning in court. Logins, paywalls, and facial recognition kept losing. And even the winners lose: Ryanair got a jury verdict and watched it vanish in six months. The full plain-English tour, including the EU contract rule from Ryanair v PR Aviation (CJEU C-30/14: an unprotected database can still be contract-locked against scrapers), is in is web scraping legal.

The data wall behind all of it

Epoch AI estimates the effective stock of public human text at roughly 300 trillion tokens, and projects that at current training trends, models will fully consume it between 2026 and 2032 (publication). That projection is the demand function for everything in this article. It is why AI bot traffic grew 12.5×, why GPTBot more than tripled its share, and why blocking keeps failing: the buyers of this data have budget curves, not preferences.

Watch that number more than the market-size one. A 17% tools market is steady business. A data wall inside most readers’ planning horizons is why agentic crawlers (fetchers that act on goals, not just sitemaps) showed up in the 2026 report as a category at all.

What I’d do with these numbers

Three moves, in order of payoff:

1. Assume your targets are defended. With 40% of traffic being bad bots, every site worth scraping ships rate limits, fingerprinting, or both. Budget for browser-quality requests from the first command.
2. Check robots.txt, then check reality. Half the top news sites block GPTBot and it doesn’t fully work for them. Before you build on politeness, verify what your target actually enforces.
3. Benchmark from your own connection. Network reports tell you what Imperva sees; they don’t tell you what a given site serves your IP. For that I run Magpie, a local open-source CLI that scrapes to markdown or JSON from my own machine. One command answers “does this site even serve me a real page,” which is ground truth no report can give you.

Then re-audit yearly. Every number on this page has a data period, and in this space the shelf life is about twelve months.

FAQ

What percentage of web traffic is bots? 53% in 2025, up from 51% in 2024, per the Thales 2026 Bad Bot Report (full-year 2025 data from Imperva’s network). Bad bots were 40% of all traffic, benign crawlers 13%, and humans 47%.

How big is the web scraping market? Mordor Intelligence estimates $1.34 billion for 2025, growing to $3.49 billion by 2031 at a 17.39% CAGR. Other analysts put the absolute figure elsewhere. Treat the shape, not the decimals, as the finding.

Is web scraping legal? US courts have repeatedly held that scraping publicly available data likely does not violate the CFAA (hiQ v. LinkedIn). Logins, paywalls, personal data, and contract prohibitions change the analysis. Start with the plain-English legal guide, then confirm with a lawyer for your case.

What is the biggest web crawler? By tracked crawl activity, Googlebot: 50% of crawl requests on Cloudflare’s network as of May 2025. If you mean the largest public crawl dataset, that’s Common Crawl, whose September 2025 monthly crawl alone covered 2.39 billion pages.