AI Bots Are 34% of All Bot Traffic Now: Bigger Than Search Engines

By Muninder Adavelli. Scraping, SEO, and proxies for over a decade.

1. AI-related bots are 34.30% of all bot traffic: now bigger than search crawlers

Cloudflare Radar, which measures verified and likely-automated requests across a global CDN with no bot-management product priced against that specific reading, puts AI-related bots at 34.30% of all bot traffic on the 28-day window ending August 1, 2026, up from 33.8% in June. The decomposition is clean:

That total is bigger than traditional search-engine crawlers at 26.13% for the first time in Cloudflare's reporting: a category flip that just happened. Absolute counts point the same direction. DataDome, whose numbers come from 400+ enterprise deployments and 5 trillion signals analysed daily, processed 17.7 billion AI agent requests in Q2 2026 alone and has counted over 30 billion year-to-date. The two networks disagree on the operator mix (DataDome's fraud-defence-weighted footprint sees more Meta than Cloudflare's general CDN does), but they agree on the direction. The concession: 34.30% is AI-related bots as a share of bot traffic, not of all web traffic. All-web bot share is a different number (Cloudflare's own broader reads run near 34-35% of total requests), and mixing the two is how "AI is X% of the web" claims spiral out of shape.

Sources: Cloudflare Radar; DataDome, AI Traffic Report Q2 2026

2. AI agent traffic grew 45% quarter over quarter, and 300% year over year

DataDome's Q2 read is a 45% jump on Q1's 12.2 billion. The monthly ramp inside Q2 tracks: April 4.77 billion, May 6.29 billion (+32% in a single month), June 6.60 billion. On a longer horizon, Akamai's 2026 State of the Internet Security report, measured across the ecommerce and application-layer surfaces Akamai defends, records AI bot activity up 300% in 2025. The two aren't in tension. A sustained 45% QoQ pace compounds to (1.45)^4 ≈ 4.4× a year: the same order of magnitude as Akamai's 300%, with room for the current quarter to run hotter than the trailing annual average. The concession: DataDome's Q3 hasn't landed, and single-quarter surges have a habit of flattening the moment the operator behind them ships an efficiency win. Meta-ExternalAgent grew +74% QoQ and Meta-WebIndexer grew +163%: most of the QoQ line is one operator, so the sustainability question depends heavily on Meta's roadmap, not on a diffuse many-vendor trend.

Sources: DataDome, AI Traffic Report Q2 2026; Akamai, State of the Internet Security 2026

3. Google runs 27% of verified bot traffic. Meta 15%. Anthropic 10.5%. OpenAI 6.5%.

Cloudflare Radar's 28-day window to August 1, 2026 puts Google at 27.11% of verified-bot requests (still the largest operator by a wide margin), with Meta at 15.01%, Anthropic at 10.53%, and OpenAI at 6.50%. The top four combined: 27.11 + 15.01 + 10.53 + 6.50 = 59.15%. Six in every ten verified bots on Cloudflare's network come from just four companies, and two of those four are AI labs rather than search engines. Anthropic held the #2 slot through June, then slipped to #3 across July as Meta's share climbed. At the individual-bot level: GoogleBot 12.87%, Claude-User 7.57% (the second-busiest single bot on the web), Meta-ExternalAgent 6.88%. On DataDome's network specifically, the mix reads differently: Meta generates the majority of AI agent traffic, with Meta-ExternalAgent and Meta-WebIndexer together making 9.1 billion requests in Q2 alone. The concession: ranks flip on 7-day windows. Cloudflare's most recent 7-day cut has Anthropic bouncing to #2 at 13.5% before the slot returns to Meta. Ranking is a snapshot. Category share is the trend.

Sources: Cloudflare Radar; DataDome, AI Traffic Report Q2 2026

4. 44% of AI crawling is for model training, not live-search indexing

Cloudflare Radar's purpose split for the same 28-day window: 44.21% of AI crawling collects data for model training, 11.66% for live-search indexing, 40.18% mixed-purpose. Two months earlier the split was 52.3% training vs 10.1% live search: the training-to-live-search ratio dropped from about 5:1 to roughly 4:1 in June-July. On the destination side, Akamai reports that more than 45% of AI bot traffic on its networks targets commerce customers, drawn to the structured product, price, and availability data ecommerce pages publish by necessity. Akamai counted over 25 billion AI bot requests hitting ecommerce sites in one two-month span in 2026. Crawl volume and downstream value have decoupled: ChatGPT-User's absolute crawl volume actually declined 6% QoQ on DataDome's network in Q2, but ChatGPT still commands 80-88% of AI referral traffic and grew +17% QoQ on that axis. Three concessions apply, one per lever: the training-to-live-search ratio moves noticeably on short windows and it will keep moving; the Akamai commerce figure is specific to Akamai-defended surfaces, not the general web; the referral-vs-crawl decoupling is a DataDome-only read, one network's signal rather than a settled cross-industry pattern.

Sources: Cloudflare Radar; Akamai, What to Do When Competitors Are Scraping Your Prices; DataDome, AI Traffic Report Q2 2026

5. What to actually track if you're sizing AI bot traffic

I run an AI visibility tool that scrapes LLM interfaces at 2-5M requests a month. Same slope at a much smaller scale: quarterly volume climbs at roughly the pace DataDome reports, and the operator mix on my own logs skews toward Anthropic and OpenAI where DataDome's skews Meta, because I query models rather than sit behind fraud defence. The four numbers worth watching from Q3 onwards:

1. The training-vs-live-search ratio on Cloudflare Radar. It dropped from 5:1 to 4:1 in two months. When live search overtakes training, the operator incentive to respect robots.txt changes: training is opportunistic, live search is customer-facing.
2. Meta's operator share vs Anthropic's. They're doing different things. Meta indexes for its own models; Anthropic's biggest volume driver (Claude-User) is user-triggered fetches. Different bots, different mitigation shapes.
3. Absolute request counts, not just share percentages. Share can rise while your absolute pain falls (better mitigations landed), or vice versa. The 17.7B to whatever-Q3-lands number is the one that decides bandwidth.
4. Referral traffic per crawl request. ChatGPT crawls less and refers more; the question that matters for a publisher is whether a given bot is paying rent.

If you're writing a robots.txt or sizing bot management in Q4 2026, the ranking is unstable but the volume trend is not. Track counts.

For the full State of Web Scraping 2026 picture this article sits inside (market size, scraper-vs-bot-vs-crawler methodology, and the broader operator ranking history), see the State of Web Scraping 2026 report.