AI Bot Traffic on Landing Pages: Agents, Crawlers and Ads (2026)

ClickTrust Research · · 11 min read · AI bot traffic · AI agents · crawlers · landing pages · invalid traffic

TL;DR Bots passed 53% of all web traffic in 2025 according to Imperva, and Cloudflare saw daily requests from AI agents grow by more than 1,700% in a year. That automation clicks ads, fills retargeting pools and fires conversion pixels, so paid landing pages need session-level scoring rather than a robots.txt file.

AI Bot Traffic on Landing Pages: Agents, Crawlers and Ads (2026)

Key takeaways

  1. Bots accounted for more than 53% of all web traffic in 2025, with human activity down to 47% and 40% of bot traffic classified as malicious, according to Imperva's Bad Bot Report 2026.
  2. Daily requests from AI agents on Cloudflare's network grew by more than 1,700% in a year, and Cloudflare now sees over 500 billion verified bot requests per week, per its September 2026 post The Internet has a second audience.
  3. The share of crawler requests made for AI training rose from 22% in spring 2025 to 52% by June 2026, according to Cloudflare, and in heavily crawled categories human traffic fell as much as 40% in under a year.
  4. In July 2025 Anthropic crawled about 38,000 pages for every visit it referred, OpenAI 1,091, Perplexity 194.8 and Google 5.4, per Cloudflare's crawl-to-click gap analysis.
  5. AI crawlers made up almost 80% of AI bot traffic on Fastly's network in mid-2025, with Meta at 52% of crawler traffic and OpenAI's ChatGPT bots at 98% of fetcher requests.
  6. Only 2.8% of about 17,000 popular domains were fully protected against test bots in 2025 and 61.2% let every test bot through, according to DataDome's Global Bot Security Report.

AI bot traffic is now a routine share of what loads your paid landing pages, and almost none of it becomes a customer. On sites behind Cloudflare, humans generated only 47% of HTML requests by early December 2025, with non-AI bots at 44% and AI bots averaging 4.2% across the year, according to Cloudflare's 2025 Radar Year in Review (link). This post covers who sends it, why it lands inside paid campaigns, how to tell an agent from a buyer, and what to block.

AI bot traffic is automated web activity generated by AI companies' crawlers, which collect pages for model training and search indexes; AI fetchers, which retrieve a page on a user's behalf to answer a question; and AI agents or agentic browsers, which browse, click and fill forms to complete a task. The Media Rating Council classifies ordinary crawlers as General Invalid Traffic, "traffic identified through routine and list-based means of filtration, such as bots, spiders, other crawlers" (MRC, Invalid Traffic Detection and Filtration Guidelines Addendum release, October 2015).

How much web traffic is AI bot traffic?

Bots are now the majority of web traffic on most measurements, and the AI share is the fastest-growing part. Imperva's Bad Bot Report 2026 (link) reports that bots accounted for more than 53% of all web traffic in 2025, with human activity down to 47% and 40% of bot traffic classified as malicious.

Different vantage points give different numbers. Cloudflare's From Googlebot to GPTBot: who's crawling your site in 2025 (link) puts bots at around 30% of global web traffic. Fastly measured automated bot traffic at 37% of observed activity on its network between April and July 2025 (Fastly, Threat Research press release, August 2025) and at 29% in Q3 2025 (Fastly, Threat Insights Report, December 2025). Vercel put bots at more than 20% of traffic across its deployments in an August 2025 week, about a quarter of it AI crawlers (Vercel, The three types of AI bot traffic).

Bot share of web traffic by who is measuring Figure 1. Bot share of web traffic as measured by Kinsta, Imperva, Fastly, Cloudflare and Vercel on their own networks in 2025 and 2026; the spread reflects what each fronts and how it counts.

Part of the gap is how each company counts, and part is which sites it fronts. Kinsta's analysis of more than 5,000 WordPress sites (link) shows the dispersion: a typical day split about 54% automated to 46% human, but the median site saw only 1.57% of bandwidth from AI bots while the 99th-percentile site saw 90.3%. Your landing page is somewhere on that curve, and only a measurement on the page itself tells you where.

The growth rates agree even where the levels differ. Cloudflare's The Internet has a second audience (link, September 2026) reports that daily requests from AI agents on its network grew by more than 1,700% in a year, and that Cloudflare now sees more than 500 billion verified bot requests per week. Its 2026 Annual Founders' Letter (link) says the forecast date for automated traffic passing human traffic moved from the second half of 2027 to May 2026. Akamai reported AI bot activity on its platform rose 300% in 2025 (Akamai, April 2026).

Who the big senders are

A handful of companies send most AI bot traffic, and they send it for different reasons. Fastly's mid-2025 research found AI crawlers made up almost 80% of AI bot traffic, with Meta at 52% of crawler traffic, Google at 23% and OpenAI at 20%; on the fetcher side, OpenAI's ChatGPT bots accounted for 98% of requests (Fastly, August 2025). Cloudflare's July 2025 crawler analysis found GPTBot's share of AI crawling jumped from 5% to 30% in a year, and GPTBot was the most-blocked bot in robots.txt.

Not all of it is benign. Imperva's 2025 Bad Bot Report (link) found ByteSpider, operated by ByteDance, alone drove 54% of AI-enabled attacks, with AppleBot at 26%, ClaudeBot at 13% and ChatGPT-User at 6%. On Akamai's network, commerce absorbed 47.9% of all AI bot traffic between July and December 2025, and commerce organizations put over 90% of AI bot activity in monitor mode rather than blocking it (Akamai, July 2026).

Crawl-to-refer ratios: what you get back

For every visitor an AI platform sends you, it fetches hundreds or thousands of pages. Cloudflare's crawl-to-click gap analysis (link, August 2025) found 80% of AI crawling over the previous 12 months was for model training, 18% for search and 2% for user actions. In July 2025, Anthropic crawled about 38,000 pages for every referred visit (down 87% from 286,930), OpenAI 1,091, Perplexity 194.8 and Google 5.4.

Pages crawled per referred visit, July 2025 Figure 2. Pages crawled for every visit referred back to the site in July 2025, by AI platform, from Cloudflare's crawl-to-click gap analysis.

By December, Cloudflare's 2025 Radar Year in Review reported Anthropic's crawl-to-refer ratio reached as high as 500,000:1 during 2025 and OpenAI's up to 3,700:1. "User action" crawling, where an AI fetches a page because a person asked, grew over 21x from January to early December.

The training share keeps rising. Cloudflare's Content Independence Day, one year on (link, July 2026) found the share of crawler requests made for AI training rose from 22% in spring 2025 to 52% by June 2026, and that in heavily crawled categories human traffic fell as much as 40% in under a year. Akamai's publishing research puts the referral side bluntly: AI chatbots drove about 96% less referral traffic than Google search in Q4 2024 (Akamai, April 2026).

Share of crawler requests made for AI training Figure 3. Share of crawler requests on Cloudflare's network made for AI model training, spring 2025 against June 2026, from Cloudflare's Content Independence Day, one year on.

For an advertiser the ratio is a measurement problem, because every one of those fetches can fire your tags.

Measure Figure Publisher and report
Bots as share of all web traffic, 2025 more than 53% Imperva, Bad Bot Report 2026
HTML requests on Cloudflare, early Dec 2025 47% human, 44% non-AI bots Cloudflare, 2025 Radar Year in Review
Daily AI-agent requests, year to Sept 2026 more than +1,700% Cloudflare, The Internet has a second audience
Purpose of AI crawling, year to Aug 2025 80% training, 18% search, 2% user actions Cloudflare, crawl-to-click gap
Anthropic crawl-to-refer ratio, 2025 peak up to 500,000:1 Cloudflare, 2025 Radar Year in Review

Why AI bot traffic lands inside paid campaigns

Agents do not buy, but they click, load pages and fire pixels, and each of those has a cost.

Clicks. Any automated agent that follows a sponsored result or a social ad link registers a click and a charge unless the platform filters it. Google Ads Help names "automated traffic" from bots or deceptive software as one of its three types of invalid click and says you are not charged for clicks it identifies as invalid (Google, Invalid clicks: Definition, Google Ads Help). The clicks it does not identify are billed, and most agents arrive undeclared: Arkose Labs found only 2.3% of legitimate AI-agent sessions between July and September 2026 carried a verified Web Bot Auth signature (Arkose Labs, State of Agentic AI Trust Report Q3 2026, as reported by PPC Land).

Retargeting pools. A crawler that loads your pricing page with cookies and JavaScript enabled joins your remarketing audiences, and you then pay to show ads to a data center. Cloudflare's 2025 Radar Year in Review found AWS and Google Cloud together originated about a quarter of global bot traffic; a retargeting pool built without an origin check fills up with exactly those ranges.

Conversion pixels and analytics. Agentic browsers complete forms. If your pixel fires on a thank-you page, the agent's visit becomes a conversion, and Smart Bidding in Google Ads or Advantage+ in Meta learns from it. F5 Labs found more than one in five add-to-cart transactions across 200 billion web and API transactions were automated (F5 Labs, 2025 Advanced Persistent Bots Report). Even non-converting crawler sessions skew bounce rate, session duration and landing-page experience, which feed Quality Score and placement decisions. The GIVT versus SIVT explainer covers how these sessions are classified.

How to tell agents from customers

There is no single signal, and most sites check none of them: DataDome's Global Bot Security Report 2025 (link) tested about 17,000 popular domains and found only 2.8% fully protected against its test bots, 36% partially and 61.2% letting every test bot through. The practical approach combines several checks on every session.

  • Declared identity. Well-behaved crawlers announce themselves (GPTBot, ClaudeBot, PerplexityBot, Googlebot) and can be verified by reverse DNS or published IP ranges. Cloudflare reports that OpenAI, Google and AWS now cryptographically sign their agents' requests. An undeclared session from the same ranges is the one to distrust.
  • Network origin. Data-center and cloud ranges rarely host real customers. Compare the origin against where your campaign is targeted.
  • Rendering behavior. Agents often skip fonts, images or third-party scripts, or load them in an unusual order. Measure what the page actually executed, not just that it was requested.
  • Interaction. Pointer movement, scroll depth, dwell time and keystroke cadence separate a person from a script that fills a form in one second.
  • Robots.txt compliance. A client that reads disallowed paths is telling you what it is. TollBit's State of the Bots for H1 2026, as covered by PPC Land (link), found about 15% of identified AI page-fetchers in Europe reached URLs that sites had disallowed.
  • Downstream outcomes. A source whose conversions never become sales is invalid whatever the session looked like.

What to block, what to allow, and why robots.txt is not enough

Robots.txt is a request, not a wall, and it does not apply to paid traffic at all. Nothing reads robots.txt before an ad click sends an agent to a URL with a gclid attached.

Cloudflare's July 2025 study found only about 14% of the top 10,000 domains with a robots.txt had any directive addressing AI bots; by September 2026 Cloudflare reported 17% of sites block training crawlers and fewer than 1% block search crawlers. TollBit's Q1 2025 State of the Bots, as presented with FIPP (link), found the share of AI-bot scrapes bypassing robots.txt rose from 3.3% in Q4 2024 to about 13% in Q1 2025. Blocking has a cost as well: TollBit's H1 2026 report, via PPC Land, cites a Rutgers/Wharton study in which publishers that blocked AI crawlers lost about 7% of weekly traffic within six weeks.

A workable policy for an advertiser:

  1. Allow and verify search crawlers such as Googlebot and Bingbot on organic pages; they feed the indexes you want.
  2. Allow but exclude from measurement declared AI fetchers that act for a user, such as ChatGPT-User, so they can read your page but never enter retargeting pools or count as conversions.
  3. Rate-limit or challenge training crawlers on content pages according to your content strategy; this is a business decision, not a fraud decision.
  4. Block from paid landing pages undeclared automation, data-center origins and sessions with no interaction, and feed those signals back as IP and audience exclusions in Google Ads, Meta and Bing.
  5. Gate conversion events behind a human-likelihood check so pixels fire on people, not agents.

Use the free tools to see what is reaching a page today, and read the full Agentic Traffic Report 2026 for the longer analysis.

How ClickTrust helps

Web Protection is a JavaScript tag that scores every session in under 50 milliseconds against 320 detection traps, each labelled GIVT or SIVT in line with MRC invalid-traffic guidance, and returns a single Traffic Quality Index (TQI) per event. It pushes IP and audience exclusions to Google Ads, Meta and Bing automatically, so agents and crawlers stop entering your retargeting pools. ClickTrust also exposes its capabilities to AI agents through an MCP server. ClickTrust audits find 14 to 42 percent of paid traffic is invalid depending on channel. It is free to start, and Traffic Rx gives a free landing-page traffic diagnosis.

Sources

Frequently asked questions

What is AI bot traffic?

AI bot traffic is automated web activity from AI companies and their users. It has three parts: crawlers that collect pages for model training and search indexes, fetchers that retrieve a page on a user's behalf to answer a question, and agents or agentic browsers that browse, click and fill forms to complete a task. The MRC treats ordinary crawlers as General Invalid Traffic.

How much web traffic is bots in 2026?

It depends on who is measuring. Imperva's Bad Bot Report 2026 puts bots at more than 53% of all web traffic in 2025. Fastly measured automated traffic at 37% of its network in mid-2025 and 29% in Q3 2025. Cloudflare's July 2025 crawler analysis puts bots at around 30% of global web traffic, and its 2025 Radar review found humans generated 47% of HTML requests on its network, with non-AI bots at 44% and AI bots at 4.2%.

Do AI agents click on ads?

They can. An agent following a sponsored link registers a click and a charge unless the platform filters it, and Google Ads Help lists automated traffic from bots as one of its three types of invalid click. Arkose Labs found that only 2.3% of legitimate AI-agent sessions in Q3 2026 carried a verified Web Bot Auth signature, so most agents arrive undeclared. Agents that load a page with JavaScript also enter retargeting audiences and can trigger conversion pixels.

Does robots.txt stop AI bots?

No. Robots.txt is a request that compliant crawlers honor, and it never applies to paid traffic because an ad click sends the visitor straight to the landing page. TollBit's data, presented with FIPP, showed the share of AI-bot scrapes bypassing robots.txt rose from 3.3% in Q4 2024 to about 13% in Q1 2025. Paid pages need session-level detection instead.

How can I tell an AI agent from a real customer on my landing page?

Combine signals. Check the declared user agent against verified IP ranges, look at the network origin for data-center and cloud ranges, measure what the page actually rendered, record pointer movement, scroll and keystroke timing, and check whether the client reads paths your robots.txt disallows. Then confirm with outcomes: a source whose conversions never become sales is invalid whatever the session looked like.

Should I block all AI crawlers?

Not necessarily. Verified search crawlers feed the indexes you want, and declared AI fetchers acting for a user can bring prospects. Blocking has a cost too: TollBit's H1 2026 report cites a Rutgers/Wharton study where publishers that blocked AI crawlers lost about 7% of weekly traffic within six weeks. Block undeclared automation and data-center origins from paid pages, and keep every bot out of your conversion events.

Start ClickTrust free · More articles