On October 9, Zen Media published a release built on one number. In Q3 2026, 13.47% of AI bot requests to sites on Cloudflare's network came back denied. A year earlier it was 5.98%. The data is Cloudflare Radar, analyzed by TechnologyChecker.io, and both parties call the Q3 figure preliminary.
The obvious read: the web is fighting back. Publishers got tired of being scraped and flipped the switch. But the question that matters for your business is narrower. Is my website blocking AI crawlers, and did anyone actually choose that?
Much of this blocking was never a decision. And the crawlers most often blocked are not the ones that decide whether you show up in an AI answer.
TL;DR
- AI bot requests denied on Cloudflare's network went from 5.98% in Q3 2025 to 13.47% in Q3 2026, per preliminary Cloudflare Radar data analyzed by TechnologyChecker.io.
- A lot of that blocking is a vendor default or a leftover rule, not a policy anyone chose.
- Robots.txt and firewalls disagree constantly. HasData found 234 of 592 sites that banned GPTBot served it a page anyway.
- Training crawlers barely matter to you. Answering crawlers decide whether you appear in the answer, and they are a different user agent.
- One test is reliable: request your own pages as the bot and read the status code.
The Mid-September Change Was a Vendor Shipping, Not Owners Deciding
Cloudflare's managed robots.txt used to prepend a block disallowing eight AI crawlers, GPTBot and ClaudeBot and Google-Extended among them. In August it announced a replacement, Bot Preference Sync, which writes the file from three dashboard policies: Search, Agent, and Training.
Around September 15 and 16, the old block vanished from files across the network. Two independent datasets caught it. Analyst Reese Calder, working from Common Crawl archives, saw its share of robots.txt files fall from roughly 2% to 0.03% overnight. HasData found the block in archived copies of 118 Cloudflare sites' files, and on none of those sites on September 16.
Sites across the network changed their stated AI policy on a Tuesday, and the change was shipped by a vendor, not decided site by site. These numbers move when a vendor ships.
Your Robots.txt and Your Server Give Different Answers
Robots.txt states a preference. Your firewall, CDN, and bot protection decide what actually gets served. Those layers diverge constantly, in both directions.
HasData tested 10,894 domains in July and re-ran the test on September 16. In its July baseline, of 592 sites disallowing GPTBot, 234 served it a live page anyway. Between July and September, the count of sites blocking GPTBot with no robots.txt declaration at all rose from 115 to 178.
Calder found the mismatch from the other side. Among sites that lost the Cloudflare block and still served a normal Chrome request, most returned 403 to GPTBot and ClaudeBot. After controlling for sites that refuse any unverified agent, 56 were still specifically refusing AI crawlers. He ran it from one residential IP with no pre-September baseline, and says so.
Your file says one thing. Your server says another. Nobody reconciled them because nobody owns both.
Training Crawlers Are Not the Ones That Matter to You
A training crawler collects text for a future model: GPTBot, ClaudeBot, CCBot. Google handles the same choice with Google-Extended, a robots.txt token rather than a separate crawler. If you run a roofing company in Fort Lauderdale, whether OpenAI trains on your service pages barely touches your revenue.
An answering crawler fetches your page for search or right now, because someone asked a question and the model needs a live source: OAI-SearchBot, ChatGPT-User, Perplexity-User. OpenAI's own documentation says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Those crawlers put your phone number in the answer.
The market has quietly sorted this out. In a September 28 snapshot of 4,242 robots.txt files on Cloudflare Radar, GPTBot carried 1.60 disallow rules per allow. OAI-SearchBot carried 0.91, allowed more often than blocked. Googlebot sat at 1.11.
A firewall rule matching "bot" in the user agent string does not know that difference. Neither does a hosting default.
Is My Website Blocking AI Crawlers? Only a Live Request Answers That
You cannot answer this from robots.txt, and you cannot answer it from your Cloudflare dashboard, because that shows the policy you set and not how it collides with your firewall rules, your rate limiting, your security plugin, and whatever a previous developer added in 2024.
You answer it by requesting your own page as the bot and reading the status code. A 200 means reachable. A 403 means blocked. Treat a challenge page as blocked too, because a crawler has no browser to pass it.
Status codes are the only evidence here. Everything else is a stated intention.
Your AI Visibility Audit May Be Measuring Nothing
If you have paid for an AI visibility report or an answer engine optimization audit in the last six months, ask one question before acting on it. Did they confirm the crawlers could reach the site?
If your edge returns 403 to OAI-SearchBot, you are invisible in every one of those reports, and the recommendation comes back as "improve your content and add schema." You spend a quarter on that. The real problem was a firewall rule, and it takes a few minutes to find. It is the same pattern as the gap between buying AI and deploying it: the spend happens, the plumbing never gets checked.
The content was never the problem.
What To Do This Week
Request your own pages as the bots and log the status codes. Homepage plus your top five landing pages, once per agent: GPTBot, OAI-SearchBot, PerplexityBot, ChatGPT-User, and a normal Chrome string as your control.
curl -I -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot)" https://yoursite.com/your-pagedoes it. Any row where Chrome gets 200 and a bot gets 403 is your answer. One caveat: some bot protection verifies a crawler by its network, so a spoofed request can be refused even where the real crawler is allowed. Confirm any 403 in your firewall logs before you change a rule.Split the three policies and date the decision. In Cloudflare, treat Search, Agent, and Training as independent. For most service businesses: allow search, allow agent, decide training on whether your content is the product. Note who approved it and when, because the next migration will rewrite your file again.
Pull 30 days of server logs filtered to OAI-SearchBot, ChatGPT-User, and Perplexity-User. Some traffic means you are in the consideration set. Zero hits across 30 days means no amount of content work helps until the request gets through.
FAQ
Should I block AI crawlers from my website?
Not as one decision. Decide training crawlers on whether your content is the thing people pay you for: a law firm's practice area pages are marketing, a paid research library is inventory. For answering crawlers, almost every local and service business should allow, because that is the pipeline into AI answers.
Is my website blocking AI crawlers if I never changed anything?
Possibly, yes. Platform defaults, security plugins, and managed CDN features add and remove AI crawler rules with no owner action, and Cloudflare's managed block disappeared from sites in September without their owners touching a file. HasData found the count of sites blocking GPTBot without declaring it rose from 115 in July to 178 in September.
Will blocking AI training crawlers hurt my Google rankings?
No. Google-Extended is separate from Googlebot, which handles search indexing, and Google states that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." If you are seeing ranking movement, look at Google's September 2026 spam update, which began September 24, first.
The Close
Most websites no longer know what their own server tells the outside world. AI visibility is not yet a content problem. It is a plumbing problem wearing a content problem's clothes.
Book a 30-minute AI visibility check. We run the crawler tests and send you the status codes. No pitch, just a read on whether the machines can see you. Get a free strategy call.