Your site may be blocking AI without anyone having decided to

Before optimizing anything so that AI cites you, check that AI can get in. Many sites return a block to the ChatGPT, Claude or Perplexity crawlers because of their CDN’s default configuration, not because anyone decided it. You verify it from the outside, with one command, in two minutes.

How to check it

Request your own page pretending to be the crawler, and look at the response code:

for bot in GPTBot ClaudeBot PerplexityBot Googlebot; do
  echo "$bot $(curl -s -o /dev/null -w '%{http_code}' -A "$bot" https://yourdomain.com/)"
done

200 means it gets in. 403 or 401 means you are blocking it. Do it against /robots.txt and against an inner page too, not just the home page: rules can apply per path.

Why the robots.txt in your repository proves nothing

Because it may not be the one the site serves. Some CDNs offer to manage robots.txt for you, and when that option is on they rewrite the file in transit. The one in the repository says one thing; the one that reaches the crawler says another.

It is the same principle as with anything that passes through an intermediary: what counts is what comes out, not what you wrote.

It happened to us

It is uncomfortable to tell, and it is the reason this post exists.

Until August 24, 2026, sonika.space was blocking the AI crawlers. In two layers at once, and nobody had put either of them there on purpose:

Layer	What was active	What it did
Bots	Blocking of AI training bots	Returned a block to the crawler
robots.txt	robots.txt managed from the CDN	Rewrote the repository’s, which said yes

The second one is the one that teaches something. Our robots.txt explicitly said we wanted to be read and cited. The CDN was replacing it before it reached anyone.

And there is a detail that is hard to find: the middle option Cloudflare offers — a content signals policy — removes the Disallow lines but leaves training marked as not allowed. To clear it we had to turn the whole management off.

On August 25 we queried fifteen user-agents against three paths and recorded the code: all fifteen answer 200. The live robots.txt is the repository’s again.

Crawlers are no longer one single thing

Cloudflare stopped treating them as one block and split them by what they do with the content after reading it. Its three categories, with their definition:

Category	What it does
Search	Collects and indexes your content so it can answer questions about it later
Agent	Acts in real time on behalf of a person who asked for something right now
Training	Takes your content to train or fine-tune a model

The distinction matters because these are different decisions. Not wanting your content inside a model’s training does not imply not wanting to show up when someone asks about your category.

The trap: blocking training also blocks Google

This is the detail almost nobody mentions, and it is Cloudflare’s own wording:

“Multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training.”

A crawler that does both things is evaluated under both policies. So ticking “block training” — which sounds like prudence — also takes you out of Google, Apple and Bing search indexing.

About September 15, 2026 it pays to be precise, because it circulates inflated: Cloudflare sets new defaults that day, only for domains onboarding from then on and only on pages that display ads, where Training and Agent are blocked. Search stays allowed. A domain that is already there does not change on its own.

What we don’t know

• We don’t know whether unblocking worked. We were not measuring before and reindexing takes weeks. We have no figure and we are not going to invent one.
• We don’t know how long we were blocked. We found it in July; when it started is not on record.
• We don’t know how many sites are like this. It is tempting to say “it happens to everyone”. We don’t have the data.
• A crawler getting in does not mean it cites you. It is a necessary condition and nothing more. Everything else — structure, direct answers, evidence — is still needed.

What to do with this

Run the command against your domain. If everything answers 200, you are done and it took two minutes. If something answers 403, you now know why AI doesn’t mention you, and the cause was probably nobody’s choice.

If you want the full picture of why this matters, it is in GEO: how to get AI to cite your company.