Connect Cloudflare: crawler visits

What it is for

Kaiku already measures what comes out of AI engines: when they cite you. Cloudflare shows what comes in: which AI crawlers (OpenAI's GPTBot, Anthropic's ClaudeBot, PerplexityBot…) and search engine crawlers (Googlebot, bingbot) read your pages, which ones, and with what response. By cross-checking the two, the Rank › Technical audit › Crawler visits tab tells whether a problem comes from access (technical) or from content. These are not human visitors: for your audience, GA4 and Search Console remain the reference. On the Home page, a discreet “Why” line under the SEO and GEO scores shows what Cloudflare explains for the selected period (AI crawlers blocked or refused, errors served to Google and Bing, or access without issues): Cloudflare explains the scores, it never enters their calculation.

Create the token step by step

• Sign in to dash.cloudflare.com and open Profile › API Tokens (profile icon at the top right).
• Click Create Token, then, at the very bottom, Create Custom Token.
• Under Permissions, add two rows: Zone · Analytics · Read and Zone · Zone · Read. Recommended, a third row: Zone · Bot Management · Read, so Kaiku can check whether Cloudflare blocks AI crawlers.
• Under Zone Resources, choose Include · Specific zone and your domain (for example exemple.ca).
• Click Continue to summary, then Create Token. Copy the token: Cloudflare shows it only once.

Enter it in Kaiku

• In Settings › Connections, scroll to the Cloudflare connection card.
• Paste the token, then click Save the token. It is encrypted at rest and never sent back to the browser in clear.
• Kaiku picks the site (the "zone") matching the project's domain. If several sites are possible, choose it from the list, then click Save the site.
• The first reading fetches the last 29 full days (Cloudflare's limit, with a margin); it usually takes a few minutes. After that, Kaiku reads new visits every Sunday.
• If the first reading fails, the Crawler visits page says so; the project owner can then click Retry the reading.
• Disconnect Cloudflare removes the token and the site from the project; visits already read are kept.

"DNS only": no data, and it is not an error

Cloudflare only sees traffic that goes through it. If the site is in DNS only mode (grey cloud in Cloudflare's DNS settings), visits go straight to your host: Cloudflare has nothing to report, and the page says so. This is common when the host runs its own network. To measure crawlers, the site's DNS record must go through Cloudflare's proxy (orange cloud), if your host allows it.

31 days at Cloudflare, the full history at Kaiku

Cloudflare keeps only 31 days of statistics. Kaiku reads them every Sunday, overlapping the days already read to catch up on a missed reading, and keeps everything: the page follows the period chosen at the top of the hub, even beyond 31 days, and says since when the history exists. Changing site or disconnecting Cloudflare erases no data.

How crawlers are recognized

Each crawler is recognized by the name it declares (its "user-agent"): GPTBot, OAI-SearchBot and ChatGPT-User for OpenAI, ClaudeBot and Claude-User for Anthropic, PerplexityBot, Google-Extended, Googlebot, bingbot, and so on. Each one shows its purpose: Training (feeds future models), Search (powers answers with sources), User request (page fetched on the fly to answer a question), Indexing (search engines).

Reading the five diagnostics

• Blocked or failing crawlers: an AI crawler gets a refusal (401, 403), a rate limit (429) or a server failure (5xx) on a page. This is technical: check the firewall, Cloudflare security rules and the server. A 404 counts as an error, not as a block.
• Read but never cited: a publisher's crawlers (for example OpenAI) read the page, but its engine (ChatGPT) cites it in none of its answers to the queries Kaiku tracks for this project, over the period. So it does not mean "never cited anywhere": only in your tracked queries. Access works: the content needs work. Only engines tracked in the project are cross-checked.
• Never visited by AI: an important page (with clicks or impressions in Search Console or Bing, or already cited by an AI) that no AI crawler has read. Link to it from pages already visited and submit it with IndexNow.
• Errors served to Google and Bing: Googlebot or bingbot gets an error (401, 403, 404, 410, 429 or 5xx) on a page; redirects are not counted. Fix the link that leads to it or restore the page; a page removed on purpose can stay at 404 or 410. With no error, the block shows the share of visits served to Google and Microsoft.
• Disallowed by robots.txt, yet visited: Kaiku reads your robots.txt at each Cloudflare reading and checks, crawler by crawler, whether the pages visited were disallowed for it (most specific rule wins, as with Google). robots.txt is only an instruction: to really block, add a Cloudflare security rule. A crawler that ignores the instruction may also be an impostor borrowing its name. Without a robots.txt, crawlers are allowed everywhere. Each visit is checked against the robots.txt read no later than the day before (a crawler may keep the file cached for 24 hours), and only on the site address where it was read: visits before the first reading are not checked. Crawlers triggered by a user (ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User, DuckAssistBot…) are not checked either: they open a page at a person's request, and their publishers state that they do not apply robots.txt, or not always.

Does Cloudflare block AI engines?

Cloudflare offers settings that block AI crawlers, sometimes turned on without anyone noticing. With the recommended Zone · Bot Management · Read permission, Kaiku reads them at each reading and when the token is saved, and shows a banner at the top of the Crawler visits page: red if all AI crawlers are blocked (ChatGPT, Claude, Perplexity and the others can neither read nor cite the site), orange if a category is blocked (AI search crawlers, visits requested by their users) or if the block only targets pages with ads, and a discreet note for what is harmless (model training excluded, anti-AI labyrinth, robots.txt extended by Cloudflare).

• To lift the block: in Cloudflare, open your domain, then AI Crawl Control (or Security › Bots), and turn off Block AI bots.
• Without this permission, the page says so in a small note: add it to the token in Cloudflare (the token stays read-only).
• The banner shows the date the settings were read. If AI crawlers still read your pages despite the block, it says so conditionally: those not yet blocked could be.