Late in October, an SEO peer mentioned something odd in their server logs. On a site they look after, the number of pages ChatGPT fetched live while answering people’s questions had fallen off a cliff. Hundreds a week through mid-October, then a fraction of that from 27 October. Nothing had been blocked. No errors. The homepage was still being visited at the usual rate, but every other page had gone quiet.
One site is one site. Crawler traffic moves around for dull reasons all the time: a schedule changes, a plugin starts rate-limiting, a server has a bad fortnight.
We are unusually well placed to check, because we look after a lot of websites and most of them sit behind Cloudflare. So we pulled a month of logs from 344 of them.
On the 37 sites with enough verified AI traffic to measure, ChatGPT’s live page fetches fell 43% per day after 27 October. Pages other than the homepage fell 62%. Bing’s crawler, over exactly the same days and the same sites, went up 5%. Nothing was blocked, and nothing was broken.
If you are doing any kind of AI visibility tracking from your own server logs, that matters rather a lot, and not for the reason most people assume.
Is ChatGPT a web crawler?
Not exactly, and the distinction turns out to be the whole story. Three different robots from OpenAI visit websites, and they do different jobs.
- ChatGPT-User fetches a page live, in the moment, because somebody asked ChatGPT a
question and it decided to go and look at your site.
- OAI-SearchBot builds OpenAI’s own search index, so ChatGPT can answer from something it
already holds rather than fetching afresh.
- GPTBot gathers data for training future models.
Three robots, three jobs, one user-agent string each. The claim we were testing was about the first one only, and it is the one that behaves most like a visitor.
What we found
Across the 37 sites, verified ChatGPT fetches ran at about 453 a day through the first half of the window, then settled at about 258 a day from 27 October onwards. The weekly totals tell the same story: 3,280 in the week of 12 October, 3,080 in the week of 19 October, then 2,178 and 1,626.
The typical site halved. Thirty of the 37 fell by more than 20%, eighteen of them by more than half, and only two rose.
It was not just a quiet fortnight
This is the test most “AI traffic is down” posts skip. If every crawler had gone quiet on our sites over the same days, the honest conclusion would be that our sites got quieter, not that anything changed at OpenAI. So we tracked Googlebot and Bingbot over exactly the same days and sites as controls.
Bing’s crawler was up 5%. GPTBot was up 4%. Googlebot’s total fell, but nearly all of that came from a single large site having a quiet spell: the typical Googlebot site moved by under 3%. ChatGPT-User is the only crawler where both the overall total and the typical site agree on a large fall.
Homepages held up. Everything deeper did not.
Homepage fetches fell 30%. Fetches to every other page fell 62%. Whatever changed, it changed most for the deep, specific pages: the service page, the comparison, the long answer you wrote precisely because somebody might ask that question.
Nothing was blocked
Before blaming anyone else, we checked ourselves. Of the verified ChatGPT requests that arrived, 95.26% got a clean 200 response before 27 October, and 95.23% after. Refusals stayed under 2% either side. Server errors stayed at zero. ChatGPT did not stop coming because we shut the door on it.
Nearly all “ChatGPT” traffic in your logs is fake
This is the part that genuinely surprised us, and it is the most useful thing in this article even if you ignore everything else.
Across those 344 sites, requests claiming to be ChatGPT-User totalled 681,466 in a month. The number that genuinely came from OpenAI was 11,298. That is 1.7%.
The rest is scrapers in costume. Anybody can put any text they like in the user-agent field of a web request, and plenty of people put “ChatGPT-User” in there because sites tend to wave AI crawlers through. One parked domain in our set produced 343,540 fake ChatGPT requests in a month, alongside near-identical volumes of fake Googlebot, fake Bingbot and fake GPTBot. It was one scraper, cycling through disguises.
To be sure, we took 56,207 individual requests and checked each source IP address against OpenAI’s published list of its own server ranges. 4,700 were genuinely OpenAI. Cloudflare had independently flagged exactly those 4,700 and no others.
So if you have seen an AI visibility report built by counting user-agent strings in server logs, treat the number with real suspicion. Ours would have been sixty times too high.
What are AI visibility metrics, and which ones still work?
Here is the uncomfortable conclusion from our own data: crawler hits are not an AI visibility metric at all. They never measured whether ChatGPT recommended you. They measured whether it needed to go and look something up. If it answers from its own index instead, you can be cited all day and see nothing in your logs.
A workable AI visibility tracking stack has three separate layers, and the mistake is letting them blur together.
- Citations and brand mentions. Does your brand appear in AI answers, and for which
prompts? Tools like Ahrefs Brand Radar and Surfer’s AI tracker sample this across ChatGPT, Perplexity, Google’s AI Overviews and Microsoft Copilot. Imperfect, sampled, and still the closest thing to a share-of-voice figure for AI search visibility.
- Referral traffic. People who actually arrive. In Google Analytics, filter referrals from
chatgpt.com and perplexity.ai. Small numbers for most businesses, but they are real people, and they convert well in our experience.
- Crawler activity. Verified only, and labelled as what it is: evidence that the machines
can reach you, not evidence that they like you. Useful as a health check, useless as a headline.
Keep the three apart in reporting. A citation is not a visit, and a visit is not a lead.
The thing we found that we had to go and fix
While we were in the data, we found something more actionable than the headline.
Across these sites, 29.5% of OAI-SearchBot’s requests were answered with “429 Too Many Requests”. Bing’s crawler got the same treatment, at 27.6%. Googlebot got zero.
Somewhere in the stack, host-level or plugin-level rate limiting had been set up to wave Google through and throttle everybody else. That configuration was probably made years ago, when “everybody else” meant scrapers. Today it means the crawler that builds the index ChatGPT answers from. If live fetching is being replaced by stored knowledge, being refused on three out of ten indexing requests is a direct visibility cost, and it is ours to fix, not OpenAI’s.
How to check your own AI visibility this month
A short, practical routine. None of it needs a new subscription.
- Check that OAI-SearchBot and GPTBot are not blocked or throttled. Look at robots.txt,
any firewall or bot-protection rules, and your host’s rate limiting. If Googlebot is the only crawler getting a clean run, somebody decided that, and it now has a price.
- Verify before you count. If you look at crawler hits at all, count only verified bots.
Cloudflare marks them for you; OpenAI publishes its IP ranges at openai.com/chatgpt-user.json.
- Keep freshness signals honest. If an index decides when to re-read you, accurate sitemaps
and last-modified dates are how you get re-read. Clean structured data helps the same cause.
- Write the deep pages to be answerable. The pages that lost the most fetches are the deep
ones. Give each one a direct answer near the top, in plain language, before the preamble.
- Start your own record. Cloudflare’s free plan forgets after 31 days, which is exactly why
we could not look back to September. We now store verified crawler counts weekly for every client, so the next time this question comes up we can answer it over a year rather than a fortnight.
The honest caveats
Thirty-seven sites is not the web. They are mostly small UK businesses and niche content sites, and together they produce about 450 verified ChatGPT fetches a day, which is a modest number to reason from. Fourteen days after the change is enough to see a step, not enough to declare a new normal. And we can see that behaviour changed on a date, not why. A stored index explains it neatly, but so would a routing change or an internal quota, and from our side of the wire those look identical. We will re-run the numbers in a fortnight.
What we can say is this: somebody noticed something on one website, we checked it against 344 of ours, and on the 37 where the question can be answered at all, the same thing happened.
We run AI visibility tracking as part of SEO for every client, using verified crawler data rather than user-agent guesswork. If you would like to know what the AI crawlers are actually doing on your site, get in touch. It takes about an hour to check properly.
