Annotate Robots.txt Changes Before You Read AI Traffic Trends

Every time you edit robots.txt, write down the date in your traffic report. Without that note, a rise or drop in AI bot hits or AI referral visits looks like a mystery, and owners often credit the wrong cause. A simple change log turns guesswork into a before-and-after comparison.

This post shows what to record, how long to wait before you judge a change, and which kinds of visits a robots.txt edit will not touch.

Why robots.txt edits change your numbers

Robots.txt tells automated crawlers which parts of your site they may request. When you allow or block a bot, its visits to your site can rise or fall. The effect can also reach the people those platforms send to you.

OpenAI documents separate bots for separate jobs. GPTBot collects content for training generative AI foundation models. OAI-SearchBot surfaces websites in ChatGPT search results. Blocking one does not block the other, so a single change can move only part of your data.

Perplexity makes a similar split. PerplexityBot is designed to surface and link websites in Perplexity search results, and Perplexity says it is not used to crawl content for AI foundation models. Check each platform's own documentation before you decide which rule to write.

Expect a delay before the effect shows

OpenAI states that it can take about 24 hours from a robots.txt update for its systems to adjust for search. That means a report pulled the same afternoon tells you nothing. Wait several days, then compare a full week before and a full week after.

Referral visits can lag even longer. A crawler must fetch your pages, the platform must update what it can show, and then people must click. Treat any early swing as a hint, not a result.

Use matching windows

Compare the same days of the week. A Monday through Sunday block before the change should line up with a Monday through Sunday block after it. Holidays, sales and email sends can swamp a small AI signal, so note those too.

Know which visits robots.txt cannot touch

Some AI visits happen because a person asked a question. OpenAI states that ChatGPT-User handles user-initiated actions, and that robots.txt rules may not apply to them. Perplexity states that Perplexity-User generally ignores robots.txt rules, since a user requested the fetch.

This matters for your report. If you block a crawler and still see hits from a user-triggered fetcher, your rule did not fail. Those requests follow a different path, so keep them in their own row and do not count them as crawling.

• Training crawlers: robots.txt is the main control.
• Search crawlers: robots.txt controls whether you can appear, with a delay.
• User-triggered fetchers: robots.txt may not apply, so expect some hits to continue.

Build a simple change log

You do not need special software. A shared spreadsheet or a notes page next to your report works well. Add one row each time anyone edits robots.txt, including your web team and any plugin that rewrites the file.

1. Date and time of the edit, in one time zone you always use.
2. The exact rule you added, removed or changed, copied as text.
3. Bots affected, listed by name.
4. The reason, such as protecting training content or allowing search visibility.
5. Check date, set at least one week out.

Then paste the same dates onto your traffic charts as annotations. A reader who sees a line on the chart will ask the right question before drawing a conclusion.

Read the result with care

On the check date, look at three numbers side by side. First, hits from the bot you changed, taken from your server logs. Second, hits from user-triggered fetchers. Third, visits that arrive with an AI assistant as the referrer.

If bot hits fell but referral visits held steady, your rule did what you wanted. If both fell, you may have blocked a search crawler that helped people find you. That is a business choice, and the log gives you the evidence to make it.

Check the file itself

Before you blame a platform, open your live robots.txt and confirm it says what you think. Plugins, hosting panels and site updates can overwrite it. A rule that was never published cannot explain a trend.

Verify the visitor

A user-agent string can be copied by anyone. Perplexity publishes IP ranges for PerplexityBot, so you can compare log entries against them. Use published ranges whenever a platform offers them, and treat unverified hits as unconfirmed.

Turn the log into decisions

A change log helps you answer plain questions. Did allowing a search crawler bring more visits within a month? Did blocking a training crawler change anything you can measure? Did a plugin quietly undo your work?

Review the log once a quarter. Remove rules nobody can explain, and keep the ones tied to a clear reason. Over time, your report shows cause and effect instead of a line that moves for unknown reasons.

Remember that referral counts are a minimum, since some AI visits arrive without a clear referrer. Use the change log to judge direction, and avoid claiming exact gains from one edit.