AI Crawler User-Agent Strings Change: How to Keep Your Traffic Filters Accurate

Yes, you should update your traffic filters. OpenAI says the version number inside its crawler user-agent strings may change. A filter that matches the full string will quietly stop working, and bot hits can slip into your visit counts.

The fix is simple. Match the stable bot name, not the whole string, and review your rules on a schedule. This post explains what changed, why it matters for your reports and how to check your own setup in about thirty minutes.

What OpenAI changed

In July 2026, OpenAI clarified its crawler documentation. A report from an established search industry publication covered two points. The version number in the user-agent string may change. OpenAI may also add a marker when its crawler fetches your robots.txt file.

The example marker looks like this: OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot. According to that report, the marker helps site owners tell robots.txt requests from other requests, especially in logs that do not show file paths.

OpenAI's own bot documentation now lists three crawlers with different jobs. Each has its own user-agent token and its own published list of IP ranges.

• OAI-SearchBot powers search features in ChatGPT. OpenAI recommends allowing it in robots.txt.
• GPTBot crawls content that may be used to train generative AI models. Disallowing it signals that your content should not be used for training.
• ChatGPT-User visits pages when a person asks ChatGPT or a custom GPT to do so. OpenAI notes that robots.txt rules may not apply to these user-initiated actions.

Why this matters for your reports

Many site owners filter bots out of reports by matching a user-agent string. If that string includes a version number, the filter works today and fails after the next update. The bot hits then return to your numbers with no warning.

The effect depends on your tool. Some tools count only browser visits that run a script, so most crawlers never appear. Server logs, however, record every request. Any report built on raw logs is exposed to this problem.

The documentation also shows that the strings differ in style. OAI-SearchBot carries a full browser-style prefix, while GPTBot and ChatGPT-User use a shorter one. A rule written for one pattern will not catch the other.

Step 1: Find the rules you already have

Start by listing every place where you separate bots from people. Most sites have more than one.

1. Check your analytics tool for any bot or internal traffic filters.
2. Check your web server or hosting settings for user-agent rules.
3. Check any firewall or security service that blocks or labels bots.
4. Check your robots.txt file for the bot names you list.

Write each rule down with the exact text it matches. This short list is your starting point.

Step 2: Match the name, not the version

Rewrite each rule so it looks for the bot token and ignores the number after the slash. For example, match the text OAI-SearchBot instead of OAI-SearchBot/1.4. Do the same for GPTBot and ChatGPT-User.

Keep matching case-insensitive where your tool allows it. Also avoid rules that require the exact link at the end of the string. OpenAI points to its documentation pages, and those addresses could change as well.

Step 3: Count the robots.txt requests separately

The new marker gives you a cleaner view. Requests that carry it are fetches of your robots.txt file. They are not page views and they are not visits.

If your logs show file paths, filter by the path. If they do not, use the marker. Then keep robots.txt fetches in their own line in your bot report. This avoids a common mistake, where a rise in raw bot requests looks like rising interest in your pages.

Step 4: Confirm with published IP ranges

A user-agent string is only a label, and anyone can copy a label. OpenAI publishes IP range lists for each of its three crawlers. Use them as a second check when a number looks wrong.

A request that claims to be OAI-SearchBot but comes from an address outside the published list should not count as that bot. Treat it as unverified traffic and keep it out of your AI numbers.

Step 5: Keep three buckets in your report

Your report reads best when it separates three kinds of activity. The first is people who clicked a link from an AI assistant and arrived on your site. The second is crawlers that read your pages for search or training. The third is fetches that happen because a person asked an assistant to read a page.

Only the first bucket is a visit in the usual sense. The other two show that AI systems are reading your content, which is useful, but they are not customers. Adding them to your visit total overstates your audience.

Step 6: Set a review date

Put a quarterly check on your calendar. Open each AI company's bot documentation page and compare it with your rules. Look for new bot names, changed strings and new IP lists.

When you find a change, record the date and what you updated. A simple note such as "filters changed after version update" will explain any step change in your charts later. Without that note, a sudden drop or jump in traffic is hard to interpret.

What to do with the numbers

Clean filters give you better decisions. If search crawler requests rise while AI-referred visits stay flat, your pages may be read but not clicked. That points to your page titles and opening answers, not your hosting.

If training crawler requests are high and you do not want that, your robots.txt rules are the place to act. If you are fine with it, leave it alone and keep it out of your visit counts. Either choice is valid. The goal is to make it on purpose.

A quick self-check

• Do my filters use bot names without version numbers?
• Do I count robots.txt fetches apart from page requests?
• Have I checked suspicious traffic against published IP ranges?
• Do I have a dated note for each filter change?

If you can answer yes to all four, your bot-versus-person line should hold up through the next documentation update.