Check AI Crawler Access Before Reading Referral Data

Before judging AI referral traffic, check whether major assistant crawlers can reach the pages people may later visit. A site can appear open in robots.txt while a firewall, CDN, or server rule returns an access denial.

That distinction matters because referral reports show completed visits, not every failed retrieval. If a crawler cannot fetch a public service page, product page, or article, an assistant may have less information to use when answering a related question.

Why crawler access belongs beside referral reports

AI referral analytics usually starts with sessions, referrers, landing pages, and conversions. Those measures describe people who reached the site after selecting a link or source shown in an assistant response.

Crawler activity happens earlier and serves a different purpose. A crawler requests pages, reads available content, and may provide information that supports future answers. It does not represent a human visit, so it should not be counted as referral traffic.

A low referral count can therefore have several explanations. The site may not receive many cited links, visitors may arrive without a recognizable referrer, or relevant crawlers may receive denials before they can read the content.

Three access checks to run

1. Review robots.txt

Start with the site's robots.txt file and record rules that apply to recognized assistant crawlers. Look for disallow statements covering the whole site, important directories, or individual files.

Do not treat an empty restriction list as proof that access works. Robots.txt gives automated systems instructions, but it does not force a server or security service to accept a request.

• Save the current file and its last change date.
• Check whether rules cover public pages that matter to customers.
• Check for broad rules that unintentionally cover those pages.
• Record differences between testing and production environments.

2. Check the response at the edge

Next, review logs from the firewall, CDN, hosting layer, or other service that receives requests before the application. Look for status codes, challenge pages, rate limits, and blocked user-agent patterns.

A 403 response means the request was denied. It differs from a robots.txt restriction because the server or security layer actively refused the request.

Review the complete request path. A crawler may receive a denial from an edge rule even when the origin server would have returned the page successfully.

• Record the requested URL and response status.
• Record the exact user-agent string.
• Record the rule or policy that produced the response when available.
• Check whether repeated requests trigger a challenge or rate limit.

3. Confirm the request identity

A user-agent string is a claim made by the requesting software. Treat it as a useful signal, not as proof of identity by itself.

Where appropriate, compare the source address with information published by the platform operating the crawler. Also check request patterns, DNS information, and whether the crawler follows the site's stated instructions.

This step helps separate a genuine platform crawler from software that copies a familiar user-agent string. It also reduces the risk of weakening security rules for an unverified requester.

Keep people and crawlers in separate report groups

Do not combine crawler requests with human sessions. Crawlers may request pages without running browser scripts, accepting cookies, or completing a form, so they do not behave like ordinary visitors.

Create separate reporting groups for at least three kinds of activity:

• Human referral visits: browser sessions that arrive from a recognizable assistant or answer engine.
• Verified crawler requests: automated requests that pass the site's identity checks and access rules.
• Unverified automation: requests that resemble crawlers but cannot be confirmed.

Keep raw logs available before applying filters. A filter can improve a dashboard while hiding evidence needed to investigate a sudden change.

How to interpret a blocked request

A blocked crawler does not prove that a page would have earned a citation or referral visit. It only shows that the request did not receive the expected content.

Use careful language in internal reports. Say that access may limit retrieval, rather than claiming that the block caused lost traffic or lost sales without supporting evidence.

Compare blocked requests with page importance. A denial for a private account area may be correct, while a denial for a public pricing, services, or contact page may deserve review.

A practical review process

1. List the public pages that answer common customer questions.
2. Review robots.txt rules affecting those pages.
3. Inspect edge and server logs for crawler requests.
4. Record status codes, user-agent strings, and blocking rules.
5. Verify crawler identity before changing security settings.
6. Test the page response again after an approved rule change.
7. Compare crawler access records with human referral reports.

Keep the review focused on specific pages and decisions. A whole-site access change may create unnecessary exposure when only a small group of public pages needs consideration.

Protect privacy while measuring access

Access testing should collect only the information needed to diagnose the request. Status code, requested path, timestamp, user-agent string, and a suitable technical identifier may be enough for a short investigation.

Limit access to raw logs and define how long the investigation data should remain available. Avoid placing private customer information into notes, dashboards, or exported reports.

Referral measurement also needs restraint. A missing referrer does not identify a person or reveal why someone visited. Treat direct, unattributed, and assistant-related visits as measurement categories, not as personal profiles.

What to report each month

A useful monthly review can fit on one page. Include crawler requests by verified status, blocked requests by reason, affected public pages, human AI referrals, landing pages, and conversions.

Separate observations from decisions. For example, a report might show that a public page returned access denials during a defined period, then recommend reviewing the responsible security rule.

This format prevents teams from confusing availability with popularity. Crawler access answers whether information could be retrieved, while referral analytics answers whether people arrived and took action.

The strongest measurement practice uses both views without merging them. Check access first, classify automated requests carefully, and then read human referral numbers with the limits of the data in mind.