An AI crawler audit starts by identifying which product and crawler a rule affects. Search discovery, user-requested fetching and model training can have different controls. Changing robots.txt without checking the served file, logs and business preference can block the wrong route.
Decision checklist
| Step | Practical check |
|---|---|
| Read the live rules | Fetch robots.txt and note user-agent groups and disallow paths. |
| Map the purpose | Match each named crawler to its official documentation. |
| Check delivery | Inspect CDN or server logs and response status where available. |
| Test a change | Stage the smallest rule change and verify the live response. |
How to apply it
Create a table with crawler, purpose, current rule, observed requests, response code, business choice and owner. For Google Search AI features, read Google’s current Search controls; for OpenAI, distinguish its documented search and training crawlers. A robots rule is a crawl instruction, not proof that a page was cited or indexed. If the site uses bot protection, a 403 may come from the CDN rather than robots.txt. Record the previous file and rollback route before deployment.
Worked example
Synthetic audit: the business wants public pages discoverable in ChatGPT Search but has blocked the search crawler while allowing a separate training crawler. The team verifies the exact user-agent rules in the live file, checks requests in logs, and proposes a targeted change for owner approval. It does not infer visibility from a single bot visit or publish a broad allow-all rule.
Limits to keep in mind
Crawler names and product controls can change. Use current primary documentation, and do not expose private paths merely to improve AI visibility.
Review checklist
- Is the crawler purpose verified from its owner?
- Could a CDN rule explain the observed block?
- Is the proposed change narrow and reversible?
Common question
Does allowing a bot guarantee AI citations?
No. Access is one prerequisite; answer selection and citations depend on the system and question.
Next useful step
Once access is understood, separate AI referrals from answer visibility: Measuring AI Referral Visits Without Treating Them as Total Visibility. To discuss the specific business decision, see Megvi’s SEO service and bring the source records for review.
Source checked 27 October 2026: OpenAI: overview of web crawlers. Verify current platform rules and interfaces before changing a live setup.
Worked examples are synthetic or illustrative unless a public case is explicitly named; they are not claimed Megvi client outcomes.
Ahmed Elneil
https://megvidigital.comAhmed Elneil is a certified digital marketer and entrepreneur with over 5 years of experience helping brands grow through Search Engine Optimization (SEO), paid advertising, and CRM automation.