AI Crawlers and Robots.txt: A Practical 2026 Guide

Do not manage every AI crawler with one blanket rule. OAI-SearchBot, GPTBot, ChatGPT-User, Googlebot and Google-Extended have different documented purposes. Decide what you want to permit—search discovery, possible training use or user-triggered access—then configure and test each layer. A robots.txt rule alone does not guarantee indexing, removal or successful access.

The biggest mistakes are usually not advanced. A team copies a generic blocklist, a CDN challenges the wrong agent, or a broad wildcard overrides an intended allow rule. The result is either unwanted access or accidental disappearance from a search surface.

Key takeaways

• OAI-SearchBot controls ChatGPT search discovery; GPTBot is a separate control for possible model-training use.
• ChatGPT-User represents user-triggered visits and does not determine search inclusion.
• Googlebot governs Google Search, including AI Overviews and AI Mode.
• Google-Extended is a separate token for certain Gemini and generative AI uses; it does not control Google Search inclusion or ranking.
• Robots.txt manages crawling, not every form of indexing or removal.
• Verify server responses, rendered content, CDN/WAF policy and logs after every rule change.
• Keep an internal crawler register with purpose, owner, policy and review date.

Table of contents

• Crawler reference table
• What robots.txt does
• OpenAI policy example
• Google controls
• Four test layers
• Monitoring

AI crawler reference table

Policies and names can change. Link your internal register to official documentation and review it on a scheduled basis.

What robots.txt actually does

robots.txt is a public file at the root of a host—for example, https://example.com/robots.txt. Compliant crawlers request it and interpret rules that apply to their user agent.

It is a crawling directive, not an authentication system. A disallowed URL remains public to anyone who knows the address. Robots rules also do not replace:

• noindex or other page-level indexing controls;
• login or access control for private information;
• CDN and firewall configuration;
• canonical tags;
• removal tools and legal requests;
• content licensing terms.

Never place secrets in robots.txt. Disallowed paths are visible to the public.

Policy example: allow ChatGPT search, disallow GPTBot

OpenAI documents OAI-SearchBot and GPTBot as independent controls. A site that wants ChatGPT search eligibility but does not want the documented GPTBot use could start with:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

This is only a starting point. Your existing file may contain more specific path rules or a wildcard group. Review the complete file, not an isolated snippet.

OpenAI also publishes IP ranges for its crawlers. If a security layer validates IPs, use the current official list and automate updates carefully. Do not assume the user-agent string alone proves identity; it can be spoofed.

OpenAI says robots changes can take roughly 24 hours to be reflected. Log the deployment time before declaring that a test has failed.

Policy example: Google Search versus Google-Extended

For visibility in normal Google Search and its AI features, Googlebot must be able to crawl eligible pages. Google states that Google-Extended does not affect Search inclusion or ranking.

A business might allow Google Search while restricting the separate Google-Extended use according to its policy. The exact rules should come from current Google documentation and legal/product requirements, not a copied social-media post.

The important concept is:

Search crawler control and generative-model usage control are not necessarily the same setting.

Document why each decision was made. This prevents a future engineer from “cleaning up” apparently contradictory rules that were intentional.

Why Allow does not guarantee visibility

Permitting a crawler clears one technical barrier. A page may still be absent because:

• the URL is not linked or discovered;
• it returns an error, redirect loop or soft 404;
• the canonical points elsewhere;
• the main answer is hidden behind client-side interactions;
• content is thin, duplicated or unhelpful;
• the page is not eligible for indexing or snippets;
• security tooling blocks the request after robots.txt is read;
• the system simply selects other sources.

Crawler access is an eligibility decision, not a ranking request.

Why Disallow is not a secure removal method

Robots.txt tells compliant crawlers not to fetch a path. It does not make that path private. Search systems may know a URL exists from links even if they cannot crawl its contents. If material must not be public, require authentication or remove it from the public web.

For index management, follow each search engine's documented directives and removal process. Avoid blocking a page in robots.txt before a crawler can see a necessary noindex directive without understanding the consequence.

The four layers you must test

1. Robots policy

Fetch the live /robots.txt, verify the status and parse the group that applies to each user agent. Test the exact URL paths that matter.

Bora's robots.txt tester helps check whether a rule permits a given path. Save the result with the date and deployed file version.

2. Network and security policy

A CDN, reverse proxy or WAF can override the apparent robots permission. Look for:

• bot-management challenges;
• rate limits;
• geo restrictions;
• IP allowlists or denylists;
• TLS failures;
• repeated 403, 429 or 5xx responses;
• inconsistent behavior between origin and edge.

Validate official IP ranges where the provider supplies them. Make security changes narrowly and review them with the responsible team.

3. URL and HTML response

For a representative set of pages, verify:

• final status is 200;
• redirects reach the canonical page efficiently;
• the canonical tag is correct;
• robots meta and X-Robots-Tag directives are intentional;
• the title, H1 and main answer are present in rendered output;
• structured data matches visible facts;
• alternate language and mobile behavior are coherent.

Use the redirect checker and SEO checker as a first pass, then inspect responses directly.

4. Discovery and evidence

Check whether the URL appears in the sitemap and has contextual internal links. A technically reachable orphan page is still difficult to discover and understand.

Connect guides to category hubs, evidence pages and conversion destinations. The GEO vs SEO guide provides the strategic architecture behind these technical controls.

A safe test procedure

1. Inventory current robots rules and security policies.
2. Define intended access by crawler and directory.
3. Test proposed rules in a non-production copy or parser.
4. Review conflicts with wildcard and more-specific groups.
5. Deploy the smallest required change.
6. Fetch the public file from outside the origin network.
7. Test a homepage, article, product page and blocked path.
8. Review edge and origin logs after the provider's stated refresh period.
9. Record results, owner and rollback procedure.

Avoid testing by impersonating a crawler in a browser and treating the result as proof of identity. A user-agent header is only one part of verification.

Common robots.txt mistakes

Blocking all crawlers to stop model training

A global block can also remove search visibility and legitimate tooling. Use the provider-specific controls that match the policy.

Assuming GPTBot controls ChatGPT search

OpenAI documents OAI-SearchBot separately. Set both rules intentionally.

Assuming Google-Extended controls AI Overviews

Google says Search AI features are governed by Googlebot and Search controls, not Google-Extended.

Forgetting subdomains

Each host has its own robots.txt. Rules on www.example.com do not automatically configure docs.example.com or shop.example.com.

Returning the wrong status

A redirect, error or HTML page at /robots.txt can create unpredictable interpretation. Serve a valid plain-text file and monitor it.

Blocking assets required for understanding

If CSS, JavaScript or other resources are essential to render the page, blocking them may hinder evaluation. Keep the main content available in robust HTML and test rendered output.

Treating a bot-management dashboard as the source of truth

The effective outcome is the live response. A rule may look enabled in one interface while another edge policy wins.

Crawler governance template

Maintain a simple table:

This turns crawler configuration into a maintained policy rather than a one-time SEO edit.

Monitoring after deployment

Create alerts or regular reports for:

• unexpected changes to robots.txt;
• rising 403, 429 or 5xx responses;
• sudden disappearance of crawler requests;
• accidental noindex or canonical changes;
• sitemap errors;
• unusual load caused by automated access;
• new provider documentation or user-agent changes.

Logs contain noisy and spoofed traffic, so validate identities before drawing conclusions. Look for patterns over time and compare both origin and CDN data.

How this supports AI recommendations

Correct crawler configuration does not make a weak page recommendable. It prevents technical policy from undermining good content. Once eligible, the page still needs a direct answer, original value, clear entities and credible evidence.

Read how to get recommended by ChatGPT for the full OpenAI-focused workflow and how to appear in Google AI Overviews for Google's Search guidance.

Where Bora helps

Bora's free tools can test robots rules, redirects, sitemaps, metadata and page-level SEO. Its automated publishing workflow can keep internal links, schema and CMS delivery consistent across a growing content library.

Crawler governance and security policy still require a responsible human. Bora can reduce repetitive verification; it should not silently decide whether an organization permits training-related access. Explore Bora's features or compare plans after the policy is defined.

Sources and freshness notes

• OpenAI crawler documentation — crawler roles, robots controls, refresh timing and IP ranges; reviewed 4 September 2026.
• Google AI features and your website — Googlebot and Google-Extended distinction; reviewed 4 September 2026.
• Recheck both primary sources before changing production rules.

Editorial implementation notes

• Have technical SEO and security review the code examples against Bora's current infrastructure.
• Add dated screenshots from the robots tester and sanitized log examples if available.
• Never publish internal IPs, private paths or security bypass instructions.
• Use a real engineer as author or reviewer and add BlogPosting schema.