How do I let AI crawlers read my site?

Check what you are blocking today

Open yoursite.com/robots.txt and read it. Many sites block AI crawlers without their owner ever deciding to - hosting providers, WAFs, CDN bot-protection rules and security plugins all do it by default.

A blanket Disallow on a broad user-agent rule is the version that catches people out, because it looks harmless until you notice it also matched every AI agent.

The crawlers worth naming explicitly

These are the agents that determine whether your own site is available as a source.

• •GPTBot - OpenAI training crawler
• •OAI-SearchBot - powers ChatGPT search results
• •ChatGPT-User - fetches a page when a user asks ChatGPT about it
• •ClaudeBot and Claude-User - Anthropic crawl and live fetch
• •PerplexityBot and Perplexity-User - Perplexity index and live fetch
• •Google-Extended - controls Gemini and AI Overviews use of your content
• •CCBot - Common Crawl, an upstream corpus for many models
• •Applebot and Applebot-Extended - Apple search and generative features
• •Bytespider, Amazonbot, MistralAI-User, YouBot, Diffbot - additional model and agent crawlers

The pattern to publish

Explicit is better than implicit. Rather than relying on a permissive default, name each agent with an empty Disallow so your intent is unambiguous, then keep genuinely private paths blocked for everyone.

Also add a Sitemap line, and a comment pointing at your llms.txt so anyone inspecting the file finds your curated map.

Access is necessary but not sufficient

Letting a crawler in does nothing if there is nothing readable when it arrives. Make sure your business name, services, phone number, hours and service area are present in the server-rendered HTML, not injected later by script.

Blocking a login area, checkout or admin path is fine and expected. Blocking your service and FAQ pages is self-defeating.

Verify rather than assume

After editing, confirm robots.txt returns HTTP 200 as text/plain and that your rules parse. Then check your server or CDN logs for actual visits from these agents over the following days - the logs are the only proof that access is really open.