What are AI crawlers (bots)?

AI crawlers are the bots AI companies send to websites. They don't all do the same job; by purpose they fall into three groups:

Google-Extended isn't a real bot but a robots.txt token: it only opts out of Gemini training. Appearing in AI Overviews is decided by Googlebot. Blocking Bingbot removes you from both Bing search and Copilot.

#Why it matters

• One wrong robots.txt line can quietly remove you from AI search altogether.
• If you don't want your content used for training, you can do that without losing visibility by keeping search and assistant bots open.

#Where you see it in SEOAsist

Open the site from Sites and switch to the AI access tab:

• Check now reads the live robots.txt and shows an Allowed, Blocked or Partial verdict for 20 AI crawlers. With the Pixel installed, the Hits (28d) column counts the bots' real requests.
• On the robots.txt AI policy card, pick the Open, Search only, No training (recommended), Closed or Custom preset. Copy the generated block, or add it with Apply via PR if GitHub is connected.

More: GEO: AI visibility.

#Example

A robots.txt block in the spirit of "No training":

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

#Common mistakes

• Blocking every AI bot at once and still expecting to show up in AI search.
• Believing that blocking Google-Extended removes you from AI Overviews.
• Forgetting that a broad Disallow in the User-agent: * group also shuts out AI bots.