What is robots.txt?

robots.txt is a text file published at https://site.com/robots.txt. User-agent lines say which bot a rule applies to, and Disallow and Allow lines say which paths may or may not be crawled. A Sitemap: line can point bots to your sitemap.

#Why it matters

robots.txt controls crawling, not indexing. By keeping bots away from low-value paths like admin areas, carts or endless filter combinations, it steers crawl budget toward important pages. One wrong line (for example Disallow: /) can stop the whole site from being crawled. Access for AI bots (GPTBot, ClaudeBot, PerplexityBot and others) is also largely managed through this file.

#Where you see it in SEOAsist

• The Technical Audit crawl respects robots.txt and skips blocked paths. It also reads the file's Sitemap: lines to discover pages. If expected pages are missing from a crawl, see Crawl problems.
• Open a site on the Sites page and switch to the AI access tab: the live robots.txt is read, the permission status of AI crawlers is shown, and the robots.txt AI policy card generates a block to append to your existing file.
• SEO Analysis raises a finding when robots.txt blocks AI bots.

#Example

User-agent: *
Disallow: /cart/
Disallow: /search?

User-agent: GPTBot
Allow: /

Sitemap: https://site.com/sitemap.xml

#Common mistakes

• Using robots.txt to remove a page from the index; a blocked page can still be indexed through links. Use noindex for that.
• Blocking CSS and JavaScript files, so Google can't render the page properly.
• Carrying a staging Disallow: / into production.