Robots.txt and AI Crawlers: What NZ Business Owners Should Check

Robots.txt tells compliant crawlers which paths they may request. It is part of crawlability, but it is not a direct recommendation switch and it is not the same as ranking or citation.

Check before changing anything

Look for broad Disallow rules, noindex tags, CDN or firewall blocks, login-only content and important information rendered only after a browser script runs. Confirm that the sitemap contains the pages you actually want discovered.

Make an intentional choice

Some owners may choose to limit particular crawlers for legal, commercial or privacy reasons. Others want public business information to be discoverable. The correct choice depends on your policy. Do not block every bot by accident and do not expose private information just for visibility.

Test the full chain

A crawler may be able to access a page that an answer engine never cites. Conversely, an engine may use other public sources. Review robots.txt, Search Console indexing and your AI visibility results separately.