robots.txt
robots.txt is a plain text file at a site's root that tells crawlers which pages they may crawl.
What it does
Stops crawling of unneeded pages, allows or blocks AI crawlers, and points to the sitemap.
Watch out
It is not a security tool: anyone can read it and bad bots ignore it. To remove a page from the index, use noindex.
Frequently asked questions
Should I block AI crawlers?
Not if you want to appear in AI answers; block selectively if you do not want your content used for training.