Robots.txt Generator
Build a robots.txt file from a few choices: what to keep crawlers out of, how to treat AI crawlers, and where your sitemap is.
Rules
Paths start with /. End a folder with /. Use * for any characters and $ for the end of the URL, as in /*.pdf$.
Your robots.txt
robots.txt
What robots.txt can and cannot do
robots.txt asks crawlers not to fetch certain URLs. Search engines and the major AI companies follow it. It does not hide pages from people, and it does not guarantee a page stays out of search results: a blocked URL can still be listed if other sites link to it, just without a description. To keep a page out of search, allow crawling and use a noindex tag instead.
Choosing what to block
- Good candidates: internal search results, cart and checkout pages, admin areas, and URLs that only differ by sorting or filter parameters.
- Do not block: CSS, JavaScript and images your pages need. Google renders pages like a browser, and blocking these can hurt how it understands them.
- Never rely on it for private files. Anyone can read robots.txt, so listing secret folders there points people straight to them.
About the AI crawler choices
Some companies use separate crawlers for training AI models and for showing pages in AI search or answers. Blocking training only still lets your pages appear and be linked in those answers. The list covers well-known crawlers that say they follow robots.txt; new ones appear regularly, and not every crawler respects the file.
Limitations
- Crawler names change. Check each company's documentation if you need to be sure of current behaviour.
- Test the finished file with the Robots.txt Tester before relying on it.