Robots.txt Tester

How crawlers read robots.txt

1. They pick one group. A crawler looks for the User-agent line that names it most specifically. Googlebot-Image uses a Googlebot-Image group if there is one, then Googlebot, then *. Only that group's rules apply; the others are ignored completely.
2. The longest matching rule wins. Allow: /admin/public/ beats Disallow: /admin/ for pages under /admin/public/ because it is longer, whatever the order in the file.
3. Allow wins a tie. If an Allow and a Disallow rule are the same length, the page is allowed.

* matches any characters and $ marks the end of the URL, so Disallow: /*.pdf$ blocks every PDF. Paths are case-sensitive.

What happens when robots.txt is missing or broken

• 404 or another 4xx: crawlers treat the whole site as allowed.
• 5xx or no response: Google treats the whole site as blocked for a while, then falls back to its last copy. A broken robots.txt can stop crawling of an otherwise healthy site.

robots.txt is not a privacy tool

A Disallow rule asks well-behaved crawlers not to fetch a URL. It does not stop anyone else, and the file itself is public, so listing secret paths there advertises them. A blocked page can also still appear in search results if other sites link to it. To keep a page out of search, allow crawling and add a noindex meta tag instead. To keep it private, put it behind a login.

Limitations

• Crawlers can change their names and rules at any time. We follow the matching rules Google documents; other crawlers may differ slightly.
• Not every AI crawler respects robots.txt. The file can only express your wishes.