robots.txt tester

What this tester evaluates

Paste a robots.txt file, type a user-agent string, and type one path. The page parses that text in the browser and reports whether this matcher would allow the path. It does not download a live robots.txt, open the path, or decide whether a search engine will index the URL.

How a group is chosen

Lines are read in order. A hash starts a comment and the rest of that line is ignored. User-agent lines that sit together form one group. The first Allow, Disallow, or other field closes that group. Matching is case-insensitive. A specific token wins over * when the user-agent you typed contains the token, or the token contains the string you typed. The longest such token wins. If none match, groups whose token is * are used. Sitemap lines are collected and then ignored for the allow decision. Crawl-delay and Host are recorded as unused directives. A line without a colon is reported as malformed and skipped.

Allow, Disallow, and the longest match

Only rules in the chosen group are considered. A path must start with/. An empty Disallow is skipped, so a blank Disallow does not block the site. An empty Allow is treated as /. In a pattern, * matches any stretch of characters and$ means the path must end there. A pattern without$ also matches longer paths that continue past it. When several rules match, the longer path pattern wins. If two matching patterns have the same length, Allow wins over Disallow. If nothing matches, the result is Allowed, with the reason “No matching Allow/Disallow rule. Default allow.”

The sample file

The box starts with User-agent: *, Disallow: /admin, and Allow: /admin/public. With user-agent Googlebot and path /admin, the heading is Blocked because only the Disallow pattern matches. Change the path to /admin/publicand the heading becomes Allowed: both rules match, and/admin/public is the longer pattern. A path such as/pricing matches neither rule, so the result is Allowed by the default.

How to read the result

Allowed and Blocked refer only to this pasted file and this matcher. The sentence under the heading names the winning line, for example “Disallowed by longest-match rule: Disallow: /admin”. Copy result copies that sentence. It does not copy a crawl from Google, and a Blocked result is not a noindex tag, a canonical, or a removal from an index.

Limits of this page

• It never requests the site named in the path.
• It does not apply Crawl-delay, Host, or Sitemap to the decision.
• It does not model every search engine’s unpublished exceptions.
• A prefix rule such as Disallow: /admin also covers /admin/settings, because the pattern has no end anchor.
• Use the robots.txt generator on this site when you want to draft a file. This page only tests a file you already have.