Robots.txt Tester & Validator
The Robots.txt Tester & Validator checks whether a robots.txt file allows or blocks a specific URL for a named crawler. Robots.txt tells crawlers which URL paths they may request, so a rule decides crawl access rather than indexing.
What robots.txt does
Robots.txt is a plain text file at the root of a host that tells crawlers which URL paths they may request. It controls crawling, and it does not control indexing.
The file is grouped by user agent. Each group carries one or more Allow or Disallow rules written as paths, and a crawler selects the group that applies to it before evaluating any rule.
A crawler reads the file before it requests other URLs. A path it may not request is never fetched, so no page-level directive in that response can be read by that crawler.
What a robots.txt file looks like
A file is a series of user-agent groups, and the longest matching path rule decides access for a matched crawler.
A typical robots.txt file
User-agent: *Disallow: /admin/Disallow: /cart$ User-agent: GooglebotAllow: /admin/public/ Sitemap: https://example.com/sitemap.xmlHow a crawler selects its robots.txt group
A crawler matches its user-agent token against the groups in the file and obeys the most specific applicable group, so a Googlebot group takes precedence over a wildcard group for Googlebot.
User-agent token matching
User-agent lines are matched by token. A group for googlebot applies to Googlebot, and a group for a wildcard applies to crawlers without a matching named group. A crawler that matches no group may request any path.
How named groups combine
Named groups add up rather than override for unrelated crawlers. Adding a Bingbot group does not change Googlebot access, and removing a wildcard group does not remove a named group.
Empty groups
An empty group, meaning a user-agent line with no following Allow or Disallow rule, leaves that crawler unrestricted. A group can therefore look restrictive in a file and still permit every path.
Which rule wins when several match a URL
When more than one Allow or Disallow rule matches a path, the rule with the longest matching path governs, regardless of the order the rules appear in the file.
Match length decides precedence
Length is measured in matched characters rather than in line number. A rule that matches a longer path beats a shorter rule, so a longer Allow can reopen one path inside a broader Disallow.
Wildcard and end-anchor patterns
Rules support two wildcard forms: an asterisk matches any sequence of characters, and a dollar sign anchors the pattern to the end of the URL. A pattern without the end anchor matches any URL that begins with the pattern.
Reopening a single path
Because precedence depends on match length, a broad Disallow followed by a targeted Allow is a supported way to keep most of a directory closed. Confirm the verdict by testing the exact path, including its query string, rather than reasoning about the file alone.
Robots.txt rules that are easy to get wrong
A path written without a leading slash, a pattern that stops short of what it should match, and a directive left over from an old URL structure are the recurring causes of an unexpected verdict.
Disallow values must be paths
A Disallow value must be a path beginning with a forward slash. A value such as admin does not match /admin/, and a value that looks like a full URL is not read as a path.
A missing end anchor widens a rule
A trailing fragment decides scope. A Disallow for /page also blocks /page-2 and /pages, because the pattern matches any URL beginning with those characters. Add the end anchor to close the pattern to one address.
Comments are not rules
Comments begin with a hash character and are ignored. A commented rule is not a rule, which is a common cause of a file whose intended block is absent.
Crawl access and indexing are separate decisions
An Allow verdict means the tested rules permit a crawler to request the URL. It does not request indexing, and a Disallow verdict does not remove a URL that is already indexed.
A blocked URL can still be listed
A blocked URL can still appear in search results when a crawler discovers it through links and never fetches the content. Removing a listed URL requires an index-side action rather than a crawl rule.
A crawl block can hide a noindex directive
A Disallow rule can also hide the directive that would have excluded a page. A crawler that cannot fetch the URL cannot read a noindex directive from the response, so a crawl block placed over a noindexed page can leave the page in the index.
Robots.txt is not access control
Robots.txt is public guidance rather than access control. A private or confidential resource needs authentication and authorization, and listing a sensitive path in the file reveals it to people and to crawlers that ignore the protocol.
Rule patterns and the paths they match
| Rule | Matches | Does not match | Winning behaviour |
|---|---|---|---|
| Disallow: /private/ | /private/report.pdf | /public/report.pdf | Blocks the whole directory |
| Disallow: /page$ | /page | /page-2 | Blocks one address only |
| Disallow: /page | /page, /page-2, /pages | Nothing at that prefix | Blocks every URL with that prefix |
| Allow: /private/open/ | /private/open/a | Other paths under /private/ | Reopens one path inside a broader Disallow |
| Disallow: /*.pdf$ | /file.pdf | /file.pdf?x=1 | Blocks PDF files with no query string |
Using the Robots.txt Tester & Validator
What the Robots.txt Tester & Validator checks and reports
Fetch the origin’s current /robots.txt file through bounded public-network checks and show the requested file URL, response status and parser evidence. A local draft can instead be evaluated entirely in the browser without publishing or sending its text to the worker.
Select the most specific applicable user-agent group for Googlebot, Bingbot and other supported crawler profiles. The verdict preserves the chosen crawler, full same-origin target and selected group so a generic wildcard result is not substituted for a more specific match.
Compare matching Allow and Disallow patterns and expose the exact longest winning rule, match length and competing rules. Supported wildcard, end-anchor and query-string matches are evaluated against the target path so a verdict can be traced to a concrete line instead of a generic warning.
Return bounded parser diagnostics, unsupported-line advisories and every declared sitemap location within the file limits. These findings help correct the tested file without treating unknown directives as access rules; current evidence remains separate from crawler caches and implementation differences.
When to use the Robots.txt Tester & Validator
Run the Robots.txt Tester to explain crawl access for one path or validate a local rules change before publishing the file.
How to use the Robots.txt Tester & Validator
Example: test https://example.com/private/report for Googlebot against the live file or a local User-agent and Disallow draft.
Common Robots.txt Tester & Validator issues and fixes
Use the reported group, exact winning line, competing rules and match lengths to edit the applicable User-agent section or the more specific path rule. Check whether another duplicate group also applies. Retest the same crawler and full target URL, including its query string, so the correction is checked against identical input.
Robots.txt is public crawler guidance, not access control. Protect confidential or private resources with authentication and authorization even when a Disallow rule exists. Publishing a sensitive path in robots.txt can reveal it to people and noncompliant bots.
An Allow verdict means the tested rules permit crawling; it does not request indexing or confirm that a crawler will visit. If a crawlable page should stay out of search, inspect its fetched meta robots and X-Robots-Tag evidence with the Noindex Checker instead of adding an unrelated crawl block that may hide those directives.
Understanding your Robots.txt Tester & Validator results
The report shows the selected user-agent group, winning and competing rules, allow or block verdict, syntax diagnostics and declared sitemap locations.
Limitation: The Robots.txt Tester evaluates one same-origin target for a selected supported crawler against either the currently fetched file or the supplied local draft. Its Google-compatible group and path matching does not certify that every crawler implements, caches or extends the protocol identically. Live retrieval can fail because of network, redirect, response-size or origin behavior; those states remain fetch evidence. Syntax advisories identify lines this parser ignores or cannot apply, not proof that a live crawler will reject the entire file. An Allow verdict does not guarantee crawling or indexing, and a Disallow verdict does not secure, remove or deindex a resource. Use authentication for access control and page-level directives for indexing decisions.
Frequently asked questions
Does robots.txt prevent indexing?
Not reliably. Robots.txt controls crawler access to a URL; it is not a removal directive. A blocked URL can still appear in search when a crawler discovers it through links without fetching the content. Use a noindex directive only where the crawler can access and read that response.
Do all crawlers interpret robots.txt identically?
No. This tester uses Google-compatible group and path matching for its verdict, but real crawlers can support different directives or cache an older file. Google caches robots.txt for up to 48 hours, so a corrected file can still be ignored for the rest of that window. The result identifies the selected profile, matched rule and current fetched evidence so the scope of the conclusion remains visible, and the Search Console robots.txt report shows which version Google last fetched and lets you request a recrawl of the file.
How do I block a directory without blocking the whole site?
Add a Disallow rule for that directory under the intended user-agent group and keep every other path unlisted, because an unlisted path is permitted. Test one URL inside the directory and one outside it, and check that a shorter Allow rule elsewhere does not reopen part of the block.
Does Disallow remove a page that is already in Google?
No. A Disallow rule prevents a future request, and a URL already in the index can remain there while the block is in place. Removing a listed URL requires an index-side action, and blocking the fetch first can prevent a crawler from reading a noindex directive that would have done the work.
Where should a sitemap be declared?
A Sitemap line in robots.txt is the conventional declaration, and it accepts an absolute URL. The tester reports every declared Sitemap line it finds so the file’s discovery instructions stay visible alongside the crawl rules.
Monitor URLs over time
A single robots.txt edit can block an entire directory for one crawler. Monitor your crawl-critical URLs and get alerted when a rule starts blocking a path that should stay open.