Robots.txt Tester & Validator
The Robots.txt Tester & Validator determines whether a live robots.txt file or local draft allows or disallows a same-origin URL for a selected crawler profile.
What the Robots.txt Tester & Validator checks
Fetch the origin’s current /robots.txt file through bounded public-network checks and show the requested file URL, response status and parser evidence. A local draft can instead be evaluated entirely in the browser without publishing or sending its text to the worker.
Select the most specific applicable user-agent group for Googlebot, Bingbot and other supported crawler profiles. The verdict preserves the chosen crawler, full same-origin target and selected group so a generic wildcard result is not substituted for a more specific match.
Compare matching Allow and Disallow patterns and expose the exact longest winning rule, match length and competing rules. Supported wildcard, end-anchor and query-string matches are evaluated against the target path so a verdict can be traced to a concrete line instead of a generic warning.
Return bounded parser diagnostics, unsupported-line advisories and every declared sitemap location within the file limits. These findings help correct the tested file without treating unknown directives as access rules; current evidence remains separate from crawler caches and implementation differences.
When to use this tool
Run the Robots.txt Tester to explain crawl access for one path or validate a local rules change before publishing the file.
Example
Example: test https://example.com/private/report for Googlebot against the live file or a local User-agent and Disallow draft.
Issues and fixes
Use the reported group, exact winning line, competing rules and match lengths to edit the applicable User-agent section or the more specific path rule. Check whether another duplicate group also applies. Retest the same crawler and full target URL, including its query string, so the correction is checked against identical input.
Robots.txt is public crawler guidance, not access control. Protect confidential or private resources with authentication and authorization even when a Disallow rule exists. Publishing a sensitive path in robots.txt can reveal it to people and noncompliant bots.
An Allow verdict means the tested rules permit crawling; it does not request indexing or confirm that a crawler will visit. If a crawlable page should stay out of search, inspect its fetched meta robots and X-Robots-Tag evidence with the Noindex Checker instead of adding an unrelated crawl block that may hide those directives.
What the result includes
The report shows the selected user-agent group, winning and competing rules, allow or block verdict, syntax diagnostics and declared sitemap locations.
Limitation: The Robots.txt Tester evaluates one same-origin target for a selected supported crawler against either the currently fetched file or the supplied local draft. Its Google-compatible group and path matching does not certify that every crawler implements, caches or extends the protocol identically. Live retrieval can fail because of network, redirect, response-size or origin behavior; those states remain fetch evidence. Syntax advisories identify lines this parser ignores or cannot apply, not proof that a live crawler will reject the entire file. An Allow verdict does not guarantee crawling or indexing, and a Disallow verdict does not secure, remove or deindex a resource. Use authentication for access control and page-level directives for indexing decisions.
Frequently asked questions
Does robots.txt prevent indexing?
Not reliably. Robots.txt controls crawler access to a URL; it is not a removal directive. A blocked URL can still appear in search when a crawler discovers it through links without fetching the content. Use a noindex directive only where the crawler can access and read that response.
Do all crawlers interpret robots.txt identically?
No. This tester uses Google-compatible group and path matching for its verdict, but real crawlers can support different directives or cache an older file. The result identifies the selected profile, matched rule and current fetched evidence so the scope of the conclusion remains visible.
Monitor URLs over time
Turn a one-off diagnostic into ongoing index monitoring and change alerts.