Robots.txt inspector

When to use this tool

Inspect pasted robots.txt directives and broad blocking rules. Use it as one focused step in your workflow, then check the result against your business details and the requirements of the platform where you plan to use it.

A practical example

For a website you administer, use its canonical public URL or the exact source requested by the form. Record the current result before changing the site so you can compare it later.

What this tool cannot tell you

This reports directives rather than simulating a specific crawler. Check individual bot groups before changing rules.