Input
Generate crawl rules, then test a path against the chosen user-agent group. Blocking a crawl does not remove an indexed URL.
Result
Text and files are processed in your browser. URL audits fetch the public page on the server.
Free tools
robots.txt generator and tester
Generate crawl rules, then test a path against the chosen user-agent group. Blocking a crawl does not remove an indexed URL.
Free. No account needed.Copy / Download