Crow Tech

Free utility

Robots.txt Parser

Group bounded robots.txt agents and directives from pasted text.

All free utilities

Processing stays on this device. Your input is not uploaded, sent to an external service, or saved by Crow Tech.

Workbench

Inputs stay on this page. Use the explicit action to validate and produce a result.

Groups declared user-agent records and directive text only. It does not fetch a site, resolve precedence, test crawler behaviour, or predict indexing.