Robots Policy Change Tester
Compare old and new robots.txt decisions for the paths and user agent you supply.
Manual analysis on this device. Your selected reports stay in this browser.
Try a worked example
Synthetic data illustrates a failure, a control and insufficient evidence. Loading an example replaces the current form inputs.
Analysis results
Preparing the local analyzer.
Download the derived report with its measurement details. It does not include the original input file.
Prepare your evidence
Paste the proposed robots.txt as the current document and the existing policy as the baseline. Under Input details, enter the crawler token and the paths that matter to your site. Review changed allow/block decisions and the winning rule for each path. This simulates the supplied policy; it does not fetch your live file or prove that a crawler visited a URL.
Read the result and check the repair
Start with the observed failures and their evidence. Follow the next action for the affected item, then collect a new report using the same scope. Use the earlier report when this tool supports comparison. Differences between sites, collection windows or test conditions can prevent a meaningful comparison.
Local processing and limits
Files and pasted evidence are analyzed in your browser. They are not uploaded, added to the page address or saved in browser storage. Each input is limited to 2 MiB, with further limits shown by the analyzer. Closing or clearing this page removes its report from the page.
Exports contain the derived findings and measurement details. Review names, paths and text before sharing. A successful check describes the supplied evidence; it does not certify a whole website or replace a staging test.
- Analysis stays in this browser. Inputs are not uploaded by this analyzer.
- Each document is limited to 2 MiB and 500 entries. Unsupported input is reported as unavailable.