Paste a web server access log and your robots.txt to see which AI training crawlers, search indexers, and live assistant fetchers visited—what they wanted, what broke, and where behavior differs from your rules.
Everything stays in your browser. Nothing is uploaded.
The consultant site loads on first visit. Presets are deterministic synthetic logs using documentation-range IPs.
Combined or Common Log Format; the last quoted string may be the user agent.
Uses longest-match precedence, * and $ wildcards, specific agent groups, and the * group as fallback.
192.0.2.42 - - [26/Oct/2026:13:55:36 -0500] "GET /pricing HTTP/1.1" 200 2326 "-" "GPTBot/1.2"User-agent substrings are matched case-insensitively, with more specific catalog entries checked first. The catalog is as of Oct 2026; user agents can be spoofed.
Google-Extended and Applebot-Extended are robots.txt controls only: they control training use, but do not appear as user agents in access logs. Unmatched strings containing bot/crawler/spider, curl, python-requests, or headless are labeled OTHER BOT; everything else is HUMAN-ish.