← Back to Daily Builds

Daily build note · October 26, 2026

Agent Visitors

Paste a web server access log and your robots.txt to see which AI agents read your site (training crawlers, AI search indexers, and assistants fetching pages for a person), what pages they wanted, what broke for them, and where your robots.txt and the bots disagree.

AI Tools Interactive demo live Static HTML/CSS/JS

What shipped

Agent Visitors is a small "Google Analytics for agents" that runs entirely in your browser. Paste an nginx or Apache access log (or drop a .log file) and, optionally, your robots.txt. It matches user agents against a catalog of about 25 known bots from OpenAI, Anthropic, Perplexity, Google, Apple, Microsoft, Meta, Amazon, ByteDance, Common Crawl and others, and sorts them into TRAINING, AI SEARCH, ASSISTANT FETCH (a person asked an assistant to read the page), CLASSIC SEARCH and other bots. You get AI share of traffic, a per-agent table with pages, error rate, first and last seen and robots status, a requests-over-time chart split by kind, the top pages AI wanted, and a Pages AI is missing list of 404s. The robots.txt checker flags IGNORED RULE, NO RULE, TRAINING OPEN, SEARCH BLOCKED and TOKEN-ONLY gaps, and a toggle-driven generator writes a robots.txt snippet you can copy. There's a markdown report stamped in Central time, and state saves under agent-visitors-v1.

How to try

  • The demo opens on Consultant site: 620 requests, 73 from four AI agents (11.8% of traffic), with a robots.txt that only says Allow: /, so it flags NO RULE and TRAINING OPEN.
  • Load Blocked but crawled to see GPTBot and Bytespider fetching paths the robots.txt disallows (IGNORED RULE), and Invisible to AI search to see a site that blocks every AI agent, including the ones that would cite it in AI answers (SEARCH BLOCKED).
  • Docs site, AI hungry is 55% AI traffic, with assistants hitting old /docs/v1/ pages that now 404. Local bakery is mostly people and Googlebot, with a single ChatGPT-User fetch.
  • Paste your own log. Nothing is uploaded, and a 50,000-line log parses in a fraction of a second.

Why this (vs neighbors)

This comes from the Startup Opportunities list ("Google Analytics for agents", via @gregisenberg). Site owners can't see agent traffic in normal analytics because bots don't run JavaScript, but it shows up in server logs. Earlier agent builds like Agent Session Replay and Agent Ledger look at what your own agents do. This one looks the other way: which outside agents are reading your site and whether your robots.txt says what you mean.

Scope trim

No live log streaming, no server install, no IP verification, no CDN log formats beyond Combined/Common, and no history across uploads.

Limitations

  • User agents can be faked. Verifying a bot for certain means checking its published IP ranges or reverse DNS.
  • The bot catalog is a snapshot as of October 2026, and bot names change often.
  • Google-Extended and Applebot-Extended are robots.txt tokens only. They never show up in logs, so the tool can only check whether you set them.

Suggested next steps

  1. Add IP-range verification for the bots that publish their ranges.
  2. Support Cloudflare, CloudFront and Vercel log exports.
  3. Turn it into a hosted service with a tiny log forwarder and a weekly "what agents read" email.