BotWitness is a crawler operated by BotWitness LLC that records what websites tell AI crawlers through their public permission files. It requests only /robots.txt, /llms.txt and /.well-known/ai.txt at a domain root, plus any license documents referenced from robots.txt, and the operator states that it never fetches content pages, never follows links and never downloads media. The operator says it makes at most one request per day per domain, covers over one million domains, and sells RFC-3161 timestamped capture certificates, change monitoring and compliance audits built on those snapshots. BotWitness states on its own bot page that it does not yet publish an egress IP list and that its requests are not yet cryptographically signed, so no IP ranges or reverse DNS hostnames exist for verification, and the crawler does not appear in public crawler directories.

At a glance

Operator: BotWitness LLC
Type: Archiver

How Centinel checks it

User agent: The request calls itself this crawler. Anyone can send the same string.

BotWitness publishes nothing Centinel can check a source against, so a match reports the name and leaves the source unconfirmed. The match tokens, verification domains, and address feeds are not published here.

Allowing or blocking it

The crawler object in the /validate response sets access_allowed to true only for a verified source that your tenant allowlists. A policy rule can allow or block this crawler by its category, Archiver.

Back to the crawler catalog