ShapBot is the crawler operated by Parallel Web Systems. It collects publicly available web content to build and maintain the search index behind Parallel's search, extraction and deep research APIs, which supply web context to AI agents. It honours the User-agent: ShapBot directive in robots.txt and its source addresses are published at https://docs.parallel.ai/resources/shapbot.json.

At a glance

Operator: Parallel Web Systems
Type: AI Search
RSL category: ai-input

How Centinel checks it

User agent: The request calls itself this crawler. Anyone can send the same string.
IP ranges: The operator publishes the addresses it crawls from, and Centinel checks the client address against that list.

A request that passes one of these checks is ShapBot itself. One that only matches the user agent is reported as unverified. The match tokens, verification domains, and address feeds are not published here.

Allowing or blocking it

The crawler object in the /validate response sets access_allowed to true only for a verified source that your tenant allowlists. A policy rule can allow or block this crawler by its category, AI Search.

Back to the crawler catalog