# Crawlers

> Crawler object schema returned by the /validate endpoint.

Source: https://docs.centinelanalytica.com/api/crawlers

## Overview

When `/validate` detects a known crawler and crawler metadata is enabled for your tenant, the response includes a `crawler` object. It identifies the crawler and reports whether the request meets source-IP verification and tenant allowlist requirements.

## Crawler object

| Field            | Type    | Required | Description                                                                                                      |
| ---------------- | ------- | -------- | ---------------------------------------------------------------------------------------------------------------- |
| `id`             | string  | yes      | Crawler identifier (UUID).                                                                                       |
| `name`           | string  | yes      | Human-readable name (e.g., `Googlebot`).                                                                         |
| `access_allowed` | boolean | yes      | `true` only when the crawler source IP is verified and the crawler is on your allowlist.                         |
| `category`       | string  | no       | The catalog crawler `type`, copied unchanged. This is not the policy `crawler.category` value.                   |
| `rsl_category`   | string  | no       | The catalog RSL category, copied unchanged. It is independent of `category` and policy matching does not use it. |

When the response contains a `crawler` object, `id`, `name`, and `access_allowed` are present.
Either optional field can be absent when the crawler catalog has no value. `access_allowed`
identifies the crawler's standing; enforce the `/validate` result from the top-level `decision`
field.

## Category metadata

The API does not publish or enforce a fixed mapping between `category` and `rsl_category`. A
crawler can return `category: "LLM"` and `rsl_category: "ai-all"`. Both fields use open string
values, copied from the catalog without normalization.

The policy matcher field `crawler.category` is different from the API response field. Policy
normalizes the catalog crawler `type` into a closed set of values. It does not read
`rsl_category`, and it does not compare the API `category` value unchanged. See
[Policy rules](https://docs.centinelanalytica.com/admin/policy.md) for the allowed policy values and operators.

When a catalog `type` has no policy normalization, a `crawler.category not_in` rule can match the
request. Do not assume that an unknown API category is `other`.