# Policy Rules > How to write rules that allow, block, rate-limit, or watch requests, and where the crawler allowlist fits in. Source: https://docs.centinelanalytica.com/admin/policy ## Overview A policy rule has two parts. The **matcher** says when the rule applies. The **directives** say what the rule does. Rules live on the [Policy page](https://dash.centinelanalytica.com/policy) in the dashboard. The validator checks your rules in priority order. For each thing a rule can set, the first rule that matches wins. Policy runs partway through the checks. By the time your rules run, the validator has already identified any known crawler and looked up the visitor's country and network. It has not scored the request for bot behavior yet. So your rules can use what the validator knows about a crawler, and a rule can decide the request before scoring runs. ## Evaluation order 1. Rules are sorted by `priority`, lowest number first. Priorities are unique per organisation, so two rules cannot share a number. 2. For each request, the validator goes down the list. A rule can set five things: `verdict`, `bot_detect`, `rate_limit`, `monitor`, and `response`. The first rule to set one of these wins, and no later rule can change it. 3. As soon as a rule sets a `verdict`, the validator stops. No later rule runs, not even the built-in crawler rules below. Anything earlier rules already set stays in place. 4. If no rule sets a verdict, policy does not decide the request. It passes the request on to bot detection, which uses the protection level from your settings. If no rule set a `verdict`, `bot_detect`, or `rate_limit`, the validator turns on monitor logging for that request. When no rule matches, policy adds nothing of its own. It does not allow the request and does not touch your protection level; it only turns on monitor logging. A rule can also set just `bot_detect` and leave the verdict to a later rule, or to scoring. > **Order matters, absolute numbers don't:** Reordering rules in the dashboard renumbers them from 1 upwards, and importing a rule set does the same. ### Rules that fail to load Policy is fail-closed per rule. If any part of a matcher is invalid, the **whole rule** is disabled and matches nothing, silently. That covers an unknown field or operator, a value of the wrong shape, a pattern that doesn't compile, a bad CIDR, and a nested group that would match everything. A disabled rule still looks active in the dashboard, so check the rule you just wrote actually fires before trusting it. ### Crawler allowlist priority The crawler allowlist is not a separate step that runs first. Instead, the validator adds two built-in rules for you. Both sit at the highest possible priority number, so they run **after** every rule you write: * An identified crawler **on** your allowlist gets `verdict: allow`. * An identified crawler **not** on your allowlist gets `verdict: block`. Both rules only apply to identified crawlers. Neither one fires unless the request matched a known crawler. Normal browser traffic is never affected. So by default, the allowlist decides what happens to crawlers. Allowlisted crawlers pass, and every other identified crawler is blocked. Because the built-in rules run last, any rule of yours that sets a verdict wins over them. To let through a crawler that is not on your allowlist, or block one that is, write your own rule with a lower priority number. Any number works, since the built-ins sit at the highest possible one. Your rule sets the verdict, evaluation stops, and the built-in never runs. ```json { "priority": 100, "note": "Let our partner's crawler through even though it is not allowlisted", "when_matcher": { "combinator": "and", "rules": [ { "field": "crawler.name", "op": "literal", "value": "PartnerBot" } ] }, "set_directives": { "verdict": "allow", "bot_detect": "off" } } ``` ## Matchers A matcher is a group of clauses. The group has a `combinator` (`and` or `or`) and a list of `rules`. Each clause in `rules` has a `field`, an operator (`op`), and a `value`. With `and`, every clause must match. With `or`, any one clause matches. ```json "when_matcher": { "combinator": "and", "rules": [ { "field": "url", "op": "glob", "value": "/checkout/**" }, { "field": "ua", "op": "regex", "value": "(?i)curl|wget" } ] } ``` A `rules` entry can also be another group. The dashboard accepts at most 10 nesting levels. The validator also has a 64-level defensive limit. Nest groups only when you need to mix `and` and `or` in one matcher. ```json "when_matcher": { "combinator": "and", "rules": [ { "field": "url", "op": "glob", "value": "/checkout/**" }, { "combinator": "or", "rules": [ { "field": "country", "op": "in", "value": ["RU", "CN"] }, { "field": "asn", "op": "eq", "value": "15169" } ] } ] } ``` Rules apply across your whole organisation. To limit one to a single site, add a `hostname` clause. The table below lists every field and the operators it accepts. You will usually build rules in the dashboard editor, but this JSON is the shape the editor stores and an import file carries, so you can copy it. You can use every field in the expression editor and in the visual builder. | Field | Operators | Matches | | ------------------ | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------ | | `url` | `literal`, `glob`, `regex` | The request path, and also the full absolute URL. A clause matches if either form matches. | | `ua` | `literal`, `regex` | The `User-Agent` header. | | `ip` | `cidr`, `literal`, `eq` | The client IP. IPv4 and IPv6. | | `hostname` | `literal`, `glob` | The host part of the request URL, without the port. | | `method` | `eq`, `in`, `not_in` | The HTTP method. Case-insensitive. | | `referer` | `literal`, `glob`, `regex`, `exists` | The `Referer` header. | | `query` | `exists`, `eq`, `regex` | A named query-string parameter's value. Takes an object value. | | `cookie` | `exists`, `eq`, `regex` | A named cookie's value. Takes an object value. | | `header` | `exists`, `eq`, `regex` | A named request header's value. Takes an object value. | | `time_of_day` | `between` | The request time inside a local-time window. | | `country` | `eq`, `in`, `not_in` | The two-letter ISO 3166-1 country of the client IP, from geo lookup. | | `asn` | `eq`, `in`, `not_in` | The autonomous-system number of the client IP. Write it as `15169` or `AS15169`. | | `crawler.verified` | `is`, `is_not` | Whether an identified crawler passed ownership verification. | | `crawler.allowed` | `is`, `is_not` | Whether an identified crawler is on your allowlist. | | `crawler.name` | `literal`, `regex` | The crawler name, such as `Googlebot` or `GPTBot`. | | `crawler.category` | `eq`, `in`, `not_in` | The crawler category (see below). | | `passport` | `exists`, `eq` | The [passport](https://docs.centinelanalytica.com/admin/passports.md) the request presented, if any. `eq` takes a passport id. | `country` and `asn` only match when the IP lookup returned a value. If the lookup has no data for an IP, the clause does not match. A rule based on country or ASN skips that request rather than guess. ### Crawler clauses Every `crawler.*` clause is false unless the request matched a known crawler. There is no `crawler.identified` field, because identification is built in: match any `crawler.*` field and you are already requiring an identified crawler. To act only on verified crawlers, add `crawler.verified` with `op: is` and `value: true`. `crawler.category` is one of `search`, `seo`, `ai_training`, `ai_assistant`, `ai_search`, `ai_agent`, `scraper`, `archive`, `monitoring`, `social_media`, `aggregator`, `accessibility`, `advertising`, `feed_reader`, `preview`, `research`, `security`, `other`. ### Operators Pick the simplest operator that fits the field. * **`literal`**: an exact text match. No wildcards, and uppercase and lowercase count as different. * **`eq`**: a match on the whole value. Only `method` and `country` ignore case. `crawler.category`, `query`, `cookie`, and the `header` value are case-sensitive. On `ip` it compares addresses rather than text, and on `asn` it accepts either the bare number or an `AS` prefix. * **`in` / `not_in`**: checks whether the value is (or is not) in a list you give. Used by `method`, `country`, `asn`, and `crawler.category`. * **`is` / `is_not`**: a true/false test, for `crawler.verified` and `crawler.allowed`. * **`exists`**: checks that the field is there at all. `referer` exists when the header is not empty; `query`, `cookie`, and `header` exist when the named parameter or header is set. * **`between`**: for `time_of_day`, a local-time window (see below). * **`glob`**, **`regex`**, and **`cidr`**: pattern operators, described next. #### Naming a query parameter, cookie, or header `query`, `cookie`, and `header` clauses take an object value with a `name` and a `value`, because the clause has to say which parameter it means. Include `name` even with `exists`, and pass `value` as an empty string there. A bare string value disables the rule. ```json { "field": "query", "op": "exists", "value": { "name": "utm_source", "value": "" } } { "field": "cookie", "op": "regex", "value": { "name": "sid", "value": "^[a-f0-9]{32}$" } } { "field": "header", "op": "eq", "value": { "name": "X-Client", "value": "mobile-app" } } ``` If a query parameter or a cookie appears more than once, the clause matches when one of the values matches. A cookie value also matches without its double quotes and after percent-decoding, and a cookie can follow a comma. A client cannot hide a cookie from a rule with a different spelling. A `header` name ignores case, so `x-client` and `X-Client` name the same header. The value is case-sensitive. #### `glob` (shell-style) Available for `url`, `hostname`, and `referer`. Supports: | Token | Meaning | | --------- | -------------------------------------------------------------------------- | | `*` | Any run of characters, including `/`. | | `**` | The same as `*` here. | | `?` | Any single character, including `/`. | | `[abc]` | Any one of the listed characters. `[a-z]` for a range, `[!abc]` to negate. | | `{a,b,c}` | Alternation. | > **Globs are not path-aware:** `*` does not stop at a `/`, so `*` and `**` behave identically. If you need to match a single path segment, end the pattern with a literal, or use a `regex` clause with an anchor. ```text /api/* matches /api/users and /api/users/42 /files/*.json matches /files/data.json and /files/v2/data.json *.example.com matches shop.example.com and api.example.com ``` #### `regex` (RE2) Available for `url`, `ua`, `referer`, `crawler.name`, and the value side of `query`, `cookie`, and `header`. Patterns compile with Go's [`regexp`](https://pkg.go.dev/regexp) package, which uses [RE2 syntax](https://github.com/google/re2/wiki/Syntax). RE2 looks like Perl or PCRE, with two differences to know about: * No backreferences (`\1`, `\2`). * No lookaround (`(?=...)`, `(?!...)`, `(?<=...)`, `(? **Regex matches are substring matches:** The matcher looks for the pattern anywhere in the value. To require a full-string match, anchor with `^` and `$`. Without anchors, `/admin` as a regex matches `/v2/admin/users` too. ```text (?i)bot|crawler|spider matches anywhere in the UA, any case ^/admin(/|$) path is /admin, or starts with /admin/ ^Mozilla/5\.0 \(.*Linux literal dot, parens are escaped [a-z]{3,8} three to eight lowercase letters ``` To ignore case, put `(?i)` at the start of the pattern. The dashboard catches most bad patterns when you save, but it checks them with the browser's regex engine. A lookaround or backreference saves cleanly and is then rejected by the validator, which disables the whole rule. The rule keeps looking active in the editor while matching nothing, so avoid both constructs. #### `cidr` Available for `ip`. Uses Go's `net.ParseCIDR`. IPv4 and IPv6 both work. For a single address, use `literal` instead. ```text 192.168.0.0/16 10.0.0.0/8 2001:db8::/32 ``` #### `between` (time of day) `time_of_day` takes a `start` and `end` time in 24-hour `HH:MM` form, plus a time zone (`tz`) like `America/New_York`. The window includes the start time and stops just before the end time. If `start` is later than `end`, the window runs past midnight into the next day. The value is an object, so the whole clause looks like this: ```json { "field": "time_of_day", "op": "between", "value": { "start": "22:00", "end": "06:00", "tz": "America/New_York" } } ``` ## Directives A rule must set at least one directive. The first rule to set a directive wins it, and no later rule can change it. | Directive | Values | Effect | | ------------ | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `verdict` | `allow`, `block` | Final decision. `allow` lets the request through. `block` returns a 403 (or your configured block response). Setting it stops evaluation. | | `bot_detect` | `off`, `low`, `normal`, `high` | How hard the validator scores for bots. `off` skips scoring. `high` is the strictest. | | `rate_limit` | object | Per-scope budget. Fields: `max_requests` (1 to 1,000,000), `window_seconds` (1 to 86,400), `scope` (`session`, `ip`, or `session_or_ip`), `phase` (currently only `pre`). | | `monitor` | `true`, `false` | Puts the request in shadow mode. Blocks are logged and reported as allow. | | `response` | object | Change the page and status returned on a block or challenge. Fields: `block_page`, `block_status`, `challenge_page`, `challenge_status`. | The status fields only accept these codes: `200`, `302`, `401`, `403`, `404`, `418`, `429`, `451`, `460`, or `503`. The page fields name a template you have set up, not raw HTML. ### What `monitor` really covers `monitor: true` is not scoped to the rule that sets it. It puts the **whole request** into shadow mode. Every terminal block returns as `allow` and is logged. This includes blocks from this rule, later rules, rate limiting, and bot scoring. Challenges still return normally. The first rule to set `monitor` wins it. A broad rule with a low priority number can put your entire organisation into shadow mode. Keep `monitor` on narrow rules while you test them. `monitor: false` does not move traffic into enforcement. When only `monitor: false` and/or `response` resolve, policy replaces `false` with its `monitor: true` fallback. `false` remains only when a `verdict`, `bot_detect`, or `rate_limit` also resolves. It also cannot override an earlier `monitor: true`. ### What `rate_limit` really does Requests over the budget are blocked, with a `rate_limit_exceeded` reason and a `retry_after` value your backend should send as `Retry-After`. Rate limiting is also fail-closed on binding: if the scope key cannot be resolved, the request is blocked outright with `policy_rate_limit_missing_binding`. Rate limiting runs before session validation, so `scope: "session"` has no session to key on for a first-time visitor. Use `session_or_ip` unless you know every request already carries a session. ## Examples ### Block obvious scraping clients on the checkout ```json { "priority": 100, "when_matcher": { "combinator": "and", "rules": [ { "field": "url", "op": "glob", "value": "/checkout/**" }, { "field": "ua", "op": "regex", "value": "(?i)curl|wget|python-requests|httpie" } ] }, "set_directives": { "verdict": "block" } } ``` ### Always allow verified search engines, no scoring ```json { "priority": 50, "when_matcher": { "combinator": "and", "rules": [ { "field": "crawler.verified", "op": "is", "value": true }, { "field": "crawler.category", "op": "eq", "value": "search" } ] }, "set_directives": { "verdict": "allow", "bot_detect": "off" } } ``` ### Rate-limit the public API per IP ```json { "priority": 200, "when_matcher": { "combinator": "and", "rules": [ { "field": "url", "op": "glob", "value": "/api/v1/**" } ] }, "set_directives": { "rate_limit": { "max_requests": 60, "window_seconds": 60, "scope": "ip", "phase": "pre" } } } ``` ### Turn detection up for a set of countries ```json { "priority": 150, "when_matcher": { "combinator": "and", "rules": [ { "field": "country", "op": "in", "value": ["CN", "RU", "BR"] } ] }, "set_directives": { "bot_detect": "high" } } ``` ### Shadow-test a new block rule for a week ```json { "priority": 80, "note": "Trial run for AI-agent block, 2026-05", "when_matcher": { "combinator": "and", "rules": [ { "field": "crawler.category", "op": "eq", "value": "ai_agent" } ] }, "set_directives": { "verdict": "block", "monitor": true } } ``` ### Catch-all fallback at the bottom The validator treats a top-level empty `and` group (`{ "combinator": "and", "rules": [] }`) as a match-everything matcher. The dashboard cannot safely create or import this matcher for a new policy rule. The editor rejects a new empty matcher, and import removes default matchers. Do not use the dashboard **Default** rule as an enforcement fallback. The validator treats its `{ "is_default": true }` matcher as invalid and skips the rule, so it enforces nothing. Use explicit matching rules for policy actions. Use your configured protection level for requests that policy does not decide, and verify that each new rule fires. ### Common mistakes * **Forgetting to anchor a regex.** `/admin` matches `/v2/admin/users`. Use `^/admin(/|$)` if you mean only `/admin` and its sub-paths. * **Treating `.` as a literal dot.** In regex, `.` is any character. Escape it as `\.` inside hostnames or paths. * **Expecting a glob to stop at a path segment.** `*` crosses `/`, so `/api/*` also matches `/api/users/42`. Use an anchored `regex` clause when you need segment boundaries. * **Expecting a rule after a verdict to run.** Once any rule sets `verdict`, evaluation stops. Put the directives you always want (like `rate_limit` or `bot_detect`) on a higher-priority rule, or on the same rule as the verdict. * **Matching a crawler without identification.** Every `crawler.*` clause only fires for identified crawlers. A rule keyed on `crawler.category` never matches ordinary browser traffic. * **Trying to use lookaround or backreferences.** RE2 does not support them. Split the matcher into a glob plus a second clause, or rewrite the pattern. * **Combining clauses expecting OR.** Clauses use the group's `combinator`. With `and` (the usual case), every clause must match. For OR, set `combinator` to `or`, nest an `or` group inside your `and` group, or use `|` inside a regex. * **Putting `monitor: true` on a broad rule.** It shadow-modes every block on the request, not just this rule's. Test new rules in monitor mode, but keep the matcher narrow while you do. * **Assuming an invalid rule fails loudly.** One bad clause disables the whole rule, and it still looks active in the editor. Confirm a new rule fires before relying on it. ### Related * [Validation](https://docs.centinelanalytica.com/install/validation.md): How the validator returns decisions and how to enforce them in your backend. * [Dashboard](https://docs.centinelanalytica.com/install/dashboard.md): Where to find your API keys and configure protected endpoints.