Policy Rules

Overview

A policy rule has two parts. The matcher says when the rule applies. The directives say what the rule does. Rules live on the Policy page in the dashboard.

The validator checks your rules in priority order. For each thing a rule can set, the first rule that matches wins.

Policy runs partway through the checks. By the time your rules run, the validator has already identified any known crawler and looked up the visitor's country and network. It has not scored the request for bot behavior yet. So your rules can use what the validator knows about a crawler, and a rule can decide the request before scoring runs.

Evaluation order

When no rule matches, policy adds nothing of its own. It does not allow the request and does not touch your protection level; it only turns on monitor logging. A rule can also set just bot_detect and leave the verdict to a later rule, or to scoring.

Rules that fail to load

Policy is fail-closed per rule. If any part of a matcher is invalid, the whole rule is disabled and matches nothing, silently. That covers an unknown field or operator, a value of the wrong shape, a pattern that doesn't compile, a bad CIDR, and a nested group that would match everything. A disabled rule still looks active in the dashboard, so check the rule you just wrote actually fires before trusting it.

Crawler allowlist priority

The crawler allowlist is not a separate step that runs first. Instead, the validator adds two built-in rules for you. Both sit at the highest possible priority number, so they run after every rule you write:

• An identified crawler on your allowlist gets verdict: allow.
• An identified crawler not on your allowlist gets verdict: block.

Both rules only apply to identified crawlers. Neither one fires unless the request matched a known crawler. Normal browser traffic is never affected. So by default, the allowlist decides what happens to crawlers. Allowlisted crawlers pass, and every other identified crawler is blocked.

Because the built-in rules run last, any rule of yours that sets a verdict wins over them. Say you want to let a crawler through that is not on your allowlist, or block one that is. Write your own rule with a lower priority number. (Any number works, since the built-ins sit at the highest possible number.) Your rule sets the verdict, evaluation stops, and the built-in never runs.

{
  "priority": 100,
  "note": "Let our partner's crawler through even though it is not allowlisted",
  "when_matcher": {
    "combinator": "and",
    "rules": [
      { "field": "crawler.name", "op": "literal", "value": "PartnerBot" }
    ]
  },
  "set_directives": { "verdict": "allow", "bot_detect": "off" }
}

Matchers

A matcher is a group of clauses. The group has a combinator (and or or) and a list of rules. Each clause in rules has a field, an operator (op), and a value. With and, every clause must match. With or, any one clause matches.

"when_matcher": {
  "combinator": "and",
  "rules": [
    { "field": "url", "op": "glob",  "value": "/checkout/**" },
    { "field": "ua",  "op": "regex", "value": "(?i)curl|wget" }
  ]
}

A rules entry can also be another group. The dashboard accepts at most 10 nesting levels. The validator also has a 64-level defensive limit. Nest groups only when you need to mix and and or in one matcher.

"when_matcher": {
  "combinator": "and",
  "rules": [
    { "field": "url", "op": "glob", "value": "/checkout/**" },
    {
      "combinator": "or",
      "rules": [
        { "field": "country", "op": "in", "value": ["RU", "CN"] },
        { "field": "asn", "op": "eq", "value": "15169" }
      ]
    }
  ]
}

Rules apply across your whole organisation. To limit one to a single site, add a hostname clause.

The table below lists every field and the operators it accepts. You will usually build rules in the dashboard editor, but this JSON is the shape the editor stores and an import file carries, so you can copy it. query and cookie are only available in the expression editor, not the visual builder.

country and asn only match when the IP lookup returned a value. If the lookup has no data for an IP, the clause does not match. A rule based on country or ASN skips that request rather than guess.

Crawler clauses

Every crawler.* clause is false unless the request matched a known crawler. There is no crawler.identified field, because identification is built in: match any crawler.* field and you are already requiring an identified crawler. To act only on verified crawlers, add crawler.verified with op: is and value: true.

crawler.category is one of search, seo, ai_training, ai_assistant, ai_search, ai_agent, scraper, archive, monitoring, social_media, aggregator, accessibility, advertising, feed_reader, preview, research, security, other.

Operators

Pick the simplest operator that fits the field.

• literal: an exact text match. No wildcards, and uppercase and lowercase count as different.
• eq: a match on the whole value. Only method and country ignore case. crawler.category, query, and cookie are case-sensitive. On ip it compares addresses rather than text, and on asn it accepts either the bare number or an AS prefix.
• in / not_in: checks whether the value is (or is not) in a list you give. Used by method, country, asn, and crawler.category.
• is / is_not: a true/false test, for crawler.verified and crawler.allowed.
• exists: checks that the field is there at all. referer exists when the header is not empty; query and cookie exist when the named parameter is set.
• between: for time_of_day, a local-time window (see below).
• glob, regex, and cidr: pattern operators, described next.

Naming a query parameter or cookie

query and cookie clauses take an object value with a name and a value, because the clause has to say which parameter it means. Include name even with exists, and pass value as an empty string there. A bare string value disables the rule.

{ "field": "query",  "op": "exists", "value": { "name": "utm_source", "value": "" } }
{ "field": "cookie", "op": "regex",  "value": { "name": "sid", "value": "^[a-f0-9]{32}$" } }

If a parameter appears more than once in the query string, the last occurrence is the one matched.

glob (shell-style)

Available for url, hostname, and referer. Supports:

/api/*           matches /api/users and /api/users/42
/files/*.json    matches /files/data.json and /files/v2/data.json
*.example.com    matches shop.example.com and api.example.com

regex (RE2)

Available for url, ua, referer, crawler.name, and the value side of query and cookie. Patterns compile with Go's regexp package, which uses RE2 syntax.

RE2 looks like Perl or PCRE, with two differences to know about:

• No backreferences (\1, \2).
• No lookaround ((?=...), (?!...), (?<=...), (?<!...)).

In exchange, matching stays fast no matter the input. A bad pattern cannot slow the validator down the way it can with PCRE.

(?i)bot|crawler|spider     matches anywhere in the UA, any case
^/admin(/|$)               path is /admin, or starts with /admin/
^Mozilla/5\.0 \(.*Linux    literal dot, parens are escaped
[a-z]{3,8}                 three to eight lowercase letters

To ignore case, put (?i) at the start of the pattern.

The dashboard catches most bad patterns when you save, but it checks them with the browser's regex engine. A lookaround or backreference saves cleanly and is then rejected by the validator, which disables the whole rule. The rule keeps looking active in the editor while matching nothing, so avoid both constructs.

cidr

Available for ip. Uses Go's net.ParseCIDR. IPv4 and IPv6 both work. For a single address, use literal instead.

192.168.0.0/16
10.0.0.0/8
2001:db8::/32

between (time of day)

time_of_day takes a start and end time in 24-hour HH:MM form, plus a time zone (tz) like America/New_York. The window includes the start time and stops just before the end time. If start is later than end, the window runs past midnight into the next day. The value is an object, so the whole clause looks like this:

{ "field": "time_of_day", "op": "between", "value": { "start": "22:00", "end": "06:00", "tz": "America/New_York" } }

Directives

A rule must set at least one directive. The first rule to set a directive wins it, and no later rule can change it.

The status fields only accept these codes: 200, 302, 401, 403, 404, 418, 429, 451, 460, or 503. The page fields name a template you have set up, not raw HTML.

What monitor really covers

monitor: true is not scoped to the rule that sets it. It puts the whole request into shadow mode. Every terminal block returns as allow and is logged. This includes blocks from this rule, later rules, rate limiting, and bot scoring. Challenges still return normally.

The first rule to set monitor wins it. A broad rule with a low priority number can put your entire organisation into shadow mode. Keep monitor on narrow rules while you test them.

monitor: false does not move traffic into enforcement. When only monitor: false and/or response resolve, policy replaces false with its monitor: true fallback. false remains only when a verdict, bot_detect, or rate_limit also resolves. It also cannot override an earlier monitor: true.

What rate_limit really does

Requests over the budget are blocked, with a rate_limit_exceeded reason and a retry_after value your backend should send as Retry-After. Rate limiting is also fail-closed on binding: if the scope key cannot be resolved, the request is blocked outright with policy_rate_limit_missing_binding.

Rate limiting runs before session validation, so scope: "session" has no session to key on for a first-time visitor. Use session_or_ip unless you know every request already carries a session.

Examples

Block obvious scraping clients on the checkout

{
  "priority": 100,
  "when_matcher": {
    "combinator": "and",
    "rules": [
      { "field": "url", "op": "glob",  "value": "/checkout/**" },
      { "field": "ua",  "op": "regex", "value": "(?i)curl|wget|python-requests|httpie" }
    ]
  },
  "set_directives": { "verdict": "block" }
}

Always allow verified search engines, no scoring

{
  "priority": 50,
  "when_matcher": {
    "combinator": "and",
    "rules": [
      { "field": "crawler.verified", "op": "is", "value": true },
      { "field": "crawler.category", "op": "eq", "value": "search" }
    ]
  },
  "set_directives": {
    "verdict": "allow",
    "bot_detect": "off"
  }
}

Rate-limit the public API per IP

{
  "priority": 200,
  "when_matcher": {
    "combinator": "and",
    "rules": [
      { "field": "url", "op": "glob", "value": "/api/v1/**" }
    ]
  },
  "set_directives": {
    "rate_limit": {
      "max_requests": 60,
      "window_seconds": 60,
      "scope": "ip",
      "phase": "pre"
    }
  }
}

Turn detection up for a set of countries

{
  "priority": 150,
  "when_matcher": {
    "combinator": "and",
    "rules": [
      { "field": "country", "op": "in", "value": ["CN", "RU", "BR"] }
    ]
  },
  "set_directives": { "bot_detect": "high" }
}

Shadow-test a new block rule for a week

{
  "priority": 80,
  "note": "Trial run for AI-agent block, 2026-05",
  "when_matcher": {
    "combinator": "and",
    "rules": [
      { "field": "crawler.category", "op": "eq", "value": "ai_agent" }
    ]
  },
  "set_directives": { "verdict": "block", "monitor": true }
}

Catch-all fallback at the bottom

The validator treats a top-level empty and group ({ "combinator": "and", "rules": [] }) as a match-everything matcher. The dashboard cannot safely create or import this matcher for a new policy rule. The editor rejects a new empty matcher, and import removes default matchers.

Do not use the dashboard Default rule as an enforcement fallback. The validator treats its { "is_default": true } matcher as invalid and skips the rule, so it enforces nothing. Use explicit matching rules for policy actions. Use your configured protection level for requests that policy does not decide, and verify that each new rule fires.

Common mistakes

• Forgetting to anchor a regex. /admin matches /v2/admin/users. Use ^/admin(/|$) if you mean only /admin and its sub-paths.
• Treating . as a literal dot. In regex, . is any character. Escape it as \. inside hostnames or paths.
• Expecting a glob to stop at a path segment. * crosses /, so /api/* also matches /api/users/42. Use an anchored regex clause when you need segment boundaries.
• Expecting a rule after a verdict to run. Once any rule sets verdict, evaluation stops. Put the directives you always want (like rate_limit or bot_detect) on a higher-priority rule, or on the same rule as the verdict.
• Matching a crawler without identification. Every crawler.* clause only fires for identified crawlers. A rule keyed on crawler.category never matches ordinary browser traffic.
• Trying to use lookaround or backreferences. RE2 does not support them. Split the matcher into a glob plus a second clause, or rewrite the pattern.
• Combining clauses expecting OR. Clauses use the group's combinator. With and (the usual case), every clause must match. For OR, set combinator to or, nest an or group inside your and group, or use | inside a regex.
• Putting monitor: true on a broad rule. It shadow-modes every block on the request, not just this rule's. Test new rules in monitor mode, but keep the matcher narrow while you do.
• Assuming an invalid rule fails loudly. One bad clause disables the whole rule, and it still looks active in the editor. Confirm a new rule fires before relying on it.

Related