Policy Rules

Overview

A policy rule has two parts. The matcher says when the rule applies. The directives say what the rule does. Rules live on the Policy page in the dashboard.

The validator checks your rules in priority order. For each thing a rule can set, the first rule that matches wins.

Policy runs partway through the checks. By the time your rules run, the validator has already identified any known crawler and looked up the visitor's country and network. It has not scored the request for bot behavior yet. So your rules can use what the validator knows about a crawler, and a rule can decide the request before scoring runs.

Evaluation order

Policy never forces a blanket allow, and it never replaces your protection level with a fixed value. A rule can set just bot_detect and leave the verdict to a later rule, or to scoring.

Crawler allowlist priority

The crawler allowlist is not a separate step that runs first. Instead, the validator adds two built-in rules for you. Both sit at the lowest priority, so they run after every rule you write:

• An identified crawler on your allowlist gets verdict: allow.
• An identified crawler not on your allowlist gets verdict: block.

Both rules only apply to identified crawlers. Neither one fires unless the request matched a known crawler. Normal browser traffic is never affected. So by default, the allowlist decides what happens to crawlers. Allowlisted crawlers pass, and every other identified crawler is blocked.

Because the built-in rules run last, any rule of yours that sets a verdict wins over them. Say you want to let a crawler through that is not on your allowlist, or block one that is. Write your own rule with a lower priority number. (Any number works, since the built-ins sit at the highest possible number.) Your rule sets the verdict, evaluation stops, and the built-in never runs.

{
  "priority": 100,
  "note": "Let our partner's crawler through even though it is not allowlisted",
  "when_matcher": {
    "combinator": "and",
    "rules": [
      { "field": "crawler.name", "op": "literal", "value": "PartnerBot" }
    ]
  },
  "set_directives": { "verdict": "allow", "bot_detect": "off" }
}

Matchers

A matcher is a group of clauses. The group has a combinator (and or or) and a list of rules. Each clause in rules has a field, an operator (op), and a value. With and, every clause must match. With or, any one clause matches.

"when_matcher": {
  "combinator": "and",
  "rules": [
    { "field": "url", "op": "glob",  "value": "/checkout/**" },
    { "field": "ua",  "op": "regex", "value": "(?i)curl|wget" }
  ]
}

The table below lists every field and the operators it accepts. You will usually build rules in the dashboard editor, but this JSON is the real shape the import and API endpoints expect, so you can copy it.

country and asn only match when the IP lookup returned a value. If the lookup has no data for an IP, the clause does not match. A rule based on country or ASN skips that request rather than guess.

Crawler clauses

Every crawler.* clause is false unless the request matched a known crawler. There is no crawler.identified field, because identification is built in: match any crawler.* field and you are already requiring an identified crawler. To act only on verified crawlers, add crawler.verified with op: is and value: true.

crawler.category is one of search, seo, ai_training, ai_assistant, ai_search, ai_agent, scraper, archive, monitoring, social_media, aggregator, accessibility, advertising, feed_reader, preview, research, security, other.

Operators

Pick the simplest operator that fits the field.

• literal: an exact text match. No wildcards, and uppercase and lowercase count as different.
• eq: a match on the whole value. method and country ignore case; query and cookie values must match exactly.
• in / not_in: checks whether the value is (or is not) in a list you give. Used by method, country, asn, and crawler.category.
• is / is_not: a true/false test, for crawler.verified and crawler.allowed.
• exists: checks that the field is there at all. referer exists when the header is not empty; query and cookie exist when the named parameter is set.
• between: for time_of_day, a local-time window (see below).
• glob, regex, and cidr: pattern operators, described next.

glob (shell-style)

Available for url, hostname, and referer. Implemented with gobwas/glob. Supports:

/api/*           matches /api/users but not /api/users/42
/api/**          matches /api/users and /api/users/42
/files/*.json    matches /files/data.json
*.example.com    matches shop.example.com and api.example.com

regex (RE2)

Available for url, ua, referer, crawler.name, and the value side of query and cookie. Patterns compile with Go's regexp package, which uses RE2 syntax.

RE2 looks like Perl or PCRE, with two differences to know about:

• No backreferences (\1, \2).
• No lookaround ((?=...), (?!...), (?<=...), (?<!...)).

In exchange, matching stays fast no matter the input. A bad pattern cannot slow the validator down the way it can with PCRE.

(?i)bot|crawler|spider     matches anywhere in the UA, any case
^/admin(/|$)               path is /admin, or starts with /admin/
^Mozilla/5\.0 \(.*Linux    literal dot, parens are escaped
[a-z]{3,8}                 three to eight lowercase letters

To ignore case, put (?i) at the start of the pattern. If a pattern does not compile, the dashboard rejects it when you save. If one somehow fails later, the rule is skipped.

cidr

Available for ip. Uses Go's net.ParseCIDR. IPv4 and IPv6 both work. For a single address, use literal instead.

192.168.0.0/16
10.0.0.0/8
2001:db8::/32

between (time of day)

time_of_day takes a start and end time in 24-hour HH:MM form, plus a time zone (tz) like America/New_York. The window includes the start time and stops just before the end time. If start is later than end, the window runs past midnight into the next day. The value is an object, so the whole clause looks like this:

{ "field": "time_of_day", "op": "between", "value": { "start": "22:00", "end": "06:00", "tz": "America/New_York" } }

Directives

A rule must set at least one directive. The first rule to set a directive wins it, and no later rule can change it.

The status fields only accept these codes: 200, 302, 401, 403, 404, 418, 429, 451, 460, or 503. The page fields name a template you have set up, not raw HTML.

Examples

Block obvious scraping clients on the checkout

{
  "priority": 100,
  "when_matcher": {
    "combinator": "and",
    "rules": [
      { "field": "url", "op": "glob",  "value": "/checkout/**" },
      { "field": "ua",  "op": "regex", "value": "(?i)curl|wget|python-requests|httpie" }
    ]
  },
  "set_directives": { "verdict": "block" }
}

Always allow verified search engines, no scoring

{
  "priority": 50,
  "when_matcher": {
    "combinator": "and",
    "rules": [
      { "field": "crawler.verified", "op": "is", "value": true },
      { "field": "crawler.category", "op": "eq", "value": "search" }
    ]
  },
  "set_directives": {
    "verdict": "allow",
    "bot_detect": "off"
  }
}

Rate-limit the public API per IP

{
  "priority": 200,
  "when_matcher": {
    "combinator": "and",
    "rules": [
      { "field": "url", "op": "glob", "value": "/api/v1/**" }
    ]
  },
  "set_directives": {
    "rate_limit": {
      "max_requests": 60,
      "window_seconds": 60,
      "scope": "ip",
      "phase": "pre"
    }
  }
}

Turn detection up for a set of countries

{
  "priority": 150,
  "when_matcher": {
    "combinator": "and",
    "rules": [
      { "field": "country", "op": "in", "value": ["CN", "RU", "BR"] }
    ]
  },
  "set_directives": { "bot_detect": "high" }
}

Shadow-test a new block rule for a week

{
  "priority": 80,
  "note": "Trial run for AI-agent block, 2026-05",
  "when_matcher": {
    "combinator": "and",
    "rules": [
      { "field": "crawler.category", "op": "eq", "value": "ai_agent" }
    ]
  },
  "set_directives": { "verdict": "block", "monitor": true }
}

Catch-all fallback at the bottom

A rule with an empty matcher ({}, an empty group) matches every request. Put one at the highest priority number to act as a final default. This example leaves the verdict to scoring and only sets the detection level.

{
  "priority": 9999,
  "when_matcher": {},
  "set_directives": { "bot_detect": "normal" }
}

Common mistakes

• Forgetting to anchor a regex. /admin matches /v2/admin/users. Use ^/admin(/|$) if you mean only /admin and its sub-paths.
• Treating . as a literal dot. In regex, . is any character. Escape it as \. inside hostnames or paths.
• Confusing * and ** in globs. * does not cross the / separator. Use ** for nested paths.
• Expecting a rule after a verdict to run. Once any rule sets verdict, evaluation stops. Put the directives you always want (like rate_limit or bot_detect) on a higher-priority rule, or on the same rule as the verdict.
• Matching a crawler without identification. Every crawler.* clause only fires for identified crawlers. A rule keyed on crawler.category never matches ordinary browser traffic.
• Trying to use lookaround or backreferences. RE2 does not support them. Split the matcher into a glob plus a second clause, or rewrite the pattern.
• Combining clauses expecting OR. Clauses use the group's combinator. With and (the usual case), every clause must match. For OR, set combinator to or, use | inside a regex, or write two rules at the same priority.
• Skipping monitor: true on first deploy. New rules can match more traffic than expected. Run them in monitor mode for a few days, watch the logs, then flip monitor off.

Related