Centinel AnalyticaCrawler Catalog
Every crawler Centinel knows, what it does, and the checks that prove the request is really it.
This page lists the whole crawler catalog and the checks that apply to each entry.
Overview
An entry has a name, an operator, a type, and a row of badges. The badges say which checks Centinel runs on a request that claims to be this crawler.
A user agent proves nothing; anyone can send any string. The other three badges are checks on where the request came from. Most entries have none of them, so Centinel names the crawler and leaves the source unconfirmed.
What the badges mean
User agent: the request calls itself this crawler.Web bot auth: the crawler signs its requests, and Centinel checks the signature against the operator's key.IP ranges: the operator publishes the addresses it crawls from, and Centinel checks the client address against that list.Reverse DNS: the client address resolves to a hostname the operator controls, and that hostname resolves back to the same address.
The values behind a badge stay off the page. Match tokens, verification domains, and range feeds read as an impersonation recipe once they sit next to each other.
What verification changes
The crawler object in the /validate response sets access_allowed to true only for a verified source that your tenant allowlists. Policy rules match on the crawler category, so search engines can pass while AI training crawlers are blocked. See Policy rules.
The catalog
Search by name, operator, or description. The type and method filters narrow the list. No source check collects the entries that carry only a user-agent badge. Open a name for that crawler's own page.
RSL marks the category used for licensing rules: search, ai-train, ai-input, or ai-all. An entry with no RSL badge carries the default, all.
1905 of 1905 crawlers. Snapshot taken 2026-08-25.
007ac9 Crawler is a web crawler developed by SISTRIX GmbH, a German SEO tools company, used for SEO analysis and website indexing.
User agent008 is a scraper operated by 80legs.
User agent2Checkout Payment Platform's webhooks integration for notifying websites about payment events. Now operated by Verifone.
User agent2ip bot is a crawler for 2ip.ru, a Russian web service providing IP address lookup, website analysis, speed tests, and various network diagnostic tools.
User agent2ip.ru is a Russian web service that provides IP address information, website analysis, and monitoring tools.
User agent360Monitoring checks website uptime, performance, and server health across multiple global locations, providing alerts for downtime and slow response times.
User agent360Spider is Qihoo 360's main search engine crawler, indexing web content for China's second largest search engine (so.com) with over 500 million users.
User agentIP ranges360Spider-Image is Qihoo 360's specialized image crawler for indexing images on China's second largest search engine.
User agent360Spider-Video is Qihoo 360's specialized video crawler for indexing video content on China's second largest search engine.
User agent5emeRue is a web crawler. Its specific purpose and operator are not publicly documented.
User agent5erue is a web crawler. Its specific purpose and operator are not publicly documented.
User agent7Siters is an uncategorized agent.
User agentA Patent Crawler is an uncategorized agent.
User agentA360-Search is a web crawler used by Area360 for collecting and analyzing property information and real estate data from websites.
User agentA6-Indexer is a search indexing crawler. Limited public documentation available about its operator.
User agentaa is a web crawler used by Agency Analytics for collecting SEO data and analyzing website performance to help marketing agencies generate client reports.
User agentAASA-Bot is Apple's crawler that fetches Apple App Site Association files to support Universal Links functionality, allowing iOS apps to handle specific URL patterns.
User agentABEvalBot is an uncategorized agent.
User agentAboundex was a search engine crawler. The service is no longer operational.
User agentabsearch-crawler is an uncategorized agent.
User agentAcademicBotRTU is an academic research crawler operated by Riga Technical University for scholarly research purposes.
User agentacapbot is a web crawler. Its specific purpose and operator are not publicly documented.
User agentAccessible Web Bot crawls customer websites to discover pages and monitor for accessibility violations on regular basis. Crawls are initiated for Accessible Web's "Page Monitoring" SaaS product.
User agentAccessStatus is a monitoring bot that checks HTTP status codes of web pages to determine if URLs are active, redirected, or returning errors for website uptime monitoring.
User agentacoonbot was a German search engine crawler. The Acoon search engine is no longer operational.
User agentAcquia Optimize (Monsido) scans websites for accessibility issues (WCAG 2.2, ADA compliance), broken links, SEO problems, and content quality issues.
User agentActiveComply is a compliance monitoring crawler that systematically scans social media platforms and websites to identify regulatory compliance violations for businesses in regulated industries like finance.
User agentAcunetix Security Scanner is an automated web application security testing tool that scans for exploitable vulnerabilities including SQL injection and XSS.
User agentAcunetix Web Vulnerability Scanner audits web applications for security vulnerabilities with 99.98% accuracy, detecting over 12,000 vulnerability types.
User agentAdagiobot is a web crawler that analyzes websites for advertising demand optimization, helping publishers maximize revenue through real-time bidding analysis and performance insights.
User agentAds.txt crawler that identifies itself as AdaptMX. It reads publisher authorised seller files. We could not confirm the operator from public documentation.
User agentadbeat_bot collects advertising intelligence data for Adbeat, a competitive intelligence platform that tracks display ad campaigns, creatives, and media buying strategies.
User agentadbeat-check-access-script is operated by Adbeat (a Pathmatics/Sensor Tower property), an ad-intelligence platform that crawls landing pages and creatives to power competitive advertising analytics.
User agentCrawler operated by ADBRO, an advertising platform. It reads publisher pages and authorised seller files to support ad placement and page analysis.
User agentAdDefend Site Context Crawler is an ad-monitoring crawler operated by AdDefend GmbH that fetches publisher pages to verify ad delivery and brand-safety context.
User agentAddSearchBot is a web crawler that indexes website content for AddSearch's AI-powered site search solution, collecting data to provide fast and accurate search results.
User agentAddThis.com is a bot operated by Oracle that fetches web content for their social sharing and website tools service. This bot visits websites to gather preview information such as titles, descriptions, and images to generate share buttons and content recommendations.
User agentadequat is an advertising technology crawler that analyzes web content for ad targeting, verification, or competitive intelligence.
User agentadequat-systems is an advertising technology crawler that analyzes web content for ad targeting, verification, or competitive intelligence.
User agentadidxbot is Microsoft's Bing Ads crawler that monitors ads and follows ad destination websites for quality control to ensure advertising standards and policy compliance.
User agentAdkernelTopicCrawler is a web crawler operated by Adkernel, a white-label ad tech solutions provider. This bot gathers information to support the company's ad network, DSP, SSP, and trading desk platforms.
User agentThe Admantx web robot indexes and analyzes website content.
User agentAdmixerBot is a web crawler operated by Admixer, a digital advertising technology company. It visits web pages to analyze and categorize their content, enabling contextual ad targeting for publishers and advertisers using Admixer's platform.
User agentAdvertising crawler that identifies itself as AdnBot. The operator is not documented publicly.
User agentads-well-known-bot is an uncategorized agent.
User agentAdsBot-Google checks landing pages for Google Ads quality, evaluating load speed, usability, and ad policy compliance to determine Quality Scores and cost-per-click rates.
User agentIP rangesAdsBot-Google-Mobile checks mobile landing pages for Google Ads quality, evaluating load speed, mobile usability, and ad policy compliance to determine Quality Scores.
User agentIP rangesadscanner is an advertising verification crawler that scans websites for ad placement and compliance.
User agentAds.txt crawler operated by AdSpirit GmbH, a German ad server company. It reads the /ads.txt file of publisher domains to confirm authorised seller entries.
User agentAds.txt parser operated by Demandbase, an account based marketing and advertising company. It reads publisher authorised seller files to validate programmatic inventory.
User agent