# BOT Intelligence Network > Public first-party access observations for crawlers, AI agents and developers. ## BOT directory - [BOT directory](https://ninizemi.com/bot-intelligence/bots/): sourced profiles and unverified User-Agent observations. - [Observation updates](https://ninizemi.com/bot-intelligence/updates/): first claimed observation per bot per UTC day. - [BOT profiles JSON](https://ninizemi.com/bot-intelligence/v1/bots.json): machine-readable profiles and counts. - [Observation updates JSON](https://ninizemi.com/bot-intelligence/v1/updates.json): paginated first claimed observation per bot per UTC day. - [Updates RSS](https://ninizemi.com/bot-intelligence/updates.xml): recent observation events and profile publications. - [Daily BOT counts](https://ninizemi.com/bot-intelligence/v1/bot-daily.json): per-bot UTC daily counts. - [Weekly BOT counts](https://ninizemi.com/bot-intelligence/v1/bot-weekly.json): per-bot UTC Monday-start weekly counts. - [GPTBot](https://ninizemi.com/bot-intelligence/bots/gptbot): Crawls web content that may be used to train generative AI foundation models. - [OAI-SearchBot](https://ninizemi.com/bot-intelligence/bots/oai-searchbot): Crawls websites to support search results in ChatGPT. - [ChatGPT-User](https://ninizemi.com/bot-intelligence/bots/chatgpt-user): Fetches web pages for user-initiated ChatGPT actions; not an automatic crawler. - [OAI-AdsBot](https://ninizemi.com/bot-intelligence/bots/oai-adsbot): Checks safety and related properties of pages submitted as ChatGPT ads. - [ClaudeBot](https://ninizemi.com/bot-intelligence/bots/claudebot): Collects web content that could contribute to generative AI model training. - [Claude-SearchBot](https://ninizemi.com/bot-intelligence/bots/claude-searchbot): Crawls web content to improve search result quality. - [Claude-User](https://ninizemi.com/bot-intelligence/bots/claude-user): Fetches web pages in response to user-directed Claude requests. - [Googlebot](https://ninizemi.com/bot-intelligence/bots/googlebot): Crawls web pages to build and update Google Search indexes. - [Googlebot Image](https://ninizemi.com/bot-intelligence/bots/googlebot-image): Crawls images for Google Images and related products. - [GoogleOther](https://ninizemi.com/bot-intelligence/bots/googleother): A generic crawler used by Google product teams. ## Dataset - [Dataset and current statistics](https://ninizemi.com/bot-intelligence/datasets): scope, observed request counts and free JSON download. - [Latest JSON](https://ninizemi.com/bot-intelligence/datasets/latest.json): hourly aggregate snapshot. ## API - [/v1/observe](https://ninizemi.com/bot-intelligence/v1/observe): Your coarse request context and recording receipt - [/v1/stats](https://ninizemi.com/bot-intelligence/v1/stats): Traffic distributions and all-time daily counts - [/v1/patterns](https://ninizemi.com/bot-intelligence/v1/patterns): Observed country/ASN/client-family combinations; groups of at least 3 - [/v1/changes](https://ninizemi.com/bot-intelligence/v1/changes): Difference between the two most recent snapshots - [/v1/bots.json](https://ninizemi.com/bot-intelligence/v1/bots.json): Known crawler profiles and unverified individual User-Agent observations - [/v1/updates.json](https://ninizemi.com/bot-intelligence/v1/updates.json): New daily User-Agent observations and catalog additions; paginated - [/v1/bot-daily.json](https://ninizemi.com/bot-intelligence/v1/bot-daily.json): Daily UTC claimed bot counts by profile - [/v1/bot-weekly.json](https://ninizemi.com/bot-intelligence/v1/bot-weekly.json): Weekly UTC claimed bot counts by profile, Monday through Sunday - [/datasets/latest.json](https://ninizemi.com/bot-intelligence/datasets/latest.json): Complete latest aggregate snapshot - [API documentation](https://ninizemi.com/bot-intelligence/docs): examples and conditional GET. - [OpenAPI](https://ninizemi.com/bot-intelligence/openapi.json): API specification. - [Data catalog](https://ninizemi.com/bot-intelligence/v1/catalog): machine-readable endpoint descriptions. ## Methodology - [Collection methodology](https://ninizemi.com/bot-intelligence/about): interpretation and data quality. - [Privacy](https://ninizemi.com/bot-intelligence/privacy): what is stored and published. No API key is required. Normal GET requests contribute coarse telemetry. No human/bot classification is performed. Client identity is unverified. Only traffic to this service is represented; counts are not unique visitors. There is no application-level observation count or retention cap. Hosting quotas and short-burst protection still apply. Snapshots are checked hourly. Use ETag / If-None-Match and poll about once per hour. ## Complete machine-readable catalog ```json { "name": "BOT Intelligence Network", "version": "0.2.0", "purpose": "A public observation exchange: requests contribute coarse telemetry; later clients retrieve aggregated knowledge.", "attribution": "Client families are unverified User-Agent claims. No human/bot classification is performed.", "scope": "Only requests received by this service. Explicit BIN verification clients are excluded; initial launch observations have been removed.", "policy": { "daily_observation_limit": null, "detail_retention_days": null, "aggregate_retention_days": null, "recommended_poll_interval_seconds": 3600, "minimum_published_group_size": 3, "burst_protection": { "requests": 60, "period_seconds": 10, "scope": "per source IP and Cloudflare location; approximate" } }, "endpoints": [ { "path": "/v1/observe", "purpose": "Your coarse request context and recording receipt", "freshness": "per request", "url": "https://ninizemi.com/bot-intelligence/v1/observe" }, { "path": "/v1/stats", "purpose": "Traffic distributions and all-time daily counts", "freshness": "hourly", "url": "https://ninizemi.com/bot-intelligence/v1/stats" }, { "path": "/v1/patterns", "purpose": "Observed country/ASN/client-family combinations; groups of at least 3", "freshness": "hourly", "url": "https://ninizemi.com/bot-intelligence/v1/patterns" }, { "path": "/v1/changes", "purpose": "Difference between the two most recent snapshots", "freshness": "hourly", "url": "https://ninizemi.com/bot-intelligence/v1/changes" }, { "path": "/v1/bots.json", "purpose": "Known crawler profiles and unverified individual User-Agent observations", "freshness": "per request", "url": "https://ninizemi.com/bot-intelligence/v1/bots.json" }, { "path": "/v1/updates.json", "purpose": "New daily User-Agent observations and catalog additions; paginated", "freshness": "per request", "url": "https://ninizemi.com/bot-intelligence/v1/updates.json" }, { "path": "/v1/bot-daily.json", "purpose": "Daily UTC claimed bot counts by profile", "freshness": "per request", "url": "https://ninizemi.com/bot-intelligence/v1/bot-daily.json" }, { "path": "/v1/bot-weekly.json", "purpose": "Weekly UTC claimed bot counts by profile, Monday through Sunday", "freshness": "per request", "url": "https://ninizemi.com/bot-intelligence/v1/bot-weekly.json" }, { "path": "/datasets/latest.json", "purpose": "Complete latest aggregate snapshot", "freshness": "hourly", "url": "https://ninizemi.com/bot-intelligence/datasets/latest.json" } ], "discovery": { "bots": "https://ninizemi.com/bot-intelligence/bots/", "updates_rss": "https://ninizemi.com/bot-intelligence/updates.xml", "dataset": "https://ninizemi.com/bot-intelligence/datasets", "docs": "https://ninizemi.com/bot-intelligence/docs", "methodology": "https://ninizemi.com/bot-intelligence/about", "openapi": "https://ninizemi.com/bot-intelligence/openapi.json", "llms": "https://ninizemi.com/bot-intelligence/llms.txt", "llms_full": "https://ninizemi.com/bot-intelligence/llms-full.txt", "sitemap": "https://ninizemi.com/bot-intelligence/sitemap.xml", "privacy": "https://ninizemi.com/bot-intelligence/privacy" }, "client_guidance": "Use conditional GET with ETag. Poll aggregates at most hourly; repeated self-requests do not create independent evidence." } ``` ## Example request ```sh curl https://ninizemi.com/bot-intelligence/datasets/latest.json ``` Stored attributes: UTC time, country, ASN, coarse client family, HTTP protocol, fixed route. Detailed records and compact aggregates have no automatic deletion period. Publication of attribute combinations requires at least three observations; smaller groups remain stored.