Crawlers such as GPTBot, ClaudeBot and Googlebot download your HTML but don't run JavaScript, so the tracking script never sees them. Your web server sees every request, so it reports them instead.
Send your requests
Create an API key in Settings → API keys, then post batches of requests to /api/crawl:
curl -X POST https://analytics.example.com/api/crawl \
-H "Authorization: Bearer oat_…" \
-H "Content-Type: application/json" \
-d '{"site":"YOUR_SITE_ID","hits":[{"path":"/blog/post","user_agent":"Mozilla/5.0 (compatible; GPTBot/1.2)","status":200}]}'| Field | Meaning |
|---|---|
path |
Request path or full URL; the query string is dropped. |
user_agent |
The request's User-Agent header. |
status |
Optional HTTP status you answered with. |
ts |
Optional RFC 3339 time, for importing logs. Defaults to now. |
Up to 1000 hits per request. Requests from regular browsers are discarded, so you can send everything. The API key must belong to the organization that owns the website.
From a Rust (axum) app
The repository includes a ready-made middleware in app/examples/crawl_forwarder.rs. It batches requests in memory and sends them every five seconds, off the request path. If the analytics server is unavailable, hits are dropped instead of slowing your site.
let tx = crawl_forwarder(format!("{ANALYTICS}/api/crawl"), site_id, api_key);
let app = app.layer(axum::middleware::from_fn_with_state(tx, forward));Categories
| Type | Examples |
|---|---|
| AI training | GPTBot, ClaudeBot, CCBot, Bytespider, Applebot-Extended |
| AI assistant | ChatGPT-User, Claude-User, PerplexityBot, OAI-SearchBot |
| Search engine | Googlebot, Bingbot, DuckDuckBot, Applebot |
| SEO tool | AhrefsBot, SemrushBot, MJ12bot |
| Link preview | Slackbot, Discordbot, Twitterbot, LinkedInBot |
| Other bot | curl, Python clients, headless browsers, unknown bots |
Crawlers are identified by their user agent. A client that pretends to be Googlebot is counted as Googlebot.