Bot Traffic & AI Crawler Tracking

See when AI assistants, search engines, and model-training crawlers request pages on your directory.

ChatGPT (GPTBot)AI Answers

Google GeminiAI Search

Anthropic ClaudeAI Answers

PerplexityBotReal-time Search

DeepSeekAI Answers

GrokReal-time Search

robots.txt & llms.txtDiscovery

1How AI Crawler Tracking Works

AI tools and search crawlers fetch your raw server HTML before users ever visit. Tracking runs server-side in your edge middleware to capture:

AI Answers

When a user queries ChatGPT or Claude about a project, the bot requests your directory page to verify current rankings and pricing.

Search Indexing

Googlebot, Bingbot, or PerplexityBot refresh search caches and indexing feeds.

Discovery Files

Monitors requests for /robots.txt, /llms.txt, and sitemaps.

2Next.js Edge Middleware Implementation

middleware.tsEdge Runtime

import { NextResponse, type NextRequest } from "next/server";

export function middleware(request: NextRequest) {
  const userAgent = request.headers.get("user-agent") || "";
  
  // Detect AI Crawlers & LLM Bots
  const isBot = /GPTBot|ClaudeBot|PerplexityBot|Google-Extended|Applebot|Bytespider/i.test(userAgent);
  
  if (isBot) {
    // Forward bot telemetry asynchronously in the background
    console.log(`[AI Bot Request]: ${userAgent} -> ${request.nextUrl.pathname}`);
  }

  return NextResponse.next();
}

export const config = {
  matcher: ["/((?!api|_next/static|_next/image|favicon.ico).*)"],
};

Ready to see incoming AI requests?

Your dashboard will automatically populate crawler logs as bots scan your leaderboard.

Return to Stats Dashboard