Guides · Crawler policy

How to Block (or Allow) AI Crawlers Without Breaking Google

Last verified: October 2026 · About 8 minutes of reading

There are four places to state or enforce a crawler rule, and only one is worth starting with. This guide gives copyable configuration for robots.txt, Cloudflare, Vercel and nginx, then shows the edit that takes a site out of Google Search — a far more common mistake than an over-permissive AI policy.

Decide first: training vs search

Block training crawlers if you want to limit future models; block search crawlers and you leave today's answers.

The two groups are not interchangeable, and a single wildcard rule hits both. Training tokens are GPTBot, ClaudeBot, Google-Extended and CCBot. Search tokens are OAI-SearchBot, ChatGPT-User, Claude-User and PerplexityBot. The background on each is in the AI crawlers list.

The last row is the tempting shortcut: one line, all eight tokens blocked. It also blocks Googlebot, which is the failure covered further down.

robots.txt syntax, crawler by crawler

One group per token, a blank line between groups, and a Disallow of / is enough to state the policy.

This block implements "block training, allow search". Multiple User-agent lines can share one group, so the three fetchers are combined and Google-Extended is listed separately for clarity:

# Block training crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /

# Google-Extended is a token, not a fetcher:
# it controls Gemini training for pages Googlebot already crawled.
User-agent: Google-Extended
Disallow: /

# Allow retrieval crawlers explicitly
User-agent: OAI-SearchBot
Allow: /

# ChatGPT-User is user-initiated: robots.txt may not apply.
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

# Everything else keeps the default rules
User-agent: *
Disallow: /admin/
Disallow: /api/

Two AI-specific details decide whether this actually works:

• Google-Extended blocks nothing. It is a usage instruction, so it belongs in the same file but does not stop a fetch. Only the other tokens have a fetcher to stop.
• ChatGPT-User is the one token robots.txt cannot reliably control. OpenAI documents that the request is user-initiated and robots.txt rules may not apply to it; enforce this group at the edge if blocking it matters.

The parsing rules underneath this block — how groups form, why the longest match wins, what happens when the file returns a 404 or a 500 — are the same for every crawler and are covered in the robots.txt guide.

Cloudflare

Since 15 September 2026 Cloudflare enforces three separate AI controls — search, training and agents — and its Block setting also stops Googlebot.

Older guides describe a single "Block AI Bots" switch, which Cloudflare retired on 15 September 2026. The replacement is three controls: Search, Training and Agent. The difference is not cosmetic — Block and Block on pages with ads now apply to mixed-use crawlers such as Applebot, Bingbot and Googlebot, so blocking Training takes you out of traditional search too. The setting that refuses training while keeping search intact is Disallow AI Training.

1. Open the zone, then Security, then Bots, then AI Crawl Control. A zone that has not migrated may still show the legacy switch.
2. Read the Search control first. If it is set to Block, Googlebot and Bingbot are affected, not only AI crawlers.
3. For Training, choose Disallow AI Training rather than Block if you want to stay discoverable.
4. Agent controls govern on-demand fetchers. Block on pages with ads is the usual middle ground.
5. Keep robots.txt consistent with the dashboard. Managed Robots.txt became Bot Preference Sync, which writes your choices into robots.txt.

New zones start with recommended defaults — ad-supported sites with search allowed, training disallowed and agents blocked on ad pages; everything else with all three allowed. Review them rather than accept them.

Vercel

On Vercel, robots.txt covers policy and a middleware or firewall rule covers enforcement; headers alone cannot match a user agent.

vercel.json can set headers and redirects per path, but it cannot branch on User-Agent. For a Next.js project on Vercel, the shortest enforcement path is middleware:

// middleware.ts
import { NextResponse } from 'next/server';
import type { NextRequest } from 'next/server';

// Google-Extended is deliberately absent: it has no user agent.
const BLOCKED = ['GPTBot', 'ClaudeBot', 'CCBot'];

export function middleware(req: NextRequest) {
  const ua = req.headers.get('user-agent') ?? '';
  if (BLOCKED.some((token) => ua.includes(token))) {
    return new NextResponse('Forbidden', { status: 403 });
  }
  return NextResponse.next();
}

export const config = { matcher: '/:path*' };

On Pro and Enterprise plans, the Vercel firewall can express the same rule without a deploy. Either way, keep the robots.txt block above: middleware returns 403 to everything that matches, including a scraper, while robots.txt is what a well-behaved crawler reads first.

nginx

An nginx map on the user agent is the cheapest server-side block, as long as you accept that the header is forgeable.

# /etc/nginx/conf.d/ai-crawlers.conf
map $http_user_agent $block_training {
    default                                0;
    ~*GPTBot                               1;
    ~*ClaudeBot                            1;
    ~*CCBot                                1;
}

map $http_user_agent $block_search {
    default                                                    0;
    ~*(OAI-SearchBot|ChatGPT-User|Claude-User|PerplexityBot)   1;
}

server {
    listen 443 ssl;
    server_name example.com;

    # Enforces the training block only. Adding $block_search here
    # would also remove you from AI answers.
    if ($block_training) {
        return 403;
    }

    # To leave AI answers as well, uncomment:
    # if ($block_search) { return 403; }

    location / {
        try_files $uri $uri/ /index.html;
    }
}

Three notes. Both map blocks must live in the http context, so a file included from nginx.conf is the right home. The patterns are case-insensitive, which matters because some clients send lowercase. $block_search is defined but not enforced by default: uncomment the second condition only if you intend to leave AI answers too.

The misconfiguration that breaks Google too

A bare wildcard Disallow of / blocks Googlebot with everything else, and a Googlebot group only rescues part of the damage.

This is the version people paste after reading that "AI crawlers ignore robots.txt":

# Do not deploy this
User-agent: *
Disallow: /

# This line does not undo the damage for every other agent,
# and some crawlers never read it.
User-agent: Googlebot
Allow: /

# Google-Extended is not Googlebot. Blocking it is safe;
# blocking * is not.
User-agent: Google-Extended
Allow: /

What happens depends on whether the Googlebot group is present.

• Wildcard alone — the common form — blocks Googlebot, including the renderer requests that fetch stylesheets and scripts. Pages already indexed stay, but new and updated pages stop being fetched.
• Wildcard plus the Googlebot group recovers Googlebot and its renderer, because renderer requests carry the Googlebot user agent. Traditional search keeps working.
• Everything else stays blocked. Bingbot, Applebot, Googlebot-Image and Googlebot-Video (separate user agents), link-preview fetchers and every AI crawler you did not name fall into the wildcard group.

The second form is the trap: Search Console stops complaining about the main page, while image search, Bing, link previews and third-party verification quietly go dark.

The safe version keeps the wildcard permissive and blocks named tokens:

# Correct: block the crawlers you mean, leave everything else alone
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /

The difference is one line of scope. The first file makes a statement about every crawler on the internet. The second names four tokens and leaves Googlebot, Bingbot and the rest untouched.

How to verify it worked

Verify the policy as data, then verify enforcement in logs; a curl with a forged user agent proves neither.

1. Fetch the file. curl -s https://example.com/robots.txt should return your rules with status 200 and text/plain. Check that blank lines separate the groups.
2. Read the policy back. For each of the eight tokens, decide whether the file allows or blocks the homepage and a deeper path. That is what the free AI readability check reports, without pretending to be a bot.
3. Watch real traffic. Keep client IPs for a week and check which agents actually arrive. A blocked crawler disappears from logs; a spoofed one does not.
4. Check Search Console after any wildcard change. Settings, then Crawl stats, then by response. A new "Blocked by robots.txt" line for Googlebot assets is the signal that you overreached; edge 403s appear in firewall events instead.

Note what this does not establish. A 200 from curl proves the file is served, not that a vendor obeys it. A 403 for a forged header proves your rule matches, not that the real crawler is affected. Those are the two claims most "I tested it" posts get wrong.

Frequently asked questions

Four questions cover the intent behind most block requests: search impact, scope, alternatives and timing.

Last verified: October 2026. Rules were checked against RFC 9309 and vendor crawler documentation on 2026-10-24; Cloudflare behaviour varies by zone age, so confirm it in your own dashboard.