Blocking AI crawlers
You spent years earning Google's trust. Then blocked the robots that replaced it.
A growing share of your customers ask ChatGPT, Claude, or Perplexity for recommendations instead of searching. Those assistants learn from crawlers with names like GPTBot and ClaudeBot — and an alarming number of business sites block them without knowing it, usually courtesy of a security plugin's default settings.
Same-day diagnosis. Flat quote before any fix.
What’s actually happening
AI assistants don't conjure recommendations from nowhere: they read the web through their own crawlers — GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended — and a site those crawlers can't reach contributes nothing to what the assistants believe. Block them and you're not protecting anything; you're volunteering for invisibility in the fastest-growing referral channel there is.
The blocking is almost never a decision anyone remembers making. Security plugins ship 'block AI bots' toggles that default on. Bot-protection services classify the crawlers as scrapers. A robots.txt copied from a blog post in 2023 disallows them wholesale. CDN firewall rules challenge anything with 'Bot' in its user agent. Each layer was installed for a sensible reason; together they've drawn the curtains on the machines your customers now consult.
The usual causes, ranked
After twenty-seven years of these calls, the odds are well mapped. Start at the top.
A security plugin's default toggle
Wordfence-class plugins and 'block bad bots' tools that lump AI crawlers in with scrapers, enabled by a default nobody reviewed.
Robots.txt disallow rules
Explicit User-agent: GPTBot / Disallow: / lines — often pasted during 2023's block-the-bots moment and forgotten as the calculus changed.
CDN and firewall bot challenges
Cloudflare-tier bot management challenging AI user agents with puzzles a crawler can't solve. The page loads for humans; the machines get a wall.
Aggressive rate limiting
Rules that throttle unfamiliar user agents until they give up — a soft block with the same hard result.
What you can safely try first
Nothing below can make things worse — that’s the selection criterion. Anything riskier belongs in professional hands, on a backup.
- 1
Read your robots.txt today
Visit yoursite.com/robots.txt and look for GPTBot, ClaudeBot, PerplexityBot, or a blanket disallow. Two minutes, and it's the most common single culprit.
- 2
Audit the security stack's bot settings
Security plugin, CDN dashboard, host firewall — look for 'AI bots,' 'bot protection,' or user-agent rules. Note what's toggled on; don't bulk-disable protection to find out.
- 3
Run the visibility scan
Our free AI visibility check asks the four big assistants about your market right now — if none of them know you exist, crawler access is the first thing worth ruling out.
Stop and call when…
- The blocking lives in CDN or server rules you didn't write and can't safely rewrite
- You want AI crawlers allowed but scrapers still blocked — that's a scalpel job, not a toggle
- Access looks fine but assistants still don't know you — the problem has moved from access to citability, which is a structured-data conversation
From there it’s our job: same-day look, flat quote, and the $229 flat repair covers most cases of exactly this.
Single Error Fix — buy it now, skip the hunt.
One error, hunted down and fixed — 500s, white screens, redirect loops, broken pages.
Covers one specific error or broken behavior on one site. Diagnosis, the fix, and a plain-English note on what happened. If we can't fix it, you get a full refund.
Questions we hear a lot.
Shouldn't I block AI crawlers to protect my content?
For a publisher selling words, maybe. For a business selling services, the trade is upside down: the 'protected' content is your marketing copy — the thing you want machines to read, digest, and repeat to buyers. Blocking GPTBot from your services page is blocking a referral source from learning what you sell.
How do I know if AI even matters for my traffic yet?
Run our free AI visibility check: it asks ChatGPT, Claude, Gemini, and Perplexity who they'd recommend in your trade and town, live. If competitors come back and you don't, the machines are already routing buyers — just not to you. That's measurable today, not a future-proofing hypothetical.
What does the $229 fix actually cover?
The full access audit — robots.txt, security plugins, CDN rules, firewall, rate limits — with AI crawlers explicitly allowed, scrapers still handled, and verification that each major crawler can actually fetch your key pages. You get the before/after and a list of exactly what changed.
Related symptoms & help
Errors travel in packs. If this one visited, check its friends.
Open the curtains to the machines that recommend you.
Send the symptom, get a same-day look and a flat quote from the developer who's fixed this exact thing more times than either of us can count.