Robots.txt blocking
Sixty bytes of text can un-list your entire business.
robots.txt is the doorman's instructions for search engines — and 'Disallow: /' is the shortest catastrophic sentence on the web. Launch-day leftovers, overzealous rules, and blocked assets quietly starve sites of the only visitor that multiplies the rest.
Same-day diagnosis. Flat quote before any fix.
What’s actually happening
Every crawler checks yourdomain.com/robots.txt before crawling. The file's rules say which paths are welcome. It's advisory — honest crawlers obey it — which makes it powerful in both directions: correctly written, it steers crawl budget; incorrectly, it's a self-imposed gag order that Google respectfully honors while your rankings starve.
The subtle modern failure isn't the total block — it's blocking assets: rules disallowing /wp-content/ or script paths mean Google fetches your HTML but can't render it as visitors see it, judging a skeleton instead of a page. Rankings sag with no obvious cause, because the page 'works fine' in every human browser.
The usual causes, ranked
After twenty-seven years of these calls, the odds are well mapped. Start at the top.
The staging block that launched
Disallow: / was correct on the development copy — and came along to production. Months of invisibility with a one-line cause. The classic.
Asset paths caught in old rules
Ancient advice said block wp-content and scripts; modern Google needs them to render. Old files carrying old wisdom, quietly sabotaging.
Overbroad pattern rules
A rule meant for one directory written loosely enough to catch half the site — Disallow: /page catching /pages/ and everything after.
Plugin or host rewriting the file
SEO plugins, security tools, and some hosts generate robots.txt dynamically — and a setting toggled somewhere rewrote yours.
What you can safely try first
Nothing below can make things worse — that’s the selection criterion. Anything riskier belongs in professional hands, on a backup.
- 1
Read it — right now, it's public
Visit yourdomain.com/robots.txt. 'Disallow: /' under User-agent: * is the catastrophe line. Two minutes, no tools, complete answer to 'am I blocked.'
- 2
Test a page in Search Console
URL Inspection says outright whether a page is 'blocked by robots.txt' — Google's own verdict on Google's own behavior.
- 3
Note whether the file is real or generated
If no robots.txt file exists on the server but the URL serves one, a plugin is generating it — and the fix lives in that plugin's settings, not in a file editor.
Stop and call when…
- The blocking rule exists but you're unsure what else that rule protects — deleting blind can expose what was correctly hidden
- The file is dynamically generated and the generator isn't obvious
- The block has been live for weeks — recovery sequencing (fix, then request recrawls, then watch) is worth doing properly
From there it’s our job: same-day look, flat quote, and the $229 flat repair covers most cases of exactly this.
Single Error Fix — buy it now, skip the hunt.
One error, hunted down and fixed — 500s, white screens, redirect loops, broken pages.
Covers one specific error or broken behavior on one site. Diagnosis, the fix, and a plain-English note on what happened. If we can't fix it, you get a full refund.
Questions we hear a lot.
How fast does Google notice the fix?
The file gets re-checked within days; recrawling your pages follows over days to weeks, accelerated by requesting indexing on priority URLs. Recovery speed tracks how established the site was — known sites re-index fast; new ones queue. Either way the un-gagging starts immediately.
Should I even have a robots.txt?
Yes, a boring one: allow everything meaningful, point at your sitemap, optionally steer crawlers away from genuinely useless paths. The best robots.txt files are three lines long and never thought about again. Cleverness in this file has a body count.
Does robots.txt keep pages private?
No — it's a courtesy sign, not a lock. Honest crawlers obey; browsers and bad actors ignore it entirely, and blocked-but-linked URLs can still appear in results as bare links. Privacy needs authentication; robots.txt just manages crawlers.
Related symptoms & help
Errors travel in packs. If this one visited, check its friends.
Un-gag the site. Sixty bytes, done right.
Send the symptom, get a same-day look and a flat quote from the developer who's fixed this exact thing more times than either of us can count.