Why Your Website Disappeared from AI Search (and How to Fix It)
Published September 13, 2026 • 7 min read
You used to show up in ChatGPT answers. Perplexity cited your pages. Then it stopped. Your traffic from AI search dropped, and you're not sure why.
This is more common than you'd think. AI search engines re-crawl and re-evaluate sources constantly, and small changes — sometimes ones you didn't even make — can knock you out. Here are the 6 most common causes and how to fix each one.
1. Your Hosting Provider Blocked AI Crawlers
This is the #1 cause. Many hosting providers, CDN services, and CMS platforms have quietly enabled AI bot blocking by default. Cloudflare added a "Block AI Scrapers and Crawlers" toggle in their dashboard — and it's ON by default for new zones. WordPress security plugins like Wordfence and Sucuri often block AI bots automatically.
How to check
Visit yoursite.com/robots.txt and look for these user agents being blocked:
GPTBot ClaudeBot PerplexityBot Google-Extended Applebot-Extended
If any of these show Disallow: /, that AI search engine literally cannot read your site.
How to fix
If using Cloudflare: go to Security > Bots > turn OFF "Block AI Scrapers and Crawlers." If using a WordPress plugin: check its bot blocking settings. Then add explicit Allow rules to your robots.txt for each AI bot.
2. A CMS Update Changed Your robots.txt
WordPress, Shopify, Wix, and Squarespace all manage robots.txt automatically. A platform update or a new plugin can silently change your crawl directives. If you're not monitoring your robots.txt, you won't notice until your AI traffic drops.
How to fix
Set up a monitoring check that alerts you when your robots.txt content changes. Our audit tool checks AI crawler access as part of its 14-point scan. Run it monthly at minimum.
3. Your Server Is Too Slow for AI Crawlers
AI crawlers have strict timeout thresholds. GPTBot gives up on pages that take more than 5 seconds to respond. If your server performance has degraded — maybe you added a heavy plugin, hit traffic spikes, or your hosting plan is throttled — crawlers may have stopped indexing your content.
How to fix
Check your server's TTFB (time to first byte). It should be under 2 seconds. Use a CDN if you're not already. Static sites and edge-rendered pages (Cloudflare Pages, Vercel, Netlify) solve this permanently.
4. Your Content Went Behind a JavaScript Wall
If you redesigned your site with a JavaScript framework (React, Vue, Angular) without server-side rendering, AI crawlers may see an empty page. Most AI bots don't execute JavaScript — they read the initial HTML response. If your content only loads after client-side rendering, it's invisible.
How to check
Run curl -s yoursite.com | grep "your main heading text" from a terminal. If your heading isn't in the raw HTML, crawlers can't see it either. You need SSR (server-side rendering) or SSG (static site generation).
5. Your Competitors' Content Got Better
AI search engines are constantly re-evaluating sources. If a competitor published a more comprehensive, more current version of content you used to be cited for, you may have been displaced. This isn't about your site getting worse — it's about the competitive landscape improving.
How to fix
Ask the AI search engine the queries you used to rank for. See who's being cited now. Read their content. Then update yours to be more current, more specific, and more comprehensive. Add original data that competitors can't replicate.
6. Your Schema Markup Broke
If you had valid JSON-LD structured data and it was removed or corrupted during a site update, AI crawlers lose a key signal about your content's type and authority. A page that AI could previously identify as an "Article by Expert X about Topic Y" becomes just another page of text.
How to check
View your page source and search for application/ld+json. If it's missing or shows JSON syntax errors, fix it immediately. Use Google's Schema Validator to test your markup.
How to Monitor Your AI Search Visibility
Prevention beats recovery. Set up these three checks:
- Monthly robots.txt audit — verify AI crawlers are allowed. Use our free audit tool
- Quarterly content refresh — update dates, statistics, and examples on your top pages
- Monthly citation check — ask ChatGPT, Perplexity, and Google for queries you target. Are you being cited? By whom?
Check Your AI Search Visibility
Free 14-point AI SEO audit. See exactly what AI search engines can and can't see on your site.
Run Free Audit →