← All articles

1. Blocking AI Crawlers in robots.txt

Many sites blindly block all bots except Googlebot. If GPTBot, PerplexityBot, or ClaudeBot are disallowed, you’re invisible to 40% of search traffic. Check your robots.txt right now — if it has User-agent: * with Disallow: rules, you may be blocking AI crawlers.

Fix: Explicitly allow AI crawlers or remove blanket blocks.

2. No Structured Data (JSON-LD)

Without schema markup, search engines have to guess what your page is about. With it, they know exactly. Sites with proper JSON-LD get 2–4x more rich results in Google and significantly higher citation rates in AI search.

Fix: Add Organization, WebSite, and page-specific schemas to every page.

3. Slow Page Speed

If your TTFB (time to first byte) exceeds 800ms, crawlers may time out before indexing your content. Users bounce after 3 seconds. Both Google and AI crawlers penalize slow sites.

Fix: Compress images, enable caching, use a CDN, minimize JavaScript.

4. Thin Content Pages

Pages with less than 300 words rarely rank for anything. AI search engines almost never cite thin content because there’s not enough substance to pull a useful answer from.

Fix: Consolidate thin pages or expand them with genuine, useful content.

5. Duplicate Title Tags and Meta Descriptions

Every page needs a unique title and description. When 10 pages share the same title, search engines can’t distinguish between them — so they rank none of them.

Fix: Write a unique, keyword-rich title (under 60 chars) and description (under 160 chars) for every page.

6. Missing or Multiple H1 Tags

Every page should have exactly one H1 that describes what the page is about. Zero H1s = search engines don’t know your main topic. Multiple H1s = confused hierarchy.

Fix: One H1 per page, with your primary keyword included naturally.

7. Keyword Stuffing

Repeating your target keyword 47 times on a page doesn’t help — it triggers spam filters. Modern AI is very good at detecting unnatural keyword density.

Fix: Write for humans. Use your keyword naturally 2–3 times and use related terms throughout.

8. Missing Canonical Tags

Without canonical URLs, search engines may index multiple versions of the same page (www vs non-www, HTTP vs HTTPS, trailing slash vs no slash). This splits your ranking power.

Fix: Add <link rel="canonical"> to every page pointing to the preferred URL.

9. Broken Internal Links

404 errors waste crawl budget and create dead ends. If your most important pages are only reachable through broken link chains, crawlers may never find them.

Fix: Audit internal links monthly. Redirect or fix every 404.

10. No Sitemap

Without a sitemap.xml, crawlers have to discover pages by following links. If your site structure is flat or pages are deeply nested, many pages won’t get indexed.

Fix: Generate a sitemap.xml, submit it in Search Console, reference it in robots.txt.

11. Not Using HTTPS

In 2026, HTTP-only sites are explicitly deprioritized by both Google and AI search. SSL certificates are free — there is zero reason to run HTTP.

Fix: Enable HTTPS with a free SSL certificate (Cloudflare, Let’s Encrypt).

12. Ignoring Mobile

Google uses mobile-first indexing. If your mobile experience is broken (tiny text, overlapping elements, horizontal scroll), that’s the version Google ranks — and AI crawlers see.

Fix: Test with Google’s Mobile-Friendly Test. Fix viewport, font sizes, and touch targets.

Find Your SEO Mistakes in 30 Seconds

Enter any URL. Get a pass/fail report for all 14 checks.

Run Free Audit →