skills.rest
kiurakku
kiurakkuCOMMUNITY
Explore all skills by this author →
𝕏@thefear078GitHub

geo-crawlers

Analyzes robots.txt, meta robots, and HTTP headers to map AI crawler access and generate robots.txt recommendations.

15|3|Updated Jul 9, 2026
Search Tags:#robots.txt#seo audit#geo#ai crawlers#crawler access#gptbot#ai search visibility
One-click install
npx skills add https://github.com/kiurakku/cursor-kit-for-ai --skill geo-crawlers-kiurakku
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: geo-crawlers
Source: https://github.com/kiurakku/cursor-kit-for-ai/tree/main/plugins/geo-seo/skills/geo-crawlers
Command: npx skills add https://github.com/kiurakku/cursor-kit-for-ai --skill geo-crawlers-kiurakku

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? AI crawlers like GPTBot, ClaudeBot, and PerplexityBot are often blocked by legacy robots.txt rules, WAF settings, or noindex headers, making a site invisible to AI search engines despite strong traditional SEO. This Skill audits crawler access end-to-end and produces a scored report with copy-paste robots.txt fixes. ## Core Features & Use Cases - Crawler Access Matrix: Fetches robots.txt, meta robots tags, and X-Robots-Tag headers, then resolves allow/block/partial status for Tier 1 and Tier 2 AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, and more). - Scoring & Severity Model: Calculates a 0-100 GEO crawler score with a severity rubric (Blocker/Major/Minor) covering catch-all Disallow rules, noindex headers, and WAF overrides. - Policy-Aligned Recommendations: Generates copy-paste robots.txt snippets for allow-all, block-training-only, or path-level policies, plus Cloudflare-specific verification commands. - Use Case: A site with strong Google rankings never appears in ChatGPT or Perplexity answers. Run this Skill to discover that a catch-all Disallow: / blocks all Tier 1 bots, then apply the generated robots.txt diff and re-verify with curl using AI bot user-agents. ## Quick Start Audit the AI crawler access for my domain example.com and generate a crawler access report with recommended robots.txt changes.

Frequently Asked Questions about geo-crawlers

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I check if my robots.txt blocks AI crawlers like GPTBot?▼

Fetch your robots.txt with curl and grep for user-agent rules matching GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot. Also check the catch-all User-agent: * section, since a Disallow: / there blocks all bots without explicit exceptions.

Which AI crawlers should I allow for AI search visibility?▼

Allow Tier 1 crawlers: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, and PerplexityBot, plus Googlebot and Bingbot which power AI Overviews and Copilot. Blocking these removes your site from ChatGPT, Claude, and Perplexity answers.

Can I block AI training but still appear in AI search results?▼

Yes. Disallow training-only crawlers like GPTBot, Google-Extended, and CCBot while allowing search crawlers such as OAI-SearchBot, ChatGPT-User, PerplexityBot, and ClaudeBot. Google-Extended does not affect standard Google ranking.

Does Cloudflare block AI bots even if robots.txt allows them?▼

Yes. Cloudflare's Block AI Scrapers toggle, Bot Fight Mode, and custom WAF rules can override a permissive robots.txt at the edge. Verify by sending curl requests with AI bot user-agents like GPTBot and PerplexityBot and checking the response status.

Why is my site invisible in ChatGPT despite good Google SEO?▼

Common causes include a catch-all Disallow: / in robots.txt, noindex meta tags or X-Robots-Tag headers on key pages, or WAF rules blocking AI user-agents. Traditional SEO success does not guarantee AI crawler access.

What happens if I block Googlebot to save crawl budget?▼

Blocking Googlebot is a fatal error for both SEO and GEO, since it powers Google Search and AI Overviews. The scoring rubric treats a sitewide Googlebot block as a Blocker-level finding that overrides all other criteria.

Related Skills

Explore More Marketing & Sales Skills→
sitharaj88sitharaj88
1

azure-front-door

Generate Azure Front Door Bicep templates for load balancing, CDN, and WAF.

Community
Advanced
erichowenserichowens
181

seo-visibility-expert

Optimize SEO and AI crawler signals for web projects.

Community
Advanced
upmarkingupmarking

seo-geo

Analyze AI visibility signals and generate a GEO-ANALYSIS.md readiness report.

Official
Advanced
Nurdiansyah0Nurdiansyah0

ai-seo

Optimizes website content for citation by AI search engines like ChatGPT, Perplexity, and Google AI Overviews.

Community
Advanced