skills.rest
Marshal-Nguyen
Marshal-NguyenCOMMUNITY
Explore all skills by this author →
GitHub

geo-crawlers

Analyzes robots.txt, meta tags, and HTTP headers to map AI crawler access for a website.

1|Updated Aug 17, 2026
Search Tags:#web analysis#robots.txt#seo audit#geo#ai visibility#ai crawlers#gptbot
One-click install
npx skills add https://github.com/Marshal-Nguyen/Skill_Claude_Agent --skill geo-crawlers-marshal-nguyen
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: geo-crawlers
Source: https://github.com/Marshal-Nguyen/Skill_Claude_Agent/tree/main/claude/skills/geo-crawlers
Command: npx skills add https://github.com/Marshal-Nguyen/Skill_Claude_Agent --skill geo-crawlers-marshal-nguyen

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Many websites unknowingly block AI crawlers like GPTBot, ClaudeBot, and PerplexityBot through legacy robots.txt rules, making their content invisible in AI-generated search results. This Skill audits a site's crawler access configuration and produces a complete access map with actionable fixes. ## Core Features & Use Cases - Crawler Access Audit: Parses robots.txt, meta robots tags, and X-Robots-Tag HTTP headers to determine whether 14 major AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more) are allowed or blocked. - AI Visibility Scoring: Computes a 0-100 AI Crawler Access Score weighted by crawler tier, blanket blocks, and presence of llms.txt and sitemaps. - Actionable Recommendations: Generates a GEO-CRAWLER-ACCESS.md report with a recommended robots.txt configuration for maximum AI search visibility. - Use Case: An SEO manager suspects their site is invisible in ChatGPT and Perplexity answers. Run this Skill against the domain to discover that a wildcard Disallow rule blocks GPTBot, then apply the generated robots.txt fix. ## Quick Start Analyze the AI crawler access configuration for example.com and generate the crawler access report.

Frequently Asked Questions about geo-crawlers

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I check if my website blocks AI crawlers like GPTBot?▼

Fetch your site's robots.txt and look for User-agent blocks targeting GPTBot, ClaudeBot, or PerplexityBot, plus wildcard Disallow rules that apply to all bots. This Skill automates that check across 14 known AI crawlers and reports each one's access status.

Which AI crawlers should I allow in robots.txt for AI search visibility?▼

Allow Tier 1 crawlers that power live AI search: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, and PerplexityBot. Tier 2 crawlers like Google-Extended, Applebot-Extended, and FacebookBot expand reach into Gemini, Apple Intelligence, and Meta AI features.

Does blocking Google-Extended affect my Google Search rankings?▼

No. Blocking Google-Extended only controls whether Google uses your content for Gemini training and AI Overviews improvement. Standard Google Search rankings are governed by the regular Googlebot, which is a separate crawler.

Can X-Robots-Tag headers block AI crawlers even if robots.txt allows them?▼

Yes. X-Robots-Tag HTTP headers such as noindex or noai override page-level access and also apply to non-HTML resources. The audit checks response headers on sample pages alongside meta robots tags to catch these overrides.

Why is my content invisible in ChatGPT even though my robots.txt allows GPTBot?▼

Common causes include noai or noindex meta tags, X-Robots-Tag headers, or heavy JavaScript rendering that GPTBot cannot process since it has limited JS rendering capability. Server-side rendering or static generation mitigates the rendering issue.

Related Skills

Explore More Marketing & Sales Skills→
enowdevenowdev
13

content-gap-analysis

Compare site sitemaps and organic search performance against competitor benchmarks.

Community
Advanced
mikelmyersmikelmyers

ai-seo

Audit web content for AI search engine extractability and citation readiness.

Community
Intermediate
mohammedalaa89mohammedalaa89

ai-seo

Audit AI search visibility across Google AI Overviews, ChatGPT, Perplexity, Claude, Gemini, and Copilot.

Community
Advanced
cursorcursor
6.6k

principle-prove-it-works

Verify completed work by directly inspecting real artifacts instead of proxies or self-reports.

Official
Basic