skills.rest
finedesignz
finedesignzCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

geo-crawlers

Analyzes robots.txt, meta tags, and HTTP headers to map AI crawler access for a website.

Updated Jun 7, 2026
Search Tags:#llms.txt#robots.txt#seo audit#geo#ai visibility#ai crawlers#gptbot
One-click install
npx skills add https://github.com/finedesignz/geo-seo-claude --skill geo-crawlers-finedesignz
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: geo-crawlers
Source: https://github.com/finedesignz/geo-seo-claude/tree/main/skills/geo-crawlers
Command: npx skills add https://github.com/finedesignz/geo-seo-claude --skill geo-crawlers-finedesignz

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Websites often unknowingly block AI crawlers like GPTBot, ClaudeBot, and PerplexityBot through legacy robots.txt rules, making their content invisible in AI-generated search results. This Skill audits a site's crawler access configuration and identifies exactly which AI bots can or cannot reach its content. ## Core Features & Use Cases - Crawler Access Mapping: Checks robots.txt, meta robots tags, and X-Robots-Tag HTTP headers against a reference of 14 AI crawlers across three tiers (search-critical, ecosystem, training-only). - AI Visibility Scoring: Produces a 0-100 AI Crawler Access Score weighted by crawler tier, blanket blocks, and presence of AI-specific files like llms.txt. - Actionable Recommendations: Generates a GEO-CRAWLER-ACCESS.md report with a complete access matrix, critical issues, and a recommended robots.txt configuration. - Use Case: A marketing team notices zero referral traffic from ChatGPT and Perplexity. Running this audit reveals their robots.txt inherited a wildcard Disallow rule blocking GPTBot and PerplexityBot, and the report provides the exact fix. ## Quick Start Ask the AI to run a crawler access audit on your domain, for example: analyze which AI crawlers can access https://example.com and give me the recommended robots.txt changes.

Frequently Asked Questions about geo-crawlers

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I check if my website blocks AI crawlers like GPTBot?▼

Fetch your site's robots.txt and look for User-agent blocks targeting GPTBot, ClaudeBot, or PerplexityBot, plus any wildcard Disallow: / rules that apply to all bots. This audit automates that check across 14 known AI crawlers and reports each one's access status.

Which AI crawlers should I allow in robots.txt for AI search visibility?▼

Allow Tier 1 crawlers that power live AI search: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, and PerplexityBot. Tier 2 crawlers like Google-Extended, Applebot-Extended, and Amazonbot are also recommended since blocking them reduces presence in AI features without protecting search rankings.

Does blocking Google-Extended affect my Google Search rankings?▼

No. Blocking Google-Extended only affects whether Google uses your content for Gemini training and AI Overviews improvement. Standard Google Search rankings are controlled by the regular Googlebot, which is a separate crawler.

Can meta tags or HTTP headers block AI crawlers even if robots.txt allows them?▼

Yes. Meta robots tags like noindex or noai and X-Robots-Tag HTTP headers can override robots.txt at the page level. The audit samples key pages for these tags and headers, including bot-specific variants like X-Robots-Tag: GPTBot: noindex.

Why is my content invisible in ChatGPT even though my robots.txt allows GPTBot?▼

Common causes include JavaScript-rendered content that GPTBot cannot execute, page-level noindex meta tags, or X-Robots-Tag headers overriding robots.txt. The audit checks rendering requirements and header-level blocks to identify these hidden issues.

Related Skills

Explore More Marketing & Sales Skills→
joanbyjoanby
25

geo-crawlers

Analyze robots.txt, meta robots tags, and HTTP headers to map AI crawler access.

Community
Intermediate
MileniumTickMileniumTick

ai-seo

Assess AI visibility across AI search engines and recommend structural improvements.

Community
Advanced
chapter42chapter42

geo-crawlers

Analyze domain AI crawler access across robots.txt, meta tags, and headers.

Community
Advanced
Marshal-NguyenMarshal-Nguyen
1

geo-crawlers

Analyzes robots.txt, meta tags, and HTTP headers to map AI crawler access for a website.

Community
Intermediate