Skills
.Work. You
Rest

robots-txt

Community

Control crawler access to your site.

Marketing & Sales#seo#robots.txt#technical seo#directives#ai crawlers#crawler control
Authorkostja94
Version1.0.0
Installs0

System Documentation

What problem does it solve?

This Skill helps you manage how search engines and AI crawlers access your website by configuring and auditing your robots.txt file.

Core Features & Use Cases

  • Configure robots.txt: Set rules for Disallow, Allow, Sitemap, and Clean-param directives.
  • Audit for accidental blocks: Identify and fix issues where important content might be unintentionally blocked from crawlers.
  • AI Crawler Strategy: Define specific rules for AI bots like GPTBot and ClaudeBot.
  • Use Case: Ensure that your staging environment is not indexed by search engines by adding a Disallow: /staging/ rule to your robots.txt.

Quick Start

Configure the robots.txt file to disallow all crawlers from the /admin/ path and specify the sitemap location.

Dependency Matrix

Required Modules

None required

Components

references

💻 Claude Code Installation

Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.

Please help me install this Skill:
Name: robots-txt
Download link: https://github.com/kostja94/marketing-skills/archive/main.zip#robots-txt

Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
View Source Repository

Agent Skills Search Helper

Install a tiny helper to your Agent, search and equip skill from 223,000+ vetted skills library on demand.