site-crawlability

What problem does it solve?

This Skill addresses issues that prevent search engines and AI crawlers from effectively discovering and indexing your website's content, such as broken links, poor site structure, and inefficient crawl budget usage.

Core Features & Use Cases

• Fixes Crawlability Issues: Identifies and provides solutions for redirect chains, broken links (4xx errors), and orphan pages.
• Optimizes Site Structure: Ensures important pages are easily accessible and recommends pagination over infinite scroll for better discoverability.
• Manages Crawl Budget: Helps reduce wasted crawl attempts on duplicate or low-value URLs.
• AI Crawler Optimization: Provides guidance for server-side rendering and URL management to improve visibility for AI bots like GPTBot and ClaudeBot.
• Use Case: A website owner notices that some of their product pages are not appearing in search results. This Skill can help diagnose if the issue is due to broken internal links, a deep site hierarchy, or problems with how AI crawlers are accessing the content.

Quick Start

Analyze the crawlability of the website and identify any issues with broken links or site structure.