skills.rest
venhdev
venhdevCOMMUNITY
Explore all skills by this author →
GitHub

link-crawler

Crawl websites to configurable depth and validate linked page accessibility.

Updated May 3, 2026
Search Tags:#web scraping#url validation#sitemap generation#website audit#link verification#link crawling#broken link check
One-click install
npx skills add https://github.com/venhdev/skills --skill link-crawler
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: link-crawler
Source: https://github.com/venhdev/skills/tree/main/skills/link-crawler
Command: npx skills add https://github.com/venhdev/skills --skill link-crawler

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Manually checking if every link on a website works is extremely time-consuming for sites with hundreds or thousands of pages, and getting a full inventory of all active content without automated tools is nearly impossible.

Core Features & Use Cases

  • Automated Link Crawling: Crawl websites up to 3 levels deep to discover all linked pages from one or more root URLs.
  • Link Validation: Automatically verify each discovered link returns a valid, content-rich page, filtering out error pages, placeholders, and dead links.
  • Use Case: Use this to audit a documentation site for broken links before a release, generate a full sitemap of active pages for a product website, or verify all links remain working after a site migration.

Quick Start

Use the link-crawler skill to crawl a specified website to your chosen depth and get a grouped list of all active working links along with counts of any dead or unreachable URLs.

Frequently Asked Questions about link-crawler

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I check a website for broken links automatically?▼

Yes, you can generate a sitemap by crawling a website to a configurable depth to discover all linked pages. The crawler outputs a JSON inventory of active, content-rich pages, which serves as a full sitemap for product websites or documentation sites.

Do I need to install external libraries to crawl URLs with Python?▼

You can validate URLs after a site migration by running the link crawler to verify all linked pages remain working and return valid content. It supports single-URL and multi-topic config file modes, outputting grouped JSON with active link counts and dead link entries.

What is the maximum crawl depth for discovering linked pages?▼

The maximum crawl depth for discovering linked pages is up to 3 levels deep from the root URL. This allows thorough website link auditing and page inventory creation without crawling indefinitely, ensuring efficient sitemap generation and broken link detection.

Can I get real-time logs for per-URL investigation during a website audit?▼

Yes, you can get real-time logs for per-URL investigation during a website audit. The crawler provides optional real-time per-URL investigation logs alongside the grouped JSON output, detailing active link counts and dead link entries as it validates accessibility.

Related Skills

Explore More Software Engineering Skills→
Shamrock2245Shamrock2245
1

python-executor

Execute Python 3.10 scripts in a sandboxed environment with configurable timeouts.

Community
Intermediate
DisposableByDefaultDisposableByDefault
1

linkedin-search

Search LinkedIn posts and people via the link-pulse CLI and produce theme-grouped summaries.

Official
Intermediate
liumeixinliumeixin
4

blogwatcher

Monitor blogs and RSS/Atom feeds with read/unread tracking and category filtering.

Community
Intermediate
smarmen86smarmen86

seo-firecrawl

Crawl and map websites via Firecrawl MCP for SEO audits.

Community
Advanced