Skills
.Work. You
Rest

web-crawler

Community

Fast, stealthy web crawling with ML-ready output

Software Engineering#reporting#Rust#robots.txt#crawl#web crawler#LLM-ready
Authorleobrival
Version1.0.0
Installs0

System Documentation

What problem does it solve?

This Skill provides a high-performance web crawler that maps site structure, discovers sitemaps, extracts links, and produces multi-format reports for quick analysis and integration with LLM/RAG pipelines.

Core Features & Use Cases

  • Performance: 60+ pages/sec crawl speed, minimal startup time, and low memory usage in Rust.
  • Robustness & Compliance: Robots.txt handling and rate-limited crawling to respect site policies.
  • Output & Intelligence: Generates JSON, Markdown (LLM-ready), HTML reports, and CSV/Links outputs; supports content filtering and stealth mode for realistic requests.
  • Use Case: Quickly map a new site to understand its structure, extract internal and external links, and prepare data for AI indexing.

Quick Start

To begin, run the prebuilt binary located at ~/.claude/skills/web-crawler/bin/rcrawler with a target URL, for example: ~/.claude/skills/web-crawler/bin/rcrawler https://example.com -w 20 -d 2

Dependency Matrix

Required Modules

None required

Components

Standard package

💻 Claude Code Installation

Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.

Please help me install this Skill:
Name: web-crawler
Download link: https://github.com/leobrival/serum-plugins-official/archive/main.zip#web-crawler

Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
View Source Repository

Agent Skills Search Helper

Install a tiny helper to your Agent, search and equip skill from 223,000+ vetted skills library on demand.