Crawl corporate websites across four sources to map URL inventory and classify content sections.
npx skills add https://github.com/MB-uc/mercury --skill ms-crawl
Please help me install this Agent Skill. Skill: ms-crawl Source: https://github.com/MB-uc/mercury/tree/main/mercury/skills/ms-crawl Command: npx skills add https://github.com/MB-uc/mercury --skill ms-crawl
This skill solves the challenge of incomplete or unreliable website auditing by automating a rigorous, four-source discovery process that ensures no content is missed during consultant research.
Run the ms-crawl skill to perform a full site discovery and generate the evidence manifest for the target company domain.
High-intent search queries and answers about installing and using this skill.
Automated corporate website discovery uses a multi-source crawl process to map URL inventory and classify content sections. It aggregates data from sitemaps, HTML navigation, full-site crawls, and pagination loops to build a complete site structure for strategic audits.
Negative verification in website crawling confirms the verified absence of specific content rather than simply missing it. This prevents false claims during evidence-based audits by strictly validating that content does not exist across the crawled site sources.
Building a complete URL inventory requires aggregating data from four sources: sitemaps, HTML navigation links, full-site crawls, and pagination loops. This multi-source approach ensures no content sections are missed during the corporate site discovery process.
Yes, the ms-crawl skill requires firecrawl to execute its comprehensive four-source discovery process. It also requires structured classification rules to ensure high-confidence data collection and accurate evidence gap flagging.
The best way to verify content presence is using a multi-source crawl that cross-references sitemaps, navigation, and full-site crawls. This generates a hierarchical site structure and an evidence manifest to accurately flag any content gaps.
Yes, the discovery process generates a structured hierarchical site structure and an evidence manifest. This structured output allows seamless integration of the classified URL inventory into downstream analysis and reporting workflows.
antgroupAccess Firecrawl API for web search, scraping, and site crawling.
kevinlinAudit website SEO health across keywords, on-page, content, technical, and competitors.
popiliadamAutomate site migration map creation and verification for URL redirections.
marchattonSearch the web, scrape content, and map site URLs via Firecrawl CLI.
