What problem does it solve? Before mapping keywords, auditing a site, or planning internal linking, you need the real list of URLs — and users often don't have an export. This Skill obtains that list from the actual source (robots.txt, sitemaps, or a live crawl) instead of guessing URLs. ## Core Features & Use Cases - Sitemap path (zero install): Reads robots.txt Sitemap directives, follows sitemap indexes, and parses all <loc> entries via curl/WebFetch or the deterministic Python script inventario_urls.py. - Screaming Frog CLI path: Runs headless crawls (free up to 500 URLs per crawl) to get status codes, content types, indexability, and inlink/outlink graphs for orphan detection. - Batch crawling for large sites: sf_crawl_all.py splits sitemap URLs into batches of ≤500, crawls each with --crawl-list, and merges the CSVs — covering entire sites with the free version. It also detects Shopify/WAF rate-limiting, revalidates false 4xx statuses with a real browser User-Agent, and enriches recovered URLs with on-page data. - Use Case: A user asks "build a keyword map for example.com" without providing URLs. The Skill extracts 84 URLs from the sitemap, delivers the list, and feeds it to the keyword-mapping workflow. ## Quick Start Ask the assistant to list all URLs of your domain, for example: "Get the full URL inventory of https://example.com so we can audit it."
