web-scraper

Extract structured content, metadata, and design tokens from websites using Python.

Updated Aug 30, 2026
One-click install
npx skills add https://github.com/prasanna591/ProblemSolvingMind --skill web-scraper-prasanna591
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-scraper
Source: https://github.com/prasanna591/ProblemSolvingMind/tree/main/.opencode/skills/web-scraper
Command: npx skills add https://github.com/prasanna591/ProblemSolvingMind --skill web-scraper-prasanna591

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve? Manually copying content, structure, and design details from websites is slow and error-prone. This Skill fetches any URL and converts it into structured markdown knowledge covering metadata, heading outlines, visible text, links, and design signals, ready for analysis or reimplementation. ## Core Features & Use Cases - Multi-mode extraction: Run the bundled Python script in meta, outline, text, links, design, or full mode to pull exactly the data you need, with optional multi-page crawling. - Competitor analysis: Follow a structured framework to capture a competitor's positioning, sitemap, design tokens, and tech stack into reusable reference notes. - Site-to-code reimplementation: Turn scraped structure and design signals into Next.js 16, React 19, and Tailwind 4 components following a defined extract-model-implement-verify pipeline. - Use Case: Point the scraper at a competitor's landing page, crawl its key pages, and generate a knowledge file with its design tokens and section structure, then rebuild an adapted version as Next.js components in this project. ## Quick Start Scrape https://example.com in full mode and save the extracted knowledge as a markdown reference file.

Frequently Asked Questions about web-scraper

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scrape a website and extract its content with Python?▼

Run the bundled scrape.py script with a URL and a mode flag, such as --mode full, to extract metadata, headings, text, links, and design signals. It uses only the Python 3 standard library, so no package installation is required.

How to analyze a competitor website's design and structure?▼

Fetch the site with --mode full to capture meta tags, heading outline, and design signals like colors and fonts, then crawl key pages with --crawl 10. Record findings in the provided competitor analysis framework covering positioning, sitemap, and tech stack.

Does the scraper work on JavaScript-rendered pages?▼

No, the stdlib HTML parser only reads server-rendered HTML. If text extraction returns nothing, the page is JS-rendered and you should fall back to the webfetch tool, noting the dependency in your notes.

Can I rebuild a scraped website as Next.js components?▼

Yes, the site-to-code reference defines a pipeline: scrape the site, model its structure into data-driven components, then implement with Next.js App Router, Tailwind, and shadcn/ui. Adapt patterns rather than copying proprietary assets or copy verbatim.

What are the limitations of crawling websites with this tool?▼

Crawling is linear and guarded: it skips media files, login and cart pages, and adds delays between requests. You should keep crawl depth low, respect robots.txt, and never scrape private or login-gated content.