Skills
.Work. You
Rest

using-web-scraping

Community

Scrape web content safely.

Software Engineering#automation#data extraction#web scraping#playwright#robots.txt#duckduckgo
Authorbesoeasy
Version1.0.0
Installs0

System Documentation

What problem does it solve?

This Skill automates the process of searching the web and extracting relevant content from public web pages, while adhering to safety and ethical scraping practices.

Core Features & Use Cases

  • Privacy-Respecting Search: Utilizes DuckDuckGo for search queries.
  • Content Extraction: Extracts title, meta description, main text, and canonical URL from visited pages.
  • Safety Compliance: Checks and respects robots.txt, implements rate-limiting, and uses polite User-Agent strings.
  • Use Case: Gather information about a specific product from multiple review sites for a market analysis report.

Quick Start

Use the web scraping skill to search for "best open source AI tools" and extract the content from the top 3 results.

Dependency Matrix

Required Modules

None required

Components

scriptsreferences

💻 Claude Code Installation

Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.

Please help me install this Skill:
Name: using-web-scraping
Download link: https://github.com/besoeasy/open-skills/archive/main.zip#using-web-scraping

Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
View Source Repository

Agent Skills Search Helper

Install a tiny helper to your Agent, search and equip skill from 223,000+ vetted skills library on demand.