skills.rest
shoshoavi
shoshoaviCOMMUNITY
Explore all skills by this author →
GitHub

web-scraping-analysis

Automates compliant web scraping pipelines from legal checks through extraction, cleaning, and reporting.

Updated Apr 15, 2026
Search Tags:#data extraction#compliance#web scraping#playwright#robots.txt#beautifulsoup#etl pipeline
One-click install
npx skills add https://github.com/shoshoavi/agentic_worflows --skill web-scraping-analysis-shoshoavi
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-scraping-analysis
Source: https://github.com/shoshoavi/agentic_worflows/tree/main/cursor/skills/web-scraping-analysis
Command: npx skills add https://github.com/shoshoavi/agentic_worflows --skill web-scraping-analysis-shoshoavi

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires requests, beautifulsoup4, lxml, scrapy, playwright, pandas, httpx, and includes scripts (resource) and references (resource) components.

What problem does it solve? Manually scraping websites involves legal risk, tool-selection guesswork, fragile selectors, and messy raw data. This Skill runs the entire pipeline autonomously: it checks robots.txt and Terms of Service compliance, analyzes the target site, picks the right tool, extracts data, cleans it, and generates an analysis report. ## Core Features & Use Cases - Legal Compliance Gate: Automatically validates robots.txt, authentication requirements, PII exposure, and Terms of Service before any scraping begins, stopping immediately if a check fails. - Automatic Tool Selection: Analyzes the target site to detect static HTML, JavaScript-rendered SPAs, data tables, pagination, and hidden API endpoints, then selects pandas, BeautifulSoup, Scrapy, Playwright, or direct API calls accordingly. - End-to-End ETL Pipeline: Extracts data with rate limiting and retry logic, cleans it with pandas (deduplication, type parsing, null reporting), and produces a Markdown analysis report with statistics and data quality scores. - Use Case: Ask it to gather competitor pricing from a public e-commerce site. It verifies scraping is permitted, detects the page structure, extracts product names and prices across paginated results, cleans the dataset, and delivers a CSV plus a report highlighting price ranges and outliers. ## Quick Start Scrape the product listings from https://example.com/products and give me a cleaned dataset with a summary report.

Frequently Asked Questions about web-scraping-analysis

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scrape a website legally with Python?▼

Check robots.txt first using urllib.robotparser, verify the Terms of Service do not prohibit scraping, avoid data behind logins, and skip personal information unless you have a legal basis under GDPR or CCPA. Always identify your bot with User-Agent and From headers and rate limit requests to 2 or more seconds.

BeautifulSoup vs Scrapy vs Playwright for web scraping?▼

Use pandas.read_html for static pages with data tables, BeautifulSoup with requests for static HTML with complex layouts, Scrapy for multi-page crawls with pagination, and Playwright for JavaScript-rendered single-page applications. If the site exposes API endpoints, call them directly with httpx or requests instead of parsing HTML.

How do I check if robots.txt allows scraping a URL?▼

Run the included robots_checker.py script with the target URL to get an ALLOWED or DISALLOWED verdict, the Crawl-delay value, and any declared sitemaps. If robots.txt cannot be read, treat the URL as disallowed as a fail-safe.

Why does my scraper return empty results on some websites?▼

Empty results usually mean the page is JavaScript-rendered, so requests only retrieves an empty shell, or your CSS selectors no longer match the site structure. Run site_analyzer.py to detect SPA frameworks like React or Vue, then switch to Playwright for browser-based rendering.

What should I do when a scraper gets HTTP 429 errors?▼

HTTP 429 means the site is rate limiting you. Back off exponentially starting around 10 seconds, reduce concurrency to one request at a time, and increase your base delay to 5 or more seconds. If errors persist after several retries, stop and report partial data.

Can I scrape websites that require a login?▼

No. Data behind authentication is not public, and scraping it carries legal risk under the CFAA and likely violates the site's Terms of Service. The compliance gate stops automatically in this case and suggests using an official API or contacting the site owner instead.

Related Skills

Explore More Data & Analytics Skills→
bobby-andrisbobby-andris

data-first

Define data requirements, gather sources, and validate quality before analysis.

Community
Advanced
LuminaVaultLuminaVault

market-alert-agent

Monitor financial markets and send configurable alerts via Discord, Telegram, or Slack.

Official
Advanced
gganbukim1gganbukim1

data-scraper-agent

Scrape public web sources on a schedule and enrich items with Gemini summaries.

Community
Advanced
larsboeslarsboes

Scraping

Automate web scraping across social media and e-commerce platforms using Bright Data proxies and Apify.

Community
Advanced