skills.rest
80portisfound
80portisfoundCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

scrapling

Scrapes web pages via HTTP, stealth browser automation, and spider crawling with Scrapling.

Updated May 3, 2026
Search Tags:#data extraction#python#web scraping#scrapling#cloudflare bypass#stealth browser#spider crawling
One-click install
npx skills add https://github.com/80portisfound/vibe-learning --skill scrapling-80portisfound
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: scrapling
Source: https://github.com/80portisfound/vibe-learning/tree/main/packages/hermes/optional-skills/research/scrapling
Command: npx skills add https://github.com/80portisfound/vibe-learning --skill scrapling-80portisfound

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires scrapling.

What problem does it solve? Extracting data from websites often fails when pages require JavaScript rendering, sit behind Cloudflare or anti-bot protection, or span many linked pages. This Skill provides CLI and Python workflows for fetching, parsing, and crawling such sites with the Scrapling framework. ## Core Features & Use Cases - Three Fetching Strategies: Use Fetcher for static HTTP pages, DynamicFetcher for JS-rendered SPAs, and StealthyFetcher for Cloudflare-protected or fingerprinted sites. - Spider Crawling Framework: Build multi-page crawlers with link following, concurrent requests, multi-session routing, and pause/resume checkpoints. - Rich Element Selection: Query pages with CSS selectors, XPath, text/regex finders, similar-element detection, and DOM navigation. - Use Case: Scrape a Cloudflare-protected product listing site by running scrapling extract stealthy-fetch with --solve-cloudflare, then export results to Markdown or JSON. ## Quick Start Ask the AI to scrape a target URL with Scrapling, for example: extract all quotes from quotes.toscrape.com and save them to a JSON file.

Frequently Asked Questions about scrapling

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scrape a JavaScript-rendered page in Python?▼

Use Scrapling's DynamicFetcher to fetch JS-rendered pages with a real browser. Call DynamicFetcher.fetch with options like wait_selector, network_idle, and disable_resources, then extract data with CSS or XPath selectors.

How to bypass Cloudflare protection when web scraping?▼

Use Scrapling's StealthyFetcher with solve_cloudflare=True to pass Cloudflare Turnstile checks. Options like block_webrtc and hide_canvas reduce browser fingerprinting, though solving adds 5-15 seconds per fetch.

What is the difference between Fetcher, DynamicFetcher, and StealthyFetcher?▼

Fetcher handles static pages over plain HTTP and is fastest. DynamicFetcher runs a real browser for JS-rendered content. StealthyFetcher adds anti-fingerprinting and Cloudflare solving for protected sites.

Why does Scrapling fail with browser errors after pip install?▼

DynamicFetcher and StealthyFetcher require browser binaries that pip does not install. Run scrapling install after pip install to download them, otherwise browser-based fetching will fail.

Can I pause and resume a Scrapling spider crawl?▼

Yes, pass a crawldir path when creating the Spider to enable checkpointing. Interrupt with Ctrl+C and re-run the same script to resume crawling from the saved checkpoint.

What are the limitations of stealth web scraping with Scrapling?▼

StealthyFetcher runs a real browser, so it consumes significant resources and limits concurrency. Cloudflare solving adds latency, and users must respect robots.txt, website terms of service, and applicable scraping laws.

Related Skills

Explore More Software Engineering Skills→
ZJU-REALZJU-REAL
195

firecrawl-map

Discover and filter website URLs using the Firecrawl command-line interface.

Official
Basic
TurnaboutHeroTurnaboutHero
12

data-wizard

Automate ETL and data transformation for CSV, JSON, and database sources using Python and pandas.

Community
Advanced
ScrapeGraphAIScrapeGraphAI
45

just-scrape

Automate web data tasks via the just-scrape CLI with an SGAI API key.

Official
Advanced
mazreanmazrean

building-static-site-rss-feeds

Generate RSS 2.0, Atom 1.0, and JSON feeds for static sites.

Community
Intermediate