skills.rest
ruban-s
ruban-sCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

BrightData

Scrapes blocked web pages by escalating through fetch, curl, Chrome render, and Bright Data proxy tiers.

Updated Sep 3, 2026
Search Tags:#markdown#web scraping#proxy#bot detection#crawl#captcha#bright data
One-click install
npx skills add https://github.com/ruban-s/DevOS --skill brightdata-ruban-s
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: BrightData
Source: https://github.com/ruban-s/DevOS/tree/main/skills/BrightData
Command: npx skills add https://github.com/ruban-s/DevOS --skill brightdata-ruban-s

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Web pages protected by bot detection, CAPTCHAs, JavaScript rendering, or fingerprinting defeat simple fetch requests, leaving agents unable to retrieve content. This Skill escalates through four retrieval rungs — plain fetch, browser-dressed curl, real-Chrome rendering, and metered Bright Data proxy — stopping at the first rung that succeeds, so metered credits are only spent when free methods demonstrably fail. ## Core Features & Use Cases - Four-Tier Single-Page Scraping: Converts one URL to Markdown by climbing WebFetch, header-dressed curl, Interceptor real-Chrome render, then Bright Data's scrape_as_markdown, with a Cloudflare Markdown-negotiation probe ahead of the ladder. - Site Crawling in Two Gears: Light Crawl walks up to 50 pages via batched MCP scraping with link-following and depth ceilings; Full Crawl uses the Bright Data Crawl API for whole estates at $1.50 per 1K pages. - Cost and Provenance Tracking: Every result is tagged with the rung that produced it, crawls return site maps with per-page bodies, spend estimates, and error rollups. - Use Case: A user says "crawl all pages under docs.example.com/api" — the Skill runs a Light Crawl with a path-prefix filter, scrapes up to 30 pages in batches of ten, and returns a hierarchical site map plus Markdown bodies. ## Quick Start Ask the agent to scrape a blocked URL or crawl a site section, for example: "Scrape this page that keeps blocking me" or "Crawl all pages under example.com/docs and return them as Markdown."

Frequently Asked Questions about BrightData

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scrape a website that blocks bots or shows CAPTCHAs?▼

Escalate through retrieval tiers: try a plain fetch first, then curl with browser headers, then a real-Chrome render, and finally a residential proxy service like Bright Data that handles CAPTCHAs and fingerprinting. Stop at the first tier that returns readable content.

How to crawl an entire website and extract all pages as Markdown?▼

Use a two-gear approach: batched scraping with link-following for sections up to about 50 pages, or a dedicated crawl API for whole estates. Bright Data's Crawl API accepts a seed URL, crawl depth, and URL filter regex, then returns all pages for roughly $1.50 per 1K pages.

When should I use Bright Data instead of a plain web fetch?▼

Only when free methods demonstrably fail. Plain fetch handles open pages, dressed curl clears header screens, and real-Chrome rendering covers JavaScript-heavy pages. Bright Data's metered proxy is the final rung for CAPTCHAs, fingerprinting, and residential-IP requirements.

Does web scraping work on JavaScript-rendered pages?▼

Static fetch and curl cannot execute JavaScript, so app-shell pages return empty shells. A real-Chrome rendering step executes scripts and returns the rendered DOM text, which handles most script-dependent pages before any paid proxy is needed.

What are the cost limits of large-scale site crawling?▼

Light crawls cap at 50 pages per run at roughly $0.006 per page, while full crawls via the Crawl API cost $1.50 per 1K pages — about $15 for 10K pages. Confirmation is required before light sweeps past 20 pages or any full crawl.

Related Skills

Explore More Software Engineering Skills→
openaiopenai
488

build-competitive-brief

Generate a competitive brief with a comparison matrix and battlecard-style outputs.

Official
Advanced
kreuzberg-devkreuzberg-dev
9.2k

chunking

Split extracted text into sized chunks for LLM context windows and RAG ingestion.

Official
Intermediate
orthogonal-shorthogonal-sh
12

olostep

Scrape and crawl web pages into structured data via configurable endpoints.

Official
Advanced
bykoleksii-hardobykoleksii-hardo

blog-discourse

Research public discourse on a topic via WebSearch and output structured reports.

Community
Advanced