skills.rest
Briian3306
Briian3306COMMUNITY
Explore all skills by this author →
GitHub

scrape

Scrape web pages into markdown, HTML, JSON, or screenshots via the Bright Data CLI.

Updated Jan 31, 2023
Search Tags:#markdown#cli#web scraping#pagination#bright data#geo-targeting#block page
One-click install
npx skills add https://github.com/Briian3306/Transporte --skill scrape-briian3306
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: scrape
Source: https://github.com/Briian3306/Transporte/tree/main/ibarra-app/.agents/skills/scrape
Command: npx skills add https://github.com/Briian3306/Transporte --skill scrape-briian3306

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Fetching web content often fails due to geo-blocking, bot detection, and JavaScript-heavy pages, and raw HTML is hard to feed into downstream tools. This Skill retrieves clean markdown, HTML, JSON, or screenshots from one or many URLs through the Bright Data CLI, with built-in verification and block-page recovery. ## Core Features & Use Cases - Single and batch scraping: Fetch one URL as clean markdown, or process dozens of URLs with a parallelism cap using xargs. - Pagination and geo-targeting: Crawl paginated listings page by page and override the exit country to bypass regional restrictions. - Verification and escalation: Detect block pages (Cloudflare, captcha) automatically, retry with different countries, and escalate to a real browser when needed. - Use Case: You have a list of 50 blog URLs and need their content as markdown for analysis. The Skill batches the requests with capped parallelism, verifies each output is not a block page, and reports successes and failures. ## Quick Start Ask the AI to scrape a specific URL into clean markdown using the Bright Data scrape skill and save the result to a file.

Frequently Asked Questions about scrape

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scrape a web page to markdown with the Bright Data CLI?▼

Run bdata scrape with the URL and -f markdown, optionally writing to a file with -o. Verify the output is non-empty and does not contain block-page signatures like "Access Denied" or "captcha" before using it.

How to scrape a large list of URLs in parallel?▼

Use xargs with -P 4 to cap concurrency at four parallel bdata scrape processes, one URL per invocation. Write each result to a hashed filename and log failures to stderr so one bad URL does not abort the batch.

What should I do when a scrape returns a Cloudflare block page?▼

Retry the same URL with a different --country exit node such as de, jp, or gb. If all country rotations still return block pages, escalate to bdata browser for full JavaScript rendering.

When should I use data-feeds instead of scraping Amazon or LinkedIn?▼

For Amazon, LinkedIn, TikTok, Instagram, YouTube, and Reddit, hand off to the data-feeds skill instead of scraping. Those platforms have structured extractors that return clean JSON directly, while scraping loses the structure.

Does the bdata scrape --mobile or --async flag work?▼

No. In CLI v0.1.8 the --mobile flag is declared but not forwarded by the request builder, and --async is rejected by the Web Unlocker API with a validation error. Do not rely on either flag.

Related Skills

Explore More Data & Analytics Skills→
r1cA18r1cA18
4

firecrawl-map

Discover and list all URLs on a website with keyword filters and JSON output.

Community
Basic
1thirteeng31thirteeng3

firecrawl-search

Search the web and extract page content as JSON or markdown.

Community
Intermediate
wulaosijiwulaosiji
28

bright-data

Scrape social media profiles, posts, comments, and engagement metrics via the Bright Data API.

Community
Intermediate
vm0-aivm0-ai
76

browserless

Execute Puppeteer and Playwright scripts in the cloud via REST endpoints.

Official
Intermediate