skills.rest
zhiyuan-zhang0206
zhiyuan-zhang0206COMMUNITY
Explore all skills by this author →
GitHub

web-sources

Fetches web pages, RSS feeds, and YouTube content into a unified S1 item schema.

1|Updated Aug 14, 2026
Search Tags:#web scraping#youtube#yt-dlp#trafilatura#content aggregation#rss#feed ingestion
One-click install
npx skills add https://github.com/zhiyuan-zhang0206/Ava --skill web-sources-zhiyuan-zhang0206
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: web-sources
Source: https://github.com/zhiyuan-zhang0206/Ava/tree/main/ava_builtins/skills/web-sources
Command: npx skills add https://github.com/zhiyuan-zhang0206/Ava --skill web-sources-zhiyuan-zhang0206

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires feedparser, trafilatura, loguru, yt-dlp, and includes references (resource) components.

What problem does it solve? Following updates across blogs, news sites, YouTube channels, and forums requires different tools per platform, producing inconsistent data that downstream digest or labeling pipelines cannot consume uniformly. ## Core Features & Use Cases - Adapter-based ingestion: Choose from youtube (yt-dlp metadata and transcripts), rss (feedparser-based feed parsing), or generic (curl + trafilatura extraction with Jina Reader fallback) adapters, all login-free. - Unified S1 schema: Every adapter mirrors raw content to $AVA_HOME/state/mirrors/ and projects it onto one shared item schema, so downstream consumers never care which adapter produced an item. - Mirror-as-cursor deduplication: An item counts as seen iff its mirror post.json exists, making re-runs idempotent with no separate state file. - Use Case: Subscribe to an AI lab's blog RSS feed and a YouTube conference playlist; run each adapter's sync to collect new entries as uniform S1 items for a daily digest. ## Quick Start Ask the agent to sync a feed or channel, for example: fetch the latest entries from this RSS feed URL and project them to S1 items using the web-sources rss adapter.

Frequently Asked Questions about web-sources

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I fetch articles from a website that has no RSS feed?▼

Use the generic adapter: run its enum command with the list page URL and a --link-pattern regex matching article links, then sync fetches each article. Extraction uses curl plus trafilatura, escalating to Jina Reader when anti-bot walls block plain requests.

How do I get YouTube video transcripts without an API key?▼

The youtube adapter uses yt-dlp to pull author-uploaded subtitles first, falling back to auto-generated captions, cleaned from VTT into plain prose. An optional whisper_fallback flag transcribes caption-less videos via OpenAI, which requires OPENAI_API_KEY and ffmpeg.

Which adapter should I use for a given source?▼

Use youtube for channels and playlists, rss for any RSS or Atom feed, and generic only when neither a platform adapter nor a feed exists. All adapters are login-free; account-bound sources need separately installed user skills.

Does the RSS adapter support fetching full article text?▼

Yes, pass --full to the rss sync command and it fetches each entry's link through the generic adapter's extraction ladder, replacing the feed summary with full-body markdown. If extraction fails, it falls back to the summary without aborting the sync.

Why does YouTube enumeration sometimes stop working?▼

yt-dlp scrapes YouTube's player surface, which breaks when YouTube changes it, producing signature errors or JS challenge failures. Upgrade with pip install -U yt-dlp and install the impersonation extra or deno as the documented mitigation.

Can these adapters bypass login walls or private content?▼

No. The Jina render rung bypasses Cloudflare and JS-render walls but not login state. Login-gated content, member-only videos, and private playlists require a logged-in browser via chrome MCP or a platform-specific adapter.

Related Skills

Explore More Software Engineering Skills→
EndUser123EndUser123

yt-is

Check tracked YouTube channels for new videos and validate channel additions.

Community
Advanced
swarm-ai-researchswarm-ai-research

RSS Digest

Aggregate fresh RSS feed items and generate concise summaries.

Official
Intermediate
math-incmath-inc
1.2k

whisper

Transcribe audio into text using Whisper multilingual speech recognition models.

Official
Advanced
mathiaschumathiaschu
92

watch

Download videos and convert them into timestamped frames and transcripts.

Community
Advanced